kubernetes-operator开发教程（基于kubebuilder脚手架）

1、Operator介绍

Operator是什么？
Kubernetes Operator是一个自定义控制器，用于通过自动化操作来管理复杂应用或服务。

实现原理是什么？
Kubernetes Operator的实现原理基于自定义控制器（Controller）和自定义资源定义（CRD）。

k8s的文档中本身没有operator这个词，operator实质是指：用户注册自己自定义的CRD，然后创建对应的资源实例（称为Custom Resource，简称CR），而后通过自己编写Controller来不断地检测当前k8s中所定义的CR的状态，如果状态和预期不一致，则调整。Controller具体做的事就是通过调用k8s api server的客户端，根据比较预期的状态和实际的状态，来对相应的资源进行 增删改查。

2、kubebuilder介绍

Kubebuilder 是一个强大的工具，使得基于 Kubernetes 构建 Operator 变得更加简单和高效。它提供了一套规范化的开发流程和模板代码，帮助开发人员快速上手并构建功能丰富的 Kubernetes Operator

具体使用教程可查看文档：https://xuejipeng.github.io/kubebuilder-doc-cn

下载教程（LINUX环境）

wget https://github.com/kubernetes-sigs/kubebuilder/releases/latest/download/kubebuilder_linux_amd64mv kubebuilder_linux_amd64 kubebuilder 
chmod +x kubebuilder
mv kubebuilder /usr/local/bin/验证：
kubebuilder version

3、demo开发

实现一个可以根据指定字段管理deployment、service的operator

3.1 开发环境

go 1.21.0
kubebuilder 3.11.1
k8s 1.19.3

兼容性这里踩了很多坑，kubebuilder、go版本使用较新版本即可。基本能够向下兼容

同时需要保证当前开发环境能够通过kubectl访问k8s集群

3.2 kubebuilder创建operator开发环境

mkdir myoperator && cd myoperatorkubebuilder init --domain timmy.com#创建API 
# GroupVersionKind（GVK） 就是 K8s 资源的坐标，也是唯一标识
kubebuilder create api --group timmy --version v1 --kind MyApp

上述操作执行完后会得到这样一个项目

我们主要修改的就是 CRD的 结构体文件 （第一个），还有就是 controller 的 调楷 方法。

3.3 定义一个CRD

myapp_types.go

package v1import (metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"appsv1 "k8s.io/api/apps/v1"
)// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.// MyAppSpec defines the desired state of MyApp
type MyAppSpec struct {// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster// Important: Run "make" to regenerate code after modifying this file// Foo is an example field of MyApp. Edit myapp_types.go to remove/update//Foo string `json:"foo,omitempty"`Image         string `json:"image"`ServicePort   int32  `json:"servicePort"`ContainerPort int32  `json:"containerPort"`Replicas      *int32  `json:"replicas"`
}// MyAppStatus defines the observed state of MyApp
type MyAppStatus struct {// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster// Important: Run "make" to regenerate code after modifying this fileappsv1.DeploymentStatus `json:",inline"`
}//+kubebuilder:object:root=true
//+kubebuilder:subresource:status// MyApp is the Schema for the myapps API
type MyApp struct {metav1.TypeMeta   `json:",inline"`metav1.ObjectMeta `json:"metadata,omitempty"`Spec   MyAppSpec   `json:"spec,omitempty"`Status MyAppStatus `json:"status,omitempty"`
}//+kubebuilder:object:root=true// MyAppList contains a list of MyApp
type MyAppList struct {metav1.TypeMeta `json:",inline"`metav1.ListMeta `json:"metadata,omitempty"`Items           []MyApp `json:"items"`
}func init() {SchemeBuilder.Register(&MyApp{}, &MyAppList{})
}

生成代码：
make manifests

安装CRD：
make install

部署CR：
kubectl apply -f config/samples/app_v1_myapp.yaml
kubectl get myapp

3.4 controller开发

一般只需要修改调楷方法Reconcile()就行

实现逻辑：

• 如果 AppService 实例不存在，则根据 AppServiceSpec 创建
  • 创建 Deployment 资源
  • 创建 Service 资源
• 如果 AppService 实例存在，则将 Annotations 中记录的 Spec 值与当前的 Spec 比较
  • 如果前后的 Spec 不同
    • 更新 Deployment 资源
    • 更新 Service 资源
  • 如果前后的 Spec 相同，则无需额外的操作
• 使用 Annotations 记录当前 Spec 的值（Annotations是资源对象中的注解，此处用来记录当前的资源对象spec，后续用来做前后对比）

myapp_controller.go

/*
Copyright 2023.Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/package controllerimport ("context""fmt""github.com/go-logr/logr"appsv1 "k8s.io/api/apps/v1"corev1 "k8s.io/api/core/v1""k8s.io/apimachinery/pkg/api/errors"metav1 "k8s.io/apimachinery/pkg/apis/meta/v1""k8s.io/apimachinery/pkg/runtime""k8s.io/apimachinery/pkg/runtime/schema""k8s.io/apimachinery/pkg/util/intstr""k8s.io/apimachinery/pkg/util/json""reflect"ctrl "sigs.k8s.io/controller-runtime""sigs.k8s.io/controller-runtime/pkg/client""sigs.k8s.io/controller-runtime/pkg/reconcile"v1 "myoperator/api/v1"
)// MyAppReconciler reconciles a MyApp object
type MyAppReconciler struct {client.ClientScheme *runtime.SchemeLog    logr.Logger
}//+kubebuilder:rbac:groups=app.timmy,resources=myapps,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=app.timmy,resources=myapps/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=app.timmy,resources=myapps/finalizers,verbs=update//+kubebuilder:rbac:groups="apps",resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups="apps",resources=deployments/status,verbs=get
//+kubebuilder:rbac:groups="",resources=services,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups="",resources=services/status,verbs=get// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the MyApp object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.15.0/pkg/reconcile
func (r *MyAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {//_ = log.FromContext(ctx)//用于在日志记录器中添加键值对的上下文信息。在这个例子中，"myapp" 是键，req.NamespacedName 是值。req.NamespacedName的值是一个 "命名空间/资源对象名称"的组合logger := r.Log.WithValues("myapp", req.NamespacedName)logger.Info("Reconciling myapp")//判断MyApp对象是否存在instance := &v1.MyApp{}if err := r.Client.Get(ctx, req.NamespacedName, instance); err != nil {if errors.IsNotFound(err) {return reconcile.Result{}, nil}return reconcile.Result{}, err}//fmt.Println(instance)logger.Info("app kind: " + instance.Kind + ", app name: " + instance.Name)if instance.DeletionTimestamp != nil {return reconcile.Result{}, nil}//deployment := &appsv1.Deployment{}if err := r.Client.Get(ctx, req.NamespacedName, deployment); err != nil {if !errors.IsNotFound(err) {return ctrl.Result{}, err}// 1. 不存在，则创建// 1-1. 创建 Deploymentdeployment = NewDeployment(instance)if err := r.Client.Create(ctx, deployment); err != nil {return ctrl.Result{}, err}// 1-2. 创建 Servicesvc := NewService(instance)if err := r.Client.Create(ctx, svc); err != nil {return ctrl.Result{}, err}} else {oldSpec := &v1.MyAppSpec{}if err := json.Unmarshal([]byte(instance.Annotations["spec"]), oldSpec); err != nil {return ctrl.Result{}, err}fmt.Println(*oldSpec)fmt.Println(instance.Spec)// 2. 对比更新if !reflect.DeepEqual(instance.Spec, *oldSpec) {// 2-1. 更新 Deployment 资源newDeployment := NewDeployment(instance)currDeployment := &appsv1.Deployment{}if err := r.Client.Get(ctx, req.NamespacedName, currDeployment); err != nil {return ctrl.Result{}, err}currDeployment.Spec = newDeployment.Specif err := r.Client.Update(ctx, currDeployment); err != nil {return ctrl.Result{}, err}// 2-2. 更新 Service 资源newService := NewService(instance)currService := &corev1.Service{}if err := r.Client.Get(ctx, req.NamespacedName, currService); err != nil {return ctrl.Result{}, err}currIP := currService.Spec.ClusterIPcurrService.Spec = newService.SpeccurrService.Spec.ClusterIP = currIPif err := r.Client.Update(ctx, currService); err != nil {return ctrl.Result{}, err}}}// 3. 关联 Annotationsdata, _ := json.Marshal(instance.Spec)if instance.Annotations != nil {instance.Annotations["spec"] = string(data)} else {instance.Annotations = map[string]string{"spec": string(data)}}if err := r.Client.Update(ctx, instance); err != nil {return ctrl.Result{}, err}return ctrl.Result{}, nil
}func NewDeployment(app *v1.MyApp) *appsv1.Deployment {labels := map[string]string{"app": app.Name}selector := &metav1.LabelSelector{MatchLabels: labels}return &appsv1.Deployment{TypeMeta: metav1.TypeMeta{APIVersion: "apps/v1",Kind:       "Deployment",},ObjectMeta: metav1.ObjectMeta{Name:      app.Name,Namespace: app.Namespace,OwnerReferences: []metav1.OwnerReference{*metav1.NewControllerRef(app, schema.GroupVersionKind{Group:   v1.GroupVersion.Group,Version: v1.GroupVersion.Version,Kind:    app.Kind,}),},},Spec: appsv1.DeploymentSpec{Replicas: app.Spec.Replicas,Selector: selector,Template: corev1.PodTemplateSpec{ObjectMeta: metav1.ObjectMeta{Labels: labels},Spec: corev1.PodSpec{Containers: []corev1.Container{{Name:            app.Name,Image:           app.Spec.Image,ImagePullPolicy: corev1.PullIfNotPresent,Ports: []corev1.ContainerPort{{Name:          "http",ContainerPort: app.Spec.ContainerPort,},},},},},},},}
}func NewService(app *v1.MyApp) *corev1.Service {return &corev1.Service{TypeMeta: metav1.TypeMeta{Kind:       "Service",APIVersion: "v1",},ObjectMeta: metav1.ObjectMeta{Name:      app.Name,Namespace: app.Namespace,OwnerReferences: []metav1.OwnerReference{*metav1.NewControllerRef(app, schema.GroupVersionKind{Group:   v1.GroupVersion.Group,Version: v1.GroupVersion.Version,Kind:    app.Kind,}),},},Spec: corev1.ServiceSpec{Type: corev1.ServiceTypeNodePort,Ports: []corev1.ServicePort{{Protocol:   corev1.ProtocolTCP,Port:       app.Spec.ServicePort,TargetPort: intstr.FromInt(int(app.Spec.ContainerPort)),},},Selector: map[string]string{"app": app.Name,},},}
}// SetupWithManager sets up the controller with the Manager.
func (r *MyAppReconciler) SetupWithManager(mgr ctrl.Manager) error {return ctrl.NewControllerManagedBy(mgr).For(&v1.MyApp{}).Complete(r)
}

测试：

1、运行控制器：
make run

2、创建一个对象
kubectl apply -f test.yml

apiVersion: app.timmy/v1
kind: MyApp
metadata:labels:app.kubernetes.io/name: myappapp.kubernetes.io/instance: myapp-samplename: myapp-sample
spec:image: nginxservicePort: 80containerPort: 80replicas: 2

3、查看deploy、svc是否创建

3.5 部署上线

1、部署kustomize
make kustomize

2、添加权限（在controller中添加注释，部署上线是会根据这些生成权限对象）

//+kubebuilder:rbac:groups="apps",resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups="apps",resources=deployments/status,verbs=get
//+kubebuilder:rbac:groups="",resources=services,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups="",resources=services/status,verbs=get

3、构建镜像
make docker-build IMG=myapp-operator:v1

注意，这是根据Dockerfile来构建的，如果无法拉取镜像，可以选择切换国内源，或者代码。go依赖下载也是如此

4、部署上线
make deploy IMG=myapp-operator:v1

本文参考：
https://github.com/schwarzeni/kubebuilder-appservice/tree/master