如何从头创建一个KubernetesOperator

2024-03-05

10分钟阅读时长

开发技术 , 集群部署 , K8s , Docker , Kubernetes , 云原生

Kubernetes体系课 , 云原生体系课 , K8s , K8s实战 , Docker实战

无论你在Kubernetes领域是个专家还是新手，你一定知道operator或者controller。你可能常听大家说“用operator安装<XYZ>”，或者“我创建一个自定义的controller来做件啥事”。这种说法究竟意味着什么？所以我先给大家基础知识。

`Operators`对比`Controllers`

对于什么是controller什么是operator可能大家有比较多的迷惑，特别对于你不是做Kubernetes领域相关工作的，可能就更像听天书。简明扼要给出我的理解，operators是一种特别的controller。区别在于operators中针对于controller可能会包含进更多特定的负载相关的知识。

那么下个问题就出现了，什么是controller?当你创建一些kubernetes object的时候，你说“这是我期待的state”。如果我们创建一个deployment并把它的replicas（副本）数量设置成3，我们实际就是在通知一系列的controller监控这个object以确保总有3个副本数量的pods在运行，与我们deployment中声明要求的状态一致。用于维持实际运行状态与预期状态一制的这些controllers，通常叫做reconciliation，简而言之，controllers使用reconciliation这种机制来确保我们集群实际运行状态与我们的预期状态始终一致。

为了让大家能体会到operator和controller之间这种差异，今天我实战创建一个operator来加强你的理解！

`Operator-SDK`

创建operator，我们将使用operator-sdk。用这个SDK的好处在于它提供了现成的框架代码，让开发工作可以迅速聚集在核心逻辑上。它在源码层面提供了创建CRD的脚手架，在集群层面安装resources，并可以对于operator运行并测试，真是个不可多得的好工具。安装又非常简单，如果你用mac，可以直接用brew进行安装：

$ brew install operator-sdk
$ operator-sdk version
operator-sdk version: "v1.27.0", commit: "5cbdad9209332043b7c730856b6302edc8996faf", kubernetes version: "v1.25.0", go version: "go1.19.5", GOOS: "darwin", GOARCH: "amd64"

配置运行环境

kubernetes集群，如果你在本地做测试，可选方案就太多了。比如说用kind或者k3d，这种都是在基于容器内再用docker做了封装，方式比较简单，本地开发已经足够了。minikube当然也可以，我用的不多。但是如果你想本地完整部署kubernetes集群，我还是推荐使用kubeadm或者是vagrant的方式，相对比较复杂，但是细枝末节的处理，能让你对于kubernetes的运行机制了解的更清晰。

我们的项目

今天实战个啥案例呢？创建一个可以针对于custom resource definitions(CRDs)给出对应响应的operator。

我们的CRD将定义ping检测功能，通过一定次数的尝试来发送我们的hostname。简而言之，将写一个做如下事的operator:

1. 当一个类型为Ping的object type被请求以后，创建一个Kubernetes工作进程
1. 这个进程将执行ping -c <number_of_attempts> <hostname>命令

所列步骤按如下顺序执行：

1. 用Operator SDK创建项目
1. 用Operator SDK创建CRD，同时也创建operator框架代码
1. 定义Kubernetes CRD Schema
1. 定义reconciliation逻辑
1. 用Operator SDK创建和部署已定义的CRD和operator

1. 用Operator SDK创建项目

$ mkdir ping-operator && cd ping-operator
╭─guruyu@guruYudeMacBook-Pro ~/go/cnguruyu/operator/ping-operator
╰─$ operator-sdk init \
    --domain=engineeringwithalex.io \
    --repo=github.com/afoley587/52-weeks-of-projects-2023/08-kubernetes-controllers/ping-operator
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.13.0
Update dependencies:
$ go mod tidy
Next: define a resource with:
$ operator-sdk create api

2. 用`Operator SDK`创建`CRD`，同时也创建`operator`框架代码

我们会给Kubernetes系统创建新的API。为了Kubernetes系统能够有效识别，我们需要声明一些信息：

API Group
API Version
Kind

对于yaml文件如果你已经比较熟悉了，它的group是app，版本是v1，类型是StatefulSet，这也是Kuberentes会如何组织我们即将添加的API的方式：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ...
  namespace: ...

开始创建我们的API：

$ operator-sdk create api \
    --group monitors \
    --version v1beta1 \
    --kind Ping \
    --namespaced \
    --controller \
    --resource \
    --make
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1beta1/ping_types.go
controllers/ping_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
mkdir -p /Users/guruyu/go/cnguruyu/operator/ping-operator/bin
test -s /Users/guruyu/go/cnguruyu/operator/ping-operator/bin/controller-gen || GOBIN=/Users/guruyu/go/cnguruyu/operator/ping-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.10.0
/Users/guruyu/go/cnguruyu/operator/ping-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests

以上命令，我们用很多标记命令要求operator SDK为我们创建了新的API，这些标记的含义如下：

–group monitors - 把API group设置为monitors
–version v1beta1 - 把API version设置为v1beta1
–kind monitors - 创建新的ojbect类型为Ping
–namespaced - resource所属的namespace
–controller - 无提示生成controller
–resource - 无提示生成resource
–make - 文件生成完成后，执行make generate

在所在目录，用命令ls -la .看下空间发生了些啥：

alt text

operator-sdk新建了很多文件，尤其重要的是api/v1beta1/ping_types.go文件，在这里面，我们要指定CRD schema，及controllers/ping_controller.go，在这里面我们要写reconciliation逻辑。

3. 定义`Kubernetes CRD Schema`

先看api/v1beta1/ping_types.go文件，默认文件中的内容如下：

//go:build !ignore_autogenerated
// +build !ignore_autogenerated

/*
Copyright 2024.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

// Code generated by controller-gen. DO NOT EDIT.

package v1beta1

import (
	runtime "k8s.io/apimachinery/pkg/runtime"
)

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *Ping) DeepCopyInto(out *Ping) {
	*out = *in
	out.TypeMeta = in.TypeMeta
	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
	out.Spec = in.Spec
	out.Status = in.Status
}

// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Ping.
func (in *Ping) DeepCopy() *Ping {
	if in == nil {
		return nil
	}
	out := new(Ping)
	in.DeepCopyInto(out)
	return out
}

// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *Ping) DeepCopyObject() runtime.Object {
	if c := in.DeepCopy(); c != nil {
		return c
	}
	return nil
}

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *PingList) DeepCopyInto(out *PingList) {
	*out = *in
	out.TypeMeta = in.TypeMeta
	in.ListMeta.DeepCopyInto(&out.ListMeta)
	if in.Items != nil {
		in, out := &in.Items, &out.Items
		*out = make([]Ping, len(*in))
		for i := range *in {
			(*in)[i].DeepCopyInto(&(*out)[i])
		}
	}
}

// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PingList.
func (in *PingList) DeepCopy() *PingList {
	if in == nil {
		return nil
	}
	out := new(PingList)
	in.DeepCopyInto(out)
	return out
}

// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *PingList) DeepCopyObject() runtime.Object {
	if c := in.DeepCopy(); c != nil {
		return c
	}
	return nil
}

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *PingSpec) DeepCopyInto(out *PingSpec) {
	*out = *in
}

// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PingSpec.
func (in *PingSpec) DeepCopy() *PingSpec {
	if in == nil {
		return nil
	}
	out := new(PingSpec)
	in.DeepCopyInto(out)
	return out
}

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *PingStatus) DeepCopyInto(out *PingStatus) {
	*out = *in
}

// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PingStatus.
func (in *PingStatus) DeepCopy() *PingStatus {
	if in == nil {
		return nil
	}
	out := new(PingStatus)
	in.DeepCopyInto(out)
	return out
}

这是新resources的默认设置，基本框架有了，也够用了！为了讲解清晰，你能看得懂，我们就只更新PingSpec部分。

从更高维来审视一番，PingSpec,PingStatus,Ping都是什么？PingSpec的作用是指定何时请求Ping Object，让我们看个Pod的例子：

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    - containerPort: 80

我们看pod的spec部分，containers这个声明部分，也属于一种schema。我们无法在CRD中嵌套任意objects，我们的设计逻辑是，针对特定hostname发ping请求，以及ping应该执行的尝试次数。我们按如下接着改改：

apiVersion: zuisishu.com/v1beta1
kind: Ping
metadata:
  labels:
    app.kubernetes.io/name: ping
    app.kubernetes.io/instance: ping-sample
    app.kubernetes.io/part-of: ping-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: ping-operator
  name: ping-sample
spec:
  hostname: "www.bing.com"
  attempts: 1

为实现我们前面定义的逻辑，我们需要按如下方式，更新默认生成的类型文件：

type PingSpec struct{
  Hostname string `json:"hostname,omitempty"`
  Attempts string `json:"attempts,omitempty"`
}
type PingStatus struct{

}

type Ping struct{
  metav1.TypeMeta `json:",inline"`
  metav1.ObjectMeta `json:"metadata,omitempty"`
  Spec PingSpec `json:"spec,inline"`
  Status PingStatus `json:"status,inline"`
}

这里我们加了个string类型的hostname和一个integer类型的attempts。

PingStatus将上报object的状态，实际它只是一个标注属性，实际的更新逻辑不在这里，但是它包含着ping行为的一些数据，比如：

ping成功了吗？
任务job完成了吗？
等等最终，Ping对象揉合了很多元素。它组合了基本类型metadata(比如group,Api Version,Kind等)，object metadata(比如name,namespace等)，PingSpec及PingStatus。

此时，我们可以执行make manifests命令：

$ make manifests
/Users/guruyu/go/cnguruyu/operator/ping-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

此刻，我们在config/crd/bases中应该能看到如下详情：

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.10.0
  creationTimestamp: null
  name: pings.zuisishu.com
spec:
  group: zuisishu.com
  names:
    kind: Ping
    listKind: PingList
    plural: pings
    singular: ping
  scope: Namespaced
  versions:
  - name: v1beta1
    schema:
      openAPIV3Schema:
        description: Ping is the Schema for the pings API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: PingSpec defines the desired state of Ping
            properties:
              foo:
                description: Foo is an example field of Ping. Edit ping_types.go to
                  remove/update
                type: string
            type: object
          status:
            description: PingStatus defines the observed state of Ping
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}

4. 定义`reconciliation`逻辑

定义这些搞完了，该搞reconciliation逻辑了，毕竟带着定义字段的请求过来，operator得有处理逻辑。比如，如果我apply了带Ping类型的resource的yaml文件，operator要怎么做响应？接着往下看。

文件controllers/ping_controller.go里的代码是啥作用，此处就不得不感叹operator-sdk生成的代码骨架，此时给我们的清晰思路了：

/*
Copyright 2024.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controllers

import (
	"context"

	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"

	monitorsv1beta1 "github.com/afoley587/52-weeks-of-projects-2023/08-kubernetes-controllers/ping-operator/api/v1beta1"
)

// PingReconciler reconciles a Ping object
type PingReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=zuisishu.com,resources=pings,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=zuisishu.com,resources=pings/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=zuisishu.com,resources=pings/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Ping object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.13.0/pkg/reconcile
func (r *PingReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)

	// TODO(user): your logic here

	return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *PingReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&monitorsv1beta1.Ping{}).
		Complete(r)
}

但这生成的代码，与我们的业务逻辑毫无关系，所以此处就必须重写了。当Ping类型的resource做create、update、delete的时候就要调用Reconcile方法。一组用户数据，比如name、namespace都打包进ctrl.Request中传给请求方法，我们的目的就是创建出的controller能够创建Kubernetes job来处理这类型的请求，我们的功能就变成如下：

func (r *PingReconciler) Reconcile(ctx context.Context,req ctrl.Request) (ctrl.Request,error){
  _ = log.FromContext(ctx)

  var ping monitorsv1beta1.Ping

  if err := r.Get(ctx,req.NamespacedName,&ping); err != nil{
    log.FromContext(ctx).Error(err,"Unable to fetch Ping")
    return ctrl.Request{},client.IgnoreNotFound(err)
  }

  job,err := r.BuildJob(ping)
  if err != nil {
		log.FromContext(ctx).Error(err, "Unable to get job definition")
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	if err := r.Create(ctx, &job); err != nil {
		log.FromContext(ctx).Error(err, "Unable to create job")
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}
	return ctrl.Result{}, nil
}

这里，我们首先从Kubernetes API获取Ping object。举例，比如说如果resource type不存在（CRD没有apply到kubernetes系统中），会导致报错。这种情况下，就无法处理相应请求了。接下来，我们调用自定义BuildJob函数。从更高的视角来看，BuildJob会创建kubernetes job的定义，这个定义会调用Reconcile功能，并对job进行应用和部署。如果job部署失败，会响应报错。

BuildJob是什么？如下是对它的定义：

func (r *PingReconciler) BuildJob(ping monitorsv1beta1.Ping) (batchv1.Job, error) {
  attempts := "-c" + strconv.Itoa(ping.Spec.Attempts)
  host := ping.Spec.Hostname
  j := batchv1.Job{
    TypeMeta: metav1.TypeMeta{
      APIVersion: batchv1.SchemaGroupVersion.String(),
      Kind: "job",
    },
    ObjectMeta: metav1.ObjectMeta{
      Name: ping.Name + "-job",
      Namespace: ping.Namespace,
    },
    Spec: batchv1.JobSpec{
      Template: corev1.PodTemplateSpec{
        Spec: corev1.PodSpec{
          RestartPolicy: corev1.RestartPolicyNever,
          Containers: []corev1.Container{
            Name: "ping",
            Image: "bash",
            Command: []string{"/bin/ping"},
            Args: []string{attempts,host},
          }
        }
      }
    }
  }
}

首先我们的Reconcile函数，从从kubernetes API的Ping resource拉取了规范的参数，然后我们创建和job。把它命名成<name of ping>-job，同时给它加一个新container，这个container将使用/bin/ping命令加-c <number of attempts> <hostname>指定的参数来运行bash型docker镜像。当Reconcile函数执行创建时，预期的新job就创建出来。这个job的作用就是创建出一个可以发出ping命令的pod。

注意：我们没有实现任何的fanalized。如果我们再次部署相同的Ping resource，会出现重复resources的报错。

5. 用`Operator SDK`创建和部署已定义的`CRD`和`operator`

关键部分都搞完了，现在用operator-sdk提供的Makefile来创建和部署我们的manifests和operator

$ make manifests                     
$ make install
$ make run

在另外的console窗口，apply一下这个kubernetes manifest，让我们看下operator是如何响应的：

kubectl apply -f - <<EOF
apiVersion: zuisishu.com/v1beta1
kind: Ping
metadata:
  labels:
    app.kubernetes.io/name: ping
    app.kubernetes.io/instance: ping-sample
    app.kubernetes.io/part-of: ping-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: ping-operator
  name: ping-sample
spec:
  hostname: "www.google.com"
  attempts: 1
EOF
ping.zuisishu.com/ping-sample created

此刻我们再检查下object即可。

如何从头创建一个KubernetesOperator

`Operators`对比`Controllers`

`Operator-SDK`

配置运行环境

我们的项目

1. 用Operator SDK创建项目

2. 用`Operator SDK`创建`CRD`，同时也创建`operator`框架代码

3. 定义`Kubernetes CRD Schema`

4. 定义`reconciliation`逻辑

5. 用`Operator SDK`创建和部署已定义的`CRD`和`operator`

Aisen

系列文章

本页内容

相关

如何从头创建一个KubernetesOperator

Operators对比Controllers

Operator-SDK

配置运行环境

我们的项目

1. 用Operator SDK创建项目

2. 用Operator SDK创建CRD，同时也创建operator框架代码

3. 定义Kubernetes CRD Schema

4. 定义reconciliation逻辑

5. 用Operator SDK创建和部署已定义的CRD和operator

Aisen

系列文章

本页内容

相关

`Operators`对比`Controllers`

`Operator-SDK`

2. 用`Operator SDK`创建`CRD`，同时也创建`operator`框架代码

3. 定义`Kubernetes CRD Schema`

4. 定义`reconciliation`逻辑

5. 用`Operator SDK`创建和部署已定义的`CRD`和`operator`