暂无图片
暂无图片
暂无图片
暂无图片
暂无图片

Programming Kubernetes读书笔记-Chapter3

程序员的Cookbook 2021-01-13
422

Chapter3. Basics of client-go

client-go
api
apimachinery
三个重要的仓库

client-go
api
apimachinery
是Kubernetes client中最核心的三个仓库。

1.client-go[1]仓库是用来访问kubernetes的client的接口。2.Pod、Deployment等对象则是放在api[2]的仓库中,例如Pod对象,它属于core group,对于v1版本来说,它的位置就在api/core/v1
目录下。Pod的类型定义就在types.go
文件中。这个目录下还包含了一些其他文件,部分文件都是通过代码生成器自动生成的。
3.最后一个仓库是apimachinery[3],包含了所有通用的用来构建类似Kubernetes风格API的模块。

Creating and Using a Client

    package main


    import (
    "fmt"
    "os"
    "path/filepath"
    "time"


    "k8s.io/apimachinery/pkg/util/wait"
    "k8s.io/client-go/informers"
    "k8s.io/client-go/rest"


    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/cache"
    "k8s.io/client-go/tools/clientcmd"
    )


    func main() {
    先从集群Pod中/var/run/secrets/kubernetes.io/serviceaccount获取到service account转换为rest.Config
    config, err := rest.InClusterConfig()
    if err != nil {
    获取service account失败,直接去读取kubeconfig文件
    kubeconfig := filepath.Join("~", ".kube", "config")
    或者是从环境变量中获取到kubeconfig的位置
    if envvar := os.Getenv("KUBECONFIG"); len(envvar) > 0 {
    kubeconfig = envvar
    }
    通过kubeconfig文件构建rest.Config
    config, err = clientcmd.BuildConfigFromFlags("", kubeconfig)
    if err != nil {
    fmt.Printf("The kubeconfig cannot be loaded: %v\n", err)
    os.Exit(1)
    }
    返回的config可以做一些自定义操作,比如自定义UserAgent、自定义AcceptContentTypes、超时实际、限流等
    config.UserAgent = fmt.Sprintf("Go %s", runtime.GOOS);
    config.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"
    }
    用reset.Config构建kubernetes client
    clientset, err := kubernetes.NewForConfig(config)
    读取book namespace下的名为example的Pod对象
    pod, err = “clientset.CoreV1().Pods("book").Get("example", metav1.GetOptions{})
    }
    复制

    Versioning和Compatibillity

    Kubernetes API是带有版本的,每个对象都有不同的版本,我们可以在api仓库中的apps
    目录下可以看到各个版本的对象存在,同样的,对于client-go来说,针对不同的对象也存在不同的版本的接口。我们可以在 client-go仓库下的kubernestes/typed/apps
    目录下找到对应版本对象的接口。Kubernestes和client-go
    是共用相同的api仓库的,因此client-go的版本需要和kubernetes具有兼容的版本才能发挥作用, 否则Api Server会拒绝掉client-go
    发出来的请求。如果client-go的版本比kubernertes的要新,那么当携带某些新增字段的时候,kubernetes可能会拒绝掉,也有可能会忽略掉,这个要看具体的字段的行为。kubernetes为了解决对象版本兼容问题,在实际将对象存储在etcd中时会按照一个称之为内部版本的对象存储进去,不同版本的API请求过来的时候,通过预定义的转换器进行转换来实现版本之间的兼容。

    Kubernetes Objects in Go

    Kubernetes中的资源,准确来说对应到Go中就是一个对象,资源的类型对应到yaml中的Kind字段,比如下面这个Pod资源。其yaml中的Kind字段就是Pod。在Kubernetest中会通过一个struct
    来表示这个Pod,我们还可以发现的Kubernetes中所有的资源都会有一些公共的字段,比如apiVersion、Kind、metadata、spec等。

      apiVersion: v1
      kind: Pod
      metadata:
      name: pod-demo
      namespace: default
      labels:
      app: myapp
      tier: fronted
      spec:
      containers:
      - name: myapp
      image: ikubernetes/myapp:v1
      - name: busybox
      image: busybox:latest
      command:
      - "bin/sh"
      - "-c"
      - "echo $(date) >> tmp/txt; sleep 5"
      复制

      Kubernetes中资源所对应的对象都默认实现了runtime.Object
      接口,这个接口很简单,定义如下。

        // Object interface must be supported by all API types registered with Scheme. Since objects in a scheme are
        // expected to be serialized to the wire, the interface an Object must provide to the Scheme allows
        // serializers to set the kind, version, and group the object is represented as. An Object may choose
        // to return a no-op ObjectKindAccessor in cases where it is not expected to be serialized.
        type Object interface {
        GetObjectKind() schema.ObjectKind
        DeepCopyObject() Object
        }




        // All objects that are serialized from a Scheme encode their type information. This interface is used
        // by serialization to set type information from the Scheme onto the serialized version of an object.
        // For objects that cannot be serialized or have unique requirements, this interface may be a no-op.
        type ObjectKind interface {
        // SetGroupVersionKind sets or clears the intended serialized kind of an object. Passing kind nil
        // should clear the current setting.
        SetGroupVersionKind(kind GroupVersionKind)
        // GroupVersionKind returns the stored group, version, and kind of an object, or nil if the object does
        // not expose or provide these fields.
        GroupVersionKind() GroupVersionKind
        }
        复制

        每一个对象都要满足Object接口的约束,可以看出通过这些接口可以设置和获取对象的GVK,以及进行深度拷贝。

        每一个Kubernetes对象都会嵌入一个metav1.TypeMeta struct
        。还有一个metav1.ObjectMeta

          // TypeMeta describes an individual object in an API response or request
          // with strings representing the type of the object and its API schema version.
          // Structures that are versioned or persisted should inline TypeMeta.
          //
          // +k8s:deepcopy-gen=false
          type TypeMeta struct {
          // Kind is a string value representing the REST resource this object represents.
          // Servers may infer this from the endpoint the client submits requests to.
          // Cannot be updated.
          // In CamelCase.
          // More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds
          // +optional
          Kind string `json:"kind,omitempty" protobuf:"bytes,1,opt,name=kind"`


          // APIVersion defines the versioned schema of this representation of an object.
          // Servers should convert recognized schemas to the latest internal value, and
          // may reject unrecognized values.
          // More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources
          // +optional
          APIVersion string `json:"apiVersion,omitempty" protobuf:"bytes,2,opt,name=apiVersion"`
          }


          type ObjectMeta struct {
          Name string `json:"name,omitempty"`
          Namespace string `json:"namespace,omitempty"`
          UID types.UID `json:"uid,omitempty"`
          ResourceVersion string `json:"resourceVersion,omitempty"`
          CreationTimestamp Time `json:"creationTimestamp,omitempty"`
          DeletionTimestamp *Time `json:"deletionTimestamp,omitempty"`
          Labels map[string]string `json:"labels,omitempty"`
          Annotations map[string]string `json:"annotations,omitempty"`
          ...
          }
          复制

          每一个对象都会包含一个metav1.TypeMeta struct
          字段和一个metav1.ObjectMeta
          字段 前者用来描述一个对象,比如是什么kind,什么APIVersion,后者则是一些元信息,比如label、注解、其中ResourceVersion就是用来实现乐观并发(optimistic-concurrency)等

          例如一个POD对象如下:

            // Pod is a collection of containers that can run on a host. This resource is created
            // by clients and scheduled onto hosts.
            type Pod struct {
            metav1.TypeMeta `json:",inline"`
            // Standard object's metadata.
            // More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata
            // +optional
            metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`


            // Specification of the desired behavior of the pod.
            // More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status
            // +optional
            Spec PodSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`


            // Most recently observed status of the pod.
            // This data may not be up to date.
            // Populated by the system.
            // Read-only.
            // More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status
            // +optional
            Status PodStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
            }
            复制

            每一个对象都有自己独立的Spec和Status,通常Spec表示用户的期望,Status则是期望的的结果,是Controller和Operator来负责填充。也存在一些异常情况,比如endpoints和RBAC

            请求分为长请求和短请求,对于长请求一般来说是诸如watch、一些Subresources(exec、sport-forward)等,对于短请求则会有60s的超时时间,当API server下线的时候会等待60s直到服务完这些短请求 对于长请求则直接断掉。从而实现所谓的优雅关闭。

            Client Sets

            在上面的example中,通过kubernetes.NewForConfig(config)
            创建了一个Clientset
            ,一个Clientset
            是可以用来访问多个API Group
            和资源的接口,其定义如下。

              // Clientset contains the clients for groups. Each group has exactly one
              // version included in a Clientset.
              type Clientset struct {
              *discovery.DiscoveryClient
              admissionregistrationV1 *admissionregistrationv1.AdmissionregistrationV1Client
              admissionregistrationV1beta1 *admissionregistrationv1beta1.AdmissionregistrationV1beta1Client
              appsV1 *appsv1.AppsV1Client
              appsV1beta1 *appsv1beta1.AppsV1beta1Client
                  appsV1beta2                  *appsv1beta2.AppsV1beta2Client
              ......
              }
              复制

              比如通过Clientset
              的appsV1接口,就可以访问apps组,v1 version下的所有资源,在这个组下有DaemonSet、ControllerRevision、Deployment、ReplcaSets、StatefulSet等资源,appsV1定义如下:

                // DeploymentsGetter has a method to return a DeploymentInterface.
                // A group's client should implement this interface.
                type DeploymentsGetter interface {
                Deployments(namespace string) DeploymentInterface
                }


                type AppsV1Interface interface {
                RESTClient() rest.Interface
                ControllerRevisionsGetter
                DaemonSetsGetter
                DeploymentsGetter
                ReplicaSetsGetter
                StatefulSetsGetter
                }


                // AppsV1Client is used to interact with features provided by the apps group.
                type AppsV1Client struct {
                restClient rest.Interface
                }


                // DeploymentInterface has methods to work with Deployment resources.
                type DeploymentInterface interface {
                Create(*v1.Deployment) (*v1.Deployment, error)
                Update(*v1.Deployment) (*v1.Deployment, error)
                UpdateStatus(*v1.Deployment) (*v1.Deployment, error)
                Delete(name string, options *metav1.DeleteOptions) error
                DeleteCollection(options *metav1.DeleteOptions, listOptions metav1.ListOptions) error
                Get(name string, options metav1.GetOptions) (*v1.Deployment, error)
                List(opts metav1.ListOptions) (*v1.DeploymentList, error)
                Watch(opts metav1.ListOptions) (watch.Interface, error)
                Patch(name string, pt types.PatchType, data []byte, subresources ...string) (result *v1.Deployment, err error)
                GetScale(deploymentName string, options metav1.GetOptions) (*autoscalingv1.Scale, error)
                UpdateScale(deploymentName string, scale *autoscalingv1.Scale) (*autoscalingv1.Scale, error)


                DeploymentExpansion
                }


                // Get takes name of the deployment, and returns the corresponding deployment object, and an error if there is any.
                func (c *deployments) Get(name string, options metav1.GetOptions) (result *v1.Deployment, err error) {
                result = &v1.Deployment{}
                err = c.client.Get().
                Namespace(c.ns).
                Resource("deployments").
                Name(name).
                VersionedParams(&options, scheme.ParameterCodec).
                Do().
                Into(result)
                return
                }
                复制

                AppsV1Client实现了AppsV1Interface接口,通过这个接口可以访问这个组下的所有资源,通过其定义可以看出,最终都是通过rest接口来访问的。注意观察你会发现上面的接口中都带有一个Options,比如ListOptions、DeleteOptions、GetOptions等,通过这些Options可以自定义一些过滤条件,比如ListOptions中可以指定label selector进行过滤。另外上面的接口中还有一个Watch接口,这个接口是用来监听对象的所有改变(添加、删除、更新),返回的watche.Interface其定义如下。

                  // Interface can be implemented by anything that knows how to watch and report changes.
                  type Interface interface {
                  // Stops watching. Will close the channel returned by ResultChan(). Releases
                  // any resources used by the watch.
                  Stop()


                  // Returns a chan which will receive all the events. If an error occurs
                  // or Stop() is called, this channel will be closed, in which case the
                  // watch should be completely cleaned up.
                  ResultChan() <-chan Event
                  }


                  // EventType defines the possible types of events.
                  type EventType string


                  const (
                  Added EventType = "ADDED"
                  Modified EventType = "MODIFIED"
                  Deleted EventType = "DELETED"
                  Bookmark EventType = "BOOKMARK"
                  Error EventType = "ERROR"


                  DefaultChanSize int32 = 100
                  )


                  // Event represents a single event to a watched resource.
                  // +k8s:deepcopy-gen=true
                  type Event struct {
                  Type EventType


                  // Object is:
                  // * If Type is Added or Modified: the new state of the object.
                  // * If Type is Deleted: the state of the object immediately before deletion.
                  // * If Type is Bookmark: the object (instance of a type being watched) where
                  // only ResourceVersion field is set. On successful restart of watch from a
                  // bookmark resourceVersion, client is guaranteed to not get repeat event
                  // nor miss any events.
                  // * If Type is Error: *api.Status is recommended; other types may make sense
                  // depending on context.
                  Object runtime.Object
                  }
                  复制

                  不鼓励直接使用watch接口,应该使用封装好的Informes。

                  Informers and Caching

                  Informers通过watch接口实现Cachae和增量更新。并能够很好的处理网络抖动,断网等场景。尽可能的每一种资源类型只创建一个Informers,否则会导致资源的浪费,为此可以通过InformerFactory
                  来创建Informer。他内部对于同一个资源类型只会创建一个informer实例。

                  It is very important to remember that any object passed from the listers to the event handlers is owned by the informers. If you mutate it in any way, you risk introducing hard-to-debug cache coherency issues into your application. Always do a deep copy (see “Kubernetes Objects in Go”) before changing an object.

                  我们在informers的event回调中切记不要修改对象,否则会导致很难排查的缓存一致性问题,如果要修改的话,请先深拷贝,然后修改。

                        informerFactory := informers.NewSharedInformerFactory(clientset, time.Second*30)
                    podInformer := informerFactory.Core().V1().Pods()
                    podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
                    AddFunc: func(new interface{}) { fmt.Println("Create a pod") },
                    UpdateFunc: func(old, new interface{}) { fmt.Println("Update a pod") },
                    DeleteFunc: func(obj interface{}) { fmt.Println("Delete a pod") },
                    })
                    informerFactory.Start(wait.NeverStop)
                    informerFactory.WaitForCacheSync(wait.NeverStop)
                    pod, _ := podInformer.Lister().Pods("default").Get("details-v1-5974b67c8-n7vdw")
                    复制

                    默认informer会监听所有namespace下的指定资源,也可以通过NewSharedInformerFactoryWithOptions
                    来进行过滤

                      informerFactory := informers.NewSharedInformerFactoryWithOptions(clientset, time.Second*30, informers.WithNamespace("default"))
                      复制

                      Object Owner

                      通常来说,在修改一个对象之前,我们总是会问自己,这个对象被谁拥有,或者是在哪个数据结构中? 一般来来说原则如下:

                      1.Informers and listers拥有他们返回的对象,要修改这个对象的时候,需要进行深拷贝2.Clients返回的新对象这个属于调用者3.Conversions返回的共享对象,如果调用者拥有输入的对象,那么它不拥有输出的共享对象。

                      API Machinery in Depth

                      API Machinery仓库实现了基本的Kubernetes类型系统,但是类型系统是什么呢? 类型这个术语并不在API Machinery仓库中存在。在API Machinery类型对应到的是Kinds。

                      Kinds

                      1.在kubernetes中,每一个GVK对应到一个具体的Go类型,相反,一个Go类型可以对应到多个GVK2.Kind并不会一对一和HTTP paths mapping,可能多个kind对应一个HTTP path,也有可能一个也不对应。

                      例如: admission.k8s.io/v1beta1.AdmissionReview不对应任何HTTP path 例如: meta.k8s.io/v1.Status 对应很多HTTP path

                      按照约定kind的命名按照驼峰命名,并且使用单数,而且对于CustomResourceDefinition类型的资源,其kind必须是符合DNS path label(REF 1035)

                      Resources

                      代表一类资源,这个资源是属于一个group,并且是有版本的,所以就有了GVR(GroupVersionResource),每一个GVR都会对应到一个HTTP path。通过GVR来应识别出一个Kubernetes的 REST API endpoint,例如GVR apps/v1.deloyment会映射到/apis/apps/v1/namespace/NAMESPACE/deployments 客户端就是通过这种映射关系来构建HTTP path的。

                      一个GVR是否是namespaced,或者是集群级别的,这是需要知道的,否则无法构建HTTP Path,按照约定,小写,并且是复数类型的kind,并且遵从DNS path label format 那么这个GVR对应的就是一个集群级别的。直接用kind来映射到HTTP path。例如: rbac.authorization.k8s.io/v1.clusterroles,映射到HTTP path就是apis/rbac.authorization.k8s.io/v1/clusterroles.

                      Scheme

                      虽然每一个Object都会包含TypeMeta
                      ,里面包含了Kind和APIVersion但是实际上,我们去访问的时候,并不会拿到对应的信息,这些信息都是空的,相反我们需要通过scheme来获取对象的Kind

                        func (s *Scheme) ObjectKinds(obj Object) ([]schema.GroupVersionKind, bool, error)
                        复制

                        通过Scheme来连接GVK和Golang类型,Scheme对两者做了映射,Scheme会预先对大量内置类型做映射。

                          scheme.AddKnownTypes(schema.GroupVersionKind{"", "v1", "Pod"}, &Pod{})
                          复制

                          最终一个Golang类型,通过shceme映射到GVK,然后GVK通过RESTMapper映射到GVR,最终通过GVR拼接出HTTP path。

                            // RESTMapping contains the information needed to deal with objects of a specific
                            // resource and kind in a RESTful manner.
                            type RESTMapping struct {
                            // Resource is the GroupVersionResource (location) for this endpoint
                            Resource schema.GroupVersionResource


                            // GroupVersionKind is the GroupVersionKind (data format) to submit to this endpoint
                            GroupVersionKind schema.GroupVersionKind


                            // Scope contains the information needed to deal with REST Resources that are in a resource hierarchy
                            Scope RESTScope
                            }
                            复制

                            References

                            [1]
                             client-go: https://github.com/kubernetes/client-go
                            [2]
                             api: https://github.com/kubernetes/api
                            [3]
                             apimachinery: https://github.com/kubernetes/apimachinery


                            文章转载自程序员的Cookbook,如果涉嫌侵权,请发送邮件至:contact@modb.pro进行举报,并提供相关证据,一经查实,墨天轮将立刻删除相关内容。

                            评论