二进制安装k8s集群(16)-安装nginx-ingress-controller

TA码字 2019-11-05

203

在上一篇文章里我们主要介绍安装k8s集群内的基础服务kube-dashboard，这里我们继续介绍安装k8s集群内基础服务nginx-ingress，这个基础服务也创建在kube-system namesapce里，是以deployment的方式运行。当然 daemonset也是可以的，这里没有硬性要求。image镜像从我们的private repo pull(以前文章里介绍过harbor private repo的创建，以及镜像的push和pull)。当然原始image来源于官方的quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.25.1，不过要下载它需要科学上网或者搭个梯子。另外对于ingress方案，一般有nginx-ingress，traefik ingress(traefik2.0也已经问世了，都是可以选择的)，haproxy ingress等，实际情况用哪种请根据团队和实际的需求来选择。

创建配置文件目录：

由于nginx-ingress-controller是以deployment的方式部署在k8s集群里的，一般都会有yaml部署文件，目前都放在此目录里。

mkdir -p opt/application/k8s/nginx-ingress
cd /opt/application/k8s/nginx-ingress

创建nginx-ingress的service-account：

nginx-ingress-controlle需要访问kube-apiserver来得到集群中的对象资源，从而展示在UI上。对于资源访问，k8s有自己的策略，这里给nginx-ingress-controlle创建单独的service-account，cluster-role-binding。目前就不详细展开了，有兴趣的同学可以看一下k8s的RBAC访问策略。

cat > opt/application/k8s/nginx-ingress/nginx-ingress-service-account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: serviceaccount-nginx-ingress
  namespace: kube-system
imagePullSecrets:
- name: container-registry
EOF


kubectl create -f opt/application/k8s/nginx-ingress/nginx-ingress-service-account.yaml 
kubectl describe serviceaccount serviceaccount-nginx-ingress -n kube-system

创建nginx-ingress的配置configmap：

nginx-ingress-controller也有自己的配置，我们把它的配置创建在k8s configmap资源里，然后在容器里挂载这个configmap的数据，从而提供给nginx-ingress-controller配置。当然配置项比较多，这里就不逐一介绍，有兴趣的同学请参考nginx-ingress的配置文档。

https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/cli-arguments.md

cat > opt/application/k8s/nginx-ingress/nginx-ingress-nginx-config-map.yaml <<EOF
kind: ConfigMap
apiVersion: v1
metadata:
  name: configmap-nginx-configuration
  namespace: kube-system
EOF


cat > opt/application/k8s/nginx-ingress/nginx-ingress-tcp-service-config-map.yaml <<EOF
kind: ConfigMap
apiVersion: v1
metadata:
  name: configmap-tcp-services
  namespace: kube-system
EOF


cat > opt/application/k8s/nginx-ingress/nginx-ingress-udp-service-config-map.yaml <<EOF
kind: ConfigMap
apiVersion: v1
metadata:
  name: configmap-udp-services
  namespace: kube-system
EOF


kubectl create -f opt/application/k8s/nginx-ingress/nginx-ingress-nginx-config-map.yaml
kubectl create -f opt/application/k8s/nginx-ingress/nginx-ingress-tcp-service-config-map.yaml
kubectl create -f opt/application/k8s/nginx-ingress/nginx-ingress-udp-service-config-map.yaml
kubectl describe configmap configmap-nginx-configuration -n kube-system
kubectl describe configmap configmap-tcp-services -n kube-system
kubectl describe configmap configmap-udp-services -n kube-system

创建nginx-ingress的cluster-role-binding：

这里为了方便，我们绑定k8s集群内置的cluster-admin账号。注意账户cluster-admin的权限比较大(从名字里就可以看出来)，对于实际应用请根据自身需求看是否创建cluster-role或者role来绑定给service-account。

cat > /opt/application/k8s/nginx-ingress/nginx-ingress-cluster-role-binding.yaml <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-role-binding-nginx-ingress
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: serviceaccount-nginx-ingress
  namespace: kube-system
EOF


kubectl create -f opt/application/k8s/nginx-ingress/nginx-ingress-cluster-role-binding.yaml
kubectl describe clusterrolebinding cluster-role-binding-nginx-ingress -n kube-system

创建nginx-ingress的证书secret：

nginx-ingress-controller负责向外暴露集群内部的service，我们这里开启ssl，给nginx-ingress-controller配置默认ssl证书(在创建ingress rule的时候同样也可以指定ssl证书，这里是在没有指定ingress rule ssl的情况下默认ssl配置)。这里的证书以secret的形式存储在k8s里，然后挂载给nginx-ingress-controller容器作为配置使用。当然请提前制作好相关证书，可以参考以前文章里制作docker server证书。

kubectl create secret tls nginx-ingress-cert-secret --namespace=kube-system \
--cert=/opt/sw/cert/k8s-ingress/tls.crt \
--key=/opt/sw/cert/k8s-ingress/tls.key 


kubectl describe secret nginx-ingress-cert-secret -n kube-system

创建nginx-ingress的kubeconfig文件secret：

nginx-ingress-controller需要访问kube-apiserver来得到集群中的对象资源，用以向外暴露服务。我们kube-apiserver的访问需要kubeconfig文件。这里我们提前制作好(可以参考以前文章里安装kubectl的时候制作kubeconfig文件)，然后以secret的形式存储在k8s里，挂载给nginx-ingress-controller容器作为配置使用。

kubectl config --kubeconfig /etc/kubernetes/kubeconfig/config-contains-cert set-cluster k8s-cluster-one --server=https://172.20.11.41:6443 --certificate-authority=/etc/kubernetes/kubeconfig/cert/ca.crt --embed-certs=true
kubectl config --kubeconfig /etc/kubernetes/kubeconfig/config-contains-cert set-credentials k8s-access-user-with-cert --client-certificate=/etc/kubernetes/kubeconfig/cert/k8sapiserver-client.crt --client-key=/etc/kubernetes/kubeconfig/cert/k8sapiserver-client.key --embed-certs=true
kubectl config --kubeconfig /etc/kubernetes/kubeconfig/config-contains-cert set-context context-one --cluster=k8s-cluster-one --user=k8s-access-user-with-cert
kubectl config --kubeconfig /etc/kubernetes/kubeconfig/config-contains-cert use-context context-one


kubectl create secret generic nginx-ingress-apiserver-access-secret --namespace=kube-system \
--from-file=kubeconfig=/etc/kubernetes/kubeconfig/config-contains-cert


kubectl describe secret nginx-ingress-apiserver-access-secret -n kube-system

创建nginx-ingress的deployment：

cat > /opt/application/k8s/nginx-ingress/nginx-ingress-deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-nginx-ingress
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: nginx-ingress
  template:
    metadata:
      labels:
        k8s-app: nginx-ingress
    spec:
      serviceAccountName: serviceaccount-nginx-ingress
      containers:
        - name: nginx-ingress
          image: 172.20.11.41:1034/infra/nginx-ingress-controller:0.25.1
          imagePullPolicy: IfNotPresent
          args:
            - /nginx-ingress-controller
            - --configmap=kube-system/configmap-nginx-configuration
            - --tcp-services-configmap=kube-system/configmap-tcp-services
            - --udp-services-configmap=kube-system/configmap-udp-services
            - --annotations-prefix=nginx.ingress.kubernetes.io
            - --http-port=80
            - --https-port=443
            - --healthz-port=10254
            - --health-check-path=/healthz
            - --health-check-timeout=15
            - --alsologtostderr
            - --default-ssl-certificate=kube-system/nginx-ingress-cert-secret
            - --kubeconfig=/config/kubeconfig
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          volumeMounts:
          - name: kube-apiserver-access-config
            mountPath: /config
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 60
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 15
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 15
      volumes:
      - name: kube-apiserver-access-config
        secret:
          secretName: nginx-ingress-apiserver-access-secret
EOF


kubectl create -f /opt/application/k8s/nginx-ingress/nginx-ingress-deployment.yaml
kubectl describe deployment deployment-nginx-ingress -n kube-system

创建nginx-ingress的service：

这里把service定义成node-port类型，以node port的方式提供给外部访问。node port定义为8443(https)和8080(http)，同时也请打开8443/8080的防火墙端口(可以参考以前文章如何打开某个端口)。

cat > /opt/application/k8s/nginx-ingress/nginx-ingress-service.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
  name: service-nginx-ingress
  namespace: kube-system
spec:
  selector:
    k8s-app: nginx-ingress
  type: NodePort
  ports:
  - name: nginx-ingress-https
    nodePort: 8443
    port: 443
    targetPort: 443
    protocol: TCP
  - name: nginx-ingress-http
    nodePort: 8080
    port: 80
    targetPort: 80
    protocol: TCP
EOF


kubectl create -f  /opt/application/k8s/nginx-ingress/nginx-ingress-service.yaml
kubectl describe service service-nginx-ingress -n kube-system

查看nginx-ingress在集群中的pod：

kubectl get pods -n kube-system|grep nginx

目前先写到这里，下一篇文章里我们继续介绍在k8s集群调试工具kubectl-debug。

数据库

文章转载自TA码字，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

二进制安装k8s集群(16)-安装nginx-ingress-controller

评论