1.安装说明

本文章将演示CentOS 7二进制方式安装高可用k8s 1.17+，相对于其他版本，二进制安装方式并无太大区别，只需要区分每个组件版本的对应关系即可。

生产环境中，建议使用小版本大于5的Kubernetes版本，比如1.19.5以后的才可用于生产环境。

2.基本环境配置

表1-1  高可用Kubernetes集群规划

软件版本信息和Pod、Service网段规划：

注意：

集群安装时会涉及到三个网段：

宿主机网段：就是安装k8s的服务器

Pod网段：k8s Pod的网段，相当于容器的IP

Service网段：k8s service网段，service用于集群容器通信。

service网段会设置为10.96.0.0/12

Pod网段会设置成192.168.0.0/12

宿主机网段可能是172.31.0.0/21

需要注意的是这三个网段不能有任何交叉。

比如如果宿主机的IP是10.105.0.x

那么service网段就不能是10.96.0.0/12，因为10.96.0.0/12网段可用IP是：

10.96.0.1 ~ 10.111.255.255

所以10.105是在这个范围之内的，属于网络交叉，此时service网段需要更换，

可以更改为192.168.0.0/16网段（注意如果service网段是192.168开头的子网掩码最好不要是12，最好为16，因为子网掩码是12他的起始IP为192.160.0.1 不是192.168.0.1）。

同样的道理，技术别的网段也不能重复。可以通过http://tools.jb51.net/aideddesign/ip_net_calc/计算：

所以一般的推荐是，直接第一个开头的就不要重复，比如你的宿主机是192开头的，那么你的service可以是10.96.0.0/12.

如果你的宿主机是10开头的，就直接把service的网段改成192.168.0.0/16

如果你的宿主机是172开头的，就直接把pod网段改成192.168.0.0/12

注意搭配，均为10网段、172网段、192网段的搭配，第一个开头数字不一样就免去了网段冲突的可能性，也可以减去计算的步骤。

主机信息，服务器IP地址不能设置成dhcp，要配置成静态IP。

VIP（虚拟IP）不要和公司内网IP重复，首先去ping一下，不通才可用。VIP需要和主机在同一个局域网内！公有云的话，VIP为公有云的负载均衡的IP，比如阿里云的SLB地址，腾讯云的ELB地址，注意公有云的负载均衡都是内网的负载均衡。

各节点设置主机名：

hostnamectl set-hostname k8s-master01.example.local
hostnamectl set-hostname k8s-master02.example.local
hostnamectl set-hostname k8s-master03.example.local
hostnamectl set-hostname k8s-ha01.example.local
hostnamectl set-hostname k8s-ha02.example.local
hostnamectl set-hostname k8s-harbot01.example.local
hostnamectl set-hostname k8s-harbor02.example.local
hostnamectl set-hostname k8s-etcd01.example.local
hostnamectl set-hostname k8s-etcd02.example.local
hostnamectl set-hostname k8s-etcd03.example.local
hostnamectl set-hostname k8s-node01.example.local
hostnamectl set-hostname k8s-node02.example.local
hostnamectl set-hostname k8s-node03.example.local
复制

各节点设置ip地址格式如下：

#CentOS
[root@k8s-master01 ~]# cat etc/sysconfig/network-scripts/ifcfg-eth0 
DEVICE=eth0
NAME=eth0
BOOTPROTO=none
ONBOOT=yes
IPADDR=172.31.3.101
PREFIX=21
GATEWAY=172.31.0.2
DNS1=223.5.5.5
DNS2=180.76.76.76

#Ubuntu
root@k8s-master01:~# cat etc/netplan/01-netcfg.yaml 
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      addresses: [172.31.3.101/21] 
      gateway4: 172.31.0.2
      nameservers:
        addresses: [223.5.5.5, 180.76.76.76]
复制

所有节点配置hosts，修改/etc/hosts如下：

cat >> etc/hosts <<EOF
172.31.3.101 k8s-master01.example.local k8s-master01
172.31.3.102 k8s-master02.example.local k8s-master02
172.31.3.103 k8s-master03.example.local k8s-master03
172.31.3.104 k8s-ha01.example.local k8s-ha01
172.31.3.105 k8s-ha02.example.local k8s-ha02
172.31.3.106 k8s-harbor01.example.local k8s-harbor01
172.31.3.107 k8s-harbor02.example.local k8s-harbor02
172.31.3.108 k8s-etcd01.example.local k8s-etcd01
172.31.3.109 k8s-etcd02.example.local k8s-etcd02
172.31.3.110 k8s-etcd03.example.local k8s-etcd03
172.31.3.111 k8s-node01.example.local k8s-node01
172.31.3.112 k8s-node02.example.local k8s-node02
172.31.3.113 k8s-node03.example.local k8s-node03
172.31.3.188 k8s-lb
172.31.3.188 harbor.raymonds.cc
EOF
复制

CentOS 7所有节点配置 yum源如下：

[root@k8s-master01 ~]# rm -f etc/yum.repos.d/*.repo
[root@k8s-master01 ~]# cat > etc/yum.repos.d/base.repo <<EOF
[base]
name=base
baseurl=https://mirrors.cloud.tencent.com/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-$releasever

[extras]
name=extras
baseurl=https://mirrors.cloud.tencent.com/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-$releasever

[updates]
name=updates
baseurl=https://mirrors.cloud.tencent.com/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-$releasever

[centosplus]
name=centosplus
baseurl=https://mirrors.cloud.tencent.com/centos/$releasever/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-$releasever

[epel]
name=epel
baseurl=https://mirrors.cloud.tencent.com/epel/$releasever/$basearch/
gpgcheck=1
gpgkey=https://mirrors.cloud.tencent.com/epel/RPM-GPG-KEY-EPEL-$releasever
EOF
复制

Rocky 8所有节点配置 yum源如下：

[root@k8s-master01 ~]# cat etc/yum.repos.d/base.repo 
[BaseOS]
name=BaseOS
baseurl=https://mirrors.sjtug.sjtu.edu.cn/rocky/$releasever/BaseOS/$basearch/os/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial

[AppStream]
name=AppStream
baseurl=https://mirrors.sjtug.sjtu.edu.cn/rocky/$releasever/AppStream/$basearch/os/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial

[extras]
name=extras
baseurl=https://mirrors.sjtug.sjtu.edu.cn/rocky/$releasever/extras/$basearch/os/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial
enabled=1

[plus]
name=plus
baseurl=https://mirrors.sjtug.sjtu.edu.cn/rocky/$releasever/plus/$basearch/os/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial

[PowerTools]
name=PowerTools
baseurl=https://mirrors.sjtug.sjtu.edu.cn/rocky/$releasever/PowerTools/$basearch/os/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial
复制

CentOS stream 8所有节点配置 yum源如下：

[root@centos8-stream ~]# cat etc/yum.repos.d/base.repo 
[BaseOS]
name=BaseOS
baseurl=https://mirrors.cloud.tencent.com/centos/$releasever-stream/BaseOS/$basearch/os/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[AppStream]
name=AppStream
baseurl=https://mirrors.cloud.tencent.com/centos/$releasever-stream/AppStream/$basearch/os/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[extras]
name=extras
baseurl=https://mirrors.cloud.tencent.com/centos/$releasever-stream/extras/$basearch/os/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[centosplus]
name=centosplus
baseurl=https://mirrors.cloud.tencent.com/centos/$releasever-stream/centosplus/$basearch/os/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[PowerTools]
name=PowerTools
baseurl=https://mirrors.cloud.tencent.com/centos/$releasever-stream/PowerTools/$basearch/os/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
复制

Ubuntu 所有节点配置 apt源如下：

root@k8s-master01:~# cat > etc/apt/sources.list <<EOF
deb http://mirrors.cloud.tencent.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.cloud.tencent.com/ubuntu/ bionic main restricted universe multiverse

deb http://mirrors.cloud.tencent.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.cloud.tencent.com/ubuntu/ bionic-security main restricted universe multiverse

deb http://mirrors.cloud.tencent.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.cloud.tencent.com/ubuntu/ bionic-updates main restricted universe multiverse

deb http://mirrors.cloud.tencent.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.cloud.tencent.com/ubuntu/ bionic-proposed main restricted universe multiverse

deb http://mirrors.cloud.tencent.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.cloud.tencent.com/ubuntu/ bionic-backports main restricted universe multiverse
EOF
复制

必备工具安装:

#CentOS安装
yum -y install vim tree lrzsz wget jq psmisc net-tools telnet yum-utils device-mapper-persistent-data lvm2 git 

#Ubuntu安装
apt -y install tree lrzsz jq
复制

所有节点关闭防火墙、selinux、swap。服务器配置如下：

#CentOS
systemctl disable --now firewalld

#CentOS 7
systemctl disable --now NetworkManager

#Ubuntu
systemctl disable --now ufw

setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' etc/selinux/config
复制

关闭swap分区

sed -ri 's/.*swap.*/#&/' etc/fstab
swapoff -a

#Ubuntu 20.04，执行下面命令
sed -ri 's/.*swap.*/#&/' etc/fstab
SD_NAME=`lsblk|awk -F"[ └─]" '/SWAP/{printf $3}'`
systemctl mask dev-${SD_NAME}.swap
swapoff -a
复制

ha01和ha02上安装chrony-server:

[root@k8s-ha01 ~]# cat install_chrony_server.sh 
#!/bin/bash
#
#**********************************************************************************************
#Author:        Raymond
#QQ:            88563128
#Date:          2021-11-22
#FileName:      install_chrony_server.sh
#URL:           raymond.blog.csdn.net
#Description:   install_chrony_server for CentOS 7/8 & Ubuntu 18.04/20.04 & Rocky 8
#Copyright (C): 2021 All rights reserved
#*********************************************************************************************
COLOR="echo -e \\033[01;31m"
END='\033[0m'

os(){
    OS_ID=`sed -rn '/^NAME=/s@.*="([[:alpha:]]+).*"$@\1@p' etc/os-release`
}

install_chrony(){
    if [ ${OS_ID} == "CentOS" -o ${OS_ID} == "Rocky" ] &> dev/null;then
        yum -y install chrony &> dev/null
        sed -i -e '/^pool.*/d' -e '/^server.*/d' -e '/^# Please consider .*/a\server ntp.aliyun.com iburst\nserver time1.cloud.tencent.com iburst\nserver ntp.tuna.tsinghua.edu.cn iburst' -e 's@^#allow.*@allow 0.0.0.0/0@' -e 's@^#local.*@local stratum 10@' etc/chrony.conf
        systemctl enable --now chronyd &> dev/null
        systemctl is-active chronyd &> dev/null ||  { ${COLOR}"chrony 启动失败,退出!"${END} ; exit; }
        ${COLOR}"chrony安装完成"${END}
    else
        apt -y install chrony &> dev/null
        sed -i -e '/^pool.*/d' -e '/^# See http:.*/a\server ntp.aliyun.com iburst\nserver time1.cloud.tencent.com iburst\nserver ntp.tuna.tsinghua.edu.cn iburst' etc/chrony/chrony.conf
        echo "allow 0.0.0.0/0" >> etc/chrony/chrony.conf
        echo "local stratum 10" >> etc/chrony/chrony.conf
        systemctl enable --now chronyd &> dev/null
        systemctl is-active chronyd &> dev/null ||  { ${COLOR}"chrony 启动失败,退出!"${END} ; exit; }
        ${COLOR}"chrony安装完成"${END}
    fi
}

main(){
    os
    install_chrony
}

main

[root@k8s-ha01 ~]# bash install_chrony_server.sh 
chrony安装完成

[root@k8s-ha02 ~]# bash install_chrony_server.sh 
chrony安装完成

[root@k8s-ha01 ~]# chronyc sources -nv
210 Number of sources = 3
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 203.107.6.88                  2   6    17    39  -1507us[-8009us] +/-   37ms
^- 139.199.215.251               2   6    17    39    +10ms[  +10ms] +/-   48ms
^? 101.6.6.172                   0   7     0     -     +0ns[   +0ns] +/-    0ns

[root@k8s-ha02 ~]# chronyc sources -nv
210 Number of sources = 3
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 203.107.6.88                  2   6    17    40    +90us[-1017ms] +/-   32ms
^+ 139.199.215.251               2   6    33    37    +13ms[  +13ms] +/-   25ms
^? 101.6.6.172                   0   7     0     -     +0ns[   +0ns] +/-    0ns
复制

master、node、harbor上安装chrony-client:

#!/bin/bash
#
#**********************************************************************************************
#Author:        Raymond
#QQ:            88563128
#Date:          2021-11-22
#FileName:      install_chrony_client.sh
#URL:           raymond.blog.csdn.net
#Description:   install_chrony_client for CentOS 7/8 & Ubuntu 18.04/20.04 & Rocky 8
#Copyright (C): 2021 All rights reserved
#*********************************************************************************************
COLOR="echo -e \\033[01;31m"
END='\033[0m'
SERVER1=172.31.3.104
SERVER2=172.31.3.105

os(){
    OS_ID=`sed -rn '/^NAME=/s@.*="([[:alpha:]]+).*"$@\1@p' etc/os-release`
}

install_chrony(){
    if [ ${OS_ID} == "CentOS" -o ${OS_ID} == "Rocky" ] &> dev/null;then
        yum -y install chrony &> dev/null
        sed -i -e '/^pool.*/d' -e '/^server.*/d' -e '/^# Please consider .*/a\server '${SERVER1}' iburst\nserver '${SERVER2}' iburst' etc/chrony.conf
        systemctl enable --now chronyd &> dev/null
        systemctl is-active chronyd &> dev/null ||  { ${COLOR}"chrony 启动失败,退出!"${END} ; exit; }
        ${COLOR}"chrony安装完成"${END}
    else
        apt -y install chrony &> dev/null
        sed -i -e '/^pool.*/d' -e '/^# See http:.*/a\server '${SERVER1}' iburst\nserver '${SERVER2}' iburst' etc/chrony/chrony.conf
        systemctl enable --now chronyd &> dev/null
        systemctl is-active chronyd &> dev/null ||  { ${COLOR}"chrony 启动失败,退出!"${END} ; exit; }
        systemctl restart chronyd
        ${COLOR}"chrony安装完成"${END}
    fi
}

main(){
    os
    install_chrony
}

main

[root@k8s-master01 ~]# cat install_chrony_client.sh 
#!/bin/bash
COLOR="echo -e \\033[01;31m"
END='\033[0m'
SERVER1=172.31.3.104
SERVER2=172.31.3.105

os(){
    OS_ID=`sed -rn '/^NAME=/s@.*="([[:alpha:]]+).*"$@\1@p' etc/os-release`
}

install_chrony(){
    if [ ${OS_ID} == "CentOS" -o ${OS_ID} == "Rocky" ] &> dev/null;then
        yum -y install chrony &> dev/null
        sed -i -e '/^pool.*/d' -e '/^server.*/d' -e '/^# Please consider .*/a\server '${SERVER1}' iburst\nserver '${SERVER2}' iburst' etc/chrony.conf
        systemctl enable --now chronyd &> dev/null
        systemctl is-active chronyd &> dev/null ||  { ${COLOR}"chrony 启动失败,退出!"${END} ; exit; }
        ${COLOR}"chrony安装完成"${END}
    else
        apt -y install chrony &> dev/null
        sed -i -e '/^pool.*/d' -e '/^# See http:.*/a\server '${SERVER1}' iburst\nserver '${SERVER2}' iburst' etc/chrony/chrony.conf
        systemctl enable --now chronyd &> dev/null
        systemctl is-active chronyd &> dev/null ||  { ${COLOR}"chrony 启动失败,退出!"${END} ; exit; }
        systemctl restart chronyd
        ${COLOR}"chrony安装完成"${END}
    fi
}

main(){
    os
    install_chrony
}

main

[root@k8s-master01 ~]# for i in k8s-master02 k8s-master03 k8s-harbor01 k8s-harbor02 k8s-node01 k8s-node02 k8s-node03;do scp install_chrony_client.sh $i:/root/ ; done

[root@k8s-master01 ~]# bash install_chrony_client.sh 
chrony安装完成

[root@k8s-master01 ~]# chronyc sources -nv
210 Number of sources = 2
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^+ k8s-ha01                      3   6    17     8    +84us[  +74us] +/-   55ms
^* k8s-ha02                      3   6    17     8    -82us[  -91us] +/-   45ms
复制

所有节点设置时区。时间同步配置如下：

ln -sf usr/share/zoneinfo/Asia/Shanghai etc/localtime
echo 'Asia/Shanghai' >/etc/timezone
复制

所有节点配置limit：

ulimit -SHn 65535

cat >>/etc/security/limits.conf <<EOF
* soft nofile 65536
* hard nofile 131072
* soft nproc 65535
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF
复制

Master01节点免密钥登录其他节点，安装过程中生成配置文件和证书均在Master01上操作，集群管理也在Master01上操作，阿里云或者AWS上需要单独一台kubectl服务器。密钥配置如下：

[root@k8s-master01 ~]# cat ssh_key.sh 
#!/bin/bash
#
#**********************************************************************************************
#Author:        Raymond
#QQ:            88563128
#Date:          2021-12-20
#FileName:      ssh_key.sh
#URL:           raymond.blog.csdn.net
#Description:   ssh_key for CentOS 7/8 & Ubuntu 18.04/24.04 & Rocky 8
#Copyright (C): 2021 All rights reserved
#*********************************************************************************************
COLOR="echo -e \\033[01;31m"
END='\033[0m'

NET_NAME=`ip addr |awk -F"[: ]" '/^2: e.*/{print $3}'`
IP=`ip addr show ${NET_NAME}| awk -F" +|/" '/global/{print $3}'`
export SSHPASS=123456
HOSTS="
172.31.3.102
172.31.3.103
172.31.3.104
172.31.3.105
172.31.3.106
172.31.3.107
172.31.3.108
172.31.3.109
172.31.3.110
172.31.3.111
172.31.3.112
172.31.3.113"

os(){
    OS_ID=`sed -rn '/^NAME=/s@.*="([[:alpha:]]+).*"$@\1@p' etc/os-release`
}

ssh_key_push(){
    rm -f ~/.ssh/id_rsa*
    ssh-keygen -f root/.ssh/id_rsa -P '' &> dev/null
    if [ ${OS_ID} == "CentOS" -o ${OS_ID} == "Rocky" ] &> dev/null;then
        rpm -q sshpass &> dev/null || { ${COLOR}"安装sshpass软件包"${END};yum -y install sshpass &> dev/null; }
    else
        dpkg -S sshpass &> dev/null || { ${COLOR}"安装sshpass软件包"${END};apt -y install sshpass &> dev/null; }
    fi
    sshpass -e ssh-copy-id -o StrictHostKeyChecking=no ${IP} &> dev/null
    [ $? -eq 0 ] && echo ${IP} is finished || echo ${IP} is false

    for i in ${HOSTS};do
        sshpass -e scp -o StrictHostKeyChecking=no -r root/.ssh root@${i}: &> dev/null
        [ $? -eq 0 ] && echo ${i} is finished || echo ${i} is false
    done

    for i in ${HOSTS};do
        scp root/.ssh/known_hosts ${i}:.ssh/ &> dev/null
        [ $? -eq 0 ] && echo ${i} is finished || echo ${i} is false
    done
}

main(){
    os
    ssh_key_push
}

main

[root@k8s-master01 ~]# bash ssh_key.sh 
安装sshpass软件包
172.31.3.101 is finished
172.31.3.102 is finished
172.31.3.103 is finished
172.31.3.104 is finished
172.31.3.105 is finished
172.31.3.106 is finished
172.31.3.107 is finished
172.31.3.108 is finished
172.31.3.109 is finished
172.31.3.110 is finished
172.31.3.111 is finished
172.31.3.112 is finished
172.31.3.113 is finished
172.31.3.102 is finished
172.31.3.103 is finished
172.31.3.104 is finished
172.31.3.105 is finished
172.31.3.106 is finished
172.31.3.107 is finished
172.31.3.108 is finished
172.31.3.109 is finished
172.31.3.110 is finished
172.31.3.111 is finished
172.31.3.112 is finished
172.31.3.113 is finished
复制

所有节点升级系统并重启，此处升级没有升级内核，下节会单独升级内核：

yum update -y --exclude=kernel* && reboot #CentOS7需要升级，CentOS8可以按需升级系统

[root@k8s-master01 ~]# cat etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)
复制

3.内核配置

CentOS7 需要升级内核至4.18+，本地升级的版本为4.19

在master01节点下载内核：

[root@k8s-master01 ~]# wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm

[root@k8s-master01 ~]# wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm
复制

从master01节点传到其他节点：

[root@k8s-master01 ~]# for i in k8s-master02 k8s-master03 k8s-node01 k8s-node02;do scp kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm $i:/root/ ; done
复制

所有节点安装内核

cd root && yum localinstall -y kernel-ml*
复制

所有节点更改内核启动顺序

grub2-set-default 0 && grub2-mkconfig -o etc/grub2.cfg

grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
复制

检查默认内核是不是4.19

grubby --default-kernel

[root@k8s-master01 ~]# grubby --default-kernel
/boot/vmlinuz-4.19.12-1.el7.elrepo.x86_64
复制

所有节点重启，然后检查内核是不是4.19

reboot

uname -a

[root@k8s-master01 ~]# uname -a
Linux k8s-master01 4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux
复制

master和node安装ipvsadm：

#CentOS
yum -y install ipvsadm ipset sysstat conntrack libseccomp

#Ubuntu
apt -y install ipvsadm ipset sysstat conntrack libseccomp-dev
复制

所有节点配置ipvs模块，在内核4.19+版本nf_conntrack_ipv4已经改为nf_conntrack， 4.18以下使用nf_conntrack_ipv4即可：

modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack #内核小于4.18，把这行改成nf_conntrack_ipv4

cat >> etc/modules-load.d/ipvs.conf <<EOF
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack #内核小于4.18，把这行改成nf_conntrack_ipv4
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF

然后执行systemctl enable --now systemd-modules-load.service即可
复制

开启一些k8s集群中必须的内核参数，所有节点配置k8s内核：

cat > etc/sysctl.d/k8s.conf <<EOF 
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF

sysctl --system
复制

Kubernetes内核优化常用参数详解：

net.ipv4.ip_forward = 1 #其值为0,说明禁止进行IP转发；如果是1,则说明IP转发功能已经打开。
net.bridge.bridge-nf-call-iptables = 1 #二层的网桥在转发包时也会被iptables的FORWARD规则所过滤，这样有时会出现L3层的iptables rules去过滤L2的帧的问题
net.bridge.bridge-nf-call-ip6tables = 1 #是否在ip6tables链中过滤IPv6包 
fs.may_detach_mounts = 1 #当系统有容器运行时，需要设置为1

vm.overcommit_memory=1  
#0， 表示内核将检查是否有足够的可用内存供应用进程使用；如果有足够的可用内存，内存申请允许；否则，内存申请失败，并把错误返回给应用进程。
#1， 表示内核允许分配所有的物理内存，而不管当前的内存状态如何。
#2， 表示内核允许分配超过所有物理内存和交换空间总和的内存

vm.panic_on_oom=0 
#OOM就是out of memory的缩写，遇到内存耗尽、无法分配的状况。kernel面对OOM的时候，咱们也不能慌乱，要根据OOM参数来进行相应的处理。
#值为0：内存不足时，启动 OOM killer。
#值为1：内存不足时，有可能会触发 kernel panic（系统重启），也有可能启动 OOM killer。
#值为2：内存不足时，表示强制触发 kernel panic，内核崩溃GG（系统重启）。

fs.inotify.max_user_watches=89100 #表示同一用户同时可以添加的watch数目（watch一般是针对目录，决定了同时同一用户可以监控的目录数量）

fs.file-max=52706963 #所有进程最大的文件数
fs.nr_open=52706963 #单个进程可分配的最大文件数
net.netfilter.nf_conntrack_max=2310720 #连接跟踪表的大小，建议根据内存计算该值CONNTRACK_MAX = RAMSIZE (in bytes)  16384  (x  32)，并满足nf_conntrack_max=4*nf_conntrack_buckets，默认262144

net.ipv4.tcp_keepalive_time = 600  #KeepAlive的空闲时长，或者说每次正常发送心跳的周期，默认值为7200s（2小时）
net.ipv4.tcp_keepalive_probes = 3 #在tcp_keepalive_time之后，没有接收到对方确认，继续发送保活探测包次数，默认值为9（次）
net.ipv4.tcp_keepalive_intvl =15 #KeepAlive探测包的发送间隔，默认值为75s
net.ipv4.tcp_max_tw_buckets = 36000 #Nginx 之类的中间代理一定要关注这个值，因为它对你的系统起到一个保护的作用，一旦端口全部被占用，服务就异常了。tcp_max_tw_buckets 能帮你降低这种情况的发生概率，争取补救时间。
net.ipv4.tcp_tw_reuse = 1 #只对客户端起作用，开启后客户端在1s内回收
net.ipv4.tcp_max_orphans = 327680 #这个值表示系统所能处理不属于任何进程的socket数量，当我们需要快速建立大量连接时，就需要关注下这个值了。

net.ipv4.tcp_orphan_retries = 3
#出现大量fin-wait-1
#首先，fin发送之后，有可能会丢弃，那么发送多少次这样的fin包呢？fin包的重传，也会采用退避方式，在2.6.358内核中采用的是指数退避，2s，4s，最后的重试次数是由tcp_orphan_retries来限制的。

net.ipv4.tcp_syncookies = 1 #tcp_syncookies是一个开关，是否打开SYN Cookie功能，该功能可以防止部分SYN攻击。tcp_synack_retries和tcp_syn_retries定义SYN的重试次数。
net.ipv4.tcp_max_syn_backlog = 16384 #进入SYN包的最大请求队列.默认1024.对重负载服务器,增加该值显然有好处.
net.ipv4.ip_conntrack_max = 65536 #表明系统将对最大跟踪的TCP连接数限制默认为65536
net.ipv4.tcp_max_syn_backlog = 16384 #指定所能接受SYN同步包的最大客户端数量，即半连接上限；
net.ipv4.tcp_timestamps = 0 #在使用 iptables 做 nat 时，发现内网机器 ping 某个域名 ping 的通，而使用 curl 测试不通, 原来是 net.ipv4.tcp_timestamps 设置了为 1 ，即启用时间戳
net.core.somaxconn = 16384  #Linux中的一个kernel参数，表示socket监听（listen）的backlog上限。什么是backlog呢？backlog就是socket的监听队列，当一个请求（request）尚未被处理或建立时，他会进入backlog。而socket server可以一次性处理backlog中的所有请求，处理后的请求不再位于监听队列中。当server处理请求较慢，以至于监听队列被填满后，新来的请求会被拒绝。
复制

所有节点配置完内核后，重启服务器，保证重启后内核依旧加载

reboot
lsmod | grep --color=auto -e ip_vs -e nf_conntrack

[root@k8s-master01 ~]# lsmod | grep --color=auto -e ip_vs -e nf_conntrack
ip_vs_ftp              16384  0 
nf_nat                 32768  1 ip_vs_ftp
ip_vs_sed              16384  0 
ip_vs_nq               16384  0 
ip_vs_fo               16384  0 
ip_vs_sh               16384  0 
ip_vs_dh               16384  0 
ip_vs_lblcr            16384  0 
ip_vs_lblc             16384  0 
ip_vs_wrr              16384  0 
ip_vs_rr               16384  0 
ip_vs_wlc              16384  0 
ip_vs_lc               16384  0 
ip_vs                 151552  24 ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip_vs_fo,ip_vs_nq,ip_vs_lblc,ip_vs_wrr,ip_vs_lc,ip_vs_sed,ip_vs_ftp
nf_conntrack          143360  2 nf_nat,ip_vs
nf_defrag_ipv6         20480  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
libcrc32c              16384  4 nf_conntrack,nf_nat,xfs,ip_vs
复制

4.高可用组件安装

（注意：如果不是高可用集群，haproxy和keepalived无需安装）

公有云要用公有云自带的负载均衡，比如阿里云的SLB，腾讯云的ELB，用来替代haproxy和keepalived，因为公有云大部分都是不支持keepalived的，另外如果用阿里云的话，kubectl控制端不能放在master节点，推荐使用腾讯云，因为阿里云的slb有回环的问题，也就是slb代理的服务器不能反向访问SLB，但是腾讯云修复了这个问题。

阿里云需要设置：slb -> haproxy -> apiserver

4.1 安装haproxy

在ha01和ha02安装HAProxy：

[root@k8s-ha01 ~]# cat install_haproxy.sh 
#!/bin/bash
#
#**********************************************************************************************
#Author:        Raymond
#QQ:            88563128
#Date:          2021-12-29
#FileName:      install_haproxy.sh
#URL:           raymond.blog.csdn.net
#Description:   The test script
#Copyright (C): 2021 All rights reserved
#*********************************************************************************************
SRC_DIR=/usr/local/src
COLOR="echo -e \\033[01;31m"
END='\033[0m'
CPUS=`lscpu |awk '/^CPU\(s\)/{print $2}'`

#lua下载地址：http://www.lua.org/ftp/lua-5.4.3.tar.gz
LUA_FILE=lua-5.4.3.tar.gz

#haproxy下载地址：https://www.haproxy.org/download/2.4/src/haproxy-2.4.10.tar.gz
HAPROXY_FILE=haproxy-2.4.10.tar.gz
HAPROXY_INSTALL_DIR=/apps/haproxy

STATS_AUTH_USER=admin
STATS_AUTH_PASSWORD=123456

VIP=172.31.3.188
MASTER01=172.31.3.101
MASTER02=172.31.3.102
MASTER03=172.31.3.103
HARBOR01=172.31.3.106
HARBOR02=172.31.3.107

os(){
    OS_ID=`sed -rn '/^NAME=/s@.*="([[:alpha:]]+).*"$@\1@p' etc/os-release`
}

check_file (){
    cd ${SRC_DIR}
    ${COLOR}'检查Haproxy相关源码包'${END}
    if [ ! -e ${LUA_FILE} ];then
        ${COLOR}"缺少${LUA_FILE}文件,请把文件放到${SRC_DIR}目录下"${END}
        exit
    elif [ ! -e ${HAPROXY_FILE} ];then
        ${COLOR}"缺少${HAPROXY_FILE}文件,请把文件放到${SRC_DIR}目录下"${END}
        exit
    else
        ${COLOR}"相关文件已准备好"${END}
    fi
}

install_haproxy(){
    [ -d ${HAPROXY_INSTALL_DIR} ] && { ${COLOR}"Haproxy已存在，安装失败"${END};exit; }
    ${COLOR}"开始安装Haproxy"${END}
    ${COLOR}"开始安装Haproxy依赖包"${END}
    if [ ${OS_ID} == "CentOS" -o ${OS_ID} == "Rocky" ] &> dev/null;then
        yum -y install gcc make gcc-c++ glibc glibc-devel pcre pcre-devel openssl openssl-devel systemd-devel libtermcap-devel ncurses-devel libevent-devel readline-devel &> dev/null
    else
        apt update &> dev/null;apt -y install gcc make openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev libreadline-dev libsystemd-dev &> dev/null
    fi
    tar xf ${LUA_FILE}
    LUA_DIR=`echo ${LUA_FILE} | sed -nr 's/^(.*[0-9]).([[:lower:]]).*/\1/p'`
    cd ${LUA_DIR}
    make all test
    cd ${SRC_DIR}
    tar xf ${HAPROXY_FILE}
    HAPROXY_DIR=`echo ${HAPROXY_FILE} | sed -nr 's/^(.*[0-9]).([[:lower:]]).*/\1/p'`
    cd ${HAPROXY_DIR}
    make -j ${CPUS} ARCH=x86_64 TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 USE_CPU_AFFINITY=1 USE_LUA=1 LUA_INC=${SRC_DIR}/${LUA_DIR}/src/ LUA_LIB=${SRC_DIR}/${LUA_DIR}/src/ PREFIX=${HAPROXY_INSTALL_DIR}
    make install PREFIX=${HAPROXY_INSTALL_DIR}
    [ $? -eq 0 ] && $COLOR"Haproxy编译安装成功"$END ||  { $COLOR"Haproxy编译安装失败,退出!"$END;exit; }
    cat > lib/systemd/system/haproxy.service <<-EOF
[Unit]
Description=HAProxy Load Balancer
After=syslog.target network.target

[Service]
ExecStartPre=/usr/sbin/haproxy -f etc/haproxy/haproxy.cfg -c -q
ExecStart=/usr/sbin/haproxy -Ws -f etc/haproxy/haproxy.cfg -p var/lib/haproxy/haproxy.pid
ExecReload=/bin/kill -USR2 $MAINPID

[Install]
WantedBy=multi-user.target
EOF
    [ -L usr/sbin/haproxy ] || ln -s ../..${HAPROXY_INSTALL_DIR}/sbin/haproxy usr/sbin/ &> dev/null
    [ -d etc/haproxy ] || mkdir etc/haproxy &> dev/null  
    [ -d var/lib/haproxy/ ] || mkdir -p var/lib/haproxy/ &> dev/null
    cat > etc/haproxy/haproxy.cfg <<-EOF
global
maxconn 100000
chroot ${HAPROXY_INSTALL_DIR}
stats socket var/lib/haproxy/haproxy.sock mode 600 level admin
uid 99
gid 99
daemon
pidfile var/lib/haproxy/haproxy.pid
log 127.0.0.1 local3 info

defaults
option http-keep-alive
option forwardfor
maxconn 100000
mode http
timeout connect 300000ms
timeout client 300000ms
timeout server 300000ms

listen stats
    mode http
    bind 0.0.0.0:9999
    stats enable
    log global
    stats uri haproxy-status
    stats auth ${STATS_AUTH_USER}:${STATS_AUTH_PASSWORD}

listen kubernetes-6443
    bind ${VIP}:6443
    mode tcp
    log global
    server ${MASTER01} ${MASTER01}:6443 check inter 3s fall 2 rise 5
    server ${MASTER02} ${MASTER02}:6443 check inter 3s fall 2 rise 5
    server ${MASTER03} ${MASTER03}:6443 check inter 3s fall 2 rise 5

listen harbor-80
    bind ${VIP}:80
    mode http
    log global
    balance source
    server ${HARBOR01} ${HARBOR01}:80 check inter 3s fall 2 rise 5
    server ${HARBOR02} ${HARBOR02}:80 check inter 3s fall 2 rise 5
EOF
    cat >> etc/sysctl.conf <<-EOF
net.ipv4.ip_nonlocal_bind = 1
EOF
    sysctl -p &> dev/null
    echo "PATH=${HAPROXY_INSTALL_DIR}/sbin:${PATH}" > etc/profile.d/haproxy.sh
    systemctl daemon-reload
    systemctl enable --now haproxy &> dev/null
    systemctl is-active haproxy &> dev/null && ${COLOR}"Haproxy 服务启动成功!"${END} ||  { ${COLOR}"Haproxy 启动失败,退出!"${END} ; exit; }
    ${COLOR}"Haproxy安装完成"${END}
}

main(){
    os
    check_file
    install_haproxy
}

main

[root@k8s-ha01 ~]# bash install_haproxy.sh

[root@k8s-ha02 ~]# bash install_haproxy.sh
复制

4.2 安装keepalived

所有master节点配置KeepAlived健康检查文件：

[root@k8s-ha02 ~]# cat check_haproxy.sh 
#!/bin/bash
#
#**********************************************************************************************
#Author:        Raymond
#QQ:            88563128
#Date:          2022-01-09
#FileName:      check_haproxy.sh
#URL:           raymond.blog.csdn.net
#Description:   The test script
#Copyright (C): 2022 All rights reserved
#*********************************************************************************************
err=0
for k in $(seq 1 3);do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 1
        continue
    else
        err=0
        break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi
复制

在ha01和ha02节点安装KeepAlived，配置不一样，注意区分 [root@k8s-master01 pki]# vim etc/keepalived/keepalived.conf ，注意每个节点的网卡（interface参数）

在ha01节点上安装keepalived-master：

[root@k8s-ha01 ~]# cat install_keepalived_master.sh 
#!/bin/bash
#
#**********************************************************************************************
#Author:        Raymond
#QQ:            88563128
#Date:          2021-12-29
#FileName:      install_keepalived_master.sh
#URL:           raymond.blog.csdn.net
#Description:   install_keepalived for CentOS 7/8 & Ubuntu 18.04/20.04 & Rocky 8
#Copyright (C): 2021 All rights reserved
#*********************************************************************************************
SRC_DIR=/usr/local/src
COLOR="echo -e \\033[01;31m"
END='\033[0m'
KEEPALIVED_URL=https://keepalived.org/software/
KEEPALIVED_FILE=keepalived-2.2.4.tar.gz
KEEPALIVED_INSTALL_DIR=/apps/keepalived
CPUS=`lscpu |awk '/^CPU\(s\)/{print $2}'`
NET_NAME=`ip addr |awk -F"[: ]" '/^2: e.*/{print $3}'`
STATE=MASTER
PRIORITY=100
VIP=172.31.3.188


os(){
    OS_ID=`sed -rn '/^NAME=/s@.*="([[:alpha:]]+).*"$@\1@p' etc/os-release`
    OS_RELEASE_VERSION=`sed -rn '/^VERSION_ID=/s@.*="?([0-9]+)\.?.*"?@\1@p' etc/os-release`
}

check_file (){
    cd  ${SRC_DIR}
    if [ ${OS_ID} == "CentOS" -o ${OS_ID} == "Rocky" ] &> dev/null;then
        rpm -q wget &> dev/null || yum -y install wget &> dev/null
    fi
    if [ ! -e ${KEEPALIVED_FILE} ];then
        ${COLOR}"缺少${KEEPALIVED_FILE}文件,如果是离线包,请放到${SRC_DIR}目录下"${END}
        ${COLOR}'开始下载Keepalived源码包'${END}
        wget ${KEEPALIVED_URL}${KEEPALIVED_FILE} || { ${COLOR}"Keepalived源码包下载失败"${END}; exit; }
    elif [ ! -e check_haproxy.sh ];then
        ${COLOR}"缺少check_haproxy.sh文件,请把文件放到${SRC_DIR}目录下"${END}
        exit
    else
        ${COLOR}"相关文件已准备好"${END}
    fi
}

install_keepalived(){
    [ -d ${KEEPALIVED_INSTALL_DIR} ] && { ${COLOR}"Keepalived已存在，安装失败"${END};exit; }
    ${COLOR}"开始安装Keepalived"${END}
    ${COLOR}"开始安装Keepalived依赖包"${END}
    if [ ${OS_ID} == "Rocky" -a ${OS_RELEASE_VERSION} == 8 ];then
        URL=mirrors.sjtug.sjtu.edu.cn
        if [ ! `grep -R "\[PowerTools\]" etc/yum.repos.d/` ];then
            cat > etc/yum.repos.d/PowerTools.repo <<-EOF
[PowerTools]
name=PowerTools
baseurl=https://${URL}/rocky/\$releasever/PowerTools/\$basearch/os/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial
EOF
        fi
    fi
    if [ ${OS_ID} == "CentOS" -a ${OS_RELEASE_VERSION} == 8 ];then
        URL=mirrors.cloud.tencent.com
        if [ ! `grep -R "\[PowerTools\]" etc/yum.repos.d/` ];then
            cat > etc/yum.repos.d/PowerTools.repo <<-EOF
[PowerTools]
name=PowerTools
baseurl=https://${URL}/centos/\$releasever/PowerTools/\$basearch/os/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
EOF
        fi
    fi
    if [[ ${OS_RELEASE_VERSION} == 8 ]] &> dev/null;then
        yum -y install make gcc ipvsadm autoconf automake openssl-devel libnl3-devel iptables-devel ipset-devel file-devel net-snmp-devel glib2-devel pcre2-devel libnftnl-devel libmnl-devel systemd-devel &> dev/null
    elif [[ ${OS_RELEASE_VERSION} == 7 ]] &> dev/null;then
        yum -y install make gcc libnfnetlink-devel libnfnetlink ipvsadm libnl libnl-devel libnl3 libnl3-devel lm_sensors-libs net-snmp-agent-libs net-snmp-libs openssh-server openssh-clients openssl openssl-devel automake iproute &> dev/null
    elif [[ ${OS_RELEASE_VERSION} == 20 ]] &> dev/null;then
        apt update &> dev/null;apt -y install make gcc ipvsadm build-essential pkg-config automake autoconf libipset-dev libnl-3-dev libnl-genl-3-dev libssl-dev libxtables-dev libip4tc-dev libip6tc-dev libipset-dev libmagic-dev libsnmp-dev libglib2.0-dev libpcre2-dev libnftnl-dev libmnl-dev libsystemd-dev
    else
        apt update &> dev/null;apt -y install make gcc ipvsadm build-essential pkg-config automake autoconf iptables-dev libipset-dev libnl-3-dev libnl-genl-3-dev libssl-dev libxtables-dev libip4tc-dev libip6tc-dev libipset-dev libmagic-dev libsnmp-dev libglib2.0-dev libpcre2-dev libnftnl-dev libmnl-dev libsystemd-dev &> dev/null
    fi
    tar xf ${KEEPALIVED_FILE}
    KEEPALIVED_DIR=`echo ${KEEPALIVED_FILE} | sed -nr 's/^(.*[0-9]).([[:lower:]]).*/\1/p'`
    cd ${KEEPALIVED_DIR}
    ./configure --prefix=${KEEPALIVED_INSTALL_DIR} --disable-fwmark
    make -j $CPUS && make install
    [ $? -eq 0 ] && ${COLOR}"Keepalived编译安装成功"${END} ||  { ${COLOR}"Keepalived编译安装失败,退出!"${END};exit; }
    [ -d etc/keepalived ] || mkdir -p etc/keepalived &> dev/null
    cat > etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived

global_defs {
    router_id LVS_DEVEL
    script_user root
    enable_script_security
}

vrrp_script check_haoroxy {
    script "/etc/keepalived/check_haproxy.sh"
    interval 5
    weight -5
    fall 2  
    rise 1
}

vrrp_instance VI_1 {
    state ${STATE}
    interface ${NET_NAME}
    virtual_router_id 51
    priority ${PRIORITY}
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        ${VIP} dev ${NET_NAME} label ${NET_NAME}:1
    }
    track_script {
       check_haproxy
    }
}
EOF
    cp ./keepalived/keepalived.service lib/systemd/system/
    cd  ${SRC_DIR}
    mv check_haproxy.sh etc/keepalived/check_haproxy.sh
    chmod +x etc/keepalived/check_haproxy.sh
    echo "PATH=${KEEPALIVED_INSTALL_DIR}/sbin:${PATH}" > etc/profile.d/keepalived.sh
    systemctl daemon-reload
    systemctl enable --now keepalived &> dev/null 
    systemctl is-active keepalived &> dev/null && ${COLOR}"Keepalived 服务启动成功!"${END} ||  { ${COLOR}"Keepalived 启动失败,退出!"${END} ; exit; }
    ${COLOR}"Keepalived安装完成"${END}
}

main(){
    os
    check_file
    install_keepalived
}

main

[root@k8s-ha01 ~]# bash install_keepalived_master.sh

[root@k8s-ha01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:05:9b:2a brd ff:ff:ff:ff:ff:ff
    inet 172.31.3.104/21 brd 172.31.7.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.31.3.188/32 scope global eth0:1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe05:9b2a/64 scope link 
       valid_lft forever preferred_lft forever
复制

在ha02节点上安装keepalived-backup：

[root@k8s-ha02 ~]# cat install_keepalived_backup.sh 
#!/bin/bash
#
#**********************************************************************************************
#Author:        Raymond
#QQ:            88563128
#Date:          2021-12-29
#FileName:      install_keepalived_backup.sh
#URL:           raymond.blog.csdn.net
#Description:   install_keepalived for CentOS 7/8 & Ubuntu 18.04/20.04 & Rocky 8
#Copyright (C): 2021 All rights reserved
#*********************************************************************************************
SRC_DIR=/usr/local/src
COLOR="echo -e \\033[01;31m"
END='\033[0m'
KEEPALIVED_URL=https://keepalived.org/software/
KEEPALIVED_FILE=keepalived-2.2.4.tar.gz
KEEPALIVED_INSTALL_DIR=/apps/keepalived
CPUS=`lscpu |awk '/^CPU\(s\)/{print $2}'`
NET_NAME=`ip addr |awk -F"[: ]" '/^2: e.*/{print $3}'`
STATE=BACKUP
PRIORITY=90
VIP=172.31.3.188


os(){
    OS_ID=`sed -rn '/^NAME=/s@.*="([[:alpha:]]+).*"$@\1@p' etc/os-release`
    OS_RELEASE_VERSION=`sed -rn '/^VERSION_ID=/s@.*="?([0-9]+)\.?.*"?@\1@p' etc/os-release`
}

check_file (){
    cd  ${SRC_DIR}
    if [ ${OS_ID} == "CentOS" -o ${OS_ID} == "Rocky" ] &> dev/null;then
        rpm -q wget &> dev/null || yum -y install wget &> dev/null
    fi
    if [ ! -e ${KEEPALIVED_FILE} ];then
        ${COLOR}"缺少${KEEPALIVED_FILE}文件,如果是离线包,请放到${SRC_DIR}目录下"${END}
        ${COLOR}'开始下载Keepalived源码包'${END}
        wget ${KEEPALIVED_URL}${KEEPALIVED_FILE} || { ${COLOR}"Keepalived源码包下载失败"${END}; exit; }
    elif [ ! -e check_haproxy.sh ];then
        ${COLOR}"缺少check_haproxy.sh文件,请把文件放到${SRC_DIR}目录下"${END}
        exit
    else
        ${COLOR}"相关文件已准备好"${END}
    fi
}

install_keepalived(){
    [ -d ${KEEPALIVED_INSTALL_DIR} ] && { ${COLOR}"Keepalived已存在，安装失败"${END};exit; }
    ${COLOR}"开始安装Keepalived"${END}
    ${COLOR}"开始安装Keepalived依赖包"${END}
    if [ ${OS_ID} == "Rocky" -a ${OS_RELEASE_VERSION} == 8 ];then
        URL=mirrors.sjtug.sjtu.edu.cn
        if [ ! `grep -R "\[PowerTools\]" etc/yum.repos.d/` ];then
            cat > etc/yum.repos.d/PowerTools.repo <<-EOF
[PowerTools]
name=PowerTools
baseurl=https://${URL}/rocky/\$releasever/PowerTools/\$basearch/os/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial
EOF
        fi
    fi
    if [ ${OS_ID} == "CentOS" -a ${OS_RELEASE_VERSION} == 8 ];then
        URL=mirrors.cloud.tencent.com
        if [ ! `grep -R "\[PowerTools\]" etc/yum.repos.d/` ];then
            cat > etc/yum.repos.d/PowerTools.repo <<-EOF
[PowerTools]
name=PowerTools
baseurl=https://${URL}/centos/\$releasever/PowerTools/\$basearch/os/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
EOF
        fi
    fi
    if [[ ${OS_RELEASE_VERSION} == 8 ]] &> dev/null;then
        yum -y install make gcc ipvsadm autoconf automake openssl-devel libnl3-devel iptables-devel ipset-devel file-devel net-snmp-devel glib2-devel pcre2-devel libnftnl-devel libmnl-devel systemd-devel &> dev/null
    elif [[ ${OS_RELEASE_VERSION} == 7 ]] &> dev/null;then
        yum -y install make gcc libnfnetlink-devel libnfnetlink ipvsadm libnl libnl-devel libnl3 libnl3-devel lm_sensors-libs net-snmp-agent-libs net-snmp-libs openssh-server openssh-clients openssl openssl-devel automake iproute &> dev/null
    elif [[ ${OS_RELEASE_VERSION} == 20 ]] &> dev/null;then
        apt update &> dev/null;apt -y install make gcc ipvsadm build-essential pkg-config automake autoconf libipset-dev libnl-3-dev libnl-genl-3-dev libssl-dev libxtables-dev libip4tc-dev libip6tc-dev libipset-dev libmagic-dev libsnmp-dev libglib2.0-dev libpcre2-dev libnftnl-dev libmnl-dev libsystemd-dev
    else
        apt update &> dev/null;apt -y install make gcc ipvsadm build-essential pkg-config automake autoconf iptables-dev libipset-dev libnl-3-dev libnl-genl-3-dev libssl-dev libxtables-dev libip4tc-dev libip6tc-dev libipset-dev libmagic-dev libsnmp-dev libglib2.0-dev libpcre2-dev libnftnl-dev libmnl-dev libsystemd-dev &> dev/null
    fi
    tar xf ${KEEPALIVED_FILE}
    KEEPALIVED_DIR=`echo ${KEEPALIVED_FILE} | sed -nr 's/^(.*[0-9]).([[:lower:]]).*/\1/p'`
    cd ${KEEPALIVED_DIR}
    ./configure --prefix=${KEEPALIVED_INSTALL_DIR} --disable-fwmark
    make -j $CPUS && make install
    [ $? -eq 0 ] && ${COLOR}"Keepalived编译安装成功"${END} ||  { ${COLOR}"Keepalived编译安装失败,退出!"${END};exit; }
    [ -d etc/keepalived ] || mkdir -p etc/keepalived &> dev/null
    cat > etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived

global_defs {
    router_id LVS_DEVEL
    script_user root
    enable_script_security
}

vrrp_script check_haoroxy {
    script "/etc/keepalived/check_haproxy.sh"
    interval 5
    weight -5
    fall 2  
    rise 1
}

vrrp_instance VI_1 {
    state ${STATE}
    interface ${NET_NAME}
    virtual_router_id 51
    priority ${PRIORITY}
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        ${VIP} dev ${NET_NAME} label ${NET_NAME}:1
    }
    track_script {
       check_haproxy
    }
}
EOF
    cp ./keepalived/keepalived.service lib/systemd/system/
    cd  ${SRC_DIR}
    mv check_haproxy.sh etc/keepalived/check_haproxy.sh
    chmod +x etc/keepalived/check_haproxy.sh
    echo "PATH=${KEEPALIVED_INSTALL_DIR}/sbin:${PATH}" > etc/profile.d/keepalived.sh
    systemctl daemon-reload
    systemctl enable --now keepalived &> dev/null 
    systemctl is-active keepalived &> dev/null && ${COLOR}"Keepalived 服务启动成功!"${END} ||  { ${COLOR}"Keepalived 启动失败,退出!"${END} ; exit; }
    ${COLOR}"Keepalived安装完成"${END}
}

main(){
    os
    check_file
    install_keepalived
}

main

[root@k8s-ha02 ~]# bash install_keepalived_backup.sh
复制

5.安装harbor

5.1 安装harbor

在harbor01和harbor02上安装harbor:

[root@k8s-harbor01 ~]# cat install_docker_binary_compose_harbor.sh
#!/bin/bash
#
#**************************************************************************************************
#Author:        Raymond
#QQ:            88563128
#Date:          2021-12-15
#FileName:      install_docker_binary_compose_harbor.sh
#URL:           raymond.blog.csdn.net
#Description:   install_docker_binary_compose_harbor for CentOS 7/8 & Ubuntu 18.04/20.04 & Rocky 8
#Copyright (C): 2021 All rights reserved
#**************************************************************************************************
SRC_DIR=/usr/local/src
COLOR="echo -e \\033[01;31m"
END='\033[0m'

URL='https://download.docker.com/linux/static/stable/x86_64/'
DOCKER_FILE=docker-19.03.9.tgz

#docker-compose下载地址:https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64
DOCKER_COMPOSE_FILE=docker-compose-linux-x86_64

#harbor下载地址:https://github.com/goharbor/harbor/releases/download/v2.4.1/harbor-offline-installer-v2.4.1.tgz
HARBOR_FILE=harbor-offline-installer-v
HARBOR_VERSION=2.4.1
TAR=.tgz
HARBOR_INSTALL_DIR=/apps
HARBOR_DOMAIN=harbor.raymonds.cc
NET_NAME=`ip addr |awk -F"[: ]" '/^2: e.*/{print $3}'`
IP=`ip addr show ${NET_NAME}| awk -F" +|/" '/global/{print $3}'`
HARBOR_ADMIN_PASSWORD=123456

os(){
    OS_ID=`sed -rn '/^NAME=/s@.*="([[:alpha:]]+).*"$@\1@p' etc/os-release`
    OS_RELEASE_VERSION=`sed -rn '/^VERSION_ID=/s@.*="?([0-9]+)\.?.*"?@\1@p' etc/os-release`
}

check_file (){
    cd ${SRC_DIR}
    if [ ${OS_ID} == "CentOS" -o ${OS_ID} == "Rocky" ] &> dev/null;then
        rpm -q wget &> dev/null || yum -y install wget &> dev/null
    fi
    if [ ! -e ${DOCKER_FILE} ];then
        ${COLOR}"缺少${DOCKER_FILE}文件,如果是离线包,请把文件放到${SRC_DIR}目录下"${END}
        ${COLOR}'开始下载DOCKER二进制源码包'${END}
        wget ${URL}${DOCKER_FILE} || { ${COLOR}"DOCKER二进制安装包下载失败"${END}; exit; }
    elif [ ! -e ${DOCKER_COMPOSE_FILE} ];then
        ${COLOR}"缺少${DOCKER_COMPOSE_FILE}文件,请把文件放到${SRC_DIR}目录下"${END}
        exit
    elif [ ! -e ${HARBOR_FILE}${HARBOR_VERSION}${TAR} ];then
        ${COLOR}"缺少${HARBOR_FILE}${HARBOR_VERSION}${TAR}文件,请把文件放到${SRC_DIR}目录下"${END}
        exit
    else
        ${COLOR}"相关文件已准备好"${END}
    fi
}

install_docker(){ 
    tar xf ${DOCKER_FILE} 
    mv docker/* usr/bin/
    cat > lib/systemd/system/docker.service <<-EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H unix://var/run/docker.sock
ExecReload=/bin/kill -s HUP \$MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target
EOF
    mkdir -p etc/docker
    cat > etc/docker/daemon.json <<-EOF
{
    "registry-mirrors": [
        "https://registry.docker-cn.com",
        "http://hub-mirror.c.163.com",
        "https://docker.mirrors.ustc.edu.cn"
    ],
    "insecure-registries": ["${HARBOR_DOMAIN}"],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "max-concurrent-downloads": 10,
    "max-concurrent-uploads": 5,
    "log-opts": {
        "max-size": "300m",
        "max-file": "2"  
    },
    "live-restore": true
}
EOF
    echo 'alias rmi="docker images -qa|xargs docker rmi -f"' >> ~/.bashrc
    echo 'alias rmc="docker ps -qa|xargs docker rm -f"' >> ~/.bashrc
    systemctl daemon-reload
    systemctl enable --now docker &> dev/null
    systemctl is-active docker &> dev/null && ${COLOR}"Docker 服务启动成功"${END} || { ${COLOR}"Docker 启动失败"${END};exit; }
    docker version && ${COLOR}"Docker 安装成功"${END} || ${COLOR}"Docker 安装失败"${END}
}

install_docker_compose(){
    ${COLOR}"开始安装 Docker compose....."${END}
    sleep 1
    mv ${SRC_DIR}/${DOCKER_COMPOSE_FILE} usr/bin/docker-compose
    chmod +x usr/bin/docker-compose
    docker-compose --version &&  ${COLOR}"Docker Compose 安装完成"${END} || ${COLOR}"Docker compose 安装失败"${END}
}

install_harbor(){
    ${COLOR}"开始安装 Harbor....."${END}
    sleep 1
    [ -d ${HARBOR_INSTALL_DIR} ] || mkdir ${HARBOR_INSTALL_DIR}
    tar xf ${SRC_DIR}/${HARBOR_FILE}${HARBOR_VERSION}${TAR} -C ${HARBOR_INSTALL_DIR}/
    mv ${HARBOR_INSTALL_DIR}/harbor/harbor.yml.tmpl ${HARBOR_INSTALL_DIR}/harbor/harbor.yml
    sed -ri.bak -e 's/^(hostname:) .*/\1 '${IP}'/' -e 's/^(harbor_admin_password:) .*/\1 '${HARBOR_ADMIN_PASSWORD}'/' -e 's/^(https:)/#\1/' -e 's/  (port: 443)/#  \1/' -e 's@  (certificate: .*)@#  \1@' -e 's@  (private_key: .*)@#  \1@' ${HARBOR_INSTALL_DIR}/harbor/harbor.yml
    if [ ${OS_ID} == "CentOS" -o ${OS_ID} == "Rocky" ] &> dev/null;then
        yum -y install python3 &> dev/null || { ${COLOR}"安装软件包失败，请检查网络配置"${END}; exit; }
    else
        apt -y install python3 &> dev/null || { ${COLOR}"安装软件包失败，请检查网络配置"${END}; exit; }
    fi
    ${HARBOR_INSTALL_DIR}/harbor/install.sh && ${COLOR}"Harbor 安装完成"${END} ||  ${COLOR}"Harbor 安装失败"${END}
    cat > lib/systemd/system/harbor.service <<-EOF
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/vmware/harbor

[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/bin/docker-compose -f apps/harbor/docker-compose.yml up
ExecStop=/usr/bin/docker-compose -f apps/harbor/docker-compose.yml down

[Install]
WantedBy=multi-user.target
EOF

    systemctl daemon-reload 
    systemctl enable harbor &>/dev/null && ${COLOR}"Harbor已配置为开机自动启动"${END}
}

set_swap_limit(){
    if [ ${OS_ID} == "Ubuntu" ];then
        ${COLOR}'设置Docker的"WARNING: No swap limit support"警告'${END}
        sed -ri '/^GRUB_CMDLINE_LINUX=/s@"$@ swapaccount=1"@' etc/default/grub
        update-grub &> dev/null
        ${COLOR}"10秒后，机器会自动重启"${END}
        sleep 10
        reboot
    fi
}

main(){
    os
    check_file
    [ -f usr/bin/docker ] && ${COLOR}"Docker已安装"${END} || install_docker
    docker-compose --version &> dev/null && ${COLOR}"Docker Compose已安装"${END} || install_docker_compose
    systemctl is-active harbor &> dev/null && ${COLOR}"Harbor已安装"${END} || install_harbor
    grep -q "swapaccount=1" etc/default/grub && ${COLOR}'"WARNING: No swap limit support"警告,已设置'${END} || set_swap_limit
}

main

[root@k8s-harbor01 ~]# bash install_docker_compose_harbor.sh

[root@k8s-harbor02 ~]# bash install_docker_compose_harbor.sh
复制

5.2 创建harbor仓库

在harbor01新建项目google_containers

在harbor02新建项目google_containers

在harbor02上新建目标

在harbor02上新建规则

在harbor01上新建目标

在harbor01上新建规则

6.部署etcd

6.1 安装etcd组件

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#server-binaries-1

下载etcd安装包

[root@k8s-etcd01 ~]# wget https://github.com/etcd-io/etcd/releases/download/v3.4.13/etcd-v3.4.13-linux-amd64.tar.gz
复制

解压etcd安装文件

[root@k8s-etcd01 ~]# tar -xf etcd-v3.4.13-linux-amd64.tar.gz --strip-components=1 -C usr/local/bin etcd-v3.4.13-linux-amd64/etcd{,ctl}
复制

版本查看

[root@k8s-etcd01 ~]# etcdctl version
etcdctl version: 3.4.13
API version: 3.4
复制

将组件发送到其他节点

[root@k8s-etcd01 ~]# for NODE in k8s-etcd02 k8s-etcd03; do echo $NODE; scp -o StrictHostKeyChecking=no usr/local/bin/etcd* $NODE:/usr/local/bin/; done
复制

etcd节点创建/opt/cni/bin目录

[root@k8s-etcd01 ~]# mkdir -p opt/cni/bin
[root@k8s-etcd02 ~]# mkdir -p opt/cni/bin
[root@k8s-etcd03 ~]# mkdir -p opt/cni/bin
复制

6.2 生成etcd证书

二进制安装最关键步骤，一步错误全盘皆输，一定要注意每个步骤都要是正确的

etcd01下载生成证书工具

[root@k8s-etcd01 ~]# wget "https://pkg.cfssl.org/R1.2/cfssl_linux-amd64" -O usr/local/bin/cfssl

[root@k8s-etcd01 ~]# wget "https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64" -O usr/local/bin/cfssljson

[root@k8s-etcd01 ~]# chmod +x usr/local/bin/cfssl usr/local/bin/cfssljson
复制

etcd节点创建etcd证书目录

[root@k8s-etcd01 ~]# mkdir etc/etcd/ssl -p
[root@k8s-etcd02 ~]# mkdir etc/etcd/ssl -p
[root@k8s-etcd03 ~]# mkdir etc/etcd/ssl -p
复制

etcd01节点生成etcd证书

生成证书的CSR文件：证书签名请求文件，配置了一些域名、公司、单位

[root@k8s-etcd01 ~]# mkdir pki
[root@k8s-etcd01 ~]# cd pki/

[root@k8s-etcd01 pki]# cat etcd-ca-csr.json 
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}

# 生成etcd CA证书和CA证书的key
[root@k8s-etcd01 pki]# cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare etc/etcd/ssl/etcd-ca
#执行结果
2022/02/25 15:32:30 [INFO] generating a new CA key and certificate from CSR
2022/02/25 15:32:30 [INFO] generate received request
2022/02/25 15:32:30 [INFO] received CSR
2022/02/25 15:32:30 [INFO] generating key: rsa-2048
2022/02/25 15:32:30 [INFO] encoded CSR
2022/02/25 15:32:30 [INFO] signed certificate with serial number 283470375480918891598291779152743682937426861184

[root@k8s-etcd01 pki]# ll etc/etcd/ssl/etcd-ca*
-rw-r--r-- 1 root root 1005 Feb 25 15:32 etc/etcd/ssl/etcd-ca.csr
-rw------- 1 root root 1679 Feb 25 15:32 etc/etcd/ssl/etcd-ca-key.pem
-rw-r--r-- 1 root root 1367 Feb 25 15:32 etc/etcd/ssl/etcd-ca.pem

[root@k8s-etcd01 pki]# cat ca-config.json
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}

[root@k8s-etcd01 pki]# cat etcd-csr.json
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}

[root@k8s-etcd01 pki]# cfssl gencert \
   -ca=/etc/etcd/ssl/etcd-ca.pem \
   -ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
   -config=ca-config.json \
   -hostname=127.0.0.1,k8s-etcd01,k8s-etcd02,k8s-etcd03,172.31.3.108,172.31.3.109,172.31.3.110 \
   -profile=kubernetes \
   etcd-csr.json | cfssljson -bare etc/etcd/ssl/etcd
#执行结果
2022/02/25 15:34:05 [INFO] generate received request
2022/02/25 15:34:05 [INFO] received CSR
2022/02/25 15:34:05 [INFO] generating key: rsa-2048
2022/02/25 15:34:05 [INFO] encoded CSR
2022/02/25 15:34:05 [INFO] signed certificate with serial number 621125122845507641888880600687542216934848214818

[root@k8s-etcd01 pki]# ll etc/etcd/ssl/etcd*
-rw-r--r-- 1 root root 1005 Feb 25 15:32 etc/etcd/ssl/etcd-ca.csr
-rw------- 1 root root 1679 Feb 25 15:32 etc/etcd/ssl/etcd-ca-key.pem
-rw-r--r-- 1 root root 1367 Feb 25 15:32 etc/etcd/ssl/etcd-ca.pem
-rw-r--r-- 1 root root 1005 Feb 25 15:34 etc/etcd/ssl/etcd.csr
-rw------- 1 root root 1679 Feb 25 15:34 etc/etcd/ssl/etcd-key.pem
-rw-r--r-- 1 root root 1501 Feb 25 15:34 etc/etcd/ssl/etcd.pem
复制

将etcd证书复制到其他etcd节点

[root@k8s-etcd01 pki]# for NODE in k8s-etcd02 k8s-etcd03; do
     ssh -o StrictHostKeyChecking=no $NODE "mkdir -p etc/etcd/ssl"
     for FILE in etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem; do
       scp -o StrictHostKeyChecking=no etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}
     done
 done
复制

6.3 Etcd配置

etcd配置大致相同，注意修改每个etcd节点的etcd配置的主机名和IP地址

6.3.1 etcd01

[root@k8s-etcd01 pki]# vim etc/etcd/etcd.config.yml
name: 'k8s-etcd01'
data-dir: var/lib/etcd
wal-dir: var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://172.31.3.108:2380'
listen-client-urls: 'https://172.31.3.108:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://172.31.3.108:2380'
advertise-client-urls: 'https://172.31.3.108:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-etcd01=https://172.31.3.108:2380,k8s-etcd02=https://172.31.3.109:2380,k8s-etcd03=https://172.31.3.110:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
复制

6.3.2 etcd02

[root@k8s-etcd02 ~]# vim etc/etcd/etcd.config.yml
name: 'k8s-etcd02'
data-dir: var/lib/etcd
wal-dir: var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://172.31.3.109:2380'
listen-client-urls: 'https://172.31.3.109:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://172.31.3.109:2380'
advertise-client-urls: 'https://172.31.3.109:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-etcd01=https://172.31.3.108:2380,k8s-etcd02=https://172.31.3.109:2380,k8s-etcd03=https://172.31.3.110:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
复制

6.3.3 etcd03

[root@k8s-etcd03 ~]# vim etc/etcd/etcd.config.yml
name: 'k8s-etcd03'
data-dir: var/lib/etcd
wal-dir: var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://172.31.3.110:2380'
listen-client-urls: 'https://172.31.3.110:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://172.31.3.110:2380'
advertise-client-urls: 'https://172.31.3.110:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-etcd01=https://172.31.3.108:2380,k8s-etcd02=https://172.31.3.109:2380,k8s-etcd03=https://172.31.3.110:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
复制

6.3.4 创建Service

所有etcd节点创建etcd service并启动

[root@k8s-etcd01 pki]# vim lib/systemd/system/etcd.service
[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service

[root@k8s-etcd01 pki]# for NODE in k8s-etcd02 k8s-etcd03; do scp lib/systemd/system/etcd.service $NODE:/lib/systemd/system/;done
复制

所有etcd节点创建etcd的证书目录

[root@k8s-etcd01 pki]# mkdir etc/kubernetes/pki/etcd -p
[root@k8s-etcd02 ~]# mkdir etc/kubernetes/pki/etcd -p
[root@k8s-etcd03 ~]# mkdir etc/kubernetes/pki/etcd -p

[root@k8s-etcd01 pki]# ln -s etc/etcd/ssl/* etc/kubernetes/pki/etcd/
[root@k8s-etcd02 ~]# ln -s etc/etcd/ssl/* etc/kubernetes/pki/etcd/
[root@k8s-etcd03 ~]# ln -s etc/etcd/ssl/* etc/kubernetes/pki/etcd/

[root@k8s-etcd01 pki]# systemctl daemon-reload && systemctl enable --now etcd
[root@k8s-etcd02 ~]# systemctl daemon-reload && systemctl enable --now etcd
[root@k8s-etcd03 ~]# systemctl daemon-reload && systemctl enable --now etcd

[root@k8s-etcd01 pki]# systemctl status etcd
● etcd.service - Etcd Service
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2022-02-25 15:45:24 CST; 12s ago
     Docs: https://coreos.com/etcd/docs/latest/
 Main PID: 12347 (etcd)
   CGroup: system.slice/etcd.service
           └─12347 usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml

Feb 25 15:45:26 k8s-etcd01.example.local etcd[12347]: health check for peer 209a1f57c506dba2 could not connect: dial tcp 172.31.3.110:...efused
Feb 25 15:45:26 k8s-etcd01.example.local etcd[12347]: health check for peer 209a1f57c506dba2 could not connect: dial tcp 172.31.3.110:...efused
Feb 25 15:45:26 k8s-etcd01.example.local etcd[12347]: peer 209a1f57c506dba2 became active
Feb 25 15:45:26 k8s-etcd01.example.local etcd[12347]: established a TCP streaming connection with peer 209a1f57c506dba2 (stream MsgApp...riter)
Feb 25 15:45:26 k8s-etcd01.example.local etcd[12347]: established a TCP streaming connection with peer 209a1f57c506dba2 (stream Message writer)
Feb 25 15:45:26 k8s-etcd01.example.local etcd[12347]: established a TCP streaming connection with peer 209a1f57c506dba2 (stream MsgApp...eader)
Feb 25 15:45:26 k8s-etcd01.example.local etcd[12347]: established a TCP streaming connection with peer 209a1f57c506dba2 (stream Message reader)
Feb 25 15:45:28 k8s-etcd01.example.local etcd[12347]: updating the cluster version from 3.0 to 3.4
Feb 25 15:45:28 k8s-etcd01.example.local etcd[12347]: updated the cluster version from 3.0 to 3.4
Feb 25 15:45:28 k8s-etcd01.example.local etcd[12347]: enabled capabilities for version 3.4
Hint: Some lines were ellipsized, use -l to show in full.

[root@k8s-etcd02 ~]# systemctl status etcd
[root@k8s-etcd03 ~]# systemctl status etcd
复制

查看etcd状态

[root@k8s-etcd01 pki]# export ETCDCTL_API=3

[root@k8s-etcd01 pki]# etcdctl --endpoints="172.31.3.108:2379,172.31.3.109:2379,172.31.3.110:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 172.31.3.108:2379 | a9fef56ff96ed75c |  3.4.13 |   20 kB |      true |      false |         3 |          9 |                  9 |        |
| 172.31.3.109:2379 | 8319ef09e8b3d277 |  3.4.13 |   20 kB |     false |      false |         3 |          9 |                  9 |        |
| 172.31.3.110:2379 | 209a1f57c506dba2 |  3.4.13 |   20 kB |     false |      false |         3 |          9 |                  9 |        |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
复制

重要：如果安装了keepalived和haproxy，需要测试keepalived是否是正常的

#测试VIP
[root@k8s-master01 pki]# ping 172.31.3.188
PING 172.31.3.188 (172.31.3.188) 56(84) bytes of data.
64 bytes from 172.31.3.188: icmp_seq=1 ttl=64 time=1.27 ms
64 bytes from 172.31.3.188: icmp_seq=2 ttl=64 time=0.585 ms
^C
--- 172.31.3.188 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.585/0.931/1.277/0.346 ms

[root@k8s-ha01 ~]# systemctl stop keepalived
[root@k8s-ha01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:05:9b:2a brd ff:ff:ff:ff:ff:ff
    inet 172.31.3.104/21 brd 172.31.7.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe05:9b2a/64 scope link 
       valid_lft forever preferred_lft forever
 
 [root@k8s-ha02 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:5e:d8:f8 brd ff:ff:ff:ff:ff:ff
    inet 172.31.3.105/21 brd 172.31.7.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.31.3.188/32 scope global eth0:1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe5e:d8f8/64 scope link 
       valid_lft forever preferred_lft forever
 
 [root@k8s-ha01 ~]# systemctl start keepalived
 [root@k8s-ha01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:05:9b:2a brd ff:ff:ff:ff:ff:ff
    inet 172.31.3.104/21 brd 172.31.7.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.31.3.188/32 scope global eth0:1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe05:9b2a/64 scope link 
       valid_lft forever preferred_lft forever

[root@k8s-master01 pki]# telnet 172.31.3.188 6443
Trying 172.31.3.188...
Connected to 172.31.3.188.
Escape character is '^]'.
Connection closed by foreign host.
复制

如果ping不通且telnet没有出现 ] ，则认为VIP不可以，不可在继续往下执行，需要排查keepalived的问题，比如防火墙和selinux，haproxy和keepalived的状态，监听端口等

所有节点查看防火墙状态必须为disable和inactive：systemctl status firewalld

所有节点查看selinux状态，必须为disable：getenforce

master节点查看haproxy和keepalived状态：systemctl status keepalived haproxy

master节点查看监听端口：netstat -lntp

7.部署docker

master和node安装docker-ce：

[root@k8s-master01 ~]# cat install_docker_binary.sh
#!/bin/bash
#
#**********************************************************************************************
#Author:        Raymond
#QQ:            88563128
#Date:          2021-12-07
#FileName:      install_docker_binary.sh
#URL:           raymond.blog.csdn.net
#Description:   install_docker_binary for centos 7/8 & ubuntu 18.04/20.04 & Rocky 8
#Copyright (C): 2021 All rights reserved
#*********************************************************************************************
SRC_DIR=/usr/local/src
COLOR="echo -e \\033[01;31m"
END='\033[0m'
URL='https://mirrors.cloud.tencent.com/docker-ce/linux/static/stable/x86_64/'
DOCKER_FILE=docker-19.03.9.tgz
HARBOR_DOMAIN=harbor.raymonds.cc

os(){
    OS_ID=`sed -rn '/^NAME=/s@.*="([[:alpha:]]+).*"$@\1@p' etc/os-release`
}

check_file (){
    cd ${SRC_DIR}
    rpm -q wget &> dev/null || yum -y install wget &> dev/null
    if [ ! -e ${DOCKER_FILE} ];then
        ${COLOR}"缺少${DOCKER_FILE}文件,如果是离线包,请把文件放到${SRC_DIR}目录下"${END}
        ${COLOR}'开始下载DOCKER二进制安装包'${END}
        wget ${URL}${DOCKER_FILE} || { ${COLOR}"DOCKER二进制安装包下载失败"${END}; exit; } 
    else
        ${COLOR}"相关文件已准备好"${END}
    fi
}

install(){ 
    [ -f usr/bin/docker ] && { ${COLOR}"DOCKER已存在，安装失败"${END};exit; }
    ${COLOR}"开始安装DOCKER..."${END}
    tar xf ${DOCKER_FILE} 
    mv docker/* usr/bin/
    cat > lib/systemd/system/docker.service <<-EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H unix://var/run/docker.sock
ExecReload=/bin/kill -s HUP \$MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target
EOF
    mkdir -p etc/docker
    cat > etc/docker/daemon.json <<-EOF
{
    "registry-mirrors": [
        "https://registry.docker-cn.com",
        "http://hub-mirror.c.163.com",
        "https://docker.mirrors.ustc.edu.cn"
    ],
    "insecure-registries": ["${HARBOR_DOMAIN}"],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "max-concurrent-downloads": 10,
    "max-concurrent-uploads": 5,
    "log-opts": {
        "max-size": "300m",
        "max-file": "2"  
    },
    "live-restore": true
}
EOF
    echo 'alias rmi="docker images -qa|xargs docker rmi -f"' >> ~/.bashrc
    echo 'alias rmc="docker ps -qa|xargs docker rm -f"' >> ~/.bashrc
    systemctl daemon-reload
    systemctl enable --now docker &> dev/null
    systemctl is-active docker &> dev/null && ${COLOR}"Docker 服务启动成功"${END} || { ${COLOR}"Docker 启动失败"${END};exit; }
    docker version && ${COLOR}"Docker 安装成功"${END} || ${COLOR}"Docker 安装失败"${END}
}

set_swap_limit(){
    if [ ${OS_ID} == "Ubuntu" ];then
        ${COLOR}'设置Docker的"WARNING: No swap limit support"警告'${END}
        sed -ri '/^GRUB_CMDLINE_LINUX=/s@"$@ swapaccount=1"@' etc/default/grub
        update-grub &> dev/null
        ${COLOR}"10秒后，机器会自动重启"${END}
        sleep 10
        reboot
    fi
}

main(){
    os
    check_file
    install
    set_swap_limit
}

main

[root@k8s-master01 ~]# bash install_docker_binary.sh

[root@k8s-master02 ~]# bash install_docker_binary.sh

[root@k8s-master03 ~]# bash install_docker_binary.sh

[root@k8s-node01 ~]# bash install_docker_binary.sh

[root@k8s-node02 ~]# bash install_docker_binary.sh

[root@k8s-node03 ~]# bash install_docker_binary.sh
复制

8.部署master

8.1 创建etcd相关目录和复制etcd证书

master节点创建etcd证书目录

[root@k8s-master01 ~]# mkdir etc/etcd/ssl -p
[root@k8s-master02 ~]# mkdir etc/etcd/ssl -p
[root@k8s-master03 ~]# mkdir etc/etcd/ssl -p
复制

将etcd证书复制到master节点

[root@k8s-etcd01 pki]# for NODE in k8s-master01 k8s-master02 k8s-master03; do
     ssh -o StrictHostKeyChecking=no $NODE "mkdir -p etc/etcd/ssl"
     for FILE in etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem; do
       scp -o StrictHostKeyChecking=no etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}
     done
 done
复制

所有master节点创建etcd的证书目录

[root@k8s-master01 ~]# mkdir etc/kubernetes/pki/etcd -p
[root@k8s-master02 ~]# mkdir etc/kubernetes/pki/etcd -p
[root@k8s-master03 ~]# mkdir etc/kubernetes/pki/etcd -p

[root@k8s-master01 ~]# ln -s etc/etcd/ssl/* etc/kubernetes/pki/etcd/
[root@k8s-master02 ~]# ln -s etc/etcd/ssl/* etc/kubernetes/pki/etcd/
[root@k8s-master03 ~]# ln -s etc/etcd/ssl/* etc/kubernetes/pki/etcd/
复制

8.2 安装kubernetes组件

下载kubernetes安装包

[root@k8s-master01 ~]# wget https://dl.k8s.io/v1.20.14/kubernetes-server-linux-amd64.tar.gz
复制

需要下载最新的1.20.x版本：

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#server-binaries
复制

打开页面后点击：

解压kubernetes安装文件

[root@k8s-master01 ~]# tar -xf kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}
复制

版本查看

[root@k8s-master01 ~]# kubelet --version
Kubernetes v1.20.14
复制

将组件发送到其他master节点

[root@k8s-master01 ~]# for NODE in k8s-master02 k8s-master03; do echo $NODE; scp -o StrictHostKeyChecking=no usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; done
复制

master节点创建/opt/cni/bin目录

[root@k8s-master01 ~]# mkdir -p opt/cni/bin
[root@k8s-master02 ~]# mkdir -p opt/cni/bin
[root@k8s-master03 ~]# mkdir -p opt/cni/bin
复制

8.3 生成k8s组件证书

二进制安装最关键步骤，一步错误全盘皆输，一定要注意每个步骤都要是正确的

master节点创建kubernetes相关目录

[root@k8s-master01 ~]# mkdir -p etc/kubernetes/pki
[root@k8s-master02 ~]# mkdir -p etc/kubernetes/pki
[root@k8s-master03 ~]# mkdir -p etc/kubernetes/pki
复制

master01下载生成证书工具

[root@k8s-master01 ~]# wget "https://pkg.cfssl.org/R1.2/cfssl_linux-amd64" -O usr/local/bin/cfssl

[root@k8s-master01 ~]# wget "https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64" -O usr/local/bin/cfssljson

[root@k8s-master01 ~]# chmod +x usr/local/bin/cfssl usr/local/bin/cfssljson
复制

8.3.1 生成ca证书

[root@k8s-master01 ~]# mkdir pki
[root@k8s-master01 ~]# cd pki/

[root@k8s-master01 pki]# cat ca-csr.json
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}

[root@k8s-master01 pki]# cfssl gencert -initca ca-csr.json | cfssljson -bare etc/kubernetes/pki/ca
#执行结果
2022/02/25 16:19:52 [INFO] generating a new CA key and certificate from CSR
2022/02/25 16:19:52 [INFO] generate received request
2022/02/25 16:19:52 [INFO] received CSR
2022/02/25 16:19:52 [INFO] generating key: rsa-2048
2022/02/25 16:19:52 [INFO] encoded CSR
2022/02/25 16:19:52 [INFO] signed certificate with serial number 50700045204155982111782984381054779655420622936

[root@k8s-master01 pki]# ll etc/kubernetes/pki/ca*
-rw-r--r-- 1 root root 1025 Feb 25 16:19 etc/kubernetes/pki/ca.csr
-rw------- 1 root root 1675 Feb 25 16:19 etc/kubernetes/pki/ca-key.pem
-rw-r--r-- 1 root root 1411 Feb 25 16:19 etc/kubernetes/pki/ca.pem
复制

8.3.2 生成apiserver证书

# 10.96.0.是k8s service的网段，如果说需要更改k8s service网段，那就需要更改10.96.0.1，

# 如果不是高可用集群，172.31.3.188为Master01的IP

[root@k8s-master01 pki]# cat ca-config.json 
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}

[root@k8s-master01 pki]# cat apiserver-csr.json 
{
  "CN": "kube-apiserver",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "Kubernetes",
      "OU": "Kubernetes-manual"
    }
  ]
}

[root@k8s-master01 pki]# cfssl gencert   -ca=/etc/kubernetes/pki/ca.pem   -ca-key=/etc/kubernetes/pki/ca-key.pem   -config=ca-config.json   -hostname=10.96.0.1,172.31.3.188,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,172.31.3.101,172.31.3.102,172.31.3.103 -profile=kubernetes   apiserver-csr.json | cfssljson -bare etc/kubernetes/pki/apiserver
#执行结果
2022/02/25 16:21:42 [INFO] generate received request
2022/02/25 16:21:42 [INFO] received CSR
2022/02/25 16:21:42 [INFO] generating key: rsa-2048
2022/02/25 16:21:42 [INFO] encoded CSR
2022/02/25 16:21:42 [INFO] signed certificate with serial number 14326114816925312981811634565226925868722808544

[root@k8s-master01 pki]# ll etc/kubernetes/pki/apiserver*
-rw-r--r-- 1 root root 1029 Feb 25 16:21 etc/kubernetes/pki/apiserver.csr
-rw------- 1 root root 1675 Feb 25 16:21 etc/kubernetes/pki/apiserver-key.pem
-rw-r--r-- 1 root root 1692 Feb 25 16:21 etc/kubernetes/pki/apiserver.pem
复制

8.3.3 生成apiserver的聚合证书

生成apiserver的聚合证书。Requestheader-client-xxx   requestheader-allowwd-xxx:aggerator

[root@k8s-master01 pki]# cat front-proxy-ca-csr.json
{
  "CN": "kubernetes",
  "key": {
     "algo": "rsa",
     "size": 2048
  }
}

[root@k8s-master01 pki]# cfssl gencert   -initca front-proxy-ca-csr.json | cfssljson -bare etc/kubernetes/pki/front-proxy-ca
#执行结果
2022/02/25 16:23:07 [INFO] generating a new CA key and certificate from CSR
2022/02/25 16:23:07 [INFO] generate received request
2022/02/25 16:23:07 [INFO] received CSR
2022/02/25 16:23:07 [INFO] generating key: rsa-2048
2022/02/25 16:23:07 [INFO] encoded CSR
2022/02/25 16:23:07 [INFO] signed certificate with serial number 60929528331736839052879833998406013639330884564

[root@k8s-master01 pki]# ll etc/kubernetes/pki/front-proxy-ca*
-rw-r--r-- 1 root root  891 Feb 25 16:23 etc/kubernetes/pki/front-proxy-ca.csr
-rw------- 1 root root 1679 Feb 25 16:23 etc/kubernetes/pki/front-proxy-ca-key.pem
-rw-r--r-- 1 root root 1143 Feb 25 16:23 etc/kubernetes/pki/front-proxy-ca.pem

[root@k8s-master01 pki]# cat ca-config.json 
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}

[root@k8s-master01 pki]# cat front-proxy-client-csr.json
{
  "CN": "front-proxy-client",
  "key": {
     "algo": "rsa",
     "size": 2048
  }
}

[root@k8s-master01 pki]# cfssl gencert   -ca=/etc/kubernetes/pki/front-proxy-ca.pem   -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem   -config=ca-config.json   -profile=kubernetes   front-proxy-client-csr.json | cfssljson -bare etc/kubernetes/pki/front-proxy-client
#返回结果（忽略警告）
2022/02/25 16:24:29 [INFO] generate received request
2022/02/25 16:24:29 [INFO] received CSR
2022/02/25 16:24:29 [INFO] generating key: rsa-2048
2022/02/25 16:24:29 [INFO] encoded CSR
2022/02/25 16:24:29 [INFO] signed certificate with serial number 625247142890350892356758545319462713918431205897
2022/02/25 16:24:29 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

[root@k8s-master01 pki]# ll etc/kubernetes/pki/front-proxy-client*
-rw-r--r-- 1 root root  903 Feb 25 16:24 etc/kubernetes/pki/front-proxy-client.csr
-rw------- 1 root root 1675 Feb 25 16:24 etc/kubernetes/pki/front-proxy-client-key.pem
-rw-r--r-- 1 root root 1188 Feb 25 16:24 etc/kubernetes/pki/front-proxy-client.pem
复制