大家好，很久没写文章了，最近一直很忙，也没多少思路，重要的是没有多少时间，就空了一段日子。长话短说，今天这篇我们聊下tiup在做check 的时候，执行了那些系统的命令。这也是一个客户的问题。因为他在执行的过程中总是报fail，所以他问：“tiup check 后台究竟执行了什么操作系统指令，我如何复现“。正因为他的问题才有了这篇文章。
我们首先执行 tiup check 的操作
[root@xxx.xx.xxx.210 ~]# tiup -v
1.10.2 tiup
Go Version: go1.18.3
Git Ref: v1.10.2
GitHash: 2de5b500c9fae6d418fa200ca150b8d5264d6b19
Node Check Result Message
---- ----- ------ -------
xxx.xx.xxx.210 cpu-governor Warn Unable to determine current CPU frequency governor policy
xxx.xx.xxx.210 memory Pass memory size is 16384MB
xxx.xx.xxx.210 disk Warn mount point / does not have 'noatime' option set
xxx.xx.xxx.210 selinux Pass SELinux is disabled
xxx.xx.xxx.210 timezone Pass time zone is the same as the first PD machine: Asia/Shanghai
xxx.xx.xxx.210 os-version Pass OS is CentOS Linux 7 (Core) 7.6.1810
xxx.xx.xxx.210 cpu-cores Pass number of CPU cores / threads: 8
xxx.xx.xxx.210 disk Fail mount point / does not have 'nodelalloc' option set
xxx.xx.xxx.210 thp Pass THP is disabled
xxx.xx.xxx.210 command Fail numactl not usable, bash: numactl: command not found
xxx.xx.xxx.125 sysctl Fail vm.swappiness = 30, should be 0
xxx.xx.xxx.125 selinux Pass SELinux is disabled
xxx.xx.xxx.125 os-version Pass OS is CentOS Linux 7 (Core) 7.6.1810
xxx.xx.xxx.125 cpu-cores Pass number of CPU cores / threads: 8
xxx.xx.xxx.125 cpu-governor Warn Unable to determine current CPU frequency governor policy
xxx.xx.xxx.125 memory Pass memory size is 16384MB
xxx.xx.xxx.125 disk Fail mount point / does not have 'nodelalloc' option set
xxx.xx.xxx.125 disk Warn mount point / does not have 'noatime' option set
xxx.xx.xxx.125 thp Fail THP is enabled, please disable it for best performance
xxx.xx.xxx.125 command Fail numactl not usable, bash: numactl: command not found
xxx.xx.xxx.125 service Fail service irqbalance is not running
xxx.xx.xxx.130 cpu-cores Pass number of CPU cores / threads: 8
xxx.xx.xxx.130 cpu-governor Warn Unable to determine current CPU frequency governor policy
xxx.xx.xxx.130 disk Fail mount point / does not have 'nodelalloc' option set
xxx.xx.xxx.130 service Fail service irqbalance is not running
xxx.xx.xxx.130 command Fail numactl not usable, bash: numactl: command not found
xxx.xx.xxx.130 timezone Pass time zone is the same as the first PD machine: Asia/Shanghai
xxx.xx.xxx.130 os-version Pass OS is CentOS Linux 7 (Core) 7.6.1810
xxx.xx.xxx.130 memory Pass memory size is 16384MB
xxx.xx.xxx.130 disk Warn mount point / does not have 'noatime' option set
xxx.xx.xxx.130 sysctl Fail vm.swappiness = 30, should be 0
xxx.xx.xxx.130 selinux Pass SELinux is disabled
xxx.xx.xxx.130 thp Fail THP is enabled, please disable it for best performance复制
现在我们要分析的是，它究竟做了哪些检查？
我们首先可以运行 tiup cluster audit，寻找到我们刚刚执行tiup记录的id，然后运行tiup cluster audit xxx，就能输出tiup cluster check 的 debug 日志信息。输出的信息还是太多了，我们需要进一步过滤，可以执行tiup cluster audit fTWwJBgz1xC | grep -i TaskBegin | grep -i command
来把tiup下发的任务命令都过滤出来，由于篇幅省略部分信息，如下所示：
2022-07-07T14:21:35 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.125, sudo=false, command=`uname -m`"}
2022-07-07T14:21:35 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.210, sudo=false, command=`uname -m`"}
2022-07-07T14:21:35 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.130, sudo=false, command=`uname -m`"}
2022-07-07T14:21:35 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.130, sudo=false, command=`uname -s`"}
2022-07-07T14:21:35 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.210, sudo=false, command=`uname -s`"}
2022-07-07T14:21:35 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.125, sudo=false, command=`uname -s`"}
2022-07-07T14:21:37 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.210, sudo=false, command=`/tmp/tiup/bin/insight`"}
2022-07-07T14:21:37 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.125, sudo=false, command=`/tmp/tiup/bin/insight`"}
2022-07-07T14:21:37 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.130, sudo=false, command=`/tmp/tiup/bin/insight`"}
2022-07-07T14:21:38 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.125, sudo=false, command=`cat /etc/security/limits.conf`"}
2022-07-07T14:21:38 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.210, sudo=false, command=`cat /etc/security/limits.conf`"}
2022-07-07T14:21:38 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.130, sudo=false, command=`cat /etc/security/limits.conf`"}
2022-07-07T14:21:38 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.125, sudo=true, command=`sysctl -a`"}
2022-07-07T14:21:39 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.210, sudo=true, command=`sysctl -a`"}
2022-07-07T14:21:39 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.130, sudo=true, command=`sysctl -a`"}
2022-07-07T14:21:40 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.125, sudo=false, command=`ss -lnt`"}
2022-07-07T14:21:40 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.130, sudo=false, command=`ss -lnt`"}
2022-07-07T14:21:40 TaskBegin {"task": "Shell: host=xx.xx.xx.xx.210, sudo=false, command=`ss -lnt`"}复制
我们稍微整理了一下，执行了以下的命令：
uname -m 检查操作系统的架构。 uname -s 检查操作系统。 下发 insight 软件包，执行 insight 收集信息。 cat etc/security/limits.conf 检查操作系统limits信息。 sysctl -a 检查操作系统参数。 ss -lnt 检查端口是否被占用。
当然你会觉得很奇怪，单纯从日志上来看，像时间服务、磁盘挂载参数这些好像是没有检查过的。所以我们从源码上在研究一下。
首先我们下载最新tiup的代码，我这里基于的是1.10.2版本，从cluster/operation/check.go
上，可以看到需要检查的项目。
// Names of checks
var (
CheckNameGeneral = "general" // errors that don't fit any specific check
CheckNameNTP = "ntp"
CheckNameChrony = "chrony"
CheckNameOSVer = "os-version"
CheckNameSwap = "swap"
CheckNameSysctl = "sysctl"
CheckNameCPUThreads = "cpu-cores"
CheckNameCPUGovernor = "cpu-governor"
CheckNameDisks = "disk"
CheckNamePortListen = "listening-port"
CheckNameEpoll = "epoll-exclusive"
CheckNameMem = "memory"
CheckNameNet = "network"
CheckNameLimits = "limits"
CheckNameSysService = "service"
CheckNameSELinux = "selinux"
CheckNameCommand = "command"
CheckNameFio = "fio"
CheckNameTHP = "thp"
CheckNameDirPermission = "permission"
CheckNameDirExist = "exist"
CheckNameTimeZone = "timezone"
)复制
这些检查项目比我们从日志观察到的要更多。我们来逐步解析一下。
CheckNameGeneral
从代码段可以看到，似乎是把 insightInfo 的信息转换成 json 格式。
var insightInfo insight.InsightInfo
if err := json.Unmarshal(rawData, &insightInfo); err != nil {
return append(results, &CheckResult{
Name: CheckNameGeneral,
Err: err,
})
}复制
insightInfo 信息又是什么？如果你仔细观察 debug 日志信息，你就会发现 tiup 会到各个节点下发 insight 软件，执行完成获得结果之后再删除 insight 软件。这个是一个很隐蔽的行为。insight 位于下面的路径下。
/.tiup/storage/cluster/packages/insight-v0.4.1-linux-amd64.tar.gz复制
可以手工解压运行一下，此处省略部分篇幅，运行结果如下：
[root@vmxxx packages]# ./insight
{
"meta": {
"timestamp": "2022-07-06T15:48:15.636854758+08:00",
"uptime": 4226411.51,
"idle_time": 33456031.63,
"sysinfo_ver": "0.9.4",
"git_branch": "tiup",
"git_commit": "v0.4.1-1-g5a79850",
"go_version": "go1.17 linux/amd64",
"tidb": null,
"tikv": null,
"pd": null
},
"sysinfo": {
"node": {
"hostname": "vm172-16-201-125",
"machineid": "9d99de235189433f954998cc2b95cbe6",
"hypervisor": "kvm",
"timezone": "Asia/Shanghai"
},
"os": {
"name": "CentOS Linux 7 (Core)",
"vendor": "centos",
"version": "7",
"release": "7.6.1810",
"architecture": "amd64"
},
"kernel": {
"release": "3.10.0-1160.53.1.el7.x86_64",
"version": "#1 SMP Fri Jan 14 13:59:45 UTC 2022",
"architecture": "x86_64"
}复制
这里可以看到它会取出操作系统的若干信息，上面的 CheckNameGeneral，其实就是执行了 JSON 解码函数 Unmarshal。
CheckNameNTP
func checkNTP(ntpInfo *insight.TimeStat) *CheckResult {
result := &CheckResult{
Name: CheckNameNTP,
}
if ntpInfo.Status == "none" {
zap.L().Info("The NTPd daemon may be not installed, skip.")
return result
}
if ntpInfo.Sync == "none" {
result.Err = fmt.Errorf("The NTPd daemon may be not start")
result.Warn = true
return result
}
// check if time offset greater than +- 500ms
if math.Abs(ntpInfo.Offset) >= 500 {
result.Err = fmt.Errorf("time offset %fms too high", ntpInfo.Offset)
}
return result
}复制
检查NTP信息，这里会从 insight 输出的信息，去判断 NTPd 守护进程有没有启动，如果没启动则会报错，同样它还会检查 ntp 当前的超时时间，如果超出500ms 也会报错。
CheckNameChrony
func checkChrony(chronyInfo *insight.ChronyStat) *CheckResult {
result := &CheckResult{
Name: CheckNameChrony,
}
if chronyInfo.LeapStatus == "none" {
zap.L().Info("The Chrony daemon may be not installed, skip.")
return result
}
// check if time offset greater than +- 500ms
if math.Abs(chronyInfo.LastOffset) >= 500 {
result.Err = fmt.Errorf("time offset %fms too high", chronyInfo.LastOffset)
}
return result
}复制
同上面类似，这里会从 insight 输出的信息去检查 chrony 当前的超时时间，如果超出500ms 也会报错。在时间这个地方，它使用了 uber 的开源日志组件 zap，对于软件没安装，它会记录到日志中。
当然我们可能会有疑问，Ntp 和 Chrony，都是时间服务，是不是二选一就行了。这部分的代码体现在CheckSystemInfo
函数里面。
// check time sync status
switch {
case insightInfo.ChronyStat.LeapStatus != "none":
results = append(results, checkChrony(&insightInfo.ChronyStat))
case insightInfo.NTP.Status != "none":
results = append(results, checkNTP(&insightInfo.NTP))
default:
results = append(results,
&CheckResult{
Name: CheckNameNTP,
Err: fmt.Errorf("The NTPd daemon or Chronyd daemon may be not installed"),
Warn: true,
},
)
}复制
CheckNameOSVer
在官方文档中，当前只支持Reahat、Centos、Oracle Linux、Amazon Linux (6.0以上版本)、Ubuntu。
func checkOSInfo(opt *CheckOptions, osInfo *sysinfo.OS) *CheckResult {
result := &CheckResult{
Name: CheckNameOSVer,
Msg: fmt.Sprintf("OS is %s %s", osInfo.Name, osInfo.Release),
}
// check OS vendor
switch osInfo.Vendor {
case "kylin":
msg := "kylin support is not fully tested, be careful"
result.Err = fmt.Errorf("%s (%s)", result.Msg, msg)
result.Warn = true
// VERSION_ID="V10"
if ver, _ := strconv.ParseFloat(strings.Trim(osInfo.Version, "V"), 64); ver < 10 {
result.Err = fmt.Errorf("%s %s not supported, use version V10 or higher(%s)",
osInfo.Name, osInfo.Release, msg)
return result
}
case "amzn":
// Amazon Linux 2 is based on CentOS 7 and is recommended for
// AWS Graviton 2 (ARM64) deployments.
if ver, _ := strconv.ParseFloat(osInfo.Version, 64); ver < 2 || ver >= 3 {
result.Err = fmt.Errorf("%s %s not supported, use version 2 please",
osInfo.Name, osInfo.Release)
return result
}
case "centos", "redhat", "rhel", "ol":
// check version
// CentOS 8 is known to be not working, and we don't have plan to support it
// as of now, we may add support for RHEL 8 based systems in the future.
if ver, _ := strconv.ParseFloat(osInfo.Version, 64); ver < 7 {
result.Err = fmt.Errorf("%s %s not supported, use version 8 please",
osInfo.Name, osInfo.Release)
return result
}
------省略-------复制
现在国产化很火热，所以我们在源码中也看到了对 kylin 操作系统的一些判断。目前是"kylin support is not fully tested, be careful"。也就是官方并没有完整的进行过测试。但是从很多网友的使用上看，似乎没有什么问题。
对于使用最多的"centos", "redhat" 系列，会检查你的版本，小于7会 not supported。但是如果你是centos 8，它也不会报错。但是这个内容在官方文档是有说明的。
不计划支持 CentOS 8 Linux，因为 CentOS 的上游支持已于 2021 年 12 月 31 日终止。
我们的操作系统的版本信息也是来源于 insight 。
CheckNameSwap && CheckNameMem
func checkMem(opt *CheckOptions, memInfo *sysinfo.Memory) []*CheckResult {
var results []*CheckResult
if memInfo.Swap > 0 {
results = append(results, &CheckResult{
Name: CheckNameSwap,
Warn: true,
Err: fmt.Errorf("swap is enabled, please disable it for best performance"),
})
}
// 32GB
if opt.EnableMem && memInfo.Size < 1024*32 {
results = append(results, &CheckResult{
Name: CheckNameMem,
Err: fmt.Errorf("memory size %dMB too low, needs 32GB or more", memInfo.Size),
})
} else {
results = append(results, &CheckResult{
Name: CheckNameMem,
Msg: fmt.Sprintf("memory size is %dMB", memInfo.Size),
})
}
return results
}复制
CheckNameSwap 和 CheckNameMem 这两个检查项是在同一个函数 checkMem 中检查的。这部分内容也是通过 insight 来收集的。如果检查到 swap > 0 ，会提示 swap is enabled，please disable it
。如果开启了内存大小校验，也就是 --enable-mem 。它会判断内存是否大于32GB。默认在 tiup 执行 check 检查的时候，几乎很少人会制定--enable-mem。
CheckNameSysctl
// CheckKernelParameters checks kernel parameter values
func CheckKernelParameters(opt *CheckOptions, p []byte) []*CheckResult {
var results []*CheckResult
for _, line := range strings.Split(string(p), "\n") {
line = strings.TrimSpace(line)
fields := strings.Fields(line)
if len(fields) < 3 {
continue
}
switch fields[0] {
case "fs.file-max":
val, _ := strconv.Atoi(fields[2])
if val < 1000000 {
results = append(results, &CheckResult{
Name: CheckNameSysctl,
Err: fmt.Errorf("fs.file-max = %d, should be greater than 1000000", val),
Msg: "fs.file-max = 1000000",
})
}
case "net.core.somaxconn":
val, _ := strconv.Atoi(fields[2])
if val < 32768 {
results = append(results, &CheckResult{
Name: CheckNameSysctl,
Err: fmt.Errorf("net.core.somaxconn = %d, should be greater than 32768", val),
Msg: "net.core.somaxconn = 32768",
})
}
case "net.ipv4.tcp_tw_recycle":
val, _ := strconv.Atoi(fields[2])
if val != 0 {
results = append(results, &CheckResult{
Name: CheckNameSysctl,
Err: fmt.Errorf("net.ipv4.tcp_tw_recycle = %d, should be 0", val),
Msg: "net.ipv4.tcp_tw_recycle = 0",
})
}
------省略-------复制
内核参数这部分检查主要是在 CheckKernelParameters
函数里面。从前面 debug 的日志，我们可以知道它是通过 sysctl -a 命令进行收集的，然后主要对下列参数进行判断。
fs.file-max 检查小于 1000000，会报错。
net.core.somaxconn 检查小于 32768，会报错。
net.ipv4.tcp_tw_recycle 检查不等于 0 会报错。
net.ipv4.tcp_syncookies 检查不等于0 会报错。
vm.overcommit_memory 需要开启 --enable-mem ，检查不等于 0 或者 1 会报错。
vm.swappiness 检查不等于0会报错。
CheckNameCPUThreads && CheckNameCPUGovernor
func checkCPU(opt *CheckOptions, cpuInfo *sysinfo.CPU) []*CheckResult {
var results []*CheckResult
if opt.EnableCPU && cpuInfo.Threads < 16 {
results = append(results, &CheckResult{
Name: CheckNameCPUThreads,
Err: fmt.Errorf("CPU thread count %d too low, needs 16 or more", cpuInfo.Threads),
})
} else {
results = append(results, &CheckResult{
Name: CheckNameCPUThreads,
Msg: fmt.Sprintf("number of CPU cores / threads: %d", cpuInfo.Threads),
})
}
// check for CPU frequency governor
if cpuInfo.Governor != "" {
if cpuInfo.Governor != "performance" {
results = append(results, &CheckResult{
Name: CheckNameCPUGovernor,
Err: fmt.Errorf("CPU frequency governor is %s, should use performance", cpuInfo.Governor),
})
} else {
results = append(results, &CheckResult{
Name: CheckNameCPUGovernor,
Msg: fmt.Sprintf("CPU frequency governor is %s", cpuInfo.Governor),
})
}
} else {
results = append(results, &CheckResult{
Name: CheckNameCPUGovernor,
Err: fmt.Errorf("Unable to determine current CPU frequency governor policy"),
Warn: true,
})
}
return results
}复制
cpu 的信息也是通过 insight 软件来收集的。CheckNameCPUThreads 检查项需要开启 --enable-cpu ，开启后会检查 cpu 的核数，小于 16 核会报错，否则只会正常的输出 cpu 的核数。CheckNameCPUGovernor
会检查 cpu 的节能策略，需要调整成 performance 模式，如果是空或者不是 performance 模式，就会警告。
CheckNameDisks
// only check for TiKV and TiFlash, other components are not that I/O sensitive
switch inst.ComponentName() {
case spec.ComponentTiKV,
spec.ComponentTiFlash:
usKey := fmt.Sprintf("%s:%s", host, blk.Mount.MountPoint)
uniqueStores[usKey] = append(uniqueStores[usKey], storePartitionInfo{
comp: inst.ComponentName(),
path: dataDir,
})
}
switch blk.Mount.FSType {
case "ext4":
if !strings.Contains(blk.Mount.Options, "nodelalloc") {
results = append(results, &CheckResult{
Name: CheckNameDisks,
Err: fmt.Errorf("mount point %s does not have 'nodelalloc' option set", blk.Mount.MountPoint),
})
}
fallthrough
case "xfs":
if !strings.Contains(blk.Mount.Options, "noatime") {
results = append(results, &CheckResult{
Name: CheckNameDisks,
Err: fmt.Errorf("mount point %s does not have 'noatime' option set", blk.Mount.MountPoint),
Warn: true,
})
}
default:
results = append(results, &CheckResult{
Name: CheckNameDisks,
Err: fmt.Errorf("mount point %s has an unsupported filesystem '%s'",
blk.Mount.MountPoint, blk.Mount.FSType),
}复制
Disk 的检查项目，只和 Tikv、Tiflash 组件相关。它会检查文件系统状态的类型，如果是ext4，则先会检查挂载是否有 nodelalloc 选项。程序中 switch case 如果顺利匹配则会正常退出。而这里的 ext4 后面加入了 fallthrough ，它仍然会执行下一个 case ，判断是否有 noatime 选项。如果你是 xfs 文件系统，它就只会进入case 中的第二段，判断是否具备 noatime 选项。如果不是 xfs 也不是 ext4，会直接报错unsupported filesytem。
CheckNamePortListen
// CheckListeningPort checks if the ports are already binded by some process on host
func CheckListeningPort(opt *CheckOptions, host string, topo *spec.Specification, rawData []byte) []*CheckResult {
var results []*CheckResult
ports := make(map[int]struct{})
topo.IterInstance(func(inst spec.Instance) {
if inst.GetHost() != host {
return
}
for _, up := range inst.UsedPorts() {
if _, found := ports[up]; !found {
ports[up] = struct{}{}
}
}
})
for p := range ports {
for _, line := range strings.Split(string(rawData), "\n") {
fields := strings.Fields(line)
if len(fields) < 5 || fields[0] != "LISTEN" {
continue
}
addr := strings.Split(fields[3], ":")
lp, _ := strconv.Atoi(addr[len(addr)-1])
if p == lp {
results = append(results, &CheckResult{
Name: CheckNamePortListen,
Err: fmt.Errorf("port %d is already in use", lp),
})
break // ss may report multiple entries for the same port
}
}
}
return results
}复制
检查端口。这里的信息是从首先读拓扑文件中的端口信息，然后执行 ss -lnt 命令去操作系统上获取现有端口，然后检查端口是否被占用。如果占用，就会报 port xx is already in use。
CheckNameEpoll
epollResult := &CheckResult{
Name: CheckNameEpoll,
}
if !insightInfo.EpollExcl {
epollResult.Err = fmt.Errorf("epoll exclusive is not supported")
}
results = append(results, epollResult)复制
这个检查项属于“惊群”概念。对于“惊群”的一般解释是：它降低了多个进程/线程通过 epoll_ctl 添加共享 fd 引发的惊群概率，使得一个事件发生时，只唤醒一个正在 epoll_wait 阻塞等待唤醒的进程/线程（而不是全部唤醒）。如果这个检查项不通过，需要查看下操作系统版本和内核版本。
CheckNameNet
func checkNetwork(opt *CheckOptions, networkDevices []sysinfo.NetworkDevice) []*CheckResult {
var results []*CheckResult
for _, netdev := range networkDevices {
// ignore the network devices that cannot be detected
if netdev.Speed == 0 {
continue
}
if netdev.Speed >= 1000 {
results = append(results, &CheckResult{
Name: CheckNameNet,
Msg: fmt.Sprintf("network speed of %s is %dMB", netdev.Name, netdev.Speed),
})
} else {
results = append(results, &CheckResult{
Name: CheckNameNet,
Err: fmt.Errorf("network speed of %s is %dMB too low, needs 1GB or more", netdev.Name, netdev.Speed),
})
}
}
return results
}复制
网络设备的信息也是通过 insight 软件来收集的，这里判断很简单，如果网卡不是千兆的，就会输出 network speed is too low。如果大于 1000MB，则会直接显示网卡的 speed 信息。
CheckNameLimits
// CheckSysLimits checks limits in /etc/security/limits.conf
func CheckSysLimits(opt *CheckOptions, user string, l []byte) []*CheckResult {
var results []*CheckResult
var (
stackSoft int
nofileSoft int
nofileHard int
)
for _, line := range strings.Split(string(l), "\n") {
line = strings.TrimSpace(line)
if strings.HasPrefix(line, "#") {
continue
}
fields := strings.Fields(line)
if len(fields) < 3 || fields[0] != user {
continue
}
switch fields[2] {
case "nofile":
if fields[1] == "soft" {
nofileSoft, _ = strconv.Atoi(fields[3])
} else {
nofileHard, _ = strconv.Atoi(fields[3])
}
case "stack":
if fields[1] == "soft" {
stackSoft, _ = strconv.Atoi(fields[3])
}
}
}
if nofileSoft < 1000000 {
results = append(results, &CheckResult{
Name: CheckNameLimits,
Err: fmt.Errorf("soft limit of 'nofile' for user '%s' is not set or too low", user),
Msg: fmt.Sprintf("%s soft nofile 1000000", user),
})
}
if nofileHard < 1000000 {
results = append(results, &CheckResult{
Name: CheckNameLimits,
Err: fmt.Errorf("hard limit of 'nofile' for user '%s' is not set or too low", user),
Msg: fmt.Sprintf("%s hard nofile 1000000", user),
})
}
if stackSoft < 10240 {
results = append(results, &CheckResult{
Name: CheckNameLimits,
Err: fmt.Errorf("soft limit of 'stack' for user '%s' is not set or too low", user),
Msg: fmt.Sprintf("%s soft stack 10240", user),
})
}
// all pass
if len(results) < 1 {
results = append(results, &CheckResult{
Name: CheckNameLimits,
})
}
return results
}复制
limits 的检查是通过 cat etc/security/limits.conf 来检查的。从源码上来看做了三项检查。分别是stackSoft、nofileSoft、nofileHard 。检查内容也很简单，就是安装用户的 softnofile，hardnofile 必须大于等于1000000，softstack 必须大于等于10240。
CheckNameSysService
// CheckServices checks if a service is running on the host
func CheckServices(ctx context.Context, e ctxt.Executor, host, service string, disable bool) *CheckResult {
result := &CheckResult{
Name: CheckNameSysService,
}
// check if the service exist before checking its status, ignore when non-exist
stdout, _, err := e.Execute(
ctx,
fmt.Sprintf(
"systemctl list-unit-files --type service | grep -i %s.service | wc -l", service),
true)
if err != nil {
result.Err = err
return result
}
if cnt, _ := strconv.Atoi(strings.Trim(string(stdout), "\n")); cnt == 0 {
if !disable {
result.Err = fmt.Errorf("service %s not found, should be installed and started", service)
}
result.Msg = fmt.Sprintf("service %s not found, ignore", service)
return result
}
active, err := GetServiceStatus(ctx, e, service+".service")
if err != nil {
result.Err = err
}
switch disable {
case false:
if !strings.Contains(active, "running") {
result.Err = fmt.Errorf("service %s is not running", service)
result.Msg = fmt.Sprintf("start %s.service", service)
}
case true:
if strings.Contains(active, "running") {
result.Err = fmt.Errorf("service %s is running but should be stopped", service)
result.Msg = fmt.Sprintf("stop %s.service", service)
}
}
return result
}复制
从源码中我们可以发现服务检查使用了 systemctl list-unit-files --type service
命令，但是我们从官方文档中并没有看到它具体会去检查哪一项服务。
从源码 pkg -> cluster - > task -> check.go 中可以看到，只增加了对 irqbalance
和 firewalld
两个服务的判断。建议是要打开 irqbalance
服务，关闭 firewalld
服务。
// check services
results = append(
results,
operator.CheckServices(ctx, e, c.host, "irqbalance", false),
// FIXME: set firewalld rules in deploy, and not disabling it anymore
operator.CheckServices(ctx, e, c.host, "firewalld", true),
)复制
CheckNameSELinux
// CheckSELinux checks if SELinux is enabled on the host
func CheckSELinux(ctx context.Context, e ctxt.Executor) *CheckResult {
result := &CheckResult{
Name: CheckNameSELinux,
}
m := module.NewShellModule(module.ShellModuleConfig{
// ignore grep errors, the file may not exist for some systems
Command: "grep -E '^\\s*SELINUX=enforcing' /etc/selinux/config 2>/dev/null | wc -l",
Sudo: true,
})
stdout, stderr, err := m.Execute(ctx, e)
if err != nil {
result.Err = fmt.Errorf("%w %s", err, stderr)
return result
}
out := strings.Trim(string(stdout), "\n")
lines, err := strconv.Atoi(out)
if err != nil {
result.Err = fmt.Errorf("can not check SELinux status, please validate manually, %s", err)
result.Warn = true
return result
}
if lines > 0 {
result.Err = fmt.Errorf("SELinux is not disabled")
} else {
result.Msg = "SELinux is disabled"
}
return result
}复制
检查 Selinux 是否关闭，主要使用了操作系统的 grep 命令去检查 /etc/selinux/config
文件中是否存在 SELINUX=enforcing
关键字。如果检查结果不为0 ，则表示 Selinux 没有 disable。
CheckNameCommand
// CheckJRE checks if java command is available for TiSpark nodes
func CheckJRE(ctx context.Context, e ctxt.Executor, host string, topo *spec.Specification) []*CheckResult {
var results []*CheckResult
topo.IterInstance(func(inst spec.Instance) {
if inst.ComponentName() != spec.ComponentTiSpark {
return
}
// check if java cli is available
// the checkpoint part of context can't be shared between goroutines
stdout, stderr, err := e.Execute(checkpoint.NewContext(ctx), "java -version", false)
if err != nil {
results = append(results, &CheckResult{
Name: CheckNameCommand,
Err: fmt.Errorf("java not usable, %s", strings.Trim(string(stderr), "\n")),
Msg: "JRE is not installed properly or not set in PATH",
})
return
}
if len(stderr) > 0 {
// java -version returns as below:
// openjdk version "1.8.0_265"
// openjdk version "11.0.8" 2020-07-14
line := strings.Split(string(stderr), "\n")[0]
fields := strings.Split(line, `"`)
ver := strings.TrimSpace(fields[1])
if strings.Compare(ver, "1.8") < 0 {
results = append(results, &CheckResult{
Name: CheckNameCommand,
Err: fmt.Errorf("java version %s is not supported, use Java 8 (1.8)+", ver),
Msg: "Installed JRE is not Java 8+",
})
} else {
results = append(results, &CheckResult{
Name: CheckNameCommand,
Msg: "java: " + strings.Split(string(stderr), "\n")[0],
})
}
} else {
results = append(results, &CheckResult{
Name: CheckNameCommand,
Err: fmt.Errorf("unknown output of java %s", stdout),
Msg: "java: " + strings.Split(string(stdout), "\n")[0],
Warn: true,
})
}
})复制
这里主要是检查java的版本。这个地方你可能会觉得很纳闷，TiDB 不是golang写的吗 ？为什么会去检查java版本。其实这主要和TiSpark组件有关。如果你的yaml文件中写了TiSpark组件，就会进行Java的检查。这里要求JDK的版本不能低于1.8。
CheckNameFio
// CheckFIOResult parses and checks the result of fio test
func CheckFIOResult(rr, rw, lat []byte) []*CheckResult {
var results []*CheckResult
// check results for rand read test
var rrRes map[string]interface{}
if err := json.Unmarshal(rr, &rrRes); err != nil {
results = append(results, &CheckResult{
Name: CheckNameFio,
Err: fmt.Errorf("error parsing result of random read test, %s", err),
})
} else if jobs, ok := rrRes["jobs"]; ok {
readRes := jobs.([]interface{})[0].(map[string]interface{})["read"]
readIOPS := readRes.(map[string]interface{})["iops"]
results = append(results, &CheckResult{
Name: CheckNameFio,
Msg: fmt.Sprintf("IOPS of random read: %f", readIOPS.(float64)),
})
} else {
results = append(results, &CheckResult{
Name: CheckNameFio,
Err: fmt.Errorf("error parsing result of random read test"),
})
}复制
这个函数的代码略长，这里省略一下输出。要检查Fio，就必须在 tiup cluster check 的后面增加--enable-disk
选项。增加之后，就会使用 fio 工具进行rand read
,rand read write
,read write latency
测试。如下图所示：
CheckTHP
// CheckTHP checks THP in /sys/kernel/mm/transparent_hugepage/{enabled,defrag}
func CheckTHP(ctx context.Context, e ctxt.Executor) *CheckResult {
result := &CheckResult{
Name: CheckNameTHP,
}
m := module.NewShellModule(module.ShellModuleConfig{
Command: fmt.Sprintf(`if [ -d %[1]s ]; then cat %[1]s/{enabled,defrag}; fi`, "/sys/kernel/mm/transparent_hugepage"),
Sudo: true,
})
stdout, stderr, err := m.Execute(ctx, e)
if err != nil {
result.Err = fmt.Errorf("%w %s", err, stderr)
return result
}
for _, line := range strings.Split(strings.Trim(string(stdout), "\n"), "\n") {
if len(line) > 0 && !strings.Contains(line, "[never]") {
result.Err = fmt.Errorf("THP is enabled, please disable it for best performance")
return result
}
}
result.Msg = "THP is disabled"
return result
}复制
检查透明大页是否关闭，这里主要是用的命令是查看/sys/kernel/mm/transparent_hugepage/{defrag,enabled}
这两个文件。然后取出其中的字符串，确认字符串中包含[never]
。[never]
表示透明大页已禁用，否则就是开启的。
CheckNameDirPermission
// CheckDirPermission checks if the user can write to given path
func CheckDirPermission(ctx context.Context, e ctxt.Executor, user, path string) []*CheckResult {
var results []*CheckResult
_, stderr, err := e.Execute(ctx,
fmt.Sprintf(
"/usr/bin/sudo -u %[1]s touch %[2]s/.tiup_cluster_check_file && rm -f %[2]s/.tiup_cluster_check_file",
user,
path,
),
false)
if err != nil || len(stderr) > 0 {
results = append(results, &CheckResult{
Name: CheckNameDirPermission,
Err: fmt.Errorf("unable to write to dir %s: %s", path, strings.Split(string(stderr), "\n")[0]),
Msg: fmt.Sprintf("%s: %s", path, err),
})
} else {
results = append(results, &CheckResult{
Name: CheckNameDirPermission,
Msg: fmt.Sprintf("%s is writable", path),
})
}
return results
}复制
验证某个用户在某个路径下具备读写权限，这里使用操作系统命来去 touch 一个 tiup_cluster_check_file
文件，然后再删除该文件来验证。
CheckNameDirExist
// CheckDirIsExist check if the directory exists
func CheckDirIsExist(ctx context.Context, e ctxt.Executor, path string) []*CheckResult {
var results []*CheckResult
if path == "" {
return results
}
req, _, _ := e.Execute(ctx,
fmt.Sprintf(
"[ -e %s ] && echo 1",
path,
),
false)
if strings.ReplaceAll(string(req), "\n", "") == "1" {
results = append(results, &CheckResult{
Name: CheckNameDirExist,
Err: fmt.Errorf("%s already exists", path),
Msg: fmt.Sprintf("%s already exists", path),
})
}
return results
}复制
CheckNameDirExist 就是检查目录是否存在，如果已经存在，则输出xxx axlready exists
报错。
CheckNameTimeZone
// CheckTimeZone performs checks if time zone is the same
func CheckTimeZone(ctx context.Context, topo *spec.Specification, host string, rawData []byte) []*CheckResult {
var results []*CheckResult
var insightInfo, pd0insightInfo insight.InsightInfo
if err := json.Unmarshal(rawData, &insightInfo); err != nil {
return append(results, &CheckResult{
Name: CheckNameTimeZone,
Err: err,
})
}
if len(topo.PDServers) < 1 {
return append(results, &CheckResult{
Name: CheckNameTimeZone,
Err: fmt.Errorf("no pd found"),
})
}
// skip compare with itself
if topo.PDServers[0].Host == host {
return nil
}
pd0stdout, _, _ := ctxt.GetInner(ctx).GetOutputs(topo.PDServers[0].Host)
if err := json.Unmarshal(pd0stdout, &pd0insightInfo); err != nil {
return append(results, &CheckResult{
Name: CheckNameTimeZone,
Err: err,
})
}
timezone := insightInfo.SysInfo.Node.Timezone
pd0timezone := pd0insightInfo.SysInfo.Node.Timezone
if timezone == pd0timezone {
results = append(results, &CheckResult{
Name: CheckNameTimeZone,
Msg: "time zone is the same as the first PD machine: " + timezone,
})
} else {
results = append(results, &CheckResult{
Name: CheckNameTimeZone,
Err: fmt.Errorf("time zone is %s, but the firt PD is %s", timezone, pd0timezone),
})
}
return results
}复制
最后一个检查项是时区。时区的信息也来源于insight
。这里首先要获取出firt PD
的时区，然后用其他机器的时区和firt PD
进行比较，确认时区是否相同，如果不同则检查会失败。
今天就到这里，这半年事情多，写作机会少，有空就会更新。
文章转载自TiDB之路，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。
评论
相关阅读
国产数据库需要扩大场景覆盖面才能在竞争中更有优势
白鳝的洞穴
273次阅读
2025-04-14 09:40:20
史诗级革新 | Apache Flink 2.0 正式发布
严少安
194次阅读
2025-03-25 00:55:05
TiDB 社区第四届专栏征文大赛联合墨天轮火热开启！TiDB 业务场景实战、运维开发攻略两大赛道，重磅礼品等你来挑战！
墨天轮编辑部
190次阅读
2025-04-15 17:01:41
PingCAP 2025 校园招聘 | 开源领航，星耀前程
PingCAP
125次阅读
2025-03-27 09:55:55
2025w11 | 图数据库了解一下
严少安
81次阅读
2025-03-21 16:51:24
从Oracle和TiDB的HTAP说起
薛晓刚
71次阅读
2025-04-07 19:09:00
最佳实践 | 在 EMR Serverless Spark 中实现 StarRocks 读写操作
阿里云大数据AI技术
65次阅读
2025-03-20 13:52:28
国产数据库如何赋能智慧医疗？3.28 TiDB 与您相约宜昌 “医院云数智技术实践研讨及成果展示交流会”
PingCAP
50次阅读
2025-03-25 18:14:39
缓存监控治理在游戏业务的实践和探索
vivo互联网技术
48次阅读
2025-03-20 09:51:10
国产数据库再获资本赋能，「九有数据库」完成A轮融资
严少安
45次阅读
2025-03-31 13:00:54
