第七篇重新编译 Promethues 修改时区

Linux点滴运维实践 2021-04-12

6182

写在之前

Prometheus 如果想报警的话，发出的邮件或者企业微信时，上面的时间是比中国东八区晚了8个小时，刚开始想通过把本地 usr/share/zoneinfo/Asia/Shanghai 通过volume的方式挂载到容器内的 /etc/localtime，尝试之后，发现不行。那么如何解决这个问题呢？其实有很多种解决方案，可以写一个插件，通过 webhook 的方式调用插件接口，把警报内容发给插件，由插件去把时间处理一下，还有一种方式，查看源码，为什么不能使用本地 etc/localtime时区，找到原因，修改源码，重新编译，这次总结一下接下从源码的角度来分析下这个问题。

原因确认并修改

源码下载链接：https://codeload.github.com/prometheus/prometheus/tar.gz/v2.17.1，下载源码，然后解压。

~/prometheus-2.17.1/cmd/prometheus/main.go 是Prometheus 入口启动文件，因为我们发送的报警都会记录在日志文件中，只需要查找到日志是如何记录的，就可以找到问题的根因，通过 Prometheus 入口文件，发现使用的第三方库promlog.New()，如下：

日志修改

通过查看github.com/prometheus/common/promlog 官方库，发现默认使用的是UTC时区，通过查看官网说使用UTC是为了防止夏令时误差，所以我们只需要重写下这个 logger 让其使用本地时区即可，这里需要注意一点，不修改的话，不会影响 Prometheus WebUI 和 Grafana 的效果，但是收邮件或者微信就有点别扭，总是需要加上8个小时，重写 logger 如下：

var logger log.Logger
if os.Getenv("LocalTZ") != "" {
    logger = func(config *promlog.Config) log.Logger {
        var (
            l log.Logger
            le level.Option
        )
        if config.Format.String() == "logfmt" {
            l = log.NewLogfmtLogger(log.NewSyncWriter(os.Stderr))
        } else {
            l = log.NewJSONLogger(log.NewSyncWriter(os.Stderr))
        }

        switch config.Level.String() {
        case "debug":
            le = level.AllowDebug()
        case "info":
            le = level.AllowInfo()
        case "warn":
            le = level.AllowWarn()
        case "error":
            le = level.AllowError()
        }
        l = level.NewFilter(l, le)
        l = log.With(l, "ts", log.TimestampFormat(
            func() time.Time { return time.Now().Local() },
            "2006-01-02T15:04:05.000Z07:00",
        ), "caller", log.DefaultCaller)
        return l
    }(&cfg.promlogConfig)
} else {
    logger = promlog.New(&cfg.promlogConfig)
}
复制

我们使用 time.Now().Local()，如果不设置环境变量 LocalTZ，还是使用默认的 UTC 时区即可。

Prometheus WebUI 修改

配置文件路径/prometheus-2.17.1/web/ui/static/js/graph/index.js，主要是修改这个index.js文件。

# 被修改行1
var date = '<span class="date">' + new Date(x * 1000).toUTCString() + '</span>';

# 修改为：
var date = '<span class="date">' + new Date(x * 1000).toString() + '</span>';

# 被修改行2
return self.endDate.data('DateTimePicker').date();

# 修改为：
return self.endDate.data('DateTimePicker').getLocalDate().getTime();
复制

修改后 diff 对比

重新编译

prometheus 需要编译静态文件，使用到了Node环境，需要安装，还需要go环境。

安装 go 环境

# 下载
wget https://studygolang.com/dl/golang/go1.14.1.linux-amd64.tar.gz

# 解压
tar -zxf go1.14.1.linux-amd64.tar.gz -C usr/loca/

# 添加环境变量并加载
echo 'PATH=$PATH:/usr/local/go/bin' >> /etc/profile
echo 'GO111MODULE=on' >> /etc/profile
echo 'GOPROXY=https://goproxy.cn,https://mirrors.aliyun.com/goproxy/,https://goproxy.io,https://athens.azurefd.net,direct' >> /etc/profile

# 测试
[root@iZm5egzzepj7apdj3vo9pjZ ~]# go version
go version go1.14.1 linux/amd64
[root@iZm5egzzepj7apdj3vo9pjZ ~]#
复制

安装 Node 环境

# 下载 Node
wget https://nodejs.org/dist/v13.12.0/node-v13.12.0-linux-x64.tar.xz

# 解压
tar -xvf node-v13.12.0-linux-x64.tar.xz -C usr/local/

# 环境变量
echo 'PATH=$PATH:/usr/loca/node-v13.12.0-linux-x64/bin'
source etc/profile

# 测试
[root@iZm5egzzepj7apdj3vo9pjZ ~]# node --version
v13.12.0
[root@iZm5egzzepj7apdj3vo9pjZ ~]#

# 安装 yarn
npm install -g yarn

# 测试
[root@iZm5egzzepj7apdj3vo9pjZ ~]# yarn --version
1.22.4
[root@iZm5egzzepj7apdj3vo9pjZ ~]#
复制

编译前端静态文件

[root@iZm5egzzepj7apdj3vo9pjZ prometheus-2.17.1]# make assets
>> writing assets
# Un-setting GOOS and GOARCH here because the generated Go code is always the same,
# but the cached object code is incompatible between architectures and OSes (which
# breaks cross-building for different combinations on CI in the same container).
cd web/ui && GO111MODULE=on GOOS= GOARCH= go generate -x -v -mod=vendor
doc.go
go run -mod=vendor assets_generate.go
writing assets_vfsdata.go
ui.go
[root@iZm5egzzepj7apdj3vo9pjZ prometheus-2.17.1]#
复制

编译 npm_licenses

[root@iZm5egzzepj7apdj3vo9pjZ prometheus-2.17.1]# make npm_licenses
>> bundling npm licenses
rm -f "npm_licenses.tar.bz2"
find web/ui/react-app/node_modules -iname "license*" | tar cfj "npm_licenses.tar.bz2" --transform 's/^/npm_licenses\//' --files-from=-
[root@iZm5egzzepj7apdj3vo9pjZ prometheus-2.17.1]#
复制

Prometheus 整体编译

[root@iZm5egzzepj7apdj3vo9pjZ prometheus-2.17.1]# make build
>> writing assets
# Un-setting GOOS and GOARCH here because the generated Go code is always the same,
# but the cached object code is incompatible between architectures and OSes (which
# breaks cross-building for different combinations on CI in the same container).
cd web/ui && GO111MODULE=on GOOS= GOARCH= go generate -x -v -mod=vendor
doc.go
go run -mod=vendor assets_generate.go
writing assets_vfsdata.go
ui.go
>> building binaries
GO111MODULE=on /root/go/bin/promu build --prefix data/prometheus-2.17.1
 > prometheus
 > promtool
 > tsdb
[root@iZm5egzzepj7apdj3vo9pjZ prometheus-2.17.1]#
复制

以上编译及安装过程，有可能会遇到各种问题，需要自行 google 解决，到此可以在目录中看到生成的二进制文件 prometheus、promtool，还有tsdb目录下面的tsdb，这些二进制在 Dockerfile 文件中会使用。

创建镜像

修改 Dockerfile 文件

[root@iZm5egzzepj7apdj3vo9pjZ prometheus-2.17.1]# cat Dockerfile
ARG ARCH="amd64"
ARG OS="linux"
FROM quay.io/prometheus/busybox-${OS}-${ARCH}:latest
LABEL maintainer="The Prometheus Authors <prometheus-developers@googlegroups.com>"

ARG ARCH="amd64"
ARG OS="linux"
COPY prometheus bin/prometheus
COPY promtool bin/promtool
COPY documentation/examples/prometheus.yml etc/prometheus/prometheus.yml
COPY console_libraries/ /usr/share/prometheus/console_libraries/
COPY consoles/ /usr/share/prometheus/consoles/
COPY LICENSE LICENSE
COPY NOTICE NOTICE
COPY npm_licenses.tar.bz2 npm_licenses.tar.bz2

RUN ln -s /usr/share/prometheus/console_libraries usr/share/prometheus/consoles/ /etc/prometheus/
RUN mkdir -p prometheus && \
    chown -R nobody:nogroup etc/prometheus prometheus

USER nobody
EXPOSE 9090
VOLUME [ "/prometheus" ]
WORKDIR prometheus
ENTRYPOINT [ "/bin/prometheus" ]
CMD [ "--config.file=/etc/prometheus/prometheus.yml", \
             "--storage.tsdb.path=/prometheus", \
             "--web.console.libraries=/usr/share/prometheus/console_libraries", \
             "--web.console.templates=/usr/share/prometheus/consoles" ]
[root@iZm5egzzepj7apdj3vo9pjZ prometheus-2.17.1]#
复制

Dockerfile 文件修改前后 diff

build镜像

[root@iZm5egzzepj7apdj3vo9pjZ prometheus-2.17.1]# docker build -t k8svip/prometheus-china-tz:v2.17.1 .
Sending build context to Docker daemon 506.1MB
Step 1/22 : ARG ARCH="amd64"
Step 2/22 : ARG OS="linux"
Step 3/22 : FROM quay.io/prometheus/busybox-${OS}-${ARCH}:latest
 ---> 0a74f8eccb8c
Step 4/22 : LABEL maintainer="The Prometheus Authors <prometheus-developers@googlegroups.com>"
 ---> Using cache
 ---> 0d056c3812f5
Step 5/22 : ARG ARCH="amd64"
 ---> Using cache
 ---> ad56539f7611
Step 6/22 : ARG OS="linux"
 ---> Using cache
 ---> bdc0dfd0de6f
Step 7/22 : COPY prometheus bin/prometheus
 ---> 815c1be1e94a
Step 8/22 : COPY promtool bin/promtool
 ---> 6a31b55ceb3f
Step 9/22 : COPY documentation/examples/prometheus.yml etc/prometheus/prometheus.yml
 ---> b73eb924da76
Step 10/22 : COPY console_libraries/ usr/share/prometheus/console_libraries/
 ---> debc7163afc7
Step 11/22 : COPY consoles/ usr/share/prometheus/consoles/
 ---> c1998bee6da8
Step 12/22 : COPY LICENSE LICENSE
 ---> 08f1057725ac
Step 13/22 : COPY NOTICE NOTICE
 ---> 45087884d5f1
Step 14/22 : COPY npm_licenses.tar.bz2 npm_licenses.tar.bz2
 ---> 1be9f58e37d5
Step 15/22 : RUN ln -s /usr/share/prometheus/console_libraries /usr/share/prometheus/consoles/ /etc/prometheus/
 ---> Running in 12cff3a6f41a
Removing intermediate container 12cff3a6f41a
 ---> 1933b14022e3
Step 16/22 : RUN mkdir -p /prometheus && chown -R nobody:nogroup etc/prometheus /prometheus
 ---> Running in e64d6bde9090
Removing intermediate container e64d6bde9090
 ---> 495c8b1b417c
Step 17/22 : USER nobody
 ---> Running in f5a22aa9df02
Removing intermediate container f5a22aa9df02
 ---> be85efcc9828
Step 18/22 : EXPOSE 9090
 ---> Running in 4df58474f9ef
Removing intermediate container 4df58474f9ef
 ---> 387ad4cffc27
Step 19/22 : VOLUME [ "/prometheus" ]
 ---> Running in a26913711311
Removing intermediate container a26913711311
 ---> 5371e95f6921
Step 20/22 : WORKDIR /prometheus
 ---> Running in d2d4e8764913
Removing intermediate container d2d4e8764913
 ---> 8a5c149f75b0
Step 21/22 : ENTRYPOINT [ "/bin/prometheus" ]
 ---> Running in 814369d4b209
Removing intermediate container 814369d4b209
 ---> d342a687b882
Step 22/22 : CMD [ "--config.file=/etc/prometheus/prometheus.yml", "--storage.tsdb.path=/prometheus", "--web.console.libraries=/usr/share/prometheus/console_libraries", "--web.console.templates=/usr/share/prometheus/consoles" ]
 ---> Running in 0cc5a7407014
Removing intermediate container 0cc5a7407014
 ---> 83fe699d5a95
Successfully built 83fe699d5a95
Successfully tagged k8svip/prometheus-china-tz:v2.17.1
[root@iZm5egzzepj7apdj3vo9pjZ prometheus-2.17.1]#
复制

镜像测试

[root@iZm5egzzepj7apdj3vo9pjZ prometheus-2.17.1]#
[root@iZm5egzzepj7apdj3vo9pjZ prometheus-2.17.1]# uptime
 22:41:12 up 1 day, 40 min, 1 user, load average: 0.11, 0.04, 0.05
[root@iZm5egzzepj7apdj3vo9pjZ prometheus-2.17.1]# uptime
 22:41:13 up 1 day, 40 min, 1 user, load average: 0.11, 0.04, 0.05
[root@iZm5egzzepj7apdj3vo9pjZ prometheus-2.17.1]# docker run --rm --net host -e LocalTZ=true -v /etc/localtime:/etc/localtime k8svip/prometheus-china-tz:v2.17.1
level=info ts=2020-04-09T22:41:15.881+08:00 caller=main.go:331 msg="no time or size retention was set so using the default time retention" duration=15d
level=info ts=2020-04-09T22:41:15.882+08:00 caller=main.go:366 msg="Starting Prometheus" version="(version=2.17.1, branch=non-git, revision=non-git)"
level=info ts=2020-04-09T22:41:15.882+08:00 caller=main.go:367 build_context="(go=go1.14.1, user=root@iZm5egzzepj7apdj3vo9pjZ, date=20200409-10:17:05)"
level=info ts=2020-04-09T22:41:15.882+08:00 caller=main.go:368 host_details="(Linux 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 iZm5egzzepj7apdj3vo9pjZ (none))"
level=info ts=2020-04-09T22:41:15.882+08:00 caller=main.go:369 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2020-04-09T22:41:15.882+08:00 caller=main.go:370 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2020-04-09T22:41:15.884+08:00 caller=main.go:700 msg="Starting TSDB ..."
level=info ts=2020-04-09T22:41:15.884+08:00 caller=web.go:514 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-04-09T22:41:15.886+08:00 caller=head.go:575 component=tsdb msg="replaying WAL, this may take awhile"
level=info ts=2020-04-09T22:41:15.887+08:00 caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2020-04-09T22:41:15.887+08:00 caller=head.go:627 component=tsdb msg="WAL replay completed" duration=346.549µs
level=info ts=2020-04-09T22:41:15.887+08:00 caller=main.go:716 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-04-09T22:41:15.887+08:00 caller=main.go:717 msg="TSDB started"
level=info ts=2020-04-09T22:41:15.888+08:00 caller=main.go:821 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2020-04-09T22:41:15.888+08:00 caller=main.go:849 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2020-04-09T22:41:15.888+08:00 caller=main.go:668 msg="Server is ready to receive web requests."
复制

日志时间已经与系统时间一致，此时如果有报警邮件的话，理论来说也应该是这个系统时间。镜像上传到了https://hub.docker.com，如果有需要的话，可以自行下载。

使用镜像

修改 yaml 文件

修改之前的prometheus-alertmanager-deployment.yaml，我这里没有从hub.docker.com下载，太慢了，传到了自己的私有镜像仓库。

重新创建下 Deployment 资源

[root@master01 monitor]# kubectl delete -f prometheus-alertmanager-deployment.yaml
deployment.apps "prometheus-server" deleted
[root@master01 monitor]# kubectl apply -f prometheus-alertmanager-deployment.yaml
deployment.apps/prometheus-server created
[root@master01 monitor]# kubectl get pods -n monitor
NAME READY STATUS RESTARTS AGE
node-exporter-ckh55 1/1     Running 1          6d9h
node-exporter-h68cp 1/1     Running 1          6d9h
node-exporter-qmfhf 1/1     Running 0          6d9h
node-exporter-sftxh 1/1     Running 0          6d9h
prometheus-server-66fb9964f5-djr6q 2/2     Running 0          11s
[root@master01 monitor]#
复制