kafka 与 zookeeper 启动 cluster.id引起的报错

背景

在做Mysql+Canal+Kafka+Greenplum 架构调研的时候，因：mysql+canal+kafka 接收消息乱码问题，想着把 zookeeper和kafka 的数据及日志全部清理掉，重跑看下效果，又遇到了 zookeeper 与 kafka 启动冲突的问题。做个记录。

声名：此环境为调研测试环境，操作有些极端，仅供参考思路。

报错：

• 先启动：zookeeper

sh zookeeper-server-start.sh ../config/zookeeper.properties 
复制

• zookeeper输出日志：

[2022-11-02 20:51:26,872] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2022-11-02 20:51:26,882] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2022-11-02 20:51:26,899] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2022-11-02 20:51:26,900] INFO Reading snapshot /usr/local/kafka_2.13-2.6.0/zookeeper_data/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileSnap)
[2022-11-02 20:51:26,906] INFO Snapshotting: 0x0 to /usr/local/kafka_2.13-2.6.0/zookeeper_data/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2022-11-02 20:51:26,925] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager)
[2022-11-02 20:51:54,996] INFO Creating new log file: log.1 (org.apache.zookeeper.server.persistence.FileTxnLog)
复制

• 再启动：kafka

sh kafka-server-start.sh ../config/server.properties 
复制

• kafka 日志输出 ：ERROR Fatal error during KafkaServer startup. Prepare to shutdown

[2022-11-02 20:51:54,973] INFO Opening socket connection to server canalkafka156/192.168.5.156:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-11-02 20:51:54,979] INFO Socket connection established, initiating session, client: /192.168.5.156:52014, server: canalkafka156/192.168.5.156:2181 (org.apache.zookeeper.ClientCnxn)
[2022-11-02 20:51:55,014] INFO Session establishment complete on server canalkafka156/192.168.5.156:2181, sessionid = 0x10015543c0b0000, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
[2022-11-02 20:51:55,018] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2022-11-02 20:51:55,431] INFO Cluster ID = pPUDVLY3SOulZAIQc4ihbA (kafka.server.KafkaServer)
[2022-11-02 20:51:55,448] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentClusterIdException: The Cluster ID pPUDVLY3SOulZAIQc4ihbA doesn't match stored clusterId Some(R7OuNY1lQG22ooPpEz5ong) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
        at kafka.server.KafkaServer.startup(KafkaServer.scala:235)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
        at kafka.Kafka$.main(Kafka.scala:82)
        at kafka.Kafka.main(Kafka.scala)
[2022-11-02 20:51:55,452] INFO shutting down (kafka.server.KafkaServer)
[2022-11-02 20:51:55,457] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2022-11-02 20:51:55,564] INFO Session: 0x10015543c0b0000 closed (org.apache.zookeeper.ZooKeeper)
[2022-11-02 20:51:55,564] INFO EventThread shut down for session: 0x10015543c0b0000 (org.apache.zookeeper.ClientCnxn)
[2022-11-02 20:51:55,567] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2022-11-02 20:51:55,575] INFO shut down completed (kafka.server.KafkaServer)
[2022-11-02 20:51:55,576] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2022-11-02 20:51:55,579] INFO shutting down (kafka.server.KafkaServer)
复制

问题复现：

• 清理 zookeeper 数据及日志：

rm -rf zookeeper_data/*
rm -rf zookeeper_logs/*
复制

• 清理 kafka 日志

rm -rf /usr/local/kafka_2.13-2.6.0/logs/*
复制

• 再启动就出现上面的报错：
  

解决：

• 从上面的报错是，发现在了这个信息：

The Cluster ID pPUDVLY3SOulZAIQc4ihbA doesn't match stored clusterId Some(R7OuNY1lQG22ooPpEz5ong) in meta.properties.
复制

• 然后找到这个文件：meta.properties
• 查看内容： Some(R7OuNY1lQG22ooPpEz5ong) in meta.properties 与下面的cluster.id=R7OuNY1lQG22ooPpEz5ong 对上了。

cat meta.properties 
#
#Wed Nov 02 21:08:47 CST 2022
cluster.id=R7OuNY1lQG22ooPpEz5ong
version=0
broker.id=0
复制

• 然后将meta.properties 清理掉

rm -rf /usr/local/kafka_2.13-2.6.0/kafka-logs/meta.properties 
复制

• 再次启动正常

sh zookeeper-server-start.sh ../config/zookeeper.properties 
sh kafka-server-start.sh ../config/server.properties 
[2022-11-02 21:14:57,802] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2022-11-02 21:14:57,824] INFO [SocketServer brokerId=0] Starting socket server acceptors and processors (kafka.network.SocketServer)
[2022-11-02 21:14:57,829] INFO [SocketServer brokerId=0] Started data-plane acceptor and processor(s) for endpoint : ListenerName(PLAINTEXT) (kafka.network.SocketServer)
[2022-11-02 21:14:57,829] INFO [SocketServer brokerId=0] Started socket server acceptors and processors (kafka.network.SocketServer)
[2022-11-02 21:14:57,842] INFO Kafka version: 2.6.0 (org.apache.kafka.common.utils.AppInfoParser)
[2022-11-02 21:14:57,842] INFO Kafka commitId: 62abe01bee039651 (org.apache.kafka.common.utils.AppInfoParser)
[2022-11-02 21:14:57,842] INFO Kafka startTimeMs: 1667394897830 (org.apache.kafka.common.utils.AppInfoParser)
[2022-11-02 21:14:57,843] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
复制

• 查看meta.properties

 cat meta.properties 
#
#Wed Nov 02 21:08:47 CST 2022
cluster.id=pPUDVLY3SOulZAIQc4ihbA
version=0
broker.id=0

复制

cluster.id= pPUDVLY3SOulZAIQc4ihbA 与上面的报错：The Cluster ID pPUDVLY3SOulZAIQc4ihbA 对上了。

总结

问题出现原因分析：之前运行kafak在该目录下创建了一些主题信息（kafka内部会在用户指定目录下存储许多与保证服务正常工作的相关文件），后来清理了zookeeper的数据及日志后，但是没有对kafka-logs目录位置下的数据进行清理，导致新启的kafka服务The Cluster ID pPUDVLY3SOulZAIQc4ihbA 与 meta.properties的旧的cluster.id=R7OuNY1lQG22ooPpEz5ong 匹配不上，所以报错。删除meta.properties或清理 kafka-logs目录，再次重启，程序恢复正常。再次查看： meta.properties cluster.id=pPUDVLY3SOulZAIQc4ihbA与上次匹配。