kafka常用命令与常见问题

常用命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
集群重启kafka：
kill -9 $(pgrep -f "kafka|zookeeper")
/usr/local/zookeeper/bin/zkServer.sh start
/usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties 

单机启动zookeeper：
nohup bin/zookeeper-server-start.sh config/zookeeper.properties > zookeeper-run.log 2>&1 &

单机启动kafka：
/usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties 

查看主题列表：
/usr/local/kafka/bin/kafka-topics.sh --zookeeper zookeeper_server:2181 --list

查看每个节点/分区的信息：
/usr/local/kafka/bin/kafka-topics.sh --describe --zookeeper zookeeper_server:2181 -topic zhouchangju4

创建主题：
/usr/local/kafka/bin/kafka-topics.sh --create --zookeeper zookeeper_server:2181 --replication-factor 1 --partitions 1 --topic zhouchangju

删除topic：
/usr/local/kafka/bin/kafka-topics --delete --zookeeper zookeeper_server:2181 --topic topic_name

创建一个发布者：
/usr/local/kafka/bin/kafka-console-producer.sh --broker-list kafka_server:9092 --topic zhouchangju

消费：
/usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka_server:9092 --topic zhouchangju   --from-beginning

常见问题

写在前面的话：排查问题，先仔细查看报错信息和日志，不要靠侥幸和瞎蒙去解决问题。

哪些情况连zookeeper，哪些情况连kafka？

除了生产和消费消息用kafka的9092之外，其他全部用zookeeper的2181端口。

可以正常生产，但是无法消费数据

这是我在测试环境搭建kafka的时候遇到的问题，卡了我好久。后来静下心来查看日志，发现失败的服务器上，日志中有报错：

1
2
3
4
5
6
7
8
9
[2018-08-28 10:08:07,020] WARN [Log partition=caole-0, dir=/usr/local/kafka/logs] Found a corrupted index file corresponding to log file /usr/local/kafka/logs/caole-0/00000000000000000000.log due to Corrupt time index found, time index file (/usr/local/kafka/logs/caole-0/00000000000000000000.timeindex) has non-zero size but the last timestamp is 0 which is less than the first timestamp 1535112867125}, recovering segment and rebuilding index files... (kafka.log.Log)

[2018-08-28 10:08:07,241] WARN [Log partition=zhouchangju-0, dir=/usr/local/kafka/logs] Found a corrupted index file corresponding to log file /usr/local/kafka/logs/zhouchangju-0/00000000000000000000.log due to Corrupt time index found, time index file (/usr/local/kafka/logs/zhouchangju-0/00000000000000000000.timeindex) has non-zero size but the last timestamp is 0 which is less than the first timestamp 1535421393086}, recovering segment and rebuilding index files... (kafka.log.Log)

[2018-08-28 10:08:08,587] ERROR Error while creating ephemeral at /brokers/ids/1003, node already exists and owner '73365454938963968' does not match current session '73365485955121152' (kafka.zk.KafkaZkClient$CheckedEphemeral)

[2018-08-28 10:08:08,846] ERROR [KafkaServer id=1003] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)

[2018-08-28 10:08:11,934] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)

发现里面caole这个topic的数据报错了；而实际上这个新搭建的kafka我根本就没写入过caole这个topic，而且报错里面还提到/brokers/ids/1003这个node的属主session有问题，因此怀疑是不是哪个配置文件有误，导致读取到了之前遗留的数据。

最终发现是因为之前装过kafka，然后/usr/local/kafka/config/zookeeper.properties文件里面的dataDir这个参数，配置的是之前的已废弃的kafka的目录，导致数据识别出问题了；修改掉这个配置的地址，就恢复正常了。

参考资料

单机安装kafka：https://www.cnblogs.com/qpf1/p/9161742.html

注意：单机模式，zookeeper的进程名称会包含kafka关键字，如果kill -9 $(pgrep -f “kafka”)，会把zookeeper也杀掉，导致kafka起不来，这个要注意。

彻底删除kafka的topic：https://blog.csdn.net/belalds/article/details/80575751

搭建kafka集群：https://www.cnblogs.com/luotianshuai/p/5206662.html