使用openmessaging-benchmark测kop性能出现的问题

1、目的:使用kafka客户端测试kop性能

2、机器配置:pulsar2.9.3 + kop2.9.3.7
3个broker、3个bookie、3个zookeeper

3、(1)broker配置:acknowledgmentAtBatchIndexLevelEnabled=true,
managedLedgerDefaultEnsembleSize=3,managedLedgerDefaultWriteQuorum=3,
managedLedgerDefaultAckQuorum=2,其余参数默认值。
bookie配置:journalSyncData=true,journalWriteData=true,其余参数默认值。
(2)kafka客户端配置:replicationFactor=3,min.insync.replicas=2,acks=all,其余参数默认值。
4、测试场景:发送速率10000,1个topic、3个分区、每个主题4个生产者、每个订阅1个消费者、1个订阅。

5、报错信息:
生产端报错:[kafka-producer-network-thread | producer-2] INFO org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2] Node -1 disconnected.
[kafka-producer-network-thread | producer-2] INFO org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2] Cancelled in-flight METADATA request with correlation id 2 due to node -1 being disonnected (elapsed time since creation: 6ms, elapsed time since send: 6ms, request timeout: 1200000ms)
[kafka-producer-network-thread | producer-2] INFO org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2] Cancelled in-flight INIT_PRODUCER_ID request with correlation id 3 due to node -1 bing disconnected (elapsed time since creation: 5ms, elapsed time since send: 5ms, request timeout: 1200000ms)
[kafka-producer-network-thread | producer-2] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-2] Bootstrap broker 172.20.140.169:9092 (id: -1 rack: null) disconnected
[kafka-producer-network-thread | producer-3] INFO org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-3] Node -1 disconnected.

消费端报错:[pool-1-thread-1] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-sub-000-jnnjO5Q-1, groupId=sub-000-jnnjO5Q] (Re-)joining group
[pool-1-thread-1] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-sub-000-jnnjO5Q-1, groupId=sub-000-jnnjO5Q] JoinGroup failed: Coordinator 172.20.140.167:9092 (id: 548361440 rack: null) is loading the group.

6、说明:在kafka客户端将acks参数从all变成1就没有这个报错,为什么设置成all会出现?

这些配置 KoP 端是没有处理的。KoP 是直接将消息写 BK,副本数遵循的是 Broker 端配置的 quorum 参数,也就是你的 332:

从生产端日志看,有一台 broker 断连了。从消费端日志看,有一台 broker(那台 Coordinator Broker)是在重启中并且在加载 offsets。但这些都是正常现象,看不出什么

关键需要检查 Broker 的错误日志,如果能稳定复现,可以改成 debug 日志看看。