配置
集群配置
- 3个Node 3个节点；
- cpu: 40c
- memory: 256G
- 带宽: 4G
JVM配置
-Xmx30G -Xms30G -XX:MaxPermSize=48m -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35
关键因素测试
Producer测试
1 和线程数的关系(threads)
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 1
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 3 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 5 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 7 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 15 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 20 --hide-header
| Threads |
MB/S |
nMsg/S |
| 1 |
81.7960 |
20939.7772 |
| 3 |
172.1875 |
44080.0053 |
| 5 |
219.7115 |
56246.1331 |
| 7 |
252.8478 |
64729.0440 |
| 10 |
258.6749 |
66607801 |
| 15 |
283.6163 |
72605.0969 |
| 20 |
257.4644 |
65910.8885 |
2 和批大小的关系(batch-size)
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 1000 --threads 10
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 5000 --threads 10 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --hide-header
| Batch-size |
MB/S |
nMsg/S |
| 1000 |
294.6779 |
214776.6323 |
| 5000 |
282.9386 |
72432.2758 |
| 10000 |
251.3108 |
32167.7872 |
3 和日志大小的关系(message-size)
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 1024 --batch-size 10000 --threads 10
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 8192 --batch-size 10000 --threads 10 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 10240 --batch-size 10000 --threads 10 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 20480 --batch-size 10000 --threads 10 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 40960 --batch-size 10000 --threads 10 --hide-header
| Message-size |
MB/S |
nMsg/S |
| 1024 |
209.7428 |
214776.6323 |
| 4096 |
282.9386 |
72432.2758 |
| 8192 |
251.3108 |
32167.7872 |
| 10240 |
233.6218 |
23922.8727 |
| 20480 |
220.0903 |
11268.6214 |
4 和响应方式的关系(acks)
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --request-num-acks 0
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --request-num-acks 1 --hide-header
| ack |
MB/S |
nMsg/S |
| 不响应 |
285.8999 |
73190.3681 |
| leader响应 |
276.9408 |
70896.8451 |
5 和压缩方式的关系(compression-codec)
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --compression-codec 0
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --compression-codec 1 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --compression-codec 2 --hide-header
| compresson-codec |
MB/S |
nMsg/S |
| 非压缩 |
274.5853 |
70293.8282 |
| Gzip |
267.6981 |
68530.7018 |
| Snappy |
827.5953 |
211864.4068 |
6 Producer配置总结
在特定资源情况下， 在日志大小为4kb, 批大小为5000, 使用15个线程，使用Snapy压缩方式，用不确认的响应方式的情况下，性能最好;
Cosumer测试
1. 和线程数的关系(threads)
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 10000 --threads 1
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 10000 --threads 3 --hide-header
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 10000 --threads 6 --hide-header
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 10000 --threads 10 --hide-header
| Threads |
MB/S |
nMsg/S |
| 1 |
61.9631 |
64973.0362 |
| 3 |
70.4807 |
73904.3677 |
| 6 |
72.2006 |
75707.8686 |
| 10 |
73.1439 |
76696.9193 |
2. 和抓取数据量的关系(fetch-size)
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 1000 --threads 10
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 2000 --threads 6 --hide-header
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 5000 --threads 6 --hide-header
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 10000 --threads 6 --hide-header
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 15000 --threads 6 --hide-header
| Fetch-Size |
MB/S |
nMsg/S |
| 2000 |
13.2777 |
13922.7289 |
| 5000 |
46.4211 |
48676.0125 |
| 10000 |
75.8128 |
79495.4688 |
| 15000 |
92.6737 |
97175.4341 |
3. cosumer配置总结
使用6-8个线程, 抓取大小为10000左右，性能相对好;
