Apache kafka 卡夫卡可验证生产者和消费者问题

我正在用卡夫卡做实验。我已经开始工作了

kafka-console-producer and 
kafka-console-consumer. 

我使用kafka producer发送消息，并在kafka控制台使用者处成功接收。 现在我想一次生成和消费大约5000条消息。我查阅了文档，了解到有两个命令

kafka-verifiable-producer.sh

kafka-verifiable-consumer.sh

我试着使用这些命令

 kafka-verifiable-producer.sh --broker-list localhost:9092 --max-messages 5000 --topic data-sending

kafka-verifiable-consumer.sh  --group-instance-id 1 --group-id data-world --topic data-sending --broker-list localhost:9092

结果如下

"timestamp":1581268289761,"name":"producer_send_success","key":null,"value":"4996","offset":44630,"topic":"try_1","partition":0}
{"timestamp":1581268289761,"name":"producer_send_success","key":null,"value":"4997","offset":44631,"topic":"try_1","partition":0}
{"timestamp":1581268289761,"name":"producer_send_success","key":null,"value":"4998","offset":44632,"topic":"try_1","partition":0}
{"timestamp":1581268289761,"name":"producer_send_success","key":null,"value":"4999","offset":44633,"topic":"try_1","partition":0}

{"timestamp":1581268289769,"name":"shutdown_complete"}
 {"timestamp":1581268289771,"name":"tool_data","sent":5000,"acked":5000,"target_throughput":-1,"avg_throughput":5285.412262156448}

在消费者控制台上，结果如下

{"timestamp":1581268089357,"name":"records_consumed","count":352,"partitions":[{"topic":"try_1","partition":0,"count":352,"minOffset":32777,"maxOffset":33128}]}
{"timestamp":1581268089359,"name":"offsets_committed","offsets":[{"topic":"try_1","partition":0,"offset":33129}],"success":true}
{"timestamp":1581268089384,"name":"records_consumed","count":500,"partitions":[{"topic":"try_1","partition":0,"count":500,"minOffset":33129,"maxOffset":33628}]}
 {"timestamp":1581268089391,"name":"offsets_committed","offsets":[{"topic":"try_1","partition":0,"offset":33629}],"success":true}
 {"timestamp":1581268089392,"name":"records_consumed","count":270,"partitions":[{"topic":"try_1","partition":0,"count":270,"minOffset":33629,"maxOffset":33898}]}
 {"timestamp":1581268089394,"name":"offsets_committed","offsets":[{"topic":"try_1","partition":0,"offset":33899}],"success":true}
 {"timestamp":1581268089415,"name":"records_consumed","count":500,"partitions":[{"topic":"try_1","partition":0,"count":500,"minOffset":33899,"maxOffset":34398}]}
 {"timestamp":1581268089416,"name":"offsets_committed","offsets":[{"topic":"try_1","partition":0,"offset":34399}],"success":true}
 {"timestamp":1581268089417,"name":"records_consumed","count":235,"partitions":[{"topic":"try_1","partition":0,"count":235,"minOffset":34399,"maxOffset":34633}]}
{"timestamp":1581268089419,"name":"offsets_committed","offsets":[{"topic":"try_1","partition":0,"offset":34634}],"success":true}

在上面的结果中，键为null。 如何使用此命令发送大量消息？ 我试图研究一个如何使用它们的例子，但没有找到。它生成类似整数的值，但我可以在何处插入消息？。 是否有任何方法可以使用此命令批量生成消息？也可以在windows中实现这样的命令，还是只针对linux？

---

如能链接到示例，将不胜感激

脚本

kafka-VerifiableProducer.sh

执行类

org.apache.kafka.tools.VerifiableProducer

。 （）

它的程序参数

--吞吐量

，

--重复键

和

--值前缀

可以满足您的需要

例如，下面生成的消息具有前缀值

111

，并且具有每5条消息旋转一次的增量键。您还可以使用

--throughput

选项配置消息的吞吐量。在本例中，它平均每秒生成5条消息

./kafka-verifiable-producer.sh --broker-list localhost:9092 --max-messages 10 --repeating-keys 5 --value-prefix 100 --throughput 5 --topic test
{"timestamp":1581271492652,"name":"startup_complete"}
{"timestamp":1581271492860,"name":"producer_send_success","key":"0","value":"100.0","offset":45,"topic":"test","partition":0}
{"timestamp":1581271492862,"name":"producer_send_success","key":"1","value":"100.1","offset":46,"topic":"test","partition":0}
{"timestamp":1581271493048,"name":"producer_send_success","key":"2","value":"100.2","offset":47,"topic":"test","partition":0}
{"timestamp":1581271493254,"name":"producer_send_success","key":"3","value":"100.3","offset":48,"topic":"test","partition":0}
{"timestamp":1581271493256,"name":"producer_send_success","key":"4","value":"100.4","offset":49,"topic":"test","partition":0}
{"timestamp":1581271493457,"name":"producer_send_success","key":"0","value":"100.5","offset":50,"topic":"test","partition":0}
{"timestamp":1581271493659,"name":"producer_send_success","key":"1","value":"100.6","offset":51,"topic":"test","partition":0}
{"timestamp":1581271493860,"name":"producer_send_success","key":"2","value":"100.7","offset":52,"topic":"test","partition":0}
{"timestamp":1581271494063,"name":"producer_send_success","key":"3","value":"100.8","offset":53,"topic":"test","partition":0}
{"timestamp":1581271494268,"name":"producer_send_success","key":"4","value":"100.9","offset":54,"topic":"test","partition":0}
{"timestamp":1581271494483,"name":"shutdown_complete"}
{"timestamp":1581271494484,"name":"tool_data","sent":10,"acked":10,"target_throughput":5,"avg_throughput":5.452562704471101}

---

如果您正在寻找更多自定义的消息键和值，最简单的方法是修改/扩展上述类。

为什么不用Java编写一个生产者，生成您想要的任意多的消息？@Giorgosmyriantous我可以，但我很想知道如何使用这些命令。卡夫卡可验证生产者和消费者。为什么可以使用这些命令，以及在什么情况下我或其他人可以使用它们。@Giorgosmyriantous我对卡夫卡不熟悉，所以我的问题可能是基本的。但如果有任何解决办法，我将不胜感激。谢谢你的回答。我会试试这个，然后回来找你。如果我没有弄错的话，这个功能只在linux中可用。正确的？在windows中使用kafka时，我不能使用相同的命令。@Rio，正如我看到的，kafka发行版有一堆BAT文件。（例如卡夫卡2.12-2.4.0/bin/windows）。上面的类可以用您自己的bat文件简单地执行。e、 g:``@echo off SetLocal set KAFKA_HEAP_OPTS=-Xmx512M“%~dp0kafka run class.bat”org.apache.KAFKA.tools.VerifiableProducer%*EndLocal``你能举个例子说明我如何在windows中将这两个类实现为我自己的bat文件吗？。我非常感谢你在这方面的帮助。Cox我看到这两个类在windows中不能作为bat文件使用。@Rio，我不能提供一个可用的bat文件，因为我不使用windows。您可以在kafka_2.XX-2.X.X/bin/windows中复制一个bat文件，并修改它执行的类名。e、 g复制kafka-console-producer.bat文件。并将

kafka.tools.ConsoleProducer

替换为

org.apache.kafka.tools.VerifiableProducer

@Rio，您在

kafka_2.XX-2.X/bin/windows

文件夹中没有

kafka run class.bat

？您甚至可以直接运行

kafka run class.bat

kafka-run-class.bat org.apache.kafka.tools.VerifiableProducer--代理列表localhost:9092--最多消息10--重复键5--值前缀100--吞吐量5--主题测试