如何使用Spring integration kafka将kafka producer主题配置为多个分区

如何使用Spring integration kafka将kafka producer主题配置为多个分区,spring,spring-integration,apache-kafka,kafka-producer-api,Spring,Spring Integration,Apache Kafka,Kafka Producer Api,我读了很多文章,但没有找到如何使用SpringIntegrationKafka配置具有多分区(在运行时创建的主题)的Producer 我使用来理解和配置我的应用程序的卡夫卡 请提供同样的解决方案 还有一件事,kafkaHeader.messageKey的用途是什么 更新:下面是我开发的测试代码 `公共类边界测试{ @试验 public void mytest()引发异常{ KafkaProducerContext<String, SMSNotificationVO> ctx

我读了很多文章,但没有找到如何使用SpringIntegrationKafka配置具有多分区(在运行时创建的主题)的Producer

我使用来理解和配置我的应用程序的卡夫卡

请提供同样的解决方案

还有一件事,kafkaHeader.messageKey的用途是什么

更新:下面是我开发的测试代码
`公共类边界测试{ @试验 public void mytest()引发异常{

    KafkaProducerContext<String, SMSNotificationVO> ctx = new KafkaProducerContext<String, SMSNotificationVO>();
    ProducerMetadata<String, SMSNotificationVO> meta = new ProducerMetadata<String, SMSNotificationVO>("sms_topic");
    meta.setValueClassType(SMSNotificationVO.class);
    meta.setKeyClassType(String.class);
    meta.setValueEncoder(new SMSObjectSerializer());

    ProducerMetadata<String, SMSNotificationVO> meta1 = new ProducerMetadata<String, SMSNotificationVO>("sms_topic_10");
    meta1.setValueClassType(SMSNotificationVO.class);
    meta1.setKeyClassType(String.class);
    meta1.setValueEncoder(new SMSObjectSerializer());
    Properties producerProperties = new Properties();
    producerProperties.put(org.apache.kafka.clients.producer.ProducerConfig.RETRIES_CONFIG, "1");//"message.send.max.retries"
    ProducerFactoryBean<String, SMSNotificationVO> producer = new ProducerFactoryBean<String, SMSNotificationVO>(meta,"192.168.1.147:9092");
    ProducerFactoryBean<String, SMSNotificationVO> producer1 = new ProducerFactoryBean<String, SMSNotificationVO>(meta1,"192.168.1.147:9092", producerProperties);


    ProducerConfiguration<String, SMSNotificationVO> p = new ProducerConfiguration<String, SMSNotificationVO>(meta, producer.getObject());
    ProducerConfiguration<String, SMSNotificationVO> p1 = new ProducerConfiguration<String, SMSNotificationVO>(meta1, producer1.getObject());

    Map<String, ProducerConfiguration<String, SMSNotificationVO>> map = new HashMap<String, ProducerConfiguration<String, SMSNotificationVO>>();
    map.put("sms_topic", p);
    map.put("sms_topic_10", p1);

    ctx.setProducerConfigurations(map);

    // java code to send message
    Map<String, String> params = new HashMap<>();
    params.put("Topic", "TEST MULTIPLE TOPIC : this is sms_topic_1");
    List<String> smsRecipients = new ArrayList<>();
    smsRecipients.add("9953225211");


     String message = "This Test message from junit topic sms_topic_1";
     Integer messageType =1;

    SMSNotificationVO vo = new SMSNotificationVO();
    vo.setMessage(message);
    vo.setMessageType(messageType);
    vo.setParams(params);
    vo.setSmsRecipients(smsRecipients);


    try{
     KafkaProducerMessageHandler<String, SMSNotificationVO> handler = new KafkaProducerMessageHandler<String, SMSNotificationVO>(ctx);
     handler.setMessageKeyExpression(new LiteralExpression("sms_topic_10"));
     handler.handleMessage(MessageBuilder.withPayload(vo)
             //if i remove the below comments I will get null pointer exception
             //.setHeader(KafkaHeaders.MESSAGE_KEY, "some key")
             //.setHeader(KafkaHeaders.PARTITION_ID, "1")
                .setHeader(KafkaHeaders.TOPIC, "sms_topic_10")
                .build());

    }catch(Exception e){
        e.printStackTrace();
        Assert.fail("Kafka SMS Producer fail");
    }

}  
}`

谢谢

主题与
制作人无关
。在向其发送或接收消息之前,您应该预先配置好主题

您可以在运行时通过Spring集成高级API使用
AdminUtils.createTopic()
创建主题,这只是为了测试方便,在生产中应该避免

KafkaHeaders.messageKey
完全映射到标准Kafka
messageKey
,可以使用yes来确定
主题中的目标
分区


因此,您肯定应该阅读ApacheKafka官方文档,了解如何使用“多分区”创建主题关于
分区
messageKey
参数,制作人方面应该做些什么。

感谢您的回复,我的想法如下:1.messageKey用于确定主题中的分区(有些类似于哈希)然后如何选择该键,因为我想随机选择键,以便每次消息发布时它都进入其他分区以利用并行性。2.在链接中提到,
消息键表达式
分区id表达式
,这些值的基础是什么3。如果提到消息键表达式,则只有我可以n使用KafkaHeaders.messageKey。谢谢如果在发送操作时指定特定的
分区
,分区确定算法将绕过
messageKey
。这些表达式正好用于Kafka
Producer
操作。请参阅其API。实际上,
KafkaHeaders.messageKey
可以用于代替
消息键表达式
。但是如果您看到我的代码,
setHeader(KafkaHeaders.message\u key,“some key”)
给出了空指针异常m-m-m。看起来您使用的库版本已经足够旧了。是否升级到最新的
spring-integration-kafka-1.3.0
?使用spring-integration-kafka-1.1.1的项目。我将尝试使用1.3.0版本。
`org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.kafka.outbound.KafkaProducerMessageHandler@7b94089b]; nested exception is java.lang.NullPointerException
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:84)
at dd.kafka.producer.OutboundTests.mytest(OutboundTests.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: java.lang.NullPointerException at kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:130)
at kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:127)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at kafka.producer.async.DefaultEventHandler.serialize(DefaultEventHandler.scala:127)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:53)
at kafka.producer.Producer.send(Producer.scala:77)
at kafka.javaapi.producer.Producer.send(Producer.scala:33)
at org.springframework.integration.kafka.support.ProducerConfiguration.send(ProducerConfiguration.java:70)
at org.springframework.integration.kafka.support.KafkaProducerContext.send(KafkaProducerContext.java:197)
at org.springframework.integration.kafka.outbound.KafkaProducerMessageHandler.handleMessageInternal(KafkaProducerMessageHandler.java:81)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
... 24 more`