Spring integration 在web应用程序中集成Kafka生产者和消费者

在tomcat web应用程序中集成Kafka生产者和消费者的最佳策略是什么？”

我使用的是最新版本的spring integration kafka

谢谢

在SpringMVC中使用相同的代码时，我无法在中生成消息。但是，它在独立Java程序中使用时可以工作

2015-03-06 15:46:46 DEBUG QueueChannel:383 - postSend (sent=true) on channel 'inputToKafka', message: GenericMessage [payload=EventLogEvent{key=com.springapp.mvc.util.EventLogEventKey@cb5d2218, numberOfEvents=10, resetCounterInDays=5, template=6, proposition=5}, headers={timestamp=1425653206240, id=eccb00b0-6617-b5c4-6de0-45a08730041e, messageKey=1, kafka_topic=test-hs}]
2015-03-06 15:46:46 DEBUG DefaultListableBeanFactory:1610 - Invoking afterPropertiesSet() on bean with name 'hello'
2015-03-06 15:46:46 DEBUG TestDispatcherServlet:1225 - Rendering view [org.springframework.web.servlet.view.InternalResourceView: name 'hello'; URL [/WEB-INF/pages/hello.jsp]] in DispatcherServlet with name ''
2015-03-06 15:46:46 DEBUG InternalResourceView:432 - Added model object 'message' of type [java.lang.String] to request in view with name 'hello'
2015-03-06 15:46:46 DEBUG InternalResourceView:166 - Forwarding to resource [/WEB-INF/pages/hello.jsp] in InternalResourceView 'hello'
2015-03-06 15:46:46 DEBUG MockRequestDispatcher:67 - MockRequestDispatcher: forwarding to [/WEB-INF/pages/hello.jsp]
2015-03-06 15:46:46 DEBUG TestDispatcherServlet:996 - Successfully completed request
2015-03-06 15:46:46 DEBUG DirtiesContextTestExecutionListener:86 - After test method: context [DefaultTestContext@68af6fa0 testClass = AppTests, testInstance = com.springapp.mvc.AppTests@55c9a711, testMethod = simple@AppTests, testException = [null], mergedContextConfiguration = [WebMergedContextConfiguration@70e08379 testClass = AppTests, locations = '{file:src/main/webapp/WEB-INF/mvc-dispatcher-servlet.xml}', classes = '{}', contextInitializerClasses = '[]', activeProfiles = '{}', propertySourceLocations = '{}', propertySourceProperties = '{}', resourceBasePath = 'src/main/webapp', contextLoader = 'org.springframework.test.context.web.WebDelegatingSmartContextLoader', parent = [null]]], class dirties context [false], class mode [null], method dirties context [false].
2015-03-06 15:46:46 DEBUG ServletTestExecutionListener:147 - Resetting RequestContextHolder for test context [DefaultTestContext@68af6fa0 testClass = AppTests, testInstance = com.springapp.mvc.AppTests@55c9a711, testMethod = simple@AppTests, testException = [null], mergedContextConfiguration = [WebMergedContextConfiguration@70e08379 testClass = AppTests, locations = '{file:src/main/webapp/WEB-INF/mvc-dispatcher-servlet.xml}', classes = '{}', contextInitializerClasses = '[]', activeProfiles = '{}', propertySourceLocations = '{}', propertySourceProperties = '{}', resourceBasePath = 'src/main/webapp', contextLoader = 'org.springframework.test.context.web.WebDelegatingSmartContextLoader', parent = [null]]].
2015-03-06 15:46:46 DEBUG DirtiesContextTestExecutionListener:118 - After test class: context [DefaultTestContext@68af6fa0 testClass = AppTests, testInstance = [null], testMethod = [null], testException = [null], mergedContextConfiguration = [WebMergedContextConfiguration@70e08379 testClass = AppTests, locations = '{file:src/main/webapp/WEB-INF/mvc-dispatcher-servlet.xml}', classes = '{}', contextInitializerClasses = '[]', activeProfiles = '{}', propertySourceLocations = '{}', propertySourceProperties = '{}', resourceBasePath = 'src/main/webapp', contextLoader = 'org.springframework.test.context.web.WebDelegatingSmartContextLoader', parent = [null]]], dirtiesContext [false].
2015-03-06 15:46:46 INFO  GenericWebApplicationContext:862 - Closing org.springframework.web.context.support.GenericWebApplicationContext@5f0dd6db: startup date [Fri Mar 06 15:46:45 CET 2015]; root of context hierarchy 
2015-03-06 15:46:46 DEBUG DefaultListableBeanFactory:247 - Returning cached instance of singleton bean 'org.springframework.integration.config.IdGeneratorConfigurer#0'
2015-03-06 15:46:46 DEBUG DefaultListableBeanFactory:247 - Returning cached instance of singleton bean 'integrationHeaderChannelRegistry'
2015-03-06 15:46:46 DEBUG DefaultListableBeanFactory:247 - Returning cached instance of singleton bean 'globalChannelInterceptorProcessor'
2015-03-06 15:46:46 DEBUG DefaultListableBeanFactory:247 - Returning cached instance of singleton bean 'kafkaOutboundChannelAdapter'
2015-03-06 15:46:46 DEBUG DefaultListableBeanFactory:247 - Returning cached instance of singleton bean 'kafkaProducerContext'
2015-03-06 15:46:46 DEBUG DefaultListableBeanFactory:247 - Returning cached instance of singleton bean '_org.springframework.integration.errorLogger'
2015-03-06 15:46:46 DEBUG DefaultListableBeanFactory:247 - Returning cached instance of singleton bean 'lifecycleProcessor'
2015-03-06 15:46:46 INFO  DefaultLifecycleProcessor:356 - Stopping beans in phase 1073741823
2015-03-06 15:46:46 INFO  DefaultLifecycleProcessor:356 - Stopping beans in phase 0
2015-03-06 15:46:46 DEBUG DefaultLifecycleProcessor:226 - Asking bean 'kafkaProducerContext' of type [class org.springframework.integration.kafka.support.KafkaProducerContext] to stop
2015-03-06 15:46:46 INFO  Producer:68 - Shutting down producer
2015-03-06 15:46:46 INFO  ProducerPool:68 - Closing all sync producers
2015-03-06 15:46:46 DEBUG DefaultLifecycleProcessor:235 - Bean 'kafkaProducerContext' completed its stop procedure
2015-03-06 15:46:46 DEBUG DefaultLifecycleProcessor:226 - Asking bean '_org.springframework.integration.errorLogger' of type [class org.springframework.integration.config.ConsumerEndpointFactoryBean] to stop
2015-03-06 15:46:46 INFO  EventDrivenConsumer:83 - Removing {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2015-03-06 15:46:46 INFO  PublishSubscribeChannel:69 - Channel 'org.springframework.web.context.support.GenericWebApplicationContext@5f0dd6db.errorChannel' has 0 subscriber(s).
2015-03-06 15:46:46 DEBUG DefaultLifecycleProcessor:235 - Bean '_org.springframework.integration.errorLogger' completed its stop procedure
2015-03-06 15:46:46 INFO  EventDrivenConsumer:131 - stopped _org.springframework.integration.errorLogger
2015-03-06 15:46:46 INFO  DefaultLifecycleProcessor:356 - Stopping beans in phase -2147483648
2015-03-06 15:46:46 DEBUG DefaultListableBeanFactory:499 - Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@53164a3b: defining beans [channelInitializer,$autoCreateChannelCandidates,IntegrationConfigurationBeanFactoryPostProcessor,integrationEvaluationContext,org.springframework.integration.expression.IntegrationEvaluationContextAwareBeanPostProcessor#0,integrationGlobalProperties,integrationHeaderChannelRegistry,globalChannelInterceptorProcessor,jsonPath,xpath,toStringFriendlyJsonNodeToStringConverter,converterRegistrar,integrationConversionService,DefaultConfiguringBeanFactoryPostProcessor,datatypeChannelMessageConverter,messageBuilderFactory,inputToKafka,org.springframework.integration.kafka.outbound.KafkaProducerMessageHandler#0,org.springframework.scheduling.support.PeriodicTrigger#0,kafkaOutboundChannelAdapter,taskExecutor,kafkaSpecificEncoder,producerProperties,kafkaProducerContext,helloController,kafkaProducer,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,org.springframework.web.servlet.view.InternalResourceViewResolver#0,org.springframework.context.annotation.ConfigurationClassPostProcessor.importAwareProcessor,org.springframework.context.annotation.ConfigurationClassPostProcessor.enhancedConfigurationProcessor,nullChannel,errorChannel,_org.springframework.integration.errorLogger,taskScheduler,org.springframework.integration.config.IdGeneratorConfigurer#0]; root of factory hierarchy
2015-03-06 15:46:46 DEBUG DefaultListableBeanFactory:555 - Retrieved dependent beans for bean '(inner bean)#220d52b8': [_org.springframework.integration.errorLogger]
2015-03-06 15:46:46 DEBUG DefaultListableBeanFactory:555 - Retrieved dependent beans for bean '(inner bean)#40452b0f': [kafkaOutboundChannelAdapter]
2015-03-06 15:46:46 DEBUG DisposableBeanAdapter:245 - Invoking destroy() on bean with name 'taskExecutor'
2015-03-06 15:46:46 INFO  ThreadPoolTaskExecutor:203 - Shutting down ExecutorService
2015-03-06 15:46:46 DEBUG DefaultListableBeanFactory:555 - Retrieved dependent beans for bean '(inner bean)#2051acec': [kafkaProducerContext]
2015-03-06 15:46:46 DEBUG DefaultListableBeanFactory:555 - Retrieved dependent beans for bean '(inner bean)#1d79997b': [(inner bean)#2051acec]
2015-03-06 15:46:46 DEBUG DefaultListableBeanFactory:555 - Retrieved dependent beans for bean '(inner bean)#1d79997b#1': [(inner bean)#7779ad44]
2015-03-06 15:46:46 DEBUG DisposableBeanAdapter:245 - Invoking destroy() on bean with name 'taskScheduler'
2015-03-06 15:46:46 INFO  ThreadPoolTaskScheduler:203 - Shutting down ExecutorService 'taskScheduler'
2015-03-06 15:46:46 DEBUG DefaultListableBeanFactory:555 - Retrieved dependent beans for bean '(inner bean)#77ce5421': [taskScheduler]

下面粘贴了相同的代码。定义了两种资源：一种是使用Kafka原生Java API，它正在工作。其他资源正在使用spring集成，但它不起作用

    <beans xmlns="http://www.springframework.org/schema/beans"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns:context="http://www.springframework.org/schema/context"
   xmlns:int="http://www.springframework.org/schema/integration"
   xmlns:int-kafka="http://www.springframework.org/schema/integration/kafka"
   xmlns:task="http://www.springframework.org/schema/task"
   xmlns:mvc="http://www.springframework.org/schema/mvc"
   xsi:schemaLocation="http://www.springframework.org/schema/beans
    http://www.springframework.org/schema/beans/spring-beans.xsd
    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
    http://www.springframework.org/schema/integration/kafka http://www.springframework.org/schema/integration/kafka/spring-integration-kafka.xsd
    http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
    http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task.xsd
     http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc.xsd
    ">


<context:component-scan base-package="com.springapp.mvc"/>
 <bean class="org.springframework.web.servlet.view.InternalResourceViewResolver">
    <property name="prefix" value="/WEB-INF/pages/"/>
    <property name="suffix" value=".jsp"/>
</bean>
<int:channel id="inputToKafka">
    <int:queue/>
</int:channel>
<int-kafka:outbound-channel-adapter id="kafkaOutboundChannelAdapter"
                                    kafka-producer-context-ref="kafkaProducerContext"
                                    auto-startup="false"
                                    channel="inputToKafka"
                                    order="1"


        >
    <int:poller fixed-delay="10" time-unit="MILLISECONDS" receive-timeout="0" task-executor="taskExecutor"/>
</int-kafka:outbound-channel-adapter>
<task:executor id="taskExecutor" pool-size="10" keep-alive="120" queue-capacity="500"/>

<bean id="producerProperties"
      class="org.springframework.beans.factory.config.PropertiesFactoryBean">
    <property name="properties">
        <props>
            <prop key="topic.metadata.refresh.interval.ms">3600000</prop>
            <prop key="message.send.max.retries">5</prop>
            <prop key="serializer.class">kafka.serializer.StringEncoder</prop>
            <prop key="request.required.acks">1</prop>
        </props>
    </property>
</bean>
<int-kafka:producer-context id="kafkaProducerContext"
                            producer-properties="producerProperties">
    <int-kafka:producer-configurations>
        <int-kafka:producer-configuration broker-list="172.16.1.13:9092"
                                          topic="test-hs"
                                         value-class-type="java.lang.Object"
                                          compression-codec="default"/>
    </int-kafka:producer-configurations>
</int-kafka:producer-context>

3600000
5.
kafka.serializer.StringEncoder
1.

控制器类

package com.springapp.mvc;
import com.springapp.mvc.util.EventLogEvent;
import com.springapp.mvc.util.EventLogEventKey;
import com.springapp.mvc.util.KafkaEventLogWriterChannel;
import com.springapp.mvc.util.KafkaProducer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.ApplicationContext;
import org.springframework.http.HttpStatus;
import org.springframework.integration.channel.QueueChannel;
import org.springframework.integration.kafka.support.KafkaHeaders;
import org.springframework.integration.kafka.support.KafkaProducerContext;
import org.springframework.integration.support.MessageBuilder;
import org.springframework.messaging.Message;
import org.springframework.messaging.MessageChannel;
import org.springframework.stereotype.Controller;
import org.springframework.ui.ModelMap;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.ResponseStatus;

import java.util.*;
import java.util.concurrent.ConcurrentHashMap;

@Controller
@RequestMapping("/")
public class KafkaController {

@Autowired
private KafkaProducer kafkaProducer;



@RequestMapping(method = RequestMethod.GET)
public String printWelcome(ModelMap model) {
    model.addAttribute("message", "Hello world!");

    kafkaProducer.sendMessageToKafka();
    return "hello";
}

@RequestMapping(value = "/native", method = RequestMethod.PUT)
@ResponseStatus(HttpStatus.NO_CONTENT)
public void nativeAPI(@RequestBody String bookCase) throws Exception {


    KafkaEventLogWriterChannel ch = new KafkaEventLogWriterChannel("test-hs");
    Map<String, String> data = new HashMap<String, String>();
    data.put("kafka.api", "native");
    ch.writeData(data);
}

@RequestMapping(value = "/spring", method = RequestMethod.PUT)
@ResponseStatus(HttpStatus.NO_CONTENT)
public void springIntegration(@RequestBody String bookCase) throws Exception     {

    kafkaProducer.sendMessageToKafka();

}
}

package com.springapp.mvc；
导入com.springapp.mvc.util.EventLogEvent；
导入com.springapp.mvc.util.EventLogEventKey；
导入com.springapp.mvc.util.KafkaEventLogWriterChannel；
导入com.springapp.mvc.util.KafkaProducer；
导入org.springframework.beans.factory.annotation.Autowired；
导入org.springframework.context.ApplicationContext；
导入org.springframework.http.HttpStatus；
导入org.springframework.integration.channel.QueueChannel；
导入org.springframework.integration.kafka.support.KafkaHeaders；
导入org.springframework.integration.kafka.support.KafkaProducerContext；
导入org.springframework.integration.support.MessageBuilder；
导入org.springframework.messaging.Message；
导入org.springframework.messaging.MessageChannel；
导入org.springframework.stereotype.Controller；
导入org.springframework.ui.ModelMap；
导入org.springframework.web.bind.annotation.RequestBody；
导入org.springframework.web.bind.annotation.RequestMapping；
导入org.springframework.web.bind.annotation.RequestMethod；
导入org.springframework.web.bind.annotation.ResponseStatus；
导入java.util.*；
导入java.util.concurrent.ConcurrentHashMap；
@控制器
@请求映射（“/”）
公共类卡夫卡控制器{
@自动连线
私人卡夫卡制作人卡夫卡制作人；
@RequestMapping（method=RequestMethod.GET）
公共字符串printWelcome（ModelMap模型）{
addAttribute（“message”，“helloworld！”）；
kafkaProducer.sendMessageToKafka（）；
回复“你好”；
}
@RequestMapping（value=“/native”，method=RequestMethod.PUT）
@ResponseStatus（HttpStatus.无内容）
public void nativeAPI（@RequestBody String bookCase）引发异常{
KafkaEventLogWriterChannel ch=新的KafkaEventLogWriterChannel（“测试hs”）；
映射数据=新的HashMap（）；
data.put（“kafka.api”，“native”）；
ch.writeData（数据）；
}
@RequestMapping（value=“/spring”，method=RequestMethod.PUT）
@ResponseStatus（HttpStatus.无内容）
public void springIntegration（@RequestBody String bookCase）引发异常{
kafkaProducer.sendMessageToKafka（）；
}
}

使用spring integration 1.0 API发送消息的类

package com.springapp.mvc.util;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.integration.kafka.support.KafkaHeaders;
import org.springframework.messaging.MessageChannel;
import org.springframework.stereotype.Component;
import java.util.HashMap;
import java.util.Map;
/**
 * Created by hs on 06-03-2015.
*/
@Component
public class KafkaProducer {

@Autowired
@Qualifier("inputToKafka")
MessageChannel inputToKafka;


public void sendMessageToKafka()
{

    Map<String, String> data = new HashMap<String, String>();
    data.put("kafka.api", "spring-integration");

   boolean status =  inputToKafka.send(org.springframework.integration.support.MessageBuilder.withPayload(data).setHeader("messageKey", String.valueOf(1)).setHeader(KafkaHeaders.TOPIC, "test-hs").build());
    try {
        Thread.sleep(10000);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }


}

}

package com.springapp.mvc.util；
导入org.springframework.beans.factory.annotation.Autowired；
导入org.springframework.beans.factory.annotation.Qualifier；
导入org.springframework.integration.kafka.support.KafkaHeaders；
导入org.springframework.messaging.MessageChannel；
导入org.springframework.stereotype.Component；
导入java.util.HashMap；
导入java.util.Map；
/**
*由hs于2015年3月6日创建。
*/
@组成部分
公共级卡夫卡制作人{
@自动连线
@限定词（“输入okafka”）
信息通道输入卡；
公共无效sendMessageToKafka（）
{
映射数据=新的HashMap（）；
data.put（“kafka.api”，“spring集成”）；
布尔状态=inputOkafka.send（org.springframework.integration.support.MessageBuilder.withPayload（data）.setHeader（“messageKey”，String.valueOf（1））.setHeader（KafkaHeaders.TOPIC，“test hs”）.build（）；
试一试{
睡眠（10000）；
}捕捉（中断异常e）{
e、 printStackTrace（）；
}
}
}

请说得更具体些

Spring集成Kafka支持只是Spring集成的扩展，而Spring集成又是Spring框架的扩展

因为您可以简单地实现SpringMVCWeb应用程序，所以不需要为它提供任何其他集成功能，比如Kafka

由于Spring Integration实现了EIP，并为众所周知的协议提供了足够的适配器，并通过

消息通道

和

消息

抽象连接所有内容，因此您只需使用

接受HTTP请求并将其发送到

从Kafka到HTTP客户机可能不那么容易，反之亦然，但您可以通过任何其他适配器和协议，如JDBC（存储Kafka消息）或WebSocket，将Kafka消息推送到连接的客户机

---

目前，我看不到在基于Spring产品组合的Web应用程序中提供任何所需集成解决方案的任何问题。

我有相同的问题，通过将配置自动启动从假更改为真

auto-startup="true"

在哪

<int-kafka:outbound-channel-adapter id="kafkaOutboundChannelAdapter"
                                kafka-producer-context-ref="kafkaProducerContext"
                                auto-startup="false"
                                channel="inputToKafka"
                                order="1"


    >

---

它起作用了

感谢阿泰姆的快速回复。我真的需要在发送消息之前调用context.start（），或者在消费之前调用context.start（）？我需要将数据直接存储到master中，然后将相同的数据发布到kafka上，以便spark和其他消费者可以使用。不确定在您的情况下什么是

context.start（）

，但如果您的Web应用程序基于Spring，则应用程序上下文将与服务器上的Web应用程序部署一起启动。因此，任何通道适配器都可以从其通道接收消息并将其发送到目标系统，如Kafka。感谢Artem回答此问题。当我在独立java应用程序中使用spring Kafka时，我能够发送消息。然而，在web应用程序中，消息永远不会传递到kafka服务器。我比较了我的独立应用程序和spring mvc日志，发现任务执行者不是starti