Apache kafka Storm 1.0.1中的Storm Kafka客户端

基于Storm文档支持的

KafkaSpout

实现基于旧的消费者API。我注意到外部包有另一个名为

storm kafka client

的实现

---

尚不清楚

1.0.1

中的新客户端版本是否已准备就绪。有人有运行它的经验吗？

我在Storm mail列表中发布了相同的问题。 新的API已经可以生产了。我们应该使用1.x分支。 我计划和你一起测试

<!-- https://mvnrepository.com/artifact/org.apache.storm/storm-kafka-client -->

<dependency>
    <groupId>org.apache.storm</groupId>
    <artifactId>storm-kafka-client</artifactId>
    <version>1.0.1</version>
</dependency>

org.apache.storm
暴风卡夫卡客户端
1.0.1

---

将更新进度。

您可以在storm 1.1.0中使用以下maven依赖项

  <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-core</artifactId>
         <version>1.1.0</version>
        <scope>provided</scope>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>log4j-over-slf4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

 <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>0.10.0.0</version>
    </dependency>
    <dependency>
    <groupId>org.apache.storm</groupId>
    <artifactId>storm-kafka</artifactId>
    <version>1.0.0</version>
</dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.10</artifactId>
         <version>0.9.0.0</version>
        <exclusions>
            <exclusion>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
            </exclusion>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

org.apache.storm
风暴核心
1.1.0
假如
org.slf4j
log4j-over-slf4j
org.apache.kafka
卡夫卡客户
0.10.0.0
org.apache.storm
卡夫卡风暴
1.0.0
org.apache.kafka
卡夫卡2.10
0.9.0.0
org.apache.zookeeper
动物园管理员
log4j
log4j

您可能会面临更多的依赖性问题，您可以通过添加所需的JAR来解决这些问题

---

java代码中的依赖关系也将从org.backtype.storm.XXXXX更改为org.apache.storm.XXXXX

下面的代码对我来说很好

public TopologyBuilder myTopology() {

    TopologyBuilder builder = new TopologyBuilder();

    try {       
        KafkaSpoutConfig<String, String> kafkaSpoutConfig = getKafkaSpoutConfig("KAFKA_IP:9092", KAFKA_TOPIC);          
        KafkaSpout kafkaSpout = new KafkaSpout<>(kafkaSpoutConfig);
        builder.setSpout("kafkaSpout", kafkaSpout, 2 * 2);
        builder.setBolt("Bolt-1", new TestBolt(), parallelism).shuffleGrouping("kafkaSpout", KAFKA_TOPIC);
    } catch (Exception ex) {

    }

    return builder;
}

---

你好@YaRiK，我正在寻找一个关于如何使用storm kafka客户端模块的快速示例！如果你已经写了一个示例代码，你能分享一下吗？谢谢

protected KafkaSpoutConfig<String, String> getKafkaSpoutConfig(String bootstrapServers ,String topic) {
    ByTopicRecordTranslator<String, String> trans = new ByTopicRecordTranslator<>(
        (r) -> new Values(r.topic(), r.partition(), r.offset(), r.key(), r.value()),
        new Fields("topic", "partition", "offset", "key", "value"), topic);

    Builder<String, String> builder = KafkaSpoutConfig.builder(bootstrapServers, new String[]{topic});

    return builder.setProp(ConsumerConfig.GROUP_ID_CONFIG, topic)
        .setProcessingGuarantee(ProcessingGuarantee.AT_LEAST_ONCE)
        .setRetry(getRetryService())
        .setRecordTranslator(trans)
        .setOffsetCommitPeriodMs(10_000)
        .setFirstPollOffsetStrategy(UNCOMMITTED_EARLIEST)
        .setMaxUncommittedOffsets(1000)
        .build();
}

protected KafkaSpoutRetryService getRetryService() {
    return new KafkaSpoutRetryExponentialBackoff(TimeInterval.microSeconds(500),
        TimeInterval.milliSeconds(2), Integer.MAX_VALUE, TimeInterval.seconds(10));
 }