Java 卡夫卡普斯工作示例

我最近熟悉了ApacheKafka，并有一个生产者-消费者的工作示例

我的下一步是将卡夫卡与喷口和螺栓结合起来，我很难在本地获得可用的示例（它们大多是旧的）

我得到了下面的示例working storm book/examples-ch02-getting_started，它从本地文本文件读取数据

同一份回购协议有一个storm book/examples-ch04-spouts的例子，但我无法让它发挥作用

我也尝试了以下示例，但出现了以下错误-

5034 [Thread-11-words] INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
5047 [Thread-11-words] ERROR backtype.storm.util - Async loop died!
java.lang.NoSuchMethodError: org.apache.zookeeper.ZooKeeper.<init>(Ljava/lang/String;ILorg/apache/zookeeper/Watcher;Z)V
        at org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29) ~[curator-client-2.4.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:169) ~[curator-framework-2.4.0.jar:na]
        at org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94) ~[curator-client-2.4.0.jar:na]
        at org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55) ~[curator-client-2.4.0.jar:na]
        at org.apache.curator.ConnectionState.reset(ConnectionState.java:219) ~[curator-client-2.4.0.jar:na]
        at org.apache.curator.ConnectionState.start(ConnectionState.java:103) ~[curator-client-2.4.0.jar:na]
        at org.apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:188) ~[curator-client-2.4.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:234) ~[curator-framework-2.4.0.jar:na]
        at storm.kafka.ZkState.<init>(ZkState.java:62) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
        at storm.kafka.KafkaSpout.open(KafkaSpout.java:85) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
        at backtype.storm.daemon.executor$fn__3371$fn__3386.invoke(executor.clj:522) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.util$async_loop$fn__460.invoke(util.clj:461) ~[storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
5049 [Thread-11-words] ERROR backtype.storm.daemon.executor -
java.lang.NoSuchMethodError: org.apache.zookeeper.ZooKeeper.<init>(Ljava/lang/String;ILorg/apache/zookeeper/Watcher;Z)V
        at org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29) ~[curator-client-2.4.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:169) ~[curator-framework-2.4.0.jar:na]
        at org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94) ~[curator-client-2.4.0.jar:na]
        at org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55) ~[curator-client-2.4.0.jar:na]
        at org.apache.curator.ConnectionState.reset(ConnectionState.java:219) ~[curator-client-2.4.0.jar:na]
        at org.apache.curator.ConnectionState.start(ConnectionState.java:103) ~[curator-client-2.4.0.jar:na]
        at org.apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:188) ~[curator-client-2.4.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:234) ~[curator-framework-2.4.0.jar:na]
        at storm.kafka.ZkState.<init>(ZkState.java:62) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
        at storm.kafka.KafkaSpout.open(KafkaSpout.java:85) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
        at backtype.storm.daemon.executor$fn__3371$fn__3386.invoke(executor.clj:522) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.util$async_loop$fn__460.invoke(util.clj:461) ~[storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
5088 [Thread-11-words] ERROR backtype.storm.util - Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
        at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
        at backtype.storm.daemon.worker$fn__4693$fn__4694.invoke(worker.clj:491) [storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.daemon.executor$mk_executor_data$fn__3272$fn__3273.invoke(executor.clj:240) [storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.util$async_loop$fn__460.invoke(util.clj:473) [storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]

5034[Thread-11-words]INFO org.apache.curator.framework.imps.CuratorFrameworkImpl-正在启动
5047[Thread-11-words]错误backtype.storm.util-异步循环终止！
java.lang.NoSuchMethodError:org.apache.zookeeper.zookeeper。（Ljava/lang/String；ILorg/apache/zookeeper/Watcher；Z）V
在org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper（DefaultZookeeperFactory.java:29）~[curator-client-2.4.0.jar:na]
在org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper（CuratorFrameworkImpl.java:169）~[curator-framework-2.4.0.jar:na]
在org.apache.curator.HandleHolder$1.getZooKeeper（HandleHolder.java:94）~[curator-client-2.4.0.jar:na]
在org.apache.curator.HandleHolder.getZooKeeper（HandleHolder.java:55）~[curator-client-2.4.0.jar:na]
在org.apache.curator.ConnectionState.reset（ConnectionState.java:219）~[curator-client-2.4.0.jar:na]
在org.apache.curator.ConnectionState.start（ConnectionState.java:103）~[curator-client-2.4.0.jar:na]
在org.apache.curator.CuratorZookeeperClient.start（CuratorZookeeperClient.java:188）~[curator-client-2.4.0.jar:na]
在org.apache.curator.framework.imps.CuratorFrameworkImpl.start（CuratorFrameworkImpl.java:234）~[curator-framework-2.4.0.jar:na]
在storm.kafka.ZkState.（ZkState.java:62）~[storm-kafka-0.9.2-cubating.jar:0.9.2-cubating]
在storm.kafka.KafkaSpout.open（KafkaSpout.java:85）~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
在backtype.storm.daemon.executor$fn__3371$fn__3386.invoke（executor.clj:522）~[storm-core-0.9.4.jar:0.9.4]
在backtype.storm.util$async\u循环$fn\uu 460.invoke（util.clj:461）~[storm-core-0.9.4.jar:0.9.4]
在clojure.lang.AFn.run（AFn.java:24）[clojure-1.5.1.jar:na]
在java.lang.Thread.run（Thread.java:745）[na:1.8.0_05]
5049[线程11字]错误backtype.storm.daemon.executor-
java.lang.NoSuchMethodError:org.apache.zookeeper.zookeeper。（Ljava/lang/String；ILorg/apache/zookeeper/Watcher；Z）V
在org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper（DefaultZookeeperFactory.java:29）~[curator-client-2.4.0.jar:na]
在org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper（CuratorFrameworkImpl.java:169）~[curator-framework-2.4.0.jar:na]
在org.apache.curator.HandleHolder$1.getZooKeeper（HandleHolder.java:94）~[curator-client-2.4.0.jar:na]
在org.apache.curator.HandleHolder.getZooKeeper（HandleHolder.java:55）~[curator-client-2.4.0.jar:na]
在org.apache.curator.ConnectionState.reset（ConnectionState.java:219）~[curator-client-2.4.0.jar:na]
在org.apache.curator.ConnectionState.start（ConnectionState.java:103）~[curator-client-2.4.0.jar:na]
在org.apache.curator.CuratorZookeeperClient.start（CuratorZookeeperClient.java:188）~[curator-client-2.4.0.jar:na]
在org.apache.curator.framework.imps.CuratorFrameworkImpl.start（CuratorFrameworkImpl.java:234）~[curator-framework-2.4.0.jar:na]
在storm.kafka.ZkState.（ZkState.java:62）~[storm-kafka-0.9.2-cubating.jar:0.9.2-cubating]
在storm.kafka.KafkaSpout.open（KafkaSpout.java:85）~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
在backtype.storm.daemon.executor$fn__3371$fn__3386.invoke（executor.clj:522）~[storm-core-0.9.4.jar:0.9.4]
在backtype.storm.util$async\u循环$fn\uu 460.invoke（util.clj:461）~[storm-core-0.9.4.jar:0.9.4]
在clojure.lang.AFn.run（AFn.java:24）[clojure-1.5.1.jar:na]
在java.lang.Thread.run（Thread.java:745）[na:1.8.0_05]
5088[Thread-11-words]ERROR backtype.storm.util-停止进程：（“工人死亡”）
java.lang.RuntimeException:（“工人死亡”）
在backtype.storm.util$exit\u process\u BANG\u doInvoke（util.clj:325）[storm-core-0.9.4.jar:0.9.4]
在clojure.lang.RestFn.invoke（RestFn.java:423）[clojure-1.5.1.jar:na]
在backtype.storm.daemon.worker$fn__4693$fn__4694.invoke（worker.clj:491）[storm-core-0.9.4.jar:0.9.4]
在backtype.storm.daemon.executor$mk_executor_data$fn__3272$fn_3273.invoke（executor.clj:240）[storm-core-0.9.4.jar:0.9.4]
在backtype.storm.util$async_循环$fn__460.invoke（util.clj:473）[storm-core-0.9.4.jar:0.9.4]
在clojure.lang.AFn.run（AFn.java:24）[clojure-1.5.1.jar:na]
在java.lang.Thread.run（Thread.java:745）[na:1.8.0_05]

当我在学习如何运行创建和运行卡夫卡喷口时遇到同样的问题，我发现这非常有用，我能够让我的卡夫卡喷口向其余的螺栓发出元组

这是一个关于如何为此创建拓扑的高级示例

public class TestTopology {

    public static void main(String[] args) {

        String zkIp = "192.168.59.103";

        String nimbusHost = "192.168.59.103";

        String zookeeperHost = zkIp +":2181";

        ZkHosts zkHosts = new ZkHosts(zookeeperHost);

        SpoutConfig kafkaConfig = new SpoutConfig(zkHosts, "myKafkaTopic", "", "storm");

        kafkaConfig.scheme = new SchemeAsMultiScheme(new JsonScheme() {
            @Override
            public Fields getOutputFields() {
                return new Fields("events");
            }
        });

        KafkaSpout kafkaSpout = new KafkaSpout(kafkaConfig);

        TopologyBuilder builder = new TopologyBuilder();

        builder.setSpout("eventsEmitter", kafkaSpout, 8);

        builder.setBolt("eventsProcessor", new RollingCountBolt(2, 1), 8)
                .fieldsGrouping("requestsEmitter", new Fields("request"));

        //More bolts stuffzz

        Config config = new Config();

        config.setMaxTaskParallelism(5);
        config.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, 2);
        config.put(Config.NIMBUS_HOST, nimbusHost);
        config.put(Config.NIMBUS_THRIFT_PORT, 6627);
        config.put(Config.STORM_ZOOKEEPER_PORT, 2181);
        config.put(Config.STORM_ZOOKEEPER_SERVERS, Arrays.asList(zkIp));

        try {
            StormSubmitter.submitTopology("my-topology", config, builder.createTopology());
        } catch (Exception e) {
            throw new IllegalStateException("Couldn't initialize the topology", e);
        }
    }

}

希望这有帮助

---

Jose Luis

你好，Jose，非常感谢你提供了到回购协议的链接。我可以在那里成功运行示例表单。当我使用LocalCluster时，数据被blots和kafka spoot占用。但当我使用StormSubmitter Kafka时，数据不会被消耗。我正在使用kafka控制台生成器和Storm vs 1.0.3将数据推送到kafka中