Node.js Docker由Storm 0.10.0和Zookeeper组成（连接中断）_Node.js_Docker_Apache Kafka_Apache Storm_Apache Zookeeper

Node.js Docker由Storm 0.10.0和Zookeeper组成（连接中断）

node.js docker apache-kafka apache-storm apache-zookeeper

Node.js Docker由Storm 0.10.0和Zookeeper组成（连接中断）,node.js,docker,apache-kafka,apache-storm,apache-zookeeper,Node.js,Docker,Apache Kafka,Apache Storm,Apache Zookeeper,部署了以下体系结构： Apache Kafka和Zookeeper在docker容器中： Storm（Nimbus、Supervisor、UI）使用以下图像：使用以下“docker compose.yml” # A scalable cache # saves data to ./data/mongo mongo: image: mongo:3.0 container_name: "mongo" volumes: - "./data/mon

部署了以下体系结构：

Apache Kafka和Zookeeper在docker容器中：

Storm（Nimbus、Supervisor、UI）使用以下图像：

使用以下“docker compose.yml”

# A scalable cache
#     saves data to ./data/mongo
mongo:
    image: mongo:3.0
    container_name: "mongo"
    volumes:
      - "./data/mongo:/data/db"
    ports:
      - "27017:27017"

# Kafka (A scalable queue) + Zookeeper (Distributed conf. service)
# as of 2015.11 -- kafka:0.8.2.1, zookeeper:3.4.5+dfsg-2
kzk:
    image: eucm/kzk
    container_name: "kzk"
    volumes:
      - "./data/zookeeper:/var/lib/zookeeper"
      - "./data/kafka:/tmp/kafka-logs"
    environment:
      - ADVERTISED_HOST=localhost
      - AUTO_CREATE_TOPICS=true
      - DELETE_TOPICS=true
    ports:
      - "9092:9092"
      - "2181:2181"

# nimbus: supervises everything; talks to zookeeper
# in e-ucm/dockerized-storm -- storm:0.10.0
#    requires: kzk
nimbus:
    image: eucm/storm-nimbus
    container_name: "nimbus"
    environment:
      - KZK_PORT_2181_TCP_ADDR=kzk
      - NIMBUS_PORT_6627_TCP_ADDR=nimbus
      - UI_PORT_8081_TCP_ADDR=ui
    ports:
      - "6627:6627"

# ui: allows access to logs
# in e-ucm/dockerized-storm -- storm:0.10.0
#    requires: nimbus, kzk
ui:
    image: eucm/storm-ui
    container_name: "ui"
    environment:
      - KZK_PORT_2181_TCP_ADDR=kzk
      - NIMBUS_PORT_6627_TCP_ADDR=nimbus
      - UI_PORT_8081_TCP_ADDR=ui
    ports: 
        - "8081:8081"

# supervisor: supervises actual workers
# in e-ucm/dockerized-storm -- storm:0.10.0
#   requires: nimbus, kzk, mongo
supervisor:
    image: eucm/storm-supervisor
    container_name: "supervisor"
    environment:
      - KZK_PORT_2181_TCP_ADDR=kzk
      - NIMBUS_PORT_6627_TCP_ADDR=nimbus
      - UI_PORT_8081_TCP_ADDR=ui
    ports:
      - "6700:6700"
      - "6701:6701"
      - "6702:6702"
      - "6703:6703"

我们还使用NodeJS和kafka节点包（）创建一个名为“sessionTest”的主题

我将这个拓扑（）从主机发送给Storm。使用以下命令：

/home/dan/storm-test/storm/apache-storm-0.10.0/bin/storm jar /home/dan/storm-test/realtime-jar-with-dependencies.jar es.eucm.gleaner.realtime.RealTime sessionTest mongodb://localhost:27017/analytics-backend-test localhost:2181 -c nimbus.host=localhost

过了一会儿，通过查看Storm workers日志（在主管图像中），我们看到以下错误

2016-03-30 09:30:05.984 b.s.d.worker [INFO] Shutting down worker sessionTest-3-1459329789 2c94d927-2341-4649-bee6-b0bb13f2706f 6700
2016-03-30 09:30:05.985 b.s.d.worker [INFO] Shutting down receive thread
2016-03-30 09:30:06.006 b.s.m.n.Client [INFO] creating Netty Client, connecting to supervisor:6700, bufferSize: 5242880
2016-03-30 09:30:06.009 b.s.m.loader [INFO] Shutting down receiving-thread: [sessionTest-3-1459329789, 6700]
2016-03-30 09:30:06.009 b.s.m.n.Client [INFO] closing Netty Client Netty-Client-supervisor/172.20.0.5:6700
2016-03-30 09:30:06.009 b.s.m.n.Client [INFO] waiting up to 600000 ms to send 0 pending messages to Netty-Client-supervisor/172.20.0.5:6700
2016-03-30 09:30:06.009 b.s.m.loader [INFO] Waiting for receiving-thread:[sessionTest-3-1459329789, 6700] to die
2016-03-30 09:30:06.010 b.s.m.loader [INFO] Shutdown receiving-thread: [sessionTest-3-1459329789, 6700]
2016-03-30 09:30:06.010 b.s.d.worker [INFO] Shut down receive thread
2016-03-30 09:30:06.010 b.s.d.worker [INFO] Terminating messaging context
2016-03-30 09:30:06.010 b.s.d.worker [INFO] Shutting down executors
2016-03-30 09:30:06.011 b.s.d.executor [INFO] Shutting down executor $spoutcoord-spout0:[2 2]
2016-03-30 09:30:06.011 b.s.util [INFO] Async loop interrupted!
2016-03-30 09:30:06.789 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/sessionTest/partitions
    at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:81) ~[stormjar.jar:?]
    at storm.kafka.trident.ZkBrokerReader.<init>(ZkBrokerReader.java:42) ~[stormjar.jar:?]
    at storm.kafka.KafkaUtils.makeBrokerReader(KafkaUtils.java:57) ~[stormjar.jar:?]
    at storm.kafka.trident.Coordinator.<init>(Coordinator.java:33) ~[stormjar.jar:?]http://e-ucm.github.io/a2/#api-Login-Login
    at storm.kafka.trident.OpaqueTridentKafkaSpout.getCoordinator(OpaqueTridentKafkaSpout.java:46) ~[stormjar.jar:?]
    at storm.trident.spout.OpaquePartitionedTridentSpoutExecutor$Coordinator.<init>(OpaquePartitionedTridentSpoutExecutor.java:44) ~[storm-core-0.10.0.jar:0.10.0]
    at storm.trident.spout.OpaquePartitionedTridentSpoutExecutor.getCoordinator(OpaquePartitionedTridentSpoutExecutor.java:183) ~[storm-core-0.10.0.jar:0.10.0]
    at storm.trident.topology.MasterBatchCoordinator.open(MasterBatchCoordinator.java:113) ~[storm-core-0.10.0.jar:0.10.0]
    at backtype.storm.daemon.executor$fn__5624$fn__5639.invoke(executor.clj:564) ~[storm-core-0.10.0.jar:0.10.0]
    at backtype.storm.util$async_loop$fn__545.invoke(util.clj:477) [storm-core-0.10.0.jar:0.10.0]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45-internal]
Caused by: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/sessionTest/partitions
    at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:94) ~[stormjar.jar:?]
    at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:65) ~[stormjar.jar:?]
    ... 11 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/sessionTest/partitions
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[stormjar.jar:?]
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[stormjar.jar:?]
    at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:214) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:203) ~[stormjar.jar:?]
    at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:199) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:191) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:38) ~[stormjar.jar:?]
    at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:91) ~[stormjar.jar:?]
    at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:65) ~[stormjar.jar:?]
    ... 11 more
2016-03-30 09:30:06.794 b.s.d.executor [ERROR] 
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/sessionTest/partitions
    at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:81) ~[stormjar.jar:?]
    at storm.kafka.trident.ZkBrokerReader.<init>(ZkBrokerReader.java:42) ~[stormjar.jar:?]
    at storm.kafka.KafkaUtils.makeBrokerReader(KafkaUtils.java:57) ~[stormjar.jar:?]
    at storm.kafka.trident.Coordinator.<init>(Coordinator.java:33) ~[stormjar.jar:?]
    at storm.kafka.trident.OpaqueTridentKafkaSpout.getCoordinator(OpaqueTridentKafkaSpout.java:46) ~[stormjar.jar:?]
    at storm.trident.spout.OpaquePartitionedTridentSpoutExecutor$Coordinator.<init>(OpaquePartitionedTridentSpoutExecutor.java:44) ~[storm-core-0.10.0.jar:0.10.0]
    at storm.trident.spout.OpaquePartitionedTridentSpoutExecutor.getCoordinator(OpaquePartitionedTridentSpoutExecutor.java:183) ~[storm-core-0.10.0.jar:0.10.0]
    at storm.trident.topology.MasterBatchCoordinator.open(MasterBatchCoordinator.java:113) ~[storm-core-0.10.0.jar:0.10.0]
    at backtype.storm.daemon.executor$fn__5624$fn__5639.invoke(executor.clj:564) ~[storm-core-0.10.0.jar:0.10.0]
    at backtype.storm.util$async_loop$fn__545.invoke(util.clj:477) [storm-core-0.10.0.jar:0.10.0]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45-internal]
Caused by: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/sessionTest/partitions
    at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:94) ~[stormjar.jar:?]
    at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:65) ~[stormjar.jar:?]
    ... 11 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/sessionTest/partitions
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[stormjar.jar:?]
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[stormjar.jar:?]
    at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:214) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:203) ~[stormjar.jar:?]
    at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:199) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:191) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:38) ~[stormjar.jar:?]
    at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:91) ~[stormjar.jar:?]
    at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:65) ~[stormjar.jar:?]
    ... 11 more
2016-03-30 09:30:06.805 b.s.util [ERROR] Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
    at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:336) [storm-core-0.10.0.jar:0.10.0]
    at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.6.0.jar:?]
    at backtype.storm.daemon.worker$fn__7188$fn__7189.invoke(worker.clj:536) [storm-core-0.10.0.jar:0.10.0]
    at backtype.storm.daemon.executor$mk_executor_data$fn__5523$fn__5524.invoke(executor.clj:261) [storm-core-0.10.0.jar:0.10.0]
    at backtype.storm.util$async_loop$fn__545.invoke(util.clj:489) [storm-core-0.10.0.jar:0.10.0]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45-internal]

2016-03-30 09:30:05.984 b.s.d.工人[信息]关闭工人会话测试-3-1459329789 2c94d927-2341-4649-bee6-b0bb13f2706f 6700
2016-03-30 09:30:05.985 b.s.d.工人[信息]正在关闭接收线程
2016-03-30 09:30:06.006 b.s.m.n.Client[信息]创建Netty客户端，连接到主管：6700，缓冲区大小：5242880
2016-03-30 09:30:06.009 b.s.m.loader[信息]关闭接收线程：[会话测试-3-14593297896700]
2016-03-30 09:30:06.009 b.s.m.n.客户[信息]结清净客户净客户主管/172.20.0.5:6700
2016-03-30 09:30:06.009 b.s.m.n.客户[信息]等待600000毫秒，向Netty客户主管发送0条未决消息/172.20.0.5:6700
2016-03-30 09:30:06.009 b.s.m.loader[INFO]等待接收线程[sessionTest-3-14593297896700]死亡
2016-03-30 09:30:06.010 b.s.m.loader[信息]关机接收线程：[会话测试-3-14593297896700]
2016-03-30 09:30:06.010 b.s.d.工人[信息]关闭接收线程
2016-03-30 09:30:06.010 b.s.d.worker[信息]终止消息上下文
2016-03-30 09:30:06.010 b.s.d.工人[信息]关闭执行器
2016-03-30 09:30:06.011 b.s.d.执行人[信息]关闭执行人$spoutcord-spout0:[2]
2016-03-30 09:30:06.011 b.s.util[INFO]异步循环中断！
2016-03-30 09:30:06.789 b.s.util[ERROR]异步循环死机！
java.lang.RuntimeException:java.lang.RuntimeException:org.apache.zookeeper.KeeperException$ConnectionLossException:KeeperErrorCode=ConnectionLoss for/brokers/topics/sessionTest/partitions
在storm.kafka.dynamicbrokerreader.getBrokerInfo（dynamicbrokerreader.java:81）~[stormjar.jar:？]
在storm.kafka.trident.ZkBrokerReader.（ZkBrokerReader.java:42）~[stormjar.jar:？]
在storm.kafka.KafkaUtils.makeBrokerReader（KafkaUtils.java:57）~[stormjar.jar:？]
在storm.kafka.trident.Coordinator.（Coordinator.java:33）~[stormjar.jar:？]http://e-ucm.github.io/a2/#api-登录
在storm.kafka.trident.opaquetridentkafkafkaspout.getCoordinator（OpaqueTridentKafkaSpout.java:46）~[stormjar.jar:？]
在storm.trident.spoot.opaquepartitionedtridentspootexecutor$协调器处。（opaquepartitionedtridentspootexecutor.java:44）~[storm-core-0.10.0.jar:0.10.0]
在storm.trident.spoot.opaQuePartitionedTridentSpootExecutor.getCoordinator（opaQuePartitionedTridentSpootExecutor.java:183）~[storm-core-0.10.0.jar:0.10.0]
打开（MasterBatchCoordinator.java:113）~[storm-core-0.10.0.jar:0.10.0]
在backtype.storm.daemon.executor$fn\u 5624$fn\u 5639.invoke（executor.clj:564）~[storm-core-0.10.0.jar:0.10.0]
在backtype.storm.util$async\u循环$fn\uu 545.invoke（util.clj:477）[storm-core-0.10.0.jar:0.10.0]
在clojure.lang.AFn.run（AFn.java:22）[clojure-1.6.0.jar:？]
在java.lang.Thread.run（Thread.java:745）[？：1.8.0_45-internal]
原因：java.lang.RuntimeException:org.apache.zookeeper.KeeperException$ConnectionLossException:KeeperErrorCode=ConnectionLoss for/brokers/topics/sessionTest/partitions
在storm.kafka.dynamicBrokerReader.getNumPartitions（dynamicBrokerReader.java:94）~[stormjar.jar:？]
在storm.kafka.dynamicbrokerreader.getBrokerInfo（dynamicbrokerreader.java:65）~[stormjar.jar:？]
... 还有11个
原因：org.apache.zookeeper.KeeperException$ConnectionLossException:KeeperErrorCode=ConnectionLoss for/brokers/topics/sessionTest/partitions
在org.apache.zookeeper.keeprexception.create（keeprexception.java:99）~[stormjar.jar:？]
在org.apache.zookeeper.keeprexception.create（keeprexception.java:51）~[stormjar.jar:？]
在org.apache.zookeeper.zookeeper.getChildren（zookeeper.java:1590）~[stormjar.jar:？]
在org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call（GetChildrenBuilderImpl.java:214）~[stormjar.jar:？]
在org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call（GetChildrenBuilderImpl.java:203）~[stormjar.jar:？]
在org.apache.curator.RetryLoop.callWithRetry（RetryLoop.java:107）~[stormjar.jar:？]
在org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground（GetChildrenBuilderImpl.java:199）~[stormjar.jar:？]
在org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath（GetChildrenBuilderImpl.java:191）~[stormjar.jar:？]
在org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath（GetChildrenBuilderImpl.java:38）~[stormjar.jar:？]
在storm.kafka.dynamicBrokerReader.getNumPartitions（dynamicBrokerReader.java:91）~[stormjar.jar:？]
在storm.kafka.dynamicbrokerreader.getBrokerInfo（dynamicbrokerreader.java:65）~[stormjar.jar:？]
... 还有11个
2016-03-30 09:30:06.794 b.s.d.执行人[错误]
java.lang.RuntimeException:java.lang.RuntimeException:org.apache.zookeeper.KeeperException$ConnectionLossException:KeeperErrorCode=ConnectionLoss for/brokers/topics/sessionTest/partitions
在storm.kafka.dynamicbrokerreader.getBrokerInfo（dynamicbrokerreader.java:81）~[stormjar.jar:？]
在storm.kafka.trident.ZkBrokerReader.（ZkBrokerReader.java:42）~[stormjar.jar:？]
在storm.kafka.KafkaUtils.makeBrokerReader（KafkaUtils.java:57）~[stormjar.jar:？]
在storm.kafka.trident.Coordinator.（Coordinator.java:33）~[stormjar.jar:？]
在storm.kafka.trident.opaquetridentkafkafkaspout.getCoordinator（OpaqueTridentKafkaSpout.java:46）~[stormjar.jar:？]
在暴风雨中。三叉戟。喷口。不透明分区。第三喷口执行器$协调器。（不透明）