Java 风暴/卡夫卡-无法获得卡夫卡的偏移滞后
我正在运行一个Storm拓扑,它在AWS上收到卡夫卡的推文 Ubuntu服务器14.04 LTS实例有4个节点-Nimbus、一个管理器、一个Kafka Zookeeper节点、一个Zookeeper(用于Storm群集)。我的Storm UI已启动并运行,我能够提交拓扑。我有两个代理,但我只使用代理。id=0一个。我有一个主题下的推特。我的卡夫卡服务器也运行良好 我是这样创建卡夫卡主题的:Java 风暴/卡夫卡-无法获得卡夫卡的偏移滞后,java,ubuntu,apache-kafka,apache-storm,Java,Ubuntu,Apache Kafka,Apache Storm,我正在运行一个Storm拓扑,它在AWS上收到卡夫卡的推文 Ubuntu服务器14.04 LTS实例有4个节点-Nimbus、一个管理器、一个Kafka Zookeeper节点、一个Zookeeper(用于Storm群集)。我的Storm UI已启动并运行,我能够提交拓扑。我有两个代理,但我只使用代理。id=0一个。我有一个主题下的推特。我的卡夫卡服务器也运行良好 我是这样创建卡夫卡主题的: bin/kafka-topics.sh--create--zookeeper localhost:218
bin/kafka-topics.sh--create--zookeeper localhost:2181/kafka--replication factor 1--partitions 1--topic twitter1 我感到困惑的是:
SpoutConfig kafkaConfig = new SpoutConfig(kafkaHosts, topicName+"-0", "/kafka", topicName+"-0");
我想我的错误正是从这一点开始的。完整代码为:
import org.apache.storm.tuple.Fields;
import org.apache.storm.kafka.BrokerHosts;
import org.apache.storm.kafka.KafkaSpout;
import org.apache.storm.kafka.SpoutConfig;
import org.apache.storm.kafka.ZkHosts;
import java.util.Arrays;
import org.apache.storm.Config;
import org.apache.storm.StormSubmitter;
import org.apache.storm.spout.SchemeAsMultiScheme;
import org.apache.storm.topology.TopologyBuilder;
import org.apache.storm.kafka.StringScheme;
public class TwitterTopology{
public static void main(String[] args) {
String topicName = "twitter1";
String topologyName = args[0];
String kafkaIp = "xxx.31.xxx.207"; //hiding the IPs here. This is the IP for my kafka-zk node. Is this ok?
String nimbusHost = "xxx.31.xxx.70";
String kafkaHost = kafkaIp + ":9092";
BrokerHosts kafkaHosts = new ZkHosts(kafkaHost);
SpoutConfig kafkaConfig = new SpoutConfig(kafkaHosts, topicName, "/kafka", topicName);
kafkaConfig.scheme = new SchemeAsMultiScheme(new StringScheme());
KafkaSpout kafkaSpout = new KafkaSpout(kafkaConfig);
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("twitter-spout", kafkaSpout, 8);
builder.setBolt("WordSplitterBolt", new JsonWordSplitterBolt(5)).shuffleGrouping("twitter-spout");
builder.setBolt("IgnoreWordsBolt", new IgnoreWordsBolt()).shuffleGrouping("WordSplitterBolt");
builder.setBolt("WordCounterBolt", new WordCounterBolt(5, 5 * 60, 50)).shuffleGrouping("IgnoreWordsBolt");
Config config = new Config();
config.setDebug(false);
config.setMaxTaskParallelism(5);
config.put(Config.NIMBUS_HOST, nimbusHost);
config.put(Config.NIMBUS_THRIFT_PORT, 6627);
config.put(Config.STORM_ZOOKEEPER_PORT, 2181);
config.put(Config.STORM_ZOOKEEPER_SERVERS, Arrays.asList(kafkaIp));
try {
config.setNumWorkers(20);
config.setMaxSpoutPending(5000);
StormSubmitter.submitTopology(topologyName, config, builder.createTopology());
} catch (Exception e) {
throw new IllegalStateException("Couldn't initialize the topology", e);
}
}
}
我在Storm UI中遇到此异常:
Unable to get offset lags for kafka. Reason: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/twitter1/partitions at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590) at org.apache.curator.shaded.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:242) at org.apache.curator.shaded.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:231) at org.apache.curator.shaded.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64) at org.apache.curator.shaded.RetryLoop.callWithRetry(RetryLoop.java:100) at org.apache.curator.shaded.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:228) at org.apache.curator.shaded.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:219) at org.apache.curator.shaded.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:41) at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getLeadersAndTopicPartitions(KafkaOffsetLagUtil.java:319) at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:256) at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:124)
错误无法获取kafka的偏移滞后
保持不变,而异常的另一部分根据我更改的zkroot路径进行更改(在SpoutConfig
中的第三个参数)。我不知道如何准确地填补这些论点,让卡夫卡从我的主题中获取推文
我使用此处提供的教程编写了提交拓扑的代码:
我对maven依赖项做了许多更改。My pom.xml具有storm core、kafka等的所有依赖项,最新版本可在maven repo中找到。zkHosts()应该包含zookeeper的配置,而不是kafka。如果你的zookeeper和kafka在同一台服务器上
尝试为zookeeper(2181)提供正确的端口
提及