Apache storm 风暴卡夫卡喷口慢慢消耗

我刚刚试过这里提到的卡夫卡风暴喷口，我使用的配置如下

    BrokerHosts brokerHosts = KafkaConfig.StaticHosts.fromHostString(
            ImmutableList.of("localhost"), 1);
    SpoutConfig spoutConfig = new SpoutConfig(brokerHosts, // list of Kafka
            "test", // topic to read from
            "/kafkastorm", // the root path in Zookeeper for the spout to
            "discovery"); // an id for this consumer for storing the
                            // consumer offsets in Zookeeper
    spoutConfig.scheme = new StringScheme();
    spoutConfig.stateUpdateIntervalMs = 1000;


    KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);

    TridentTopology topology = new TridentTopology();
    InetSocketAddress inetSocketAddress = new InetSocketAddress(
            "localhost", 6379);
    TridentState wordsCount = topology
            .newStream(SPOUT_FIRST, kafkaSpout)
            .parallelismHint(1)
            .each(new Fields("str"), new TestSplit(), new Fields("words"))
            .groupBy(new Fields("words"))
            .persistentAggregate(
                    RedisState.transactional(inetSocketAddress),
                    new Count(), new Fields("counts")).parallelismHint(100);

    Config conf = new Config();
    conf.setMaxTaskParallelism(200);
    // conf.setDebug( true );
    // conf.setMaxSpoutPending(20);

    // This topology can only be run as local because it is a toy example
    LocalDRPC drpc = new LocalDRPC();
    LocalCluster cluster = new LocalCluster();
    cluster.submitTopology("symbolCounter", conf, topology.build());

但是上面的喷口从卡夫卡主题获取消息的速度大约是7000/秒，但我预计每秒的负载大约是50000条消息。我尝试了各种方法来增加spoutConfig中的提取缓冲区大小，但没有明显的结果

---

有没有人遇到过类似的问题，他无法以制作人生成消息的速度通过storm获取卡夫卡主题？

我将配置中的“topology.spout.max.batch.size”值更新为大约64*1024，然后storm处理变得很快

只是一个想法，可能是因为暴风雨中的阿克林，而不是因为卡夫卡斯波特，你看到关闭阿克林有什么不同吗？您的主题的分区数是多少。降低maxSpoutPending值还有什么变化？@user2720864问题是kafspout能够获取所有大量的消息，但它在更新zookeeper阶段上花费了一些时间，而zookeeper阶段在其自身阶段继续进行。我将尝试您的建议。@user2720864:我将配置中的“topology.spout.max.batch.size”值更新为大约64*1024，然后storm处理变得很快。感谢您共享配置！