Apache kafka filebeat到kafka:无法连接到代理

Apache kafka filebeat到kafka:无法连接到代理,apache-kafka,apache-zookeeper,hadoop2,filebeat,Apache Kafka,Apache Zookeeper,Hadoop2,Filebeat,我是apache环境中的新手,目前我正试图将日志数据从filebeat producer发送到kafka broker 环境: 卡夫卡2.11(通过ambari安装) filebeat 7.4.2(windows) 我试图将日志从filebeat发送到ambari,我启动了kafka服务器并创建了名为“test”的主题,它被列在--list上。我对卡夫卡经纪人的港口很困惑。在一些教程中,我看到他们使用的是9092而不是2181。现在,我应该使用哪个端口从filebeat发送日志 这是我的file

我是apache环境中的新手,目前我正试图将日志数据从filebeat producer发送到kafka broker

环境:
  • 卡夫卡2.11(通过ambari安装)
  • filebeat 7.4.2(windows)
  • 我试图将日志从filebeat发送到ambari,我启动了kafka服务器并创建了名为“test”的主题,它被列在--list上。我对卡夫卡经纪人的港口很困惑。在一些教程中,我看到他们使用的是9092而不是2181。现在,我应该使用哪个端口从filebeat发送日志

    这是我的filebeat.conf 后果 这让我怀疑我是否真的得到了9092端口。因此,我查看了server.properties。我最关心的一些是:

    port=6667
    listeners=PLAINTEXT://x.x.x.x:6667
    
    然后我再次尝试执行filebeat.conf并将端口9092更改为6667,结果如下 问题 发生了什么事?应该使用哪个端口?每个端口的用途是什么

    任何回复都将不胜感激。多谢各位

    使现代化
    根据这一点,正确的来源是6667,因为kafka是通过ambari安装的

    可以使用的端口没有限制,它只取决于可用性。 在第一种情况下,正如您所说,代理可以在6667上启动,因此9092上没有运行任何进程

    2020-06-10T09:18:01.451+0700信息卡夫卡/日志。转到:53未能 连接到代理x.x.x.x:6667:拨号tcp:lookup x.x.x.x:没有这样的主机

    接下来,当您提到
    播发的.listeners
    属性时,您应该确保在
    播发的.listeners
    中提到的IP是分配给该机器的IP。您不能分配
    1.1.1.1:9092
    (仅举一些示例)

    执行
    ifconfig
    (linux)、
    ipconfig
    (windows)并在可从应用程序计算机访问的网络接口上查看计算机的IP。 在linux中,主要是eth0

    必须可以从运行应用程序的计算机访问此IP。
    因此,运行应用程序的计算机应该能够解析该IP。您可能还需要检查您的Kafka broker与正在运行应用程序的计算机之间的网络连接。

    您好,非常感谢您的回复,非常感谢。我已经通过ping确保制作人能够看到卡夫卡的主机名。似乎我必须编辑server.properties并添加播发的侦听器。我添加了advised.listeners=(与侦听器默认值相同),然后重新启动它。然后我就跑了。没有任何变化。我的卡夫卡是通过ambari安装的。除了编辑server.properties之外,我还有其他方法可以添加adverted.listeners吗?您的机器的IP是多少,可以从您的生产者那里ping?您在播发的
    侦听器中是否使用了相同的IP?是的,但我将主机名作为播发的.listeners,因为侦听器使用的是主机名而不是IP。知识产权是必须的吗?谢谢,我想不是这样,只要主机名可以由生产者应用程序解析。你必须确保你所付出的一切都是你在制作人身上付出的,并且必须是可以解决的。不过,您可以尝试将其更改为IP并进行检查。
    
    2020-06-10T09:00:32.214+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
    
    2020-06-10T09:00:32.214+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
    
    2020-06-10T09:00:32.215+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
    2020-06-10T09:00:32.215+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
    2020-06-10T09:00:32.215+0700    INFO    kafka/log.go:53 client/metadata retrying after 250ms... (3 attempts remaining)
    
    2020-06-10T09:00:32.466+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092
    
    2020-06-10T09:00:34.475+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
    
    2020-06-10T09:00:34.475+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
    
    2020-06-10T09:00:34.477+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
    2020-06-10T09:00:34.477+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
    2020-06-10T09:00:34.478+0700    INFO    kafka/log.go:53 client/metadata retrying after 250ms... (2 attempts remaining)
    
    2020-06-10T09:00:34.729+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092
    
    2020-06-10T09:00:36.737+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
    
    2020-06-10T09:00:36.737+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
    
    2020-06-10T09:00:36.738+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
    2020-06-10T09:00:36.738+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
    2020-06-10T09:00:36.738+0700    INFO    kafka/log.go:53 client/metadata retrying after 250ms... (1 attempts remaining)
    
    2020-06-10T09:00:36.989+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092
    
    2020-06-10T09:00:39.002+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
    
    2020-06-10T09:00:39.002+0700    INFO    kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
    
    2020-06-10T09:00:39.004+0700    INFO    kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
    2020-06-10T09:00:39.004+0700    INFO    kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
    2020-06-10T09:00:39.004+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092
    
    port=6667
    listeners=PLAINTEXT://x.x.x.x:6667
    
    2020-06-10T09:18:01.448+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:6667
    
    2020-06-10T09:18:01.450+0700    INFO    kafka/log.go:53 producer/broker/1001 starting up
    
    2020-06-10T09:18:01.451+0700    INFO    kafka/log.go:53 producer/broker/1001 state change to [open] on test/0
    
    2020-06-10T09:18:01.451+0700    INFO    kafka/log.go:53 producer/leader/test/0 selected broker 1001
    
    2020-06-10T09:18:01.451+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:6667: dial tcp: lookup x.x.x.x: no such host
    
    2020-06-10T09:18:01.452+0700    INFO    kafka/log.go:53 producer/broker/1001 state change to [closing] because dial tcp: lookup x.x.x.x: no such host
    
    2020-06-10T09:18:01.453+0700    DEBUG   [kafka] kafka/client.go:264     finished kafka batch
    2020-06-10T09:18:01.453+0700    DEBUG   [kafka] kafka/client.go:278     Kafka publish failed with: dial tcp: lookup x.x.x.x: no such host
    2020-06-10T09:18:01.454+0700    INFO    kafka/log.go:53 producer/leader/test/0 state change to [flushing-3]
    
    2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/leader/test/0 state change to [normal]
    
    2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/leader/test/0 state change to [retrying-3]
    
    2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/leader/test/0 abandoning broker 1001
    
    2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/broker/1001 shut down