Fatal编程技术网
  • docker/
  • 在docker中运行clickhouse和Kafka

在docker中运行clickhouse和Kafka

docker apache-kafka

在docker中运行clickhouse和Kafka,docker,apache-kafka,clickhouse,Docker,Apache Kafka,Clickhouse,我试图将卡夫卡与clickhouse结合起来。我想从restapi向kafka发送一些数据，并将其传递给clickhouse，它将成为我的数据库。到目前为止，我能够让卡夫卡独立工作，并将数据从一端传递到另一端。但当我试图在docker中使用它时，我遇到了一些问题。我可以看到卡夫卡端使用卡夫卡工具使用的数据。但我的clickhouse显然没有连接。这就是错误，当我运行docker时，我不断得到： clickhouse-server_1 | %3|1584611439.583|FAIL|Click

我试图将卡夫卡与clickhouse结合起来。我想从restapi向kafka发送一些数据，并将其传递给clickhouse，它将成为我的数据库。到目前为止，我能够让卡夫卡独立工作，并将数据从一端传递到另一端。但当我试图在docker中使用它时，我遇到了一些问题。我可以看到卡夫卡端使用卡夫卡工具使用的数据。但我的clickhouse显然没有连接。这就是错误，当我运行docker时，我不断得到：

clickhouse-server_1  | %3|1584611439.583|FAIL|ClickHouse 20.3.3.6#consumer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 1ms in state CONNECT)
clickhouse-server_1  | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 1ms in state CONNECT)
clickhouse-server_1  | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-1| [thrd:localhost:9092/bootstrap]: 1/1 brokers are down
clickhouse-server_1  | %3|1584611439.583|FAIL|ClickHouse 20.3.3.6#consumer-2| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
clickhouse-server_1  | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-2| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
clickhouse-server_1  | %3|1584611439.583|ERROR|ClickHouse 20.3.3.6#consumer-2| [thrd:localhost:9092/bootstrap]: 1/1 brokers are down
我还添加了以下日志文件：

2020.03.19 10:16:11.637331 [ 1 ] {} <Error> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.3.6 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2020.03.19 10:16:11.637658 [ 1 ] {} <Error> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.3.6 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2020.03.19 10:16:11.637854 [ 1 ] {} <Error> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = DNS error: EAI: -9 (version 20.3.3.6 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2020.03.19 10:16:11.678748 [ 74 ] {} <Error> void DB::DDLWorker::runMainThread(): Code: 999, e.displayText() = Coordination::Exception: All connection tries failed while connecting to ZooKeeper. Addresses: 192.168.144.3:2181
Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 20.3.3.6 (official build)), 192.168.144.3:2181
Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 20.3.3.6 (official build)), 192.168.144.3:2181
Poco::Exception. Code: 1000, e.code() = 111, e.displayText() = Connection refused (version 20.3.3.6 (official build)), 192.168.144.3:2181
 (Connection loss), Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x102d352c in /usr/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x8f2d989 in /usr/bin/clickhouse
2. Coordination::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) @ 0xdd300e4 in /usr/bin/clickhouse
3. Coordination::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0xdd3070e in /usr/bin/clickhouse
4. ? @ 0xdd638b2 in /usr/bin/clickhouse
5. Coordination::ZooKeeper::ZooKeeper(std::__1::vector<Poco::Net::SocketAddress, std::__1::allocator<Poco::Net::SocketAddress> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Poco::Timespan, Poco::Timespan, Poco::Timespan) @ 0xdd6099b in /usr/bin/clickhouse
6. zkutil::ZooKeeper::init(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xdd3fd53 in /usr/bin/clickhouse
7. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xdd405bd in /usr/bin/clickhouse
8. DB::Context::getZooKeeper() const @ 0xce5fb97 in /usr/bin/clickhouse
9. DB::DDLWorker::getAndSetZooKeeper() @ 0xce84c53 in /usr/bin/clickhouse
10. DB::DDLWorker::runMainThread() @ 0xce8eae3 in /usr/bin/clickhouse
11. ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::'lambda'()::operator()() const @ 0xce96031 in /usr/bin/clickhouse
12. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x8f50b07 in /usr/bin/clickhouse
13. ? @ 0x8f4f00f in /usr/bin/clickhouse
14. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
15. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so
 (version 20.3.3.6 (official build))
我的卡夫卡引导部分：

producer = KafkaProducer(bootstrap_servers=['kafka://localhost:9092'], value_serializer=lambda m: json.dumps(m).encode('ascii'))
和我的docker配置部分：

version: "3.6"

services:
  clickhouse-server:
    image: yandex/clickhouse-server
    volumes:
      - ./clickhouse/config.xml:/etc/clickhouse-server/config.xml
      - ./clickhouse/zookeeper-servers.xml:/etc/clickhouse-server/conf.d/zookeeper-servers.xml
      - ./shared/ch-data/clickhouse:/var/lib/clickhouse
    depends_on:
      - kafka
    links:
      - kafka
    ports:
      - 9000:9000

  clickhouse-client:
    image: yandex/clickhouse-client
    entrypoint:
      - /bin/sleep
    command:
      - infinity

  kafka:
    image: wurstmeister/kafka:2.11-1.0.2
    volumes:
      - ./shared/ch-data/kafka:/data
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: 192.168.1.86
    links:
     - zookeeper
    ports:
     - 9092:9092
     - 9094:9094

  zookeeper:
    image: zookeeper
    volumes:
      - ./shared/ch-data/zookeeper:/data
    ports:
      - 2181:2181
我对引导服务器和卡夫卡监听器部分特别困惑。据我所知，我作为卡夫卡的听众提供了我的主机IP，从我的理解来看，你似乎误解了Docker网络。最重要的是，
链接
功能已被弃用，因此我不确定您从何处复制了撰写文件

localhost
将始终是当前容器，而不是另一个容器

修正：
kafka://localhost:9092
&
引导服务器=['kafka://localhost:9092']
以使用实际的Docker服务名称

据我所知，我已经作为卡夫卡听众提供了我的主机IP

  • 只有在使用多台物理计算机时，才需要主机IP
  • 如果您在同一台物理机器上使用Docker之外的Kafka客户端，则只需公布localhost
  • 您应该始终在Docker网络中为客户端公布kafka容器服务名称
    • 总之，
    KAFKA_侦听器将有3-4个端口，第四个端口用于复制的内部安全协议。你做多少广告取决于你自己
    
您需要添加协议映射，正如在该博客文章中所讨论的
    
请注意，您不需要使用任何特定的卡夫卡或Zookeeper docker图像，并且博客中提供的这些组成文件可以正常工作。另外，永远不要相信两个不同的Docker映像共享同一组环境变量
    
wurstmeister映像有自己的Github页面讨论侦听器
    

    version: "3.6"

services:
  clickhouse-server:
    image: yandex/clickhouse-server
    volumes:
      - ./clickhouse/config.xml:/etc/clickhouse-server/config.xml
      - ./clickhouse/zookeeper-servers.xml:/etc/clickhouse-server/conf.d/zookeeper-servers.xml
      - ./shared/ch-data/clickhouse:/var/lib/clickhouse
    depends_on:
      - kafka
    links:
      - kafka
    ports:
      - 9000:9000

  clickhouse-client:
    image: yandex/clickhouse-client
    entrypoint:
      - /bin/sleep
    command:
      - infinity

  kafka:
    image: wurstmeister/kafka:2.11-1.0.2
    volumes:
      - ./shared/ch-data/kafka:/data
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: 192.168.1.86
    links:
     - zookeeper
    ports:
     - 9092:9092
     - 9094:9094

  zookeeper:
    image: zookeeper
    volumes:
      - ./shared/ch-data/zookeeper:/data
    ports:
      - 2181:2181