带Docker Swarm的卡夫卡-如何执行卡夫卡命令

我刚接触卡夫卡，有点困惑

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

到目前为止，我已经用以下命令创建了两台docker机器，一台是manager，一台是worker：

docker-machine create manager
docker-machine create worker1

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

我在docker群中添加了这两个节点

docker@manager:~$ docker node ls                                                                                                                                                                             
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
6bmovp3hr0j2w5irmexvvjgzq *   manager             Ready               Active              Leader              19.03.5
mtgbd9bg8d6q0lk9ycw10bxos     worker1             Ready               Active                                  19.03.5

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

我想在docker swarm中创建一个卡夫卡集群。我的docker-compose.yml看起来像这样

version: '3.2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - target: 9094
        published: 9094
        protocol: tcp
        mode: host
    environment:
      HOSTNAME_COMMAND: "hostname | awk -F'-' '{print $$2}'"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

其中hostname | awk-F'-''{print$$2}命令返回工作进程的管理器

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

然后我使用命令docker stack deploy-c docker-compose.yml kafka进行部署，该命令返回

Creating network kafka_default
Creating service kafka_zookeeper
Creating service kafka_kafka

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

我检查了th服务是否与docker服务ls一起运行

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

然后我用docker service scale Kafkau kafka=2来缩放服务。当我执行docker服务ps kafka_kafka时，我得到

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

到目前为止，我有一个动物园管理员和两个卡夫卡经纪人。一个经纪人负责经理，一个经纪人负责工人。这意味着工作者上的代理复制manager上的代理

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

---

我还想知道在这一步中如何运行Kafka命令。假设我想创建一个主题。我怎样才能做到这一点？我必须从这里下载卡夫卡二进制文件：？如果答案是肯定的，我得去动物园管理员服务器？它还没有运行吗

要在容器内运行kafka命令，首先需要获取kafka容器id

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

$ docker ps -a  
  CONTAINER ID                 IMAGE  
   1239075412         wurstmeister/kafka:latest  

现在拿这个容器来说，卡夫卡的情况会有所不同

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

$ docker exec -it 1239075412 sh  

日志位于以下文件夹中

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

# cd kafka/kafka-logs-1239075412  

kafka命令将出现在此容器类型kafka中，然后按tab键两次 您将获得用于创建主题等的所有kafka命令

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

这意味着工作者上的代理复制manager上的代理

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

就你的问题而言，卡夫卡可以说是无大师。所有代理的功能都是相同的，它们的进程仅为复制而相互通信

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

如果您想深入了解内部，可以研究运行在其中一个代理上的控制器进程

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

我得去动物园管理员服务器？它还没有运行吗

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

是的。参考-kafka_动物园管理员图片

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

假设我想创建一个主题。我怎样才能做到这一点

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

当然，您可以完全跳过使用CLI命令，而使用另一个容器，如

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

我必须从这里下载卡夫卡二进制文件

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

---

否则，是/否。如果您想从主机创建主题，则是。容器内部也可以工作提示：了解docker compose exec

docker机器基本上不推荐使用，好了，为什么要使用它？另外，您不需要挂载docker.sock文件。谢谢您的回答。所以动物园管理员和卡夫卡跑得一样好。如何创建主题？我不明白清楚这部分…好吧，你可以跳过使用CLI命令。。。如果我不想跳过CLI命令，该怎么办？抱歉，我是docker和kafka的新手。您运行我链接到的服务。是否作为Docker容器。如果您对这两种技术都不熟悉，那么不要同时使用这两种技术。只需在卡夫卡官方网站上安装卡夫卡和Zookeeper即可。我强烈建议不要通过卡夫卡或动物园管理员开始学习Docker

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp