如何将Python使用者连接到AWS MSK
我正在尝试将python使用者连接到AWS MSK集群。我该怎么做 运行AWS MSK群集 我正在尝试使用python和kafka python使用来自MSK集群的消息 我犯了一个错误如何将Python使用者连接到AWS MSK,python,amazon-web-services,apache-kafka,aws-msk,Python,Amazon Web Services,Apache Kafka,Aws Msk,我正在尝试将python使用者连接到AWS MSK集群。我该怎么做 运行AWS MSK群集 我正在尝试使用python和kafka python使用来自MSK集群的消息 我犯了一个错误 Traceback (most recent call last): File "consumer.py", line 23, in <module> for message in consumer: File "/home/ubuntu/.local/l
Traceback (most recent call last):
File "consumer.py", line 23, in <module>
for message in consumer:
File "/home/ubuntu/.local/lib/python3.6/site-packages/kafka/consumer/group.py", line 1193, in __next__
return self.next_v2()
File "/home/ubuntu/.local/lib/python3.6/site-packages/kafka/consumer/group.py", line 1201, in next_v2
return next(self._iterator)
File "/home/ubuntu/.local/lib/python3.6/site-packages/kafka/consumer/group.py", line 1116, in _message_generator_v2
record_map = self.poll(timeout_ms=timeout_ms, update_offsets=False)
File "/home/ubuntu/.local/lib/python3.6/site-packages/kafka/consumer/group.py", line 655, in poll
records = self._poll_once(remaining, max_records, update_offsets=update_offsets)
File "/home/ubuntu/.local/lib/python3.6/site-packages/kafka/consumer/group.py", line 675, in _poll_once
self._coordinator.poll()
File "/home/ubuntu/.local/lib/python3.6/site-packages/kafka/coordinator/consumer.py", line 270, in poll
self.ensure_coordinator_ready()
File "/home/ubuntu/.local/lib/python3.6/site-packages/kafka/coordinator/base.py", line 258, in ensure_coordinator_ready
self._client.poll(future=future)
File "/home/ubuntu/.local/lib/python3.6/site-packages/kafka/client_async.py", line 582, in poll
self._maybe_connect(node_id)
File "/home/ubuntu/.local/lib/python3.6/site-packages/kafka/client_async.py", line 392, in _maybe_connect
conn.connect()
File "/home/ubuntu/.local/lib/python3.6/site-packages/kafka/conn.py", line 429, in connect
if self._try_handshake():
File "/home/ubuntu/.local/lib/python3.6/site-packages/kafka/conn.py", line 508, in _try_handshake
self._sock.do_handshake()
File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
OSError: [Errno 0] Error
回溯(最近一次呼叫最后一次):
文件“consumer.py”,第23行,在
消费者信息:
文件“/home/ubuntu/.local/lib/python3.6/site packages/kafka/consumer/group.py”,第1193行,下一步__
返回self.next_v2()
文件“/home/ubuntu/.local/lib/python3.6/site packages/kafka/consumer/group.py”,第1201行,下一个_v2
返回下一个(self.\u迭代器)
文件“/home/ubuntu/.local/lib/python3.6/site packages/kafka/consumer/group.py”,第1116行,在消息生成器v2中
record\u map=self.poll(timeout\u ms=timeout\u ms,update\u offset=False)
文件“/home/ubuntu/.local/lib/python3.6/site packages/kafka/consumer/group.py”,第655行,投票中
记录=自我。轮询一次(剩余、最大记录、更新偏移=更新偏移)
文件“/home/ubuntu/.local/lib/python3.6/site packages/kafka/consumer/group.py”,第675行,在一次投票中
self.\u协调人poll()
文件“/home/ubuntu/.local/lib/python3.6/site packages/kafka/coordinator/consumer.py”,第270行,投票中
self.确保协调器准备就绪()
文件“/home/ubuntu/.local/lib/python3.6/site packages/kafka/coordinator/base.py”,第258行,在确保协调员准备就绪
self.\u client.poll(未来=未来)
文件“/home/ubuntu/.local/lib/python3.6/site packages/kafka/client_async.py”,第582行,在轮询中
self.\u可能\u连接(节点\u id)
文件“/home/ubuntu/.local/lib/python3.6/site packages/kafka/client\u async.py”,第392行,在连接中
连接
文件“/home/ubuntu/.local/lib/python3.6/site packages/kafka/conn.py”,第429行,在connect中
如果是self.\u尝试握手():
文件“/home/ubuntu/.local/lib/python3.6/site packages/kafka/conn.py”,第508行,在“尝试握手”中
self.\u sock.do\u握手()
文件“/usr/lib/python3.6/ssl.py”,第1077行,在do_握手中
赛尔夫:握手
文件“/usr/lib/python3.6/ssl.py”,第689行,在do_握手中
赛尔夫:握手
操作错误:[错误号0]错误
使用kafka python:
from kafka import KafkaConsumer
if __name__ == '__main__':
topic_name = 'example-topic'
consumer = KafkaConsumer(topic_name, auto_offset_reset='earliest',
bootstrap_servers=['kafka2:9092'], api_version=(0, 10), consumer_timeout_ms=1000)
for msg in consumer:
print(msg.value)
if consumer is not None:
consumer.close()
你试过什么?你被困在哪里?您在搜索引擎中搜索过“Python Kafka”吗?目前,我在EC2中运行Kafka,并让Python消费者收听多个主题和处理。但是我需要运行AWS MSK集群并连接到它,所以将Python代码中的Kafka连接地址更改为MSK集群地址。如果这不起作用,请在问题中添加更多信息。这不起作用,这就是问题出现的原因…如果它不起作用,请在问题中提供错误消息。这可能根本不是Python的问题,而是AWS网络配置的问题,所以您也需要提供这些信息
from time import sleep
from kafka import KafkaProducer
# publish messages on topic
def publish_message(producer_instance, topic_name, key, value):
try:
key_bytes = bytes(key, encoding='utf-8')
value_bytes = bytes(value, encoding='utf-8')
producer_instance.send(topic_name, key=key_bytes, value=value_bytes)
producer_instance.flush()
print('Message ' + key + ' published successfully.')
except Exception as ex:
print('Exception in publishing message')
print(str(ex))
# establish kafka connection
def connect_kafka_producer():
_producer = None
try:
_producer = KafkaProducer(bootstrap_servers=['kafka1:9092'])
except Exception as ex:
print('Exception while connecting Kafka')
print(str(ex))
finally:
return _producer
if __name__ == '__main__':
kafka_producer = connect_kafka_producer()
x = 0
while True:
publish_message(kafka_producer, 'raw_recipes', str(x), 'This is message ' + str(x))
x += 1
if kafka_producer is not None:
kafka_producer.close()