Apache kafka 如何覆盖融合python客户端kafka中的默认配置值？

作为初学者，我正在探索ApacheKafka和融合的Kafka python客户端。当我尝试从生产者发送简单消息时，消费者能够成功地消费消息。我想我会尝试将图像作为有效载荷发送。因此，继续使用1MB（png）图像，我的制作人无法生成消息。我遇到的错误是

  p.produce('mytopic', callback=delivery_report, key='hello', value=str_value)
cimpl.KafkaException: KafkaError{code=MSG_SIZE_TOO_LARGE,val=10,str="Unable to produce message: Broker: Message size too large"}

虽然我在谷歌上搜索了一下发现了 因此，我修改了我的server.props（代理端），如下所示：

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
max.message.bytes=1048576 
message.max.bytes=1048576 
replica.fetch.max.bytes=1048576

但我仍然无法解决这个问题

producer.py

from confluent_kafka import Producer
import base64
import time

# some_data_source = ['hey', 'hi']

with open("1mb.png", "rb") as imageFile:
    str_value = base64.b64encode(imageFile.read())

p = Producer({'bootstrap.servers': 'localhost:9092', 'compression.type': 'snappy'})

def delivery_report(err, msg):
    """ Called once for each message produced to indicate delivery result.
        Triggered by poll() or flush(). """
    if err is not None:
        print('Message delivery failed: {}'.format(err))
    else:
        print('Message delivered to {} [{}]'.format(msg.topic(), msg.partition()))

for _ in range(2):
    # Trigger any available delivery report callbacks from previous produce() calls
    p.poll(0)

    # Asynchronously produce a message, the delivery report callback
    # will be triggered from poll() above, or flush() below, when the message has
    # been successfully delivered or failed permanently.
    p.produce('mytopic', callback=delivery_report, key='hello', value=str_value)

# Wait for any outstanding messages to be delivered and delivery report
# callbacks to be triggered.
p.flush()

from confluent_kafka import Consumer


c = Consumer({
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'mygroup',
    'auto.offset.reset': 'earliest'
})

c.subscribe(['mytopic'])

while True:
    msg = c.poll(1.0)

    if msg is None:
        continue
    if msg.error():
        print("Consumer error: {}".format(msg.error()))
        continue

    print('Received message: {}'.format(msg.value().decode('utf-8')))

c.close()

consumer.py

from confluent_kafka import Producer
import base64
import time

# some_data_source = ['hey', 'hi']

with open("1mb.png", "rb") as imageFile:
    str_value = base64.b64encode(imageFile.read())

p = Producer({'bootstrap.servers': 'localhost:9092', 'compression.type': 'snappy'})

def delivery_report(err, msg):
    """ Called once for each message produced to indicate delivery result.
        Triggered by poll() or flush(). """
    if err is not None:
        print('Message delivery failed: {}'.format(err))
    else:
        print('Message delivered to {} [{}]'.format(msg.topic(), msg.partition()))

for _ in range(2):
    # Trigger any available delivery report callbacks from previous produce() calls
    p.poll(0)

    # Asynchronously produce a message, the delivery report callback
    # will be triggered from poll() above, or flush() below, when the message has
    # been successfully delivered or failed permanently.
    p.produce('mytopic', callback=delivery_report, key='hello', value=str_value)

# Wait for any outstanding messages to be delivered and delivery report
# callbacks to be triggered.
p.flush()

from confluent_kafka import Consumer


c = Consumer({
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'mygroup',
    'auto.offset.reset': 'earliest'
})

c.subscribe(['mytopic'])

while True:
    msg = c.poll(1.0)

    if msg is None:
        continue
    if msg.error():
        print("Consumer error: {}".format(msg.error()))
        continue

    print('Received message: {}'.format(msg.value().decode('utf-8')))

c.close()

我是否需要添加任何参数，或者我是否在配置中丢失了某些内容？任何帮助都将不胜感激

---

谢谢

看来你并没有对经纪人的默认值做太多改变；它仍然在1MB左右

对于客户端错误，您需要将

message.max.bytes

添加到Producer配置中

如果您需要任何其他客户机属性，例如消费者最大获取字节数，请在此处进行说明

---

---

总体而言，建议您将图像上载到一个集中的文件存储，然后通过Kafka以普通字符串的形式发送它们的URI位置。这将提高吞吐量并减少代理的存储需求，特别是当您通过多个主题发送/复制相同的图像数据时

刚刚在producer config中添加了“message.max.bytes”：“10485780”，工作速度非常快！Thanks@OneCricketeerFeel使用帖子旁边的复选标记可以自由接受答案