Apache kafka 无法在Kafka 0.10.2(命令行)中使用SSL配置授权-生产者和消费者无法写入或读取主题
我无法在Kafka 0.10.2中使用SSL配置授权。我正在为代理、生产者和消费者使用命令行客户端。当kafka服务器配置文件中注释掉了Apache kafka 无法在Kafka 0.10.2(命令行)中使用SSL配置授权-生产者和消费者无法写入或读取主题,apache-kafka,Apache Kafka,我无法在Kafka 0.10.2中使用SSL配置授权。我正在为代理、生产者和消费者使用命令行客户端。当kafka服务器配置文件中注释掉了allow.everybody.if.no.acl.found=true时,生产者和消费者无法写入或读取测试主题(否则,他们可以读取和写入) 我已经搜索了、、和-,但仍然无法获得授权(尽管我通过TLS进行了身份验证) 我的证书来自IdenTrust/Letsencrypt。如果我取消注释allow.everybody.If.no.acl.found=true,我
allow.everybody.if.no.acl.found=true
时,生产者和消费者无法写入或读取测试主题(否则,他们可以读取和写入)
我已经搜索了、、和-,但仍然无法获得授权(尽管我通过TLS进行了身份验证)
我的证书来自IdenTrust/Letsencrypt。如果我取消注释allow.everybody.If.no.acl.found=true
,我会在生产者连接时的代理日志中看到:
DEBUG SslTransportLayer:358 - SSL handshake completed successfully with
peerHost 'devel-2.sjml.com' peerPort 56099 peerPrincipal 'CN=testkafkaconsumer1.eigenroute.com' cipherSuite
'TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384'
制作人可以写测试主题,消费者可以阅读测试主题。但是,当注释掉上述行时,该输出不会出现在日志中。在这种情况下,producer命令行客户端将输出以下内容:
WARN Error while fetching metadata with correlation id 10588 :
{test100=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
下面是主题test100的Zookeeper的ACL,以及列出它的命令:
$ bin/kafka-acls.sh --list --authorizer-properties zookeeper.connect=localhost:2181 --topic test100
Current ACLs for resource `Topic:test100`:
User:CN=testkafkaconsumer1.eigenroute.com has Allow permission for operations: Read from hosts: *
User:CN=kafka.eigenroute.com has Allow permission for operations: All from hosts: *
User:CN=testkafkaproducer1.eigenroute.com has Allow permission for operations: Write from hosts: *
以下是我用于将用户添加到ACL的命令:
./bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:CN=kafka.eigenroute.com --operation All --topic test100
./bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:CN=testkafkaproducer1.eigenroute.com --operation Write --topic test100
./bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:CN=testkafkaconsumer1.eigenroute.com --operation Read --topic test100
OS:Debian 8 Jessie
以下是代理、使用者和生产者的配置文件:
代理配置:
# secure-server-letsencrypt.properties
broker.id=0
delete.topic.enable=true
listeners=SSL://kafka.eigenroute.com:9093
port=9093
advertised.host.name=kafka.eigenroute.com
ssl.keystore.location=/home/kafka/keystore/kafka.keystore.jks
ssl.keystore.password=some-password
ssl.key.password=some-password
ssl.truststore.location=/usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts
ssl.truststore.password=some-password
ssl.endpoint.identification.algorithm=HTTPS
ssl.client.auth=required
security.inter.broker.protocol=SSL
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:CN=testkafkaproducer1.eigenroute.com
# allow.everyone.if.no.acl.found=true
advertised.listeners=SSL://kafka.eigenroute.com:9093
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
# secure-consumer.properties
zookeeper.connect=127.0.0.1:2181
# timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
#consumer group id
group.id=test-consumer-group
#consumer timeout
#consumer.timeout.ms=5000
security.protocol=SSL
ssl.truststore.location=/usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts
ssl.truststore.password=some-password
ssl.keystore.location=/home/kafka/keystore/testkafkaconsumer1.keystore.jks
ssl.keystore.password=some-password
ssl.key.password=some-password
bootstrap.servers=kafka.eigenroute.com:9093
security.protocol=SSL
ssl.truststore.location=/usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts
ssl.truststore.password=some-password
ssl.keystore.location=/home/kafka/keystore/testkafkaproducer1.keystore.jks
ssl.keystore.password=some-password
ssl.key.password=some-password
compression.type=none
消费者配置:
# secure-server-letsencrypt.properties
broker.id=0
delete.topic.enable=true
listeners=SSL://kafka.eigenroute.com:9093
port=9093
advertised.host.name=kafka.eigenroute.com
ssl.keystore.location=/home/kafka/keystore/kafka.keystore.jks
ssl.keystore.password=some-password
ssl.key.password=some-password
ssl.truststore.location=/usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts
ssl.truststore.password=some-password
ssl.endpoint.identification.algorithm=HTTPS
ssl.client.auth=required
security.inter.broker.protocol=SSL
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:CN=testkafkaproducer1.eigenroute.com
# allow.everyone.if.no.acl.found=true
advertised.listeners=SSL://kafka.eigenroute.com:9093
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
# secure-consumer.properties
zookeeper.connect=127.0.0.1:2181
# timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
#consumer group id
group.id=test-consumer-group
#consumer timeout
#consumer.timeout.ms=5000
security.protocol=SSL
ssl.truststore.location=/usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts
ssl.truststore.password=some-password
ssl.keystore.location=/home/kafka/keystore/testkafkaconsumer1.keystore.jks
ssl.keystore.password=some-password
ssl.key.password=some-password
bootstrap.servers=kafka.eigenroute.com:9093
security.protocol=SSL
ssl.truststore.location=/usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts
ssl.truststore.password=some-password
ssl.keystore.location=/home/kafka/keystore/testkafkaproducer1.keystore.jks
ssl.keystore.password=some-password
ssl.key.password=some-password
compression.type=none
生产者配置:
# secure-server-letsencrypt.properties
broker.id=0
delete.topic.enable=true
listeners=SSL://kafka.eigenroute.com:9093
port=9093
advertised.host.name=kafka.eigenroute.com
ssl.keystore.location=/home/kafka/keystore/kafka.keystore.jks
ssl.keystore.password=some-password
ssl.key.password=some-password
ssl.truststore.location=/usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts
ssl.truststore.password=some-password
ssl.endpoint.identification.algorithm=HTTPS
ssl.client.auth=required
security.inter.broker.protocol=SSL
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:CN=testkafkaproducer1.eigenroute.com
# allow.everyone.if.no.acl.found=true
advertised.listeners=SSL://kafka.eigenroute.com:9093
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
# secure-consumer.properties
zookeeper.connect=127.0.0.1:2181
# timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
#consumer group id
group.id=test-consumer-group
#consumer timeout
#consumer.timeout.ms=5000
security.protocol=SSL
ssl.truststore.location=/usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts
ssl.truststore.password=some-password
ssl.keystore.location=/home/kafka/keystore/testkafkaconsumer1.keystore.jks
ssl.keystore.password=some-password
ssl.key.password=some-password
bootstrap.servers=kafka.eigenroute.com:9093
security.protocol=SSL
ssl.truststore.location=/usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts
ssl.truststore.password=some-password
ssl.keystore.location=/home/kafka/keystore/testkafkaproducer1.keystore.jks
ssl.keystore.password=some-password
ssl.key.password=some-password
compression.type=none
我认为将producer用户设置为超级用户,就像我在broker/server配置中所做的那样,应该允许producer写入主题;唉,事实并非如此。代理似乎无法从Zookeeper中找到ACL。有人能建议如何解决这个问题吗?谢谢 如中所述,对于--消费者
和--生产者
选项,您需要允许对消费者的主题进行描述和阅读,以及对消费者组进行阅读
对于制作人,您需要描述并撰写制作人的主题,同时在集群上创建。您找到解决方案了吗?仍然没有:-(.我甚至尝试过Kerberos路由,但也无法实现。刚刚添加了一个赏金-希望这有助于获得一些建议。谢谢。这很有效-谢谢!我执行了以下行:
/bin/kafka-acls.sh--authorizer properties zookeer.connect=localhost:2181--add--allow主体用户:CN=testkafka-sumer1.eigenroute.com--operation description--topic test100
和/bin/kafka-acls.sh--authorizer properties zookeer.connect=localhost:2181--add--allow principal User:CN=testkafkaproducer1.eigenroute.com--operation description--topic test100
按照您的建议添加description权限,现在不再出现错误。