Plugins librdkafka注册OAuthBearrer_token_refresh_cb的自定义插件未被调用_Plugins_Apache Kafka_Oauth_Sasl_Librdkafka

Plugins librdkafka注册OAuthBearrer_token_refresh_cb的自定义插件未被调用

plugins apache-kafka oauth

Plugins librdkafka注册OAuthBearrer_token_refresh_cb的自定义插件未被调用,plugins,apache-kafka,oauth,sasl,librdkafka,Plugins,Apache Kafka,Oauth,Sasl,Librdkafka,我正在尝试为librdkafka编写自定义插件，它将提供OAuthBearrer\u token\u refresh\u cb的自定义实现在我们公司，我们使用kafka和定制的OAUTHBEARER sasl机制，在java/kotlin中运行良好，在.Net中也运行良好。下一步是为librdkafka实现相同的功能，因为我们需要它来实现Clickhouse DB使用librdkafka的卡夫卡连接器我尝试了以下插件驱动：自定义oauth.c 创建共享库： gcc -shared oaut

我正在尝试为

librdkafka

编写自定义插件，它将提供

OAuthBearrer\u token\u refresh\u cb

的自定义实现

在我们公司，我们使用kafka和定制的OAUTHBEARER sasl机制，在java/kotlin中运行良好，在.Net中也运行良好。下一步是为

librdkafka

实现相同的功能，因为我们需要它来实现Clickhouse DB使用

librdkafka

的卡夫卡连接器

我尝试了以下插件驱动：

自定义oauth.c 创建共享库：

gcc -shared oauth-cb.o -lrdkafka -o lib-oauth-cb.so

通过以下方式使用

kafkacat

kafkacat -L \
    -X debug="all" \
    -X log_level=0 \
    -b my-broker-1:9094 \
    -X security.protocol=SASL_SSL \
    -X ssl.ca.location=/path-to-my-cer.pem \
    -X sasl.mechanisms=OAUTHBEARER \
    -X plugin.library.paths=lib-oauth-cb.so \
    -X sasl.oauthbearer.config="my-custom-oauth-config"

结果：

============== registering callback
============== callback registered
% ERROR: No such configuration property: "plugin.library.paths" (plugin lib-oauth-cb.so)

如果我注释掉在

rd\u kafka\u conf\u set\u oauthbearer\u token\u refresh\u cb

之后发生的所有printf-s，那么我不会得到

%错误：没有这样的配置属性：“plugin.library.path”（plugin lib oauth cb.so）

但这样也不起作用，我设置oauth承载令牌的回调从未被调用。这是输出：

============== registering callback
%7|1605365045.759|SASL|rdkafka#producer-1| [thrd:app]: Selected provider OAUTHBEARER (builtin) for SASL mechanism OAUTHBEARER
%7|1605365045.759|OPENSSL|rdkafka#producer-1| [thrd:app]: Using OpenSSL version OpenSSL 1.1.1g  21 Apr 2020 (0x1010107f, librdkafka built with 0x1010107f)
%7|1605365045.762|SSL|rdkafka#producer-1| [thrd:app]: Loading CA certificate(s) from file /path-to-my-cer.pem
%7|1605365045.763|WAKEUPFD|rdkafka#producer-1| [thrd:app]: sasl_ssl://my-broker-1:9094/bootstrap: Enabled low-latency ops queue wake-ups
%7|1605365045.763|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1605365045.763|BROKER|rdkafka#producer-1| [thrd:app]: sasl_ssl://my-broker-1:9094/bootstrap: Added new broker with NodeId -1
%7|1605365045.763|BRKMAIN|rdkafka#producer-1| [thrd:sasl_ssl://my-broker-1:9094/bootstrap]: sasl_ssl://my-broker-1:9094/bootstrap: Enter main broker thread
%7|1605365045.763|CONNECT|rdkafka#producer-1| [thrd:app]: sasl_ssl://my-broker-1:9094/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1605365045.763|CONNECT|rdkafka#producer-1| [thrd:sasl_ssl://my-broker-1:9094/bootstrap]: sasl_ssl://my-broker-1:9094/bootstrap: Received CONNECT op
%7|1605365045.763|STATE|rdkafka#producer-1| [thrd:sasl_ssl://my-broker-1:9094/bootstrap]: sasl_ssl://my-broker-1:9094/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1605365045.763|BROADCAST|rdkafka#producer-1| [thrd:sasl_ssl://my-broker-1:9094/bootstrap]: Broadcasting state change
%7|1605365045.763|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.5.0 (0x10500ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, CC CXX PKGCONFIG OSXLD LIBDL PLUGINS ZLIB SSL SASL_CYRUS ZSTD HDRHISTOGRAM LZ4_EXT SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0xfffff)
%7|1605365045.763|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1605365046.763|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
%7|1605365047.763|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
%7|1605365048.763|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
%7|1605365049.764|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
%7|1605365050.766|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
%7|1605365050.766|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
% ERROR: Failed to acquire metadata: Local: Broker transport failure

笔记：

此kafka群集有3个不同的侦听器
- 端口9092纯文本协议（试用过kafkacat，它可以正常工作）
- 端口9093 SSL协议（试用过kafkacat，效果良好）
- 端口9094 SASL_SSL协议，OAuthBearner作为SASL机制
我已经使用
```
brew
```
在我的Mac上安装了
```
librdkafka
```
和
```
kafkat
```
，正如我所见，librdkafka的版本是
```
1.5.0
```

问题:

为什么从未调用过我的回调

oauth\u refresh

（no

================初始化oauthbearer配置

…打印在

stdout

）

为什么我会得到一个错误

没有这样的配置属性：“plugin.library.path”（plugin lib oauth cb.so）

只要在
```
printf
```

即使多次设置也不会导致此错误：

```
printf
```

rd\u kafka\u conf\u set\u oauthbearer\u token\u refresh\u cb

```
printf
```

rd\u kafka\u conf\u set\u oauthbearer\u token\u refresh\u cb

导致此错误的
```
printf
```
的副作用是什么
问题的这一部分不那么重要，因为我不打算保留那些printf-s，part
```
1）
```
很重要

目标

我的目标是拥有OAuthBearrer方案的小型可插拔库，它可以插入任何使用
```
librdkafka
```
（最终目标是的kafka连接器）
我使用了卡夫卡塔，只是因为它是一个很容易实验的工具
关键的一点是可插入性，这样就不需要重新编译某些使用librdkafka的应用程序，只需使用自定义的刷新oauth令牌逻辑即可

编辑我遇到了几个问题：

conf_init的签名：

需要是：

rd_kafka_resp_err_t conf_init（rd_kafka_conf_t*conf，const char*path，char*errstr，size\u errstr\u size）

而不是：

void conf\u init（rd\u kafka\u conf\u t*conf，const char*path，char*errstr，size\u t errstr\u size）

在我的示例中，设置OAuthBear令牌时，到期日需要是
```
nowMs+someTTL
```
，而不是
```
0
```

我最终使用了而不是安装的

kafkacat

，因为我用

brew

安装的最新版本kafka cat是

1.5.0

在对这个较旧的

kafkacat

运行时，我遇到以下错误

kafkacat（76875,0x11b279dc0）malloc:**对象0x10bad7bd0的错误：被释放的指针未分配kafkacat（76875,0x11b279dc0）malloc:**在malloc_error_break中设置一个断点以进行调试


我让它在消费模式-C
下工作，但在打印元数据模式-L
下不工作，因为-L
模式不调用rd_kafka_poll，所以我的回调从未被调用。

我最后加入了《卡夫卡民调》（rk，5000）-L开关时，代码>进入

该plugin.library.paths
错误表示您的librdkafka库是在不支持插件的情况下构建的，即缺少WITH_DLOPEN
。这应该在Linux和OSX上用标准的/configure
自动检测到。这就是您正在使用的吗？不，实际上我自己根本没有编译librdkafka，我只是用brew预编译安装了它（其中包括删除brew的功能）将librdkafka/rdkafka.h
放在我的/usr/local/include
中，我只是使用gcc
编译了有问题的代码片段，没有任何额外的工具，比如make
或cMake
，现在我在想。。。如果我需要在启用插件支持的情况下编译librdkafka，以便能够附加一些自定义插件，那么只有在使用librdkafka的应用程序实际使用启用插件的版本时，这种自定义插件的附加才能起作用。我担心，如果它是使用librdkafka预编译并禁用插件支持的，那么它将不适用于任何预编译的应用程序（如kafkacat或ClickhouseDB）。我需要自己构建kafkacat（和Clickhouse Db）吗？librdkafka将在几乎所有现代平台上启用插件支持，所以我不知道为什么您的构建没有启用。什么版本的卡夫卡？另外，请尝试启动应用程序并将librdkafka属性debug
设置为generic
…，它应该会发出类似的日志行。。。librdkafka v1.6.0-PRE3-32-g134984-dirty-devel-O0（0x1060000）rdkafka#producer-1已初始化（内置功能gzip、snappy…、plugins…、GCC…LIBDL plugins…
。检查插件和============== registering callback
============== callback registered
% ERROR: No such configuration property: "plugin.library.paths" (plugin lib-oauth-cb.so)

============== registering callback
%7|1605365045.759|SASL|rdkafka#producer-1| [thrd:app]: Selected provider OAUTHBEARER (builtin) for SASL mechanism OAUTHBEARER
%7|1605365045.759|OPENSSL|rdkafka#producer-1| [thrd:app]: Using OpenSSL version OpenSSL 1.1.1g  21 Apr 2020 (0x1010107f, librdkafka built with 0x1010107f)
%7|1605365045.762|SSL|rdkafka#producer-1| [thrd:app]: Loading CA certificate(s) from file /path-to-my-cer.pem
%7|1605365045.763|WAKEUPFD|rdkafka#producer-1| [thrd:app]: sasl_ssl://my-broker-1:9094/bootstrap: Enabled low-latency ops queue wake-ups
%7|1605365045.763|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1605365045.763|BROKER|rdkafka#producer-1| [thrd:app]: sasl_ssl://my-broker-1:9094/bootstrap: Added new broker with NodeId -1
%7|1605365045.763|BRKMAIN|rdkafka#producer-1| [thrd:sasl_ssl://my-broker-1:9094/bootstrap]: sasl_ssl://my-broker-1:9094/bootstrap: Enter main broker thread
%7|1605365045.763|CONNECT|rdkafka#producer-1| [thrd:app]: sasl_ssl://my-broker-1:9094/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1605365045.763|CONNECT|rdkafka#producer-1| [thrd:sasl_ssl://my-broker-1:9094/bootstrap]: sasl_ssl://my-broker-1:9094/bootstrap: Received CONNECT op
%7|1605365045.763|STATE|rdkafka#producer-1| [thrd:sasl_ssl://my-broker-1:9094/bootstrap]: sasl_ssl://my-broker-1:9094/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1605365045.763|BROADCAST|rdkafka#producer-1| [thrd:sasl_ssl://my-broker-1:9094/bootstrap]: Broadcasting state change
%7|1605365045.763|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.5.0 (0x10500ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, CC CXX PKGCONFIG OSXLD LIBDL PLUGINS ZLIB SSL SASL_CYRUS ZSTD HDRHISTOGRAM LZ4_EXT SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0xfffff)
%7|1605365045.763|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1605365046.763|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
%7|1605365047.763|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
%7|1605365048.763|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
%7|1605365049.764|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
%7|1605365050.766|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection
%7|1605365050.766|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
% ERROR: Failed to acquire metadata: Local: Broker transport failure