Apache kafka 如何转换mqtt';将有效负载数据转换为kafka字符串类型
现在我正在尝试从mqtt消息连接到kafka(实际上是kafka中的spark流) 我用了这个连接器 和Spark-2.1.0,卡夫卡-0.10.1.1 像这样的火花流输出Apache kafka 如何转换mqtt';将有效负载数据转换为kafka字符串类型,apache-kafka,mqtt,apache-kafka-connect,Apache Kafka,Mqtt,Apache Kafka Connect,现在我正在尝试从mqtt消息连接到kafka(实际上是kafka中的spark流) 我用了这个连接器 和Spark-2.1.0,卡夫卡-0.10.1.1 像这样的火花流输出 ({"schema":{"type":"string","optional":false},"payload":"mqtt"},{"schema":{"type":"bytes","optional":false},"payload":"MTIzMTIz"}) 和生产者代码 object mqttProducer { de
({"schema":{"type":"string","optional":false},"payload":"mqtt"},{"schema":{"type":"bytes","optional":false},"payload":"MTIzMTIz"})
和生产者代码
object mqttProducer {
def main(args: Array[String]) {
val brokerUrl = "tcp://ip"
val topic = "mqtt"
val msg = "123123"
var client: MqttClient = null
// Creating new persistence for mqtt client
val persistence = new MqttDefaultFilePersistence("/tmp")
try {
// mqtt client with specific url and client id
client = new MqttClient(brokerUrl, MqttClient.generateClientId, persistence)
client.connect()
val msgTopic = client.getTopic(topic)
val message = new MqttMessage(msg.getBytes("utf-8"))
while (true) {
msgTopic.publish(message)
println("Publishing Data, Topic : %s, Message : %s".format(msgTopic.getName, message))
Thread.sleep(1000)
}
}
catch {
case e: MqttException => println("Exception Caught: " + e)
}
finally {
client.disconnect()
}
和spark流媒体卡夫卡消费代码
package hb.test1
import org.apache.kafka.common.serialization.StringDeserializer
import org.apache.spark.SparkConf
import org.apache.spark.streaming.Seconds
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.kafka010.ConsumerStrategies.Subscribe
import org.apache.spark.streaming.kafka010.KafkaUtils
import org.apache.spark.streaming.kafka010.LocationStrategies.PreferConsistent
object test2 {
def main(args: Array[String]): Unit = {
val sparkConf = new SparkConf().setAppName("app")
val ssc = new StreamingContext(sparkConf, Seconds(1))
val kafkaParams = Map[String, Object](
"bootstrap.servers" ->
"servers ip",
"key.deserializer" -> classOf[StringDeserializer],
"value.deserializer" -> classOf[StringDeserializer],
"group.id" -> "use_a_separate_group_id_for_each_stream",
"auto.offset.reset" -> "latest",
"enable.auto.commit" -> (false: java.lang.Boolean)
)
val topics = Array("mqtt-kafka")
val stream = KafkaUtils.createDirectStream[String, String](
ssc,
PreferConsistent,
Subscribe[String, String](topics, kafkaParams)
)
val testStream = stream.map(x => (x.key, x.value))
testStream.print()
ssc.start()
ssc.awaitTermination()
}
}
我怎样才能得到字符串而不是字节?
请帮助大家,有效负载“MTIzMTIz”是刚刚用base64编码的字符串“123123”。如果您只想获取MQTT负载并将其发送到Kafka,而不使用base64编码,那么应该使用ByteArrayConverter。在同一mqtt连接器的配置中,我将值转换器设置为:
“value.converter”:“io.confluent.connect.replicator.util.ByteArrayConverter”
上面的ByteArrayConverter与Confluent Enterprise发行版一起提供,但也有其他开源Kafka Connect ByteArrayConverter,如qubole/streamx Kafka-Connect-s3连接器附带的
KIP-128将标准ByteArrayConverter添加到Kafka Connect框架中
更新:卡夫卡0.11现已发布,并附带ByteArrayConverter。配置“value.converter”:“org.apache.kafka.connect.converters.ByteArrayConverter”
,您应该获得未经Base64编码的原始mqtt有效负载。该有效负载“MTIzMTIz”是仅经Base64编码的字符串“123123”。如果您只想获取MQTT负载并将其发送到Kafka,而不使用base64编码,那么应该使用ByteArrayConverter。在同一mqtt连接器的配置中,我将值转换器设置为:
“value.converter”:“io.confluent.connect.replicator.util.ByteArrayConverter”
上面的ByteArrayConverter与Confluent Enterprise发行版一起提供,但也有其他开源Kafka Connect ByteArrayConverter,如qubole/streamx Kafka-Connect-s3连接器附带的
KIP-128将标准ByteArrayConverter添加到Kafka Connect框架中
更新:卡夫卡0.11现已发布,并附带ByteArrayConverter。配置“value.converter”:“org.apache.kafka.connect.converters.ByteArrayConverter”
,您应该获得原始mqtt负载,而不使用Base64编码