Python 2.7 “原因”；java.lang.NoSuchMethodError:org.eclipse.paho.client.mqttv3.MqttConnectOptions.setAutomaticReconnect（Z）V"；

我试图通过修改提供的示例wordcount来使用ApacheBahir运行spark结构化流式MQTT

SPARK版本：SPARK-2.2.0-bin-hadoop2.7

我正在使用此命令运行程序：

bin\spark submit--packages org.apache.bahir:spark-sql-streaming-mqtt_2.11:2.2.0 mqtt.py

下面是我的代码：

# mqtt.py

from __future__ import print_function
import sys
from pyspark.sql import SparkSession
from pyspark.sql.functions import explode
from pyspark.sql.functions import split
if __name__ == "__main__":
    spark = SparkSession\
        .builder\
        .appName("StructuredNetworkWordCount")\
        .getOrCreate()
    broker_uri = 'xyz.com:1883'
    lines = (spark\
        .readStream\
        .format("org.apache.bahir.sql.streaming.mqtt.MQTTStreamSourceProvider")\
        .option("topic","xyz")\
        .load("tcp://{}".format(broker_uri)))\

    # Split the lines into words
    words = lines.select(
        # explode turns each item in an array into a separate row
        explode(
            split(lines.value, ' ')
        ).alias('word')
    )

    # Generate running word count
    wordCounts = words.groupBy('word').count()

    # Start running the query that prints the running counts to the console
    query = wordCounts\
        .writeStream\
        .outputMode('complete')\
        .format('console')\
        .start()

    query.awaitTermination()

但我在获取查询时遇到以下错误：

17/11/09 19:48:14 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
17/11/09 19:48:16 INFO StreamExecution: Starting [id = e0335f31-f3e0-4ee1-a774-52582268845c, runId = f6a87268-164c-4eab-82db-1ac0bacd2bad]. Use C:\Users\xyz\AppData\Local\Temp\temporary-42cbc22f-7c1d-413c-b81c-3d4496f8e297 to store the query checkpoint.
17/11/09 19:48:16 WARN MQTTStreamSourceProvider: If `clientId` is not set, a random value is picked up.
Recovering from failure is not supported in such a case.
17/11/09 19:48:16 ERROR StreamExecution: Query [id = e0335f31-f3e0-4ee1-a774-52582268845c, runId = f6a87268-164c-4eab-82db-1ac0bacd2bad] terminated with error
java.lang.NoSuchMethodError: org.eclipse.paho.client.mqttv3.MqttConnectOptions.setAutomaticReconnect(Z)V
        at org.apache.bahir.sql.streaming.mqtt.MQTTStreamSourceProvider.createSource(MQTTStreamSource.scala:219)
        at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:243)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2$$anonfun$applyOrElse$1.apply(StreamExecution.scala:158)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2$$anonfun$applyOrElse$1.apply(StreamExecution.scala:155)
        at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:194)
        at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2.applyOrElse(StreamExecution.scala:155)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2.applyOrElse(StreamExecution.scala:153)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:256)
        at org.apache.spark.sql.execution.streaming.StreamExecution.logicalPlan$lzycompute(StreamExecution.scala:153)
        at org.apache.spark.sql.execution.streaming.StreamExecution.logicalPlan(StreamExecution.scala:147)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches(StreamExecution.scala:276)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:206)
Exception in thread "stream execution thread for [id = e0335f31-f3e0-4ee1-a774-52582268845c, runId = f6a87268-164c-4eab-82db-1ac0bacd2bad]" java.lang.NoSuchMethodError: org.eclipse.paho.client.mqttv3.MqttConnectOptions.setAutomaticReconnect(Z)V
        at org.apache.bahir.sql.streaming.mqtt.MQTTStreamSourceProvider.createSource(MQTTStreamSource.scala:219)
        at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:243)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2$$anonfun$applyOrElse$1.apply(StreamExecution.scala:158)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2$$anonfun$applyOrElse$1.apply(StreamExecution.scala:155)Traceback (most recent call last):
  File "C:/Users/xyz/Documents/Fall-17/Transportation/spark/mqtt.py", line 84, in <module>

    query.awaitTermination()
  File "C:\spark\spark-2.2.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\sql\streaming.py", line 106, in awaitTermination
        at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:194)  File "C:\spark\spark-2.2.0-bin-hadoop2.7\python\lib\py4j-0.10.4-src.zip\py4j\java_gateway.py", line 1133, in __call__
  File "C:\spark\spark-2.2.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\sql\utils.py", line 75, in deco
pyspark.sql.utils.StreamingQueryException: u'org.eclipse.paho.client.mqttv3.MqttConnectOptions.setAutomaticReconnect(Z)V\n=== Streaming Query ===\nIdentifier: [id = e0335f31-f3e0-4ee1-a774-52582268845c, runId = f6a87268-164c-4eab-82db-1ac0bacd2bad]\nCurrent Committed Offsets: {}\nCurrent Available Offsets: {}\n\nCurrent State: INITIALIZING\nThread State: RUNNABLE'

        at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2.applyOrElse(StreamExecution.scala:155)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2.applyOrElse(StreamExecution.scala:153)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:272)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:256)
        at org.apache.spark.sql.execution.streaming.StreamExecution.logicalPlan$lzycompute(StreamExecution.scala:153)
        at org.apache.spark.sql.execution.streaming.StreamExecution.logicalPlan(StreamExecution.scala:147)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches(StreamExecution.scala:276)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:206)
17/11/09 19:48:16 INFO SparkContext: Invoking stop() from shutdown hook

17/11/09 19:48:14信息StateStoreCordinatorRef:Registered StateStoreCordinator端点
2009年11月17日19:48:16信息流执行：启动[id=e0335f31-f3e0-4ee1-a774-52582268845c，runId=f6a87268-164c-4eab-82db-1ac0bacd2bad]。使用C:\Users\xyz\AppData\Local\Temp\temporary-42cbc22f-7c1d-413c-b81c-3d4496f8e297存储查询检查点。
17/11/09 19:48:16警告MQTTStreamSourceProvider:如果未设置'clientId'，将拾取一个随机值。
在这种情况下，不支持从故障中恢复。
2009年11月17日19:48:16错误流执行：查询[id=e0335f31-f3e0-4ee1-a774-52582268845c，runId=f6a87268-164c-4eab-82db-1ac0bacd2bad]因错误而终止
java.lang.NoSuchMethodError:org.eclipse.paho.client.mqttv3.MqttConnectOptions.setAutomaticReconnect（Z）V
位于org.apache.bahir.sql.streaming.mqtt.MQTTStreamSourceProvider.createSource（MQTTStreamSource.scala:219）
位于org.apache.spark.sql.execution.datasources.DataSource.createSource（DataSource.scala:243）
位于org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2$$anonfun$applyOrElse$1.apply（StreamExecution.scala:158）
位于org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2$$anonfun$applyOrElse$1.apply（StreamExecution.scala:155）
位于scala.collection.mutable.MapLike$class.getOrElseUpdate（MapLike.scala:194）
位于scala.collection.mutable.AbstractMap.GetOrelsUpdate（Map.scala:80）
位于org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2.applyOrElse（StreamExecution.scala:155）
位于org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2.applyOrElse（StreamExecution.scala:153）
位于org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply（TreeNode.scala:267）
位于org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply（TreeNode.scala:267）
位于org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin（TreeNode.scala:70）
位于org.apache.spark.sql.catalyst.trees.TreeNode.transformDown（TreeNode.scala:266）
位于org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply（TreeNode.scala:272）
位于org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply（TreeNode.scala:272）
位于org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply（TreeNode.scala:306）
位于org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator（TreeNode.scala:187）
位于org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren（TreeNode.scala:304）
位于org.apache.spark.sql.catalyst.trees.TreeNode.transformDown（TreeNode.scala:272）
位于org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply（TreeNode.scala:272）
位于org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply（TreeNode.scala:272）
位于org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply（TreeNode.scala:306）
位于org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator（TreeNode.scala:187）
位于org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren（TreeNode.scala:304）
位于org.apache.spark.sql.catalyst.trees.TreeNode.transformDown（TreeNode.scala:272）
位于org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply（TreeNode.scala:272）
位于org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply（TreeNode.scala:272）
位于org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply（TreeNode.scala:306）
位于org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator（TreeNode.scala:187）
位于org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren（TreeNode.scala:304）
位于org.apache.spark.sql.catalyst.trees.TreeNode.transformDown（TreeNode.scala:272）
位于org.apache.spark.sql.catalyst.trees.TreeNode.transform（TreeNode.scala:256）
位于org.apache.spark.sql.execution.streaming.StreamExecution.logicalPlan$lzycompute（StreamExecution.scala:153）
位于org.apache.spark.sql.execution.streaming.StreamExecution.logicalPlan（StreamExecution.scala:147）
位于org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$streaming$StreamExecution$$runBatches（StreamExecution.scala:276）
位于org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run（StreamExecution.scala:206）
线程“针对[id=e0335f31-f3e0-4ee1-a774-52582268845c，runId=f6a87268-164c-4eab-82db-1ac0bacd2bad]的流执行线程”java.lang.NoSuchMethodError:org.eclipse.paho.client.mqttv3.MqttConnectOptions.setAutomaticReconnect（Z）V中出现异常
位于org.apache.bahir.sql.streaming.mqtt.MQTTStreamSourceProvider.createSource（MQTTStreamSource.scala:219）
位于org.apache.spark.sql.execution.datasources.DataSource.createSource（DataSource.scala:243）
位于org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2$$anonfun$applyOrElse$1.apply（StreamExecution.scala:158）
在org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2$$anonfun$applyOrElse$1.apply（StreamExecution.scala:155）回溯（最后一次调用）：
文件“C:/Users/xyz/Documents/Fall-17/Transportation/spark/mqtt.py”，第84行，在
查询
文件“C:\spark\spark-2.2.0-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\sql\streaming.py”，第106行，第1行
在scala.collection.mutable.MapLike$class.getOrElseUpdate（MapLike.scala:194）文件“C:\spark\spark-2.2.0-bin-