Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Apache spark SODAAPI的错误消息中缺少什么?_Apache Spark_Pyspark_Soda - Fatal编程技术网

Apache spark SODAAPI的错误消息中缺少什么?

Apache spark SODAAPI的错误消息中缺少什么?,apache-spark,pyspark,soda,Apache Spark,Pyspark,Soda,stackoverflow的人们好 我有一个问题还没有回答?我想这不是我要找的答案。我一直在构建这个API以获取一些数据,但在调用时它似乎失败了。我有所需的Jar文件,并且它正在集群中读取 File "spark_app.py", line 205, in <module> sodasDf = spark.read.format('org.apache.dsext.spark.datasource.rest.RestDataSource').options

stackoverflow的人们好

我有一个问题还没有回答?我想这不是我要找的答案。我一直在构建这个API以获取一些数据,但在调用时它似乎失败了。我有所需的Jar文件,并且它正在集群中读取

File "spark_app.py", line 205, in <module>
    sodasDf = spark.read.format('org.apache.dsext.spark.datasource.rest.RestDataSource').options(**prms_Soda).load()
  File "/app/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 172, in load
  File "/app/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/app/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
  File "/app/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o109.load.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 9.0 failed 4 times, most recent failure: Lost task 0.3 in stage 9.0 (TID 12, cvglpcdh13.td.afg, executor 4): java.lang.NullPointerException
        at org.apache.dsext.spark.datasource.rest.RESTRelation$$anonfun$3.apply(RestRelation.scala:108)
        at org.apache.dsext.spark.datasource.rest.RESTRelation$$anonfun$3.apply(RestRelation.scala:108)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
        at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
        at org.apache.dsext.spark.datasource.rest.RESTRelation.org$apache$dsext$spark$datasource$rest$RESTRelation$$callRest(RestRelation.scala:108)
        at org.apache.dsext.spark.datasource.rest.RESTRelation$$anonfun$2.apply(RestRelation.scala:100)
        at org.apache.dsext.spark.datasource.rest.RESTRelation$$anonfun$2.apply(RestRelation.scala:100)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$11$$anon$1.hasNext(WholeStageCodegenExec.scala:624)
        at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
        at scala.collection.Iterator$class.isEmpty(Iterator.scala:331)
        at scala.collection.AbstractIterator.isEmpty(Iterator.scala:1334)
        at scala.collection.TraversableOnce$class.reduceLeftOption(TraversableOnce.scala:203)
        at scala.collection.AbstractIterator.reduceLeftOption(Iterator.scala:1334)
        at scala.collection.TraversableOnce$class.reduceOption(TraversableOnce.scala:210)
        at scala.collection.AbstractIterator.reduceOption(Iterator.scala:1334)
文件“spark_app.py”,第205行,在
sodasDf=spark.read.format('org.apache.dsext.spark.datasource.rest.RestDataSource')。选项(**prms_Soda).load()
文件“/app/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py”,第172行,已加载
文件“/app/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py”,调用中的第1257行__
文件“/app/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py”,第63行,deco格式
文件“/app/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py”,第328行,在get\u返回值中
py4j.protocol.Py4JJavaError:调用o109.load时出错。
:org.apache.spark.SparkException:作业因阶段失败而中止:阶段9.0中的任务0失败4次,最近的失败:阶段9.0中的任务0.3丢失(TID 12,cvglpcdh13.td.afg,执行器4):java.lang.NullPointerException
在org.apache.dsext.spark.datasource.rest.RESTRelation$$anonfun$3.apply上(RESTRelation.scala:108)
在org.apache.dsext.spark.datasource.rest.RESTRelation$$anonfun$3.apply上(RESTRelation.scala:108)
在scala.collection.TraversableLike$$anonfun$map$1.apply处(TraversableLike.scala:234)
在scala.collection.TraversableLike$$anonfun$map$1.apply处(TraversableLike.scala:234)
在scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
位于scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
位于scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
位于scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
在org.apache.dsext.spark.datasource.rest.rest.RESTRelation.org$apache$dsext$spark$datasource$rest$RESTRelation$$callRest(RESTRelation.scala:108)
在org.apache.dsext.spark.datasource.rest.RESTRelation$$anonfun$2.apply上(RESTRelation.scala:100)
在org.apache.dsext.spark.datasource.rest.RESTRelation$$anonfun$2.apply上(RESTRelation.scala:100)
位于scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
位于scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
位于org.apache.spark.sql.catalyst.expressions.GeneratedClass$GenerateEditorForCodeGenStage1.processNext(未知源)
位于org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
位于org.apache.spark.sql.execution.whisttagecodegenexec$$anonfun$11$$anon$1.hasNext(whisttagecodegenexec.scala:624)
位于scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
位于scala.collection.Iterator$class.isEmpty(Iterator.scala:331)
位于scala.collection.AbstractIterator.isEmpty(Iterator.scala:1334)
位于scala.collection.TraversableOnce$class.reduceLeftOption(TraversableOnce.scala:203)
位于scala.collection.AbstractIterator.ReduceEleftOption(迭代器.scala:1334)
在scala.collection.TraversableOnce$class.reduceOption(TraversableOnce.scala:210)
位于scala.collection.AbstractIterator.reduceOption(迭代器.scala:1334)

我不确定我错过了什么。看起来之前有人问了同样的问题,所以(),我不确定他们的和我的是相同的还是我遇到了不同的错误,但是如果有人能帮助我,我将不胜感激?我仔细检查了我的工作,我拥有github回购协议中的所有内容,但它仍然不起作用。