Python “rdd.count（）”有效，但“rdd.first（）”失败，Jupyter笔记本中出现Py4JJava错误_Python_Apache Spark_Pyspark_Jupyter Notebook

Python “rdd.count（）”有效，但“rdd.first（）”失败，Jupyter笔记本中出现Py4JJava错误

python apache-spark pyspark jupyter-notebook

Python “rdd.count（）”有效，但“rdd.first（）”失败，Jupyter笔记本中出现Py4JJava错误,python,apache-spark,pyspark,jupyter-notebook,Python,Apache Spark,Pyspark,Jupyter Notebook,我对Spark完全陌生。我使用的是Spark 2.3版和Python 3.7版。顺便说一下，在Windows10上。我正在启动一个Jupyter笔记本来执行PypSpark操作。我正在学习一门关于Pluralsight的课程，学习Spark 2.0入门我正在Jupyter中使用Anaconda命令提示符中的以下命令启动pyspark：设置PYSPARK\u DRIVER\u PYTHON=jupyter 设置PYSPARK\u DRIVER\u PYTHON\u OPTS=notebook

我对Spark完全陌生。我使用的是Spark 2.3版和Python 3.7版。顺便说一下，在Windows10上。我正在启动一个Jupyter笔记本来执行PypSpark操作。我正在学习一门关于Pluralsight的课程，学习Spark 2.0入门

我正在Jupyter中使用Anaconda命令提示符中的以下命令启动pyspark：

设置PYSPARK\u DRIVER\u PYTHON=jupyter 设置PYSPARK\u DRIVER\u PYTHON\u OPTS=notebook 皮斯帕克

笔记本打开后：

我运行以下命令：

sc

from pyspark.sql.types import Row
from datetime import datetime

simple_data = sc.parallelize([1, "Alice", 50])
simple_data

simple_data.count()

simple_data.first()

现在，这里是它失败的地方：simple_data。首先出现以下错误：

Py4JJavaError                             Traceback (most recent call last)
<ipython-input-5-cc577dea1d9b> in <module>
----> 1 simple_data.first()

    C:\spark\python\pyspark\rdd.py in first(self)
   1374         ValueError: RDD is empty
   1375         """
-> 1376         rs = self.take(1)
   1377         if rs:
   1378             return rs[0]

C:\spark\python\pyspark\rdd.py in take(self, num)
   1356 
   1357             p = range(partsScanned, min(partsScanned + numPartsToTry, totalParts))
-> 1358             res = self.context.runJob(self, takeUpToNumLeft, p)
   1359 
   1360             items += res

C:\spark\python\pyspark\context.py in runJob(self, rdd, partitionFunc, partitions, allowLocal)
    999         # SparkContext#runJob.
   1000         mappedRDD = rdd.mapPartitions(partitionFunc)
-> 1001         port = self._jvm.PythonRDD.runJob(self._jsc.sc(), mappedRDD._jrdd, partitions)
   1002         return list(_load_from_socket(port, mappedRDD._jrdd_deserializer))
   1003 

C:\spark\python\lib\py4j-0.10.6-src.zip\py4j\java_gateway.py in __call__(self, *args)
   1158         answer = self.gateway_client.send_command(command)
   1159         return_value = get_return_value(
-> 1160             answer, self.gateway_client, self.target_id, self.name)
   1161 
   1162         for temp_arg in temp_args:

C:\spark\python\pyspark\sql\utils.py in deco(*a, **kw)
     61     def deco(*a, **kw):
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:
     65             s = e.java_exception.toString()

C:\spark\python\lib\py4j-0.10.6-src.zip\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)
    318                 raise Py4JJavaError(
    319                     "An error occurred while calling {0}{1}{2}.\n".
--> 320                     format(target_id, ".", name), value)
    321             else:
    322                 raise Py4JError(

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 4, localhost, executor driver): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "C:\spark\python\pyspark\rdd.py", line 1354, in takeUpToNumLeft
    yield next(iterator)
StopIteration

您可能需要执行以下操作：

simple_data=sc.parallelize[[1，Alice，50]].toDF 简单数据计数简单数据优先 simple_data.show

请注意parallelize内部的更改。

您是在本地运行Spark还是在集群上运行Spark？您对Spark的配置是什么？什么版本的Spark？你能试试simple_data.show吗？您是否看到正在UI上运行spark作业？我正在本地运行spark。版本是2.3。我不知道你说的Spark配置是什么意思。我运行了simple_data.show，它抛出了一个错误：AttributeError:'RDD'对象没有属性'show'，你能在UI上看到你的作业吗？数据帧通常优先于RDD。你能试试简单的数据收集吗？然后打印出来。因为你原来问题中的错误是指向一个空的RDD，这对我来说意味着spark不能做任何事情。您能在Spark UI上验证作业是否已完成吗？简单的数据收集工作。它打印了[1，'Alice'，50]和simple_data.count打印了3。但是simple_data.first失败。@Learner也许您应该尝试将Spark更新到最新版本或重新安装。在Jupyter上使用Spark 2.4.3，我无法重现您的问题。否则，你可能应该一直呆在那里，直到有更具洞察力的人出现。有一件事可能是，也可能不是根本原因，那就是sc的使用。我认为在Spark 2.X中，标准的连接设置是使用Spark=SparkSession.builder.configconf=conf.getOrCreate，然后使用Spark.sparkContext.Parallelize*。我看到您将其作为列表列表，并且已经将rdd转换为数据帧。但即使这样做了，我还是犯了同样的错误。错误是什么？在我的本地机器上，使用Spark 2.4.5和Python 3.7，这件事对我来说非常好。我不明白为什么这不起作用……它说：Py4JJavaError：调用z:org.apache.spark.api.python.PythonRDD.runJob时出错：org.apache.spark.SparkException:作业因阶段失败而中止：阶段0.0中的任务0失败1次，最近的失败：阶段0.0中的任务0.0丢失TID 0，本地主机，执行器驱动程序：org.apache.spark.api.python.python异常：回溯最近一次调用：文件C:\spark\python\pyspark\rdd.py，第1354行，在takeupNumleft yield nextiterator stopIteration中，堆栈中会有更多的消息跟踪错误消息。请在我提到你的用户名的粗体部分后面的问题中贴上完整的stacktraceAdded错误信息。

Py4JJavaError                             Traceback (most recent call last)
<ipython-input-3-4dbbd81a7c5c> in <module>
      2 #simple_data
      3 
----> 4 simple_data = sc.parallelize([[1, "Alice", 50]]).toDF()
      5 simple_data.count()
      6 simple_data.first()

C:\spark\python\pyspark\sql\session.py in toDF(self, schema, sampleRatio)
     56         [Row(name=u'Alice', age=1)]
     57         """
---> 58         return sparkSession.createDataFrame(self, schema, sampleRatio)
     59 
     60     RDD.toDF = toDF

C:\spark\python\pyspark\sql\session.py in createDataFrame(self, data, schema, samplingRatio, verifySchema)
    685 
    686         if isinstance(data, RDD):
--> 687             rdd, schema = self._createFromRDD(data.map(prepare), schema, samplingRatio)
    688         else:
    689             rdd, schema = self._createFromLocal(map(prepare, data), schema)

C:\spark\python\pyspark\sql\session.py in _createFromRDD(self, rdd, schema, samplingRatio)
    382         """
    383         if schema is None or isinstance(schema, (list, tuple)):
--> 384             struct = self._inferSchema(rdd, samplingRatio, names=schema)
    385             converter = _create_converter(struct)
    386             rdd = rdd.map(converter)

C:\spark\python\pyspark\sql\session.py in _inferSchema(self, rdd, samplingRatio, names)
    353         :return: :class:`pyspark.sql.types.StructType`
    354         """
--> 355         first = rdd.first()
    356         if not first:
    357             raise ValueError("The first row in RDD is empty, "

C:\spark\python\pyspark\rdd.py in first(self)
   1374         ValueError: RDD is empty
   1375         """
-> 1376         rs = self.take(1)
   1377         if rs:
   1378             return rs[0]

C:\spark\python\pyspark\rdd.py in take(self, num)
   1356 
   1357             p = range(partsScanned, min(partsScanned + numPartsToTry, totalParts))
-> 1358             res = self.context.runJob(self, takeUpToNumLeft, p)
   1359 
   1360             items += res

C:\spark\python\pyspark\context.py in runJob(self, rdd, partitionFunc, partitions, allowLocal)
    999         # SparkContext#runJob.
   1000         mappedRDD = rdd.mapPartitions(partitionFunc)
-> 1001         port = self._jvm.PythonRDD.runJob(self._jsc.sc(), mappedRDD._jrdd, partitions)
   1002         return list(_load_from_socket(port, mappedRDD._jrdd_deserializer))
   1003 

C:\spark\python\lib\py4j-0.10.6-src.zip\py4j\java_gateway.py in __call__(self, *args)
   1158         answer = self.gateway_client.send_command(command)
   1159         return_value = get_return_value(
-> 1160             answer, self.gateway_client, self.target_id, self.name)
   1161 
   1162         for temp_arg in temp_args:

C:\spark\python\pyspark\sql\utils.py in deco(*a, **kw)
     61     def deco(*a, **kw):
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:
     65             s = e.java_exception.toString()

C:\spark\python\lib\py4j-0.10.6-src.zip\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)
    318                 raise Py4JJavaError(
    319                     "An error occurred while calling {0}{1}{2}.\n".
--> 320                     format(target_id, ".", name), value)
    321             else:
    322                 raise Py4JError(

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "C:\spark\python\pyspark\rdd.py", line 1354, in takeUpToNumLeft
    yield next(iterator)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\spark\python\lib\pyspark.zip\pyspark\worker.py", line 229, in main
  File "C:\spark\python\lib\pyspark.zip\pyspark\worker.py", line 224, in process
  File "C:\spark\python\lib\pyspark.zip\pyspark\serializers.py", line 372, in dump_stream
    vs = list(itertools.islice(iterator, batch))
RuntimeError: generator raised StopIteration

    at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:298)
    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:438)
    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:421)
    at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:252)
    at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
    at scala.collection.Iterator$class.foreach(Iterator.scala:893)
    at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
    at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
    at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
    at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)
    at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
    at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
    at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
    at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
    at org.apache.spark.api.python.PythonRDD$$anonfun$1.apply(PythonRDD.scala:141)
    at org.apache.spark.api.python.PythonRDD$$anonfun$1.apply(PythonRDD.scala:141)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2067)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2067)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:109)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1599)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1587)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1586)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1586)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
    at scala.Option.foreach(Option.scala:257)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1820)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1769)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1758)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2027)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2048)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2067)
    at org.apache.spark.api.python.PythonRDD$.runJob(PythonRDD.scala:141)
    at org.apache.spark.api.python.PythonRDD.runJob(PythonRDD.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:214)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "C:\spark\python\pyspark\rdd.py", line 1354, in takeUpToNumLeft
    yield next(iterator)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\spark\python\lib\pyspark.zip\pyspark\worker.py", line 229, in main
  File "C:\spark\python\lib\pyspark.zip\pyspark\worker.py", line 224, in process
  File "C:\spark\python\lib\pyspark.zip\pyspark\serializers.py", line 372, in dump_stream
    vs = list(itertools.islice(iterator, batch))
RuntimeError: generator raised StopIteration

    at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:298)
    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:438)
    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:421)
    at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:252)
    at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
    at scala.collection.Iterator$class.foreach(Iterator.scala:893)
    at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
    at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
    at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
    at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)
    at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
    at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
    at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
    at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
    at org.apache.spark.api.python.PythonRDD$$anonfun$1.apply(PythonRDD.scala:141)
    at org.apache.spark.api.python.PythonRDD$$anonfun$1.apply(PythonRDD.scala:141)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2067)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2067)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:109)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    ... 1 more