Pyspark 为什么不能在spark sql中缓存后调用show方法？
我用HiveContext（而不是SQLContext）在pyspark中创建了一个名为df的数据帧 但是我发现在调用df.cache（）之后，我将无法调用df.show（）。例如：Pyspark 为什么不能在spark sql中缓存后调用show方法？,pyspark,apache-spark-sql,Pyspark,Apache Spark Sql,我用HiveContext（而不是SQLContext）在pyspark中创建了一个名为df的数据帧 但是我发现在调用df.cache（）之后，我将无法调用df.show（）。例如： >>> df.show(2) +--------+-------------+--------+--------------+--------+-------+---------+--------+--------+-------------+--------+-----+ | bits|
>>> df.show(2)
+--------+-------------+--------+--------------+--------+-------+---------+--------+--------+-------------+--------+-----+
| bits| dst_ip|dst_port|flow_direction|in_iface|ip_dscp|out_iface| pkts|protocol| src_ip|src_port| tag|
+--------+-------------+--------+--------------+--------+-------+---------+--------+--------+-------------+--------+-----+
|16062594|42.120.84.166| 11291| 1| 3| 36| 2|17606406| pnni|42.120.84.115| 14166|10008|
|13914480|42.120.82.254| 13667| 0| 4| 32| 1|13953516| ax.25| 42.120.86.49| 19810|10002|
+--------+-------------+--------+--------------+--------+-------+---------+--------+--------+-------------+--------+-----+
only showing top 2 rows
>>>
>>> df.cache()
DataFrame[bits: bigint, dst_ip: string, dst_port: bigint, flow_direction: string, in_iface: bigint, ip_dscp: string, out_iface: bigint, pkts: bigint, protocol: string, src_ip: string, src_port: bigint, tag: string]
>>> df.show(2)
16/05/16 15:59:32 ERROR Executor: Exception in task 0.0 in stage 14.0 (TID 14)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/opt/spark-1.6.0-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/worker.py", line 111, in main
process()
File "/opt/spark-1.6.0-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/worker.py", line 106, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/opt/spark-1.6.0-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/serializers.py", line 263, in dump_stream
vs = list(itertools.islice(iterator, batch))
File "<stdin>", line 1, in <lambda>
IndexError: list index out of range