Spark-toDebugString在python中不好
这是我在scala中使用编辑字符串时得到的:Spark-toDebugString在python中不好,python,scala,apache-spark,Python,Scala,Apache Spark,这是我在scala中使用编辑字符串时得到的: scala> val a = sc.parallelize(Array(1,2,3)).distinct a: org.apache.spark.rdd.RDD[Int] = MappedRDD[3] at distinct at <console>:12 scala> a.toDebugString res0: String = (4) MappedRDD[3] at distinct at <console&g
scala> val a = sc.parallelize(Array(1,2,3)).distinct
a: org.apache.spark.rdd.RDD[Int] = MappedRDD[3] at distinct at <console>:12
scala> a.toDebugString
res0: String =
(4) MappedRDD[3] at distinct at <console>:12
| ShuffledRDD[2] at distinct at <console>:12
+-(4) MappedRDD[1] at distinct at <console>:12
| ParallelCollectionRDD[0] at parallelize at <console>:12
>>> a = sc.parallelize([1,2,3]).distinct()
>>> a.toDebugString()
'(4) PythonRDD[6] at RDD at PythonRDD.scala:43\n | MappedRDD[5] at values at NativeMethodAccessorImpl.java:-2\n | ShuffledRDD[4] at partitionBy at NativeMethodAccessorImpl.java:-2\n +-(4) PairwiseRDD[3] at RDD at PythonRDD.scala:261\n | PythonRDD[2] at RDD at PythonRDD.scala:43\n | ParallelCollectionRDD[0] at parallelize at PythonRDD.scala:315'
如您所见,python中的输出不如scala中的好。有什么技巧可以让这个函数有更好的输出吗
我正在使用Spark 1.1.0 尝试添加
print
语句,以便实际打印调试字符串,而不是显示其\uuuu repr\uuu
:
>>> a = sc.parallelize([1,2,3]).distinct()
>>> print a.toDebugString()
(8) PythonRDD[27] at RDD at PythonRDD.scala:44 [Serialized 1x Replicated]
| MappedRDD[26] at values at NativeMethodAccessorImpl.java:-2 [Serialized 1x Replicated]
| ShuffledRDD[25] at partitionBy at NativeMethodAccessorImpl.java:-2 [Serialized 1x Replicated]
+-(8) PairwiseRDD[24] at distinct at <stdin>:1 [Serialized 1x Replicated]
| PythonRDD[23] at distinct at <stdin>:1 [Serialized 1x Replicated]
| ParallelCollectionRDD[21] at parallelize at PythonRDD.scala:358 [Serialized 1x Replicated]
a=sc.parallelize([1,2,3]).distinct()
>>>打印a.toDebugString()
(8) PythonRDD.scala:44[序列化1x复制]
|NativeMethodAccessorImpl.java处的值处的MappedRDD[26]:-2[序列化1x复制]
|NativeMethodAccessorImpl.java分区中的Shuffledd[25]:-2[序列化1x复制]
+-(8) 不同位置处的成对ISERDD[24]:1[序列化1x复制]
|PythonRDD[23]位于不同的位置:1[序列化1x复制]
|PythonRDD.scala:358[序列化1x复制]
它没有被执行,只是被缓存 你应使用:
a = sc.parallelize([1,2,3]).distinct()
a.collect()
[1, 2, 3]