Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/6.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Python 如何找到两个rdd';皮斯帕克的钥匙?_Python_Apache Spark_Pyspark - Fatal编程技术网

Python 如何找到两个rdd';皮斯帕克的钥匙?

Python 如何找到两个rdd';皮斯帕克的钥匙?,python,apache-spark,pyspark,Python,Apache Spark,Pyspark,我有两个RDD作为: rdd1 = sc.parallelize([("www.page1.html", "word1"), ("www.page2.html", "word1"), ("www.page1.html", "word3")]) rdd2 = sc.parallelize([("www.page1.html", 7.3), ("www.page2.html", 1.25), ("www.page3.html", 5.41)]) intersection_rd

我有两个RDD作为:

rdd1 = sc.parallelize([("www.page1.html", "word1"), ("www.page2.html", "word1"), 
    ("www.page1.html", "word3")])

rdd2 = sc.parallelize([("www.page1.html", 7.3), ("www.page2.html", 1.25), 
    ("www.page3.html", 5.41)])

intersection_rdd = rdd1.keys().intersection(rdd2.keys())       
//当我这样做时,我得到的只是键的交叉点,即(www.page1.html,www.page2.html)

但是我需要两个RDD的键和两个值。 输出应如下所示:

[www.page1.html, (word1, word3, 7.3)]

[www.page2.html, (word1, 1.25)]

例如,您可以
cogroup
和筛选:

## This depends on empty resultiterable.ResultIterable
## evaluating to False

intersection_rdd = rdd1.cogroup(rdd2).filter(lambda x: x[1][0] and x[1][1])
intersection_rdd.map(lambda x: (x[0], (list(x[1][0]), list(x[1][1])))).collect()

## [('www.page1.html', (['word1', 'word3'], [7.3])),
##  ('www.page2.html', (['word1'], [1.25]))]

由于仅对关键帧使用设置操作,因此输出仅包括关键帧

rdd1.union(rdd2).groupByKey().mapValues(tuple).collect()
联合GroupByKey


('www.page1.html','word1')('www.page1.html',['word1','word3',7.3])
('www.page2.html','word1')('www.page2.html',['word1',1.25])
('www.page1.html','word3')('www.page3.html',[5.41])
('www.page1.html',7.3)
('www.page2.html',1.25)

('www.page3.html',5.41)

如果您包括OP尝试的解决方案不正确的原因,以及您的解决方案是如何解决的,这将非常有用。