Python pyspark rdd将多个json文件数据合并到一个rdd中

我正在尝试将两个文件数据合并到一个rdd中。假设我有两个文件

file1.txt

，这是一个大的json格式文件，

file2.txt

这是一个小的csv格式文件

file1.txt

（json格式）如下所示：

{

{"a":1 , "b":{"ba":1 , "bb":"babc", "bc":"babc2", "d":"babc3"}, "c":"abc2"}, 

{"a":2 , "b":{"ba":2 , "bb":"babc2", "bc":"babc22", "d":"babc32"}, "c":"abc22"}

}

key   value

xyz   xyz1

pqr   pqr1

{

{"a":1 , "b":{"ba":1 , "bb":"babc", "bc":"babc2", "d":"babc3", "e":{{"key": "xyz", "value":"xyz1"},{"key": "pqr", "value":"pqr1"}}, "c":"abc2"}, 

{"a":2 , "b":{"ba":2 , "bb":"babc2", "bc":"babc22", "d":"babc32", "e":{{"key": "xyz", "value":"xyz1"},{"key": "pqr", "value":"pqr1"}}, "c":"abc22"}

}

def joinfunc(file1_json, file2_json):

    file1_detail = json.loads(file1_json)
    file2_detail = json.loads(file2_json)

    b_file1_detail_list = file1_detail["b"]
    append_detail =[]

    if b_file1_detail_list is not None:
        for b_file1_detail in b_file1_detail_list:
            append_detail = {"e" : file2_detail}
            b_file1_detail.append(append_detail)

    return json.dumps(dict(file1_detail))

file2.txt

（csv格式）如下所示：

{

{"a":1 , "b":{"ba":1 , "bb":"babc", "bc":"babc2", "d":"babc3"}, "c":"abc2"}, 

{"a":2 , "b":{"ba":2 , "bb":"babc2", "bc":"babc22", "d":"babc32"}, "c":"abc22"}

}

key   value

xyz   xyz1

pqr   pqr1

{

{"a":1 , "b":{"ba":1 , "bb":"babc", "bc":"babc2", "d":"babc3", "e":{{"key": "xyz", "value":"xyz1"},{"key": "pqr", "value":"pqr1"}}, "c":"abc2"}, 

{"a":2 , "b":{"ba":2 , "bb":"babc2", "bc":"babc22", "d":"babc32", "e":{{"key": "xyz", "value":"xyz1"},{"key": "pqr", "value":"pqr1"}}, "c":"abc22"}

}

def joinfunc(file1_json, file2_json):

    file1_detail = json.loads(file1_json)
    file2_detail = json.loads(file2_json)

    b_file1_detail_list = file1_detail["b"]
    append_detail =[]

    if b_file1_detail_list is not None:
        for b_file1_detail in b_file1_detail_list:
            append_detail = {"e" : file2_detail}
            b_file1_detail.append(append_detail)

    return json.dumps(dict(file1_detail))

现在我希望我的输出是rdd，如下所示：

{

{"a":1 , "b":{"ba":1 , "bb":"babc", "bc":"babc2", "d":"babc3"}, "c":"abc2"}, 

{"a":2 , "b":{"ba":2 , "bb":"babc2", "bc":"babc22", "d":"babc32"}, "c":"abc22"}

}

key   value

xyz   xyz1

pqr   pqr1

{

{"a":1 , "b":{"ba":1 , "bb":"babc", "bc":"babc2", "d":"babc3", "e":{{"key": "xyz", "value":"xyz1"},{"key": "pqr", "value":"pqr1"}}, "c":"abc2"}, 

{"a":2 , "b":{"ba":2 , "bb":"babc2", "bc":"babc22", "d":"babc32", "e":{{"key": "xyz", "value":"xyz1"},{"key": "pqr", "value":"pqr1"}}, "c":"abc22"}

}

def joinfunc(file1_json, file2_json):

    file1_detail = json.loads(file1_json)
    file2_detail = json.loads(file2_json)

    b_file1_detail_list = file1_detail["b"]
    append_detail =[]

    if b_file1_detail_list is not None:
        for b_file1_detail in b_file1_detail_list:
            append_detail = {"e" : file2_detail}
            b_file1_detail.append(append_detail)

    return json.dumps(dict(file1_detail))

我尝试的是将

file2.txt

转换为json格式，然后执行如下操作：

output_rdd = file1_rdd.map(lambda x: joinfunc(x, file2_rdd))

然后尝试写入

输出\u rdd

但这给了我一些错误，比如

cPickle.PicklingError:无法序列化对象：Py4JError:错误 调用o72时发生。getnewargs。跟踪：py4j.Py4JException: 方法getnewargs（[]）不存在

有没有建议将这两个文件合并为一个输出rdd？任何建议都会有帮助

附言：我对星火是完全陌生的

编辑：

我的joinfunc如下所示：

{

{"a":1 , "b":{"ba":1 , "bb":"babc", "bc":"babc2", "d":"babc3"}, "c":"abc2"}, 

{"a":2 , "b":{"ba":2 , "bb":"babc2", "bc":"babc22", "d":"babc32"}, "c":"abc22"}

}

key   value

xyz   xyz1

pqr   pqr1

{

{"a":1 , "b":{"ba":1 , "bb":"babc", "bc":"babc2", "d":"babc3", "e":{{"key": "xyz", "value":"xyz1"},{"key": "pqr", "value":"pqr1"}}, "c":"abc2"}, 

{"a":2 , "b":{"ba":2 , "bb":"babc2", "bc":"babc22", "d":"babc32", "e":{{"key": "xyz", "value":"xyz1"},{"key": "pqr", "value":"pqr1"}}, "c":"abc22"}

}

def joinfunc(file1_json, file2_json):

    file1_detail = json.loads(file1_json)
    file2_detail = json.loads(file2_json)

    b_file1_detail_list = file1_detail["b"]
    append_detail =[]

    if b_file1_detail_list is not None:
        for b_file1_detail in b_file1_detail_list:
            append_detail = {"e" : file2_detail}
            b_file1_detail.append(append_detail)

    return json.dumps(dict(file1_detail))

---

修正你的

lambda

打字错误。共享您的

joinfunc

代码，同时修复打字错误并共享joinfunc代码