Python 如何处理大型json文件(将其展平到tsv)
我正在处理一个大型JSON文件,特别是persona数据集() Persona Chat中的每个条目都是一个dict,带有两个键personal和personal 数据集是一个条目列表Python 如何处理大型json文件(将其展平到tsv),python,json,pandas,dataframe,Python,Json,Pandas,Dataframe,我正在处理一个大型JSON文件,特别是persona数据集() Persona Chat中的每个条目都是一个dict,带有两个键personal和personal 数据集是一个条目列表 personality: list of strings containing the personality of the agent utterances: list of dictionaries, each of which has two keys which are lists of strings.
personality: list of strings containing the personality of the agent
utterances: list of dictionaries, each of which has two keys which are lists of strings.
candidates: [next_utterance_candidate_1, ..., next_utterance_candidate_19]
The last candidate is the ground truth response observed in the conversational data
history: [dialog_turn_0, ... dialog_turn N], where N is an odd number since the other user starts every conversation.
我试图实现的是将其展平并以以下格式转换为tsv:
col_index, string (where string is the personality, candidates and history
但是
每当我尝试加载它并将其转换为数据帧时
import pandas as pd
df = pd.read_json(r'path')
display(df)
我得到以下错误:
ValueError: arrays must all be same length
无论是文章还是其他libs/框架和方法,甚至是面包屑,我们都非常感谢您的帮助
编辑:
我正在将它提供给另一个需要tsv的api,我正在考虑一种连接和保留结构的方法,以便重新构造它。要完全展平该文件,您需要
import json
def read_personachat_file(name="personachat_self_original.json"):
with open(name, "r") as f:
data = json.load(f)
for entry_type, chats in data.items():
for chat_id, chat in enumerate(chats):
personality = "|".join(chat["personality"])
for utt_id, utt in enumerate(chat["utterances"]):
for key in ("candidates", "history"):
for phrase_id, phrase in enumerate(utt[key]):
yield (entry_type, chat_id, personality, utt_id, key, phrase_id, phrase)
for entry in read_personachat_file():
print(entry)
('train', 313, 'i like to wear red .|i wear a red purse .|i like to wear red shoes also .|i use red lipstick .|i drive a red car .', 5, 'candidates', 7, 'my sister will be my mom , she wants me to get married')
('train', 313, 'i like to wear red .|i wear a red purse .|i like to wear red shoes also .|i use red lipstick .|i drive a red car .', 5, 'candidates', 8, 'hi , how are ya ?')
('train', 313, 'i like to wear red .|i wear a red purse .|i like to wear red shoes also .|i use red lipstick .|i drive a red car .', 5, 'candidates', 9, 'sounds good . i am just sitting here with my dog . i love animals .')
('train', 313, 'i like to wear red .|i wear a red purse .|i like to wear red shoes also .|i use red lipstick .|i drive a red car .', 5, 'candidates', 10, "sure i'll go with you but i am baking a pizza right now , my favorite . come eat .")
('train', 313, 'i like to wear red .|i wear a red purse .|i like to wear red shoes also .|i use red lipstick .|i drive a red car .', 5, 'candidates', 11, 'where do you work then soccer person ?')
('train', 313, 'i like to wear red .|i wear a red purse .|i like to wear red shoes also .|i use red lipstick .|i drive a red car .', 5, 'candidates', 12, 'it is so pretty in the fall and winter , my favorite time to go')
('train', 313, 'i like to wear red .|i wear a red purse .|i like to wear red shoes also .|i use red lipstick .|i drive a red car .', 5, 'candidates', 13, 'i to travel and meet new people')
输出将类似于
import json
def read_personachat_file(name="personachat_self_original.json"):
with open(name, "r") as f:
data = json.load(f)
for entry_type, chats in data.items():
for chat_id, chat in enumerate(chats):
personality = "|".join(chat["personality"])
for utt_id, utt in enumerate(chat["utterances"]):
for key in ("candidates", "history"):
for phrase_id, phrase in enumerate(utt[key]):
yield (entry_type, chat_id, personality, utt_id, key, phrase_id, phrase)
for entry in read_personachat_file():
print(entry)
('train', 313, 'i like to wear red .|i wear a red purse .|i like to wear red shoes also .|i use red lipstick .|i drive a red car .', 5, 'candidates', 7, 'my sister will be my mom , she wants me to get married')
('train', 313, 'i like to wear red .|i wear a red purse .|i like to wear red shoes also .|i use red lipstick .|i drive a red car .', 5, 'candidates', 8, 'hi , how are ya ?')
('train', 313, 'i like to wear red .|i wear a red purse .|i like to wear red shoes also .|i use red lipstick .|i drive a red car .', 5, 'candidates', 9, 'sounds good . i am just sitting here with my dog . i love animals .')
('train', 313, 'i like to wear red .|i wear a red purse .|i like to wear red shoes also .|i use red lipstick .|i drive a red car .', 5, 'candidates', 10, "sure i'll go with you but i am baking a pizza right now , my favorite . come eat .")
('train', 313, 'i like to wear red .|i wear a red purse .|i like to wear red shoes also .|i use red lipstick .|i drive a red car .', 5, 'candidates', 11, 'where do you work then soccer person ?')
('train', 313, 'i like to wear red .|i wear a red purse .|i like to wear red shoes also .|i use red lipstick .|i drive a red car .', 5, 'candidates', 12, 'it is so pretty in the fall and winter , my favorite time to go')
('train', 313, 'i like to wear red .|i wear a red purse .|i like to wear red shoes also .|i use red lipstick .|i drive a red car .', 5, 'candidates', 13, 'i to travel and meet new people')
(无论这对您是否有用)。不要使用
pd.read\u json()
尝试读取非表格数据。只需使用json.load()
。此外,考虑到个性
/候选项
/历史记录
,您的CSV文件将非常奇怪。我正在将其提供给另一个需要tsv的api,我正在考虑一种连接和保留结构的方法,以重新构建它。谢谢你的json.load()评论。谢谢,这是一个很好的开始,请在我身上喝点咖啡。