Python 从json对象创建数据帧
我终于从一个包含许多json对象的文件中获得了所需的数据输出,但我需要一些帮助,以便在数据循环时将下面的输出转换为单个数据帧。以下是生成输出的代码,包括输出外观的示例: 原始数据:Python 从json对象创建数据帧,python,json,pandas,Python,Json,Pandas,我终于从一个包含许多json对象的文件中获得了所需的数据输出,但我需要一些帮助,以便在数据循环时将下面的输出转换为单个数据帧。以下是生成输出的代码,包括输出外观的示例: 原始数据: { "zipcode":"08989", "current"{"canwc":null,"cig":4900,"class":"observation","clds":"OVC","day_ind":"D","dewpt":19,"expireTimeGMT":1385486700,"feels_like":34,"
{
"zipcode":"08989",
"current"{"canwc":null,"cig":4900,"class":"observation","clds":"OVC","day_ind":"D","dewpt":19,"expireTimeGMT":1385486700,"feels_like":34,"gust":null,"hi":37,"humidex":null,"icon_code":26,"icon_extd":2600,"max_temp":37,"wxMan":"wx1111"},
"triggers":[53,31,9,21,48,7,40,178,55,179,176,26,103,175,33,51,20,57,112,30,50,113]
}
{
"zipcode":"08990",
"current":{"canwc":null,"cig":4900,"class":"observation","clds":"OVC","day_ind":"D","dewpt":19,"expireTimeGMT":1385486700,"feels_like":34,"gust":null,"hi":37,"humidex":null,"icon_code":26,"icon_extd":2600,"max_temp":37, "wxMan":"wx1111"},
"triggers":[53,31,9,21,48,7,40,178,55,179,176,26,103,175,33,51,20,57,112,30,50,113]
}
def lines_per_n(f, n):
for line in f:
yield ''.join(chain([line], itertools.islice(f, n - 1)))
for fin in glob.glob('*.txt'):
with open(fin) as f:
for chunk in lines_per_n(f, 5):
try:
jfile = json.loads(chunk)
zipcode = jfile['zipcode']
datetime = jfile['current']['proc_time']
triggers = jfile['triggers']
print pd.Series(jfile['zipcode']),
pd.Series(jfile['current']['proc_time']),\
jfile['triggers']
except ValueError, e:
pass
else:
pass
当我运行上面的代码时得到的示例输出,我想将其作为3列存储在pandas数据框架中
08988 20131126102946 []
08989 20131126102946 [53, 31, 9, 21, 48, 7, 40, 178, 55, 179]
08988 20131126102946 []
08989 20131126102946 [53, 31, 9, 21, 48, 7, 40, 178, 55, 179]
00544 20131126102946 [178, 30, 176, 103, 179, 112, 21, 20, 48]
下面的代码看起来更接近,因为如果我在列表中传递并转置df,它会给我一个时髦的df。你知道我怎样才能把它正确地整形吗
def series_chunk(chunk):
jfile = json.loads(chunk)
zipcode = jfile['zipcode']
datetime = jfile['current']['proc_time']
triggers = jfile['triggers']
return jfile['zipcode'],\
jfile['current']['proc_time'],\
jfile['triggers']
for fin in glob.glob('*.txt'):
with open(fin) as f:
for chunk in lines_per_n(f, 7):
df1 = pd.DataFrame(list(series_chunk(chunk)))
print df1.T
[u'08988', u'20131126102946', []]
[u'08989', u'20131126102946', [53, 31, 9, 21, 48, 7, 40, 178, 55, 179]]
[u'08988', u'20131126102946', []]
[u'08989', u'20131126102946', [53, 31, 9, 21, 48, 7, 40, 178, 55, 179]]
数据帧:
0 1 2
0 08988 20131126102946 []
0 1 2
0 08989 20131126102946 [53, 31, 9, 21, 48, 7, 40, 178, 55, 179, 176, ...
0 1 2
0 08988 20131126102946 []
0 1 2
0 08989 20131126102946 [53, 31, 9, 21, 48, 7, 40, 178, 55, 179, 176, ...
这是我的最终代码和输出。如何捕获它通过循环创建的每个数据帧,并将它们动态连接为一个数据帧对象
for fin in glob.glob('*.txt'):
with open(fin) as f:
print pd.concat([series_chunk(chunk) for chunk in lines_per_n(f, 7)], axis=1).T
0 1 2
0 08988 20131126102946 []
1 08989 20131126102946 [53, 31, 9, 21, 48, 7, 40, 178, 55, 179, 176, ...
0 1 2
0 08988 20131126102946 []
1 08989 20131126102946 [53, 31, 9, 21, 48, 7, 40, 178, 55, 179, 176, ...
注意:对于那些想将json解析成pandas的人来说,如果你有有效的json(这个问题没有),那么你应该使用pandas函数:
查看以下示例、可以传递给此函数的参数以及规范化结构化程度较低的json的方法
例如,如果没有有效的json,在作为json读入之前咀嚼字符串通常是有效的
如果您有多个json文件,则应将数据帧连接在一起(类似于此答案中的内容):
这个例子的原始答案是:
在正则表达式中对传递给read_csv的分隔符使用查找:
In [11]: df = pd.read_csv('foo.csv', sep='(?<!,)\s', header=None)
In [12]: df
Out[12]:
0 1 2
0 8988 20131126102946 []
1 8989 20131126102946 [53, 31, 9, 21, 48, 7, 40, 178, 55, 179, 176, ...
2 8988 20131126102946 []
3 8989 20131126102946 [53, 31, 9, 21, 48, 7, 40, 178, 55, 179, 176, ...
4 544 20131126102946 [178, 30, 176, 103, 179, 112, 21, 20, 48, 7, 5...
5 601 20131126094911 []
6 602 20131126101056 []
7 603 20131126101056 []
8 604 20131126101056 []
9 544 20131126102946 [178, 30, 176, 103, 179, 112, 21, 20, 48, 7, 5...
10 601 20131126094911 []
11 602 20131126101056 []
12 603 20131126101056 []
13 604 20131126101056 []
[14 rows x 3 columns]
注意:您也可以将try/except移动到
系列\u块中
,但是如果我不是从csv工作呢?我需要处理一个json对象的大文件,然后根据问题中的三个键生成值。我正试图将数据放入数据框中进行进一步处理。@prometheus2305这是如何从您提供的输出创建数据框(虽然严格来说不是“csv”!)。我想你可以更直接地把一些东西粘在一起,但如果没有感谢的话就很难了。我已经添加了所有代码和示例数据。希望能够以某种方式将对象输出合并到数据帧中。如果您有任何帮助,我们将不胜感激。您想要的输出是什么?与regex的read_csv数据帧方法中的输出完全相同。只有3列,其中包含指定3个键的键和值。我认为您可以使用concat来包装它,类似于:pd.concat([series_chunk(chunk)for chunk in line_per_n(f,5)]),其中series_chunk是将每行作为一个序列(try/except块中的位)返回的函数。@AndyHayden感谢您的帮助。我被连接到df对象的最后一步卡住了。我已经更新了我的代码和输出。如果您能提供任何指导,我将不胜感激。随附于我的答案后,您将走上正确的道路。
pd.concat([pd.read_json(file) for file in ...], ignore_index=True)
In [11]: df = pd.read_csv('foo.csv', sep='(?<!,)\s', header=None)
In [12]: df
Out[12]:
0 1 2
0 8988 20131126102946 []
1 8989 20131126102946 [53, 31, 9, 21, 48, 7, 40, 178, 55, 179, 176, ...
2 8988 20131126102946 []
3 8989 20131126102946 [53, 31, 9, 21, 48, 7, 40, 178, 55, 179, 176, ...
4 544 20131126102946 [178, 30, 176, 103, 179, 112, 21, 20, 48, 7, 5...
5 601 20131126094911 []
6 602 20131126101056 []
7 603 20131126101056 []
8 604 20131126101056 []
9 544 20131126102946 [178, 30, 176, 103, 179, 112, 21, 20, 48, 7, 5...
10 601 20131126094911 []
11 602 20131126101056 []
12 603 20131126101056 []
13 604 20131126101056 []
[14 rows x 3 columns]
def series_chunk(chunk):
jfile = json.loads(chunk)
zipcode = jfile['zipcode']
datetime = jfile['current']['proc_time']
triggers = jfile['triggers']
return pd.Series([jfile['zipcode'], jfile['current']['proc_time'], jfile['triggers']])
dfs = []
for fin in glob.glob('*.txt'):
with open(fin) as f:
df = pd.concat([series_chunk(chunk) for chunk in lines_per_n(f, 5)], axis=1)
dfs.append(dfs)
df = pd.concat(dfs, ignore_index=True)