Python 如何使用pandas中现有列之一的列表中的名称创建新列,并从另一列的列表中指定值?
我对大数据帧有一个问题,大约1kk行,180列。它以3列开始。第一列包含ID。第二行和第三行包含每行中的列表-它们是连接的(第一行-第一列列表中的第一个元素与第二列列表中的第一个元素连接:Python 如何使用pandas中现有列之一的列表中的名称创建新列,并从另一列的列表中指定值?,python,python-3.x,pandas,list,time,Python,Python 3.x,Pandas,List,Time,我对大数据帧有一个问题,大约1kk行,180列。它以3列开始。第一列包含ID。第二行和第三行包含每行中的列表-它们是连接的(第一行-第一列列表中的第一个元素与第二列列表中的第一个元素连接: ids | fruits | count | 1 | [grape, apple, banana] | [7.0, 4.0, 3.0] 2 | [mango, banana, strawberry, grape] | [5.0, 8.0, 15.0, 2.0] 3 | [apple, avocado]
ids | fruits | count |
1 | [grape, apple, banana] | [7.0, 4.0, 3.0]
2 | [mango, banana, strawberry, grape] | [5.0, 8.0, 15.0, 2.0]
3 | [apple, avocado] | [9.0, 1.0]
4 | NaN | NaN
5 | [pummelo] | [12.0]
我想使用'fruits'列中的列表元素作为新列的名称,这些新列的值将分配给row和fruits。但列不能重复,例如:
ids | grape | apple | banana | mango | strawberry | avocado | pummelo
1 | 7.0 | 4.0 | 3.0 | NaN | NaN | NaN | NaN
2 | 2.0 | NaN | 8.0 | 5.0 | 15.0 | NaN | NaN
3 | NaN | 9.0 | NaN | NaN | NaN | 1.0 | NaN
4 | NaN | NaN | NaN | NaN | NaN | NaN | NaN
5 | NaN | NaN | NaN | NaN | NaN | NaN | 12.0
集合中唯一元素的计数(所有列表的非重复和)'fruits'是180,这就是为什么我希望最后有180列
问题是速度。我尝试了pandas iterrows(),但当涉及到所有1kk行时,它就变成了无休止的故事。下面是我尝试过的代码
#making an example dataframe
import numpy as np
fruit_df = pd. DataFrame(columns=['ids','fruits','count'])
ids = [1,2,3,4,5]
fruits = [['grape', 'apple', 'banana'], ['mango', 'banana', 'strawberry', 'grape'], ['apple', 'avocado'], np.nan, ['pummelo']]
count = [[7.0, 4.0, 3.0],[5.0, 8.0, 15.0, 2.0], [9.0, 1.0], np.nan, [12.0]]
#creating fruits columns in dataframe - this one timing is ok , fine for me (about 15 mins)
fruits_columns=[]
for row in fruit_df['fruits']:
if type(row)==list:
fruits_columns.append(row)
else:
fruits_columns.append(list())
import itertools
all_fruits = list(itertools.chain(*fruits_columns))
all_fruits = set(all_fruits)
for fruit in all_fruits:
fruit_df[fruit]=np.nan
#iterating over the data - here is main problem - takes very, very long time.. works well for this tiny dataset but when it comes to 1000000 rows and 180 columns...
def iter_over_rows(data):
for index, row in data.iterrows():
if type(row['fruits'])!=float:
for cat in range(len(row['fruits'])):
data[row['fruits'][cat]][index] = row['count'][cat]
我想加快数据处理的速度。考虑过用所有180个水果作为键并将它们作为值来制作字典,但在最后,顺序会被破坏。如果你知道如何更快地处理,那就太好了。干杯!这会做你想做的一切,但它会删除
ID4
,因为它们只会同时使用ntainNA
值
设置相同:
fruit_df = pd. DataFrame(columns=['ids','fruits','count'])
ids = [1,2,3,4,5]
fruits = [['grape', 'apple', 'banana'], ['mango', 'banana', 'strawberry', 'grape'], ['apple', 'avocado'], np.nan, ['\
pummelo']]
count = [[7.0, 4.0, 3.0],[5.0, 8.0, 15.0, 2.0], [9.0, 1.0], np.nan, [12.0]]
fruit_df['ids'] = ids
fruit_df['fruits'] = fruits
fruit_df['count'] = count
我们希望将包含列表的行转换为堆叠系列(这基本上只是将列表扩展为新行,同时保留行的ID:
fruit_df.set_index(['ids'], inplace=True)
fruit_series = fruit_df.apply(lambda x: pd.Series(x['fruits']), axis=1).stack()
count_series = fruit_df.apply(lambda x: pd.Series(x['count']), axis=1).stack()
final_df = pd.DataFrame()
final_df['Fruits'] = fruit_series
final_df['Counts'] = count_series
print(final_df)
所以我们看到最终的_df看起来像:
Fruits Counts
ids
1 0 grape 7.0
1 apple 4.0
2 banana 3.0
2 0 mango 5.0
1 banana 8.0
2 strawberry 15.0
3 grape 2.0
3 0 apple 9.0
1 avocado 1.0
5 0 pummelo 12.0
好的,很酷,现在我们已经扩展了列表行以匹配它们的id,但是我们现在看到了这个我们不想要的multi_索引df,所以我们将删除它,然后旋转我们的表,使id成为索引,并生成列:
final_df = final_df.reset_index().drop('level_1', axis=1)
final_df = final_df.pivot(index='ids', columns = 'Fruits', values = 'Counts')
print(final_df)
返回:
Fruits apple avocado banana grape mango pummelo strawberry
ids
1 4.0 NaN 3.0 7.0 NaN NaN NaN
2 NaN NaN 8.0 2.0 5.0 NaN 15.0
3 9.0 1.0 NaN NaN NaN NaN NaN
5 NaN NaN NaN NaN NaN 12.0 NaN
非常接近,我希望这对你有用!
整个代码组合:
import pandas as pd
import numpy as np
fruit_df = pd. DataFrame(columns=['ids','fruits','count'])
ids = [1,2,3,4,5]
fruits = [['grape', 'apple', 'banana'], ['mango', 'banana', 'strawberry', 'grape'], ['apple', 'avocado'], np.nan, ['\
pummelo']]
count = [[7.0, 4.0, 3.0],[5.0, 8.0, 15.0, 2.0], [9.0, 1.0], np.nan, [12.0]]
fruit_df['ids'] = ids
fruit_df['fruits'] = fruits
fruit_df['count'] = count
fruit_df.set_index(['ids'], inplace=True)
fruit_series = fruit_df.apply(lambda x: pd.Series(x['fruits']), axis=1).stack()
count_series = fruit_df.apply(lambda x: pd.Series(x['count']), axis=1).stack()
final_df = pd.DataFrame()
final_df['Fruits'] = fruit_series
final_df['Counts'] = count_series
final_df = final_df.reset_index().drop('level_1', axis=1)
final_df = final_df.pivot(index='ids', columns = 'Fruits', values = 'Counts')
print(final_df)
这将完成您想要的所有操作,但它将删除IDs4,因为它们只包含NA值 设置相同:
fruit_df = pd. DataFrame(columns=['ids','fruits','count'])
ids = [1,2,3,4,5]
fruits = [['grape', 'apple', 'banana'], ['mango', 'banana', 'strawberry', 'grape'], ['apple', 'avocado'], np.nan, ['\
pummelo']]
count = [[7.0, 4.0, 3.0],[5.0, 8.0, 15.0, 2.0], [9.0, 1.0], np.nan, [12.0]]
fruit_df['ids'] = ids
fruit_df['fruits'] = fruits
fruit_df['count'] = count
我们希望将包含列表的行转换为堆叠系列(这基本上只是将列表扩展为新行,同时保留行的ID:
fruit_df.set_index(['ids'], inplace=True)
fruit_series = fruit_df.apply(lambda x: pd.Series(x['fruits']), axis=1).stack()
count_series = fruit_df.apply(lambda x: pd.Series(x['count']), axis=1).stack()
final_df = pd.DataFrame()
final_df['Fruits'] = fruit_series
final_df['Counts'] = count_series
print(final_df)
所以我们看到最终的_df看起来像:
Fruits Counts
ids
1 0 grape 7.0
1 apple 4.0
2 banana 3.0
2 0 mango 5.0
1 banana 8.0
2 strawberry 15.0
3 grape 2.0
3 0 apple 9.0
1 avocado 1.0
5 0 pummelo 12.0
好的,很酷,现在我们已经扩展了列表行以匹配它们的id,但是我们现在看到了这个我们不想要的multi_索引df,所以我们将删除它,然后旋转我们的表,使id成为索引,并生成列:
final_df = final_df.reset_index().drop('level_1', axis=1)
final_df = final_df.pivot(index='ids', columns = 'Fruits', values = 'Counts')
print(final_df)
返回:
Fruits apple avocado banana grape mango pummelo strawberry
ids
1 4.0 NaN 3.0 7.0 NaN NaN NaN
2 NaN NaN 8.0 2.0 5.0 NaN 15.0
3 9.0 1.0 NaN NaN NaN NaN NaN
5 NaN NaN NaN NaN NaN 12.0 NaN
非常接近,我希望这对你有用!
整个代码组合:
import pandas as pd
import numpy as np
fruit_df = pd. DataFrame(columns=['ids','fruits','count'])
ids = [1,2,3,4,5]
fruits = [['grape', 'apple', 'banana'], ['mango', 'banana', 'strawberry', 'grape'], ['apple', 'avocado'], np.nan, ['\
pummelo']]
count = [[7.0, 4.0, 3.0],[5.0, 8.0, 15.0, 2.0], [9.0, 1.0], np.nan, [12.0]]
fruit_df['ids'] = ids
fruit_df['fruits'] = fruits
fruit_df['count'] = count
fruit_df.set_index(['ids'], inplace=True)
fruit_series = fruit_df.apply(lambda x: pd.Series(x['fruits']), axis=1).stack()
count_series = fruit_df.apply(lambda x: pd.Series(x['count']), axis=1).stack()
final_df = pd.DataFrame()
final_df['Fruits'] = fruit_series
final_df['Counts'] = count_series
final_df = final_df.reset_index().drop('level_1', axis=1)
final_df = final_df.pivot(index='ids', columns = 'Fruits', values = 'Counts')
print(final_df)
工作得很好!我在设置之后用替换的方式处理了NaNs:fruit_df['fruits']=fruit_df['fruits'].应用(lambda d:d如果是instance(d,list)或者其他['no_fruit'])fruit_df['count']=fruit_df['count'].应用(lambda d:d如果是instance(d,list)或者[-1]),然后再次替换它们(对我来说,重要的是只保留NaNs的数据):final_df=final_df.replace(to_replace=['not_four'],value=np.nan)final_df=final_df.replace(to_replace=-1,value=np.nan)谢谢你的回答,我给了+1,但似乎我在这里太新了,它被计算了,但没有显示这是处理NaN's的一个好方法!我很高兴你发现这个答案很有用,并且我很高兴处理这个问题。效果很好!我在设置后用替换的方式处理NaN's:fruit_df['fruits']=fruit_df['fruits']。申请(lambda d:d如果isinstance(d,list)else['no_fruit'])fruit_df['count']=fruit_df['count'])应用(lambda d:d如果isinstance(d,list)else[-1]),然后再次替换它们(对我来说,重要的是仅用nans保存数据):final_df=final_df.replace(to_replace=['not_fruit'],value=np.nan)final_df=final_df.replace(to_replace=-1,value=np.nan)谢谢你的回答,我给出了+1,但似乎我在这里太新了,它被计算了,但没有显示这是处理nan的一个好方法!我很高兴你发现答案很有用,并且我在这方面很开心。