Pandas 熊猫中的自定义布尔过滤?
我有一个数据帧Pandas 熊猫中的自定义布尔过滤?,pandas,filtering,Pandas,Filtering,我有一个数据帧 0 1 2 3 Marketcap 0 1.707280 0.666952 0.638515 -0.061126 2.291747 1.71B 1 -1.017134 1.353627 0.618433 0.008279 0.148128 1.82B 2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851
0 1 2 3 Marketcap
0 1.707280 0.666952 0.638515 -0.061126 2.291747 1.71B
1 -1.017134 1.353627 0.618433 0.008279 0.148128 1.82B
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
是否有某种定制的过滤器方法,可以让Python知道B>M>K
假设我想过滤,df[df.Marketcap>35.00M]
,有没有一种聪明或干净的方法可以做到这一点?
使用M或B可以使值非常可读且易于区分
多谢各位
编辑:重新打开线程作为Max U的答案,而优秀似乎产生了一个熊猫bug,我们在Github上打开了一个问题 这不是超级干净,但它做到了这一点,并且没有使用任何python迭代: 代码:
# Create a separate column (which you can omit later) that converts 'Marketcap' strings to numbers
df['cap'] = df.loc[df['Marketcap'].str.contains('B'), 'Marketcap'].str.replace('B','').astype(float) * 1000
df['cap'].fillna(df.loc[df['Marketcap'].str.contains('M'), 'Marketcap'].str.replace('M',''), inplace = True)
# For pandas pre-0.20.0 (<May 2017)
print df.ix[df['cap'].astype(float) > 35, :-1]
# For pandas 0.20.0+ (.ix[] deprecated)
print df.iloc[df[df['cap'].astype(float) > 35].index, :-1]
# Or, alternate pandas 0.20.0+ option (thanks @Psidom)
print df[df['cap'].astype(float) > 35].iloc[:,:-1]
0 1 2 3 4 Marketcap
0 1.707280 0.666952 0.638515 -0.061126 2.291747 1.71B
1 -1.017134 1.353627 0.618433 0.008279 0.148128 1.82B
4 2.029370 0.899612 0.261146 1.474148 -1.663970 100.9M
In [44]: df
Out[44]:
0 1 2 3 4 Marketcap
0 1.707280 0.666952 0.638515 -0.061126 2.291747 1.71B
1 -1.017134 1.353627 0.618433 0.008279 0.148128 1.82B
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
In [45]: df[pd.eval(df.Marketcap.replace(['[Kk]','[Mm]','[Bb]'],
['*10**3','*10**6','*10**9'], regex=True) \
.add(' < 35*10**6'))]
Out[45]:
0 1 2 3 4 Marketcap
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
In [176]: df
Out[176]:
0 1 2 3 Market Cap
0 1.707280 0.666952 0.638515 -0.061126 2.291747 1.71B
1 -1.017134 1.353627 0.618433 0.008279 0.148128 1.82B
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
to_replace = ['\d+\s*[Kk]','\d+\s*[Mm]','\d+\s*[Bb]', '-1', 'N/A']
value = [1000,1000000,1000000000, 1, 1]
mask = df.assign(
f=df['Market Cap'].replace(to_replace, value, regex=True),
Marketcap=pd.to_numeric(df['Market Cap'].str.replace(r'[^\d\.]', ''), errors='coerce')
).eval("Marketcap * f < 35000000")
df[mask]
In [178]: df[mask]
Out[178]:
0 1 2 3 Market Cap
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
更新:
# Create a separate column (which you can omit later) that converts 'Marketcap' strings to numbers
df['cap'] = df.loc[df['Marketcap'].str.contains('B'), 'Marketcap'].str.replace('B','').astype(float) * 1000
df['cap'].fillna(df.loc[df['Marketcap'].str.contains('M'), 'Marketcap'].str.replace('M',''), inplace = True)
# For pandas pre-0.20.0 (<May 2017)
print df.ix[df['cap'].astype(float) > 35, :-1]
# For pandas 0.20.0+ (.ix[] deprecated)
print df.iloc[df[df['cap'].astype(float) > 35].index, :-1]
# Or, alternate pandas 0.20.0+ option (thanks @Psidom)
print df[df['cap'].astype(float) > 35].iloc[:,:-1]
0 1 2 3 4 Marketcap
0 1.707280 0.666952 0.638515 -0.061126 2.291747 1.71B
1 -1.017134 1.353627 0.618433 0.008279 0.148128 1.82B
4 2.029370 0.899612 0.261146 1.474148 -1.663970 100.9M
In [44]: df
Out[44]:
0 1 2 3 4 Marketcap
0 1.707280 0.666952 0.638515 -0.061126 2.291747 1.71B
1 -1.017134 1.353627 0.618433 0.008279 0.148128 1.82B
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
In [45]: df[pd.eval(df.Marketcap.replace(['[Kk]','[Mm]','[Bb]'],
['*10**3','*10**6','*10**9'], regex=True) \
.add(' < 35*10**6'))]
Out[45]:
0 1 2 3 4 Marketcap
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
In [176]: df
Out[176]:
0 1 2 3 Market Cap
0 1.707280 0.666952 0.638515 -0.061126 2.291747 1.71B
1 -1.017134 1.353627 0.618433 0.008279 0.148128 1.82B
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
to_replace = ['\d+\s*[Kk]','\d+\s*[Mm]','\d+\s*[Bb]', '-1', 'N/A']
value = [1000,1000000,1000000000, 1, 1]
mask = df.assign(
f=df['Market Cap'].replace(to_replace, value, regex=True),
Marketcap=pd.to_numeric(df['Market Cap'].str.replace(r'[^\d\.]', ''), errors='coerce')
).eval("Marketcap * f < 35000000")
df[mask]
In [178]: df[mask]
Out[178]:
0 1 2 3 Market Cap
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
说明:
In [14]: df.Marketcap.replace(['M','B'],['','*1000'], regex=True)
Out[14]:
0 1.71*1000
1 1.82*1000
2 1.1
3 30.92
4 100.9
Name: Marketcap, dtype: object
In [15]: df.Marketcap.replace(['M','B'],['','*1000'], regex=True).add(' > 35')
Out[15]:
0 1.71*1000 > 35
1 1.82*1000 > 35
2 1.1 > 35
3 30.92 > 35
4 100.9 > 35
Name: Marketcap, dtype: object
In [16]: pd.eval(df.Marketcap.replace(['M','B'],['','*1000'], regex=True).add(' > 35'))
Out[16]: array([True, True, False, False, True], dtype=object)
源DF:
# Create a separate column (which you can omit later) that converts 'Marketcap' strings to numbers
df['cap'] = df.loc[df['Marketcap'].str.contains('B'), 'Marketcap'].str.replace('B','').astype(float) * 1000
df['cap'].fillna(df.loc[df['Marketcap'].str.contains('M'), 'Marketcap'].str.replace('M',''), inplace = True)
# For pandas pre-0.20.0 (<May 2017)
print df.ix[df['cap'].astype(float) > 35, :-1]
# For pandas 0.20.0+ (.ix[] deprecated)
print df.iloc[df[df['cap'].astype(float) > 35].index, :-1]
# Or, alternate pandas 0.20.0+ option (thanks @Psidom)
print df[df['cap'].astype(float) > 35].iloc[:,:-1]
0 1 2 3 4 Marketcap
0 1.707280 0.666952 0.638515 -0.061126 2.291747 1.71B
1 -1.017134 1.353627 0.618433 0.008279 0.148128 1.82B
4 2.029370 0.899612 0.261146 1.474148 -1.663970 100.9M
In [44]: df
Out[44]:
0 1 2 3 4 Marketcap
0 1.707280 0.666952 0.638515 -0.061126 2.291747 1.71B
1 -1.017134 1.353627 0.618433 0.008279 0.148128 1.82B
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
In [45]: df[pd.eval(df.Marketcap.replace(['[Kk]','[Mm]','[Bb]'],
['*10**3','*10**6','*10**9'], regex=True) \
.add(' < 35*10**6'))]
Out[45]:
0 1 2 3 4 Marketcap
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
In [176]: df
Out[176]:
0 1 2 3 Market Cap
0 1.707280 0.666952 0.638515 -0.061126 2.291747 1.71B
1 -1.017134 1.353627 0.618433 0.008279 0.148128 1.82B
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
to_replace = ['\d+\s*[Kk]','\d+\s*[Mm]','\d+\s*[Bb]', '-1', 'N/A']
value = [1000,1000000,1000000000, 1, 1]
mask = df.assign(
f=df['Market Cap'].replace(to_replace, value, regex=True),
Marketcap=pd.to_numeric(df['Market Cap'].str.replace(r'[^\d\.]', ''), errors='coerce')
).eval("Marketcap * f < 35000000")
df[mask]
In [178]: df[mask]
Out[178]:
0 1 2 3 Market Cap
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
解决方案:
# Create a separate column (which you can omit later) that converts 'Marketcap' strings to numbers
df['cap'] = df.loc[df['Marketcap'].str.contains('B'), 'Marketcap'].str.replace('B','').astype(float) * 1000
df['cap'].fillna(df.loc[df['Marketcap'].str.contains('M'), 'Marketcap'].str.replace('M',''), inplace = True)
# For pandas pre-0.20.0 (<May 2017)
print df.ix[df['cap'].astype(float) > 35, :-1]
# For pandas 0.20.0+ (.ix[] deprecated)
print df.iloc[df[df['cap'].astype(float) > 35].index, :-1]
# Or, alternate pandas 0.20.0+ option (thanks @Psidom)
print df[df['cap'].astype(float) > 35].iloc[:,:-1]
0 1 2 3 4 Marketcap
0 1.707280 0.666952 0.638515 -0.061126 2.291747 1.71B
1 -1.017134 1.353627 0.618433 0.008279 0.148128 1.82B
4 2.029370 0.899612 0.261146 1.474148 -1.663970 100.9M
In [44]: df
Out[44]:
0 1 2 3 4 Marketcap
0 1.707280 0.666952 0.638515 -0.061126 2.291747 1.71B
1 -1.017134 1.353627 0.618433 0.008279 0.148128 1.82B
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
In [45]: df[pd.eval(df.Marketcap.replace(['[Kk]','[Mm]','[Bb]'],
['*10**3','*10**6','*10**9'], regex=True) \
.add(' < 35*10**6'))]
Out[45]:
0 1 2 3 4 Marketcap
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
In [176]: df
Out[176]:
0 1 2 3 Market Cap
0 1.707280 0.666952 0.638515 -0.061126 2.291747 1.71B
1 -1.017134 1.353627 0.618433 0.008279 0.148128 1.82B
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
to_replace = ['\d+\s*[Kk]','\d+\s*[Mm]','\d+\s*[Bb]', '-1', 'N/A']
value = [1000,1000000,1000000000, 1, 1]
mask = df.assign(
f=df['Market Cap'].replace(to_replace, value, regex=True),
Marketcap=pd.to_numeric(df['Market Cap'].str.replace(r'[^\d\.]', ''), errors='coerce')
).eval("Marketcap * f < 35000000")
df[mask]
In [178]: df[mask]
Out[178]:
0 1 2 3 Market Cap
2 -0.774057 -0.165566 -0.083345 0.741598 -0.139851 1.1M
3 -0.630724 0.250737 1.308556 -1.040799 1.064456 30.92M
4 2.029370 0.899612 0.261146 1.474148 -1.663970 476.74k
5 2.029370 0.899612 0.261146 1.474148 -1.663970 -1
PS如果要在结果数据集更改中保留非数值(如N/A
):
pd.to_numeric(df['Market Cap'].str.replace(r'[^\d\.]', ''), errors='coerce')
到
为什么启用了
regex=True
?如果启用了regex=True
,则会遇到此错误<代码>“PandaSexpisitor”对象没有属性“visit\u省略号”。如果我关闭它,我会遇到一个不同的错误,我把它作为一个图像发布在OP中。关于错误有什么想法吗?@moondra,你的熊猫版本是什么?我跑了这行,只找到了这两个;k
我负责在代码中加入“k”。我会更新OP以便你能清楚地看到一切。好的,我把它作为bug提交了。希望他们能看看。谢谢你的帮助谢谢。我刚刚意识到我的数据帧中也有一个k
(1000个),所以我在OP中更新了数据帧,以反映这一点。您是否能够更新代码以反映这一点?非常感谢。moondra-@MaxU的解决方案比我的要干净得多,我认为没有必要再改造他的轮子。谢谢!今天晚些时候我会看一看,因为它看起来有点复杂,需要一些时间。顺便说一句,要获得这些外观整洁的输出单元格(out[178]等),您是否通过命令行在Ipython中执行所有操作,并只复制单元格?我试着复制Jupyter笔记本的输出单元格,但当我把它粘贴到这里时,它非常不整洁。@moondra,是的,对不起,我更喜欢iPython,因为我是一个控制台管理员;-)嗨,Max,我有一个关于掩码
部分代码的问题;df.assign中的第一个f
创建一个新列,对吗?第二部分,Marketcap=pd.to_numeric
也在创建一个新列?我理解那部分有点困难。谢谢@moondra,是的,f
-是一个新的列(因子:1、1000、1000000等),Marketcap
是市值的干净数字表示形式