Python 如何处理大熊猫中的复制警告设置 背景
我刚刚把我的熊猫从0.11升级到了0.13.0rc1。现在,该应用程序正在弹出许多新的警告。其中一个是这样的:Python 如何处理大熊猫中的复制警告设置 背景,python,pandas,dataframe,chained-assignment,Python,Pandas,Dataframe,Chained Assignment,我刚刚把我的熊猫从0.11升级到了0.13.0rc1。现在,该应用程序正在弹出许多新的警告。其中一个是这样的: E:\FinReporter\FM_EXT.py:449: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_index,col_indexer] = value instead quote_df['TVol']
E:\FinReporter\FM_EXT.py:449: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
quote_df['TVol'] = quote_df['TVol']/TVOL_SCALE
quote_df = quote_df.ix[:,[0,3,2,1,4,5,8,9,30,31]]
我想知道它到底是什么意思?我需要改变什么吗
如果我坚持使用quote_df['TVol']=quote_df['TVol']/TVol\u SCALE
,我应该如何暂停警告
给出错误的函数
更多错误消息
创建
设置WithCopyWarning
是为了标记可能会混淆的“链接”分配,例如以下分配,但这些分配并不总是按预期工作,尤其是在第一个选择返回副本时。[背景讨论见和。]
df[df['A'] > 2]['B'] = new_val # new_val not set in df
警告建议重写如下:
df.loc[df['A'] > 2, 'B'] = new_val
# some code here
with ChainedAssignent():
df2['A'] /= 2
# more code follows
但是,这不适合您的使用,这相当于:
df = df[df['A'] > 2]
df['B'] = new_val
很明显,您不关心写操作返回到原始帧(因为您正在覆盖对它的引用),但不幸的是,此模式无法与第一个链式赋值示例区分开来。因此出现了(假阳性)警告。如果您想进一步阅读,可能出现误报的问题将在中讨论。您可以通过以下分配安全地禁用此新警告
import pandas as pd
pd.options.mode.chained_assignment = None # default='warn'
df # Some DataFrame
df = df.loc[:, 0:2] # Some filtering (unsure whether a view or copy is returned)
df = df.copy() # Ensuring a copy is made
df[df["Name"] == "John"] = "Johny" # Assignment can be done now (no warning)
其他资源
设置CopyWarning
的目的是向用户(尤其是新用户)显示,他们可能正在操作副本,而不是他们认为的原始版本。有误报(如果你知道你在做什么,那就可以了)。一种可能是按照@Garrett的建议关闭(默认警告)警告
这是另一个选择:
In [1]: df = DataFrame(np.random.randn(5, 2), columns=list('AB'))
In [2]: dfa = df.ix[:, [1, 0]]
In [3]: dfa.is_copy
Out[3]: True
In [4]: dfa['A'] /= 2
/usr/local/bin/ipython:1: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
#!/usr/local/bin/python
您可以将is_copy
标志设置为False
,这将有效地关闭该对象的检查:
如果显式复制,则不会发生进一步的警告:
In [7]: dfa = df.ix[:, [1, 0]].copy()
In [8]: dfa['A'] /= 2
OP上面显示的代码虽然是合法的,可能我也会这么做,但从技术上讲,它是针对这个警告的,而不是误报。另一种不出现警告的方法是通过reindex
执行选择操作,例如:
quote_df = quote_df.reindex(columns=['STK', ...])
或者
数据帧复制警告
当你去做这样的事情时:
E:\FinReporter\FM_EXT.py:449: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
quote_df['TVol'] = quote_df['TVol']/TVOL_SCALE
quote_df = quote_df.ix[:,[0,3,2,1,4,5,8,9,30,31]]
pandas.ix
在本例中,返回一个新的独立数据帧
您决定在此数据帧中更改的任何值都不会更改原始数据帧
这就是熊猫试图警告你的
为什么
.ix
是个坏主意
.ix
对象尝试做不止一件事,对于任何阅读过干净代码的人来说,这是一种强烈的气味
给定此数据帧:
df = pd.DataFrame({"a": [1,2,3,4], "b": [1,1,2,2]})
两种行为:
dfcopy = df.ix[:,["a"]]
dfcopy.a.ix[0] = 2
行为一:dfcopy
现在是一个独立的数据帧。更改它不会更改df
df.ix[0, "a"] = 3
行为二:这将更改原始数据帧
改用
.loc
pandas开发人员认识到.ix
对象[推测性地]很臭,因此创建了两个新对象,这有助于数据的存取和分配。(另一个是.iloc
)
.loc
更快,因为它不尝试创建数据的副本
.loc
旨在修改现有的数据帧,这将提高内存效率
.loc
是可预测的,它有一个行为
解决方案 在代码示例中,您正在加载一个包含许多列的大文件,然后将其修改为更小的文件
pd.read\u csv
功能可以帮助您解决很多问题,还可以加快文件的加载速度
因此,与其这样做
quote_df = pd.read_csv(StringIO(str_of_all), sep=',', names=list('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefg')) #dtype={'A': object, 'B': object, 'C': np.float64}
quote_df.rename(columns={'A':'STK', 'B':'TOpen', 'C':'TPCLOSE', 'D':'TPrice', 'E':'THigh', 'F':'TLow', 'I':'TVol', 'J':'TAmt', 'e':'TDate', 'f':'TTime'}, inplace=True)
quote_df = quote_df.ix[:,[0,3,2,1,4,5,8,9,30,31]]
这样做
columns = ['STK', 'TPrice', 'TPCLOSE', 'TOpen', 'THigh', 'TLow', 'TVol', 'TAmt', 'TDate', 'TTime']
df = pd.read_csv(StringIO(str_of_all), sep=',', usecols=[0,3,2,1,4,5,8,9,30,31])
df.columns = columns
这将只读取您感兴趣的列,并正确命名它们。无需使用邪恶的
.ix
对象来做神奇的事情。如果您已将切片分配给变量,并希望使用变量进行设置,如下所示:
df2 = df[df['A'] > 2]
df2['B'] = value
df.loc[df2.index.tolist(), 'B'] = value
您不想使用Jeffs解决方案,因为您的条件计算df2
太长或其他原因,那么您可以使用以下方法:
df2 = df[df['A'] > 2]
df2['B'] = value
df.loc[df2.index.tolist(), 'B'] = value
df2.index.tolist()
返回df2中所有条目的索引,然后用于在原始数据帧中设置B列。为了消除任何疑问,我的解决方案是制作切片的深度副本,而不是常规副本。
这可能不适用,具体取决于您的上下文(内存限制/片的大小、性能下降的可能性-尤其是如果复制发生在循环中,就像对我一样,等等)
要明确的是,以下是我收到的警告:
/opt/anaconda3/lib/python3.6/site-packages/ipykernel/__main__.py:54:
SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
插图
我怀疑这个警告是因为我把一个专栏放在了这个片段的副本上。从技术上讲,虽然没有尝试在切片副本中设置值,但这仍然是对切片副本的修改。
以下是我为证实这一怀疑而采取的(简化)步骤,我希望这将有助于我们这些试图理解这一警告的人
>> data1 = {'A': [111, 112, 113], 'B':[121, 122, 123]}
>> df1 = pd.DataFrame(data1)
>> df1
A B
0 111 121
1 112 122
2 113 123
>> df2 = df1
>> df2
A B
0 111 121
1 112 122
2 113 123
# Dropping a column on df2 can affect df1
# No slice involved here, but I believe the principle remains the same?
# Let me know if not
>> df2.drop('A', axis=1, inplace=True)
>> df1
B
0 121
1 122
2 123
示例1:在原始列上删除列会影响副本
我们已经知道了,但这是一个健康的提醒。这不是警告的内容
>> data1 = {'A': [111, 112, 113], 'B':[121, 122, 123]}
>> df1 = pd.DataFrame(data1)
>> df1
A B
0 111 121
1 112 122
2 113 123
>> df2 = df1
>> df2
A B
0 111 121
1 112 122
2 113 123
# Dropping a column on df1 affects df2
>> df1.drop('A', axis=1, inplace=True)
>> df2
B
0 121
1 122
2 123
可以避免对df1所做的更改影响df2。注意:您可以通过执行df.copy()
来避免导入copy.deepcopy
>> data1 = {'A': [111, 112, 113], 'B':[121, 122, 123]}
>> df1 = pd.DataFrame(data1)
>> df1
A B
0 111 121
1 112 122
2 113 123
>> import copy
>> df2 = copy.deepcopy(df1)
>> df2
A B
0 111 121
1 112 122
2 113 123
# Dropping a column on df1 does not affect df2
>> df1.drop('A', axis=1, inplace=True)
>> df2
A B
0 111 121
1 112 122
2 113 123
示例2:在副本上删除列可能会影响原始
这实际上说明了这一警告
>> data1 = {'A': [111, 112, 113], 'B':[121, 122, 123]}
>> df1 = pd.DataFrame(data1)
>> df1
A B
0 111 121
1 112 122
2 113 123
>> df2 = df1
>> df2
A B
0 111 121
1 112 122
2 113 123
# Dropping a column on df2 can affect df1
# No slice involved here, but I believe the principle remains the same?
# Let me know if not
>> df2.drop('A', axis=1, inplace=True)
>> df1
B
0 121
1 122
2 123
可以避免对df2所做的更改影响df1
>> data1 = {'A': [111, 112, 113], 'B':[121, 122, 123]}
>> df1 = pd.DataFrame(data1)
>> df1
A B
0 111 121
1 112 122
2 113 123
>> import copy
>> df2 = copy.deepcopy(df1)
>> df2
A B
0 111 121
1 112 122
2 113 123
>> df2.drop('A', axis=1, inplace=True)
>> df1
A B
0 111 121
1 112 122
2 113 123
干杯 我相信,你可以避免这样的问题:
return (
pd.read_csv(StringIO(str_of_all), sep=',', names=list('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefg')) #dtype={'A': object, 'B': object, 'C': np.float64}
.rename(columns={'A':'STK', 'B':'TOpen', 'C':'TPCLOSE', 'D':'TPrice', 'E':'THigh', 'F':'TLow', 'I':'TVol', 'J':'TAmt', 'e':'TDate', 'f':'TTime'}, inplace=True)
.ix[:,[0,3,2,1,4,5,8,9,30,31]]
.assign(
TClose=lambda df: df['TPrice'],
RT=lambda df: 100 * (df['TPrice']/quote_df['TPCLOSE'] - 1),
TVol=lambda df: df['TVol']/TVOL_SCALE,
TAmt=lambda df: df['TAmt']/TAMT_SCALE,
STK_ID=lambda df: df['STK'].str.slice(13,19),
STK_Name=lambda df: df['STK'].str.slice(21,30)#.decode('gb2312'),
TDate=lambda df: df.TDate.map(lambda x: x[0:4]+x[5:7]+x[8:10]),
)
)
使用分配。从:将新列分配给数据帧,返回一个新对象(副本),其中包含除新列之外的所有原始列
参见Tom Augspurger关于pandas中的方法链接的文章:对于我来说,这个问题发生在以下>简化
np.random.seed(0)
df = pd.DataFrame(np.random.choice(10, (3, 5)), columns=list('ABCDE'))
df
A B C D E
0 5 0 3 3 7
1 9 3 5 2 4
2 7 6 8 8 1
df[df.A > 5]['B']
1 3
2 6
Name: B, dtype: int64
df.loc[df.A > 5, 'B']
1 3
2 6
Name: B, dtype: int64
df.loc[df.A > 5, 'B'] = 4
# becomes
df.__setitem__((df.A > 5, 'B'), 4)
df[df.A > 5]['B'] = 4
# becomes
df.__getitem__(df.A > 5).__setitem__('B", 4)
df.loc[df.A > 5, 'B'] = 4
df.iloc[(df.A > 5).values, 1] = 4
df.loc[1, 'A'] = 100
df.iloc[1, 0] = 100
df2 = df[['A']]
df2['A'] /= 2
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/IPython/__main__.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
df2
A
0 2.5
1 4.5
2 3.5
df2 = df.loc[:, ['A']]
df2['A'] /= 2 # Does not raise
pd.options.mode.chained_assignment = None
df2['A'] /= 2
df2 = df[['A']].copy(deep=True)
df2['A'] /= 2
class ChainedAssignent:
def __init__(self, chained=None):
acceptable = [None, 'warn', 'raise']
assert chained in acceptable, "chained must be in " + str(acceptable)
self.swcw = chained
def __enter__(self):
self.saved_swcw = pd.options.mode.chained_assignment
pd.options.mode.chained_assignment = self.swcw
return self
def __exit__(self, *args):
pd.options.mode.chained_assignment = self.saved_swcw
# some code here
with ChainedAssignent():
df2['A'] /= 2
# more code follows
with ChainedAssignent(chained='raise'):
df2['A'] /= 2
SettingWithCopyError:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
df
A B C D E
0 5 0 3 3 7
1 9 3 5 2 4
2 7 6 8 8 1
A B C D E
0 5 0 3 3 7
1 1000 3 5 2 4
2 1000 6 8 8 1
df.A[df.A > 5] = 1000 # works, because df.A returns a view
df[df.A > 5]['A'] = 1000 # does not work
df.loc[df.A 5]['A'] = 1000 # does not work
df.loc[df.A > 5, 'A'] = 1000
A B C D E
0 5 0 3 3 7
1 9 3 5 12345 4
2 7 6 8 8 1
df.loc[1, 'D'] = 12345
df.iloc[1, 3] = 12345
df.at[1, 'D'] = 12345
df.iat[1, 3] = 12345
A B C D E
1 9 3 5 2 4
2 7 6 8 8 1
df2.loc[df2.C == 5, 'D'] = 123
df2 = df[df.A > 5]
df2 = df[df.A > 5].copy()
# Or,
# df2 = df.loc[df.A > 5, :]
A B C D E
1 9 3 5 2 4
2 7 6 8 8 1
df2.drop('C', axis=1, inplace=True)
df2 = df[df.A > 5]
def plot(pdb, df, title, **kw):
df['target'] = (df['ogg'] + df['ugg']) / 2
# ...
df = data[data['anz_emw'] > 0]
pixbuf = plot(pdb, df, title)
df['target'] = ...
data[data['anz_emw'] > 0]['target'] = ...
def plot(pdb, df, title, **kw):
df.loc[:,'target'] = (df['ogg'] + df['ugg']) / 2
import pandas as pd
df = pd.DataFrame({'x':[1,2,3]})
df0 = df[df.x>2]
df0['foo'] = 'bar'
df1 = df[df.x>2].copy(deep=False)
df1['foo'] = 'bar'
import inspect
slice= df[df.x>2]
slice_copy = df[df.x>2].copy(deep=False)
inspect.getmembers(slice)
inspect.getmembers(slice_copy)
| | slice | slice_copy |
| _is_copy | weakref | None |
class SupressSettingWithCopyWarning:
def __enter__(self):
pd.options.mode.chained_assignment = None
def __exit__(self, *args):
pd.options.mode.chained_assignment = 'warn'
with SupressSettingWithCopyWarning():
#code that produces warning
df # Some DataFrame
df = df.loc[:, 0:2] # Some filtering (unsure whether a view or copy is returned)
df = df.copy() # Ensuring a copy is made
df[df["Name"] == "John"] = "Johny" # Assignment can be done now (no warning)
from contextlib import contextmanager
@contextmanager
def SuppressPandasWarning():
with pd.option_context("mode.chained_assignment", None):
yield
import pandas as pd
from string import ascii_letters
a = pd.DataFrame({"A": list(ascii_letters[0:4]), "B": range(0,4)})
mask = a["A"].isin(["c", "d"])
# Even shallow copy below is enough to not raise the warning, but why is a mystery to me.
b = a.loc[mask] # .copy(deep=False)
# Raises the `SettingWithCopyWarning`
b["B"] = b["B"] * 2
# Does not!
with SuppressPandasWarning():
b["B"] = b["B"] * 2
prop_df = df.query('column == "value"')
prop_df['new_column'] = prop_df.apply(function, axis=1)
prop_df = df.copy(deep=True)
prop_df = prop_df.query('column == "value"')
prop_df['new_column'] = prop_df.apply(function, axis=1)
prop_df = df.query('column == "value"').reset_index(drop=True)
prop_df['new_column'] = prop_df.apply(function, axis=1)