Python 为什么这个脚本要运行这么长时间?
我有一个包含200000行的csv文件。我已将其加载到数据帧中,并希望使用faker使用以下脚本对其进行匿名化:Python 为什么这个脚本要运行这么长时间?,python,python-3.x,pandas,jupyter-notebook,faker,Python,Python 3.x,Pandas,Jupyter Notebook,Faker,我有一个包含200000行的csv文件。我已将其加载到数据帧中,并希望使用faker使用以下脚本对其进行匿名化: for i in range(MasterDE1.FirstName.size): MasterDE1.loc[(MasterDE1["Gender__pc"] == 'Female'), ['FirstName','LastName']] = fake.first_name_female(),fake.last_name_female() MasterDE1.loc
for i in range(MasterDE1.FirstName.size):
MasterDE1.loc[(MasterDE1["Gender__pc"] == 'Female'), ['FirstName','LastName']] = fake.first_name_female(),fake.last_name_female()
MasterDE1.loc[(MasterDE1["Gender__pc"] == 'Male'), ['FirstName','LastName']] = fake.first_name_male(),fake.last_name_male()
MasterDE1.loc[(MasterDE1["Gender__pc"] == 'Unknown'), ['FirstName','LastName']] = fake.first_name(),fake.last_name()
MasterDE1['Name'] = MasterDE1['FirstName'] + ' ' + MasterDE1['LastName']
MasterDE1['EmailAddress'] = 'smithandthunder' + str(i+1) + '@gmail.com'
它已经运行了20分钟(我不认为内核已经死了) 您可以省略循环:
MasterDE1 = pd.DataFrame({'Gender__pc':['Female','Male','Unknown'],
'FirstName':['s','d','f'],
'LastName': ['d','f','r']})
MasterDE1 = pd.concat([MasterDE1]*3).reset_index(drop=True)
print (MasterDE1)
FirstName Gender__pc LastName
0 s Female d
1 d Male f
2 f Unknown r
3 s Female d
4 d Male f
5 f Unknown r
6 s Female d
7 d Male f
8 f Unknown r
def f1():
return 'first_name_female' + str(np.random.randint(100))
def f2():
return 'last_name_female' + str(np.random.randint(100))
maskfem = (MasterDE1["Gender__pc"] == 'Female')
a = pd.Series(((np.arange(len(MasterDE1.index))) + 1).astype(str))
MasterDE1.loc[maskfem, 'FirstName'] = [f1() for x in np.arange(maskfem.sum())]
MasterDE1.loc[maskfem, 'LastName'] = [f2() for x in np.arange(maskfem.sum())]
MasterDE1['Name'] = MasterDE1['FirstName'] + ' ' + MasterDE1['LastName']
MasterDE1['EmailAddress'] = 'smithandthunder' + a + '@gmail.com'
我不知道要告诉你为什么要花那么长时间,但可能是因为文件的大小 但是,您可以找到一种方法来监视该循环,以了解它是否仍在工作:
signal = 0
for i in range(0,200000):
....
# something going on in the loop
....
# signal the loop
signal += 1
if signal == 50000 or signal == 100000 or signal == 150000:
print('It\'s still going!')
elif signal > 200000:
print('It\'s over 200000 already!')
break # or you can raise an error instead of break (raise RuntimeError)
不必在每次迭代中更新数据帧,您可以先生成名称,然后分配:
df = pd.DataFrame({'Gender': np.random.choice(['Female', 'Male', 'Unknown'], p=[0.45, 0.45, 0.1], size=2*10**5),
'First Name': np.nan, 'Last Name': np.nan})
df.head()
Out:
First Name Gender Last Name
0 NaN Female NaN
1 NaN Male NaN
2 NaN Female NaN
3 NaN Male NaN
4 NaN Male NaN
df.shape
Out: (200000, 3)
现在,以下内容应在几分钟内完成:
df.loc[df['Gender']=='Female', ('First Name', 'Last Name')] = [(fake.first_name_female(), fake.last_name_female()) for _ in range(df[df['Gender']=='Female'].shape[0])]
df.loc[df['Gender']=='Male', ('First Name', 'Last Name')] = [(fake.first_name_male(), fake.last_name_male()) for _ in range(df[df['Gender']=='Male'].shape[0])]
df.loc[df['Gender']=='Unknown', ('First Name', 'Last Name')] = [(fake.first_name(), fake.last_name()) for _ in range(df[df['Gender']=='Unknown'].shape[0])]
df.head()
Out:
First Name Gender Last Name
0 Ruth Female Moore
1 Christina Female Jones
2 Lindsey Female Davis
3 Aaron Unknown Watkins
4 Joshua Male Henry
在那之后,像
df['Name']=df['First Name']+'''+df['Last Name']]
这样的事情应该会很快完成。谢谢。当我尝试此操作时,会出现以下错误:TypeError:ufunc“add”不包含具有签名匹配类型dtype的循环(“我相信false.first\u name\u female()
(和其他)每次调用时都要生成新名称。因此需要一个循环或应用程序。是的@ayhan。我尝试过这种方法,它为整个表提供了相同的名称。我希望所有名称都不同。但最好的方法是使用change函数返回假数据数组,然后在np.arange(maskfem.sum())中循环[f1()]不是必需的。这不符合我的使用案例,因为它是假的。first\u name\u female不是字符串,但我从Faker库中找到了一个方法,生成了一个真正的女性姓氏,而不是字符串。不要介意否定点,但希望得到评论,以便我可以改进未来的问题。非常感谢!谢谢。非常有用为了将来的目的
df.loc[df['Gender']=='Female', ('First Name', 'Last Name')] = [(fake.first_name_female(), fake.last_name_female()) for _ in range(df[df['Gender']=='Female'].shape[0])]
df.loc[df['Gender']=='Male', ('First Name', 'Last Name')] = [(fake.first_name_male(), fake.last_name_male()) for _ in range(df[df['Gender']=='Male'].shape[0])]
df.loc[df['Gender']=='Unknown', ('First Name', 'Last Name')] = [(fake.first_name(), fake.last_name()) for _ in range(df[df['Gender']=='Unknown'].shape[0])]
df.head()
Out:
First Name Gender Last Name
0 Ruth Female Moore
1 Christina Female Jones
2 Lindsey Female Davis
3 Aaron Unknown Watkins
4 Joshua Male Henry