在python中查找与特定条件匹配的重复项
下面是我正在处理的示例数据在python中查找与特定条件匹配的重复项,python,python-2.7,pandas,Python,Python 2.7,Pandas,下面是我正在处理的示例数据 sender receiver date id salman akhtar 20161201 1111 akhtar salman 20161201 1112 nabeel ahmed 20161201 1113 salman akhtar 20161201 1114 salman akhtar 20161202 1115 nabeel ahmed 20161202 1116 ahmed
sender receiver date id
salman akhtar 20161201 1111
akhtar salman 20161201 1112
nabeel ahmed 20161201 1113
salman akhtar 20161201 1114
salman akhtar 20161202 1115
nabeel ahmed 20161202 1116
ahmed nabeel 20161202 1117
nabeel ahmed 20161202 1118
nabeel ahmed 20161202 1119
我试图实现的是根据条件查找重复条目,在同一日期内相同的发送者和接收者
为此,我编写了以下代码
import pandas as pd
import xlsxwriter
print 'Script for Finding duplicate entries\n'
path = raw_input('Enter file name: ')
print 'Loading file. Please wait...'
xlsx = pd.ExcelFile(path+'.xlsx')
print 'File loaded successfully.\n'
sheet = raw_input('Enter Sheet Name: ')
df = pd.read_excel(xlsx, sheet)
df['is_duplicated'] = df.duplicated(['sender', 'receiver','date'],keep=False)
df_dup = df.loc[df['is_duplicated'] == True]
print 'Found Below Duplicates'
print df_dup
writer = pd.ExcelWriter("pandas_column_formats.xlsx", engine='xlsxwriter')
df_dup.to_excel(writer, sheet_name='Sheet1')
writer.save()
print 'File created successfully.'
现在,我还想合并fuzzyfuzzy
,因为当前代码只返回精确的重复项,并且我希望根据上述条件得到所有可能的重复行
有人能帮忙吗?像这样的吗
>>> fuzz_ratio = 50
>>> df_rem = df[~df.duplicated(['sender', 'receiver','date'],keep=False)]
>>> df_possible_dup = pd.merge(df_rem, df, on='date', suffixes=['', '_j'])
>>> df_possible_dup.apply(lambda x: fuzz.ratio(x['sender'], x['sender_j']) >= 50 and x['id'] != x['id_j'], axis=1)
我不知道您的确切要求,但您可能想检查发送方或接收方是否完全匹配,其他部分是否可能匹配。然后,您可以使用自定义功能:
def worker(x, fuzz_ratio):
if x['id'] != x['id_j']:
return False
if x['sender'] == x['sender_j'] and fuzz.ratio(x['receiver'], x['receiver_j']) > fuzz_ratio:
return True
if x['receiver'] == x['receiver_j'] and fuzz.ratio(x['sender'], x['sender_j']) > fuzz_ratio:
return True
return False
>>> df_possible_dup.apply(lambda x: worker(x, fuzz_ratio))