Python 如何使这段代码更高效（KS测试）
我的数据是这样的Python 如何使这段代码更高效（KS测试）,python,pandas,csv,pyspark,kolmogorov-smirnov,Python,Pandas,Csv,Pyspark,Kolmogorov Smirnov,我的数据是这样的 id1,id2,similarity CHEMBL1,CHEMBL1,1 CHEMBL2,CHEMBL1,0.18 CHEMBL3,CHEMBL1,0.56 CHEMBL4,CHEMBL1,0.64 CHEMBL5,CHEMBL1,0.12 CHEMBL1,CHEMBL2,0.18 CHEMBL2,CHEMBL2,1 CHEMBL3,CHEMBL2,0.26 CHEMBL4,CHEMBL2,0.78 CHEMBL5,CHEMBL2,0.33 CHEMBL1,CHEMBL3,0.
id1,id2,similarity
CHEMBL1,CHEMBL1,1
CHEMBL2,CHEMBL1,0.18
CHEMBL3,CHEMBL1,0.56
CHEMBL4,CHEMBL1,0.64
CHEMBL5,CHEMBL1,0.12
CHEMBL1,CHEMBL2,0.18
CHEMBL2,CHEMBL2,1
CHEMBL3,CHEMBL2,0.26
CHEMBL4,CHEMBL2,0.78
CHEMBL5,CHEMBL2,0.33
CHEMBL1,CHEMBL3,0.56
CHEMBL2,CHEMBL3,0.26
CHEMBL3,CHEMBL3,1
CHEMBL4,CHEMBL3,0.04
CHEMBL5,CHEMBL3,0.85
CHEMBL1,CHEMBL4,0.64
CHEMBL2,CHEMBL4,0.78
CHEMBL3,CHEMBL4,0.04
CHEMBL4,CHEMBL4,1
CHEMBL5,CHEMBL4,0.49
CHEMBL1,CHEMBL5,12
CHEMBL2,CHEMBL5,0.33
CHEMBL3,CHEMBL5,0.85
CHEMBL4,CHEMBL5,0.49
CHEMBL5,CHEMBL5,1
import pandas as pd
from scipy.stats import ks_2samp
import re
with open('example.csv', 'r') as f, open('Metrics.tsv', 'a') as f_out:
f_out.write('compound_1' + '\t' + 'compound_2' + '\t' + 'Similarity' + '\t' + 'KS Distance' + '\n')
df = pd.read_csv(f, delimiter = ',', lineterminator = '\n', header = None)
d = {}
l_id1 = []
l_id2 = []
l_sim = []
uniq_comps = df.iloc[:, 0].unique().tolist()
for i in uniq_comps:
d[i] = []
for j in range(df.shape[0]):
d[df.iloc[j, 0]].append(df.iloc[j, 2])
l_id1.append(df.iloc[j, 0])
l_id2.append(df.iloc[j, 1])
l_sim.append(df.iloc[j, 2])
for k in range(len(l_id1)):
sim = round(l_sim[k]*100, 0)/100
ks = re.findall(r"statistic=(.*)\,.*$", str(ks_2samp(d[l_id1[k]], d[l_id2[k]])))
f_out.write(l_id1[k] + '\t' + l_id2[k] + '\t' + str(sim) + '\t' + str(''.join(ks)) + '\n')
compound_1,compound_2,Similarity,KS Distance
CHEMBL1,CHEMBL1,1.0,0.0
CHEMBL2,CHEMBL1,0.18,0.4
CHEMBL3,CHEMBL1,0.56,0.2
CHEMBL4,CHEMBL1,0.64,0.2
CHEMBL5,CHEMBL1,0.12,0.4
CHEMBL1,CHEMBL2,0.18,0.4
CHEMBL2,CHEMBL2,1.0,0.0
CHEMBL3,CHEMBL2,0.26,0.2
CHEMBL4,CHEMBL2,0.78,0.4
CHEMBL5,CHEMBL2,0.33,0.2
CHEMBL1,CHEMBL3,0.56,0.2
CHEMBL2,CHEMBL3,0.26,0.2
CHEMBL3,CHEMBL3,1.0,0.0
CHEMBL4,CHEMBL3,0.04,0.2
CHEMBL5,CHEMBL3,0.85,0.2
CHEMBL1,CHEMBL4,0.64,0.2
CHEMBL2,CHEMBL4,0.78,0.4
CHEMBL3,CHEMBL4,0.04,0.2
CHEMBL4,CHEMBL4,1.0,0.0
CHEMBL5,CHEMBL4,0.49,0.2
CHEMBL1,CHEMBL5,12.0,0.4
CHEMBL2,CHEMBL5,0.33,0.2
CHEMBL3,CHEMBL5,0.85,0.2
CHEMBL4,CHEMBL5,0.49,0.2
CHEMBL5,CHEMBL5,1.0,0.0
- 在pandas中打开CSV时，您已经在RAM中加载了所有数据，因此不需要将该数据的副本放入列表中（例如
，l_id1
，等等）。尽可能避免使用多个数据副本，这会降低性能并使代码更难调试l_id2 - 在处理Pandas DataFrame时，尽量避免编写显式循环，应该有一种方法可以为您执行此操作，例如
groupby - Scipy statistic包返回一个结果对象，该对象总是公开
和statistic
成员，使用它而不是强制转换为字符串，然后使用正则表达式提取值pvalue - 避免对昂贵的函数进行不必要的调用，在最后一个循环中，您将为两个样本KS测试计算相同的数量，而不是每次计算一次，然后将结果与数据集合并
8 bytes*197e6 rows/1024**3 ~ 1.47 Gb
id1import itertools
import numpy as np
import pandas as pd
from scipy import stats
N = 10**6
df = pd.DataFrame({
"id1": np.random.choice([f"CHEMBL{i:d}" for i in np.arange(1, 6)], N),
"id2": np.random.choice([f"CHEMBL{i:d}" for i in np.arange(1, 6)], N),
"value": np.random.uniform(0, 12, N)
})
id1 id2 value
0 CHEMBL4 CHEMBL3 10.719870
1 CHEMBL2 CHEMBL5 2.911339
2 CHEMBL4 CHEMBL4 0.001595
3 CHEMBL2 CHEMBL3 0.148120
4 CHEMBL5 CHEMBL2 4.683689
def apply_test(df, idkey="id", valuekey="value", test=stats.ks_2samp):
"""
Apply statistical test to each possible pair of identifier
"""
# Group by identifier:
g = df.groupby(idkey)
# Generate all 2-combination of identifier:
for k1, k2 in itertools.combinations(g.groups.keys(), 2):
# Apply Statistical Test to grouped data:
t = test(df.loc[g.groups[k1],valuekey], df.loc[g.groups[k2],valuekey])
# Store Identifier pair:
res = {"id1": k1, "id2": k2}
# Store statistics and p-value:
res.update({k: getattr(t, k) for k in t._fields})
# Yield result:
yield res
r = pd.DataFrame([x for x in apply_test(df)])
df.merge(r)
id1 id2 value statistic pvalue
0 CHEMBL2 CHEMBL5 2.911339 0.003035 0.315677
1 CHEMBL2 CHEMBL5 6.583948 0.003035 0.315677
2 CHEMBL2 CHEMBL5 10.237092 0.003035 0.315677
3 CHEMBL2 CHEMBL5 8.049175 0.003035 0.315677
4 CHEMBL2 CHEMBL5 3.977925 0.003035 0.315677
... ... ... ... ... ...
400776 CHEMBL4 CHEMBL5 4.339528 0.002661 0.479805
400777 CHEMBL4 CHEMBL5 5.353133 0.002661 0.479805
400778 CHEMBL4 CHEMBL5 10.599985 0.002661 0.479805
400779 CHEMBL4 CHEMBL5 9.701375 0.002661 0.479805
400780 CHEMBL4 CHEMBL5 7.951454 0.002661 0.479805
id1 id2 statistic pvalue
0 CHEMBL1 CHEMBL2 0.002312 0.657859
1 CHEMBL1 CHEMBL3 0.002125 0.756018
2 CHEMBL1 CHEMBL4 0.001701 0.934290
3 CHEMBL1 CHEMBL5 0.002560 0.527594
4 CHEMBL2 CHEMBL3 0.002155 0.741524
5 CHEMBL2 CHEMBL4 0.001766 0.914602
6 CHEMBL2 CHEMBL5 0.003035 0.315677
7 CHEMBL3 CHEMBL4 0.001668 0.944053
8 CHEMBL3 CHEMBL5 0.002603 0.507482
9 CHEMBL4 CHEMBL5 0.002661 0.479805
r = pd.DataFrame([x for x in apply_test(df)])
df.merge(r)
id1 id2 value statistic pvalue
0 CHEMBL2 CHEMBL5 2.911339 0.003035 0.315677
1 CHEMBL2 CHEMBL5 6.583948 0.003035 0.315677
2 CHEMBL2 CHEMBL5 10.237092 0.003035 0.315677
3 CHEMBL2 CHEMBL5 8.049175 0.003035 0.315677
4 CHEMBL2 CHEMBL5 3.977925 0.003035 0.315677
... ... ... ... ... ...
400776 CHEMBL4 CHEMBL5 4.339528 0.002661 0.479805
400777 CHEMBL4 CHEMBL5 5.353133 0.002661 0.479805
400778 CHEMBL4 CHEMBL5 10.599985 0.002661 0.479805
400779 CHEMBL4 CHEMBL5 9.701375 0.002661 0.479805
400780 CHEMBL4 CHEMBL5 7.951454 0.002661 0.479805
我投票决定结束这个问题，因为它应该被问到。你能把每个文件的几行都贴出来看看格式吗。为了减少数据量，一个选项是生成直方图或ECDF以限制内存中的de大小。@jlandercy我的数据图像文件不可见吗？数据图像是非常糟糕的数据通信方式。您应该复制粘贴可重用代码，使您的问题符合SO标准。你也可以通过阅读来了解更多。是的，拥有文件的结构很有趣，因为您正在运行正则表达式。@MarcinOrlowski虽然这可能是关于CR的主题，但在将来，请不要以代码审阅站点的存在作为结束问题的理由。评估请求并使用一个原因，比如需求焦点（正如我在这里所做的），主要是基于意见的，等等。然后你可以向OP提到，如果需要，它可以发布在代码审查上。请看你好，谢谢你的回答。是的，你正确地理解了我。我想比较每一对的相似性分布，并将它们与该对的相似性值进行比较。因此，最好在输出文件中具有相似性值。我应该在你的函数中的什么地方添加这个？在更新res之前，只需更改为
valuekey=“similarity”