Python 在scikit学习中获得二元概率分类器的最大精度
scikit learn中是否有任何内置函数可以获得二进制概率分类器的最大精度 例如,要获得F1的最高分数,我需要:Python 在scikit学习中获得二元概率分类器的最大精度,python,machine-learning,scikit-learn,classification,Python,Machine Learning,Scikit Learn,Classification,scikit learn中是否有任何内置函数可以获得二进制概率分类器的最大精度 例如,要获得F1的最高分数,我需要: # AUCPR precision, recall, thresholds = sklearn.metrics.precision_recall_curve(y_true, y_score) auprc = sklearn.metrics.auc(recall, precision) max_f1 = 0 for r, p, t in zip(recall, preci
# AUCPR
precision, recall, thresholds = sklearn.metrics.precision_recall_curve(y_true, y_score)
auprc = sklearn.metrics.auc(recall, precision)
max_f1 = 0
for r, p, t in zip(recall, precision, thresholds):
if p + r == 0: continue
if (2*p*r)/(p + r) > max_f1:
max_f1 = (2*p*r)/(p + r)
max_f1_threshold = t
我可以用类似的方式计算最大精度:
accuracies = []
thresholds = np.arange(0,1,0.1)
for threshold in thresholds:
y_pred = np.greater(y_score, threshold).astype(int)
accuracy = sklearn.metrics.accuracy_score(y_true, y_pred)
accuracies.append(accuracy)
accuracies = np.array(accuracies)
max_accuracy = accuracies.max()
max_accuracy_threshold = thresholds[accuracies.argmax()]
但我不知道是否有任何内置函数。我开始改进解决方案,将
阈值=np.arange(0,1,0.1)
转换为一种更智能、二分法的方法来寻找最大值
然后我意识到,经过两个小时的工作后,获得所有的准确度比仅仅找到最大值要便宜得多!!(是的,这完全违反直觉)
我在下面写了很多评论来解释我的代码。请随意删除所有这些内容,以使代码更具可读性
import numpy as np
# Definition : we predict True if y_score > threshold
def ROC_curve_data(y_true, y_score):
y_true = np.asarray(y_true, dtype=np.bool_)
y_score = np.asarray(y_score, dtype=np.float_)
assert(y_score.size == y_true.size)
order = np.argsort(y_score) # Just ordering stuffs
y_true = y_true[order]
# The thresholds to consider are just the values of score, and 0 (accept everything)
thresholds = np.insert(y_score[order],0,0)
TP = [sum(y_true)] # Number of True Positives (For Threshold = 0 => We accept everything => TP[0] = # of postive in true y)
FP = [sum(~y_true)] # Number of True Positives (For Threshold = 0 => We accept everything => TP[0] = # of postive in true y)
TN = [0] # Number of True Negatives (For Threshold = 0 => We accept everything => we don't have negatives !)
FN = [0] # Number of True Negatives (For Threshold = 0 => We accept everything => we don't have negatives !)
for i in range(1, thresholds.size) : # "-1" because the last threshold
# At this step, we stop predicting y_score[i-1] as True, but as False.... what y_true value say about it ?
# if y_true was True, that step was a mistake !
TP.append(TP[-1] - int(y_true[i-1]))
FN.append(FN[-1] + int(y_true[i-1]))
# if y_true was False, that step was good !
FP.append(FP[-1] - int(~y_true[i-1]))
TN.append(TN[-1] + int(~y_true[i-1]))
TP = np.asarray(TP, dtype=np.int_)
FP = np.asarray(FP, dtype=np.int_)
TN = np.asarray(TN, dtype=np.int_)
FN = np.asarray(FN, dtype=np.int_)
accuracy = (TP + TN) / (TP + FP + TN + FN)
sensitivity = TP / (TP + FN)
specificity = TN / (FP + TN)
return((thresholds, TP, FP, TN, FN))
整个过程只是一个循环,算法也很简单。
事实上,这个愚蠢的简单函数比我之前提出的解决方案快10倍(计算阈值的精度=np.arange(0,1,0.1)
),比我以前的智能ass动态算法快30倍
然后,您可以轻松计算所需的任何KPI,例如:
def max_accuracy(thresholds, TP, FP, TN, FN) :
accuracy = (TP + TN) / (TP + FP + TN + FN)
return(max(accuracy))
def max_min_sensitivity_specificity(thresholds, TP, FP, TN, FN) :
sensitivity = TP / (TP + FN)
specificity = TN / (FP + TN)
return(max(np.minimum(sensitivity, specificity)))
如果要测试它,请执行以下操作:
y_score = np.random.uniform(size = 100)
y_true = [np.random.binomial(1, p) for p in y_score]
data = ROC_curve_data(y_true, y_score)
%matplotlib inline # Because I personnaly use Jupyter, you can remove it otherwise
import matplotlib.pyplot as plt
plt.step(data[0], data[1])
plt.step(data[0], data[2])
plt.step(data[0], data[3])
plt.step(data[0], data[4])
plt.show()
print("Max accuracy is", max_accuracy(*data))
print("Max of Min(Sensitivity, Specificity) is", max_min_sensitivity_specificity(*data))
享受;) 你好,弗兰克,你找到它的内置函数了吗,因为我现在正在搜索它。@GeorgeSolymosi我没有找到它的内置函数。谢谢你提供的信息,请注意,行
accurity=np.array(accurity)
应该改为accurity=np.array(accuries)
或类似内容:)@GeorgeSolymosi感谢您的精彩捕捉!对了,好的,清晰的,透明的代码弗兰克!这样做的缺点是,特别是对于不平衡的数据集,分数的大部分变化可能位于第一个或最后一个bin中。更好的方法是计算每个唯一值(tp,fp,fn,tn)的阈值tp,fp,fn,tn。这可以在一次过程中高效完成(scikit在计算AUCROC时在内部执行此操作。)
from sklearn.metrics import accuracy_score
from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y_true, probs)
accuracy_scores = []
for thresh in thresholds:
accuracy_scores.append(accuracy_score(y_true, [m > thresh for m in probs]))
accuracies = np.array(accuracy_scores)
max_accuracy = accuracies.max()
max_accuracy_threshold = thresholds[accuracies.argmax()]