Python sklearn中遗漏一项交叉验证的混淆矩阵

Python sklearn中遗漏一项交叉验证的混淆矩阵,python,machine-learning,scikit-learn,logistic-regression,cross-validation,Python,Machine Learning,Scikit Learn,Logistic Regression,Cross Validation,但我不知道在使用遗漏交叉验证时如何创建混淆矩阵: 我应该如何为LOOCV创建混淆矩阵,以便可视化每类的准确性?借用您的方法,您可以通过创建一个在迭代过程中接收元数据的自定义记分器来解决这个问题。这些元数据可用于查找:F1分数、准确度、召回率、准确度以及混淆矩阵 在这里,我们需要另一个技巧,是使用它接受一个自定义记分员,所以我们来了 以下是一个示例,您可以根据自己的绝对要求进行更多工作: import pandas as pd import numpy as np import matplot

但我不知道在使用遗漏交叉验证时如何创建混淆矩阵:


我应该如何为LOOCV创建混淆矩阵,以便可视化每类的准确性?

借用您的方法,您可以通过创建一个在迭代过程中接收元数据的自定义记分器来解决这个问题。这些元数据可用于查找:F1分数、准确度、召回率、准确度以及混淆矩阵


在这里,我们需要另一个技巧,是使用它接受一个自定义记分员,所以我们来了


以下是一个示例,您可以根据自己的绝对要求进行更多工作:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import make_scorer, accuracy_score, confusion_matrix
from sklearn.model_selection import GridSearchCV, StratifiedKFold


# Your method from the link you provided
def cm_analysis(y_true, y_pred, labels, ymap=None, figsize=(10,10)):
    if ymap is not None:
        y_pred = [ymap[yi] for yi in y_pred]
        y_true = [ymap[yi] for yi in y_true]
        labels = [ymap[yi] for yi in labels]
    cm = confusion_matrix(y_true, y_pred, labels=labels)
    cm_sum = np.sum(cm, axis=1, keepdims=True)
    cm_perc = cm / cm_sum.astype(float) * 100
    annot = np.empty_like(cm).astype(str)
    nrows, ncols = cm.shape
    for i in range(nrows):
        for j in range(ncols):
            c = cm[i, j]
            p = cm_perc[i, j]
            if i == j:
                s = cm_sum[i]
                annot[i, j] = '%.1f%%\n%d/%d' % (p, c, s)
            elif c == 0:
                annot[i, j] = ''
            else:
                annot[i, j] = '%.1f%%\n%d' % (p, c)
    cm = pd.DataFrame(cm, index=labels, columns=labels)
    cm.index.name = 'Actual'
    cm.columns.name = 'Predicted'
    fig, ax = plt.subplots(figsize=figsize)
    sns.heatmap(cm, annot=annot, fmt='', ax=ax)
    #plt.savefig(filename)
    plt.show()


# Custom Scorer
def my_scorer(y_true, y_pred):
    acc = accuracy_score(y_true, y_pred)
    # you can either save  y_true, y_pred and accuracy into a file
    # for later use with the info in clf.cv_results_
    # or plot the confusion matrix right here!
    # for labels, you can create a class attribute to make it more dynamic
    # i.e. changes automatically with every new dataset!
    cm_analysis(y_true, y_pred, labels=[0,1], ymap=None, figsize=(10, 10))
    # N.B as long as you have y_true and y_pred from every round here, you can
    # do with them all the metrics that want such as F1 Score, Precision, Recall, A
    # ccuracy and the Confusion Matrix!
    return acc


url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv"
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
df = pd.read_csv(url, names=names)
array = df.values
X = np.array(array[:,0:8])
Y = np.array(array[:,8]).astype(int)

# I'll make it two just for submitting the result here!
num_folds = 2
skf = StratifiedKFold(n_splits=num_folds, random_state=0)

# this is just a trick because the list contains 
# the default parameter only (i.e. useless)
param_grid = {'C': [1.0]}
model = LogisticRegression()
# create custom scorer
custom_scorer = make_scorer(my_scorer)
# pass it to the GridSearchCV
clf = GridSearchCV(model, param_grid, scoring=custom_scorer, cv=skf, return_train_score=True)
# Fit and Go
clf.fit(X,Y)

# cv_results_ is a dict with all CV results during the iterations!
# IDK, you may need it to combine its content with the metrics ..etc
print(clf.cv_results_)
后果 分割0

第1部分


编辑 如果您严格想要
LOOCV
,那么您可以将其应用于上述代码中,只需将
StratifiedKFold
替换为
LeaveOneOut
函数即可;但请记住,
LeaveOneOut
将循环684次!所以计算起来非常昂贵。但是,这将在迭代过程中为您提供详细的混淆矩阵(即元数据)

尽管如此,如果您正在寻找整个(即最终)流程的混淆矩阵,那么您仍然需要使用
GridSearchCV
,但如下所示:

......
loocv = LeaveOneOut()
clf = GridSearchCV(model, param_grid, scoring='accuracy', cv=loocv)
clf.fit(X,Y)

y_pred = clf.best_estimator_.predict(X)
cm_analysis(Y, y_pred, labels=[0, 1], ymap=None, figsize=(10,10))
结果

好吧,我已经提到,我正在寻找LOOCV的混淆矩阵。你能修改你的答案以便它回答问题吗?我看不到连接。你为什么不为所问的问题创建一个混乱矩阵呢?我不熟悉你说的话,而且答案中没有足够的解释来说明两者之间的联系
{'mean_score_time': array([0.09023476]), 'split0_train_score': 
 array([0.79166667]), 'mean_train_score': array([0.77864583]), 
'params': [{'C': 1.0}], 'std_test_score': array([0.01953125]), 
'mean_fit_time': array([0.00235796]), 
'param_C': masked_array(data=[1.0], mask=[False], fill_value='?',
dtype=object), 'rank_test_score': array([1], dtype=int32), 
'split1_test_score': array([0.7734375]), 
'std_fit_time': array([0.00032902]), 'mean_test_score': array([0.75390625]), 
'std_score_time': array([0.00237632]), 'split1_train_score': array([0.765625]), 
'split0_test_score': array([0.734375]), 'std_train_score': array([0.01302083])}
......
loocv = LeaveOneOut()
clf = GridSearchCV(model, param_grid, scoring='accuracy', cv=loocv)
clf.fit(X,Y)

y_pred = clf.best_estimator_.predict(X)
cm_analysis(Y, y_pred, labels=[0, 1], ymap=None, figsize=(10,10))