在有监督的机器学习算法中，为考察model的generalization ability，我们需要对model进行评估。为防止过拟合，不可以使用训练集数据进行模型评估，此时可以采取cross validation方法：将原始数据分为训练集、测试机
常见cross validation：
- Handout validation 留出法
- K-Fold validation k折交叉验证
- Leave-one-out 留一法
一、我们先介绍handout：该方法为k-fold的特殊情况：
思想：
- 将原始数据分为训练集、测试集，常见比例为0.8:0.2
- 使用训练集训练模型
- 使用测试集来评估模型
import numpy as np
from sklearn.linear_model import LogisticRegression
np.random.seed(8)
shuffled_index = np.random.permutation(admissions.index)
shuffled_admissions = admissions.loc[shuffled_index]
train = shuffled_admissions.iloc[0:515]
test = shuffled_admissions.iloc[515:len(shuffled_admissions)]
#make regression model using training data.
model=LogisticRegression()
model.fit(train[['gpa']],train['actual_label'])
test['predicted_label']=model.predict(test[['gpa']])
accuracy=len(test[(test['predicted_label']==test['actual_label'])])/len(test)
print(accuracy)
ROC曲线：
import matplotlib.pyplot as plt
from sklearn import metrics
probabilities = model.predict_proba(test[["gpa"]])
fpr, tpr, thresholds = metrics.roc_curve(test["actual_label"], probabilities[:,1])
plt.plot(fpr, tpr)
AUC：
在实际应用中，我们更关心的时TPR，即正确预测到正例的比率。为了对该模型进行更有效的评估，可以计算roc曲线下部的面积，即AUC，当auc越接近1时，我们说该模型的效果越好。
from sklearn.metrics import roc_auc_score
auc_score=roc_auc_score(test['actual_label'],probabilities[:,1])
print(auc_score)
二、K-Fold
原理：将原始数据分为k分，k-1份作为训练集，剩下的1份作为测试集，一次迭代；
下面给出简单的源码：
分割数据：
import pandas as pd
admissions = pd.read_csv("admissions.csv")
admissions["actual_label"] = admissions["admit"]
admissions = admissions.drop("admit", axis=1)
shuffled_index = np.random.permutation(admissions.index)
shuffled_admissions = admissions.loc[shuffled_index]
admissions = shuffled_admissions.reset_index()
admissions.ix[0:128,'fold']=1 #将0-128 row作为第1次iteration
admissions.ix[129:257,'fold']=2
admissions.ix[258:386,'fold']=3
admissions.ix[387:514,'fold']=4
admissions.ix[515:644,'fold']=5
admissions['fold']=admissions['fold'].astype(int)
print(admissions.head())
print(admissions.tail())
生成模型，对每个模型进行评估，最后计算平均准确率accuracy：
import numpy as np
fold_ids = [1,2,3,4,5]
lr=LogisticRegression()
def train_and_test(admissions,fold_ids):
accuracies=[]
for i in fold_ids:
train_iteration=admissions[admissions['fold']!=i]
test_iteration=admissions[admissions['fold']==i]
# make model using training data set.
lr.fit(train_iteration[['gpa']],train_iteration['actual_label'])
#predicting test data set.
test_iteration['labels']=lr.predict(test_iteration[['gpa']])
iteration_accuracy=len(test_iteration[test_iteration['labels']==test_iteration['actual_label']])/len(test_iteration)
accuracies.append(iteration_accuracy)
return accuracies
accuracies=train_and_test(admissions,fold_ids)
print(accuracies)
average_accuracy=sum(accuracies)/len(fold_ids)
print(average_accuracy)
python中的sklearn库已经封装好了该方法，使用sklearn库完成：
from sklearn.cross_validation import KFold
from sklearn.cross_validation import cross_val_score
from sklearn.linear_model import LogisticRegression
admissions = pd.read_csv("admissions.csv")
admissions["actual_label"] = admissions["admit"]
admissions = admissions.drop("admit", axis=1)
kf=KFold(len(admissions),5,shuffle=True,random_state=8)
lr=LogisticRegression()
accuracies=cross_val_score(lr,admissions[['gpa']],admissions['actual_label'],scoring='accuracy',cv=kf)
average_accuracy=sum(accuracies)/len(accuracies)
print(accuracies)
print(average_accuracy)
注意：初始化KFold类并不会生成模型、预测数据，仅仅是对数据做了分组。
在进行线性/逻辑回归时常常使用交叉验证来评估模型。
如果是对于单变量模型--即特征只有一个，往往采用handout方法；
多余多变量模型--具有多个特征列，往往会采用kfold方法。
参考地址：https://zh.wikipedia.org/wiki/%E4%BA%A4%E5%8F%89%E9%A9%97%E8%AD%89
