<jupyter_start><jupyter_text>二分类器从一个判断手写体是否是5简单二分类出发<jupyter_code># 调整数据的label为2class
y_train_5 = (y_train == 5)
y_test_5 = (y_test == 5)
from sklearn.linear_model import SGDClassifier

sgd_clf = SGDClassifier(random_state=42,max_iter=100, tol=None)

sgd_clf.fit(X_train, y_train_5)
sgd_clf.predict([some_digit])<jupyter_output><empty_output><jupyter_text>性能评估  手动完成交叉验证<jupyter_code>from sklearn.model_selection import StratifiedKFold
from sklearn.base import clone

skfolds = StratifiedKFold(n_splits=3, random_state=42)

for train_index, test_index in skfolds.split(X_train, y_train_5):
    clone_clf = clone(sgd_clf)
    X_train_folds = X_train  [train_index]
    y_train_folds = y_train_5[train_index]
    X_test_fold   = X_train  [test_index ]
    y_test_fold   = y_train_5[test_index ]
    
    clone_clf.fit(X_train_folds, y_train_folds)
    y_pred = clone_clf.predict(X_test_fold)
    n_correct = sum(y_pred == y_test_fold)
    
    print(n_correct / len(y_pred))
from sklearn.model_selection import cross_val_score

cross_val_score(sgd_clf, X_train, y_train_5, cv=3, scoring="accuracy")<jupyter_output><empty_output><jupyter_text>看起来预测出是5的准确性很高，但是注意我们评估的方法是拿 预测对的除以总数下面看一个*非5*二分器表现<jupyter_code>from sklearn.base import BaseEstimator

class Never5Classifier(BaseEstimator):
    def fit(self, X, y=None):
        pass
    def predict(self, X):
        return np.zeros((len(X), 1), dtype=bool)
never_5_clf = Never5Classifier()
cross_val_score(never_5_clf, X_train, y_train_5, cv=3, scoring="accuracy")<jupyter_output><empty_output><jupyter_text>可想而知我们都预测是false正确率也有90%, 原因在于只有一小部分正例，引入其他评估方法  混淆矩阵对于矩阵元素$(i,j)$ 表示$i$被识别称$j$的数量<jupyter_code>from sklearn.model_selection import cross_val_predict

y_train_pred = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3) # cross_val_predict输出具体预测的结果
y_train_pred<jupyter_output><empty_output><jupyter_text>通过比较y_train_5和y_train_pred计算混淆矩阵<jupyter_code>from sklearn.metrics import confusion_matrix

_confusion_matrix = confusion_matrix(y_train_5, y_train_pred) # confusion_matrix(真实label, 预测)
_confusion_matrix<jupyter_output><empty_output><jupyter_text>$(0,0)=54055$ 非$5$被识别成非$5$ 真反例 $true\ positives  $  $(0,1)=524$  非$5$被识别成5   假反例 $false\ positives  $  $(1,0)=1449$  $5$被识别成非$5$ 假反例 $false\ negatives  $  $(1,1)=3972$  $5$被识别成$5$   真正例 $true\ positives  $    更简洁的测度 1. 正例的精确度，考量的范围不再是全体样本$\displaystyle precosion = \frac{TP}{TP+FP}$  做一次正例的判断保证预测正确的概率  2. 被分类器正确预测为正例的比例  $\displaystyle recall = \frac{TP}{TP+FN}$  $\begin{bmatrix} TN & FP \\ FN & TP\end{bmatrix}$  Precision and Recall<jupyter_code>from sklearn.metrics import precision_score, recall_score

print(precision_score(y_train_5, y_train_pred))
print(_confusion_matrix[1][1] / (_confusion_matrix[1][1] + _confusion_matrix[0][1]))
print(recall_score(y_train_5, y_train_pred))
print(_confusion_matrix[1][1] / (_confusion_matrix[1][1] + _confusion_matrix[1][0]))<jupyter_output>0.6389964951116031
0.6389964951116031<jupyter_text>$F_1$量度定义为准确和查全率的调和平均数$\displaystyle F_1 = \frac{2}{\frac{1}{precision}+\frac{1}{recall}}=2 \times \frac{precision \times recall}{precision + recall}$<jupyter_code>from sklearn.metrics import f1_score

f1_score(y_train_5, y_train_pred)<jupyter_output><empty_output><jupyter_text>$F_1$量度综合考虑了准确度和查全率，但往往我们需要给他们分配权重，在不同的项目中，我们有自己的偏好。  但实情是当我们提高一方面会削减另一个方面。这称为$precision/recall\ tradeoff$ 分类器为其预测基于$decision\ function$打分，如果分数高于阈值归为正例，反之亦然。  选取不同的$decision\ threshold$会影响$precision$和$recall$得分    一般而言提高阈值随之提高了准确率，降低了召回率，反之，降低准确率，提高召回率。  sklearn不提供直接修改阈值，但我们可以通过调节`descision_function()`达到目的<jupyter_code>y_scores = sgd_clf.decision_function([some_digit])
y_scores
threshold = 0
y_some_digit_pred = (y_scores > threshold)
y_some_digit_pred
threshold = 20000
y_some_digit_pred = (y_scores > threshold)
y_some_digit_pred
y_scores = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3, method="decision_function")  # 输出具体的得分
y_scores<jupyter_output><empty_output><jupyter_text>基于以上的得分我们可以调整阈值达到不同的准确率和查全率<jupyter_code>from sklearn.metrics import precision_recall_curve

precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores) 
# 预测结果,交叉验证得分来 得到在不同阈值设置下查全率和查准率
# 对于不同的阈值，绘制precisions和recalls曲线
def plot_precision_recall_vs_threshold(precisions, recalls, thresholds):
    plt.plot(thresholds, precisions[:-1], "b--", label="Precision", linewidth=2) # 查准率
    plt.plot(thresholds, recalls[:-1],"g--", label="Recall", linewidth=2) # 查全率
    plt.xlabel("Threshold", fontsize=16)
    plt.ylim([0, 1])
    plt.legend(loc="upper left", fontsize=16)
plt.figure(figsize=(10, 4))
plt.xlim([-60000, 60000])
plot_precision_recall_vs_threshold(precisions, recalls, thresholds)
plt.show()
#### 试试直接绘制precision vs recall
def plot_precision_vs_recall(precisions, recalls):
    plt.plot(recalls, precisions, "b-", linewidth=2)
    plt.xlabel("Recall", fontsize=16)
    plt.ylabel("Precision", fontsize=16)
    plt.xlim([0,1])
    plt.ylim([0,1])
    #  plt.axis([0, 1, 0, 1]) # 合并为一句
plot_precision_vs_recall(precisions, recalls)
plt.show()
y_train_pred_90 = (y_scores > 5000)
precision_score(y_train_5, y_train_pred_90)
recall_score(y_train_5, y_train_pred_90)<jupyter_output><empty_output><jupyter_text>ROC曲线  $receiver\ operating\ characteristic\ curve$   反应模型的泛化能力，绘制$TPR\ against\ FPR$   $\displaystyle TPR(true\ positive\ rate) = \frac{TP}{TP+FN}$  , $\displaystyle FPR(false\ positive\ rate) = \frac{FP}{FP+TN}$<jupyter_code>from sklearn.metrics import roc_curve

fpr, tpr ,thresholds = roc_curve(y_train_5, y_scores)
def plot_roc_curve(fpr, tpr, label=None):
    plt.plot(fpr, tpr, linewidth=2, label=label)
    plt.plot([0,1],[0,1],'k--')
    plt.axis([0,1,0,1])
    plt.xlabel("False Positive Rate")
    plt.ylabel("True Positive Rate")
plot_roc_curve(fpr, tpr) # 绘制ROC
plt.show()
from sklearn.metrics import roc_auc_score # 计算AUC ROC 面积
roc_auc_score(y_train_5, y_scores)<jupyter_output><empty_output><jupyter_text>何时使用ROC与PR当正例很少或者我们关心$FP$超过$FN$时选用PR curve   想想反之选择ROC curve 随机森林分类器注意其工作方式并没与`decision_function()`显示分数，其使用`dict_proba()`方法    对于sklearn的分类器都有其中一种可以显示估计器得分的方法<jupyter_code>from sklearn.ensemble import RandomForestClassifier

forest_clf = RandomForestClassifier(random_state=42)

y_proba_forest = cross_val_predict(forest_clf, X_train, y_train_5, cv=3, method="predict_proba")
y_proba_forest # 输出两个 label<jupyter_output><empty_output><jupyter_text>Multiclass Classification多类别分类器随机森林，朴素贝叶斯分类器等都能直接处理多类别，其他的例如支持向量机和线性模型是严格二分类器   然而仍有方法去实现线性模型个多分类。   一种称为$(one-versus-all)OvA$, 对手写体做十个二分类器组合预测       从每个分类器得到该图像的`decision score`取得分最高作为预测结果   另一种称为(`one-versus-one`)$OvO$ ，对每对数字做一个分类器,共计需要$\displaystyle\frac{N*(N-1)}{2}$个分类器    预测时从这些分类器中查询结果，这种策略的好处在于每个分类器只需在部分训练数据上跑模型。  对于有些算法例如支持向量机分类器在大数据下表现不优，此时就可以采取OvO方式训练出很多小模型。   对于大部分机器学习算法优先采用OvA方式<jupyter_code>sgd_clf.fit(X_train, y_train)
sgd_clf.predict([some_digit]) # sklearn 对于y_train多class 自动OvA多类分类(除了SVM)
some_digit_scores = sgd_clf.decision_function([some_digit])
some_digit_scores # 实际上sklearn 底下训练了10组分类器
np.argmax(some_digit_scores)
sgd_clf.classes_ # 按序存储class
sgd_clf.classes_[5]<jupyter_output><empty_output><jupyter_text>强制使用$OvO$跑线性多分类器   从`OneVsOneClassifier` 或 `OneVsRestClassifier` 类自己构造即可<jupyter_code>import warnings 
warnings.filterwarnings('ignore') # 强用ovo会报Further Warning这里忽略下！

from sklearn.multiclass import OneVsOneClassifier

ovo_clf = OneVsOneClassifier(SGDClassifier(random_state=42))
ovo_clf.fit(X_train, y_train)
ovo_clf.predict([some_digit])

len(ovo_clf.estimators_) # 10 * 9 / 2 = 45
# 随机森林并没有使用ovo or ova，其可以直接多分类
forest_clf.fit(X_train, y_train)
forest_clf.predict([some_digit])
forest_clf.predict_proba([some_digit])
forest_clf.classes_
cross_val_score(sgd_clf, X_train, y_train, cv=3, scoring="accuracy")
# array([0.87567487, 0.86234312, 0.87038056])<jupyter_output><empty_output><jupyter_text>尝试特征缩放 优化<jupyter_code>from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))
cross_val_score(sgd_clf, X_train_scaled, y_train, cv=3, scoring="accuracy")
# array([0.91261748, 0.91069553, 0.91113667])<jupyter_output><empty_output><jupyter_text>分析错误 假设我们有了几个较好的模型，现在来分析其错误是什么造成的<jupyter_code>y_train_pred = cross_val_predict(sgd_clf, X_train_scaled, y_train, cv=3)
y_train_pred # 返回预测结果score 返回预测得分
conf_mx = confusion_matrix(y_train, y_train_pred)
conf_mx
# 主对角线位为正确的预测
plt.matshow(conf_mx, cmap=plt.cm.gray)
plt.show()
#### 使用错误率而非绝对错误的数量描述图像
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = conf_mx / row_sums
np.fill_diagonal(norm_conf_mx, 0)
plt.matshow(norm_conf_mx, cmap=plt.cm.gray)
plt.show()<jupyter_output><empty_output><jupyter_text>行为实际的class，列为预测的class   观察错误来源$(8, *), (*, 8), (5, 3), (3, 5), (7, 9), (9, 7)$  通过混淆矩阵帮我们后期优化致命方向，应该改善8,9与其他数字的区别，调整3,5的预测方式 个体分析3,5<jupyter_code>def plot_digits(instances, images_per_row=10, **options):
    size = 28
    images_per_row = min(len(instances), images_per_row)
    images = [instance.reshape(size,size) for instance in instances]
    n_rows = (len(instances) - 1) // images_per_row + 1
    row_images = []
    n_empty = n_rows * images_per_row - len(instances)
    images.append(np.zeros((size, size * n_empty)))
    for row in range(n_rows):
        rimages = images[row * images_per_row : (row + 1) * images_per_row]
        row_images.append(np.concatenate(rimages, axis=1))
    image = np.concatenate(row_images, axis=0)
    plt.imshow(image, cmap = matplotlib.cm.binary, **options)
    plt.axis("off")
cl_a, cl_b = 3, 5
# 手工混淆矩阵
X_aa = X_train[(y_train == cl_a) & (y_train_pred == cl_a)]
X_ab = X_train[(y_train == cl_a) & (y_train_pred == cl_b)]
X_ba = X_train[(y_train == cl_b) & (y_train_pred == cl_a)]
X_bb = X_train[(y_train == cl_b) & (y_train_pred == cl_b)]

# 默认子图命令是subplot(111)
plt.subplot(221); plot_digits(X_aa[:25], images_per_row=5)
plt.subplot(222); plot_digits(X_ab[:25], images_per_row=5)
plt.subplot(223); plot_digits(X_ba[:25], images_per_row=5)
plt.subplot(224); plot_digits(X_bb[:25], images_per_row=5)
plt.show()
# 左侧为预测为a=3, 右侧为预测为b=5<jupyter_output><empty_output><jupyter_text>Multilabel Classification 多标签分类器 单标签分类器把训练数据扔进一个class中，多标签分类器对任务要扔到多个class中<jupyter_code>y_train_large = (y_train >= 7)
y_train_odd =(y_train % 2 == 1)
y_multiabel = np.c_[y_train_large, y_train_odd]
print("y_train_large: ", y_train_large)
print("y_train_odd: ", y_train_odd)
print("y_multiabel: \n", y_multiabel)
from sklearn.neighbors import KNeighborsClassifier

knn_clf = KNeighborsClassifier()
knn_clf.fit(X_train, y_multiabel)
knn_clf.predict([some_digit]) # some_digit = 5
y_train_knn_pred = cross_val_predict(knn_clf, X_train, y_train, cv=3)
import pickle
with open("../y_train_knn_pred.pk", "rb") as f:
     y_train_knn_pred = pickle.load(f)
f1_score(y_train, y_train_knn_pred, average="macro") # average=macro label同等重要， weighted加权
import pickle

with open('../y_train_knn_pred.pk', 'wb') as f:
#     np.ndarray.dump(y_train_knn_pred, f)
     pickle.dump(y_train_knn_pred, f)<jupyter_output><empty_output><jupyter_text>Multioutput Classifiction 多输出分类器  模拟一个系统输入混乱的图片输出干净的图片<jupyter_code>### 产生噪声

noise = np.random.randint(0, 100, (len(X_train), 784))
X_train_mod = X_train + noise
noise = np.random.randint(0, 100, (len(X_test), 784))
X_test_mod = X_test + noise

y_train_mod = X_train
y_test_mod = X_test
def plot_digit(data):
    image = data.reshape(28, 28) # 别忘了reshape成(28*28)
    plt.imshow(image, cmap = plt.cm.binary,
               interpolation="nearest")
    plt.axis("off")
some_index = 5500
plt.subplot(121); plot_digit(X_test_mod[some_index])
plt.subplot(122); plot_digit(y_test_mod[some_index])
plt.show()
knn_clf.fit(X_train_mod, y_train_mod)
clean_digit = knn_clf.predict([X_test_mod[some_index]])
plot_digit(clean_digit)<jupyter_output><empty_output><jupyter_text>Extra material  伪分类器 提供基线参考<jupyter_code>from sklearn.dummy import DummyClassifier # 随机分类器
dmy_clf = DummyClassifier()
y_probas_dmy = cross_val_predict(dmy_clf, X_train, y_train, cv=3, method="predict_proba")
y_scores_dmy = y_probas_dmy[:,1]
fprr, tprr, thresholdsr = roc_curve(y_train_5, y_scores_dmy)
plot_roc_curve(fprr, tprr)<jupyter_output><empty_output><jupyter_text>KNN classifie<jupyter_code>from sklearn.neighbors import KNeighborsClassifier

knn_clf = KNeighborsClassifier(n_jobs=-1, weights="distance", n_neighbors=4)
knn_clf.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Exercises  1. Knn超参优化knn在此类问题山表现优秀，尝试用网格搜索做超参优化  knn分类器的主要超参：weights和n_neighbors  2. 数据增强 data augmentation尝试对原有手写体进行旋转，增强/扩充数据集<jupyter_code>from scipy.ndimage.interpolation import shift

def shift_image(img, dx, dy):
    img = img.reshape(28, 28)
    shifted_image = shift(img, [dy, dx], cval=0, mode="constant")
    return shifted_image.reshape([-1])
image_set = X_train[233]
print(y_train[233])
shifted_image_down = shift_image(image_set, dx=0, dy=5)
shifted_image_left = shift_image(image_set, dx=-5, dy=0)

plt.figure(figsize=(5,3))

plt.subplot(131)
plt.title("Original", fontsize=14)
plt.imshow(image_set.reshape(28,28), interpolation="nearest", cmap="Greys")

plt.subplot(132)
plt.title("Shifted down", fontsize=14)
plt.imshow(shifted_image_down.reshape(28, 28), interpolation="nearest", cmap="Greys")

plt.subplot(133)
plt.title("Shifted left", fontsize=14)
plt.imshow(shifted_image_left.reshape(28, 28), interpolation="nearest", cmap="Greys")
plt.show()
X_train_augmented = [img for img in X_train]
y_train_augmented = [img for img in y_train]
for dx, dy in ((1,0), (-1,0), (0,1), (0,-1)):
    for image, label in zip(X_train, y_train):
        X_train_augmented.append(shift_image(image, dx=dx, dy=dy))
        y_train_augmented.append(label)
        
X_train_augmented = np.array(X_train_augmented)
y_train_augmented = np.array(y_train_augmented)
knn_clf = KNeighborsClassifier()
knn_clf.fit(X_train_augmented, y_train_augmented)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12779552715654952, 'fraction_numerical': 0.031004937554458323, 'mean_word_length': 7.728136882129277, 'pattern_counts': {'":': 0, '<': 84, '<?xml version=': 0, '>': 88, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '662962', 'license': 'MIT', 'n_tokens_mistral': 6815, 'n_tokens_neox': 6745, 'n_words': 799, 'path': 'handson-ml/3_Classification.ipynb', 'repo_name': 'Luoyayu/Machine-Learning', 'size': '13772'}