<jupyter_start><jupyter_text>用 NumPy 就能辨識數字<jupyter_code>from PIL import Image
import numpy as np<jupyter_output><empty_output><jupyter_text>先下載 MNIST 資料<jupyter_code>import os
import urllib
from urllib.request import urlretrieve
dataset = 'mnist.pkl.gz'
def reporthook(a,b,c):
    print("\rdownloading: %5.1f%%"%(a*b*100.0/c), end="")
    
if not os.path.isfile(dataset):
        origin = "https://github.com/mnielsen/neural-networks-and-deep-learning/raw/master/data/mnist.pkl.gz"
        print('Downloading data from %s' % origin)
        urlretrieve(origin, dataset, reporthook=reporthook)
import gzip
import pickle
with gzip.open(dataset, 'rb') as f:
    # dataset is built in Python 2, and Python 3 needs to read in latin1
    train_set, validation_set, test_set = pickle.load(f, encoding='latin1')<jupyter_output><empty_output><jupyter_text>Q先看看這些資料是什麼吧！<jupyter_code># use scipy.stats to describe data
from scipy import stats
print("train_set[0]:\n{}\n".format(stats.describe(train_set[0].ravel())))
print("validation_set[0]:\n{}\n".format(stats.describe(validation_set[0].ravel())))
print("test_set[0]:\n{}".format(stats.describe(test_set[0].ravel())))
# ravle() will transform matrix to flat array
print("train_set[0]:      ravel={}".format(train_set[0].ravel().shape))
print("train_set[0]:      shape={}\t dtype={}".format(train_set[0].shape, train_set[0].dtype))
print("train_set[1]:      shape={}\t dtype={}".format(train_set[1].shape, train_set[1].dtype))
print("validation_set[0]: shape={}\t dtype={}".format(validation_set[0].shape, validation_set[0].dtype))
print("validation_set[1]: shape={}\t dtype={}".format(validation_set[1].shape, validation_set[1].dtype))
print("test_set[0]:       shape={}\t dtype={}".format(test_set[0].shape, test_set[0].dtype))
print("test_set[1]:       shape={}\t dtype={}".format(test_set[1].shape, test_set[1].dtype))<jupyter_output>train_set[0]:      ravel=(39200000,)
train_set[0]:      shape=(50000, 784)	 dtype=float32
train_set[1]:      shape=(50000,)	 dtype=int64
validation_set[0]: shape=(10000, 784)	 dtype=float32
validation_set[1]: shape=(10000,)	 dtype=int64
test_set[0]:       shape=(10000, 784)	 dtype=float32
test_set[1]:       shape=(10000,)	 dtype=int64<jupyter_text>Supervised Learning類比：類比：中文房間<jupyter_code>train_X, train_y = train_set
test_X, test_y = test_set<jupyter_output><empty_output><jupyter_text>看一下 MNIST 的 y 是什麼<jupyter_code># 訓練資料， y 的前 20 筆
train_y[:20]<jupyter_output><empty_output><jupyter_text>看一下 MNIST 的 X<jupyter_code>from IPython.display import display
def showX(X):
    int_X = (X*255).clip(0,255).astype('uint8')
    # N*784 -> N*28*28 -> 28*N*28 -> 28 * 28N
    int_X_reshape = int_X.reshape(-1,28,28).swapaxes(0,1).reshape(28,-1)
    display(Image.fromarray(int_X_reshape))
# 訓練資料， X 的前 20 筆
showX(train_X[:20])<jupyter_output><empty_output><jupyter_text>先從簡單的方法開始笨方法，直接比較，找最接近的圖。先試試看使用方差好了  Q 計算 u, v 方差```pythonu = train_X[0]v = train_X[1]```<jupyter_code>u = train_X[0]
v = train_X[1]
print(np.sum((u-v)**2))
print(np.sum(np.square(u-v)))
print(np.linalg.norm(u-v)**2)<jupyter_output>86.94919
86.94919
86.949183020698<jupyter_text>Q 試著* 顯示 test_X[0]* 在 `train_X` 中找出最像 `test_X[0]` 的圖片編號* display 那張最圖片* 然後查看對應的 `train_y`* 看看 test_y[0]<jupyter_code>def nearest_neighbor_draw(length):
    predictions = np.zeros(shape=(1, length))
    for d_index in range(length):
        # show test_X
        showX(test_X[d_index])
        # find the most similar train data index
        s_index = np.sum(np.square(test_X[d_index]-train_X), axis=1).argmin()
        print("train_X[{}]".format(s_index))
        showX(train_X[s_index])
        # check train_y is equal to test_y or not
        print("train_y[{}] = {}".format(s_index, train_y[s_index]))
        print("test_y[{}] = {}".format(d_index, test_y[d_index]))
        predictions[0][d_index] = train_y[s_index]
    return predictions
def nearest_neighbor(length):
    predictions = np.zeros(shape=(1, length))
    for d_index in range(length):
        s_index = np.sum(np.square(test_X[d_index]-train_X), axis=1).argmin()
        predictions[0][d_index] = train_y[s_index]
    return predictions
prediction = nearest_neighbor_draw(1)<jupyter_output><empty_output><jupyter_text>Q拿前面10/100個 `test_X` 做同樣的事情，然後統計一準確度。<jupyter_code>prediction = nearest_neighbor_draw(10)
print("accuracy: {}%".format(np.mean(prediction==test_y[:10])*100))
prediction = nearest_neighbor(100)
print("accuracy: {}%".format(np.mean(prediction==test_y[:100])*100))<jupyter_output>accuracy: 100.0%<jupyter_text>Q如果 train_X 只有 500 筆資料呢？利用 reshaping,  broadcasting 技巧， 算出對 `test_X[:100]` 的準確度！Hint: `np.expand_dims` 來取代 `np.reshape`<jupyter_code>def nearest_neighbor_500(length):
    predictions = np.zeros(shape=(1, length))
    for d_index in range(length):
        s_index = np.sum(np.square(test_X[d_index]-train_X[:500]), axis=1).argmin()
        predictions[0][d_index] = train_y[s_index]
    return predictions
prediction = nearest_neighbor_500(100)
print("accuracy: {}%".format(np.mean(prediction==test_y[:100])*100))
u = np.expand_dims(train_X[:500], 1)
v = np.expand_dims(test_X[:100], 0)
prediction = train_y[np.sum(np.square(u - v), axis=2).argmin(axis=0)]
print("accuracy: {}%".format(np.mean(prediction==test_y[:100])*100))<jupyter_output>accuracy: 85.0%<jupyter_text>Q用其他距離函數? e.g.  `np.abs(...).sum()`<jupyter_code>def nearest_neighbor_l1(length):
    predictions = np.zeros(shape=(1, length))
    for d_index in range(length):
        s_index = np.linalg.norm(test_X[d_index]-train_X, 1, axis=1).argmin()
#         s_index = np.sum(np.abs(test_X[d_index]-train_X), axis=1).argmin()
        predictions[0][d_index] = train_y[s_index]
    return predictions
prediction = nearest_neighbor_l1(100)
print("accuracy: {}%".format(np.mean(prediction==test_y[:100])*100))<jupyter_output>accuracy: 98.0%<jupyter_text>改用來內積取代方差$$\begin{align*}\left\Vert \mathbf{u}-\mathbf{v}\right\Vert ^{2} & =\left(\mathbf{u}-\mathbf{v}\right)\cdot\left(\mathbf{u}-\mathbf{v}\right)\\ & =\left\Vert \mathbf{u}\right\Vert ^{2}-2\mathbf{u}\cdot\mathbf{v}+\left\Vert \mathbf{v}\right\Vert ^{2}\\\end{align*}$$ Note* 改善用掉太多 memory 的問題    * square error 所需 memory = $ 10000 \cdot 5000 \cdot 784 $    * cosine similarity 所需 memory = $ 10000 \cdot 5000 $* 內積結果值越大表示兩向量夾角越小越相近* normalize 用意在於不要駛向量長度影響結果    * 使用限制是資料不會因為做 normalize 而無法辨識<jupyter_code># 資料 normalize
train_X  = train_X / np.linalg.norm(train_X, axis=1, keepdims=True)
test_X  = test_X / np.linalg.norm(test_X, axis=1, keepdims=True)
# 矩陣乘法 == 大量計算內積
A = test_X @ train_X.T
print(A.shape)
# 內積結果值越大表示越相近
A.argmax(axis=1)
predict_y = train_y[A.argmax(axis=1)]
# 測試資料， X 的前 20 筆
# could not use showX(test_X[:20]) because we did the normalization before
showX(test_set[0][:20])
# 猜測的 Y 前20筆
predict_y[:20]
#測試資料的 y 前 20 筆
test_y[:20]
# 正確率
print("accuracy: {}%".format(np.mean(predict_y == test_y)*100))<jupyter_output>accuracy: 97.08%<jupyter_text>用 PCA 降低維度<jupyter_code>from sklearn.decomposition import PCA
pca = PCA(n_components=60)
train_X = pca.fit_transform(train_set[0])
test_X = pca.transform(test_set[0])
train_X.shape
train_X /= np.linalg.norm(train_X, axis=1, keepdims=True)
test_X /= np.linalg.norm(test_X, axis=1, keepdims=True)
# 矩陣乘法
A = test_X @ train_X.T
predict_y = train_y[A.argmax(axis=1)]
# 正確率
print("accuracy: {}%".format(np.mean(predict_y == test_y)*100))<jupyter_output>accuracy: 97.07000000000001%<jupyter_text>Q* 試試看不同的維度* 檢查看看 PCA 前後，定義的誤差函數之間的差別。* 使用 sklearn knn<jupyter_code># knn
from sklearn.neighbors import KNeighborsClassifier
neigh = KNeighborsClassifier(n_neighbors=10)
neigh.fit(train_X, train_y)
predict_y = neigh.predict(test_X)
print("accuracy: {}%".format(np.mean(predict_y == test_y)*100))<jupyter_output>accuracy: 97.03%
{'fraction_non_alphanumeric': 0.17055167055167056, 'fraction_numerical': 0.04700854700854701, 'mean_word_length': 5.9702166064981945, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 52, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '168481', 'license': 'MIT', 'n_tokens_mistral': 3608, 'n_tokens_neox': 3473, 'n_words': 483, 'path': 'Basic_ML/01-From NumPy to MNIST.ipynb', 'repo_name': 'PinmanHuang/CrashCourseML', 'size': '7722'}