<jupyter_start><jupyter_text>Datacamp: Introduction to Numpy    <NAME> : <EMAIL> The main goal of this notebook is to get familiar with Python and NumPy by manipulating a famous dataset in machine learning.The data are embedded in the `scikit-learn` library.This dataset is known as digits and contains images of hand-written digits with theirs associated labels.  I - Manipulations and visualization of the `digits` dataset  Imports et intialisation<jupyter_code>%matplotlib inline                      

import numpy as np                      # charge un package pour le numérique
import matplotlib.pyplot as plt         # charge un package pour les graphiques<jupyter_output><empty_output><jupyter_text>Dataset descriptionWe will load the `digits` dataset available in `scikit-learn` (the import name is `sklearn`). This dataset contains images of hand-written digits.<jupyter_code># Load the dataset from scikit-learn
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.data, digits.target
X
X.shape
X.ndim
y.shape
y.size
X.size, 1797*64
X[0, :].size
y
y.max(), y.min()
np.unique(y)
print(f"Number of pixels (features):      {X.shape[1]}")
print(f"Number of images (samples):       {X.shape[0]}")
print(f"Number of classes:                {len(np.unique(y))}")
# Choose any image (sample)
idx_to_test = 15

print("Show a line of the array (i.e., image):")
print(X[idx_to_test, :])
print("Show the associted class (i.e., associated class):")
print(y[idx_to_test])<jupyter_output>Show a line of the array (i.e., image):
[ 0.  5. 12. 13. 16. 16.  2.  0.  0. 11. 16. 15.  8.  4.  0.  0.  0.  8.
 14. 11.  1.  0.  0.  0.  0.  8. 16. 16. 14.  0.  0.  0.  0.  1.  6.  6.
 16.  0.  0.  0.  0.  0.  0.  5. 16.  3.  0.  0.  0.  1.  5. 15. 13.  0.
  0.  0.  0.  4. 15. 16.  2.  0.  0.  0.]
Show the associted class (i.e., associated class):
5<jupyter_text>EXERCISE:           What is the data type of X? y?      Change `idx_to_test`. Without showing y[idx_to_test], can you recognize the number of this sample?<jupyter_code>X.dtype
y.dtype
idx_to_test = 10
X[idx_to_test]<jupyter_output><empty_output><jupyter_text>Data visualization:The digitized images have a size of 8 x 8 accounting for a total of 64 pixels. They are stored into a row vector which need to be reshaped to be visualized as an image. You can use the function `numpy.reshape` to transform the 1D-array into a 2D-array of 8 x 8 values.<jupyter_code># We use `imshow` to visualize the the image
plt.imshow(np.reshape(X[0, :], (8, 8)));
# Use a grayscale colormap for a better visualization
plt.imshow(np.reshape(X[idx_to_test, :], (8, 8)),
           cmap='gray', aspect='equal', interpolation='nearest')
plt.colorbar()
plt.title(f"The associated class with the image at index {idx_to_test} is {y[idx_to_test]}")<jupyter_output><empty_output><jupyter_text>EXERCICE:           Show an image with 1 line and 1 column over 2.      Show the previous image by removing the pixel on the border?      Show the pixel distribution using `plt.hist`).<jupyter_code>img = X[0].reshape((8, 8))
plt.imshow(img[::2, ::2])
plt.imshow(img[1:-1, 1:-1])
plt.hist(img.ravel());<jupyter_output><empty_output><jupyter_text>Basic statistics:Pour mieux comprendre la base de données on va s'intéresser à quelques statistiques. On commence par calculer les moyennes et variances par classes pour chacun des chiffres. La moyenne par classe se visualise comme une image qui est une représentantion moyenne pour chaque chiffre de zéro à neuf. Idem pour la variance, ce qui permet alors de voir les parties avec les plus grandes variations entre les membres d'une même classe.To better understand the database, we will check a couple of statistics.We can start by looking at the mean and variances for each class digit.We can plot the mean and variance which should be a 8 x 8 image.* What is the representation of the mean and variance images?<jupyter_code>classes_list = np.unique(y).astype(int)
print("Liste des classes en présence: ", classes_list)<jupyter_output>Liste des classes en présence:  [0 1 2 3 4 5 6 7 8 9]<jupyter_text>EXERCICE:           Calculer un représentant moyen du chiffre 0 (l'image qui en pixel i,j contient la valeur moyenne du pixel i,j parmis tous les 0)      Avec une boucle `for` calculer le représentant moyen pour chaque chiffre      Faire la même chose en remplaçant la moyenne par l'écart type      Afficher toutes les images sur une grille à l'aide de la fonction `plt.subplots`<jupyter_code>fig, axs = plt.subplots(ncols=5, nrows=2)
for klazz, ax in zip(classes_list, axs.ravel()):
    mask_img = y == klazz
    ax.imshow(
        X[mask_img].mean(axis=0).reshape((8, 8)),
        cmap="gray"
    )
    ax.axis("off")
    ax.set_title(f"Label: {klazz}")
fig, axs = plt.subplots(ncols=5, nrows=2)
for klazz, ax in zip(classes_list, axs.ravel()):
    mask_img = y == klazz
    ax.imshow(
        X[mask_img].std(axis=0).reshape((8, 8)),
        cmap="gray"
    )
    ax.axis("off")
    ax.set_title(f"Label: {klazz}")<jupyter_output><empty_output><jupyter_text>II - Nearest centroids classificationThe aim of this exercise is to implement your own classifier based on an intuitive idea.For a new image, we will predict the class for which the mean digit is the closest (in the feature space).    EXERCISE:           Split the dataset into 2 part. We will denote the with the following variable X_train, y_train, X_test, and y_test for the training data and labels and testing data and labels, respectively.      For each class, compute the mean digit image on the training set. We will denote the resulting variable `centroids_train`.      For each sample in the testing set, compute the nearest centroids. Compute the percentage of good predictions to evaluate the performance of your classifier.<jupyter_code>mask_train = np.ones(y.shape, dtype=bool)
mask_train[len(y) // 2:] = False
mask_test = ~mask_train
X_train, X_test = X[mask_train], X[mask_test]
y_train, y_test = y[mask_train], y[mask_test]
centroids_train = []
for klazz in classes_list:
    mask_img = y_train == klazz
    centroids_train.append(X_train[mask_img].mean(axis=0))
centroids_train = np.array(centroids_train)
fig, axs = plt.subplots(ncols=5, nrows=2)
for img, ax in zip(centroids_train, axs.ravel()):
    ax.imshow(
        img.reshape((8, 8)), cmap="gray"
    )
    ax.axis("off")
centroids_train.shape
from scipy.spatial.distance import cdist

y_pred = cdist(X_test, centroids_train).argmin(axis=1)
(np.zeros(len(y_test)) == y_test).mean()
img_dst = []
for img_test in X_test:
    dist = []
    for centroid in centroids_train:
        dist.append(np.sqrt(np.sum((img_test - centroid)**2)))
    img_dst.append(dist)
img_dst = np.array(img_dst)
img_dst.argmin(axis=1)
y_test
from scipy.spatial.distance import cdist
from sklearn.base import BaseEstimator, ClassifierMixin


class NearestCentroidClassifier(BaseEstimator, ClassifierMixin):
    
    def fit(self, X, y):
        centroids_train = []
        self.classes_ = np.unique(y)
        for klazz in self.classes_:
            mask_img = y == klazz
            centroids_train.append(X[mask_img].mean(axis=0))
        self.centroids_train = np.array(centroids_train)
        return self
        
    def predict(self, X):
        return cdist(X_test, centroids_train).argmin(axis=1)
clf = NearestCentroidClassifier()
clf.fit(X_train, y_train).predict(X_test)
clf.score(X_test, y_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10403225806451613, 'fraction_numerical': 0.02110215053763441, 'mean_word_length': 3.7913715389568576, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '553950', 'license': 'CC0-1.0', 'n_tokens_mistral': 2644, 'n_tokens_neox': 2545, 'n_words': 858, 'path': '04_datacamp/datacamp_numpy_corrected.ipynb', 'repo_name': 'HirahTang/datascience_starter_course', 'size': '7472'}