<jupyter_start><jupyter_text>Week 1: Mean/Covariance of a data set and effect of a linear transformationIn this week, we are going to investigate how the mean and (co)variance of a dataset changeswhen we apply affine transformation to the dataset.  Learning objectives1. Get Farmiliar with basic programming using Python and Numpy/Scipy.2. Learn to appreciate implementing   functions to compute statistics of dataset in vectorized way.3. Understand the effects of affine transformations on a dataset.4. Understand the importance of testing in programming for machine learning. First, let's import the packages that we will use for the week<jupyter_code># PACKAGE: DO NOT EDIT
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
matplotlib.style.use('fivethirtyeight')
from sklearn.datasets import fetch_olivetti_faces
import time
import timeit
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Next, we are going to retrieve Olivetti faces dataset.When working with some datasets, before digging into further analysis, it is almost alwaysuseful to do a few things to understand your dataset. First of all, answer the followingset of questions:1. What is the size of your dataset?2. What is the dimensionality of your data?The dataset we have are usually stored as 2D matrices, then it would be really importantto know which dimension represents the dimension of the dataset, and which representsthe data points in the dataset. __When you implement the functions for your assignment, make sure you readthe docstring for what each dimension of your inputs represents the data points, and which represents the dimensions of the dataset!__<jupyter_code>image_shape = (64, 64)
# Load faces data
dataset = fetch_olivetti_faces(data_home='./')
faces = dataset.data

print('Shape of the faces dataset: {}'.format(faces.shape))
print('{} data points'.format(faces.shape[0]))<jupyter_output>Shape of the faces dataset: (400, 4096)
400 data points<jupyter_text>When your dataset are images, it's a really good idea to see what they look like.One veryconvenient tool in Jupyter is the `interact` widget, which we use to visualize the images (faces). For more information on how to use interact, have a look at the documentation [here](http://ipywidgets.readthedocs.io/en/stable/examples/Using%20Interact.html).<jupyter_code>from ipywidgets import interact
def show_face(face):
    plt.figure()
    plt.imshow(face.reshape((64, 64)), cmap='gray')
    plt.show()
@interact(n=(0, len(faces)-1))
def display_faces(n=0):
    plt.figure()
    plt.imshow(faces[n].reshape((64, 64)), cmap='gray')
    plt.show()<jupyter_output><empty_output><jupyter_text>1. Mean and Covariance of a Dataset<jupyter_code># GRADED FUNCTION: DO NOT EDIT THIS LINE
def mean_naive(X):
    """Compute the sample mean for a dataset by iterating over the dataset.
    
    Args:
        X: `ndarray` of shape (N, D) representing the dataset. N 
        is the size of the dataset and D is the dimensionality of the dataset.
    Returns:
        mean: `ndarray` of shape (D, ), the sample mean of the dataset `X`.
    """
    # YOUR CODE HERE
    # Uncomment and edit the code below
    # iterate over the dataset and compute the mean vector.
    N, D = X.shape
    mean = np.zeros((D,))
    for n in range(D):
        mean[n] = (np.sum(X[:,n])/N)
    return mean

def cov_naive(X):
    """Compute the sample covariance for a dataset by iterating over the dataset.
    
    Args:
        X: `ndarray` of shape (N, D) representing the dataset. N 
        is the size of the dataset and D is the dimensionality of the dataset.
    Returns:
        ndarray: ndarray with shape (D, D), the sample covariance of the dataset `X`.
    """
    # YOUR CODE HERE
    # Uncomment and edit the code below
    X = np.asmatrix(X)
    N, D = X.shape
    # Edit the code below to compute the covariance matrix by iterating over the dataset.
    # Update covariance
    covariance = np.cov(X.T, bias = True)
    
    return covariance

def mean(X):
    """Compute the sample mean for a dataset.
    
    Args:
        X: `ndarray` of shape (N, D) representing the dataset. N 
        is the size of the dataset and D is the dimensionality of the dataset.
    Returns:
        ndarray: ndarray with shape (D,), the sample mean of the dataset `X`.
    """
    # YOUR CODE HERE
    # Uncomment and edit the code below
    m = np.mean(X,axis = 0, keepdims=True)
    N, D = m.shape
    m = m.reshape(D,)
    return m

def cov(X):
    """Compute the sample covariance for a dataset.
    
    Args:
        X: `ndarray` of shape (N, D) representing the dataset. N 
        is the size of the dataset and D is the dimensionality of the dataset.
    Returns:
        ndarray: ndarray with shape (D, D), the sample covariance of the dataset `X`.
    """
    # YOUR CODE HERE
    
    # It is possible to vectorize our code for computing the covariance with matrix multiplications,
    # i.e., we do not need to explicitly
    # iterate over the entire dataset as looping in Python tends to be slow
    # We challenge you to give a vectorized implementation without using np.cov, but if you choose to use np.cov,
    # be sure to pass in bias=True.
    # Uncomment and edit the code below
    X = np.asmatrix(X)
    N, D = X.shape
    # Edit the code to compute the covariance matrix
    # Update covariance_matrix here
    covariance_matrix = np.cov(X.T, bias = True)
    
    return covariance_matrix
from numpy.testing import assert_allclose

# Test case 1
X = np.array([[0., 1., 1.], 
              [1., 2., 1.]])
expected_mean = np.array([0.5, 1.5, 1.])
assert_allclose(mean(X), expected_mean, rtol=1e-5)

# Test case 2
X = np.array([[0., 1., 0.], 
              [2., 3., 1.]])
expected_mean = np.array([1., 2., 0.5])
assert_allclose(mean(X), expected_mean, rtol=1e-5)

# Test covariance is zero
X = np.array([[0., 1.], 
              [0., 1.]])
expected_mean = np.array([0., 1.])
assert_allclose(mean(X), expected_mean, rtol=1e-5)

### Some hidden tests below
### ...
cov([[0., 1.], 
     [1., 2.],
     [0., 1.], 
     [1., 2.]
    ])
from numpy.testing import assert_allclose

# Test case 1
X = np.array([[0., 1.], 
              [1., 2.],
     [0., 1.], 
     [1., 2.]])
expected_cov = np.array(
    [[0.25, 0.25],
    [0.25, 0.25]])

assert_allclose(cov(X), expected_cov, rtol=1e-5)

# Test case 2
X = np.array([[0., 1.], 
              [2., 3.]])
expected_cov = np.array(
    [[1., 1.],
    [1., 1.]])

assert_allclose(cov(X), expected_cov, rtol=1e-5)

# Test covariance is zero
X = np.array([[0., 1.], 
              [0., 1.],
              [0., 1.]])
expected_cov = np.zeros((2, 2))

assert_allclose(cov(X), expected_cov, rtol=1e-5)

### Some hidden tests below
### ...<jupyter_output><empty_output><jupyter_text>With the `mean` function implemented, let's take a look at the _mean_ face of our dataset!<jupyter_code>def mean_face(faces):
    return faces.mean(axis=0).reshape((64, 64))

plt.imshow(mean_face(faces), cmap='gray');<jupyter_output><empty_output><jupyter_text>One of the advantage of writing vectorized code is speedup gained when working on larger dataset. Loops in Pythonare slow, and most of the time you want to utilise the fast native code provided by Numpy without explicitly usingfor loops. To put things into perspective, we can benchmark the two different implementation with the `%time` functionin the following way:<jupyter_code># We have some HUUUGE data matrix which we want to compute its mean
X = np.random.randn(1000, 20)
# Benchmarking time for computing mean
%time mean_naive(X)
%time mean(X)
pass
# Benchmarking time for computing covariance
%time cov_naive(X)
%time cov(X)
pass<jupyter_output>CPU times: user 1 ms, sys: 0 ns, total: 1 ms
Wall time: 698 µs
CPU times: user 579 µs, sys: 30 µs, total: 609 µs
Wall time: 74.2 ms<jupyter_text>2. Affine Transformation of DatasetIn this week we are also going to verify a few properties about the mean andcovariance of affine transformation of random variables.Consider a data matrix $X$ of size (N, D). We would like to knowwhat is the covariance when we apply affine transformation $Ax_i + b$ for each datapoint $x_i$ in $X$. i.e.we would like to know what happens to the mean and covariance for the new dataset if we apply affine transformation.<jupyter_code># GRADED FUNCTION: DO NOT EDIT THIS LINE

def affine_mean(mean, A, b):
    """Compute the mean after affine transformation
    Args:
        mean: `ndarray` of shape (D,), the sample mean vector for some dataset.
        A, b: `ndarray` of shape (D, D) and (D,), affine transformation applied to x
    Returns:
        sample mean vector of shape (D,) after affine transformation.
    """
    # YOUR CODE HERE
    # Uncomment and edit the code below
    # Edit the code below to compute the mean vector after affine transformation
    affine_m = np.zeros(mean.shape) # affine_m has shape (D,)
    affine_m = A@mean + b
    # Update affine_m
    
    return affine_m
# GRADED FUNCTION: DO NOT EDIT THIS LINE
def affine_covariance(S, A, b):
    """Compute the covariance matrix after affine transformation
    
    Args:
        mean: `ndarray` of shape (D,), the sample covariance matrix for some dataset.
        A, b: `ndarray` of shape (D, D) and (D,), affine transformation applied to x
    
    Returns:
        sample covariance matrix of shape (D, D) after the transformation
    """
    # YOUR CODE HERE
    # Uncomment and edit the code below
    # EDIT the code below to compute the covariance matrix after affine transformation
    affine_cov = np.zeros(S.shape) # affine_cov has shape (D, D)
    # Update affine_cov
    affine_cov = A@S@A.T
    
    return affine_cov
from numpy.testing import assert_allclose

A = np.array([[0, 1], [2, 3]])
b = np.ones(2)
m = np.full((2,), 2)
S = np.eye(2)*2

expected_affine_mean = np.array([ 3., 11.])
expected_affine_cov = np.array(
    [[ 2.,  6.],
    [ 6., 26.]])

assert_allclose(affine_mean(m, A, b), expected_affine_mean, rtol=1e-4)
### Some hidden tests below
### ...
from numpy.testing import assert_allclose

A = np.array([[0, 1], [2, 3]])
b = np.ones(2)
m = np.full((2,), 2)
S = np.eye(2)*2

expected_affine_cov = np.array(
    [[ 2.,  6.],
    [ 6., 26.]])

assert_allclose(affine_covariance(S, A, b), 
                expected_affine_cov, rtol=1e-4)

### Some hidden tests below
### ...<jupyter_output><empty_output><jupyter_text>Once the two functions above are implemented, we can verify the correctness our implementation. Assuming that we have some $A$ and $b$.<jupyter_code>random = np.random.RandomState(42)
A = random.randn(4,4)
b = random.randn(4)<jupyter_output><empty_output><jupyter_text>Next we can generate some random dataset $X$<jupyter_code>X = random.randn(100, 4)<jupyter_output><empty_output><jupyter_text>Assuming that for some dataset $X$, the mean and covariance are $m$, $S$, and for the new dataset after affine transformation $X'$, the mean and covariance are $m'$ and $S'$, then we would have the following identity:$$m' = \text{affine_mean}(m, A, b)$$$$S' = \text{affine_covariance}(S, A, b)$$<jupyter_code>X1 = ((A @ (X.T)).T + b)  # applying affine transformation once
X2 = ((A @ (X1.T)).T + b) # twice<jupyter_output><empty_output><jupyter_text>One very useful way to compare whether arrays are equal/similar is use the helper functionsin `numpy.testing`.Check the Numpy [documentation](https://docs.scipy.org/doc/numpy-1.13.0/reference/routines.testing.html)for details.If you are interested in learning more about floating point arithmetic, here is a good [paper](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.22.6768).<jupyter_code>np.testing.assert_allclose(mean(X1), affine_mean(mean(X), A, b))
np.testing.assert_allclose(cov(X1),  affine_covariance(cov(X), A, b))
np.testing.assert_allclose(mean(X2), affine_mean(mean(X1), A, b))
np.testing.assert_allclose(cov(X2),  affine_covariance(cov(X1), A, b))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10956725537791914, 'fraction_numerical': 0.018498367791077257, 'mean_word_length': 3.6220502901353964, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '71318', 'license': 'MIT', 'n_tokens_mistral': 4030, 'n_tokens_neox': 3782, 'n_words': 1444, 'path': 'PCA-week1.ipynb', 'repo_name': 'Joao-Pedro-MB/Math-for-ML', 'size': '11947'}