<jupyter_start><jupyter_text>Programming task 10: Dimensionality Reduction<jupyter_code>import warnings
warnings.filterwarnings('ignore')

import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Exporting the results to PDFOnce you complete the assignments, export the entire notebook as PDF and attach it to your homework solutions. The best way of doing that is1. Run all the cells of the notebook.2. Export/download the notebook as PDF (File -> Download as -> PDF via LaTeX (.pdf)).3. Concatenate your solutions for other tasks with the output of Step 2. On a Linux machine you can simply use `pdfunite`, there are similar tools for other platforms too. You can only upload a single PDF file to Moodle.Make sure you are using `nbconvert` Version 5.5 or later by running `jupyter nbconvert --version`. Older versions clip lines that exceed page width, which makes your code harder to grade.   PCA Given the data in the matrix X your tasks is to:* Calculate the covariance matrix $\Sigma$.* Calculate eigenvalues and eigenvectors of $\Sigma$.* Plot the original data $X$ and the eigenvectors to a single diagram. What do you observe? Which eigenvector corresponds to the smallest eigenvalue?* Determine the smallest eigenvalue and remove its corresponding eigenvector. The remaining eigenvector is the basis of a new subspace.* Transform all vectors in X in this new subspace by expressing all vectors in X in this new basis.	  The given data X<jupyter_code>X = np.array([(-3,-2),(-2,-1),(-1,0),(0,1),
              (1,2),(2,3),(-2,-2),(-1,-1),
              (0,0),(1,1),(2,2), (-2,-3),
              (-1,-2),(0,-1),(1,0), (2,1),(3,2)])<jupyter_output><empty_output><jupyter_text>Task 1: Calculate the covariance matrix $\Sigma$<jupyter_code>def get_covariance(X):
    """Calculates the covariance matrix of the input data.
    
    Parameters
    ----------
    X : array, shape [N, D]
        Data matrix.
        
    Returns
    -------
    Sigma : array, shape [D, D]
        Covariance matrix
        
    """
    Sigma = (X.T @ X) / (X.shape[0]-1)
    return Sigma<jupyter_output><empty_output><jupyter_text>Task 2: Calculate eigenvalues and eigenvectors of  $\Sigma$.<jupyter_code>def get_eigen(S):
    """Calculates the eigenvalues and eigenvectors of the input matrix.
    
    Parameters
    ----------
    S : array, shape [D, D]
        Square symmetric positive definite matrix.
        
    Returns
    -------
    L : array, shape [D]
        Eigenvalues of S
    U : array, shape [D, D]
        Eigenvectors of S
    
    """
    L, U = np.linalg.eigh(Sigma)
    return L, U<jupyter_output><empty_output><jupyter_text>Task 3: Plot the original data X and the eigenvectors to a single diagram.Note that, in general if $u_i$ is an eigenvector of the matrix $M$ with eigenvalue $\lambda_i$ then $\alpha \cdot u_i$ is also an eigenvector of $M$ with the same eigenvalue $\lambda_i$, where $\alpha$ is an arbitrary scalar (including $\alpha=-1$). Thus, the signs of the eigenvectors are arbitrary, and you can flip them without changing the meaning of the result. Only their direction matters. The particular result depends on the algorithm used to find them.<jupyter_code># plot the original data
plt.scatter(X[:, 0], X[:, 1])

# plot the mean of the data
mean_d1, mean_d2 = X.mean(0)
plt.plot(mean_d1, mean_d2, 'o', markersize=10, color='red', alpha=0.5)

# calculate the covariance matrix
Sigma = get_covariance(X)
# calculate the eigenvector and eigenvalues of Sigma
L, U = get_eigen(Sigma)

plt.arrow(mean_d1, mean_d2, U[0, 0], U[1, 0], width=0.01, color='red', alpha=0.5)
plt.arrow(mean_d1, mean_d2, U[0, 1], U[1, 1], width=0.01, color='red', alpha=0.5)<jupyter_output><empty_output><jupyter_text>What do you observe in the above plot? Which eigenvector corresponds to the smallest eigenvalue?Write your answer here:[YOUR ANSWER]  Task 4: Transform the data Determine the smallest eigenvalue and remove its corresponding eigenvector. The remaining eigenvector is the basis of a new subspace. Transform all vectors in X in this new subspace by expressing all vectors in X in this new basis.<jupyter_code>def transform(X, U, L):
    """Transforms the data in the new subspace spanned by the eigenvector corresponding to the largest eigenvalue.
    
    Parameters
    ----------
    X : array, shape [N, D]
        Data matrix.
    L : array, shape [D]
        Eigenvalues of Sigma_X
    U : array, shape [D, D]
        Eigenvectors of Sigma_X
        
    Returns
    -------
    X_t : array, shape [N, 1]
        Transformed data
        
    """
    X_t = X @ U[:,1]
    return None
X_t = transform(X, U, L)<jupyter_output><empty_output><jupyter_text>SVD  Task 5: Given the matrix $M$ find its SVD decomposition $M= U \cdot \Sigma \cdot V$ and reduce it to one dimension using the approach described in the lecture.<jupyter_code>M = np.array([[1, 2], [6, 3],[0, 2]])
def reduce_to_one_dimension(M):
    """Reduces the input matrix to one dimension using its SVD decomposition.
    
    Parameters
    ----------
    M : array, shape [N, D]
        Input matrix.
        
    Returns
    -------
    M_t: array, shape [N, 1]
        Reduce matrix.
    
    """
    
    U, sigma, V = np.linalg.svd(M, 0)
    Sigma = np.diag(sigma)
    
    # truncated Sigma
    trunc_sigma = Sigma.copy()
    trunc_sigma[1, 1] = 0
    
    # M_t = U @ trunc_sigma
    M_t = M @ V[:, 0]
    
    return M_t
M_t = reduce_to_one_dimension(M)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10996563573883161, 'fraction_numerical': 0.017543859649122806, 'mean_word_length': 3.1925701288855195, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '108552', 'license': 'MIT', 'n_tokens_mistral': 1794, 'n_tokens_neox': 1683, 'n_words': 664, 'path': 'Informatics/Machine Learning/ML (IN2064) - TUM/Week_10/exercise_10_notebook.ipynb', 'repo_name': 'MarcosSalib/Cocktail_MOOC', 'size': '5529'}