<jupyter_start><jupyter_text>Matrix Decomposition Exercise<jupyter_code># import numpy
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd<jupyter_output><empty_output><jupyter_text>**Task:** Generate a random matrix 'A' with a shape of 150x50, using the '**np.random.rand**' function.<jupyter_code>A = np.random.rand(150,50)<jupyter_output><empty_output><jupyter_text>**Task:** Check the shape of 'A'.<jupyter_code>A.shape
pd.DataFrame(A).head()<jupyter_output><empty_output><jupyter_text>**Task:** Decompose matrix 'A' with SVD using numpy (decompose into matrices U,D,V).<jupyter_code>U, D, V = np.linalg.svd(A, full_matrices=False)<jupyter_output><empty_output><jupyter_text>**Task:** Check the shapes of matrices U,D,V.<jupyter_code>print(U.shape, D.shape, V.shape)<jupyter_output>(150, 50) (50,) (50, 50)<jupyter_text>**Task:** Reconstruct matrix 'A' from matrices U,D,V.<jupyter_code># for i in range(1, 51): # Gotta do 1 to size of D(sigma)
#     A_rec = np.matrix(U[:, :i]) * np.diag(D[:i]) * np.matrix(V[:i, :]) # udot * diagV
A_rec = U.dot(np.diag(D)).dot(V) # this is way faster and easier
A_rec<jupyter_output><empty_output><jupyter_text>**Task:** Compare matrices 'A' and 'A_rec' with the [**np.allclose**](https://numpy.org/doc/stable/reference/generated/numpy.allclose.html) function.<jupyter_code>np.allclose(A, A_rec)<jupyter_output><empty_output><jupyter_text>**Task:** Reduce the dimnesion of matrix 'A' to shape 150x20 with SVD (try both equations).<jupyter_code>aa = U[:,:20].dot(np.diag(D[:20])) # one way, just drop the unwanted columns and rows.
aa.shape
A_reconstruct = A.dot(V[:20,:].T) # 150,50 A dot product... times only 20 of V. 
# Need to transpose for dot to work<jupyter_output><empty_output><jupyter_text>**Task:** Print the shape.<jupyter_code>A_reconstruct.shape<jupyter_output><empty_output><jupyter_text>**Task:** Compare the two results with the [**np.allclose**](https://numpy.org/doc/stable/reference/generated/numpy.allclose.html) function.<jupyter_code>np.allclose(aa,A_reconstruct)<jupyter_output><empty_output><jupyter_text>**Task:** Filter the noise from matrix 'A' with 20 largest singular vectors (keep the shape of (150x50)).  check noise by plotting<jupyter_code>plt.matshow(A.T)<jupyter_output><empty_output><jupyter_text>first attempt with summing rows?<jupyter_code>row_sums = A.sum(axis=1)
new_matrix = A / row_sums[:, np.newaxis]
plt.matshow(new_matrix.T)<jupyter_output><empty_output><jupyter_text>**Task:** Check the shape.<jupyter_code>new_matrix.shape
row_sums2 = new_matrix.sum(axis=1)
new_matrix2 = new_matrix / row_sums[:, np.newaxis]
new_matrix2.shape
plt.matshow(new_matrix2.T)<jupyter_output><empty_output><jupyter_text>second attempt by multiplying only 1 row/column USING SVD<jupyter_code>noise_reduction = np.matrix(U[:, :1]) * np.diag(D[:1]) * np.matrix(V[:1, :])
plt.matshow(noise_reduction.T) # wtf way better<jupyter_output><empty_output><jupyter_text>**Task:** Define the function `'SVD'` which perform singular values decomposition.Do not forget to hceck the shape of the input matrix.<jupyter_code>"""
PARAMS:
    A (numpy.ndarray) - matrix to decompose
RETURN:
    U (numpy.ndarray) - left singular vectors
    SV (numpy.ndarray) - singular values
    V (numpy.ndarray) - right singular vectors
"""
def SVD(mx):
    print("mx shape: ", mx.shape)
    U, D, V = np.linalg.svd(mx)
    return U.shape, D.shape, V.shape<jupyter_output><empty_output><jupyter_text>**Task:** Call the function `'SVD'` on matrix 'A'.<jupyter_code>SVD(A)<jupyter_output>mx shape:  (150, 50)
{'fraction_non_alphanumeric': 0.15393258426966291, 'fraction_numerical': 0.018258426966292134, 'mean_word_length': 6.826373626373626, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 63, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '411552', 'license': 'MIT', 'n_tokens_mistral': 1413, 'n_tokens_neox': 1346, 'n_words': 284, 'path': 'w3/w3-day_5/.ipynb_checkpoints/matrix_decomposition_exercise-checkpoint.ipynb', 'repo_name': 'bmskarate/lighthouseMain', 'size': '3560'}