<jupyter_start><jupyter_text>14. Matrix DecompositionsMatrix decompositions are methods that reduce a matrix into constituent parts that make it easier to calculate more complex matrix operations. Matrix decomposition methods, also called matrix factorization methods, are a foundation of linear algebra in computers, even for basic operations such as solving systems of linear equations, calculating the inverse, and calculating the determinant of a matrix. 14.2 What is a Matrix DecompositionA matrix decomposition is a way of reducing a matrix into its constituent parts. It is an approach that can simplify more complex matrix operations that can be performed on the decomposed matrix rather than on the original matrix itself. A common analogy for matrix decomposition is the factoring of numbers, such as the factoring of 10 into 2 x 5. For this reason, matrix decomposition is also called matrix factorization. Like factoring real values, there are many ways to decompose a matrix, hence there are a range of different matrix decomposition techniques. Two simple and widely used matrix decomposition methods are the LU matrix decomposition and the QR matrix decomposition. 14.3 LU DecompositionThe LU decomposition is for square matrices and decomposes a matrix into L and U components. Where A is the square matrix that we wish to decompose, L is the lower triangle matrix and U is the upper triangle matrix.                            A = L  U                             The LU decomposition is found using an iterative numerical process and can fail for those matrices that cannot be decomposed or decomposed easily. A variation of this decomposition that is numerically more stable to solve in practice is called the LUP decomposition, or the LU decomposition with partial pivoting.                                                        A = L  U  P                             The rows of the parent matrix are re-ordered to simplify the decomposition process and the additional P matrix specifies a way to permute the result or return the result to the original order.                             14.4 QR DecompositionThe QR decomposition is for n x m matrices (not limited to square matrices) and decomposes a matrix into Q and R components.                            A = Q  R                             Where A is the matrix that we wish to decompose, Q a matrix with the size m x m, and R is an upper triangle matrix with the size m x n. The QR decomposition is found using an iterative numerical method that can fail for those matrices that cannot be decomposed, or decomposed easily. Like the LU decomposition, the QR decomposition is often used to solve systems of linear equations, although is not limited to square matrices.                            The QR decomposition can be implemented in NumPy using the qr() function. By default, the function returns the Q and R matrices with smaller or reduced dimensions that is more economical. 14.5 Cholesky DecompositionThe Cholesky decomposition is for square symmetric matrices where all values are greater than zero, so-called positive definite matrices. The decomposition is defined as                             A = L  L^T                             Where A is the matrix being decomposed, L is the lower triangular matrix and L^T is the transpose of L.                            The Cholesky decomposition is used for solving linear least squares for linear regression, as well as simulation and optimization methods. When decomposing symmetric matrices, the Cholesky decomposition is nearly twice as ecient as theLU decomposition and should be preferred in these cases. The Cholesky decomposition can be implemented in NumPy by calling the cholesky() function.<jupyter_code>from numpy import array
from scipy.linalg import lu
from numpy.linalg import qr
from numpy.linalg import cholesky

# The LU decomposition is for square matrices and decomposes a matrix into L and U components.
# decomposition is defined as  A = L  U  P

A = array([
[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
P, L, U = lu(A) # LUP decomposition method and P, L, and U components of the PLU decomposition

# print(P) # P matrix specifies a way to permute the result or return the result to the original order. 
# print(L) # L is the lower triangle matrix  
# print(U) # U is the upper triangle matrix.
# print(P.dot(L).dot(U)) # original matrix is reconstructed


# The QR decomposition is for n x m matrices (not limited to m x m matrices) and decomposes a matrix into Q and R 
# components. decomposition is defined as A = Q  R

A = array([
[1, 2],
[3, 4],
[5, 6]])
Q, R = qr(A, 'complete') # Factorize rectangular matrix A and Q, R are the component of QR decompoosition

# print(Q) # Q a matrix with the size m x m
# print(R) # R is an upper triangle matrix with the size m x n.
# print(Q.dot(R)) # reconstruct original matrix A

# The Cholesky decomposition is for square symmetric matrices where all values are greater than zero, so-called 
# positive definite matrices. decomposition is defined as A = L  L^T

A = array([
[2, 1, 1],
[1, 2, 1], 
[1, 1, 2]])

L = cholesky(A) # Cholesky decomposition method factorize matrix A and L is component of Cholesky decomposition
print(L) # L is the lower triangular matrix  
print(L.T) #L^T is the transpose of matrix L.
print(L.dot(L.T)) # Reconstruct Orginal Matrix<jupyter_output>[[1.41421356 0.         0.        ]
 [0.70710678 1.22474487 0.        ]
 [0.70710678 0.40824829 1.15470054]]
[[1.41421356 0.70710678 0.70710678]
 [0.         1.22474487 0.40824829]
 [0.         0.         1.15470054]]
[[2. 1. 1.]
 [1. 2. 1.]
 [1. 1. 2.]]<jupyter_text>15. EigendecompositionMatrix decompositions are a useful tool for reducing a matrix to their constituent parts in order to simplify a range of more complex operations. Perhaps the most used type of matrix decomposition is the eigendecomposition that decomposes a matrix into eigenvectors and eigenvalues. This decomposition also plays a role in methods used in machine learning, such as in the Principal Component Analysis method or PCA.15.2 Eigendecomposition of a MatrixEigendecomposition of a matrix is a type of decomposition that involves decomposing a square matrix into a set of eigenvectors and eigenvalues. i.e. matrix decomposition is called eigendecomposition, in which we decompose a matrix into a set of eigenvectors and eigenvalues                                A  v = (Lambda)  v                                  This is  he eigenvalue equation, where A is the parent square matrix that we are decomposing, v is the eigenvector of the matrix, and 'lambda' is the lowercase Greek letter lambda and represents the eigenvalue scalar.   A matrix could have one eigenvector and eigenvalue for each dimension of the parent matrix. Not all square matrices can be decomposed into eigenvectors and eigenvalues, and some can only be decomposed in a way that requires complex numbers. The parent matrix can be shown to be a product of the eigenvectors and eigenvalues.                                          A = Q  /\  Q^T                                          Where Q is a matrix comprised of the eigenvectors, /\ is the uppercase Greek letter lambda and is the diagonal matrix comprised of the eigenvalues, and Q^T is the transpose of the matrix comprised of the eigenvectors. Eigendecomposition is used as an element to simplify the calculation of other more complex matrix operations.15.3 Eigenvectors and EigenvaluesEigenvectors are unit vectors, which means that their length or magnitude is equal to 1.0, Eigenvectors are unit vectors, which means that their length or magnitude is equal to 1.0, referred as right vectors, which simply means a column vector.Eigenvalues are coeficients applied to eigenvectors that give the vectors their length or magnitude. 15.4 Calculation of EigendecompositionAn eigendecomposition is calculated on a square matrix using an efficient iterative algorithm. Often an eigenvalue is found first, then an eigenvector is found to solve the equation as a set of coefficients. he eigendecomposition can be calculated in NumPy using the eig() function.                                          15.5 Confirm an Eigenvector and EigenvalueWe can confirm that a vector is indeed an eigenvector of a matrix by by multiplying the candidate eigenvector by the value vector and comparing the result with the eigenvalue. First, we will define a matrix, then calculate the eigenvalues and eigenvectors then test whether the first vector and value are in fact an eigenvalue and eigenvector for the matrix. The eigenvectors are returned as a matrix with the same dimensions as the parent matrix, where each column is an eigenvector, e.g. the first eigenvector is vectors[:, 0]. Eigenvalues are returned as a list, where value indices in the returned array are paired with eigenvectors by column index, e.g. the first eigenvalue at values[0] is paired with the first eigenvector at vectors[:, 0].15.6 Reconstruct MatrixWe can reverse the process and reconstruct the original matrix given only the eigenvectors and eigenvalues. First, the list of eigenvectors must be taken together as a matrix, where each vector becomes a row. The eigenvalues need to be arranged into a diagonal matrix. The NumPy diag() function can be used for this. Next, we need to calculate the inverse of the eigenvector matrix, which we can achieve with the inv() NumPy function. Finally, these elements need tobe multiplied together with the dot() function.<jupyter_code>from numpy import array
from numpy import diag
from numpy.linalg import eig
from numpy.linalg import inv

A = array([
[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])

values, vectors = eig(A) # factorize matrix A into eigen value and vector

print(A)
print(values)
print(vectors) # 
print(vectors[:,0])
print(vectors[0])
print(A.dot(vectors[:,0])) # confirm first eigenvector i.e. original matrix with the first eigenvector 
print(vectors[:,0] * values[0]) # first eigenvector multiplied by the first eigenvalue. 

# the results of these two multiplications show the same resulting vector, as we would expect.
Q = vectors #create matrix from eigenvectors
R = inv(vectors) #create inverse of eigenvectors matrix
L = diag(values) #create diagonal matrix from eigenvalues

#print(Q.dot(L).dot(R)) #reconstruct the original matrix<jupyter_output>[[1 2 3]
 [4 5 6]
 [7 8 9]]
[ 1.61168440e+01 -1.11684397e+00 -9.75918483e-16]
[[-0.23197069 -0.78583024  0.40824829]
 [-0.52532209 -0.08675134 -0.81649658]
 [-0.8186735   0.61232756  0.40824829]]
[-0.23197069 -0.52532209 -0.8186735 ]
[-0.23197069 -0.78583024  0.40824829]
[ -3.73863537  -8.46653421 -13.19443305]
[ -3.73863537  -8.46653421 -13.19443305]<jupyter_text>16. Singular Value DecompositionThe most known and widely used matrix decomposition or matrix factorization method is the Singular-Value Decomposition, or SVD. All matrices have an SVD, which makes it more stable than other methods, such as the eigendecomposition.  it is often used in a wide array of applications including compressing, denoising, and data reduction. 16.2 What is the Singular-Value DecompositionIt is a matrix decomposition method for educing a matrix to its constituent parts in order to make certain subsequent matrix calculations simpler. For the case of simplicity we will focus on the SVD for real-valued matrices                         A = U  E  V^T                        Where A is the real n x m matrix that we wish to decompose, U is an m x m matrix, E (sigma) is an m x n diagonal matrix, and V^T is the V transpose of an n x n matrix where T is a superscript.The singular value decomposition (SVD) provides another way to factorize a matrix, into singular vectors and singular values. The SVD allows us to discover some of the same kind of information as the eigendecomposition. However, the SVD is more generally applicable.16.3 Calculate Singular-Value DecompositionThe SVD can be calculated by calling the svd() function. The function takes a matrix and returns the U, E and V^T elements. The E diagonal matrix is returned as a vector of singular values. The V matrix is returned in a transposed form, e.g. V^T. 16.4 Reconstruct MatrixThe original matrix can be reconstructed from the U, E, and V^T elements. The s vector must be converted into a diagonal matrix using the diag() function. By default, this function will create a square matrix that is m x m, relative to our original matrix. This causes a problem as the size of the matrices do not fit the rules of matrix multiplication, where the number of columns in a matrix must match the number of rows in the subsequent matrix. After creating the square E diagonal matrix, the sizes of the matrices are relative to the original n x m matrix that we are decomposing as                        U(m x m)  E(m x m)  V^T(n x n)                         16.5 PseudoinverseThe pseudoinverse is the generalization of the matrix inverse for square matrices to rectangular matrices where the number of rows and columns are not equal. It is also called the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse.The pseudoinverse is denoted as A+, where A is the matrix that is being inverted and + is superscript. The pseudoinverse is calculated using the singular value decomposition of A:                        A+= V  D+ U^T                        Where A+ is the pseudoinverse, D+ is the pseudoinverse of the diagonal matrix  and V is the transpose of V^T . We can get U and V from the SVD operation                        16.6 Dimensionality Reduction A popular application of SVD is for dimensionality reduction. Data with a large number of features, such as more features (columns) than observations (rows) may be reduced to a smaller subset of features that are most relevant to the prediction problem. The result is a matrix with a lower rank that is said to approximate the original matrix.  To do this we can perform an SVD operation on the original data and select the top k largest singular values in . These columns can be selected from E and the rows selected from V^T. An approximate B of the original vector A can then be reconstructed.  B = U  k V kn natural language processing, this approach can be used on matrices of word occurrencesor word frequencies in documents and is called Latent Semantic Analysis or Latent Semantic Indexing. The scikit-learn provides a TruncatedSVD class that implements this capability directly. The TruncatedSVD class can be created in which you must specify the number of desirable features or components to select, e.g. 2. Once created, you can t the transform (e.g. calculate V) by calling the fit() function, then apply it to the original matrix by calling the transform() function. The result is the transform of A called T above.<jupyter_code>from numpy import array
from numpy import diag
from numpy import zeros
from scipy.linalg import svd
from numpy.linalg import pinv
from sklearn.decomposition import TruncatedSVD

A = array([[1, 2],[3, 4],[5, 6]])

# The singular value decomposition (SVD) method for reducing a matrix to its constituent parts and  make certain 
# subsequent matrix calculations simpler

U,s,V = svd(A) # svd factorize matrix A in to its constituent parts
# print(U) # U is an m x m matrix
# print(s) #(sigma) is an m x n diagonal matrix
# print(V) # V matrix is returned in a transposed form, e.g. V^T

#print(zeros([A.shape[0], A.shape[1]]))
#print(A.shape[0], A.shape[1])
sigma = zeros([A.shape[0], A.shape[1]]) #create mxn Sigma matrix
#print(sigma)

sigma[:A.shape[1], :A.shape[1]] = diag(s) #populate Sigma with n x n diagonal matrix
#print(diag(s))

sigma = diag(s)
# print(U.dot(sigma.dot(V))) #reconstruct matrix

A = array([
[1, 2, 3],
[4, 5, 6],
[7, 8, 9]]) 

U,s,V = svd(A) # svd factorize

sigma = diag(s)
#print(U.dot(sigma.dot(V))) #reconstruct matrix

## The pseudoinverse is the generalization of the matrix inverse for square matrices to rectangular matrices where 
## the number of rows and columns are not equal.
A = array([
[0.1, 0.2],
[0.3, 0.4],
[0.5, 0.6],
[0.7, 0.8]])

# print(pinv(A)) # print pseudoinverse

##### PCA Example

# A popular application of SVD is for dimensionality reduction. Data with a large number of features, such as more 
# features (columns) than observations (rows) may be reduced to a smaller subset of features that are most relevant 
# to the prediction problem.

A = array([
[1,2,3,4,5,6,7,8,9,10],
[11,12,13,14,15,16,17,18,19,20],
[21,22,23,24,25,26,27,28,29,30]])

U,s,V = svd(A) # svd factorize operation on the original matrix A and select the top k largest singular values 

sigma = zeros([A.shape[0], A.shape[1]]) #create mxn Sigma matrix
sigma[:A.shape[0], :A.shape[0]] = diag(s) #populate Sigma with n x n diagonal matrix
n_element = 2 
sigma = sigma[:,:n_element]
V = V[:n_element,:]
B = U.dot(sigma.dot(V))
T = U.dot(sigma)
T = A.dot(V.T)
#print(T)

####svd data reduction in scikit-learn

# The scikit-learn provides a TruncatedSVD class that implements this capability directly.

svd = TruncatedSVD(n_components = 2) #TruncatedSVD class created to specify the no of features (components) to select(2)
svd.fit(A) #calculate V by calling the fit() function

print(svd.transform(A)) #apply fit function to matrix A by calling the transform() function<jupyter_output>[[18.52157747  6.47697214]
 [49.81310011  1.91182038]
 [81.10462276 -2.65333138]]
{'fraction_non_alphanumeric': 0.05998624957029907, 'fraction_numerical': 0.03471983499484359, 'mean_word_length': 3.664617851416355, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '94791', 'license': 'Unlicense', 'n_tokens_mistral': 4981, 'n_tokens_neox': 4459, 'n_words': 2540, 'path': 'Factorization.ipynb', 'repo_name': '5atish/Jupyter-Notebook', 'size': '17454'}