<jupyter_start><jupyter_text>Singular Value Decomposition What is the singular value decomposition?The Singular-Value Decomposition, or SVD for short, is a matrixdecomposition method for reducing a matrix to its constituent partsin order to make certain subsequent matrix calculations simpler. Forthe case of simplicity we will focus on the SVD for real-valued matricesand ignore the case for complex numbers.    $$A = U . \Sigma . V^{T}$$Where $A$ is the real $n \times m$ matrix that we wish to decompose, $U$ is an$m \times m$ matrix, $\Sigma$ represented by the uppercase Greek letter sigma)is an $m \times n$ diagonal matrix, and $V^{T}$ is the $V$ transpose of an $n \times n$ matrix where $T$ is a superscript. Calculate Singular-Value Decomposition:<jupyter_code>from numpy import array
from scipy.linalg import svd

# defining a matrix
A = array([
    [1, 2],
    [3, 4],
    [5, 6]
])

print(f"A: \n{A}\n")

# factorizing
U, s, V = svd(A)
print(f"U: \n{U}\n")
print(f"s: {s}\n")
print(f"V: \n{V}\n")<jupyter_output>A: 
[[1 2]
 [3 4]
 [5 6]]

U: 
[[-0.2298477   0.88346102  0.40824829]
 [-0.52474482  0.24078249 -0.81649658]
 [-0.81964194 -0.40189603  0.40824829]]

s: [9.52551809 0.51430058]

V: 
[[-0.61962948 -0.78489445]
 [-0.78489445  0.61962948]]<jupyter_text>Reconstruct Matrix:The original matrix can be reconstructed from the $U$ , $\Sigma$ and $V^{T}$ elements.The $U$, $s$, and $V$ elements returned from the $svd()$ cannot be multiplied directly.The $s$ vector must be converted into a diagonal matrix using the $diag()$ function.By default, this function will create a square matrix that is $m \times m$, relative toour original matrix. This causes a problem as the size of the matrices do not fit the rules of matrix multiplication, where the number of columns in a matrix must match thenumber of rows in the subsequent matrix. After creating the square $\Sigma$ diagonal matrix,the sizes of the matrices are relative to the original $n \times m$ matrix that we aredecomposing, as follows:$$U(m \times m).\Sigma(m \times m). V^{T}(n \times n)$$Where, in fact, we require:$$U(m \times m).\Sigma(m \times n). V^{T}(n \times n)$$We can achieve this by creating a new $\Sigma$ matrix of all zero values that is $m \times n$ (e.g. morerows) and populate the first $n \times n$ part of the matrix with the square diagonal matrix calculatedvia $diag()$.<jupyter_code># reconstructing rectangular matrix fro svd

from numpy import array
from numpy import diag
from numpy import zeros
from scipy.linalg import svd

# defining a matrix
A = array([
    [1, 2],
    [3, 4],
    [5, 6]
])
print(f"A: \n{A}\n")

# factorizing
U, s, V = svd(A)

# creating m*n Sigma matrix
Sigma = zeros((A.shape[0], A.shape[1]))

# populate Sigma with n*n diagonal matrix
Sigma[:A.shape[1], :A.shape[1]] = diag(s)

# reconstructing matrix
B = U.dot(Sigma.dot(V))
print(f"B: \n{B}")<jupyter_output>A: 
[[1 2]
 [3 4]
 [5 6]]

B: 
[[1. 2.]
 [3. 4.]
 [5. 6.]]<jupyter_text>The above complication with the $\Sigma$ diagonal only exists with the case where m and n arenot equal. The diagonal matrix can be used directly when reconstructing a square matrix, asfollows:<jupyter_code>from numpy import array
from numpy import diag
from scipy.linalg import svd

# defining a matrix
A = array([
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
])
print(f"A: \n{A}\n")

# factorizing
U, s, V = svd(A)

# creating n*n Sigma matrix
Sigma = diag(s)

# reconstructing matrix
B = U.dot(Sigma.dot(V))
print(f"B: \n{B}")<jupyter_output>A: 
[[1 2 3]
 [4 5 6]
 [7 8 9]]

B: 
[[1. 2. 3.]
 [4. 5. 6.]
 [7. 8. 9.]]<jupyter_text>PseudoinverseThe pseudoinverse is the generalization of the matrix inverse for square matrices to rectangularmatrices where the number of rows and columns are not equal. It is also called the Moore-PenroseInverse after two independent discoverers of the method or the Generalized Inverse.The pseudoinverse is denoted as $A^{+}$, where $A$ is the matrix that is being inverted and $+$ is asuperscript. The pseudoinverse is calculated using the singular value decomposition of $A$:$$A^{+} = V . D^{+} . U^{T}$$The $D^{+}$ can be calculated by creating a diagonal matrix from $\Sigma$, calculating the reciprocalof each non-zero element in $\Sigma$, and taking the transpose if the original matrix was rectangular.$$\begin{equation*}\Sigma = \begin{pmatrix}s_{1,1} & 0 & 0 \\0 & s_{2,2} & 0 \\0 & 0 & s_{3,3}\end{pmatrix}\end{equation*}$$$$\begin{equation*}D^{+} = \begin{pmatrix}\frac{1}{s_{1,1}} & 0 & 0 \\0 & \frac{1}{s_{2,2}} & 0 \\0 & 0 & \frac{1}{s_{3,3}}\end{pmatrix}\end{equation*}$$<jupyter_code># pseudoinverse
from numpy.linalg import pinv

# defining an array
A = array([
    [0.1, 0.2],
    [0.3, 0.4],
    [0.5, 0.6],
    [0.7, 0.8]
])
print(f"A: \n{A}\n")

# calculating pseudoinverse
B = pinv(A)
print(f"B: \n{B}")<jupyter_output>A: 
[[0.1 0.2]
 [0.3 0.4]
 [0.5 0.6]
 [0.7 0.8]]

B: 
[[-1.00000000e+01 -5.00000000e+00  1.42385628e-14  5.00000000e+00]
 [ 8.50000000e+00  4.50000000e+00  5.00000000e-01 -3.50000000e+00]]<jupyter_text>We can calculate the pseudoinverse manually via the SVD and compare the results to the$pinv()$ function. First we must calculate the $SVD$. Next we must calculate the reciprocal ofeach value in the s array. Then the $s$ array can be transformed into a diagonal matrix with anadded row of zeros to make it rectangular. Finally, we can calculate the pseudoinverse from theelements. The specific implementation is:$$A^{+} = V^{T} . D^{T} . U^{T}$$<jupyter_code># pseudoinverse using svd
# defining an array
A = array([
    [0.1, 0.2],
    [0.3, 0.4],
    [0.5, 0.6],
    [0.7, 0.8]
])
print(f"A: \n{A}\n")

# factorizing
U, s, V = svd(A)

#reciprocals of s
d = 1.0 / s

# creating m*n D matrix
D = zeros(A.shape)

# populate D with n*n diagonal matrix
D[:A.shape[1], :A.shape[1]] = diag(d)

# calculating pseudoinverse
B = V.T.dot(D.T).dot(U.T)
print(f"B: \n{B}")<jupyter_output>A: 
[[0.1 0.2]
 [0.3 0.4]
 [0.5 0.6]
 [0.7 0.8]]

B: 
[[-1.00000000e+01 -5.00000000e+00  1.42578328e-14  5.00000000e+00]
 [ 8.50000000e+00  4.50000000e+00  5.00000000e-01 -3.50000000e+00]]<jupyter_text>Dimensionality ReductionA popular application of $SVD$ is for dimensionality reduction. Data with a large numberof features, such as more features (columns) than observations (rows) may be reduced to a smaller subset of features that are most relevant to the prediction problem. The result is a matrix with a lower rank that is said to approximate the original matrix. To do this we can perform an $SVD$ operation on the original data and select the top $k$ largest singular values in $\Sigma$. These columns can be selected from $\Sigma$ and the rows selectedfrom $V^{T}$ . An approximate $B$ of the original vector $A$ can then be reconstructed.$$B = U . \Sigma_{k} . V_{k}^{T}$$<jupyter_code># data reduction using svd

from numpy import array
from numpy import diag
from numpy import zeros
from scipy.linalg import svd

# defining an array
A = array([
    [ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10],
    [11, 12, 13, 14, 15, 16, 17, 18, 19, 20],
    [21, 22, 23, 24, 25, 26, 27, 28, 29, 30]
])
print(f"A: \n{A}\n")

# factorizing
U, s, V = svd(A)

# creating m*n Sigma matrix
Sigma = zeros((A.shape[0], A.shape[1]))

# populating Sigma with n*n diagonal matrix
Sigma[:A.shape[0], :A.shape[0]] = diag(s)

# selecting
n_elements = 2
Sigma = Sigma[:, :n_elements]
V = V[:n_elements, :]

# reconstructing
B = U.dot(Sigma.dot(V))
print(f"B: \n{B}\n")

# transforming
T = U.dot(Sigma)
print(f"T: \n{T}\n")
T = A.dot(V.T)
print(f"T: \n{T}")
# svd data reduction in sklearn

from sklearn.decomposition import TruncatedSVD

# creating transform
svd = TruncatedSVD(n_components=2)

# fitting transform
svd.fit(A)

# Applying transform
result = svd.transform(A)
print(f"result: \n{result}")<jupyter_output>result: 
[[18.52157747  6.47697214]
 [49.81310011  1.91182038]
 [81.10462276 -2.65333138]]
{'fraction_non_alphanumeric': 0.14593089482344007, 'fraction_numerical': 0.07340842931274522, 'mean_word_length': 4.023521932612842, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 14, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '113807', 'license': 'MIT', 'n_tokens_mistral': 3277, 'n_tokens_neox': 2883, 'n_words': 1037, 'path': 'src/Number_13.ipynb', 'repo_name': 'EverLookNeverSee/numpy_and_algebra', 'size': '7901'}