<jupyter_start><jupyter_text>Matrices  -----  Notation and Operations  Matrix notationMatrix Notation is a notation system that allows succinct representation of complex operations, such as a change of basis.  * **Matlab** is based on Matrix Notation.* **Python**: similar functionality by using **numpy** Recall that a **vector** can be represented as a one dimensional array of numbers. A **matrix** is a two dimensional rectangle of numbers. A matrix consists of rows, indexed from the top to the bottom and of columns, indexed from the left to the right. As is described in the  figure.A matrix with $n$ rows and $m$ columns is said to be an "$m$ by $n$ matrix".In numpy we will say that the **shape** of the matrix is $(m,n)$. We will also use the LaTeX notation $M_{m \times n}$ to indicate that $M$ is an $m \times n$ matrix.  Transposing a MatrixAt times it is useful to switch the rows and column dimensions of matrices.  Consider the matrix$$\begin{equation}	A=\begin{bmatrix}	  a_{11} & a_{12} \\	  a_{21} & a_{22} \\	  a_{31} & a_{32} 		\end{bmatrix}\end{equation}$$The transpose of A  is$$\begin{equation}   A^{\mathsf{T}}=\begin{bmatrix}	  a_{11} & a_{21} & a_{31} \\	  a_{12} & a_{22} & a_{32} \\	\end{bmatrix}\end{equation}$$<jupyter_code>import numpy as np
# The .reshape command reorganized the elements of a matrix into a new shape
A = np.array(range(6))
print('A=',A)
B=A.reshape(2,3)
print("B is a 2X3 matrix:\n",B)
print("the shape of B is:",B.shape)
print("The transpose of B is\n",B.T)
print("the shape of B.T is:",B.T.shape)<jupyter_output><empty_output><jupyter_text>Vectors as matrices.When using matrix notation, vectors can be represented as either [row or column vectors](https://en.wikipedia.org/wiki/Row_and_column_vectors). In a matrix context, a vector $\vec{v}$ is denoted by a bold-face letter. ${\bf v}$ for a column vector and ${\bf v}^\top$ for row vector: * By default a vector is represented as a **column vector** which is a matrix consisting of a single column:$$\begin{equation}{\bf v}=	\begin{bmatrix}	  v_1 \\      v_2 \\      \vdots \\	  v_d	\end{bmatrix}\end{equation}$$ * If $\vec{v}$ is a column vector then the **transpose** of $\vec{v}$, denoted by $\vec{v}^\top$ is a **row vector** which is a matrix consists of a single row:$$\begin{equation}{\bf v}^{\top}=	\begin{bmatrix}	  v_1 & v_2 & \cdots & v_d	\end{bmatrix}\end{equation}$$  A vector as a matrixRow and Column vectors can be thought of as matrices.* The column vector ${\bf v}$ is a $d \times 1$ matrix.* The row vector ${\bf v}^{\top}$ is a $1 \times d$ matrix.  A matrix as a collection of vectorsMatrices can be represented as a collection of vectors. For example, consider the $2\times 3$ matrix ${\bf A}=\begin{bmatrix}	  a_{11} & a_{12} & a_{13}\\	  a_{21} & a_{22} & a_{23}		\end{bmatrix}$ We can represent ${\bf A}=\begin{bmatrix}	  a_{11} & a_{12} & a_{13}\\	  a_{21} & a_{22} & a_{23}		\end{bmatrix}$ as vectors in one of two ways:* As a row of column vectors:$$ {\bf A} = \begin{bmatrix} {\bf c}_1 , {\bf c}_2 , {\bf c}_3 \end{bmatrix}$$where$$   {\bf c}_1=\begin{bmatrix} a_{11}\\ a_{21} \end{bmatrix},   {\bf c}_2=\begin{bmatrix} a_{12}\\ a_{22} \end{bmatrix},    {\bf c}_3=\begin{bmatrix} a_{13}\\ a_{23} \end{bmatrix}$$ * Or as a column of row vectors: ${\bf A} = \begin{bmatrix} {\bf r}_1 \\ {\bf r}_2 \end{bmatrix}$  where $   {\bf r}_1=\begin{bmatrix} a_{11}, a_{12}, a_{13} \end{bmatrix},   {\bf r}_2=\begin{bmatrix} a_{21}, a_{22}, a_{23} \end{bmatrix}, $<jupyter_code>A=np.array(range(6)).reshape(2,3)
print('A=\n',A)
print("Splitting A into columns:")
Columns=np.split(A,3,axis=1)
for i in range(len(Columns)):
    print('column %d'%i)
    print(Columns[i])
A_recon=np.concatenate(Columns,axis=1)
print('reconstructing the matrix from the columns:')
print(A_recon)
print('Checking that the reconstruction is equal to the original')
print(A_recon==A)
print("Splitting A into rows:")
Rows=np.split(A,2,axis=0)

for i in range(len(Rows)):
    print('row %d'%i)
    print(Rows[i])
A_recon=np.concatenate(Rows,axis=0)

print('reconstructing the matrix from the rows:')
print(A_recon)
print('Checking that the reconstruction is equal to the original')
print(A_recon==A)<jupyter_output><empty_output><jupyter_text>Numpy functionsBeyond the commands `reshape`, `split` and `concatanate` numpy has a rich set of functions to manipulate arrays, for a complete list see [Numpy Array Manipulation routines](https://docs.scipy.org/doc/numpy/reference/routines.array-manipulation.html)  Matrix - scalar operationsYou can add/subtract multiply/divide a scalar from a matrix  Adding a scalar value to a matrixLet $A$=$\bigl[ \begin{smallmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{smallmatrix} \bigr]$.  Here is how we would add the scalar $3$ to $A$:$$\begin{equation}	A+3=\begin{bmatrix}	  a_{11} & a_{12} \\	  a_{21} & a_{22} 		\end{bmatrix}+3	=\begin{bmatrix}	  a_{11}+3 & a_{12}+3 \\	  a_{21}+3 & a_{22}+3 		\end{bmatrix}\end{equation}$$  Subtracting a scalar value to a matrixSubstraction is similar$$\begin{equation}	A-3=\begin{bmatrix}	  a_{11} & a_{12} \\	  a_{21} & a_{22} 		\end{bmatrix}-3	=\begin{bmatrix}	  a_{11}-3 & a_{12}-3 \\	  a_{21}-3 & a_{22}-3 		\end{bmatrix}\end{equation}$$  Product of a scalar and a matrixMultiplication is also similar$$\begin{equation}	3 \times A = 3 \times \begin{bmatrix}	  a_{11} & a_{12} \\	  a_{21} & a_{22} 		\end{bmatrix}	=	\begin{bmatrix}	  3a_{11} & 3a_{12} \\	  3a_{21} & 3a_{22} 		\end{bmatrix}\end{equation}$$  Dividing a matrix by a scalarDivision by $a$ is the same as multiplying by $1/a$. Note that you cn divide a matrix by a scalar, but dividing a scalar by a matrix is not defined.$$\begin{equation}	A/5= A \times \frac{1}{5}= \begin{bmatrix}	  a_{11}/5 & a_{12}/5 \\	  a_{21}/5 & a_{22}/5 		\end{bmatrix}\end{equation}$$<jupyter_code># Some examples of matrix-scalar operations using numpy
print('A=\n',A)
print('A+3=3+A=\n',A+3)  # addition

print('A*3=\n',A*3)  # product

print('A/2=\n',A/2)  # integer division
print('A/2.=\n',A/2.)  # floating point division<jupyter_output><empty_output><jupyter_text>Adding and subtracting two matricesLet $A$=$\bigl[ \begin{smallmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{smallmatrix} \bigr]$ and $B$=$\bigl[ \begin{smallmatrix} b_{11} & b_{12} \\ b_{21} & b_{22} \end{smallmatrix} \bigr]$.  To compute $A-B$, subtract each element of B from the corresponding element of A:$	A -B =	\begin{bmatrix}	  a_{11} & a_{12} \\	  a_{21} & a_{22} 		\end{bmatrix} -	\begin{bmatrix} b_{11} & b_{12} \\	  b_{21} & b_{22}	\end{bmatrix} $$	=	\begin{bmatrix}	  a_{11}-b_{11} & a_{12}-b_{12} \\	  a_{21}-b_{21} & a_{22}-b_{22} 		\end{bmatrix}$ Addition works exactly the same way:$	A + B =	\begin{bmatrix}	  a_{11} & a_{12} \\	  a_{21} & a_{22} 		\end{bmatrix} +	\begin{bmatrix} b_{11} & b_{12} \\	  b_{21} & b_{22}	\end{bmatrix} $    $	=	\begin{bmatrix}	  a_{11}+b_{11} & a_{12}+b_{12} \\	  a_{21}+b_{21} & a_{22}+b_{22} 		\end{bmatrix}$ An important point to know about matrix addition and subtraction is that it is only defined when $A$ and $B$ are of the same size.  Here, both are $2 \times 2$.  Since operations are performed element by element, these two matrices must be conformable- and for addition and subtraction that means they must have the same numbers of rows and columns.  I like to be explicit about the dimensions of matrices for checking conformability as I write the equations, so write$$A_{2 \times 2} + B_{2 \times 2}= \begin{bmatrix}  a_{11}+b_{11} & a_{12}+b_{12} \\  a_{21}+b_{21} & a_{22}+b_{22} 	\end{bmatrix}_{2 \times 2}$$Notice that the result of a matrix addition or subtraction operation is always of the same dimension as the two operands.Let's define another matrix, B, that is also $2 \times 2$ and add it to A:<jupyter_code>B = np.random.randn(2,2)
print(B)
try:
    result = A + B
except Exception as e: 
    print(e)<jupyter_output><empty_output><jupyter_text>Matrix-Matrix products  The dot product of two vectors* Recall that a vector is just a skinny matrix.* Consider the dot product $(1,2,3) \cdot (1,1,0) = 1 \times 1 + 2 \times 1 +3 \times 0= 3$. Conventions of dot product in matrix notation:  * The first vector is a row vector and the second vector is a column vector.  * There is no operator ($\cdot$) between the two vectors  $$   \begin{bmatrix} 1,2,3 \end{bmatrix}   \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} = 1 \times 1 + 2 \times 1 +3 \times 0= 3$$  The dot product of a matrix and a vectorTo multiply the matrix ${\bf A}=\begin{bmatrix}	  a_{11} & a_{12} & a_{13}\\	  a_{21} & a_{22} & a_{23}		\end{bmatrix}$by the column vector ${\bf c}=\begin{bmatrix} c_1 \\ c_2 \\ c_3 \end{bmatrix}$   We think of ${\bf A}$ as consisting or two row vectors:${\bf A} = \begin{bmatrix} {\bf r}_1 \\ {\bf r}_2 \end{bmatrix}$  where $   {\bf r}_1=\begin{bmatrix} a_{11}, a_{12}, a_{13} \end{bmatrix},   {\bf r}_2=\begin{bmatrix} a_{21}, a_{22}, a_{23} \end{bmatrix}, $and take the dot products of ${\bf r}_1,{\bf r}_2$ with ${\bf c}$ to create a column vector of dimension 2:${\bf A} {\bf c} = \begin{bmatrix} {\bf r}_1 {\bf c} \\ {\bf r}_2 {\bf c} \end{bmatrix} = \begin{bmatrix}	  a_{11}c_1 + a_{12}c_2 + a_{13} c_3 \\	  a_{21}c_1 + a_{22}c_2 + a_{23} c_3		\end{bmatrix}$  Dot product of two matricesMultiplying a matrix and a column vector can be generalized to multiplying two matrices.To do so we think of Alternatively, consider a matrix ${\bf C}$ of size $2 \times 3$ and a matrix ${\bf A}$ of size $3 \times 2$$$\begin{equation}	{\bf A}=\begin{bmatrix}	  a_{11} & a_{12} \\	  a_{21} & a_{22} \\	  a_{31} & a_{32} 		\end{bmatrix}	,	{\bf C} = 	\begin{bmatrix}		  c_{11} & c_{12} & c_{13} \\		  c_{21} & c_{22} & c_{23} 	\end{bmatrix}	\end{equation}$$ To compute ${\bf AC}$ we think of ${\bf A}$ as a column of row vectors:${\bf A} =\begin{bmatrix}	  {\bf a}_1 \\	  {\bf a}_2 \\	  {\bf a}_3		\end{bmatrix}    $    and of ${\bf C}$ as a row of column vectors: ${\bf C} =\begin{bmatrix}	  {\bf c}_1,	  {\bf c}_2,	  {\bf c}_3		\end{bmatrix}    $ ${\bf AC}$ is the matrix generated from taking the dot product of each row vector in ${\bf A}$ with each column vector in ${\bf C}$${\bf AC}=	\begin{bmatrix}	  {\bf a}_1 \\	  {\bf a}_2 \\	  {\bf a}_3			\end{bmatrix}	\begin{bmatrix}	  {\bf c}_1,	  {\bf c}_2,	  {\bf c}_3	\end{bmatrix}= \begin{bmatrix}    {\bf a}_1 \cdot {\bf c}_1 & {\bf a}_1 \cdot {\bf c}_2 & {\bf a}_1 \cdot {\bf c}_3 \\    {\bf a}_2 \cdot {\bf c}_1 & {\bf a}_2 \cdot {\bf c}_2 & {\bf a}_2 \cdot {\bf c}_3 \\    {\bf a}_3 \cdot {\bf c}_1 & {\bf a}_3 \cdot {\bf c}_2 & {\bf a}_3 \cdot {\bf c}_3    \end{bmatrix} =    $ $= \begin{bmatrix}	  a_{11} c_{11}+a_{12} c_{21} & a_{11} c_{12}+a_{12} c_{22} & a_{11} c_{13}+a_{12} c_{23} \\	  a_{21} c_{11}+a_{22} c_{21} & a_{21} c_{12}+a_{22} c_{22} & a_{21} c_{13}+a_{22} c_{23} \\	  a_{31} c_{11}+a_{32} c_{21} & a_{31} c_{12}+a_{32} c_{22} & a_{31} c_{13}+a_{32} c_{23}	\end{bmatrix}$    For more information on the topic of matrix multiplication, see http://en.wikipedia.org/wiki/Matrix_multiplication.<jupyter_code># Matrix - Vector product
A = np.arange(6).reshape((3,2))
C = np.array([-1,1])

print(A.shape)
print(C.shape)
print(np.dot(A,C.T))
# Matrix - Matrix product

# Define the matrices A and C
A = np.arange(6).reshape((3,2))
C = np.random.randn(2,2)

print('A=\n',A)
print('C=\n',C)<jupyter_output><empty_output><jupyter_text>We will use the numpy dot operator to perform the these multiplications.  You can use it two ways to yield the same result:<jupyter_code>print('A.dot(C)=\n',A.dot(C))
print('np.dot(A,C)=\n',np.dot(A,C))<jupyter_output><empty_output><jupyter_text>ConformityNote that the number of columns in the first matrix has to be equal to the number of columns in the second matrix. Otherwise, the matrix product is not defined. When this condition holds we say that the two matrices **conform**.Taking the product of two matrices that don't conform results in an exception:<jupyter_code>np.dot(C,A)<jupyter_output><empty_output><jupyter_text>Orthonormal matrices and change of Basis**As was explained in the notebook: "Linear Algebra Review"** We say that the vectors $\vec{u}_1,\vec{u}_2,\ldots,\vec{u}_d \in R^d$ form an **orthonormal basis** of $R^d$. If:* **Normality:** $\vec{u}_1,\vec{u}_2,\ldots,\vec{u}_d$ are unit vectors: $\forall 1 \leq i \leq d: \vec{u}_i \cdot \vec{u}_i =1 $* **Orthogonality:** Every pair of vectors are orthogonal: $\forall 1 \leq i\neq j \leq d: \vec{u}_i \cdot \vec{u}_j =0 $** Orthonormal basis can be used to rotate the vector space:*** $\vec{v}$ is **represented** as a list of $d$ dot products: $$[\vec{v}\cdot\vec{u_1},\vec{v}\cdot\vec{u_2},\ldots,\vec{v}\cdot\vec{u_d}]$$* $\vec{v}$ is **reconstructed** by summing its projections on the basis vectors:$$\vec{v} = (\vec{v}\cdot\vec{u_1})\vec{u_1} + (\vec{v}\cdot\vec{u_2})\vec{u_2} + \cdots + (\vec{v}\cdot\vec{u_d})\vec{u_d}$$  Change of Basis using matrix notationTo use matrix notation, we think of $\vec{u}_i$ as a row vector:$$   {\bf u}_i=\begin{bmatrix} u_{i1}, u_{i2},\ldots, u_{id} \end{bmatrix},$$ We can combine the orthonormal vectors to create an *orthonormal matrix*$$ {\bf U} = \begin{bmatrix} {\bf u}_1 \\ {\bf u}_2 \\ \vdots \\ {\bf u}_d \end{bmatrix}= \begin{bmatrix} u_{11}, u_{12},\ldots, u_{1d} \\ u_{21}, u_{22},\ldots, u_{2d} \\ \vdots\\u_{d1}, u_{d2},\ldots, u_{dd} \end{bmatrix}$$Orthonormality: ${\bf UU^{\top} = I}$ Using this notation, the representation of a column vector $\bf v$ in the orthonormal basis corresponsing to the rows of ${\bf U}$ is equal to $${\bf Uv} = \begin{bmatrix} {\bf u}_1 {\bf v} \\ {\bf u}_2 {\bf v} \\ \vdots \\ {\bf u}_d {\bf v} \end{bmatrix}$$   And the reconstruction of $\bf v$ is equal to ${\bf U U^{\mathsf{T}} v}$  The Identity  MatrixThe identity matrix behaves like the number $1$: The dot product of any matrix ${\bf A}$ by the identity matrix ${\bf I}$ yields ${\bf A}$.$$ {\bf A I = I A = A} $$ The identity matrix is zero everywhere other than the diagonal, where it is $1$.$${\bf I} = \begin{bmatrix} 1, 0,\ldots, 0 \\ 0, 1,\ldots, 0 \\ \ddots \\0,0,\ldots, 1 \end{bmatrix}$$**Excercise:** Check that ${\bf A I = I A = A}$.  Inverting a Matrix Recall that the multiplicative inverse of the number $a$ is $a^{-1}=1/a$The property of $a^{-1}$ is that $a a^{-1}=1$.Recall also that $0$ does not have a multiplicative inverse. **Some** square matrices ${\bf A}$ have a multiplicative inverse ${\bf A^{-1}}$  such that ${\bf A A^{-1} = A^{-1} A =I}$ Finding the inverse of a matrix is called *inverting* the matrix.   An $n\times n$ matrix $\bf A$ represents a linear transformation from $R^n$ to $R^n$. If the matrix is [**invertible**](https://en.wikipedia.org/wiki/Invertible_matrix) then there is another transformation ${\bf A}^{-1}$ that represents the inverse transformation, such that for any column vctor ${\bf v} \in R^n$:$${\bf A}^{-1}{\bf A}{\bf v} = {\bf A}{\bf A}^{-1}{\bf v} = {\bf v} $$  Inverting a 2X2 matrixConsider the square $2 \times 2$ matrix ${\bf A} = \bigl( \begin{smallmatrix} a_{11} & a_{12} \\ a_{21} & a_{22}\end{smallmatrix} \bigr)$.  The inverse of matrix ${\bf A}$ is$$\begin{equation}	{\bf A}^{-1}=\begin{bmatrix}             a_{11} & a_{12} \\		     a_{21} & a_{22}            \end{bmatrix}^{-1}=\frac{1}{a_{11}a_{22}-a_{12}a_{21}}	\begin{bmatrix}		             a_{22} & -a_{12} \\				     -a_{21} & a_{11} 		           \end{bmatrix}\end{equation}$$ **Excercise:** Check that $ {\bf A A^{-1}=A^{-1} A=I }$ For more information on inverting matrices, see thishttp://en.wikipedia.org/wiki/Matrix_inversion.<jupyter_code># An example of computing the inverse using numpy.linalg.inv
# note, we need a square matrix (# rows = # cols), use C:
C = np.random.randn(2,2)
print("C=\n",C)
C_inverse = np.linalg.inv(C)
print("C_inverse=\n",C_inverse)<jupyter_output><empty_output><jupyter_text>Checking that $C\times C^{-1} = I$:<jupyter_code>I = np.eye(2)
print("identity matrix=\n",I)
print("C.dot(C_inverse)-I=\n",C.dot(C_inverse)-I)
print("C_inverse.dot(C)-I=\n",C_inverse.dot(C)-I)<jupyter_output><empty_output><jupyter_text>Singular matricesNot all matrices have an inverse. Those that do not are called **singular**<jupyter_code>C=np.array([[1,0],[1,0]])
print("C=\n",C)
try:
    C_inverse = np.linalg.inv(C)
except: 
    print('C cannot be inverted: it is a singular matrix')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.182363747703613, 'fraction_numerical': 0.038946723821187995, 'mean_word_length': 4.004903463070794, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '567269', 'license': 'MIT', 'n_tokens_mistral': 6362, 'n_tokens_neox': 5839, 'n_words': 1955, 'path': 'Week 12 _ Regression and PCA/more_lectures/2.Matrix_notation_and_operations.ipynb', 'repo_name': 'ebishwaraj/ProbabilityStatisticsPython_DSE210x_UCSD_DataScienceMicroMasters', 'size': '16330'}