<jupyter_start><jupyter_text>Video W1 12: Matrices and Vectors[YouTube Video Link](https://www.youtube.com/watch?v=04IlpTiBTsk&index=12&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW) MatricesA matrix is a retangular array of numbers:$$\begin{bmatrix}1402 &  191 \\ 1371 &  821 \\  949 & 1437 \\  147 & 1448 \\ \end{bmatrix}$$The dimension of a matrix is the number of rows by the number of columns.  In the previous matrix,we have 4 rows by 2 columns.In Python, we use `NumPy` arrays to represent 1, 2 (and higher dimensional) vectors and matrices.<jupyter_code>A = np.array([[1402, 191], 
              [1371, 821], 
              [949, 1437], 
              [147, 1448]])
print(A)
print(A.ndim) # it is a 2 dimensional matrix
print(A.shape) # notice this is a 2 dimensional matrix, of size 4 x 2
# a 2 X 3  matrix
B = np.array([[1, 2, 3],
              [4, 5, 6]])

print(B)
print(B.ndim)
print(B.shape)<jupyter_output>[[1 2 3]
 [4 5 6]]
2
(2, 3)<jupyter_text>We refer to entries of 2 dimensional matrices using `row,column` indices. So given:$$A=\begin{bmatrix}1402 &  191 \\ 1371 &  821 \\  949 & 1437 \\  147 & 1448 \\ \end{bmatrix}$$$$A_{ij} = i,j \; \mathrm{entry \; in \; the} \; i^{th} \; \mathrm{row,} \; j^{th} \; \mathrm{column}.$$So$$A_{12} = 191$$**NOTE:** In Python, lists and `NumPy` arrays are indexed starting at 0 (0 based indexing).  This meansthat when you translate any code from a 1 based mathematical indexing, to the 0 based indexing, you always need to subtract one.  So for example, to get the $A_{12}$ entry in our matrix we need to:<jupyter_code>print(A[0,1])<jupyter_output>191<jupyter_text>The lecturer goes in to this a bit more in the video.  In general, in programminglanguages, and in mathematical treatments, you can treat matrices and vectorsas being 1 indexed (starting at index 1) or 0 indexed (starting at index 0).Because the class from our videos uses Matlab, which is 1 indexed, the lecturertends to use 1 indexed vectors.  However, since Python and NumPy are 0indexed representations, I will usually convert any notations from the videosinto equivalent 0 indexed versions, because that is simplier and more directlyimplementable into Python and NumPy 0 indexed lists and arrays.  VectorA vector is simply a 1 dimensional vector, or you can think of it as a matrix with n rows and 1 column:$$y=\begin{bmatrix}460 \\ 232 \\ 315 \\ 178 \\ \end{bmatrix}$$Notice that in `NumPy`, there is a slight difference between a vector, and an array with 1 column:<jupyter_code># a numpy vector is 1 dimensional, we can initialize with a 1d list
y = np.array([460, 232, 315, 178])
print(y)
print(y.ndim)
print(y.shape)

# we can create a 4x1 column matrix, like this:
z = np.array([[460],
              [232],
              [315],
              [178]])
print(z)
print(z.ndim)
print(z.shape)<jupyter_output>[460 232 315 178]
1
(4,)
[[460]
 [232]
 [315]
 [178]]
2
(4, 1)<jupyter_text>Sometimes in writing code we can use `NumPy` 1d vectors, but sometimes we need to use a $n \times 1$column matrix instead.  Video W1 13: Addition and Scalar Multiplication[YouTube Video Link](https://www.youtube.com/watch?v=eCJpIfMrm6U&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW&index=13) Matrix AdditionAddition and subtraction of matrices are simply element by element wise operations.  The matricesneed to have exactly the same dimensions for this to be defined:$$\begin{bmatrix}1 &  0 \\ 2 &  5 \\ 3 & 1 \\ \end{bmatrix}+\begin{bmatrix}4 & 0.5 \\ 2 & 5 \\ 0 & 1 \\ \end{bmatrix}=\begin{bmatrix}5 & 0.5 \\ 4 & 10 \\ 3 & 2 \\ \end{bmatrix}$$<jupyter_code>A = np.array([[1, 0],
              [2, 5],
              [3, 1]])
B = np.array([[4, 0.5],
              [2, 5],
              [0, 1]])

print(A + B)
print(A - B)<jupyter_output>[[ 5.   0.5]
 [ 4.  10. ]
 [ 3.   2. ]]
[[-3.  -0.5]
 [ 0.   0. ]
 [ 3.   0. ]]<jupyter_text>Scalar MultiplicationI will probably slip into the habit of using the term scalar here and there.  Scalar is just a fancyterm for a regular, single (real valued) value.  Basically the opposite of a Matrix is a scalar value.When you multiply (or divide) a matrix by a scalar value, you simply multiple each value in the matrixby the scalar value:$$3\times\begin{bmatrix}1 &  0 \\ 2 &  5 \\ 3 & 1 \\ \end{bmatrix}=\begin{bmatrix}3 & 0 \\ 6 & 15 \\ 9 & 3 \\ \end{bmatrix}$$By default, `NumPy` arrays support scalar operations against vectors and matrices.So if we want to do the equivalent scalar multiplication, we simply multiplythe `NumPy` array by the scalar value, like this:<jupyter_code>A = np.array([[1, 0],
              [2, 5],
              [3, 1]])

print(3.0 * A)
B = np.array([[4, 0],
              [6, 3]])

print(B / 4.0)
print(1.0/4.0 * B)<jupyter_output>[[1.   0.  ]
 [1.5  0.75]]
[[1.   0.  ]
 [1.5  0.75]]<jupyter_text>Also `NumPy` is perfectly capable of combining multiple operations togetherfrom single complex expressions using vectorized operation.  For example:<jupyter_code>a = np.array([[1],
              [4],
              [2]])
b = np.array([[0],
              [0],
              [5]])
c = np.array([[3],
              [0],
              [2]])

d = 3.0 * a + b - c / 3.0
print(d)
print(d.shape)<jupyter_output>[[ 2.        ]
 [12.        ]
 [10.33333333]]
(3, 1)<jupyter_text>Video W1 14: Matrix Vector Multiplication[YouTube Video Link](https://www.youtube.com/watch?v=bA3wxP5AtQk&index=14&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW) Example of Matrix x Vector Multiplication$$\begin{bmatrix}1 &  3 \\ 4 &  2 \\ 2 & 1 \\ \end{bmatrix}\begin{bmatrix}1 \\ 5 \\ \end{bmatrix}=\begin{bmatrix}16 \\4 \\7 \\\end{bmatrix}$$Here we multiple a $3 \times 2$ matrix by a $2 \times 1$ vector.  The result is a $3 \times 1$vector.In `NumPy`, the `*` operator only works to perform scalar multiplication of a matrix with a scalarvalue (or scalar multiplications of two matrices of the same dimensions), thus the followingis not going to work.<jupyter_code>A = np.array([[1, 3],
              [4, 0],
              [2, 1]])
x = np.array([[1],
              [5]])

try:
    print(A * x)
except ValueError:
    print('Operator * is not overloaded to perform Matrix - Matrix multiplication, need to use np.dot() function')
    
# uncomment this to see the actual ValueError exception
#print(A * x)<jupyter_output>Operator * is not overloaded to perform Matrix - Matrix multiplication, need to use np.dot() function<jupyter_text>We instead need to use a `NumPy` library function if we want to do matrix multiplication:<jupyter_code>y = np.dot(A, x)
print(y)<jupyter_output>[[16]
 [ 4]
 [ 7]]<jupyter_text>Notice that the result of multiple A of shape 3 X 2 times x of shape2 x 1 is a vector of shape 3 x 1.<jupyter_code>print(A.shape)
print(x.shape)
print(y.shape)<jupyter_output>(3, 2)
(2, 1)
(3, 1)<jupyter_text>Another example, lets multiple a 3 x 4 matrix times a 4 x 1 vector to geta 3 x 1 result.<jupyter_code>A = np.array([[ 1,  2, 1, 5],
              [ 0,  3, 0, 4],
              [-1, -2, 0, 0]])
x = np.array([[1],
              [3],
              [2],
              [1]])

print(A.shape)
print(x.shape)
y = np.dot(A, x)
print(y)
print(y.shape)<jupyter_output>[[14]
 [13]
 [-7]]
(3, 1)<jupyter_text>A Neat TrickWe can compute all of the model hypothesis (given a particular $\theta_0, \theta_1$) for a number ofhouses simultaneously using matrix vector multiplication.Say our hypothesis is:$$h_\theta(x) = -40 + 0.25x$$Where of course $\theta_0$ the intercept is -40 and $\theta_1$ the slope is 0.25.We can do the following for a set of houses, to compute the model prices forall of the houses at the same time.  Notice that we need to add a dummycolumn to our X array of feature input data, to represent the value tomultiple times the intercept (we always just add the intercept to theother term, so multiplying by 1 gives equivalent.)<jupyter_code>X = np.array([[1, 2104],
              [1, 1416],
              [1, 1534],
              [1, 852]])
theta = np.array([[-40],
                  [0.25]])

y = np.dot(X, theta) # y is our hypothesis/model prices for the 4 houses of the given sizes
print(y)<jupyter_output>[[486. ]
 [314. ]
 [343.5]
 [173. ]]<jupyter_text>So in a single operation, we calculated the hypothesis for all (4) of our housesin our input feature data set.Using a vectorized matrix vector multiplication like this is much fasterthan doing this in a for loop. Vectorized operations are much morecomputationally efficient.  For example:<jupyter_code>%timeit y = np.dot(X, theta)

def compute_hypothesis_using_loop(X, theta):
    y = np.empty((X.shape[0], 1)) # create empty array to hold and return result
    
    for i in range(X.shape[0]):
        x = X[i, 1] # extract ith house square feet
        y[i,0] = theta[0, 0] + theta[1, 0] * x # theta[0,0] is our intercept and theta[1, 0] is our slope

    return y

%timeit y = compute_hypothesis_using_loop(X, theta)<jupyter_output>3.32 µs ± 57 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
5.36 µs ± 73.1 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)<jupyter_text>This will usually be about 2 or 3 times slower using an explicit loop.  And thepenalty is much much worse if your input data X has a large number ofhouses.  Here with only 4 houses you don't get a real good feel for how muchfaster the vectorized version can be compared to explicitly iteratingin a loop to calculate the hypothesis.  Video W1 15: Matrix Vector MultiplicationThere was a duplicate in the playlist of the Matrix Vector multiplication video, so ignore W1 15. Video W1 16: Matrix Matrix Multiplication[YouTube Video Link](https://www.youtube.com/watch?v=aqK2Y0l2rXE&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW&index=16) Multiplying two matrices togetherThis is really a simple extension of the matrix by vector multiplication:$$\begin{bmatrix}1 &  3 & 2 \\ 4 &  0 & 1 \\ \end{bmatrix}\begin{bmatrix}1 & 3 \\ 0 & 1 \\5 & 2 \\\end{bmatrix}=\begin{bmatrix}11 & 10 \\9  & 14 \\\end{bmatrix}$$In `NumPy` we still use the same `np.dot()` function to perform matrix by matrix multiplication:<jupyter_code>A = np.array([[1, 3, 2],
              [4, 0, 1]])
B = np.array([[1, 3],
              [0, 1],
              [5, 2]])

print(np.dot(A, B))
# another example, a 2x2 times a 2x2 matrix
A = np.array([[1, 3],
              [2, 5]])
B = np.array([[0, 1],
              [3, 2]])

print(np.dot(A, B))<jupyter_output>[[ 9  7]
 [15 12]]<jupyter_text>More Neat TricksLets say instead of 1 hypothesis, we instead have 3 hypotheses that we want to calculate the predicted price for a set of housesfor:$$h_\theta(x) = -40 + 0.25x$$$$h_\theta(x) = 200 + 0.1x$$$$h_\theta(x) = -150 + 0.4x$$We can use matrix matrix multiplication to perform all of the prediction calculations in a single vectorized operation.<jupyter_code>X = np.array([[1, 2104],
              [1, 1416],
              [1, 1534],
              [1, 852]])

Theta = np.array([[-40, 200, -150],
                  [0.25, 0.1, 0.4]])

print(np.dot(X, Theta))<jupyter_output>[[486.  410.4 691.6]
 [314.  341.6 416.4]
 [343.5 353.4 463.6]
 [173.  285.2 190.8]]<jupyter_text>The resulting matrix gives the hypothesized price for each of the 4 houses(the rows of the matrix).  Each column represents a different hypothesizedprice, thus we have 3 columns for each of the 3 hypothesis being calculated.This multiplication can of course be extended to any number of houses(with any number of input features) and for any number of hypothesis wewant to evaluate.Also, as discussed previously, the vectorized operation of calculating allhypothesis for all houses shown here will be much faster than if we explicitlywrote some nested loops to calculate each of the hypothesized house pricesone by one.  Video W1 17: Matrix Multiplication Properties[YouTube Video Link](https://www.youtube.com/watch?v=00GJM2Js7AI&index=17&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW)A discussion of some things you should be aware when using linear algebra matrix operations.- Matrix multipication is not commutative $ A \times B \ne B \times A$- However, matrix multiplication is associative   $(A \times B) \times C = A \times (B \times C)$<jupyter_code># matrix multiplication IS NOT commutative
A = np.array([[1, 1],
              [0, 0]])
B = np.array([[0, 0],
              [2, 0]])

print(np.dot(A, B))
print(np.dot(B, A))
# Matrix multiplication IS associative
C = np.array([[2, 0],
              [2, 1]])

Tmp = np.dot(A, B)
print(np.dot(Tmp, C))

Tmp = np.dot(B,C)
print(np.dot(A, Tmp))<jupyter_output>[[4 0]
 [0 0]]
[[4 0]
 [0 0]]<jupyter_text>Identity MatrixFor scalar values, the value 1 is the identity, e.g $1 \times z = z \times 1 = z$.For matrices there is an identity matrix, denotied as $I$ or $(I_{n \times n})$.  The identity matrix has 1's on the diagonal, andzeros every else.<jupyter_code># the 4x4 identity matrix
print(np.eye(4))<jupyter_output>[[1. 0. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 0. 1.]]<jupyter_text>For any matrix $A$,$$A \cdot I = I \cdot A = A$$<jupyter_code>A = np.array([[3, 4, 7],
              [1, 8, 2],
              [9, 6, 5]])

I = np.eye(3)

print(np.dot(A, I))
print(np.dot(I, A))<jupyter_output>[[3. 4. 7.]
 [1. 8. 2.]
 [9. 6. 5.]]
[[3. 4. 7.]
 [1. 8. 2.]
 [9. 6. 5.]]<jupyter_text>Video W1 18: Inverse and Transpose[YouTube Video Link](https://www.youtube.com/watch?v=9McrlFqn-gg&index=18&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW) Matrix InverseThe inverse of a scalar is simply the value such that when you multiple the scalar by its inverse you get 1.For example:$$3 \times 3^{-1} = 3 \times \frac{1}{3} = 1$$For a matrix, the inverse of a matrix is a matrix such that:$$A \cdot (A^{-1}) = A^{-1} \cdot A = I$$Only square matrices can have inverses.  But not all square matrices are guaranted to have an inverse.  Finding the inverse ofa square matrix (if it exists) can be difficult.  Here is an example of a matrix and its inverse:<jupyter_code>A = np.array([[3, 4],
              [2, 16]])
Ainv = np.array([[0.4, -0.1],
                 [-0.05, 0.075]])

# show that Ainv is the inverse of A
print(np.dot(A, Ainv))<jupyter_output>[[ 1.00000000e+00 -5.55111512e-17]
 [ 0.00000000e+00  1.00000000e+00]]<jupyter_text>Don't be confused if the numbers (especially the 0's), are't exactly 1 or 0, the result is very close to the Identity matrix (within machine precision of the calculation).How do we compute the Inverse of a matrix?  There are algorithms to do this (of course), and it is difficult to do by hand.The `NumPy` library has functions that can be used to compute the inverse of a matrix:<jupyter_code>print(np.linalg.inv(A))
print(np.dot(A, np.linalg.inv(A)))<jupyter_output>[[1. 0.]
 [0. 1.]]<jupyter_text>Some scalar values don't have an inverse, for example 0 doesn't have an inverse.  Likewise, some matrices don't have an inverse.However for machine learning this is usually not an issue, because often there are pseudo inverses that are close enough formachine learning purposes.  Matrix TransposeThe transpose of matrix simply flips the row and column indexes of a matrix:$$A =\begin{bmatrix}1 &  2 & 0 \\ 3 &  5 & 9 \\ \end{bmatrix}\;\;\;A^T =\begin{bmatrix}1 &  3 \\ 2 &  5\\0 & 9 \\\end{bmatrix}$$<jupyter_code>A = np.array([[1, 2, 0],
              [3, 5, 9]])

# we can use the transpose function from NumPy
print(np.transpose(A))

# or we can use the convenience attribute x.T
print(A.T) 
import sys
sys.path.append("../../src") # add our class modules to the system PYTHON_PATH

from ml_python_class.custom_funcs import version_information
version_information()<jupyter_output>Module   Versions
--------------------   ------------------------------------------------------------
              numpy:   ['1.18.5']
             pandas:   ['1.0.5']
            seaborn:   ['0.10.1']
{'fraction_non_alphanumeric': 0.13344799031785465, 'fraction_numerical': 0.05713739728645137, 'mean_word_length': 3.2420967306133477, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 6, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '368317', 'license': 'CC-BY-3.0', 'n_tokens_mistral': 5932, 'n_tokens_neox': 5256, 'n_words': 1986, 'path': 'lectures/review/U00-4-Linear-Algebra-Review.ipynb', 'repo_name': 'tgrasty/CSCI574-Machine-Learning', 'size': '15699'}