<jupyter_start><jupyter_text>+ This notebook is part of lecture 14 *Orthogonal vectors and subspaces* in the OCW MIT course 18.06 by Prof <NAME> [1]+ Created by me, Dr <NAME>    + Head of Acute Care Surgery    + Groote Schuur Hospital    + University Cape Town    + Email me with your thoughts, comments, suggestions and corrections Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr <NAME> is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.+ [1] OCW MIT 18.06+ [2] <NAME>, <NAME>, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org<jupyter_code>from IPython.core.display import HTML, Image
# css_file = 'style.css'
# HTML(open(css_file, 'r').read())
from sympy import init_printing, symbols, Matrix
from warnings import filterwarnings
init_printing(use_latex='mathjax')
filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Orthogonal vectors and subspaces Rowspace orthogonal to nullspace and columnspace to nullspace of AT N(ATA) = N(A)  Orthogonal vectors * Two vectors are orthogonal if their dot product is zero* If they are written as column vectors **x** and **y**, their dot product is **x**T**y**    * For orthogonal (perpendicular) vectors **x**T**y** = 0* From the Pythagorean theorem they are orthogonal if$$ { \left\| \overline { x }  \right\|  }^{ 2 }+{ \left\| \overline { y }  \right\|  }^{ 2 }={ \left\| \overline { x } +\overline { y }  \right\|  }^{ 2 }\\ { \left\| \overline { x }  \right\|  }=\sqrt { { x }_{ 1 }^{ 2 }+{ x }_{ 2 }^{ 2 }+\dots +{ x }_{ b }^{ 2 } }  $$ * The length squared of a (column) vector **x** can be calculated by **x**T**x*** This achieves exactly the same as the sum of the squares of each element in the vector$$ { x }_{ 1 }^{ 2 }+{ x }_{ 2 }^{ 2 }+\dots +{ x }_{ n }^{ 2 }$$ * Following from the Pythagorean theorem we have$$ { \left\| \overline { x }  \right\|  }^{ 2 }+{ \left\| \overline { y }  \right\|  }^{ 2 }={ \left\| \overline { x } +\overline { y }  \right\|  }^{ 2 }\\ { \underline { x }  }^{ T }\underline { x } +{ \underline { y }  }^{ T }\underline { y } ={ \left( \underline { x } +\underline { y }  \right)  }^{ T }\left( \underline { x } +\underline { y }  \right) \\ { \underline { x }  }^{ T }\underline { x } +{ \underline { y }  }^{ T }\underline { y } ={ \underline { x }  }^{ T }\underline { x } +{ \underline { x }  }^{ T }\underline { y } +{ \underline { y }  }^{ T }\underline { x } +{ \underline { y }  }^{ T }\underline { y } \\ \because \quad { \underline { x }  }^{ T }\underline { y } ={ \underline { y }  }^{ T }\underline { x } \\ { \underline { x }  }^{ T }\underline { x } +{ \underline { y }  }^{ T }\underline { y } ={ \underline { x }  }^{ T }\underline { x } +2{ \underline { x }  }^{ T }\underline { y } +{ \underline { y }  }^{ T }\underline { y } \\ 2{ \underline { x }  }^{ T }\underline { y } =0\\ { \underline { x }  }^{ T }\underline { y } =0 $$* This states that the dot product of orthogonal vectors equal zero * The zero vector is orthogonal to all other similar dimensional vectors  Orthogonality of subspaces * Consider two subspaces *S* and *T** To be orthogonal every vector in *S* must be orthogonal to any vector in *T* * Consider the *XY* and *YZ* planes in 3-space* They are not orthogonal, since many combinations of vectors (one in each plane) are not orthogonal* Vectors in the intersection, even though, one each from each plane can indeed be the same vector* We can say that any planes that intersect cannot be orthogonal to each other  Orthogonality of the rowspace and the nullspace * The nullspace contains vectors **x** such that A**x** = **0*** Now remembering that **x**T**y** = 0 for orthogonal column vectors and considering each row in A as a transposed column vector and **x** (indeed a column vector) and their product being zero meaning that they are orthogonal, we have:$$ \begin{bmatrix} { { a }_{ 11 } } & { a }_{ 12 } & \dots  & { a }_{ 1n } \\ { a }_{ 21 } & { a }_{ 22 } & \dots  & { a }_{ 2n } \\ \vdots  & \vdots  & \vdots  & \vdots  \\ { a }_{ m1 } & { a }_{ m2 } & \dots  & { a }_{ mn } \end{bmatrix}\begin{bmatrix} { x }_{ 1 } \\ { x }_{ 2 } \\ \vdots  \\ { x }_{ n } \end{bmatrix}=\begin{bmatrix} 0 \\ 0 \\ \vdots  \\ 0 \end{bmatrix}\\ \begin{bmatrix} { a }_{ 11 } & { a }_{ 12 } & \dots  & { a }_{ 1n } \end{bmatrix}\begin{bmatrix} { x }_{ 1 } \\ { x }_{ 2 } \\ \vdots  \\ { x }_{ n } \end{bmatrix}=0\\ \dots  $$ * The rows (row vectors) in A are NOT the only vectors in the rowspace, since we also need to show that ALL linear combinations of them are also orthogonal to **x*** This is easy to see by the structure above  Orthogonality of the columnspace and the nullspace of AT * The proof is the same as above * The orthogonality of the rowspace and the nullspace is creating two orthogonal subspaces in &8477;n* The orthogonality of the columnspace and the nullspace of AT is creating two orthogonal subspaces in &8477;m * Note how the dimension add up to the degree of the space &8477;    * The rowspace (a fundamental subspace in &8477;n) is of dimension *r*    * The dimension of the nullspace (a fundamental subspace in &8477;n) is of dimension *n* - *r*    * Addition of these dimensions gives us the dimension of the total space *n* as in &8477;n    * AND    * The columnspace is of dimension *r* and the nullspace of AT is of dimension *m* - *r*, which adds to *m* as in &8477;m * This means that two lines that may be orthogonal in &8477;3 cannot be two orthogonal subspaces of &8477;3 since the addition of the dimensions of these two subspaces (lines) is not 3 (as in &8477;3) * We call this complementarity, i.e. the nullspace and rowspace are orthogonal *complements* in &8477;n  ATA * We know that    * The result is square    * The result is symmetric, i.e. (*n*&215;*m*)(*m*&215;*n*)=*n*&215;*n*    * (ATA)T = ATATT = ATA * When A**x** = **b** is not solvable we use ATA**x** = AT**b*** **x** in the first instance did not have a solution, but after multiplying both side with AT, we hope that the second **x** has an solution, now called$$ {A}^{T}{A}\hat{x} = {A}^{T}{b} $$ * Consider the matrix below with *m* = 4 equation in *n* = 2 unknowns* The only **b** solutions must be linear combinations of the columnspace of A<jupyter_code>A = Matrix([[1, 1], [1, 2], [1, 5]])
A<jupyter_output><empty_output><jupyter_text>$$ {x}_{1} \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} + {x}_{2} \begin{bmatrix} 1 \\ 2 \\ 5 \end{bmatrix} = \begin{bmatrix} {b}_{1} \\ {b}_{2} \\ {b}_{3} \end{bmatrix} $$<jupyter_code>A.transpose() * A<jupyter_output><empty_output><jupyter_text>* Note how the nullspace of ATA is equal to the nullspace of A<jupyter_code>(A.transpose() * A).nullspace() == A.nullspace()<jupyter_output><empty_output><jupyter_text>* The same goes for the rank<jupyter_code>A.rref(), (A.transpose() * A).rref()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15809033733562036, 'fraction_numerical': 0.02387078330474557, 'mean_word_length': 3.531735751295337, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '134439', 'license': 'MIT', 'n_tokens_mistral': 2443, 'n_tokens_neox': 2181, 'n_words': 937, 'path': '_math/MIT_OCW_18_06_Linear_algebra/II_01_Orthogonality_of_vectors_and_subspaces.ipynb', 'repo_name': 'aixpact/data-science', 'size': '7038'}