<jupyter_start><jupyter_text>Vertical Line Test  1.1 Create two graphs, one that passes the vertical line test and one that does not.<jupyter_code>import matplotlib.pyplot as plt
import pandas as pd
import numpy as np


# Plot Linearly Independent Vectors

# Axis Bounds
plt.xlim(-2,5)          
plt.ylim(-1,3)

# Original Vector
a = [-1.5,.5] 
b = [3, 1]

# Plot Vectors
plt.arrow(1,1, a[0], a[1], linewidth=3, head_width=.05, head_length=0.05, color ='blue')
plt.arrow(0,0, b[0], b[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')

plt.title("This graph does not pass the vertical line test")
plt.show()
# Plot Linearly Independent Vectors

# Axis Bounds
plt.xlim(-3,4)          
plt.ylim(-1,3)

# Original Vector
a = [-1.5,.5] 
b = [3, 1]

# Plot Vectors
plt.arrow(-0.5,0, a[0], a[1], linewidth=3, head_width=.05, head_length=0.05, color ='blue')
plt.arrow(.5,0, b[0], b[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')

plt.title("This graph passes the vertical line test")
plt.show()<jupyter_output><empty_output><jupyter_text>1.2 Why are graphs that don't pass the vertical line test not considered "functions?" Because a function is defined where no two outputs are mapped to the same point. In graph 1, there are places where the two vectors share x, while in graph 2 that does not occur anywhere.   Functions as Relations  2.1 Which of the following relations are functions? Why?\begin{align}\text{Relation 1: } \{(1, 2), (3, 2), (1, 3)\}\\\text{Relation 2: } \{(1, 3), (2, 3), (6, 7)\}\\\text{Relation 3: } \{(9, 4), (2, 1), (9, 6)\}\\\text{Relation 4: } \{(6, 2), (8, 3), (6, 4)\}\\\text{Relation 5: } \{(2, 6), (2, 7), (2, 4)\}\end{align} (Zhenya) ---Only Relation 2 is a function because none of the coordinates share an X value so it passes the vertical line test.  Functions as a mapping between dimensions  3.1 for the following functions what is the dimensionality of the domain (input) and codomain (range/output)?\begin{align}m(𝑥_1,𝑥_2,𝑥_3)=(x_1+x_2, x_1+x_3, x_2+x_3)\\n(𝑥_1,𝑥_2,𝑥_3,𝑥_4)=(x_2^2 + x_3, x_2x_4)\end{align}  (Zhenya)---1st function: R^3 to R^22nd function: R^4 to R^2   3.2 Do you think it's possible to create a function that maps from a lower dimensional space to a higher dimensional space? If so, provide an example.Zhenya---This goes from R^1 to R^3$m(x_1) = (x_1 / 2 + x_1 / 3,x_1 / 3 + x_1 / 4),x_1 / 4 + x_1 / 5) $  Vector Transformations  4.1 Plug the corresponding unit vectors into each function. Use the output vectors to create a transformation matrix.\begin{align}p(\begin{bmatrix}x_1 \\ x_2 \end{bmatrix}) = \begin{bmatrix} x_1 + 3x_2 \\2 x_2 - x_1 \\  \end{bmatrix}\\\\q(\begin{bmatrix}x_1 \\ x_2 \\ x_3\end{bmatrix}) = \begin{bmatrix} 4x_1 + x_2 + 2x_3 \\2 x_2 - x_1 + 3x_3 \\ 5x_1 - 2x_3 + x_2  \end{bmatrix}\end{align} (Zhenya)THIS IS CORRECT YAYYY!!!!---First way to transform the function into a transformation matrix:\begin{align}p(\begin{bmatrix}0 \\ 1 \end{bmatrix}) = \begin{bmatrix} x_1 + 3x_2 \\2 x_2 - x_1 \\  \end{bmatrix} =\begin{bmatrix}3 \\ 2 \end{bmatrix} \end{align} \begin{align}p(\begin{bmatrix}1 \\ 0 \end{bmatrix}) = \begin{bmatrix} x_1 + 3x_2 \\2 x_2 - x_1 \\  \end{bmatrix}=\begin{bmatrix}1 \\ -1 \end{bmatrix}\end{align}\begin{align}q(\begin{bmatrix}1 \\ 0 \\ 0 \end{bmatrix}) = \begin{bmatrix} 4x_1 + x_2 + 2x_3 \\2 x_2 - x_1 + 3x_3 \\ 5x_1 - 2x_3 + x_2  \end{bmatrix}=\begin{bmatrix}4 \\ -1 \\ 5 \end{bmatrix}\end{align}\begin{align}q(\begin{bmatrix}0 \\ 1 \\ 0 \end{bmatrix}) = \begin{bmatrix} 4x_1 + x_2 + 2x_3 \\2 x_2 - x_1 + 3x_3 \\ 5x_1 - 2x_3 + x_2  \end{bmatrix}=\begin{bmatrix}1 \\ 2 \\ 1 \end{bmatrix}\end{align}\begin{align}q(\begin{bmatrix}0 \\ 0 \\ 1 \end{bmatrix}) = \begin{bmatrix} 4x_1 + x_2 + 2x_3 \\2 x_2 - x_1 + 3x_3 \\ 5x_1 - 2x_3 + x_2  \end{bmatrix}=\begin{bmatrix}2 \\ 3 \\ -2 \end{bmatrix}\end{align}---Here are the transformation matrices\begin{align}p = \begin{bmatrix} 1 & 3 \\ -1 & 2 \end{bmatrix}\end{align}\begin{align}q = \begin{bmatrix} 4 & 1 & 2 \\ -1 & 2 & 3 \\ 5 & 1 & -2 \end{bmatrix}\end{align}  4.2 Verify that your transformation matrices are correct by choosing an input matrix and calculating the result both via the traditional functions above and also via vector-matrix multiplication.<jupyter_code>#not sure how to check my work

x1 = 1
x2 = 2 


p = array([[x1,3*x2],
           [2,2]])<jupyter_output><empty_output><jupyter_text>Eigenvalues and Eigenvectors Zhenya - need to learn don't understand!!!!!! FUCK 5.1 In your own words, give an explanation for the intuition behind eigenvalues and eigenvectors.What about the vector represented by the earth's axis? Does it move? We could call this vector the **eigenvector** of the earth's rotation brecause the transformation does not reorient it. Any transformation that you could conceive of possesses eigenvectors - dimensions of space that are not pointed in a new direction by the transformation described by a certain transformation matrix.  So what's an eigenvalue?An eigenvector is any vector that doesn't change its orientation during a given transformation, but an eigenvector might still get scaled by some scalar. It can get stretched or squished, it just can't change the direction in which it is pointing. An **eigenvalue** is the scalar that represents how its corresponding eigenvector was scaled during the transformation. Eigenvectors and eigenvalues always come in pairs.  The Curse of Dimensionality  6.1 What are some of the challenges of working with high dimensional spaces?* Cannot visualize beyond 3 dimensions reliably. This limitation is a challenge.  * Increased computational workload. * Measures of distance begin to lose meaning.   6.2 What is the rule of thumb for how many observations you should have compared to parameters in your model?<jupyter_code>#You should have at least 5 observations for every parameter (feature) in your model<jupyter_output><empty_output><jupyter_text>Principal Component Analysis  7.1 Code for loading and cleaning the 2013 national dataset from the [Housing Affordability Data System (HADS)](https://www.huduser.gov/portal/datasets/hads/hads.html) --housing data, can be found below.  Perform PCA on the processed dataset `national_processed` (Make sure you standardize your data!) and then make a scatterplot of PC1 against PC2. Some of our discussion and work around PCA with this dataset will continue during tomorrow's lecture and assignment.Not only does this dataset have decent amount columns to begin with (99), but in preparing the data for PCA we have also [one-hot-encoded](https://hackernoon.com/what-is-one-hot-encoding-why-and-when-do-you-have-to-use-it-e3c6186d008ftargetText=One%20hot%20encoding%20is%20a,the%20entry%20in%20the%20dataset.) all of the categorical variables. This has the effect of creating a new column for each individual category of each categorical variable. After processing this dataset has 64738 columns. --Das a lot of columns.Don't worry too much about the mechanics of one-hot encoding right now, you will learn and experiment with a whole bunch of categorical encoding approaches in unit 2. The code below will read in the dataset and perform the one-hot encoding of the categorical variables. Start adding your PCA code at the bottom of the provided code.<jupyter_code>import os.path
from urllib.request import urlopen
from zipfile import ZipFile
from io import BytesIO
import pandas as pd

# Read National Data 
national_url = 'https://www.huduser.gov/portal/datasets/hads/hads2013n_ASCII.zip'
national_file = 'thads2013n.txt'

if os.path.exists(national_file):
    national = pd.read_csv(national_file)
else: 
    z_national = urlopen(national_url)
    zip_national = ZipFile(BytesIO(z_national.read())).extract(national_file)
    national = pd.read_csv(zip_national)
# Look at first 5 rows
print(national.shape)
national.head()
# Look at datatypes
# a lot of object datatypes even though they look to be strings of numbers.
national.dtypes
# check for null values
national.isnull().sum().any()
# check for number of categorical vs numeric columns
cat_cols = national.columns[national.dtypes=='object']
num_cols = national.columns[national.dtypes!='object']

print(f'{len(cat_cols)} categorical columns')
print(f'{len(num_cols)} numerical columns')
national_processed = national.copy()

# Get One-Hot-Encoding of categorical variables
national_processed = pd.get_dummies(national_processed)
print(national_processed.shape)
national_processed.head()
national1 = national.select_dtypes(include=["float64","int64"])
mean(national1.T, axis=1)
#having trouble finding Z

from numpy import array
from numpy import mean
from numpy import std
from numpy import cov
from numpy.linalg import eig


print("Data: \n",national.head(1) )
# calculate the mean of each column
means = mean(national1.T, axis=1)
print("\n Means: \n", means)
# center columns by subtracting column means
centered_data = national1 - means
print("\n Centered Data: \n", centered_data)
# calculate the standard deviation of each column
std_devs = std(national1.T, axis=1)
print('\n Standard Deviations: \n', std_devs)
# complete the standardization by dividing by std_dev
standardized_data = centered_data / std_devs
print("\n Standardized Data: \n", standardized_data)
# calculate covariance matrix of centered matrix
# We're not going to find the covaraicne matrix by the (Z^T)(Z) method, I trust this function more
covariance_matrix = cov(standardized_data.T)
print("\n Covariance Matrix: \n", covariance_matrix)
# eigendecomposition of covariance matrix
values, vectors = eig(covariance_matrix)
print("\n Eigenvectors: \n", vectors)
print("\n Eigenvalues: \n", values)
# project data
P = vectors.T.dot(covariance_matrix.T)
print("\n Projected Data: \n", P.T)
# Principal Component Analysis
from numpy import array
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
# define a matrix
print("Data: \n", national1)
# Standardize the Data
# Instantiate a Standard Scaler object
scaler = StandardScaler()
# Use the object to fit_transform our data
Z = scaler.fit_transform(national1)
print("\n Standardized Data: \n", Z)
# create the PCA instance
# The integer value that we're passing in says 
# how many new columns we want to retain from PCA
pca = PCA(2)
# fit on data
pca.fit(Z)
# access values and vectors
print("\n Eigenvectors: \n", pca.components_)
print("\n Eigenvalues: \n",pca.explained_variance_)
# transform data
B = pca.transform(Z)
print("\n Projected Data: \n", B)
len(B)
PC1 = pd.DataFrame(B)
plt.scatter(PC1[0], PC1[1])
plt.title("Data After PCA")
plt.xlabel('PC1')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11730950141110066, 'fraction_numerical': 0.03386641580432737, 'mean_word_length': 4.571802935010482, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '611477', 'license': 'MIT', 'n_tokens_mistral': 3719, 'n_tokens_neox': 3398, 'n_words': 1244, 'path': 'module3-dimensionality-reduction/Zhenya_Warshavsky_High_Dimensional_Data_Assignment.ipynb', 'repo_name': 'zwarshavsky/AB-Demo', 'size': '10630'}