<jupyter_start><jupyter_text>INFSCI 2915 Foundations- Machine Learning - Spring 2018 Assignment 1Issued Tuesday, 2/06/2018; Due Tuesday, 11:59pm, 2/13/2018Total points: 100<jupyter_code># Type in your information in the double quotes
firstName = "YUHAO"
lastName = "WU"
pittID = "yuw121"<jupyter_output><empty_output><jupyter_text>--   Problem 1. K-nearest neighbors  [20 points]  The table below provides a training data set containing six observations, three predictors, and one qualitative response variable.Suppose we wish to use this data set to make a prediction for Y when X1 = X2 = X3 = 0 using K-nearest neighbors. |Obs.|X1|X2|X3|Y|--|-------------------------------||1 |0|3|0|Green||2 |2|0|0|Green||3|0|1|3|Green||4|0|1|2|Red||5|-1|0|1|Red||6|1|1|1|Green| Problem 1-1. Compute the Euclidean distance between each observation and the test point, X1 = X2 = X3 = 0<jupyter_code>#Write your narrative answer here
import numpy as np
x=np.array([[0,3,0],
           [2,0,0],
           [0,1,3],
           [0,1,2],
           [-1,0,1],
           [1,1,1]])
test=[0,0,0]
Obs=1
for i in x:
    dist=np.linalg.norm(i-test)
    print("The distance between observation ",Obs," and test is ",dist)
    Obs=Obs+1;<jupyter_output>The distance between observation  1  and test is  3.0
The distance between observation  2  and test is  2.0
The distance between observation  3  and test is  3.1622776601683795
The distance between observation  4  and test is  2.23606797749979
The distance between observation  5  and test is  1.4142135623730951
The distance between observation  6  and test is  1.7320508075688772<jupyter_text>Problem 1-2. What is our prediction with K = 1? and explain why.<jupyter_code>#Write your narrative answer here
import pandas as pd
from sklearn.neighbors import KNeighborsClassifier 
from sklearn.model_selection import train_test_split
from sklearn import preprocessing

y=np.array(["Green","Green","Green","Red","Red","Green"])
# y=y[:,np.newaxis]
# dataset=np.concatenate((x,y),axis=1)

k=1
knn=KNeighborsClassifier(n_neighbors=k)
knn.fit(x,y)
print(knn.predict([[0,0,0]]))
# predict_pro=knn.predict_proba([[0,0,0]])
# print("When k= ",k," Accuracy is ",predict_pro)
When we just find the one of the nearest points, the red one is the shortest distance one.<jupyter_output><empty_output><jupyter_text>Problem 1-3. What is our prediction with K = 3? and explain why.<jupyter_code>#Write your narrative answer here
k=3
knn=KNeighborsClassifier(n_neighbors=k)
knn.fit(x,y)
print(knn.predict([[0,0,0]]))
When we choose three of the nearest points, we find two points are green, 
whereas the remain one is red, so we choose the more points' color as our result of prediction<jupyter_output><empty_output><jupyter_text>Problem 1-4.  How using a different number of K has an impact on the performance? explain.<jupyter_code>#Write your narrative answer here
K is K points which are closest to test point,
if we just choose few points, may be our prediction will be limited in a small range(overfitting) and have high variance.
But if we choose too many points, may be our prediction will be underfitting and have high bias.<jupyter_output><empty_output><jupyter_text>---   Problem 2. Answer those questions about linear regression [30 points]   - Writing a code is not required for this question- Feel free to select any type of submission you are comfortable with (Since it may require some mathematical formula and symbols; MS Word, or scanned version of your writing will be fine)  Problem 2-1. Find the least squared fit of a linear regression model using the following traning data |Smoke index(x)|Lung cancer mortality index(y)||:--:|:-------------------------------:||127|115||121|128||94|128||126|156||102|101||111|128||95|115||89|105||101|87|<jupyter_code>#Write your narrative answer here
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

x=np.array([127,121,94,126,102,111,95,89,101])
y=np.array([115,128,128,156,101,128,115,105,87])
x_ave=np.mean(x)
y_ave=np.mean(y)
up=0;
down=0

for index in range(len(x)):
    up=up+(y[index]-y_ave)*(x[index]-x_ave)
    down=down+pow(x[index]-x_ave,2)
B1=up/down
B2=y_ave-B1*x_ave
print(B1)
print(B2)


# method two
x=x.reshape(-1,1)
y=y.reshape(-1,1)
linreg= LinearRegression().fit(x, y)
print("The coefficient of TV feature is:",linreg.coef_)
print("The intercept is: ", linreg.intercept_)<jupyter_output>0.7500998003992017
37.6003992015968
The coefficient of TV feature is: [[0.7500998]]
The intercept is:  [37.6003992]<jupyter_text>Problem 2-2. Given the test data below, compute the R-squared metric of the fitted model |Xi|Yi||:--:|:-------------------------------:||90|103||106|131||105|85||115|99||113|144|<jupyter_code>#Write your narrative answer here
from sklearn.metrics import r2_score
xi=np.array([90,106,105,115,113])
yi=np.array([103,131,85,99,144])


y_pred=B2+B1*xi
RSS=sum(pow((y_pred-yi),2))
TSS=sum(pow((yi-np.mean(yi)),2))
R2=1-RSS/TSS
print(R2)


# method two
print(r2_score(yi,y_pred))<jupyter_output>0.040627525222455496
0.040627525222455496<jupyter_text>---   Problem 3. This question involves the use of multiple linear regression on the Boston dataset [30 points] In this part, you should download and analyze **"Boston House Prices"** dataset. Here use a code below to download the  dataset:<jupyter_code>from sklearn.datasets import load_boston
import pandas as pd
import numpy as np
dataset = load_boston()
print(dataset.keys())<jupyter_output>dict_keys(['data', 'target', 'feature_names', 'DESCR'])<jupyter_text>Problem 3-1 Print the description of the dataset.  Answer the following question: - Whole dataset contains 14 attributes, (13 numeric/categorical predictive and 1 target value)  what is a abbreviation of the target value?<jupyter_code>#Write your narrative answer here
print(dataset.DESCR)
 MEDV  Median value of owner-occupied homes in $1000's<jupyter_output><empty_output><jupyter_text>Problem 3-2  Generate descriptive statistics using DataFrame. (hint: use "DataFrame . describe" function)Follow two steps to answer questions.- Create a DataFrame usnig "data" from the dataset with columns using "feature_names".- Generate descriptive statistics  Answer the following questions: -  Which feature has the highest range (minimum and maximum value)?-  Which feature has the lowest mean?  -  Which feature has a lowest standard deviation?<jupyter_code>#Write your narrative answer here
df=pd.DataFrame(dataset.data,columns=dataset.feature_names)
df.describe()
Answer:
TAX have the highest range.
CHAS have lowest mean.
CHAS have lowest standard deviation<jupyter_output><empty_output><jupyter_text>Problem 3-3  Feature Scaling Answer the following questions: - From the information above, Do you recommend **feature scaling** to improve performance? Explain.  - What is a difference between MinMaxScaler and StandardScaler?<jupyter_code>#Write your narrative answer here

I recommend feature scaling because the value of each feature have a large difference.
If we don't do job in feature scaling, some features will cause larger influence on the prediction.
We should keep balance in weight
MinMaxScaler: scales features to be in range 0 -1
StandardScaler: scales features so that they are all with zero mean and unit
variance<jupyter_output><empty_output><jupyter_text>Problem 3-4  Calculate and report **correlations** between features and the target  Answer the following questions: - What is a difference between positive and negative numbers on the correlation table? Explain.- What is the lowest number in the table? Do you think it is the most or the least helpful features in predicting thetarget class? Explain.<jupyter_code>#Write your narrative answer here
df['target'] = dataset.target
print(df.corr())

Answer:
the positive number means this feature has positive correlation with target, when it add, target will add
the negative number means this feature has negative correlation with target, when it add, target will decrease
the lowest number is -0.737663 (LSTAT) includes negative number,it has the most helpful influnence on target.
or we do not consider negative number, the lowest number is 0.175260 (CHAS) which is least helpful in target.<jupyter_output><empty_output><jupyter_text>Problem 3-5  Follow two steps to answer questions.- Add 1 more column (dataset.target) to your DataFrame (give a name).- Find the correlation matrix that shows the correlation between each pair of features. - Plot a correlation matrix You can use the code below or write your own code to plot a correlation matrix *(extra point for writing your own code)*  Answer the following questions: - What is the correlation between the feature RM and the LSTAT? - What does this value of correlation indicate?- Can you detect most or the least helpful features in predicting the target class from correlation matrix or from plot? How?<jupyter_code>%matplotlib notebook
%matplotlib inline
from sklearn.datasets import load_boston
dataset = load_boston()
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
sns.set(style="white")
# Get correlation matrix
df = (pd.DataFrame(dataset.data, columns= dataset.feature_names))
df['target'] = dataset.target

plt.matshow(df.corr())

corr = df.corr()
# # Generate a mask for the upper triangle
# mask = np.zeros_like(corr, dtype=np.bool)
# mask[np.triu_indices_from(mask)] = True
# # Set up the matplotlib figure
# f, ax = plt.subplots(figsize=(11, 9))
# # Generate a custom diverging colormap
# cmap = sns.diverging_palette(220, 10, as_cmap=True)
# # Draw the heatmap with the mask and correct aspect ratio
# sns.heatmap(corr, mask=mask, cmap=cmap, vmax=1, center=0, square=True, linewidths=.9, cbar_kws={"shrink": .7})
#Write your narrative answer here
We can see in the graph the correlation between LSTAT and RM is smaller than negative 0.6, 
which means these two features have large negative correlation.
When we talked about the most helpful features, we just need to find the maximum absolute value in the row of target in matrix,
or the most blue or the most red one in the row of target in the plot.
On the contrary, when we find the least helpful features, we just need to find the minimum absolute value in the row of target in matrix,
or the least blue or the least red one in the row of target in the plot<jupyter_output><empty_output><jupyter_text>Problem 3-6  Scatter plot.  Follow three steps to answer questions  - Plot LSTAT versus target variable - Plot RM versus target variable - Plot CHAS versus target variable   Answer the following questions: - Is the relationship linear?- Which looks the most nonlinear? Explain your chouce<jupyter_code>#Write your narrative answer here
import matplotlib.pyplot as plt

plt.scatter(df.LSTAT,df.target,marker='o')
plt.xlabel('LSTAT')
plt.ylabel('target')
plt.title('LSTAT vs target')
plt.show()

plt.scatter(df.RM,df.target,marker='o')
plt.xlabel('RM')
plt.ylabel('target')
plt.title('RM vs target')
plt.show()

plt.scatter(df.CHAS,df.target,marker='o')
plt.xlabel('CHAS')
plt.ylabel('target')
plt.title('CHAS vs target')
plt.show()

x=df.drop('target',axis=1)
y=df['target']
linreg= LinearRegression().fit(x, y)
print("The coefficient of TV feature is:",linreg.coef_)
print("The intercept is: ", len(linreg.coef_))


RM looks like liner and CHAS looks the most nonliner, because the value of CHAS focus on the 0.0 or 1.0<jupyter_output><empty_output><jupyter_text>Problem 3-7. Follow steps to answer questions.> *Use train_test_split() with the option "random_state=0".1. Fit a linear regression model with RM and LSTAT features only. Find the R-squared metric. 2. Fit a linear regression model using RM, LSTAT and include the interaction term (RM * LSTAT). How R-squared metric differs from the previous model without interaction term?3. Fit a linear regression model using LSTAT and include the polynomial term ( $LSTAT^2$).  Find the R-squared metric.4. Fit linear regression model using LSTAT and include the polynomial term ( $LSTAT^2$ and $LSTAT^3$ ). Find the R-squared metric. - How does R-squared metric differ in the previous models ?  Comment your observation.<jupyter_code>RM=df.RM.values.reshape(-1,1)
LSTAT=df.LSTAT.values.reshape(-1,1)
x=np.concatenate((RM,LSTAT),axis=1)
y=df.target.values.reshape(-1,1)
X_train, X_test, Y_train, Y_test= train_test_split(x, y, random_state= 0)
model=LinearRegression()
fitted_model=model.fit(X_train,Y_train)
print("1. RM and LSTAT, R-squared metric is ",fitted_model.score(X_test,Y_test))


x2=np.concatenate((x,RM*LSTAT),axis=1)
y=df.target.values.reshape(-1,1)
X_train, X_test, Y_train, Y_test= train_test_split(x2, y, random_state= 0)
model2=LinearRegression()
fitted_model2=model2.fit(X_train,Y_train)
print("2. RM, LSTAT and RM*LSTAT, R-squared metric is ",fitted_model2.score(X_test,Y_test))
print("   R-squared metric increase with the interaction term")



x3=np.concatenate((LSTAT,LSTAT**2),axis=1)
y=df.target.values.reshape(-1,1)
X_train, X_test, Y_train, Y_test= train_test_split(x3, y, random_state= 0)
model=LinearRegression()
fitted_model=model.fit(X_train,Y_train)
print("3. LSTAT and LSTAT*LSTAT, R-squared metric is ",fitted_model.score(X_test,Y_test))


x4=np.concatenate((x3,LSTAT**3),axis=1)
y=df.target.values.reshape(-1,1)
X_train, X_test, Y_train, Y_test= train_test_split(x4, y, random_state= 0)
model=LinearRegression()
fitted_model=model.fit(X_train,Y_train)
print("4. LSTAT.LSTAT*LSTAT and LSTAT*LSTAT*LSTAT, R-squared metric is ",fitted_model.score(X_test,Y_test))
print("   the second model has the highest score which means it is the best one. With the add higher polynomial parameter in LASTA, R-squared metric become higher and higher")<jupyter_output>1. RM and LSTAT, R-squared metric is  0.5692445415835343
2. RM, LSTAT and RM*LSTAT, R-squared metric is  0.6615207706770674
   R-squared metric increase with the interaction term
3. LSTAT and LSTAT*LSTAT, R-squared metric is  0.5518753354137798
4. LSTAT.LSTAT*LSTAT and LSTAT*LSTAT*LSTAT, R-squared metric is  0.5768833545180131
   the second model has the highest score which means it is the best one. With the add higher polynomial parameter in LASTA, R-squared metric become higher and higher<jupyter_text>Problem 3-8. Fit all features (13 features) in the dataset to a multiple linear regression model, and report > (1) p-values for each feature.  Answer the following questions: - What does p-value means- What are the important features?<jupyter_code>import statsmodels.formula.api as smf
dataset=load_boston()

X=dataset.data
Y=dataset.target
model = smf.OLS(Y,X)
m1=model.fit()
print("P-values are: ",m1.pvalues)
the features which have small p-values are important features<jupyter_output><empty_output><jupyter_text>---   Problem 4. Ordinary Least Square (OLS) [20 points]   Problem 4-1.  Ordinary Least Square (OLS), calculate parametersFollow two steps - From your DataFrame generate new DataFrame with 2 columns ('LSTAT' and target)  - Using the least squared formula below write a function to calulate $B_0$  and $B_1$  from your new DataFrame where 'LSTAT' = x and target = y. *(hint you can add 2 colums to your DataFrame($x*x$  and  $x*y$), check the problem 2-1)* \begin{array} \\    \hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x}, \\ \\    \hat{\beta}_1 = \frac{\sum_{i=1}^n y_i x_i - \bar{y}\sum_{i=1}^n x_i}{\sum_{i=1}^n x^2 - \bar{x}\sum_{i=1}^n x_i} \\ \\    \text{where }\bar{x} = \frac{\sum_{i=1}^y x_i}{n} \text{ and } \bar{y} = \frac{\sum_{i=1}^y y_i}{n} \\\end{array}<jupyter_code>#Write your narrative answer here
df = (pd.DataFrame(dataset.data, columns= dataset.feature_names))
df['target']=dataset.target
df2=df[['LSTAT','target']]

LSTAT=df2.LSTAT.values.reshape(-1,1)
target=df2.target.values.reshape(-1,1)
# X_train, X_test, Y_train, Y_test= train_test_split(LSTAT,target, random_state= 0)
# linreg= LinearRegression().fit(X_train, Y_train)
# print("The coefficient of TV feature is:",linreg.coef_)
# print("The intercept is: ", linreg.intercept_)


# method two
L_ave=np.mean(LSTAT)
t_ave=np.mean(target)
up=0;
down=0

for index in range(len(LSTAT)):
    up=up+(target[index]-t_ave)*(LSTAT[index]-L_ave)
    down=down+pow(LSTAT[index]-L_ave,2)
B1=up/down
B0=t_ave-B1*L_ave
print("B1 is ",B1)
print("B0 is ",B0)<jupyter_output>B1 is  [-0.95004935]
B0 is  [34.55384088]<jupyter_text>Problem 4-2.  Ordinary Least Square (OLS) Using formula below and using $B_0$ and $B_1$ from previous problem calculate $\hat{y}$ $where$ x = 'LSTAT'- Add your predicted data to the DataFrame  \begin{array} \\    \hat{y} = \hat{\beta}_0 + \hat{\beta}_1*x, \\ \\\end{array}<jupyter_code>#Write your narrative answer here

t_pred=B2+B1*LSTAT
# RSS=sum(pow((t_pred-target),2))
# TSS=sum(pow((target-np.mean(target)),2))
# R2=1-RSS/TSS
# print(R2)


t_pred=t_pred.reshape(-1)
df3 = (pd.DataFrame(df2))
df3['predict']=pd.Series(t_pred)
df3<jupyter_output><empty_output><jupyter_text>Problem 4-3.  Verification Predict $\hat{y}$ using linear regression model *(for this part you don't need to use train test split) *- Add your predicted data to the DataFrameCompare 3 column:- target - predicted column using our own code, (problem 2-9)- predicted column using linear regression model Comment your observations<jupyter_code>#Write your narrative answer here
linreg= LinearRegression().fit(LSTAT, target)
print("The coefficient of TV feature is:",linreg.coef_)
print("The intercept is: ", linreg.intercept_)

predict=linreg.predict(LSTAT)
predict=predict.reshape(-1)
df4 = (pd.DataFrame(df3))
df4['liner regression predict']=pd.Series(predict)
df4
my own code and liner regression have same prediction
it proves that when using same dataset, liner regression will have same result.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11390765765765766, 'fraction_numerical': 0.041385135135135136, 'mean_word_length': 5.3026969481902055, 'pattern_counts': {'":': 1, '<': 71, '<?xml version=': 0, '>': 73, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '489885', 'license': 'Apache-2.0', 'n_tokens_mistral': 6299, 'n_tokens_neox': 5770, 'n_words': 1966, 'path': 'assignment1/YUHAO_WU-yuw121-Assignment1.ipynb', 'repo_name': 'BeMoreMature/ml_course', 'size': '17760'}