<jupyter_start><jupyter_text>INFSCI 2915 Foundations- Machine Learning - Spring 2018 Assignment 2Issued Monday, 2/26/2018; Due Monday, 11:59pm, 3/12/2018Total points: 100<jupyter_code># Type in your information in the double quotes
firstName = ""
lastName = ""
pittID = ""<jupyter_output><empty_output><jupyter_text>Problem 1. Train Test Split [20 points] In this part, you should download **"Iris dataset"**. Use a code below to download the  dataset:<jupyter_code>from sklearn.datasets import load_iris
dataset=load_iris()
print(dataset.keys())<jupyter_output>dict_keys(['data', 'target', 'target_names', 'DESCR', 'feature_names'])<jupyter_text>Problem 1-1.  Split iris dataset into training and test datasets. (use train_test_split function and random_state=0)- Report the shape of each (training and test) datasets. What is the proportion of each set?<jupyter_code>#Write your narrative answer here<jupyter_output><empty_output><jupyter_text>Problem 1-2.   Split iris dataset into training and test datasets. Make the proportion of training dataset 60% (use train_test_split function and random_state=0) - Report the shape of each (training and test) datasets.<jupyter_code>#Write your narrative answer here<jupyter_output><empty_output><jupyter_text>Problem 1-3.   Split iris dataset into training and test datasets. Make the proportion of test dataset 33.3% (use train_test_split function and random_state=0)  - Report the shape of each (training and test) dataset.<jupyter_code>#Write your narrative answer here<jupyter_output><empty_output><jupyter_text>Problem 1-4. Random State  Split iris dataset into training and test datasets, Use different Random State values (0,  10, 253, 1000, 0 and 10) - Analyze the shape and values of the train and test datasets  generated from different Random State values (you can use descriptive statistics) - What happens if you do not specify Random State? - What is a purpose of using Random State?<jupyter_code>#Write your narrative answer here<jupyter_output><empty_output><jupyter_text>Problem 1-5. Optional (Extra Points)  Write a python function that takes the dataset X  (matrix) and Y (numpy array) as an input and returns two non-overlapping datasets: the training and testing data, such that every entry to training set should be selected randomly. The proportion of the training and testing data should be 75%(training)/25%(test). Test your function using the synthetic dataset below.<jupyter_code>import numpy as np 
import pandas as pd
X = np.stack((np.arange(100), 10*np.array(np.arange(100)),100*np.array(np.arange(100)),1000*np.array(np.arange(100))), axis=-1)
Y = np.random.randint(2, size=100)
df = pd.DataFrame(np.concatenate((X,Y[:,None]),axis=1), columns = ("X1","X2","X3","X4", "Y"))
df.head(10)
#Write your narrative answer here<jupyter_output><empty_output><jupyter_text>Problem 2. Feature Selection  [10 points]   In this part, you should download and analyze **"Boston House Prices" ** dataset **with only 4 features**. (use code below to download  and change dataset)- Find the best two features using forward and backward selection to the Boston House Prices" dataset with only 4 features ('CHAS','RM','TAX', 'LSTAT'), use a linear regression model to find a best subset of features.- Comment your choice<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn.datasets import load_boston
dataset = load_boston()
import numpy as np 
import pandas as pd
#print(dataset.keys())
# Data preprocessing
new_data= pd.DataFrame(dataset.data, columns=dataset.feature_names)[['CHAS','RM','TAX', 'LSTAT']]

X_train,X_test,Y_train,Y_test=train_test_split(new_data.values,dataset.target,random_state=0)
#Write your narrative answer here<jupyter_output><empty_output><jupyter_text>Problem 3. Qualitative Variables  [10 points] In this part, you should download and analyze **"Boston House Prices"** dataset. Use a code below to download the  dataset:<jupyter_code>from sklearn.datasets import load_boston
dataset = load_boston()
print(dataset.keys())<jupyter_output>dict_keys(['data', 'target', 'feature_names', 'DESCR'])<jupyter_text>Problem 3-1.  Follow steps to answer questions.> *Use train_test_split() with the option "random_state=0".- Identify qualitative feature in the dataset (a feature with dummy variable). What is an abbreviation for the qualitative feature? (What is a definition of that  abbreviation ?)- Fit a linear regression model with qualitative variable feature only. Report intercept and coefficient.  - What can you infer about the qualitative feature from the coefficient? (Hint check lecture:  "Regression with Qualitative Features")<jupyter_code>#Write your narrative answer here<jupyter_output><empty_output><jupyter_text>Problem 4. Regularization [30 points]    Problem 4-1.  Answer the following questions - What is the objective using regularization?- Do you need to use regularization if you have only 1 feature (B0 and B1)?<jupyter_code>#Write your narrative answer here<jupyter_output><empty_output><jupyter_text>Problem 4-2.Ridge Regression   In this part, you should download and analyze **"Boston House Prices"** dataset. Use a code below to download the  dataset:<jupyter_code>dataset = load_boston()
print(dataset.keys())<jupyter_output>dict_keys(['data', 'target', 'feature_names', 'DESCR'])<jupyter_text>Follow steps to answer questions.- Fit a linear regression model with all features and report the estimated coefficients for the fitted model (do not just print summary, make a table with feature names and estimated coefficients) -  Fit a ridge regression model with λ = 1. Report the estimated coefficients for the fitted model. -  Fit a ridge regression model with λ = 0. Report the estimated coefficients for the fitted model. - Compare estimated coefficients of ridge regression models with λ = 1 and λ = 0  and linear regression model. (you can use descriptive statistics)- What did you observe from this comparison? - Comment your observation.<jupyter_code>#Write your narrative answer here<jupyter_output><empty_output><jupyter_text>Problem 4-3.Ridge Regression (part 2)   Fit a ridge regression with λ = 5, 10, 50, 100, and 1000. For each value, report the estimated coefficients for the fitted model (do not just print summary, make a DataFrame with feature names and estimated coefficients)- What happens to the coefficients as you increase λ?- What happens to the flexibility of the model as you increase λ?<jupyter_code>#Write your narrative answer here<jupyter_output><empty_output><jupyter_text>Problem 4-4 Lasso Regression   In this part, you should download and analyze **"Boston House Prices"** dataset.  Follow steps to answer questions.- Fit a linear regression model with all features and report the estimated coefficients for the fitted model (do not just print summary, make a DataFrame with feature names and estimated coefficients) -  Fit a lasso regression model with λ = 1. Report the estimated coefficients for the fitted model. -  Fit a lasso regression model with λ = 0. Report the estimated coefficients for the fitted model. - Compare estimated coefficients of lasso regression models with λ = 1 and λ = 0, ridge regression models with λ = 1 and λ = 0 and linear regression model. (you can use descriptive statistics)- What did you observe from this comparison? - Comment your observation.<jupyter_code>#Write your narrative answer here<jupyter_output><empty_output><jupyter_text>Problem 4-5.Lasso Regression (part 2)   Fit a lasso regression with λ = 5, 10, 50, 100, and 1000. For each value, report the estimated coefficients for the fitted model (do not just print summary, make a DataFrmae with feature names and estimated coefficients)- What happens to the coefficients as you increase λ?- What happens to the flexibility of the model as you increase λ?<jupyter_code>#Write your narrative answer here<jupyter_output><empty_output><jupyter_text>Problem 5.Logistic Regression [30 points]    Problem 5-1.   We fit a logistic regression model to predict the probability that an individual will default on his/her credit card balance. We used the total balance (single feature) to fit the model and got the results shown in the table below. |        |Coefficient| Std.error|Z -statistic|P-Value||:--:|:-------------------------------:||Intercept|-10.6513|0.3612|-29.5|<0.0001||balance|0.0055|0.002|24.9|<0.0001| - What is the parametric model used in logistic regression?- What is the probability that an individual with a balance equal to 15000 dollar will default?- What is the probability that an individual with balance equals to 800 dollar will not default?<jupyter_code>#Write your narrative answer here<jupyter_output><empty_output><jupyter_text>Problem 5-2.   The coefficients of logistic regression are obtained by maximizing the likelihood function\begin{array} \\l(\beta) = \prod_{i:y_{i}=1} P(y_{i} = 1|x)\prod_{i{}':y_{{i}'}=0} (1-P(y_{{i}'} = 1|x))\end{array}Show that maximizing thelikelihood function is equivalent to minimizing the cost function $J(\beta)$, such that.\begin{array} \\J(\beta) = -\sum [y_{i} log(P(y_{i} = 1|x)) + (1- y_{i})log(1- P(y_{i} = 1|x))]\end{array}Here $n$ is the number training examples. Mention one possible method for obtaining theoptimal coefficients.<jupyter_code>#Write your narrative answer here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09042211270623526, 'fraction_numerical': 0.02410542104135419, 'mean_word_length': 5.615875265768958, 'pattern_counts': {'":': 0, '<': 72, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '468711', 'license': 'Apache-2.0', 'n_tokens_mistral': 2817, 'n_tokens_neox': 2556, 'n_words': 1097, 'path': 'assignment2/Assignment2Spring2018v_2(1).ipynb', 'repo_name': 'BeMoreMature/ml_course', 'size': '9334'}