<jupyter_start><jupyter_text>Theory1. Problem: Estimate why do cactai grow in the parts of the desert. We don't really know what distinguishes high and low growth clusters. What is the ML system most suitable for the problem?We need to use non-supervised learning to estimate clusters. 2. How can the lack of validation set create a bias in the fit measurement?If we do not use validation and jump from training to testing this may result in over-testing where we find the model that fits peculiarities of testing data.3. How can we reduce overfitting?Simplify the model, add more data, drop model restriction.  Practice<jupyter_code># To plot pretty figures directly within Jupyter
%matplotlib inline
import matplotlib as mpl
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)
import os
# Code example
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import sklearn.linear_model
# Download the data
import urllib
DOWNLOAD_ROOT = "https://raw.githubusercontent.com/ageron/handson-ml2/master/"
for filename in ("oecd_bli_2015.csv", "gdp_per_capita.csv"):
    print("Downloading", filename)
    url = DOWNLOAD_ROOT + "datasets/lifesat/" + filename
    urllib.request.urlretrieve(url,filename)
# Code example
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import sklearn.linear_model

# Load the data
oecd_bli = pd.read_csv("oecd_bli_2015.csv", thousands=',')
gdp_per_capita = pd.read_csv("gdp_per_capita.csv",thousands=',',delimiter='\t',
                             encoding='latin1', na_values="n/a")

#1. Load dataset oecd_bli
import os
import pandas as pd
oecd_bli = pd.read_csv("oecd_bli_2015.csv", thousands=',')
oecd_bli = oecd_bli[oecd_bli["INEQUALITY"]=="TOT"]
oecd_bli = oecd_bli.pivot(index="Country", columns="Indicator", values="Value")
oecd_bli.head(2)<jupyter_output><empty_output><jupyter_text>1 Plot the relationship Linear model predicting Homicide rate with 'Educational Attaintment' and 'Life satisfaction' to the propostion: "...in much wisdom is much grief, and he who increases knowledge increases sorrow." Use a scatter plot: y='Life satisfaction', x='Educational attainment'  2 Estimate and report relationship Linear model predicting 'Life satisfaction'  using 'Educational attainment'. 3 Plot regression line and scatter plot together. Does the greated eduction is correlation with greated life satisfaction? 4 Drop all Scandinavian countries: ['Denmark', 'Estonia', 'Finland', 'Norway', 'Iceland'] and re-estimate the relationship between life_satisfcation and educational attainment. How did the results change?<jupyter_code>NoScanCountries.index<jupyter_output><empty_output><jupyter_text>5 Plot regression lines for all countries and countries without Scandinavia1.   List item2.   List item<jupyter_code># Plot regression lines for all countries and countries without Scanfinavia<jupyter_output><empty_output><jupyter_text>6. Load Boston. Dataset description is here: https://scikit-learn.org/stable/datasets/index.htmlboston-dataset    1. CRIM      per capita crime rate by town    2. ZN        proportion of residential land zoned for lots over                  25,000 sq.ft.    3. INDUS     proportion of non-retail business acres per town    4. CHAS      Charles River dummy variable (= 1 if tract bounds                  river; 0 otherwise)    5. NOX       nitric oxides concentration (parts per 10 million)    6. RM        average number of rooms per dwelling    7. AGE       proportion of owner-occupied units built prior to 1940    8. DIS       weighted distances to five Boston employment centres    9. RAD       index of accessibility to radial highways    10. TAX      full-value property-tax rate per $10,000    11. PTRATIO  pupil-teacher ratio by town    12. B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks                  by town    13. LSTAT    % lower status of the population    14. MEDV     Median value of owner-occupied homes in $1000's<jupyter_code># load libraries
from sklearn.datasets import load_boston
import pandas as pd
import random
# set random seed to get consistent results
random.seed(123)
np.random.seed(123) 
# Load Boston citi price data 
X, y = load_boston(return_X_y=True)
df = pd.DataFrame(data=X,columns=['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSAT'])
# create random indicator for pipeline
group_of_items = ['Good', 'Bad', 'Terrible', 'Excellient']           
df['QLTY'] = np.random.choice(group_of_items, len(df), p=[0.5, 0.1, 0.1, 0.3])
df_full = df.copy()
# Randomly delete observations for imputation
for i in range(0,13):
  c = df.columns[i]
  df.loc[df.sample(frac=0.1).index, c] = None
df_miss = df<jupyter_output><empty_output><jupyter_text>Create a pipeline that will:- create binary indicator for QLTY- Impute all missing variables based on means- Scale all number data to using Standard scalarBe careful not to scale categorical data: cat = ['CHAS', 'RAD', 'QLTY']To impute categorical variables use  SimpleImputer(strategy='most_frequent')<jupyter_code>from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer
# WE can also impute the missing data
from sklearn.impute import SimpleImputer
imputer = SimpleImputer()
from sklearn.preprocessing import OneHotEncoder<jupyter_output><empty_output><jupyter_text>7. Estimate linear regression predicting MEDV using  full data and imputed data: df_full_prepared, df_prepared. Report ther resulting RMSEs<jupyter_code>from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
lin_reg = LinearRegression()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0847988855998607, 'fraction_numerical': 0.017760752220094028, 'mean_word_length': 4.692765113974232, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '542804', 'license': 'MIT', 'n_tokens_mistral': 1831, 'n_tokens_neox': 1706, 'n_words': 625, 'path': 'lesson_2_ML_Project/HW2.ipynb', 'repo_name': 'JiaheXu/Statistical_ML', 'size': '5743'}