<jupyter_start><jupyter_text>Chapter 2<jupyter_code>import pandas as pd
import os
import numpy as np
import matplotlib.pyplot as plt
pwd = os.path.dirname(os.getcwd())
print(pwd)
datafolder = os.path.join(pwd,'datasets/housing')
print(datafolder)
filepath = os.path.join(datafolder, 'housing.csv')
print(filepath)
housing_data = pd.read_csv(filepath)
housing_data
housing_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 20640 entries, 0 to 20639
Data columns (total 10 columns):
 #   Column              Non-Null Count  Dtype  
---  ------              --------------  -----  
 0   longitude           20640 non-null  float64
 1   latitude            20640 non-null  float64
 2   housing_median_age  20640 non-null  float64
 3   total_rooms         20640 non-null  float64
 4   total_bedrooms      20433 non-null  float64
 5   population          20640 non-null  float64
 6   households          20640 non-null  float64
 7   median_income       20640 non-null  float64
 8   median_house_value  20640 non-null  float64
 9   ocean_proximity     20640 non-null  object 
dtypes: float64(9), object(1)
memory usage: 1.6+ MB<jupyter_text>As we get the data loaded, let's explore some of the sections. There are 9 columns of numbers, and then one object column, ocean_proximity. Object here is a string of some kind that could be a category or feature which we can use to organize the dataset.<jupyter_code>housing_data['ocean_proximity'].value_counts()
housing_data['ocean_proximity'].value_counts().plot(kind='barh')
housing_data.describe()<jupyter_output><empty_output><jupyter_text>So the experts you are working with tell you that median income is a good predictor of the price of the house. So let's bin that column to get some insight into that distribution<jupyter_code>housing_data['median_income'].hist()
housing_data['income_cat'] = pd.cut(x=housing_data['median_income'], 
                                                    bins=[0, 1.5, 3.0, 4.5, 6, np.inf],
                                                    labels=[1,2,3,4,5])
housing_data['income_cat'].value_counts()
housing_data['income_cat'].hist()<jupyter_output><empty_output><jupyter_text>When someone mentions a variable is intersting, it is a good idea to spend some time looking at that variable. Now let's start preparing our data set for machine learning. Let's set the y value to median_house_value since that is what we want to ultimately predict, and then the x axes are everything else in the data set. In ML, 'features' are interchangably used with columns of a dataset. so ocean_proximity is a 'feature'<jupyter_code>y = housing_data['median_house_value']
X = housing_data.drop('median_house_value', axis=1)
X<jupyter_output><empty_output><jupyter_text>Split DatasetsWe now need to take the data and make a training and testing set. The training set will be used to train the model, and the test set will be used to verify the accuracy of the model once it is done training. We want to make sure the model doesn't overfit.We know that this is good data and so we want to keep the training and the testing sets distinct from each other. We also want to randomize the selection from the good data so our training set is weighted appropriately and doesn't skew things to one side or the other. Read up about this in the books<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)
X_train<jupyter_output><empty_output><jupyter_text>Let's double check the distribtions of the total data set and the training set, just to make sure the distributions are approximate.<jupyter_code>housing_data['income_cat'].hist()
X_train['income_cat'].hist()<jupyter_output><empty_output><jupyter_text>Stratified Split<jupyter_code>from sklearn.model_selection import StratifiedShuffleSplit

split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_index, test_index in split.split(housing_data,housing_data['income_cat']):
    strat_train_set = housing_data.loc[train_index]
    strat_test_set = housing_data.loc[test_index]<jupyter_output><empty_output><jupyter_text>Let's chekc that the stratified split gave of close to the same ditribution of data.<jupyter_code>strat_test_set['income_cat'].value_counts() / len(strat_test_set)
housing_data['income_cat'].value_counts()/ len(housing_data)
# Since we used income_cat as a way to easily verify the distrubtions, we don't really need it, so we're going to drop it

strat_test_set.drop('income_cat', axis='columns', inplace=True)
strat_train_set.drop('income_cat', axis='columns', inplace=True)<jupyter_output><empty_output><jupyter_text>Visualizing our DataAlways visualize data, don't just just the stats. There are some datasets that can have the same median, mean, and mode, but be wildly different shapes, this is especially true as the data gets more and more dimensions.<jupyter_code># a good practice is to create a copy of the data to fiddle around with it first, if you make mistake you dont have to run a
# bunch of code to recreate the pandas dataset. 

# Avoid looking at the test dataset as much as possible. 

housing = strat_train_set.copy()<jupyter_output><empty_output><jupyter_text>Histograms<jupyter_code>housing.hist(bins=50, figsize=(20,15))<jupyter_output><empty_output><jupyter_text>Geographic Data<jupyter_code>housing.plot(kind='scatter', x ='longitude', y='latitude', alpha=0.4,
                s=housing['population']/100, label='population', figsize=(10,7), 
                c='median_house_value', cmap=plt.get_cmap('jet'), colorbar=True, 
                sharex=False)<jupyter_output><empty_output><jupyter_text>Correlation Matrix<jupyter_code>housing.corr()
import seaborn as sns

corr = housing.corr()

mask = np.triu(np.ones_like(corr, dtype=bool))

f, ax = plt.subplots(figsize=(11, 9))
cmap = sns.diverging_palette(230, 20, as_cmap=True)
sns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0, 
            square=True, linewidths=.5, cbar_kws={'shrink': .5})<jupyter_output><empty_output><jupyter_text>Preparing Data for Machine Learning<jupyter_code># set housing to the feature columns by dropping the median house value column
housing = strat_train_set.drop('median_house_value', axis=1)

# the labels we want to use are the house values (eg fit the features to these categories)
housing_labels = strat_train_set['median_house_value'].copy()<jupyter_output><empty_output><jupyter_text>Imuptation - Dealing with missing valuesFind missing or numll values, Option 1: Drop the rows<jupyter_code>missing_rows = housing[housing.isnull().any(axis=1)].head()
missing_rows
missing_rows.dropna(subset=['total_bedrooms'])
missing_rows.drop('total_bedrooms',axis=1)<jupyter_output><empty_output><jupyter_text>Option 3, Imputation. Whereever there is a missing value, we are going to insert the median value as a way to fill in the data. Here is some skill as a data scientist comes in. IS median the best for the data set or is there another figure of merit which is more in line with the data. For this example median is simple enough, but a better metric may be a number correlated with the house price.<jupyter_code>median = housing['total_bedrooms'].median()
missing_rows['total_bedrooms'].fillna(median, inplace=True)
missing_rows
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy='median')
#get rid of the categorical column 
housing_num = housing_data.select_dtypes(include=[np.number])
housing_num
imputer.fit(housing_num)
X = imputer.transform(housing_num)
housing_tr = pd.DataFrame(X, columns=housing_num.columns, index=housing_num.index)
housing_tr
housing_tr[housing_tr.isnull().any(axis=1)].head()<jupyter_output><empty_output><jupyter_text>Encode Categorical Variables<jupyter_code>from sklearn.preprocessing import OrdinalEncoder

housing_cat = housing[['ocean_proximity']]
housing_cat.head(10)


ordinal_encoder = OrdinalEncoder()
housing_cat_encoded = ordinal_encoder.fit_transform(housing_cat)
housing_cat_encoded[:10]
from sklearn.preprocessing import OneHotEncoder

cat_encoder = OneHotEncoder(sparse=False,)
housing_cat_1hot = cat_encoder.fit_transform(housing_cat)
housing_cat_1hot[:10]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0899830220713073, 'fraction_numerical': 0.019524617996604415, 'mean_word_length': 5.305045871559633, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '462497', 'license': 'Apache-2.0', 'n_tokens_mistral': 2764, 'n_tokens_neox': 2592, 'n_words': 786, 'path': 'MyNotes/Ch2.ipynb', 'repo_name': 'bigm141414/handson-ml', 'size': '8246'}