<jupyter_start><jupyter_text>Generating the test setNow let's extract some data as the test set, and we do not look at it.<jupyter_code>np.random.seed(42)
import numpy as np

def split_train_test(data, test_ratio):
    shuffled_indices = np.random.permutation(len(data))
    test_set_size = int(len(data) * test_ratio)
    test_indices = shuffled_indices[:test_set_size]
    train_indices = shuffled_indices[test_set_size:]
    return data.iloc[train_indices], data.iloc[test_indices]
train_set, test_set = split_train_test(housing, 0.2)
len(train_set)
len(test_set)
from zlib import crc32

def test_set_check(identifier, test_ratio):
    return crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2 ** 32
def split_train_test_by_id(data, test_ratio, id_column):
    ids = data[id_column]
    in_test_set = ids.apply(lambda id_: test_set_check(id_, test_ratio))
    return data.loc[~in_test_set], data.loc[in_test_set] 
housing_with_id = housing.reset_index()
train_set, test_set = split_train_test_by_id(housing_with_id, 0.2, "index")<jupyter_output><empty_output><jupyter_text>or we could do the following, to use a stable feature as the ID<jupyter_code>housing_with_id["id"] = housing["longitude"] * 1000 + housing["latitude"]
train_set, test_set = split_train_test_by_id(housing_with_id, 0.2, "id")<jupyter_output><empty_output><jupyter_text>or we could do it with Native Scikit way<jupyter_code>from sklearn.model_selection import train_test_split

train_set, test_set = train_test_split(housing, test_size=0.2, random_state=42)<jupyter_output><empty_output><jupyter_text>in order to avoid sampling bias using a totally random sample. We want to ensure that the test set in representative of the popuklation. We know that *medium income* is an important attribute. so we create **spectrum** for it.<jupyter_code>housing["income_cat"] = pd.cut(housing["median_income"],
                              bins   = [0., 1.5, 3.0, 4.5, 6.0, np.inf],
                              labels = [1, 2, 3, 4, 5])
housing["income_cat"].hist()<jupyter_output><empty_output><jupyter_text>now we would to to do **stratified sampling** based on the income category.<jupyter_code>from sklearn.model_selection import StratifiedShuffleSplit

split = StratifiedShuffleSplit(n_splits = 1, test_size = 0.2, random_state = 42)
for train_index, test_index in split.split(housing, housing["income_cat"]):
    strat_train_set = housing.loc[train_index]
    strat_test_set  = housing.loc[test_index]
strat_test_set["income_cat"].hist()
for set_ in (strat_train_set, strat_test_set):
    set_.drop("income_cat", axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Exploring the data<jupyter_code>housing = strat_train_set.copy()
housing.plot(kind="scatter", x="longitude", y="latitude", alpha=0.1)
housing.plot(kind="scatter", x="longitude", y="latitude", alpha=0.4,
            s=housing["population"] / 100, label="population", figsize=(10,7),
            c="median_house_value", cmap=plt.get_cmap("jet"), colorbar=True)
plt.legend()<jupyter_output><empty_output><jupyter_text>We then can calculate the *standard correlation coefficient* between every pair of attributes.<jupyter_code>corr_matrix = housing.corr()
corr_matrix["median_house_value"].sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>we can also check for correlation between attributes using pandas `scatter_matrix()` function<jupyter_code>from pandas.plotting import scatter_matrix

attributes = ["median_house_value", "median_income", "total_rooms", "housing_median_age"]

scatter_matrix(housing[attributes], figsize=(12,8))<jupyter_output><empty_output><jupyter_text>It seems like the most promising attribute to predict the median house value is the median income<jupyter_code>housing.plot(kind="scatter", x="median_income", y="median_house_value", alpha=0.1)<jupyter_output><empty_output><jupyter_text>Next we would like to create some extra features by combining the attributes<jupyter_code>housing["room_per_household"] = housing["total_rooms"] / housing["households"]
housing["bedrooms_per_room"] = housing["total_bedrooms"] / housing["total_rooms"]
housing["population_per_household"] = housing["population"] / housing["households"]
corr_matrix = housing.corr()
corr_matrix["median_house_value"].sort_values(ascending = False)<jupyter_output><empty_output><jupyter_text>Prepare the data for ML<jupyter_code>housing = strat_train_set.drop("median_house_value", axis=1)
housing_labels = strat_train_set["median_house_value"].copy()<jupyter_output><empty_output><jupyter_text>some ML algorithms cannot work with missing features, where the total_bedrooms attributes have some null value, therefore we can do the following<jupyter_code># housing.dropna(subset=["total_bedrooms"]) # drop the corresponding distrcits
# housing.drop("total_bedrooms", axis=1) # drop the feature completely
median = housing["total_bedrooms"].median()
housing["total_bedrooms"].fillna(median, inplace = True)<jupyter_output><empty_output><jupyter_text>or we can use SciKit's way to tackle NA values<jupyter_code>from sklearn.impute import SimpleImputer

imputer = SimpleImputer(strategy = "median")

housing_num = housing.drop("ocean_proximity", axis = 1) # housing_proximity contains text field where median cannot be applied

imputer.fit(housing_num)

# we can view the medians stored within the imputer
print(imputer.statistics_)

print(housing_num.median().values)

X = imputer.transform(housing_num)
# X is a plain NumPy array containing the transformed feature

housing_tr = pd.DataFrame(X, columns=housing_num.columns, index=housing_num.index)<jupyter_output>[-118.51     34.26     29.     2119.5     433.     1164.      408.
    3.5409]
[-118.51     34.26     29.     2119.5     433.     1164.      408.
    3.5409]
{'fraction_non_alphanumeric': 0.1121866897147796, 'fraction_numerical': 0.021780466724286948, 'mean_word_length': 6.465806451612903, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 52, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '43791', 'license': 'Apache-2.0', 'n_tokens_mistral': 2053, 'n_tokens_neox': 1930, 'n_words': 424, 'path': 'Untitled.ipynb', 'repo_name': 'Running-VincentW/handson-ml2', 'size': '5785'}