<jupyter_start><jupyter_text>Visualizing the Boston Housing DataNow that we have seen some of the basic mechanics of `Matplotlib`, let's work with some actual data. For that, we are going to use the Boston Housing dataset.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt

from sklearn import datasets

%matplotlib inline<jupyter_output><empty_output><jupyter_text>I have added a couple dependencies that your environment may not have yet. Uncomment the following lines to install them.<jupyter_code># import sys
# !conda install --yes --prefix {sys.prefix} scikit-learn
# !conda install --yes --prefix {sys.prefix} pandas
# Matplotlib has some built in style sheets
mpl.style.use('fivethirtyeight')
boston = datasets.load_boston()

# Sklearn uses a dictionary like object to hold its datasets
X = boston['data']
y = boston['target']

feature_names = list(boston.feature_names)
X.shape
# Some info about our data
print(boston['DESCR'])
print(f'Dataset features contains {X.shape[0]} samples, and {X.shape[1]} data dimensions.')
print(f'Each of the {y.shape[0]} samples has an real valued label.')<jupyter_output>Dataset features contains 506 samples, and 13 data dimensions.
Each of the 506 samples has an real valued label.<jupyter_text>What are These Data Features?There are thirteen dimensions in this data. What types of<jupyter_code># Take a look at the type of every *column* of our array
for idx, feature in enumerate(X.T):
    print(f'{feature_names[idx]} is of type {feature.dtype}')<jupyter_output>CRIM is of type float64
ZN is of type float64
INDUS is of type float64
CHAS is of type float64
NOX is of type float64
RM is of type float64
AGE is of type float64
DIS is of type float64
RAD is of type float64
TAX is of type float64
PTRATIO is of type float64
B is of type float64
LSTAT is of type float64<jupyter_text>A Small CheatWe haven't introduced the pandas library yet, but I am going to sneak it in here to give us a quick sense of our data.<jupyter_code>X_df = pd.DataFrame(X)
X_df.columns = boston.feature_names
X_df.describe()<jupyter_output><empty_output><jupyter_text>A Bird's Eye ViewOne of my first goto visualizations is the histogram. This will give us a quick visual check to see if there are outliers.<jupyter_code># Check out the distributions of our features
# Note our y-axes across the subplots. Which is more informative,
# sharing that y-axis information across subplots or not?
fig, axs = plt.subplots(ncols=7, nrows=2, figsize=(20, 10), sharey=False)
axs = axs.flatten()

for idx, feature in enumerate(X.T):
    axs[idx].hist(feature)
    axs[idx].set_title(f'{feature_names[idx]}')

# Since we have 2 rows each with 3 columns,
# and only 13 features, then we can visualize
# our target in the last `Axes` index.
axs[-1].hist(y, color='orange')
axs[-1].set_title(f'Target: MEDV')

plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=5.0)<jupyter_output><empty_output><jupyter_text>At first glance we can see that a few of these distributions are skewed. Though we haven't talked about machine learning algorithms yet, can you guess how important `Feature 3` is to our data? On one hand, I can imagine that a model might not be great at distinguishing the importance of this feature. Many of the samples have a low value for this feature, and it may be difficult to distinguish one sample from another given this feature. On the other hand, knowing that a sample has a high value for this feature might be informative.Let's take a look at the same information, using a boxplot instead.<jupyter_code>fig, axs = plt.subplots(ncols=7, nrows=2, figsize=(20, 10), sharey=False)
axs = axs.flatten()

for idx, feature in enumerate(X.T):
    axs[idx].boxplot(feature)
    axs[idx].set_title(f'{feature_names[idx]}')

axs[-1].boxplot(y)
_ = axs[-1].set_title(f'Target: MEDV')

plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=5.0)
# There are better ways of keeping track of this sort of thing 
# with pandas, but since we have been working with numpy up until
# this point, forgive me for indexing in this way. Indexing errors
# are common in programming, so we have to make sure we know what we
# are getting when we slice into an array like this.
proportion_non_retail = X[:, 2]
pupil_teacher_ratio = X[:, 10]

hist_features = {
    'proportion_non_retail': proportion_non_retail, 
    'pupil_teacher_ratio': pupil_teacher_ratio
}

fig, ax = plt.subplots(figsize=(20, 10))

for label, feature in hist_features.items():
    ax.hist(feature, label=label, alpha=0.3)

plt.legend(fontsize='x-large')
plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=5.0)<jupyter_output><empty_output><jupyter_text>How Do These Features Relate to our Target?Let's take a closer look at some of our features and how they relate to the median home value in Boston. Let's first plot one of features along the x-axis with the median home value along the y-axis.<jupyter_code># Simple approach with state machine approach
plt.figure(figsize=(20, 10))

plt.scatter(
    pupil_teacher_ratio, 
    y, 
    s=100,            # keeping a uniform size for all points
    alpha=0.5,        # transparency for overlapping points
    edgecolor="black" # give definition to each point
)

plt.xlabel("Pupil to Teacher Ratio")
plt.ylabel(r"Median Home Value ($\times \$1000$)")
plt.suptitle("Pupil to Teacher Ratio By Housing Value")
plt.show()<jupyter_output><empty_output><jupyter_text>Sometimes we want to see how two or more features relate to one another. Let's create a scatter plot showing how the proportion of owner-occupied units built prior to 1940 relates to the weighted distances to five Boston employment centres. We can layer in extra information by coloring our points by our target variable, the median value of homes in Boston. Add layers of design features like color and the size of our plotted points can give us a bit more nuance in our visualizations. Just be careful, we should always consider whether each element in our visualization adds information.<jupyter_code>proportion_owner_occupied = X[:, 6]
distances_to_employment = X[:, 7]

# Fancier example using the state machine approach
plt.figure(figsize=(20, 10))

plt.scatter(
    proportion_owner_occupied, 
    distances_to_employment, 
    s=100,        # keeping a uniform size for all points
    c=y,          # color the points by our target
    cmap='magma', # colormaps can make things easier to read
    alpha=0.5     # transparency for overlapping points
)

cbar = plt.colorbar()
cbar.set_label(r"Median Home Value ($\times \$1000$)")

plt.xlabel("Proportion of Owner Occupied Units Built Prior to 1940")
plt.ylabel("Weighted Mean Distance to Employment Center")
plt.suptitle("Owner Occupied Homes and Distance to Employment")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08356669123065585, 'fraction_numerical': 0.017833456153279293, 'mean_word_length': 4.834909716251075, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '435658', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2217, 'n_tokens_neox': 2058, 'n_words': 862, 'path': 'examples/notebooks/module_02_03_boston_housing_data.ipynb', 'repo_name': 'yngtodd/eleven', 'size': '6785'}