<jupyter_start><jupyter_text>Regression in `scikit-learn` Goals- Practice with the `fit` and `predict` interface of sklearn models- Get a visual sense of how different regression models work. RationaleNeural nets are strong performers for data that lacks clear features. But for well-structured tabular data with meaningful features (or data that can be translated to that form), simple models can sometimes perform very well, and can be much faster and sometimes more interpretable. Even if you plan to fit a neural net model, training a decision tree or random forest first can be a good quick first pass.The Scikit-Learn (sklearn) `fit`-`predict` interface for modeling has become the *de facto* industry standard for this sort of modeling, so it's highly likely that what you see here will be useful in your future work. DocumentationThe sklearn documentation is exemplary. See:- [Linear Models](https://scikit-learn.org/stable/modules/linear_model.html) for, e.g., linear regression- [Decision Trees](https://scikit-learn.org/stable/modules/tree.html)- [Ensemble Methods](https://scikit-learn.org/stable/modules/ensemble.html) for, e.g., random forests  Setup Let's import necessary modules: *pandas* and NumPy for data wrangling, Matplotlib for plotting, and some sklearn models.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor
import sklearn.tree
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from sklearn.metrics import mean_absolute_error, mean_squared_error, log_loss, accuracy_score, classification_report
from sklearn.model_selection import train_test_split<jupyter_output><empty_output><jupyter_text>We'll load the data. We're using a dataset of home sale prices from the Ames, Iowa assessor's database, described in [this paper](http://ww2.amstat.org/publications/jse/v19n3/decock.pdf). (DATA 202 students may remember seeing this dataset.)Pandas (typically imported as `pd`, see above) is a very useful library for working with tabular datasets. We'll see here that we can easily read a CSV file directly off the Internet...<jupyter_code>ames = pd.read_csv('https://github.com/kcarnold/AmesHousing/blob/master/data/ames.csv.gz?raw=true', compression="gzip")<jupyter_output><empty_output><jupyter_text>The main object from pandas is a `DataFrame`. It holds a table of data:<jupyter_code>ames.head()<jupyter_output><empty_output><jupyter_text>Each column of data generally has a consistent data type. (Note: `object` columns are the exception. They usually mean "string", but could actually hold any Python object.)<jupyter_code>ames.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2930 entries, 0 to 2929
Data columns (total 81 columns):
 #   Column              Non-Null Count  Dtype  
---  ------              --------------  -----  
 0   MS_SubClass         2930 non-null   object 
 1   MS_Zoning           2930 non-null   object 
 2   Lot_Frontage        2930 non-null   int64  
 3   Lot_Area            2930 non-null   int64  
 4   Street              2930 non-null   object 
 5   Alley               2930 non-null   object 
 6   Lot_Shape           2930 non-null   object 
 7   Land_Contour        2930 non-null   object 
 8   Utilities           2930 non-null   object 
 9   Lot_Config          2930 non-null   object 
 10  Land_Slope          2930 non-null   object 
 11  Neighborhood        2930 non-null   object 
 12  Condition_1         2930 non-null   object 
 13  Condition_2         2930 non-null   object 
 14  Bldg_Type           2930 non-null   object 
 15  House_Style         2930 non-null   object 
 16  Overal[...]<jupyter_text>It behaves like a dictionary of its columns. Each column is a `Series` object.<jupyter_code>type(ames['Sale_Price'])<jupyter_output><empty_output><jupyter_text>`Series` support broadcast operations, similar to NumPy arrays and Torch `tensor`s; they also have other functionality.<jupyter_code>ames['price'] = ames["Sale_Price"] / 100_000 # Make `price` be in units of $100k, to be easier to interpret.<jupyter_output><empty_output><jupyter_text>Now we'll look into this dataset:We'll define some functions to plot the data and models. Since we have latitude and longitude for each home, we can plot this data in 2D with a color for the sale price.(Sorry, you'll just have to imagine there's a map underneath.)<jupyter_code>def plot_data():
    # You don't have to know how this function works.
    plt.scatter(ames['Longitude'], ames['Latitude'], c=ames["price"], s=.5)
    plt.xlabel("Longitude"); plt.ylabel("Latitude")
    plt.colorbar(label="Sale Price ($100k)")
plot_data()<jupyter_output><empty_output><jupyter_text>We'll try to predict home price based on *location* (which the realtors assure us is the most important factor anyway). So we'll grab the Latitude and Longitude columns of the data. We'll call that input data `X`, by convention. There are several different ways to index into a pandas `DataFrame`; using a `list` gives us a `DataFrame` with just the columns with those names. We'll then access the underlying NumPy data by using `.values`.<jupyter_code>X = ames[['Longitude', 'Latitude']].values
X.shape<jupyter_output><empty_output><jupyter_text>Our target, called `y` by convention, will be the home price (we'll soon introduce a different *y*, but start with this one).<jupyter_code>y = ames['price'].values
y.shape<jupyter_output><empty_output><jupyter_text>Notice that `X` has two axes and thus is written in uppercase; `y` has 1 and thus is written in lowercase. (This is `sklearn` convention; other libraries are less consistent about this.) Now let's split the data into a `train` and `valid` set (which sklearn calls train-*test*, but that's fine). `random_state` is how `sklearn` specifies the random seed (it's actually slightly more flexible than a seed).<jupyter_code>X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=.2, random_state=42)<jupyter_output><empty_output><jupyter_text>We'll verify that the shapes make sense. Note how many items are in each of the sets.<jupyter_code>X_train.shape, y_train.shape
X_valid.shape, y_valid.shape<jupyter_output><empty_output><jupyter_text>Here's a function to plot our regression model in "data space" (i.e., what it would predict everywhere on the map).This function is pretty customized to our specific use case, though you can get inspiration from it for use in other situations.<jupyter_code>def plot_model(clf, fig=None):
    # Compute extents
    lat_min = ames.Latitude.min()
    lat_max = ames.Latitude.max()
    lon_min = ames.Longitude.min()
    lon_max = ames.Longitude.max()
    price_min = ames.price.min()
    price_max = ames.price.max()

    # Ask the classifier for predictions on a grid
    xx, yy = np.meshgrid(np.linspace(lon_min, lon_max, 250), np.linspace(lat_min, lat_max, 250))
    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)

    # Show the predictions. Superimpose the original data.
    if fig is None:
        plt.figure(figsize=(16, 8))
    plt.contourf(xx, yy, Z, alpha=.5, cmap=plt.cm.viridis, vmin=price_min, vmax=price_max)
    plt.scatter(ames['Longitude'], ames['Latitude'], c=ames["price"], s=1, cmap='viridis', vmin=price_min, vmax=price_max)
    plt.xlabel("Longitude"); plt.ylabel("Latitude")
    plt.colorbar(label="Sale Price ($100k)")<jupyter_output><empty_output><jupyter_text>Task  Part A: Linear regression Step A1: **Fit a linear regression model (call it `linreg`) to the training set (`X_train`, `y_train`).**<jupyter_code># linreg = LinearRegression().fit(...)
linreg = LinearRegression().fit(X_train, y_train)
print("Prediction equation:")
print('y_pred = '
    + ' + ' .join(f'{coef:.3f} * {name}' for coef, name in zip(linreg.coef_, ['Latitude', 'Longitude']))
    + f' + {linreg.intercept_:.3f}')<jupyter_output>Prediction equation:
y_pred = -7.955 * Latitude + 11.886 * Longitude + -1242.742<jupyter_text>Step A2: **Plot the model's predictions in data space.** The code for step is filled in for you because there's not a generic way to do this; our approach here is customized to our particular model and task so you don't have to understand the details of how it works.The main thing to observe here is **what shapes do you see**. Think about why you might see those shapes in light of the prediction equation.<jupyter_code>plot_model(linreg)<jupyter_output><empty_output><jupyter_text>Step A3: **Compute the model's predictions on the validation set (call them `y_pred`). What does the model predict for the first house in the validation set? How does that compare with the actual price for that home?**<jupyter_code># y_pred = linreg.predict(...)
y_pred = linreg.predict(X_valid)
# your code here
print(f"Prediction: {y_pred[0]:.2f}, Actual: {y_valid[0]:.2f}, difference: {y_valid[0] - y_pred[0]:.2f}")<jupyter_output>Prediction: 1.55, Actual: 1.61, difference: 0.06<jupyter_text>Step A4: **Compute and show the mean squared error and the mean absolute error for the validation set.**- You may use the `mean_absolute_error` and `mean_squared_error` functions (imported from `sklearn.metrics` above).- Use the predictions you already made above.- Use Shift-TAB or `?` to get the documentation for these functions to ensure you're passing the arguments in the correct order.<jupyter_code># your code here
print("Mean absolute error: {:.3f}".format(mean_absolute_error(y_valid, y_pred)))
print("Mean squared error:  {:.3f}".format(mean_squared_error(y_valid, y_pred)))<jupyter_output>Mean absolute error: 0.582
Mean squared error:  0.679<jupyter_text>Part B: Decision tree regression Step B1: **Fit a decision tree model (call it `dtree_reg`) to the training set (`X_train`, `y_train`).**We'll use a small `max_depth` to be able to plot the tree. We'll then fit another one with full depth.Notice how the tree makes its prediction starting at the top (root) and checking one feature at a time. If the check is `True`, it goes left; otherwise, it goes right. When it hits a node with no check (a "leaf"), it predicts the value stored there. (Think: how do you think it might have computed that value?)<jupyter_code># dtree_reg_small = DecisionTreeRegressor(max_depth=3)...
dtree_reg_small = DecisionTreeRegressor(max_depth=3).fit(X_train, y_train)
plt.figure(figsize=(20, 15))
sklearn.tree.plot_tree(dtree_reg_small, feature_names=["Latitude", "Longitude"], filled=True);<jupyter_output><empty_output><jupyter_text>Now let's let the tree grow as big as it wants.<jupyter_code># dtree_reg = DecisionTreeRegressor()...
dtree_reg = DecisionTreeRegressor().fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>If the tree is big, the graphic may get unreadable. A text export may be easier to read:<jupyter_code>print(sklearn.tree.export_text(dtree_reg, feature_names=["Latitude", "Longitude"], max_depth=2))<jupyter_output>|--- Longitude <= 42.05
|   |--- Latitude <= -93.63
|   |   |--- Longitude <= 42.00
|   |   |   |--- truncated branch of depth 9
|   |   |--- Longitude >  42.00
|   |   |   |--- truncated branch of depth 22
|   |--- Latitude >  -93.63
|   |   |--- Longitude <= 42.04
|   |   |   |--- truncated branch of depth 28
|   |   |--- Longitude >  42.04
|   |   |   |--- truncated branch of depth 20
|--- Longitude >  42.05
|   |--- Latitude <= -93.65
|   |   |--- Latitude <= -93.66
|   |   |   |--- truncated branch of depth 12
|   |   |--- Latitude >  -93.66
|   |   |   |--- truncated branch of depth 24
|   |--- Latitude >  -93.65
|   |   |--- Latitude <= -93.63
|   |   |   |--- truncated branch of depth 22
|   |   |--- Latitude >  -93.63
|   |   |   |--- truncated branch of depth 16<jupyter_text>Step B2: **Plot the decision tree model in data space**. Observe what shapes of you see.<jupyter_code>plot_model(dtree_reg)
# your code here
y_pred = dtree_reg.predict(X_valid)
print("Mean absolute error:", mean_absolute_error(y_valid, y_pred))
print("Mean squared error:", mean_squared_error(y_valid, y_pred))<jupyter_output>Mean absolute error: 0.38370621160409557
Mean squared error: 0.3993000024034508<jupyter_text>Part C: Random Forest regressionRandom Forests take random subsets of the data and fit decision trees to each one. As each tree is fit, it also considers only a random subset of features for each decision. The combination of these two reduces the *variance* of the model, that is, how much the model's predictions change if it's fit on different subsets of data. **Fit a random forest regression model to this data.** Use the default hyperparameters.<jupyter_code># rf_reg = ...
rf_reg = RandomForestRegressor().fit(X_train, y_train)
print(f"We just fit a random forest with {rf_reg.n_estimators} trees.")
plot_model(rf_reg)<jupyter_output><empty_output><jupyter_text>Note: you can use code like this to show all of the different trees in the forest. It may or may not work in your computer, though.<jupyter_code>if False:
    from matplotlib.animation import FuncAnimation
    from IPython.display import HTML
    def frame(i):
        plt.clf()
        plot_model(rf_reg.estimators_[i], fig=fig)
        plt.title(f"Tree {i:03d}")
    fig = plt.figure(figsize=(16, 10))
    anim = FuncAnimation(fig=fig, func=frame, frames=len(rf_reg.estimators_))
    # One of these two should work:
    display(HTML(anim.to_html5_video()))
    #display(HTML(anim.to_jshtml()))<jupyter_output><empty_output><jupyter_text>Again, compute the predictions and errors.<jupyter_code># your code here
y_pred = rf_reg.predict(X_valid)
print("Mean absolute error:", mean_absolute_error(y_valid, y_pred))
print("Mean squared error:", mean_squared_error(y_valid, y_pred))<jupyter_output>Mean absolute error: 0.28854747700243444
Mean squared error: 0.20237182807213674
{'fraction_non_alphanumeric': 0.10658691265494379, 'fraction_numerical': 0.025223407321994813, 'mean_word_length': 4.446232339089482, 'pattern_counts': {'":': 0, '<': 94, '<?xml version=': 0, '>': 94, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '275904', 'license': 'MIT', 'n_tokens_mistral': 4681, 'n_tokens_neox': 4313, 'n_words': 1482, 'path': 'static/fundamentals/u7n1-sklearn-regression_soln.ipynb', 'repo_name': 'VelSharon/cs344', 'size': '13876'}