<jupyter_start><jupyter_text>CS4618: Artificial Intelligence IRegression    <NAME>    School of Computer Science and Information Technology    University College Cork Initialization$\newcommand{\Set}[1]{\{1\}}$ $\newcommand{\Tuple}[1]{\langle1\rangle}$ $\newcommand{\v}[1]{\pmb{1}}$ $\newcommand{\cv}[1]{\begin{bmatrix}1\end{bmatrix}}$ $\newcommand{\rv}[1]{[1]}$ $\DeclareMathOperator{\argmax}{arg\,max}$ $\DeclareMathOperator{\argmin}{arg\,min}$ $\DeclareMathOperator{\dist}{dist}$$\DeclareMathOperator{\abs}{abs}$<jupyter_code>%reload_ext autoreload
%autoreload 2
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.neighbors import KNeighborsRegressor<jupyter_output><empty_output><jupyter_text>Regression    We want to create programs that make predictions.    Specifically in this part of the module we will consider predicting numeric values.    Prediction of numeric values is called regression:                    Given a vector $\v{x}$ of feature values that describes some object, we predict a target value, which is numeric (real-valued).                        E.g.given a vector of feature values that describe a house, predict the selling price of the house.                (In everyday use, prediction is about the future; but we use the word more generally in AI.) Notation    We continue to use $\v{x}$ for an object.    We will use $y$ for the target value.        Actually, we will be even more precise:                    We will use $y$ for the actual target value.            We will use $\hat{y}$ for a predicted target value.             Datasets of labeled examples    We don't want to write a program to make these predictions.    We want to learn how to make them.    To learn how to make predictions, we need a dataset.                    As before: $m$ examples, $n$ features                But we need a different kind of dataset: we need a dataset of labeled examples.        A labeled example is a pair, comprising a vector of feature values and the target value        $$\Tuple{\v{x}, y}$$        So a labeled dataset looks like this:         $$\Set{\Tuple{\v{x}^{(1)}, y^{(1)}}, \Tuple{\v{x}^{(2)}, y^{(2)}}, \ldots,         \Tuple{\v{x}^{(m)}, y^{(m)}}}$$                     E.g. the features are floor area, bedrooms and bathrooms; the target is the selling price                (thousands of &euro;)                 $$\Set{\Tuple{\cv{92.9\\3\\2}, 175},\,\, \Tuple{\cv{171.9\\4\\3}, 435},\,\, \Tuple{\cv{79\\3\\1}, 85}\cdots}$$                               From a labeled dataset, we can construct a matrix $\v{X}$ and a vector $\v{y}$ as follows:        $$\v{X} = \begin{bmatrix}              \v{x}_1^{(1)} & \v{x}_2^{(1)} & \ldots & \v{x}_n^{(1)} \\              \v{x}_1^{(2)} & \v{x}_2^{(2)} & \ldots & \v{x}_n^{(2)} \\              \vdots        & \vdots        & \vdots & \vdots \\              \v{x}_1^{(m)} & \v{x}_2^{(m)} & \ldots & \v{x}_n^{(m)} \\              \end{bmatrix}\,\,\,\,\,          \v{y} = \cv{y^{(1)}\\ y^{(2)}\\ \vdots\\ y^{(m)}}        $$                    In the matrix $\v{X}$, rows are examples, columns are features.            The vector $\v{y}$ gives the corresponding target values.            E.g                $$\v{X} =                     \begin{bmatrix}                        92.9 & 3 & 2 \\                        171.9 & 4 & 3 \\                        79 & 3 & 1                    \end{bmatrix}\,\,\,\,\,                  \v{y} = \cv{175\\ 435\\ 85}                $$                             Let's read our Cork property prices dataset inot a pandas DataFrame.    Then, we'll extract $\v{X}$ and $\v{y}$ and convert them to numpy arrays.<jupyter_code># Use pandas to read the CSV file into a DataFrame
df = pd.read_csv("../datasets/dataset_corkA.csv")

# The features we want to select
features = ["flarea", "bdrms", "bthrms"]

# Extract these features and convert to numy 2D array
X = df[features].values

# The target values, converted to a numpy 1D array
y = df["price"].values<jupyter_output><empty_output><jupyter_text>Supervised Learning    Learning from a labeled dataset is called supervised learning.    The goal is to generalize from the labeled dataset so that we can predict target values        when given the feature values of objects that were not in the labeled dataset (unseen examples).        One way to categorize the algorithms for doing this is by how they generalize:                    instance-based learning; and            model-based learning.             Instance-Based Learning    Instance-based learners learn by heart: they simply store the examples in the labeled dataset.    The way they generalize is using similarity (or distance): given an unseen example $\v{x}$,         they predict $\hat{y}$ from the $y$-values of examples in the dataset that are similar to $\v{x}$.        Let's look at two concrete examples of this: nearest-neighbour regression and         $k$-nearest-neighbours regression.     Nearest-neighbour regression    To predict the target value $\hat{y}$ for unseen example $\v{x}$,                    we find the example $\Tuple{\v{x}', y'}$ in the labeled dataset whose distance from $\v{x}$ is                 smallest; and                          we use $y'$ as our prediction.                              We also refer to this as a 1-nearest-neighbour regressor or        just 1NN or kNN for $k=1$.        We can easily implement this using code from the previous lecture.<jupyter_code>def euc(x, xprime):
    return np.sqrt(np.sum((x - xprime)**2))

def nn_regressor(x, X, y):
    # Find which example in X is the most similar to x
    index = np.argmin([euc(x, xprime) for xprime in X])
    # Return the corresponding target value from y
    return y[index]
# Standardization of X
scaler = StandardScaler()
scaler.fit(X)
X_scaled = scaler.transform(X)

# Standardization of our houses
our_houses = np.array([[126.0, 3, 1], [107.0, 2, 1]])
our_houses_scaled = scaler.transform(our_houses)


# Prediction
[nn_regressor(x_scaled, X_scaled, y) for x_scaled in our_houses_scaled]<jupyter_output><empty_output><jupyter_text>k-nearest-neighbours regression    The problems with 1NN is that it can be incorrectly influenced by noisy examples:                    If there are examples in the labeled dataset where we have incorrectly recorded the feature values,                then we may not find the best example from which to make our prediction.                        If there are examples in the labeled dataset where we have incorrectly recorded the target value,                then these will result in incorrect predictions.                            To reduce the influence of noisy examples, we use more than one neighbour:                    we find $k$ examples whose distance from unseen example $\v{x}$ is smallest; and            we use the mean of their $y$-values as our prediction.                            We abbreviate the name of this to kNN, e.g. 3NN is where we use 3 nearest-neighbours.    It too is easily implemented.<jupyter_code>def knn_regressor(x, X, y, k=3):
    # Find which k examples in X are the most similar to x
    indexes = np.argsort([euc(x, xprime) for xprime in X])[:k]
    # Return the corresponding target value from y
    return y[indexes].mean()
# Prediction (with k=3, the default)
[knn_regressor(x_scaled, X_scaled, y) for x_scaled in our_houses_scaled]<jupyter_output><empty_output><jupyter_text>There many variants of kNN.                    A common one, for example, is to use a weighted average of the neighbour's target values.            The weights could be the inverse of the distances so that more similar examples count for more.                We don't need to implement them for ourselves: scikit-learn has a class KNeighborsRegressor.     Model-Based Learning    To get an intuition for what we mean by the word model, we'll imagine the following.    We want to build a system that can predict the selling prices of houses in Cork.    We don't know how to do it, so we ask Ann!                    She's an auctioneer &mdash; an expert at predicting Cork city house prices.            But we don't ask her to predict your house price or mine.            We ask her for a general method for predicting Cork house prices.                She tells us that her rule-of-thumb is that prices start at 25k&euro; and increase by 1.5k&euro; for every         extra square metre of floor area:        $$\hat{y} = 25 + 1.5\mathit{flarea}$$                    So, she predicts your house (floor area of 126 square metres) will sell for                 $\hat{y} = 25 + 1.5 \times 126 = 214$k&euro;                            Ann has given us a model.                                    In very abstract terms, a model is an approximation of some part of reality that enables us to make                 predictions about that reality.                                        In very concrete terms for this module, a model is a formula (or function or procedure or set of                 rules&hellip;) that expresses the relationship between an object's features and the thing that is                being predicted (in regression, the target value).                            (It so happens that Ann's is a linear model &mdash; see next lecture.) Implementing and visualizing Ann's model<jupyter_code># Ann's model
def h_ann(flarea):
    return 25 + 1.5 * flarea
# Predicting the selling price of your house
h_ann(126)
# Plotting the predictions made by Ann's model
fig = plt.figure()
plt.title("Ann's model")
xvals = np.linspace(0, 500, 2)
plt.plot(xvals, h_ann(xvals))
plt.xlabel("Floor area (sq metres)")
plt.xlim(0, 500)
plt.ylabel("Price (000 euros)")
plt.ylim(0, 1000)
plt.show()<jupyter_output><empty_output><jupyter_text>Pre-Covid, I met this guy called Dev in a pub. He isn't a Cork auctioneer but he's really opinionated.    He gave me a different model:        $$\hat{y} = 20 + 50\mathit{bdrms} + 10\mathit{bdrms}^2$$        (Dev's model is not a linear model &mdash; it is a non-linear model. In fact, it's a quadratic model.) Implementing and visualizing Dev's model<jupyter_code># Dev's model
def h_dev(bdrms):
    return 20 + 50 * bdrms + 10 * bdrms ** 2
# Predicting the selling price of your house
h_dev(3)
# Plotting the predictions made by Dev's model
fig = plt.figure()
plt.title("Dev's model")
xvals = np.linspace(0, 10, 10)
plt.plot(xvals, h_dev(xvals))
plt.xlabel("Bdrms")
plt.xlim(0, 10)
plt.ylabel("Price (000 euros)")
plt.ylim(0, 1500)
plt.show()<jupyter_output><empty_output><jupyter_text>Building a model: learn from data    Rather than ask an expert, we want to learn models like Ann's and Dev's from our labeled dataset.            We would take some or all of the examples, call them the training set, and give them to a         learning algorithm.        As best it can, the learning algorithm would find a model based on the labeled examples in the training set.     Learning from data     Terminology            We will say that the algorithm learns a model.                We could also say that we are training the algorithm on the data.            We could also say that the algorithm fits a model to the training set.                We could also call it function approximation.     Model-based learning in scikit-learn    So let's use scikit-learn to learn a linear model from our labeled dataset.    The class for this is called LinearRegression.<jupyter_code># Fit the model

linear_model = LinearRegression()

linear_model.fit(X, y)
# Make a prediction
linear_model.predict(our_houses)<jupyter_output><empty_output><jupyter_text>As you case see, for the LinearRegression class, we don't need to scale the data (although         it wouldn't do any harm if we did).        You might be wondering what linear model it actually learned. Is it the same as Ann's?<jupyter_code>linear_model.intercept_, linear_model.coef_<jupyter_output><empty_output><jupyter_text>So this is whatb it learned:        $$\hat{y} = 38 + 2.5 \times \mathit{flarea} + -0.3 \times \mathit{bdrms} + 0.8 \times \mathit{bthrms}$$        Very different from Ann's!     scikit-learn has a nice API:                    All classes for transforming data, e.g. scaling it, have methods called fit                and transform:                                    fit computes anything that is needed to carry out the                        transformation. For example, in the case of StandardScaler, it calculates                        the mean and standard deviation.                                        transform performs the actual transformation. In the case of                        StandardScaler, it subtracts the mean and divides by the standard deviation.                                                            All classes for making predictions have methods called fit and predict:                                    fit builds the model.                                                     For example, for KNeighborsRegressor, it chooses a data structure                                for storing the labeld dataset.                                                        For linearRegression, it learns the linear model from the labeled                                dataset.                                                                                            predict uses the model to make predictions.                                                    For example, for KNeighborsRegressor, it finds neighbours and                                averages their $y$-values.                                                        For LinearRegression, it plugs values into the formula learned                                by the fit method.                                                                                                                    We can see how neat it is by showing the code for the KNeighborsRegressor:<jupyter_code># Fit the model

knn = KNeighborsRegressor(n_neighbors=3)

knn.fit(X_scaled, y)
# Make a prediction
knn.predict(our_houses_scaled)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08698717044973918, 'fraction_numerical': 0.01318201043282109, 'mean_word_length': 2.2561395455588706, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 36, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '562282', 'license': 'MIT', 'n_tokens_mistral': 3914, 'n_tokens_neox': 3591, 'n_words': 1617, 'path': 'ai1/lectures/AI1_04_regression.ipynb', 'repo_name': 'derekgbridge/artificial_intelligence', 'size': '14192'}