<jupyter_start><jupyter_text>Copyright 2019 Google LLC.<jupyter_code># Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.<jupyter_output><empty_output><jupyter_text>Introduction to scikit-learn [Scikit-learn](https://scikit-learn.org) is a machine learning library in Python.Scikit-learn is the first of the several machine learning libraries we will explore in this course. It is relatively approachable, supports a wide variety of traditional machine learning models, and is ubiquitous in the world of data science.  Datasets Scikit-learn contains methods for loading, fetching, and making (generating) data. The methods for doing this all fall under the [datasets](https://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.datasets) subpackage. Most of the functions in this package have `load`, `fetch`, or `make` in the name to let you know what the method is doing under the hood.**Loading functions** bring static datasets into your program. The data comes pre-packaged with scikit-learn, so no network access is required.**Fetching functions** also bring static datasets into your program. However, the data is pulled from the internet, so if you don't have network access, these functions might fail.**Generating functions** make dynamic datasets based on some equation.These pre-packaged dataset functions exist for many popular datasets, such as the [MNIST digits dataset](https://en.wikipedia.org/wiki/MNIST_database) and the [Iris flower dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set). The generation functions reference classic dataset "shape" formations such as [moons](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.htmlsklearn.datasets.make_moons) and [swiss rolls](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_swiss_roll.htmlsklearn.datasets.make_swiss_roll). These datasets are perfect for getting familiar with machine learning.  LoadingLet us first look at an example of loading data. We will load the iris flowers dataset using the [load_iris](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.htmlsklearn.datasets.load_iris) function.<jupyter_code>from sklearn.datasets import load_iris

iris_data = load_iris()
iris_data<jupyter_output><empty_output><jupyter_text>That's a lot to take in. Let's examine this loaded data a little more closely. First we'll see what data type this dataset is:<jupyter_code>type(iris_data)<jupyter_output><empty_output><jupyter_text>`sklearn.utils.Bunch` is a type that you'll see quite often when working with datasets built into scikit-learn. It is a dictionary-like container for feature and target data within a dataset.You won't find much documentation about `Bunch` objects because they are not really meant for usage beyond containing data native to scikit-learn.Let's look at the attributes of the iris dataset:<jupyter_code>dir(iris_data)<jupyter_output><empty_output><jupyter_text>`DESCR` is a description of the dataset.<jupyter_code>print(iris_data['DESCR'])<jupyter_output><empty_output><jupyter_text>`filename` is the name of the source file where the data is stored.<jupyter_code>print(iris_data['filename'])<jupyter_output><empty_output><jupyter_text>`feature_names` is the name of the feature columns.<jupyter_code>print(iris_data['feature_names'])<jupyter_output><empty_output><jupyter_text>`target_names`, despite the name, is not the names of the target columns. There is only one column of targets.Instead, `target_names` is the human-readable names of the classes in the target list within the bunch. In this case,`target_names` is the names of the three species of iris in this dataset.<jupyter_code>print(iris_data['target_names'])<jupyter_output><empty_output><jupyter_text>We can now examine `target` and see that it contains zeros, ones, and twos. These correspond to the target names 'setosa', 'versicolor', and 'virginica'.<jupyter_code>print(iris_data['target'])<jupyter_output><empty_output><jupyter_text>Last, we'll look at the `data` within the bunch. The data is an array of arrays. Each sub-array contains four values. These values match up with the `feature_names`. The first item in each sub-array is 'sepal length (cm)', the next is 'sepal width (cm)', and so on.<jupyter_code>iris_data['data']<jupyter_output><empty_output><jupyter_text>The number of target values should always equal the number of rows in the data.<jupyter_code>print(len(iris_data['data']))
print(len(iris_data['target']))<jupyter_output><empty_output><jupyter_text>`Bunch` objects are an adequate container for data. They can be used directly to feed models. However, `Bunch` objects are *not* very good for analyzing and manipulating your data.In this course, we will typically convert `Bunch` objects into Pandas `DataFrame` objects to make analysis, data cleaning, visualization, and train/test splitting easier.To do this, we will take the matrix of feature data and append the target data to it to create a single matrix of data. We also take the list of feature names and append the word 'species' to represent the target classes in the matrix.<jupyter_code>import pandas as pd
import numpy as np

iris_df = pd.DataFrame(
  data=np.append(
    iris_data['data'], 
    np.array(iris_data['target']).reshape(len(iris_data['target']), 1), 
    axis=1),
  columns=np.append(iris_data['feature_names'], ['species'])
)

iris_df.sample(n=10)<jupyter_output><empty_output><jupyter_text>You might notice that the integer representation of species got converted to a floating point number along the way. We can change that back.<jupyter_code>iris_df['species'] = iris_df['species'].astype('int64')

iris_df.sample(n=10)<jupyter_output><empty_output><jupyter_text>Exercise 1 Load the [Boston house price dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html) into a Pandas `DataFrame`. Append the target values to the last column of the `DataFrame` called `boston_df`. Name the target column 'PRICE'.  **Student Solution**<jupyter_code># Your answer goes here<jupyter_output><empty_output><jupyter_text>---  FetchingFetching is similar to loading. Scikit-learn will first see if it can find the dataset locally, and, if so, it will simply load the data. Otherwise, it will attempt to pull the data from the internet.We can see fetching in action with the [fetch_california_housing](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_california_housing.htmlsklearn.datasets.fetch_california_housing) function below.<jupyter_code>from sklearn.datasets import fetch_california_housing

housing_data = fetch_california_housing()

type(housing_data)<jupyter_output><empty_output><jupyter_text>The dataset is once again a `Bunch`.If you follow the link to the [fetch_california_housing](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_california_housing.htmlsklearn.datasets.fetch_california_housing) documentation, you notice that the dataset is a **regression** dataset as opposed to the iris dataset, which was a **classification** dataset.We can see the difference in the dataset by checking out the attributes of the `Bunch`.<jupyter_code>dir(housing_data)<jupyter_output><empty_output><jupyter_text>We see that four of the attributes that we expect are present, but 'target_names' is missing. This is because our target is now a continuous variable (home price) and not a discrete value (iris species).<jupyter_code>print(housing_data['target'][:10])<jupyter_output><empty_output><jupyter_text>Converting a `Bunch` of regression data to a `DataFrame` is no different than for a `Bunch` of classification data.<jupyter_code>import pandas as pd
import numpy as np

housing_df = pd.DataFrame(
  data=np.append(
    housing_data['data'], 
    np.array(housing_data['target']).reshape(len(housing_data['target']), 1), 
    axis=1),
  columns=np.append(housing_data['feature_names'], ['price'])
)

housing_df.sample(n=10)<jupyter_output><empty_output><jupyter_text>GeneratingIn the example datasets we've seen so far in this Colab, the data is static and loaded from a file. Sometimes it makes more sense to generate a dataset. For this, we can use one of the many [generator functions](https://scikit-learn.org/stable/modules/classes.htmlsamples-generator). `make_regression` is a generator that creates a dataset with an underlying regression that you can then attempt to discover using various machine learning models.In the example below, we create a dataset with 10 data points. For the sake of visualization, we have only one feature per datapoint, but we could ask for more.The return values are the $X$ and $y$ values for the regression. $X$ is a matrix of features. $y$ is a list of targets.Since a generator uses randomness to generate data, we are going to set a `random_state` in this Colab for reproducibility. This ensures we get the same result every time we run the code. **You won't do this in your production code.**<jupyter_code>from sklearn.datasets import make_regression

features, targets = make_regression(n_samples=10, n_features=1, random_state=42)

features, targets<jupyter_output><empty_output><jupyter_text>We can use a visualization library to plot the regression data.<jupyter_code>import matplotlib.pyplot as plt

plt.plot(features, targets, 'b.')
plt.show()<jupyter_output><empty_output><jupyter_text>That data appears to have a very linear pattern!If we want to make it more realistic (non-linear), we can add some noise during data generation.**Remember that random_state is for reproducibility only. Don't use this in your code unless you have a good reason to.**<jupyter_code>from sklearn.datasets import make_regression

features, targets = make_regression(n_samples=10, n_features=1, random_state=42, noise=5.0)

plt.plot(features, targets, 'b.')
plt.show()<jupyter_output><empty_output><jupyter_text>There are dozens of dataset loaders and generators in the scikit-learn [datasets](https://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.datasets) package. When you want to play with a new machine learning algorithm, they are a great source of data for getting started.  Exercise 2 Search the [scikit-learn datasets documentation](https://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.datasets) and find a function to make a "Moons" dataset. Create a dataset with 75 samples. Use a random state of 42 and a noise of 0.08. Store the $X$ return value in a variable called `features` and the $y$ return value in a variable called `targets`.  **Student Solution**<jupyter_code># Your answer goes here<jupyter_output><empty_output><jupyter_text>---  Exercise 3 In Exercise Two, you created a "moons" dataset. In that dataset, the features are $(x,y)$-coordinates that can be graphed in a scatterplot. The targets are zeros and ones that represent a binary classification.Use matplotlib's [scatter method](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.scatter.html) to visualize the data as a scatterplot. Use the `c` argument to make the dots for each class a different color.  **Student Solution**<jupyter_code># Your answer goes here<jupyter_output><empty_output><jupyter_text>---  Models Machine learning involves training a model to gain insight and predictive power from a dataset. Scikit-learn has support for many different types of models, ranging from classic algebraic models to more modern deep learning models.Throughout the remainder of this course, you will learn about many of these models in much more depth. This section will walk you through some of the overarching concepts across all models.  EstimatorsMost of the models in scikit-learn are considered [estimators](https://scikit-learn.org/stable/modules/generated/sklearn.base.BaseEstimator.htmlsklearn.base.BaseEstimator). An estimator is expected to implement two methods: `fit` and `predict`.`fit` is used to train the model. At a minimum, it is passed the feature data used to train the model. In supervised models, it is also passed the target data.`predict` is used to get predictions from the model. This method is passed features and returns target predictions. Let's see an example of this in action.Linear regression is a simple model that you might have encountered in a statistics class in the past. The model attempts to draw a straight line through a set of data points, so the line is as close to as many points as possible.We'll use scikit-learn's [LinearRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) class to fit a line to the regression data that we generated earlier in this Colab. To do that, we simply call the `fit(features, targets)` method.After fitting, we can ask the model for predictions. To do this, we use the `predict(features)` method.<jupyter_code>from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression

regression = LinearRegression()
regression.fit(features, targets)
predictions = regression.predict(features)

plt.plot(features, targets, 'b.')
plt.plot(features, predictions, 'r-')
plt.show()<jupyter_output><empty_output><jupyter_text>At this point, don't worry too much about the details of what `LinearRegression` is doing. There is a deep-dive into regression problems coming up soon.For now, just note the `fit`/`predict` pattern for training estimators, and know that you'll see it throughout our adventures with scikit-learn.  TransformersIn practice, it is rare that you will get perfectly clean data that is ready to feed into your model for training. Most of the time, you will need to perform some type of cleaning on the data first.You've had some hands-on experience doing this in our Pandas Colabs. Scikit-learn can also be used to perform some data preprocessing.Transformers are spread about within the scikit-learn library. Some are in the [preprocessing](https://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.preprocessing) module while others are in more specialized packages like [compose](https://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.compose), [feature_extraction](https://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.feature_extraction), [impute](https://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.impute), and others.All transformers implement the `fit` and `transform` methods. The `fit` method calculates parameters necessary to perform the data transformation. The `transform` method actually applies the transformation. There is a convenience `fit_transform` method that performs both fitting and transformation in one method call. Let's see a transformer in action.We will use the [MinMaxScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.htmlsklearn.preprocessing.MinMaxScaler) to scale our feature data between zero and one. This scales the data with a linear transform so that the minimum value becomes 0 and the maximum value becomes 1, so all values are within 0 and 1.Looking at our feature data pre-transformation, we can see values that are below zero and above one.<jupyter_code>features<jupyter_output><empty_output><jupyter_text>We will now create a `MinMaxScaler` and fit it to our feature data.Each transformer has different information that it needs in order to perform a transformation. In the case of the `MinMaxScaler`, the smallest and largest values in the data are needed.<jupyter_code>from sklearn.preprocessing import MinMaxScaler

transformer = MinMaxScaler()
transformer.fit(features)
transformer.data_min_, transformer.data_max_<jupyter_output><empty_output><jupyter_text>You might notice that the values are stored in arrays. This is because transformers can operate on more than one feature. In this case, however, we have only one. Next, we need to apply the transformation to our features. After the transformation, we can now see that all of the features fall between the range of zero to one. Moreover, you might notice that the minimum and maximum value in the untransformed `features` array correspond to the 0 and 1 in the transformed array, respectively.<jupyter_code>features = transformer.transform(features)
features<jupyter_output><empty_output><jupyter_text>PipelinesA [pipeline](https://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.pipeline) is simply a series of transformers, often with an estimator at the end. In the example below, we use a [Pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) class to perform min-max scaling or our feature data and then train a linear regression model using the scaled features.<jupyter_code>from sklearn.pipeline import Pipeline

features, targets = make_regression(
    n_samples=10, n_features=1, random_state=42, noise=5.0)

pipeline = Pipeline([
  ('scale', MinMaxScaler()),
  ('regression', LinearRegression())
])

pipeline.fit(features, targets)

predictions = pipeline.predict(features)

plt.plot(features, targets, 'b.')
plt.plot(features, predictions, 'r-')
plt.show()<jupyter_output><empty_output><jupyter_text>MetricsSo far we have seen ways that scikit-learn can help you get data, modify that data, train a model, and finally, make predictions. But how do we know how good these predictions are?Scikit-learn also comes with many functions for measuring model performance in the [metrics](https://scikit-learn.org/stable/modules/classes.htmlsklearn-metrics-metrics) package. Later in this course, you will learn about different ways to measure the performance of regression and classification models, as well as tradeoffs between the different metrics. We can use the [mean_squared_error](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html) function to find the mean squared error (MSE) between the target values that we used to train our linear regression model and the predicted values.<jupyter_code>from sklearn.metrics import mean_squared_error

mean_squared_error(targets, predictions)<jupyter_output><empty_output><jupyter_text>In this case, the MSE value alone doesn't have much meaning. Since the data that we fit the regression to isn't related to any real-world metrics, the MSE is hard to interpret alone.As we learn more about machine learning and begin training models on real data, you'll learn how to interpret MSE and other metrics in the context of the data being analyzed and the problem being solved. There are also metrics that come with each estimator class. These metrics can be extracted using the `score` method.The `regression` class we created earlier can be scored, as can the `pipeline`.<jupyter_code>print(regression.score(features, targets))
print(pipeline.score(features, targets))<jupyter_output><empty_output><jupyter_text>The return value of the `score` method depends on the estimator being used. In the case of `LinearRegression`, the score is the $R^2$ score, where scores closer to 1.0 are better. You can find the metric that `score` returns in the [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.htmlsklearn.linear_model.LinearRegression.score) for the given estimator you're using.  Exercise 4 Use the [Pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) class to combine a data pre-processor and an estimator.To accomplish this:1. Find a [preprocessor](https://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.preprocessing) that uses the max absolute value for scaling.1. Find a [linear_model](https://scikit-learn.org/stable/modules/classes.htmlmodule-sklearn.linear_model) based on the Huber algorithm.1. Combine this preprocessor and estimator into a pipeline.1. Make a sample regression dataset with 200 samples and 1 feature. Use a random state of 85 and a noise of 5.0. Save the features in a variable called `features` and the targets in a variable called `targets`.1. Fit the model.1. Using the features that were created when the regression dataset was created, make predictions with the model and save them into a variable called `predictions`.1. Plot the features and targets used to train the model on a scatterplot with blue dots.1. Plot the features and predictions over the scatterplot as a red line.  **Student Solution**<jupyter_code># Your answer goes here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07880877142583549, 'fraction_numerical': 0.0037824379967442306, 'mean_word_length': 5.854939284542173, 'pattern_counts': {'":': 0, '<': 125, '<?xml version=': 0, '>': 125, 'https://': 30, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '31', 'dir': 'jupyter-structured-clean-dedup', 'id': '90307', 'license': 'Apache-2.0', 'n_tokens_mistral': 5856, 'n_tokens_neox': 5676, 'n_words': 2362, 'path': 'content/03_regression/01_introduction_to_sklearn/colab.ipynb', 'repo_name': 'skykykykykykykykykykykys/applied-machine-learning-intensive', 'size': '20886'}