<jupyter_start><jupyter_text>Neural network methodsAuthor: <NAME>  Learning objectives* Understand what an artificial neural network (ANN) is and how it can be used.* Implement ANNs for use in prediction and classification based on multiple input features. Learning deeply[Artificial Neural Networks (ANNs)](https://en.wikipedia.org/wiki/Artificial_neural_network) and [deep learning](https://en.wikipedia.org/wiki/Deep_learning) are currently getting a lot of interest, both as a subject of research and as a tool for analyzing datasets. A big difference from other machine learning techniques we've looked at so far is that ANNs can identify characteristics of interest by themselves, rather than having to be chosen by data scientists. Some of the other advantages of ANNs are related specifically to interpreting video and audio data, such as by using [convolutional neural networks](https://en.wikipedia.org/wiki/Convolutional_neural_network), but today we will focus on simple ANNs so you understand their struction and function. UnitsANNs are designed as *layers* of *units* (or nodes, or [artificial neurons](https://en.wikipedia.org/wiki/Artificial_neuron)). Each unit accepts multiple inputs, each of which has a different weight, including one bias input, which it combines into a single value. That single value is passed to an [activation function](https://en.wikipedia.org/wiki/Activation_function), which provides an output only if the combined value is greater than a particular threshold (usually, zero).In effect, each unit focuses on a particular aspect of the layer underneath it, and then summarizing that information for the layer above it. Every input to every unit has a *weight* and the unit has a *bias* input, and so is effectively doing a linear regression on the incoming data to obtain an output. The use of a non-linear activation function allows the ANN to predict and classify data that are not [linearly separable](https://en.wikipedia.org/wiki/Linear_separability). ANNs used to use the same sigmoid function we saw before, but these days the [rectified linear unit (ReLU) function](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) has become much more popular. It simply returns zero when the real output value is less than zero.![ReLU and softmax rectifiers](../nb-images/800px-Rectifier_and_softplus_functions.svg.png) Layers![A diagram showing input, output and hidden layers in an ANN](../nb-images/colored_neural_network.svg.png)Every neural network has three layers:* An input layer, where each unit corresponds to a particular input feature. This could be categorical data, continuous data, or even colour values from images.* An output layer. We will be running two examples today: in the first, we will use a single output unit (the predicted price for a particular house in California). In the second, we will use seven output units, each corresponding to a particular type of forest cover.* A hidden layer. Without a hidden layer, an ANN can only pick up linear relationships: how changes in the input layer correspond to values in the output layer. Thanks to the hidden layer, an ANN can also pick up non-linear relationships, where different groups of input values interact in complicated ways to get the correct response on the output layer. The "deep" in [deep learning](https://en.wikipedia.org/wiki/Deep_learning) refers to the hidden layers that allow the model to identify intermediate patterns between the input and output layers.Putting it all together, we end up with a type of ANN called a *multilayer perceptron (MLP) network*, which looks like the following:![A visualization of a multi-layer perceptron (MLP) from the Scikit-learn manual](../nb-images/multilayerperceptron_network.png)[Google's Machine Learning course](https://developers.google.com/machine-learning/crash-course/introduction-to-neural-networks/anatomy) puts this very nicely: "Stacking nonlinearities on nonlinearities lets us model very complicated relationships between the inputs and the predicted outputs. In brief, each layer is effectively learning a more complex, higher-level function over the raw inputs."  So what do we need?To create an ANN, we need to choose:1. The number of input units (= the number of input features)2. The number of output units:    - When using the ANN to predict, we generally only need a single output unit.    - When using the ANN to classify, we generally set the number of output units to the number of possible labels.3. The number of hidden layers    - More hidden layers allows for more complex models -- which, as you've learned, also increases your risk of overfitting! So you want to go for the simplest model that meets your needs.4. A loss function. The scikit-learn classes we use always use [logistic loss](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html), also known as cross-entropy loss.4. The solver to use. The solver controls learning by searching for local minima in the parameter space. ANNs generally use [stochastic gradient descent (SGD)](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) such as [RMSProp](https://en.wikipedia.org/wiki/Stochastic_gradient_descentRMSProp), but today we will use [Adaptive Moment Estimation (Adam)](https://en.wikipedia.org/wiki/Stochastic_gradient_descentAdam).5. The regularization protocol. We will use L2 regularization.Note that these are the [hyperparameters](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)) of our model: we will adjust these hyperparameters to improve how quickly and accurately we can determine the actual parameters of our model, which is the set of weights and biases on all units across all layers.  Reminders of the ground rules* Always have *training* data and *testing* data, and make sure the ANN *never* sees the testing data.* Always shuffle your data.* ANNs don't work well when the features are in different ranges: it's usually a good idea to *normalize* it before use.  ANN for prediction: how much might this house cost? There aren't very good datasets for showcasing prediction on biological data, so we will use one of the classic machine learning datasets: a dataset of [California house prices](https://www.kaggle.com/camnugent/california-housing-prices), based on the 1990 census and published in [Pace and Barry, 1997](https://doi.org/10.1016/S0167-7152(96)00140-X). Scikit-Learn can download this dataset for us, so let's start with that. Let's start by shuffling our data and splitting into testing data and training data. The input data comes in many different ranges: compare the ranges of latitude (32.54 to 41.95), longitude (-124.35 to -114.31), median income (0.50 to 15.0) and population (3 to 35682). As we described earlier, it's a good idea to normalize these values. Scikit-Learn has several built-in scalers that do just this. We will use the [StandardScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html), which changes the data so it is normally distributed, with a mean of 0 and a variance of 1, but other [standardization methods are available](https://scikit-learn.org/stable/modules/preprocessing.htmlstandardization-or-mean-removal-and-variance-scaling). We won't normalize the output labels for simplicity and because it's usually not necessary if you have a single output. However, if you have multiple output labels, you will want to normalize them to each other. Alright, we're ready to run our model! We will use the [MLPRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html) -- MLP stands for [multilevel perceptron](https://en.wikipedia.org/wiki/Multilayer_perceptron), a description of this kind of neural network. Three things to note: - Some models will never converge, in which case you will see a warning. This suggests that learning is *not* completely, and is likely because something is wrong with learning using this dataset with these hyperparameters. - We learn iteratively. In Scikit-Learn, each iteration is further broken up into "batches" of data. - We expect to see loss going down over time and validation score going up over time. We can visualize these in a graph if we want. Finally, we can use our test data to check how our model performs on data that it has not been previously exposed to. Let's see how we did! Not bad, but it could definitely be better. Let's make a predictionNote that we can use this ANN to make a prediction. To come up with one, let's look at those values again: So what if we knew that a house was in a block that: - had a median income of 30,000 USD - had a median house age of 12 years - had an average of 2 rooms - had an average of 1 bedroom - had a population of 2,000 in the block - had an average occupancy of 2 - was located at (33.93 N, -118.49 E) How much would our model predict it would cost?<jupyter_code>house_blocks = [[
    3.,
    12,
    2,
    1,
    2000,
    2.,
    33.93,
    -118.49
]]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06533996683250415, 'fraction_numerical': 0.012493090105030403, 'mean_word_length': 4.725316455696203, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 20, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '543440', 'license': 'MIT', 'n_tokens_mistral': 2363, 'n_tokens_neox': 2194, 'n_words': 1235, 'path': 'assets/nb-lessons/13_neural_network_methods_worksheet.ipynb', 'repo_name': 'USDA-ARS-GBRU/ml-training-site', 'size': '9052'}