<jupyter_start><jupyter_text>![](https://github.com/Ai-Adventures/rmd/blob/master/notebooks/images/logo_full.png?raw=1)The course is offered by [Ai Adventures](https://aiadventures.in/). The notebooks are created by [<NAME>](https://twitter.com/I_ankursingh). This material is subject to the terms and conditions of the [Creative Commons CC BY-NC-SA 4.0 license](https://creativecommons.org/licenses/by-nc-sa/4.0/). Any use for commercial purpose is strictly prohibited.   **Introduction to Neural Networks** **Approach**Machine learning is taught by academics, for academics. That’s why most material is so dry and math-heavy.Developers need to know what works and how to use it. We need less math and more tutorials with working code.___ **Anatomy of a Neural Network**Buildind and Training a Neural Network revolves around the following objects:- **Layers**, which are combined into a network (or model)- **Input** data and corresponding **targets**- **Loss function**, which defines the feedback signal used for learning- **Optimizer**, which determines how learning proceeds![](https://github.com/Ai-Adventures/rmd/blob/master/notebooks/images/model.png?raw=1)___ LayersThe fundamental data structure in neural networks is the **layer**. You can think of layers as the LEGO bricks of Deep learning, a metaphor that is made explicit by frameworks like keras. Some of the layers are- Dense or Fully connected layers (Dense).- Convolution layers (Conv2D, Conv1D).- Recurrent layers (RNN, LSTM, GRU).In Keras you don't have to care about **compatibility** between the layers because the layers you add to your models are dynamically built to match the shape of the incoming layer.<jupyter_code>from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

model = Sequential()
model.add(Dense(32, input_shape=(28*28,)))<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.<jupyter_text>Loss Functions and OptimizersOnce the network architecture is defined, you have to choose two major things:- **Loss function (objective function)** - The quantity that will be minmized during training. It represents a measure of successs for the task at hand.- **Optimizer** - Determines how the network will be updated based on the loss function. It implements a specific variant of stocastic gradient descent(SGD).Choosing the right objective function for the right problem is extremely important: your network will take any shortcut it can, to minimize the loss; so if the objective doesn’t fully correlate with success for the task at hand, your network will end up doing things you may not have wanted.Fortunately, when it comes to common problems such as classification, regression, and sequence prediction, there are simple guidelines you can follow to choose the correct loss. For instance, you’ll use binary_crossentropy for a two-class classification problem, categorical_crossentropy for a many-class classification problem, mean-squared error for a regression problem, connectionist temporal classification(CTC) for a sequence-learning problem, and so on.**Note:** Crossentropy is a quantity from the field of Information Theory that measures the distance between probability distributions or, in this case, between the ground-truth distribution and your predictions.<jupyter_code>## Using optimizer with default settings 
model.compile('rmsprop', 'mse', 
               metrics=['accuracy'])
## Using optimizer with custom argument values
from tensorflow.keras.optimizers import RMSprop

model.compile(optimizer=RMSprop(),
              loss='mse',
              metrics = ['accuracy']
             )<jupyter_output><empty_output><jupyter_text>Building your networkYou can intuitively understand teh dimensionality of your model as "how much freedom you're allowing the network to have when learning internal representations". Having more hidden units allows the model to learn more-complex representations/functions, but it makes the network more computationally expensive and may lead to learning unwanted patterns.There are two key architecture decisions to be made about such a stack of Dense layers:- How many layers to use- How many hidden units to choose for each layerAll the intermediate layers will use an **Activation Function** like **relu**. Without an activation function, The Dense layer would consist of two linear operations - a dot product and an addition.So the layer could only learn linear transformations of the input data. Such a model is to restricted and wouldn't benefit from multiple layers fo representations, because a deep stack of linear layers would still implement a linear operation: adding more layers wouldn't extend the learning ability of the model.In order to get access to a much richer network that would benefit from deep representaions, you need a non-linearity or activation function. relu is the most popular activation function, but there are many other candidates. **Exercise:** Visit [Tensorflow Playground](https://playground.tensorflow.org/) and try building (writing keras code) some of the models that you see.    Regression problem : Predicting house prices<jupyter_code>from tensorflow.keras.datasets import boston_housing

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

from tensorflow.keras.optimizers import RMSprop

import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Loading the dataset<jupyter_code>(X_train, y_train), (X_test, y_test) = boston_housing.load_data()
X_train.shape, y_train.shape, X_test.shape, y_test.shape
X_train[:2] # first 2 samples
y_train[:2] # first 2 labels<jupyter_output><empty_output><jupyter_text>The dataset contains 13 different features:1. Per capita crime rate.2. The proportion of residential land zoned for lots over 25,000 square feet.3. The proportion of non-retail business acres per town.4. Charles River dummy variable (= 1 if tract bounds river; 0 otherwise).5. Nitric oxides concentration (parts per 10 million).6. The average number of rooms per dwelling.7. The proportion of owner-occupied units built before 1940.8. Weighted distances to five Boston employment centers.9. Index of accessibility to radial highways.10. Full-value property-tax rate per $10,000.11. Pupil-teacher ratio by town.12. 1000 * (Bk - 0.63) ** 2 where Bk is the proportion of Black people by town.13. Percentage lower status of the population.Each one of these input data features is stored using a different scale. Some features are represented by a proportion between 0 and 1, other features are ranges between 1 and 12, some are ranges between 0 and 100, and so on. This is often the case with real-world data, and understanding how to explore and clean such data is an important skill to develop.**Labels**The labels are the house prices in thousands of dollars. (You may notice the mid-1970s prices.)  Building the model<jupyter_code>model = Sequential()
model.add(Dense(32, activation='relu', input_shape=(13,)))
model.add(Dense(16, activation='relu'))
model.add(Dense(1))
model.summary()
model.compile('rmsprop', 'mse', metrics=['mae']) # Mean Absolute Error
history = model.fit(X_train, y_train, epochs=20, validation_data=(X_test, y_test))
model.predict(X_test[0:2]), y_test[0:2]
history.history.keys()
plt.figure(figsize=(10,8))
plt.plot(list(range(20)), history.history['mean_absolute_error'])
plt.plot(list(range(20)), history.history['val_mean_absolute_error'])
plt.title('Mean Absolute Error')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07656050955414012, 'fraction_numerical': 0.012993630573248408, 'mean_word_length': 5.445812807881773, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '349414', 'license': 'MIT', 'n_tokens_mistral': 2112, 'n_tokens_neox': 1979, 'n_words': 932, 'path': 'notebooks/01_intro.ipynb', 'repo_name': 'Ankur-singh/mitwpu', 'size': '7855'}