<jupyter_start><jupyter_text>Deep learning basics: a simple regression caseIn this tutorial, we train a Multi Layer Perceptron (MLP) to fit a noisy sinusoid curve.  Step 1: Prepare the environment<jupyter_code>import torch
import numpy as np
import matplotlib.pyplot as plt
from ipywidgets import IntSlider, interact
%matplotlib nbagg
# Define computing device
use_cuda = False

if torch.cuda.is_available and use_cuda:
    print('We have GPU !')
    device = torch.device('cuda')
else:
    print('We will use CPU')
    device = torch.device('cpu')
# Fix random seed for reproducibility
torch.manual_seed(0)<jupyter_output><empty_output><jupyter_text>Step 2: Generate training data<jupyter_code># Training data
x = torch.linspace(-10, 10, 1000).unsqueeze(1).to(device)
# Label
y = torch.sin(x) + 0.2 * torch.rand_like(x)<jupyter_output><empty_output><jupyter_text>experiment with a different function that works when modifying a specific parameter ?  Step 3: Build the neural network We build a MLP, i.e., composed of Fully Connected layers, with 2 hidden layers.![T1_MLP-2.png](attachment:T1_MLP-2.png)<jupyter_code>neurons_lin1 = 200
neurons_lin2 = 100

model = torch.nn.Sequential(
    # hidden layers
    torch.nn.Linear(in_features=x.shape[1], out_features=neurons_lin1),
    torch.nn.LeakyReLU(),
    torch.nn.Linear(in_features=neurons_lin1, out_features=neurons_lin2),
    torch.nn.LeakyReLU(),
    # output layers
    torch.nn.Linear(in_features=neurons_lin2, out_features=y.shape[1])
)

model.to(device)<jupyter_output><empty_output><jupyter_text>Step 4: Define training hyperparameters<jupyter_code># Loss function
criterion = torch.nn.MSELoss()<jupyter_output><empty_output><jupyter_text>ExerciseTweak the learning rate to improve the performance of the model<jupyter_code>learning_rate = 0.5
# Optimizer (Gradient descent)
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Training duration
num_epochs = 500<jupyter_output><empty_output><jupyter_text>Step 5: Train the network<jupyter_code>predictions = []
losses = []

for epoch in range(num_epochs):
    # Reinitialize gradient of the model weights
    optimizer.zero_grad()
    
    # Prediction
    y_pred = model(x)
    
    # Error measurement
    loss = criterion(y_pred, y)
    
    # Backpropagation
    loss.backward()
    
    # Weight update
    optimizer.step()
#     scheduler.step()
    
    predictions.append(y_pred.detach().to('cpu'))
    losses.append(loss.detach().to('cpu'))<jupyter_output><empty_output><jupyter_text>Training results<jupyter_code>plt.figure(figsize=(4,3))
plt.xlabel('Epoch')
plt.title('Training loss')
plt.plot(losses)
plt.tight_layout()
fig, ax = plt.subplots()

ax.scatter(x.to('cpu'), y.to('cpu'), s=1, c='b', label='training data')
pred, = ax.plot(x.to('cpu'), predictions[0], c='orange', label='prediction')
fig.legend()

@interact(epoch=IntSlider(min=0, max=num_epochs-1, step=50, value=0))
def update(epoch=0):
    pred.set_ydata(predictions[epoch])
    print('Loss: {:0.4f}'.format(losses[epoch]))
    fig.canvas.draw()<jupyter_output><empty_output><jupyter_text>Step 6: Test the model on new data<jupyter_code># Test data
x_test = torch.linspace(-10, 10, 1500).unsqueeze(1).to(device)
# Label
y_test = torch.sin(x_test) + 0.2 * torch.rand_like(x_test)
# Switch the model to test mode
# This is important for some kinds of layers, such as BatchNorm, that have 
# different behavior at test and training time
model.eval()

# We don't need to build the gradient graph, so let's save some memory !
with torch.no_grad():
    y_pred_test = model(x_test)
    test_loss = criterion(y_pred_test, y_test)
fig_test, ax_test = plt.subplots()

ax_test.scatter(x_test.to('cpu'), y_test.to('cpu'), s=1, c='b', label='test data')
ax_test.plot(x_test.to('cpu'), y_pred_test, c='orange', label='test prediction')
ax_test.annotate(text='Test Loss: {:0.4f}'.format(test_loss),  xy=(6, -1))
fig_test.legend()<jupyter_output><empty_output><jupyter_text>Do you need a "solution" ?![down_arrow-3.png](attachment:down_arrow-3.png)  Hint You can explore learning rates in the range [0.1 ; 0.001]  Solution![down_arrow.png](attachment:down_arrow.png)<jupyter_code># Quite good parameter
learning_rate = 0.05<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13083451202263083, 'fraction_numerical': 0.019094766619519095, 'mean_word_length': 5.619344773790951, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '406912', 'license': 'MIT', 'n_tokens_mistral': 1581, 'n_tokens_neox': 1515, 'n_words': 333, 'path': 'deep-learning/T1_fitting_sin_with_MLP.ipynb', 'repo_name': 'feroner/Escapeschool2020', 'size': '4242'}