<jupyter_start><jupyter_text>Training a mean regressorIn this notebook we are going to train a neural network able to predict the mean value of a signal.<jupyter_code>from IPython.core.display import display, HTML
display(HTML("<style>.container { width:75% !important; }</style>"))
%matplotlib inline
%load_ext autoreload
%autoreload 2<jupyter_output><empty_output><jupyter_text>Training and testing datasets<jupyter_code>from data import TimeSeriesDataset, RandomTimeWindow, ToTensor, Metrics, Compose
from torch.utils.data import DataLoader<jupyter_output><empty_output><jupyter_text>We configure a transformation to have a mean value calculated on the signal:- a random window of 256 sample is extracted from the time series- the only metric calculated is the mean value- the sample is then converted to a torch tensor<jupyter_code>transform = Compose([
    RandomTimeWindow(256), 
    Metrics(metrics=["mean"]), 
    ToTensor()])<jupyter_output><empty_output><jupyter_text>We use univariate time-series and limit the training and testing samples to a single group:<jupyter_code>data_train = TimeSeriesDataset("/files/cedric/datasets/personnal/time_series/train_time_series.hdf5", 
                               nb_groups=1, nb_parameters=1, transform=transform)
data_test = TimeSeriesDataset("/files/cedric/datasets/personnal/time_series/test_time_series.hdf5", 
                              nb_groups=1, nb_parameters=1, transform=transform)
print("Number of training samples: ", len(data_train))
print("Number of testing samples: ", len(data_test))
batch_size = 64
data_train_loader = DataLoader(data_train, batch_size=batch_size, shuffle=True, num_workers=4)
data_test_loader = DataLoader(data_test, batch_size=batch_size, shuffle=True, num_workers=4)<jupyter_output><empty_output><jupyter_text>Visualizing samplesThe following function enables to visualize the samples with metrics. The plot also makes appear a convolution of signal that has been applied to calculate derivatives and integrals, it is not used in this simple example.<jupyter_code>from data import plot_sample
data_train.dataloader(False)
sample = data_train[42]
plot_sample(sample, fig_width=14)<jupyter_output><empty_output><jupyter_text>A fully-connected networkThis problem is easy for a fully connected network, it requires only a single layer. An obvious solution is to have all weights equal to $\frac{1}{256}$ and the network will output the mean of all input parameters.<jupyter_code>from torch import nn
import torch
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.flatten = nn.Flatten(1)
        self.dense1 = nn.Linear(256, 1)
    def forward(self, x):
        return self.dense1(self.flatten(x))
model = Model()
print(model)<jupyter_output>Model(
  (flatten): Flatten()
  (dense1): Linear(in_features=256, out_features=1, bias=True)
)<jupyter_text>We can then configure a fitting object:<jupyter_code>from model import ModelFitting, sub_model
fitter = ModelFitting(model, data_train_loader, data_test_loader)
fitter.set_optimizer(torch.optim.Adam, lr=0.001)
fitter.set_loss(nn.MSELoss(reduction="mean"))
fitter.check_model()
fitter.fit(epochs=30)
fitter.plot_convergence();<jupyter_output>Fitting the model using cuda:0 as device.
Epoch 0091 training: 100.0% Samples per second: 5.434e+03 Running loss: 3.8768e-06 Elapsed time: 0.1 seconds 
Epoch 0091 testing:  100.0% Samples per second: 5.205e+03 Testing loss: [32m3.2398e-06[0m Elapsed time: 0.3 seconds 
Epoch 0092 training: 100.0% Samples per second: 6.010e+03 Running loss: 4.0729e-06 Elapsed time: 0.4 seconds 
Epoch 0092 testing:  100.0% Samples per second: 6.050e+03 Testing loss: [32m3.0216e-06[0m Elapsed time: 0.6 seconds 
Epoch 0093 training: 100.0% Samples per second: 6.209e+03 Running loss: 3.8705e-06 Elapsed time: 0.7 seconds 
Epoch 0093 testing:  100.0% Samples per second: 6.154e+03 Testing loss: [32m2.6873e-06[0m Elapsed time: 0.8 seconds 
Epoch 0094 training: 100.0% Samples per second: 6.374e+03 Running loss: 2.7858e-06 Elapsed time: 1.0 seconds 
Epoch 0094 testing:  100.0% Samples per second: 6.166e+03 Testing loss: [32m3.0014e-06[0m Elapsed time: 1.1 seconds 
Epoch 0095 training: 100.0% Samples per se[...]<jupyter_text>Let's have a look at model trained weights, in orange is the theoretical weight that shall be found by the network:<jupyter_code>import matplotlib.pyplot as plt
plt.hist(model.dense1.weight.detach().numpy().reshape(256), bins=21)
plt.axvline(1./256., color='orange');<jupyter_output><empty_output><jupyter_text>Let's take a testing sample and visualize the prediction of the network:<jupyter_code>data_test.dataloader(False);
sample = data_test[132]
plot_sample(sample, fig_width=14, prediction=model.forward(sample["data"]))<jupyter_output><empty_output><jupyter_text>The precision is good but it takes quite a few epochs to go there. Besides we are not at optimium solution. The local minimum we have found is quite goo though.  Forcing weights and bias initializationIn this example, we are going to intialize the weight ands bias to the optimum value and see what the training process does with it:<jupyter_code>model = Model()
model
def weight_init(x):
    if type(x) == nn.Linear:
        print(x)
        nn.init.constant_(x.weight, 1./256.)
        nn.init.constant_(x.bias, 0.)
    return x
model = model.apply(weight_init)
data_test.dataloader(False)
sample = data_test[132]
plot_sample(sample, fig_width=14, prediction=model.forward(sample["data"]))
fitter = ModelFitting(model, data_train_loader, data_test_loader)
fitter.set_optimizer(torch.optim.Adam, lr=0.0001)
fitter.set_loss(nn.MSELoss(reduction="mean"))
loss = fitter.validate(data_test_loader)
print(loss)<jupyter_output>Testing:    8.3% Samples per second: 6.705e+02 Testing loss: [32m6.4900e-16[0m Elapsed time: 0.1 seconds 
Testing:   16.7% Samples per second: 1.328e+03 Testing loss: [32m5.6574e-16[0m Elapsed time: 0.1 seconds 
Testing:   25.0% Samples per second: 1.983e+03 Testing loss: [32m6.4286e-16[0m Elapsed time: 0.1 seconds 
Testing:   33.3% Samples per second: 2.249e+03 Testing loss: [32m6.6082e-16[0m Elapsed time: 0.1 seconds 
Testing:   41.7% Samples per second: 2.803e+03 Testing loss: [32m6.5200e-16[0m Elapsed time: 0.1 seconds 
Testing:   50.0% Samples per second: 3.355e+03 Testing loss: [32m6.1420e-16[0m Elapsed time: 0.1 seconds 
Testing:   58.3% Samples per second: 3.906e+03 Testing loss: [32m6.9894e-16[0m Elapsed time: 0.1 seconds 
Testing:   66.7% Samples per second: 4.334e+03 Testing loss: [32m7.4157e-16[0m Elapsed time: 0.1 seconds 
Testing:   75.0% Samples per second: 4.810e+03 Testing loss: [32m8.1607e-16[0m Elapsed time: 0.1 seconds 
Testing:   83.3% Samples pe[...]<jupyter_text>With weights and bias forced to optimum value, the loos calculated on the testing dataset is very low.<jupyter_code>loss = fitter.validate(data_train_loader)
print(loss)<jupyter_output>Testing:    8.3% Samples per second: 7.595e+02 Testing loss: [32m1.2950e-15[0m Elapsed time: 0.1 seconds 
Testing:   16.7% Samples per second: 1.464e+03 Testing loss: [32m1.3795e-15[0m Elapsed time: 0.1 seconds 
Testing:   25.0% Samples per second: 1.885e+03 Testing loss: [32m1.2504e-15[0m Elapsed time: 0.1 seconds 
Testing:   33.3% Samples per second: 2.504e+03 Testing loss: [32m1.1901e-15[0m Elapsed time: 0.1 seconds 
Testing:   41.7% Samples per second: 3.122e+03 Testing loss: [32m1.1111e-15[0m Elapsed time: 0.1 seconds 
Testing:   50.0% Samples per second: 3.737e+03 Testing loss: [32m1.0105e-15[0m Elapsed time: 0.1 seconds 
Testing:   58.3% Samples per second: 4.015e+03 Testing loss: [32m9.8226e-16[0m Elapsed time: 0.1 seconds 
Testing:   66.7% Samples per second: 4.577e+03 Testing loss: [32m9.7668e-16[0m Elapsed time: 0.1 seconds 
Testing:   75.0% Samples per second: 5.075e+03 Testing loss: [32m9.6453e-16[0m Elapsed time: 0.1 seconds 
Testing:   83.3% Samples pe[...]<jupyter_text>And it's the same for the loss on the training samples.What happen if we try to train the model from there ?<jupyter_code>fitter.fit(epochs=30)
fitter.plot_convergence();<jupyter_output>Fitting the model using cuda:0 as device.
Epoch 0001 training: 100.0% Samples per second: 5.468e+03 Running loss: 1.8174e-05 Elapsed time: 0.1 seconds 
Epoch 0001 testing:  100.0% Samples per second: 5.311e+03 Testing loss: [32m3.8176e-06[0m Elapsed time: 0.3 seconds 
Epoch 0002 training: 100.0% Samples per second: 5.378e+03 Running loss: 4.6733e-06 Elapsed time: 0.4 seconds 
Epoch 0002 testing:  100.0% Samples per second: 6.814e+03 Testing loss: [32m7.4037e-06[0m Elapsed time: 0.6 seconds 
Epoch 0003 training: 100.0% Samples per second: 6.133e+03 Running loss: 2.9921e-06 Elapsed time: 0.7 seconds 
Epoch 0003 testing:  100.0% Samples per second: 5.882e+03 Testing loss: [32m8.6237e-07[0m Elapsed time: 0.8 seconds 
Epoch 0004 training: 100.0% Samples per second: 5.902e+03 Running loss: 9.4429e-07 Elapsed time: 1.0 seconds 
Epoch 0004 testing:  100.0% Samples per second: 6.096e+03 Testing loss: [32m1.8161e-07[0m Elapsed time: 1.1 seconds 
Epoch 0005 training: 100.0% Samples per se[...]<jupyter_text>We get a very fast convergence, and the weights of the network are getting close of the optimum value:<jupyter_code>import matplotlib.pyplot as plt
plt.hist(model.dense1.weight.detach().numpy().reshape(256), bins=21)
plt.axvline(1./256., color='orange');
data_test.dataloader(False);
sample = data_test[132]
plot_sample(sample, fig_width=14, prediction=model.forward(sample["data"]))<jupyter_output><empty_output><jupyter_text>The prediction of the neural network is much better now ! And the convergeence is fast. But how can we help the network converge to this solution without initializing the weights to the solution ?Besides, if we want to re-use the knowledge of this network, we need to have some kind of feature extraction. We will implement this with a convolutional neuron network.  Convolutional Neuron NetworkOnce again, there is an obvious solution with a convolutional neuron network. With a single convolutional layer, the convolution shall be doing a "rolling average" of the data. We can then apply a average pooling to reduce the dimensionality of the problem and then send this information to e fully connected network that will make the prediction. Let's build such a model with a single convolutional layer with 1 output channel.<jupyter_code>class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(1, 3), stride=1, padding=0, bias=True)
        self.activ1 = nn.ReLU()
        self.pool1 = nn.AvgPool2d(kernel_size=(1, 2), stride=(1, 2))        
        self.flatten = nn.Flatten(1)
        self.linear = nn.Linear(127, 1)
    def forward(self, x):
        x = self.pool1(self.activ1(self.conv1(x)))
        x = self.flatten(x)
        x = self.linear(x)
        return x
model = Model()
model
fitter = ModelFitting(model, data_train_loader, data_test_loader)
fitter.set_optimizer(torch.optim.AdamW, lr=0.001, weight_decay=0.1)
fitter.set_loss(nn.MSELoss(reduction="mean"))
fitter.check_model()
fitter.fit(epochs=10)
fitter.plot_convergence();<jupyter_output>Fitting the model using cuda:0 as device.
Epoch 0111 training: 100.0% Samples per second: 5.247e+03 Running loss: 1.0097e-04 Elapsed time: 0.1 seconds 
Epoch 0111 testing:  100.0% Samples per second: 5.548e+03 Testing loss: [32m8.5068e-05[0m Elapsed time: 0.3 seconds 
Epoch 0112 training: 100.0% Samples per second: 6.138e+03 Running loss: 8.1809e-05 Elapsed time: 0.4 seconds 
Epoch 0112 testing:  100.0% Samples per second: 6.597e+03 Testing loss: [32m8.0553e-05[0m Elapsed time: 0.5 seconds 
Epoch 0113 training: 100.0% Samples per second: 5.899e+03 Running loss: 5.8651e-05 Elapsed time: 0.7 seconds 
Epoch 0113 testing:  100.0% Samples per second: 6.861e+03 Testing loss: [32m8.3383e-05[0m Elapsed time: 0.8 seconds 
Epoch 0114 training: 100.0% Samples per second: 6.337e+03 Running loss: 8.7965e-05 Elapsed time: 0.9 seconds 
Epoch 0114 testing:  100.0% Samples per second: 6.524e+03 Testing loss: [32m8.0677e-05[0m Elapsed time: 1.1 seconds 
Epoch 0115 training: 100.0% Samples per se[...]<jupyter_text>We see that the model can perfectly be trained but at this stage the accuracy is quite good.<jupyter_code>data_test.dataloader(False);
sample = data_test[40]
plot_sample(sample, fig_width=14, prediction=model.forward(sample["data"].view(1, 1, 1, -1)));<jupyter_output><empty_output><jupyter_text>We can extract the first layer of the model and pass it a sample:<jupyter_code>submodel = sub_model(model, "conv1")
submodel
x_conv = submodel.forward(sample["data"].view(1, 1, 1, -1)).view(1, -1).detach().numpy()
plt.plot(x_conv[0])
plt.plot(sample["data"][0][0], color="k")<jupyter_output><empty_output><jupyter_text>A 2-layer convolutional neuron networkLet's introduce a second convolutional layer to further reduce the dimensionality of the feature extractor:<jupyter_code>class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(1, 3), stride=1, padding=0, bias=True)
        self.activ1 = nn.ReLU()
        self.pool1 = nn.AvgPool2d(kernel_size=(1, 2), stride=(1, 2))
        self.conv2 = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(1, 3), stride=1, padding=0, bias=True)
        self.activ2 = nn.ReLU()
        self.pool2 = nn.AvgPool2d(kernel_size=(1, 2), stride=(1, 2))        
        self.flatten = nn.Flatten(1)
        self.linear = nn.Linear(62, 1)
    def forward(self, x):
        x = self.pool1(self.activ1(self.conv1(x)))
        x = self.pool2(self.activ2(self.conv2(x)))
        x = self.flatten(x)
        x = self.linear(x)
        return x
model = Model()
model
fitter = ModelFitting(model, data_train_loader, data_test_loader)
fitter.set_optimizer(torch.optim.AdamW, lr=0.001, weight_decay=0.1)
fitter.set_loss(nn.MSELoss(reduction="mean"))
fitter.check_model()
fitter.fit(epochs=10)
fitter.plot_convergence();
data_test.dataloader(False);
sample = data_test[123]
plot_sample(sample, fig_width=14, prediction=model.forward(sample["data"].view(1, 1, 1, -1)));<jupyter_output><empty_output><jupyter_text>Unsurprisingly this architecture works well for a mean regressor.  Using max-pooling layers insteadWe replace average pooling by max-pooling to see if this might have an impact on the accuracy of the network.<jupyter_code>class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(1, 3), stride=1, padding=0, bias=True)
        self.activ1 = nn.ReLU()
        self.pool1 = nn.MaxPool2d(kernel_size=(1, 2), stride=(1, 2))
        self.conv2 = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(1, 3), stride=1, padding=0, bias=True)
        self.activ2 = nn.ReLU()
        self.pool2 = nn.MaxPool2d(kernel_size=(1, 2), stride=(1, 2))        
        self.flatten = nn.Flatten(1)
        self.linear = nn.Linear(62, 1)
    def forward(self, x):
        x = self.pool1(self.activ1(self.conv1(x)))
        x = self.pool2(self.activ2(self.conv2(x)))
        x = self.flatten(x)
        x = self.linear(x)
        return x
model = Model()
model
fitter = ModelFitting(model, data_train_loader, data_test_loader)
fitter.set_optimizer(torch.optim.AdamW, lr=0.001, weight_decay=0.1)
fitter.set_loss(nn.MSELoss(reduction="mean"))
fitter.check_model()
fitter.fit(epochs=30)
fitter.plot_convergence();
data_test.dataloader(False);
sample = data_test[123]
plot_sample(sample, fig_width=14, prediction=model.forward(sample["data"].view(1, 1, 1, -1)));<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11646917178886002, 'fraction_numerical': 0.07946264495279133, 'mean_word_length': 4.853857566765579, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 69, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '655140', 'license': 'MIT', 'n_tokens_mistral': 6363, 'n_tokens_neox': 5482, 'n_words': 1554, 'path': 'notebooks/04_Mean_Regressor.ipynb', 'repo_name': 'ccampguilhem/TimeSeriesLearning', 'size': '15781'}