<jupyter_start><jupyter_text>Learn the noise  Overview Can a Neural Net make predictions about the noise not just the signal?In other words, can a ML algorithm learn how much it does NOT know?I think so!To gain intuition we use the simple example of fitting the (noisy) data and check wheather the ML algorithm can predict both the signal and the noise. SetupConsider a noisy labelled data $(x,y)$. The data is generated by drawing samples from the equation$$    y_i= f(x_i) + \eta_i,$$where $f(x_i)$ is some fixed, but (possibly unknown) function, and $\eta_i$ is a Gaussian, uncorrelate noise variable such that$$\langle \eta_i \rangle=0 \\\langle \eta_i \eta_j \rangle = \delta_{ij} r^2$$The NN architecture is as following:1. one input unit (containing x_i)2. one or more hidden layers with a number of hidden units hidden={n1,n2,n3,...}3. there are two output units. One is predicting the signal $\hat{y}_i$ and has a linear activation function, while the other is predicting the noise $\sigma^2$ and has a Relu activation function.The proposed loss function is:$$\mathcal{Loss}=\sum_i \frac{(y_i-\hat{y}_i)^2}{\sigma^2_i+\epsilon} + \sigma^2_i$$where $\epsilon=1E-5$ is a small number used to prevent division by zero. We note that the minimum of the Loss function is obtained for $$\hat{y}_i \approx y_i \quad\text{and}\quad \sigma^2_i = (y_i-\hat{y}_i)^2$$Of course a model will try to overfit the data (i.e. fit the noise as well) to produce $\hat{y}_i = y_i$ and $\sigma^2_i=0$ which is the global minimum of the loss function. However overfitting is easy to identify since leads to the Test error being much larger than the Train error (high variance problem). When the model is optimally trainined (neither overfit not underfit) the the train error and test error should be similar and hopefully the NN will predict $\sigma^2_i \approx r^2$ for ANY $i$. ParametersThe parameters of the model are:1. data generation: range of ${x_i}$, functional form of $f(x_i)=\sin(20*x_i)$, value of the noise $r^2=0, 0.1, 0.5$, dataset size $m$2. NN architecture: values of $n1,n2,n3,...$3. hyperparameters: epochs, gradient descent method.<jupyter_code># Generate the data that we are trying to fit
import numpy as np
import matplotlib as mpl
from matplotlib import pyplot as plt

def f(x):
    return np.sin(20*x)

N_train=1000 # size of train set
N_test=100   # size of test set
xmin=0 #minimum value of x 
xmax=1 #maximum value of x
r2=0.2 #noise strenght in the data

# Train and Test dataset
np.random.seed(1) # for reproducibility
x_train_raw = xmin + np.random.rand(N_train)*(xmax-xmin)
x_test_raw  = xmin + np.random.rand(N_test)*(xmax-xmin)
y_train_raw = f(x_train_raw)+np.random.randn(N_train)*r2
y_test_raw  = f(x_test_raw)+np.random.randn(N_test)*r2
# Plotting 
ordered=np.linspace(xmin,xmax,100)
plt.plot(x_train_raw,y_train_raw, "o",ms=3, label='Training')
plt.plot(x_test_raw,y_test_raw, "o",ms=3, label='Test')
plt.plot(ordered,f(ordered),ms=2, label='Exact')
plt.legend(loc='upper left')
plt.show()
#import os
#os.environ["THEANO_FLAGS"] = "mode=FAST_RUN,device=gpu,floatX=float32"
import theano
from theano import tensor as T
theano.config.compute_test_value = 'off'

from keras.layers import Input, Dense, concatenate
from keras.models import Model
import keras.backend as K



# This returns a tensor
inputs = Input(shape=(1,),name='input')

# a layer instance is callable on a tensor, and returns a tensor
x = Dense(40, activation='relu', name='layer1')(inputs)
x = Dense(40, activation='relu', name='layer2')(x)
x = Dense(40, activation='relu', name='layer3')(x)
y_hat = Dense(1, activation='linear', name='y_hat')(x)
s2    = Dense(1, activation='relu',name='s2')(x)
predictions = concatenate([y_hat,s2],name='predictions')


# Define the model
model = Model(inputs=inputs, outputs=predictions)
model.summary()
def custom_loss(y_true, y_pred):
    y_only_true  = y_true[:,0]
    y_only_pred  = y_pred[:,0]
    s2_only_pred = y_pred[:,1]
    return T.mean(T.square(y_only_true-y_only_pred)/(s2_only_pred+1E-5)+s2_only_pred)
    
def mse_y_only(y_true, y_pred):
    y_only_true  = y_true[:,0]
    y_only_pred  = y_pred[:,0]
    return T.mean(T.square(y_only_true-y_only_pred))   

model.compile(loss=[custom_loss], optimizer='adam',metrics=[custom_loss,mse_y_only])
labels_train=np.zeros((y_train_raw.shape[0],2))
labels_train[:,0]=y_train_raw[:]

labels_test=np.zeros((y_test_raw.shape[0],2))
labels_test[:,0]=y_test_raw[:]

history=model.fit([x_train_raw],[labels_train], epochs=5000, batch_size=100,verbose=0)
#print(history.history.keys())

plt.figure()
plt.plot(history.history['custom_loss'],label='loss')
plt.plot(history.history['mse_y_only'],label='mse')
plt.xlabel('epoch')
plt.legend(loc='upper left')
plt.show()
#score = model.evaluate(x_train_raw,[y_train_raw,s2_train], verbose=0)
print(model.metrics_names)
train_score = model.evaluate(x_train_raw,labels_train, verbose=0)
test_score = model.evaluate(x_test_raw,labels_test, verbose=0)

print("train_score",train_score)
print("test_score",test_score)

predictions =model.predict(x_test_raw)
y_pred=predictions[:,0]
s2_pred=predictions[:,1]
mean_s2=np.mean(s2_pred)
print("mean_s2 =>",mean_s2)
print(s2_pred)

plt.figure()
plt.errorbar(x_test_raw,y_pred, yerr=np.sqrt(s2_pred),label='Test',fmt='o')
plt.plot(ordered,f(ordered),ms=2, label='Exact')<jupyter_output>['loss', 'custom_loss', 'mse_y_only']
train_score [0.40256698656082152, 0.40256698656082152, 0.040720481149852279]
test_score [0.40619349956512452, 0.40619349956512452, 0.040881703794002536]
mean_s2 => 0.204262
[ 0.21229216  0.21058449  0.22685601  0.22026917  0.20826006  0.20059997
  0.17592375  0.20882553  0.17336321  0.22986843  0.22136727  0.21144865
  0.21715997  0.20063767  0.17496018  0.21829468  0.19825229  0.22087252
  0.18292651  0.20105484  0.20469907  0.20372136  0.21129467  0.17341763
  0.20952052  0.2178629   0.20197108  0.21664676  0.18359429  0.22370335
  0.18200922  0.20040826  0.19015142  0.20852453  0.19157162  0.20212498
  0.20796317  0.18812329  0.21408707  0.17061323  0.17702341  0.20687363
  0.19906372  0.223894    0.22154957  0.20254469  0.22963519  0.22888146
  0.21135871  0.21111156  0.20017816  0.22071251  0.2262952   0.19117855
  0.19899587  0.23265845  0.20129351  0.23851204  0.19471708  0.20686907
  0.18892479  0.18410599  0.20249546  0.17029512  0.2040165[...]
{'fraction_non_alphanumeric': 0.1295858919855141, 'fraction_numerical': 0.12753897024090693, 'mean_word_length': 5.38391959798995, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '28539', 'license': 'MIT', 'n_tokens_mistral': 2877, 'n_tokens_neox': 2430, 'n_words': 606, 'path': 'learn_the_noise/learn_the_noise_custom.ipynb', 'repo_name': 'dalessioluca/ML_educational', 'size': '6351'}