<jupyter_start><jupyter_text>OverviewThe purpose of this notebook is to illustrate a neural network interpretability method which is compatible with linear regression. In linear regression, provided the independent variables are scaled, one can view the regression coefficients as a measure of importance of the variables. Equivalently, the dependent variable can be differentiated w.r.t. the inputs to give the coefficient. Similarly, the derivatives of the network w.r.t. the inputs are a non-linear generalization of interpretability in linear regression. Moreover, we should expect the neural network gradients to approximate the linear regression coefficients when the data is generated by a linear regression model. Various simple experimental tests, corresponding to Section 3 of Chpt 5, are performed to illustrate the properties of network interpretability.<jupyter_code>import numpy as np
from keras.models import Sequential
from keras.layers import Dense
from keras.callbacks import EarlyStopping
from keras.wrappers.scikit_learn import KerasRegressor
import statsmodels.api as sm
import sklearn<jupyter_output><empty_output><jupyter_text>Simple Data Generation Process (DGP)Let us generate data from the following linear regression model$Y=X_1+X_2 + \epsilon~, ~~X_1, X_2 \sim N(0,1)~, ~~\epsilon \sim N(0,1)$<jupyter_code>M = 5000 # Number of samples
np.random.seed(7) # Set NumPy's random seed for reproducibility
X = np.zeros(shape=(M, 2))
X[:int(M/2), 0] = np.random.randn(int(M/2))
X[:int(M/2), 1] = np.random.randn(int(M/2))

# Use antithetic sampling to reduce the bias in the mean
X[int(M/2):, 0] = -X[:int(M/2), 0]
X[int(M/2):, 1] = -X[:int(M/2), 1]

eps = np.zeros(shape=(M,1))
eps[:int(M/2)] = np.random.randn(int(M/2), 1)
eps[int(M/2):] = -eps[:int(M/2)]
Y = X[:, 0] + X[:, 1] + eps.flatten()<jupyter_output><empty_output><jupyter_text>Use ordinary least squares to fit a linear model to the dataFor a baseline, let us compare the neural network with OLS regression. We fit statsmodels' OLS model to the data<jupyter_code>ols_results = sm.OLS(Y, sm.add_constant(X)).fit()<jupyter_output><empty_output><jupyter_text>For each input, get the predicted $Y$ value according to the model<jupyter_code>y_ols = ols_results.predict(sm.add_constant(X))<jupyter_output><empty_output><jupyter_text>View characteristics of the resulting model. You should observe that the intercept is close to zero and the other coefficients are close to one.<jupyter_code>ols_results.summary()<jupyter_output><empty_output><jupyter_text>Compare with a feedforward NN with no hidden layersRecall that the feedforward network with no hidden layers or activation function is a linear regression model.Create a build function for the linear perceptron, which transforms the inputs directly to a single output<jupyter_code>def linear_NN0_model(l1_reg=0.0):    
    model = Sequential()
    model.add(Dense(1, input_dim=2, kernel_initializer='normal'))
    model.compile(loss='mean_squared_error', optimizer='adam', metrics=['mae', 'mse'])
    return model<jupyter_output><empty_output><jupyter_text>An early stopping callback to terminate training once the weights appear to have converged to an optimum.<jupyter_code>es = EarlyStopping(monitor='loss', mode='min', verbose=1, patience=10)<jupyter_output><empty_output><jupyter_text>Passing the build function for our model and training parameters to the `KerasRegressor` constructor to create a Scikit-learn-compatible regression model. This allows you to take advantage of the library's built-in tools and estimator methods, and to incorporate it into Scikit-learn pipelines.<jupyter_code>lm = KerasRegressor(build_fn=linear_NN0_model, epochs=40, batch_size=10, verbose=1, callbacks=[es])<jupyter_output><empty_output><jupyter_text>Train the model<jupyter_code>lm.fit(X, Y)<jupyter_output>Epoch 1/40
5000/5000 [==============================] - 3s 601us/step - loss: 2.4874 - mae: 1.2604 - mse: 2.4874
Epoch 2/40
5000/5000 [==============================] - 2s 364us/step - loss: 1.5458 - mae: 0.9998 - mse: 1.5458
Epoch 3/40
5000/5000 [==============================] - 2s 318us/step - loss: 1.1422 - mae: 0.8552 - mse: 1.1422
Epoch 4/40
5000/5000 [==============================] - 1s 261us/step - loss: 1.0092 - mae: 0.8027 - mse: 1.0092
Epoch 5/40
5000/5000 [==============================] - 2s 303us/step - loss: 0.9773 - mae: 0.7890 - mse: 0.9773 1s - loss: 1.0402 - ma
Epoch 6/40
5000/5000 [==============================] - 2s 319us/step - loss: 0.9719 - mae: 0.7869 - mse: 0.9719
Epoch 7/40
5000/5000 [==============================] - 2s 303us/step - loss: 0.9712 - mae: 0.7865 - mse: 0.9712
Epoch 8/40
5000/5000 [==============================] - 1s 286us/step - loss: 0.9713 - mae: 0.7865 - mse: 0.9713
Epoch 9/40
5000/5000 [==============================] - 1s 290us/step - l[...]<jupyter_text>Check that the weights are close to oneThe weights should be close to unity. The bias term is the second entry and should be close to zero.<jupyter_code>print("weights: " + str(lm.model.layers[0].get_weights()[0]))
print("bias: " + str(lm.model.layers[0].get_weights()[1]))<jupyter_output>weights: [[0.9869313]
 [1.0236046]]
bias: [0.00017654]<jupyter_text>Compare with a FFW Neural Network with one hidden layer (unactivated) This time we create a neural network with a hidden layer with 10 units.<jupyter_code>n = 10 # number of hidden units
def linear_NN1_model(l1_reg=0.0):    
    model = Sequential()
    model.add(Dense(n, input_dim=2, kernel_initializer='normal')) 
    model.add(Dense(1, kernel_initializer='normal', activation='linear'))
    model.compile(loss='mean_squared_error', optimizer='adam', metrics=['mae', 'mse'])
    return model
lm = KerasRegressor(build_fn=linear_NN1_model, epochs=50, batch_size=10, verbose=1, callbacks=[es])
lm.fit(X, Y)
W1 = lm.model.get_weights()[0]
b1 = lm.model.get_weights()[1]
W2 = lm.model.get_weights()[2]
b2 = lm.model.get_weights()[3]
print(W1, W2)<jupyter_output>[[ 0.21191542 -0.3510564   0.32653695  0.30687252  0.27739337  0.30587277
   0.26292333  0.24803898  0.250718   -0.3381098 ]
 [ 0.33603936 -0.28845066  0.25735727  0.3006563   0.32002118  0.28473574
   0.35539797  0.28512466  0.28092325 -0.26294112]] [[ 0.34959066]
 [-0.33088237]
 [ 0.29290077]
 [ 0.3498653 ]
 [ 0.26861414]
 [ 0.3326019 ]
 [ 0.35045084]
 [ 0.37188262]
 [ 0.40377927]
 [-0.34295896]]<jupyter_text>Check that the coefficients are close to one and the intercept is close to zero<jupyter_code>beta_0 = np.dot(np.transpose(W2), b1) + b2
beta_1 = np.dot(np.transpose(W2), W1[0])
beta_2 = np.dot(np.transpose(W2), W1[1])
print(beta_0, beta_1, beta_2)<jupyter_output>[-0.04406428] [0.97107023] [1.0083461]<jupyter_text>Compare with a feedforward NN with one hidden layer ($tanh$ activated) Finally, we create another model with a 10 unit hidden layer, this time with a $tanh$ activation function.<jupyter_code># number of hidden neurons
n = 10
# with non-linear activation
def linear_NN1_model_act(l1_reg=0.0):    
    model = Sequential()
    model.add(Dense(n, input_dim=2, kernel_initializer='normal', activation='tanh'))
    model.add(Dense(1, kernel_initializer='normal')) 
    model.compile(loss='mean_squared_error', optimizer='adam', metrics=['mae', 'mse'])
    return model
lm = KerasRegressor(build_fn=linear_NN1_model_act, epochs=100, batch_size=10, verbose=1, callbacks=[es])<jupyter_output><empty_output><jupyter_text>Train the model<jupyter_code>lm.fit(X, Y)<jupyter_output>Epoch 1/100
5000/5000 [==============================] - 2s 491us/step - loss: 1.6220 - mae: 0.9961 - mse: 1.6220
Epoch 2/100
5000/5000 [==============================] - 1s 281us/step - loss: 0.9934 - mae: 0.7969 - mse: 0.9934
Epoch 3/100
5000/5000 [==============================] - 1s 284us/step - loss: 0.9909 - mae: 0.7954 - mse: 0.9909
Epoch 4/100
5000/5000 [==============================] - 1s 278us/step - loss: 0.9882 - mae: 0.7949 - mse: 0.9882
Epoch 5/100
5000/5000 [==============================] - 1s 288us/step - loss: 0.9871 - mae: 0.7942 - mse: 0.9871
Epoch 6/100
5000/5000 [==============================] - 1s 290us/step - loss: 0.9851 - mae: 0.7934 - mse: 0.9851
Epoch 7/100
5000/5000 [==============================] - 1s 292us/step - loss: 0.9842 - mae: 0.7928 - mse: 0.9842
Epoch 8/100
5000/5000 [==============================] - 1s 295us/step - loss: 0.9828 - mae: 0.7916 - mse: 0.9828
Epoch 9/100
5000/5000 [==============================] - 2s 316us/step - loss: 0.9825 - [...]<jupyter_text>Compute the Sensitivities<jupyter_code># Assumes that the activation function is tanh
def sensitivities(lm, X):
    
    W1 = lm.model.get_weights()[0]
    b1 = lm.model.get_weights()[1]
    W2 = lm.model.get_weights()[2]
    b2 = lm.model.get_weights()[3]
    
    
    M = np.shape(X)[0]
    p = np.shape(X)[1]

    beta = np.array([0]*M*(p+1), dtype='float32').reshape(M,p+1)
    
    beta[:, 0] = (np.dot(np.transpose(W2), np.tanh(b1)) + b2)[0] # intercept \beta_0= F_{W,b}(0)
    for i in range(M):
 
        Z1 = np.tanh(np.dot(np.transpose(W1),np.transpose(X[i,])) + b1)
      
        D = np.diag(1 - Z1**2)
        
        for j in range(p):  
            beta[i, j+1] = np.dot(np.transpose(W2), np.dot(D, W1[j]))
            
    return beta
beta = sensitivities(lm, X)<jupyter_output><empty_output><jupyter_text>Check that the intercept is close to one and the coefficients are close to one<jupyter_code>print(np.mean(beta, axis=0))
print(np.std(beta, axis=0))<jupyter_output>[5.578546e-07 6.809256e-02 7.658932e-02]
{'fraction_non_alphanumeric': 0.18225772545722094, 'fraction_numerical': 0.10710531847803237, 'mean_word_length': 4.79123554473524, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 59, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '142245', 'license': 'Unlicense', 'n_tokens_mistral': 4079, 'n_tokens_neox': 3374, 'n_words': 888, 'path': 'Chapter5-Interpretability/ML_in_Finance-Interpretability.ipynb', 'repo_name': 'JJJerome/ML_Finance_Codes', 'size': '9514'}