<jupyter_start><jupyter_text>Visualizing Hidden Layers<jupyter_code>%matplotlib notebook

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.model_selection import train_test_split

def plot_decision_boundary(model, X, y):
    X_max = X.max(axis=0)
    X_min = X.min(axis=0)
    xticks = np.linspace(X_min[0], X_max[0], 100)
    yticks = np.linspace(X_min[1], X_max[1], 100)
    xx, yy = np.meshgrid(xticks, yticks)
    ZZ = model.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = ZZ[:,0] >= 0.5
    Z = Z.reshape(xx.shape)
    fig, ax = plt.subplots()
    ax = plt.gca()
    ax.contourf(xx, yy, Z, cmap=plt.cm.bwr, alpha=0.2)
    ax.scatter(X[:,0], X[:,1], c=y, alpha=0.4)

df = pd.read_csv('../data/geoloc_elev.csv')

# we only use the 2 features that matter
X = df[['lat', 'lon']].values
y = df['target'].values

X_train, X_test, y_train, y_test = train_test_split(X, y, 
    test_size = 0.3, random_state=0)
# we are using notebook interactive plotting.
# make sure to snap the plot to the notebook before proceeding
df.plot(kind='scatter',
        x='lat',
        y='lon',
        c='target',
        cmap='bwr');
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
model = Sequential()

model.add(Dense(4, input_dim=2, activation='tanh'))
model.add(Dense(3, activation='tanh'))
model.add(Dense(1, activation='sigmoid'))

model.compile(SGD(lr=0.5), 'binary_crossentropy', metrics=['accuracy'])

h = model.fit(X_train, y_train, epochs=20, verbose=0, validation_split=0.1)
# we are using notebook interactive plotting.
# make sure to snap the plot to the notebook before proceeding
plot_decision_boundary(model, X, y)<jupyter_output><empty_output><jupyter_text>Representation Learning: inspecting the output of the hidden layer<jupyter_code>model.layers<jupyter_output><empty_output><jupyter_text>In order to extract the activations at the hidden layer from Keras, we can [create a function](http://keras.io/getting-started/faq/how-can-i-visualize-the-output-of-an-intermediate-layer) where we specify what layer we would like to "extract" the value of like so:<jupyter_code>from keras import backend as K
input_t = model.layers[0].input
inner_t = model.layers[1].output
get_hidden_layer_output = K.function([input_t], [inner_t])
get_hidden_layer_output
H = get_hidden_layer_output([X_test])[0]

H.shape
from mpl_toolkits.mplot3d import Axes3D
# A helper function to make a 3d scatter plot
def plot_3d_representation(X, y):
    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    ax.scatter(X[:,0], X[:,1], X[:,2], c=y, alpha=0.2)
    ax.view_init(60, 30)
plot_3d_representation(H, y_test)<jupyter_output><empty_output><jupyter_text>What do you see? Are the classes linearly separable now? Discuss what you think is happening with a partner or instructor.  Exercise 1- Reset the above model to random weights and inspect the hidden layer representation- Are the two classes well separated without training?<jupyter_code>model = Sequential()

model.add(Dense(4, input_dim=2, activation='tanh'))
model.add(Dense(3, activation='tanh'))
model.add(Dense(1, activation='sigmoid'))

model.compile(SGD(lr=0.5), 'binary_crossentropy', metrics=['accuracy'])
get_hidden_layer_output = K.function([model.layers[0].input],
                                     [model.layers[1].output])

H = get_hidden_layer_output([X_test])[0]
plot_3d_representation(H, y_test)<jupyter_output><empty_output><jupyter_text>Execise 2Let's separate two True from False banknotes and look how the model learns the inner representation.- Load the `../data/banknotes.csv` dataset into a pandas dataframe- Inspect it using Seaborn Pairplot- Separate features from labels. Labels are contained in the `class` column- Split data into train and test sets, using a 30% test size and random_state=42- Create a model with the following architecture:        Input: 4 features        Inner layer: 2 nodes, relu activation        Output layer: 1 node, sigmoid- Compile the model- Set the model weights to the initial weights provided below using `model.set_weights`- Train the model one epoch at a time, and at each epoch visualize the test data as it appears at the output of the inner layer `model.layers[0].output` on a 2D scatter plot.You should see model gradually learn to separate the 2 classes.<jupyter_code>weights = [np.array([[-0.26285839,  0.82659411],
                     [ 0.65099144, -0.7858932 ],
                     [ 0.40144777, -0.92449236],
                     [ 0.87284446, -0.59128475]]),
           np.array([ 0.,  0.]),
           np.array([[-0.7150408 ], [ 0.54277754]]),
           np.array([ 0.])]
df = pd.read_csv('../data/banknotes.csv')
df.head()
import seaborn as sns
sns.pairplot(df, hue="class");
X = df.drop('class', axis=1).values
y = df['class'].values
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42)
from keras.optimizers import RMSprop
model = Sequential()
model.add(Dense(2, input_shape=(4,), activation='relu'))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer=RMSprop(lr=0.01),
              metrics=['accuracy'])
model.set_weights(weights)
features_function = K.function([model.layers[0].input],
                               [model.layers[0].output])
plt.figure(figsize=(15,10))

for i in range(1, 26):
    plt.subplot(5, 5, i)
    h = model.fit(X_train, y_train, batch_size=16,
                  epochs=1, verbose=0)
    test_acc = model.evaluate(X_test, y_test,
                              verbose=0)[1]
    features = features_function([X_test])[0]
    plt.scatter(features[:, 0], features[:, 1],
                c=y_test, cmap='coolwarm', marker='.')
    plt.xlim(-0.5, 15)
    plt.ylim(-0.5, 15)
    
    acc_ = test_acc * 100.0
    t = 'Epoch: {}, Test Acc: {:3.1f} %'.format(i, acc_)
    plt.title(t, fontsize=11)

plt.tight_layout();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13325530847684333, 'fraction_numerical': 0.03628155826784819, 'mean_word_length': 3.972568578553616, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '424132', 'license': 'MIT', 'n_tokens_mistral': 2193, 'n_tokens_neox': 2061, 'n_words': 537, 'path': 'solutions_do_not_open/Lab_12_DL Visualizing Hidden Layers_solution.ipynb', 'repo_name': 'differentmatt/ztdl-5-day-bootcamp', 'size': '5981'}