<jupyter_start><jupyter_text>Intro to PyTorch for classification tasksWe'll use Pytorch together with supporting libraries `tensorlayers` and `skorch` to train a regressor that will map synthetic accoustic seismic waveforms and their corresponding velocity profiles. The dataset was put together by <NAME> and is hosted on github here: [https://github.com/LukasMosser/SNIST](https://github.com/LukasMosser/SNIST)<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import datetime
import os
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
urls = [
        'https://raw.githubusercontent.com/LukasMosser/SNIST/master/data/train/train_amplitudes.npy',
        'https://raw.githubusercontent.com/LukasMosser/SNIST/master/data/train/train_velocities.npy',
        'https://raw.githubusercontent.com/LukasMosser/SNIST/master/data/test/test_amplitudes.npy',
        'https://raw.githubusercontent.com/LukasMosser/SNIST/master/data/test/test_velocities.npy',
        'https://raw.githubusercontent.com/LukasMosser/SNIST/master/data/test/test_amplitudes_noise_1.npy',
        'https://raw.githubusercontent.com/LukasMosser/SNIST/master/data/test/test_amplitudes_noise_2.npy'
    ]<jupyter_output><empty_output><jupyter_text>Numpy allows you to point at URL data sources. It'll take care of downloading them and keeping reference of where they are with respect to a root folder specified by the user.<jupyter_code>ds = np.DataSource('../data/')

train_amplitudes = np.load(ds.open(urls[0], mode='rb'))
train_velocities = np.load(ds.open(urls[1], mode='rb'))
plt.imshow(train_amplitudes[0], aspect=0.06)
plt.plot(train_velocities[0])<jupyter_output><empty_output><jupyter_text>Let's define some dataset parameters. Note that these come from the [SNIST](https://github.com/LukasMosser/SNIST) properties.<jupyter_code>N_train = 600    # Number of total training examples
N_val = 150      # Number of samples used for validation
N_samples = 271  # Number of samples in time
N_recorders = 20 # Number of recording stations
N_target = 9     # Number of layers in the target velocity model
N_z = 360        # Number of grid blocks in z-dimension (only used for visualisation)<jupyter_output><empty_output><jupyter_text>Now some neural network parameters:<jupyter_code>lr = 1e-2                  # Learning rate
batch_size = N_train-N_val # Batchsize used in training - do full batch evaluation because of small data
N_epochs = 200             # Number of epochs to train for
import torch
from torch import nn

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
device
# Number of neurons in hidden layer
n_hidden = 50

# Setup a sequential network
model = nn.Sequential(
            nn.Linear(N_recorders*N_samples, n_hidden),
            nn.Sigmoid(),
            nn.Linear(n_hidden, N_target),
        )

model.to(device)
print(model)<jupyter_output><empty_output><jupyter_text>We need to make sure the inputs are standarized:<jupyter_code>train_mean, train_std = train_amplitudes.mean(), train_amplitudes.std()
train_vel_max = train_velocities.max()
X = train_amplitudes - train_mean
X /= train_std

y = train_velocities / train_vel_max

X_train = X[:-N_val]
y_train = y[:-N_val]


X_test = X[N_train-N_val:]
y_test = y[N_train-N_val:]<jupyter_output><empty_output><jupyter_text>Let's confirm that the shapes of these matrices are OK:<jupyter_code>X_train.shape, y_train.shape, X_test.shape, y_test.shape<jupyter_output><empty_output><jupyter_text>We want to make the training data to be 1D to be allowed through this fully connected neural network.<jupyter_code>X_train = X_train.reshape((N_train-N_val, N_samples*N_recorders))
X_test = X_test.reshape((N_val, N_samples*N_recorders))<jupyter_output><empty_output><jupyter_text>Now we can implement<jupyter_code>import torch.utils.data as utils

X_train_ = torch.tensor(X_train, dtype=torch.float).to(device)
y_train_ = torch.tensor(y_train, dtype=torch.float).to(device)

traindata = utils.TensorDataset(X_train_, y_train_)
trainloader = utils.DataLoader(traindata)
import torch.optim as optim

optimizer = optim.SGD(model.parameters(), lr=1e-3)
criterion = nn.MSELoss()
epochs = 10

for epoch in range(epochs):
    epoch_loss = 0.0
    for i, data in enumerate(trainloader):
        inputs, labels = data
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        epoch_loss += loss.item()
    print(f"# {epoch+1}  Loss {epoch_loss}")
print('Finished Training')<jupyter_output><empty_output><jupyter_text>We can make predictions with the trained model:<jupyter_code>X_test_ = torch.tensor(X_test, dtype=torch.float).to(device)
y_test_ = torch.tensor(y_test, dtype=torch.float).to(device)

valdata = utils.TensorDataset(X_test_, y_test_)
valloader = utils.DataLoader(valdata)
y_pred = np.array([model(xi).cpu().detach().numpy() for xi, yi in valloader])
y_pred = np.squeeze(y_pred)
plt.plot(y_pred[0], label='Predicted velocity')
plt.plot(y_test[0], label='True velocity')
plt.legend()<jupyter_output><empty_output><jupyter_text>You can save the model like this:<jupyter_code>torch.save(model.state_dict(), 'model.ckpt')<jupyter_output><empty_output><jupyter_text>Exercise- Create a training function to perform the whole training loop. For now, it should receive as parameters `X_train` and `y_train` and return a trained model- Put as parameter the number of neurons that are part of the single hidden layer and see how the training performance (loss) varies- Add more hidden layers and train models with different network configurations<jupyter_code># Your code here.<jupyter_output><empty_output><jupyter_text>A couple of extra libraries can help the process of making neural networks with Pytorch:- `Torchlayers`: Figures out dimensionality of input and output of each layer- `skorch`: Provides a `scikit-learn` compatible object from a `Pytorch` network to incorporate in scikit-learn workflows<jupyter_code>import torchlayers as tl
from skorch import NeuralNetRegressor<jupyter_output><empty_output><jupyter_text>A model can now be created like this:<jupyter_code># Define the architecture of the network
net_arch = torch.nn.Sequential(
    tl.Linear(50),  # specify ONLY out_channels
    tl.Sigmoid(),  # activation from first hidden layer
    tl.Linear(10),  # specify ONLY out_channels
    tl.Sigmoid(), # activation from second hidden layer
    tl.Linear(N_target),  # Output for 10 classes
)

# Build the network
net = tl.build(net_arch, torch.randn(1, *X_train[0].shape)) # torchlayers needs an input example to figure out the internal dimensions of the network<jupyter_output><empty_output><jupyter_text>Now we can create the scikit-learn compatible object:<jupyter_code>model = NeuralNetRegressor(
    net,
    max_epochs=10,
    lr=0.01,
)

model.fit(X_train, y_train)
y_proba = model.predict(X_test)
y_proba[0]
plt.plot(y_pred[0], label='Predicted velocity')
plt.plot(y_test[0], label='True velocity')
plt.legend()<jupyter_output><empty_output><jupyter_text>Convolutional Neural NetworksWhat'as a Convolutional Neural Network? - Keep these cheatsheets at hand: https://github.com/afshinea/stanford-cs-230-deep-learning ![alt text](../images/convolution-layer-a.png)  What's max-pooling?![alt text](../images/max-pooling-a.png)<jupyter_code>X = train_amplitudes - train_mean
X /= train_std

y = train_velocities / train_vel_max

X_train = X[:-N_val]
y_train = y[:-N_val]


X_test = X[N_train-N_val:]
y_test = y[N_train-N_val:]
# torch.nn and torchlayers can be mixed easily
net_arch = torch.nn.Sequential(
    tl.Conv(32),  # specify ONLY out_channels
    nn.ReLU(),  # use torch.nn wherever you wish
    tl.GlobalMaxPool(),  # Known from Keras
    tl.Linear(5), # Add a fully connected hidden layer
    tl.ReLU(), # Activate the hidden layer
    tl.Linear(9),  # Output for target linear output
)

net = tl.build(net_arch, torch.randn(1, *X_train[0].shape))
model = NeuralNetRegressor(
    net,
    max_epochs=10,
    lr=0.01,
)

model.fit(X_train, y_train)
y_proba = model.predict(X_test)
y_proba[0]
plt.plot(y_pred[0], label='Predicted velocity')
plt.plot(y_test[0], label='True velocity')
plt.legend()<jupyter_output><empty_output><jupyter_text>Model persistenceThe easiest way to save a model is to `pickle` the trained model object.<jupyter_code>import pickle

# saving
with open('torch_regressor.pkl', 'wb') as f:
    pickle.dump(model, f)

# loading
with open('torch_regressor.pkl', 'rb') as f:
    model = pickle.load(f)
model.predict(X_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10917335193582142, 'fraction_numerical': 0.008719916288803628, 'mean_word_length': 5.197406340057637, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 66, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '150690', 'license': 'Apache-2.0', 'n_tokens_mistral': 3022, 'n_tokens_neox': 2920, 'n_words': 714, 'path': 'notebooks/Regression_with_PyTorch.ipynb', 'repo_name': 'seismic-shift/geoml', 'size': '8607'}