<jupyter_start><jupyter_text>We're going to use PyViz's [Bokeh](https://bokeh.pydata.org/en/latest/) to build interactive plots in this notebook. Bokeh uses a js kernel to serve data to the browser, so we need to initialize it.<jupyter_code>output_notebook()<jupyter_output><empty_output><jupyter_text>Import data and shape it for MLLet's start by importing the .csv we wrote out in `data_prep.ipynb`. We'll then shuffle the data **(WHY?)**. Then we'll peel the labels off of the features **(AGAIN, WHY?)**.<jupyter_code># read in .csv file as a Pandas DataFrame
rocks = pd.read_csv('../dat/csv/clean_rocks.csv')

# break out the names of the features and rocks for plotting
chemical_names = list(rocks.columns)
rock_names     = ['andesite', 'basalt', 'carbonatite', 'kimberlite', 'rhyolite']

# cast the Pandas dataframe as an array, for use with PyTorch
rocks = rocks.as_matrix()

print('import shape:', rocks.shape, '\n')

# randomly shuffle the samples (rows) so we don't overcondition
# the network by training on one rock type at a time
np.random.shuffle(rocks)

# split off the labels from the features for training
ftrs = rocks[:, :-1]
lbls = rocks[:, -1]

print('features matrix shape:', ftrs.shape)
print('labels matrics shape:', lbls.shape)<jupyter_output>import shape: (36232, 144) 

features matrix shape: (36232, 143)
labels matrics shape: (36232,)<jupyter_text>Next we'll split the data up into training and testing datasets. We'll train only on the training split, and hold the testing split out blind. When the model is trained, we'll expose the test data to the network to see if it has really learned anything. If the network has learned (to generalize), it should be able to predict the rock type for samples of chemical compositions it's never seen before.<jupyter_code>train_percentage = 0.9
train_num = int(np.round(rocks.shape[0] * train_percentage))

train_dat = ftrs[:train_num, :]
train_lbl = lbls[:train_num]

test_dat  = ftrs[train_num:, :]
test_lbl  = lbls[train_num:]
print('train_dat shape:', train_dat.shape)
print('train_lbl shape:', train_lbl.shape)
print('test_dat shape:', test_dat.shape)
print('test_lbl shape:', test_lbl.shape)<jupyter_output>train_dat shape: (32609, 143)
train_lbl shape: (32609,)
test_dat shape: (3623, 143)
test_lbl shape: (3623,)<jupyter_text>Build neural networkThis model is slightly different from the one in the regression task of notebook `2.0-Simple-Neural-Network.ipynb` for two reasons:1. The input and output of this network has mulitiple features, defined by `input_size` and `num_classes`. This is because we have a multifeature input, or "feature vector." In the regression task, we input one value, and expected one output value. In this task, we input as many values as we have chemical composition values. 2. The output of the model is a 5 element vector which represents the probability of the input sample corresponding to each of the 5 "classes," or rock types. The `nn.Softmax` layer on the back of the model calculates these probabilities. Instead of "regressing" one input x value to one output y value, we "classify" one feature vector to one class.We've also included a set of commented out layers. By adding these layers back in at home, you'll add more depth to your network, and increase the accuracy of your predictions. The cost of calculating and backpropagating gradients increases dramatically, so don't expect to be able to train it in an hour on a single thread.<jupyter_code>class FirstNet(torch.nn.Module):
    def __init__(self, input_size, num_classes):
        super(FirstNet, self).__init__()
        self.fc1  = torch.nn.Linear(input_size, 3000)
        self.relu1 = torch.nn.ReLU()
        #self.fc2  = torch.nn.Linear(3000, 2000)
        #self.relu2 = torch.nn.ReLU()
        #self.fc3  = torch.nn.Linear(2000, 1000)
        #self.relu3 = torch.nn.ReLU()
        self.fc4  = torch.nn.Linear(3000, num_classes)
        self.soft = torch.nn.Softmax(dim=1)
    
    def forward(self, x):
        out = self.fc1(x)
        out = self.relu1(out)
        #out = self.fc2(out)
        #out = self.relu2(out)
        #out = self.fc3(out)
        #out = self.relu3(out)
        out = self.fc4(out)
        out = self.soft(out)
        return out
    
net = FirstNet(input_size=train_dat.shape[1], num_classes=5)

print(net)<jupyter_output>FirstNet(
  (fc1): Linear(in_features=143, out_features=3000, bias=True)
  (relu1): ReLU()
  (fc4): Linear(in_features=3000, out_features=5, bias=True)
  (soft): Softmax()
)<jupyter_text>Test the ModelBelow, we'll run the data through the untrained model to see what the output looks like. First we check for a GPU:<jupyter_code>use_cuda = torch.cuda.is_available()
if use_cuda:
    net = net.cuda()<jupyter_output><empty_output><jupyter_text>Next we perform a forward pass through the untrained network. To do so, we must _(gasp, I thought this was Python)_ type the matrices as PyTorch tensors.<jupyter_code>if use_cuda:
    predictions = net.forward(torch.FloatTensor(train_dat).cuda()).cpu()
else:
    predictions = net.forward(torch.FloatTensor(train_dat))<jupyter_output><empty_output><jupyter_text>Now we plot the output from the untrained model against the groundtruth. You can click the labels in the legend to turn off the predictions or the true class values. If you click the mouse wheel control on the right side of the diagram, you can zoom and pan simultaneously.<jupyter_code># set up the plot
p1 = figure(y_range=rock_names, plot_width=900, plot_height=500, title="First 100 Rocks")
p1.title.text_font_size = '24pt'
p1.xaxis.axis_label = 'Sample Number'
p1.yaxis.axis_label = 'Rock type'
p1.xaxis.major_label_orientation = 1.57

# plot the rock type data
r1 = p1.circle(range(100), train_lbl[:100] + 0.5, fill_alpha=0.6, line_alpha=0.6, legend='groundtruth')
# plot the predictions from the network
r2 = p1.circle(range(100), np.argmax(predictions.data.numpy(), axis=1) + 0.5, fill_alpha=0.2, line_alpha=0.2, 
               fill_color='red', line_color='red', legend='prediction')

# set up the legend
p1.legend.location = "top_left"
p1.legend.click_policy="hide"

# show the plot inline
show(p1, notebook_handle=True)<jupyter_output>/home/gram/anaconda3/lib/python3.6/site-packages/bokeh/models/sources.py:114: BokehUserWarning: ColumnDataSource's columns must be of the same length. Current lengths: ('x', 100), ('y', 32609)
  "Current lengths: %s" % ", ".join(sorted(str((k, len(v))) for k, v in data.items())), BokehUserWarning))<jupyter_text>Train the ModelNow let's train the model. Even the tiny neural network we defined above will take longer than we have time to train during this class, but let's kick it off, watch the loss, and see if it's learning anything:<jupyter_code>%%time

# here, again we type the data and labels as tensors
train_dat = torch.FloatTensor(train_dat)
train_lbl = torch.LongTensor(train_lbl)

# define hyperparameters
learning_rate = 0.001
num_epochs = 30
loss_hist = []

# build a multiclass cross entropy loss function
criterion = torch.nn.CrossEntropyLoss()
# instantiate a stochastic gradient descent optimizer class
optimizer = torch.optim.SGD(net.parameters(), lr=learning_rate)
# set the model parameters for training mode
net.train()

# build a loss plot
p2 = figure(plot_width=900, plot_height=500)
r2 = p2.line(range(len(loss_hist)), loss_hist)
p2.legend.location = "top_left"
p2.legend.click_policy="hide"
loss_plot = show(p2, notebook_handle=True)

# send data to GPU, if appropriate
if use_cuda:
    criterion = criterion.cuda()
    train_dat = train_dat.cuda()
    train_lbl = train_lbl.cuda()
    
# train for many epochs
for epoch in tnrange(num_epochs):
    # forward pass through the model
    predictions = net.forward(train_dat)
    # calculate local value on the loss surface
    loss = criterion(predictions, train_lbl)

    # clear the gradient buffer
    optimizer.zero_grad()
    # backward pass through the model to calculate gradients
    loss.backward()
    # take one step towards a minimum in the loss surface
    optimizer.step()

    # replot the network loss for one epoch
    loss_hist.append(loss.data.numpy())
    r2 = p2.line(range(len(loss_hist)), loss_hist)
    push_notebook(handle=loss_plot)
    
# set the model parameters for inference mode
net.eval()<jupyter_output><empty_output><jupyter_text>Save Trained ModelAlways (always!) save your trained model weights. You'll thank yourself laters.<jupyter_code>import datetime

np.save('../dat/checkpoints/loss.npy', np.array(loss_hist), allow_pickle=False)

checkpoint_name = ''.join(('../dat/checkpoints/', str(datetime.datetime.now()).replace(' ','_'), '.bin'))
torch.save(net.cpu().state_dict(), checkpoint_name)<jupyter_output><empty_output><jupyter_text>Predict rock types on unseen data! Finally, we'll use our trained model to predict the rock type of data that the model has never seen and calculate its accuracy.First, we build the predictions:<jupyter_code>%%time

# here, again we type the data and labels as tensors
if use_cuda:
    predictions = net.forward(torch.FloatTensor(test_dat).cuda()).cpu()
else:
    predictions = net.forward(torch.FloatTensor(test_dat))<jupyter_output>CPU times: user 137 ms, sys: 11.1 ms, total: 148 ms
Wall time: 46.5 ms<jupyter_text>Next, we type convert our predictions back to a standard numpy array and evaluate the maximum probability for each class, elementwise (samplewise).<jupyter_code>predictions = np.argmax(predictions.data.numpy(), axis=1)<jupyter_output><empty_output><jupyter_text>Now that we have our predictions, we can calculate the accuracy simply by differencing the groudtruth with the predictions. If the prediction is different than the groundtruth, we assign an "incorrect". If the prediction is the same as the groudtruth we assign a "corrrect".<jupyter_code>accuracy = np.equal(predictions, test_lbl)
accuracy = np.round(100 * np.sum(accuracy) / predictions.shape[0])

print('total accuracy:', accuracy, '%')<jupyter_output>total accuracy: 32.0 %<jupyter_text>Now, let's analyze classwise (rock-type-wise) accuracy. We'll perform the same analysis as above, but we'll do it for each rock type to see if the model performs better on a certain rock type.<jupyter_code>for i, one_rock_type in enumerate(rock_names):
    idx = np.where(test_lbl == i)[0]

    accuracy = predictions[idx] == i
    accuracy = np.round(100 * np.sum(accuracy) / idx.shape[0])

    print('total accuracy on rock type {}:'.format(one_rock_type), accuracy, '%')
# set up the plot
p1 = figure(y_range=rock_names, plot_width=900, plot_height=500, title="Rock Type Predictions")
p1.title.text_font_size = '24pt'
p1.xaxis.axis_label = 'Sample Number'
p1.yaxis.axis_label = 'Rock type'
p1.xaxis.major_label_orientation = 1.57

# plot the rock type data
r1 = p1.circle(range(test_lbl.shape[0]), test_lbl + 0.5, fill_alpha=0.6, line_alpha=0.6, legend='groundtruth')
# plot the predictions from the network
r2 = p1.circle(range(test_lbl.shape[0]), predictions + 0.5, fill_alpha=0.2, line_alpha=0.2, 
               fill_color='red', line_color='red', legend='prediction')

# set up the legend
p1.legend.location = "top_left"
p1.legend.click_policy="hide"

# show the plot inline
show(p1, notebook_handle=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10714920860750489, 'fraction_numerical': 0.02276364929752801, 'mean_word_length': 4.842597402597403, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '46180', 'license': 'Apache-2.0', 'n_tokens_mistral': 3812, 'n_tokens_neox': 3612, 'n_words': 1188, 'path': 'nb/3.0-Simple-Neural-Network_realData.ipynb', 'repo_name': 'unjymslf/pytorch-geo-intro', 'size': '11246'}