<jupyter_start><jupyter_text>Classifying newswires 🗞<jupyter_code>data_length=10000<jupyter_output><empty_output><jupyter_text>Loading the Reuters datasetThe argument *num_words* means we’ll only keep the top *num_words* most frequently occurring words in the training data. Rare words will be discarded. This allows us to work with vector data of manageable size.<jupyter_code>from tensorflow.keras.datasets import reuters

(train_data, train_labels), (test_data, test_labels) = reuters.load_data(
    num_words=data_length)<jupyter_output><empty_output><jupyter_text>Preparing the dataWe will turn the lists into vectors of 0s and 1s. This would mean, for instance, turning the sequence [3, 5] into a 10,000-dimensional vector that would be all 0s except for indices 3 and 5, which would be 1s. Then we will use as the first layer in our network a Dense layer, capable of handling floating-point vector data.<jupyter_code>import numpy as np

def vectorize_sequences(sequences, dimension=10000):
    results = np.zeros((len(sequences), dimension))
    for i, sequence in enumerate(sequences):
        results[i, sequence] = 1.
    return results

x_train = vectorize_sequences(train_data)
x_test = vectorize_sequences(test_data)<jupyter_output><empty_output><jupyter_text>To vectorize the labels we'll use one-hot encoding which is a widely used format for categorical data, also called categorical encoding. In this case, one-hot encoding of the labels consists of embedding each label as an all-zero vector with a 1 in the place of the label index. There is a built-in way to do this in Keras.<jupyter_code>from tensorflow.keras.utils import to_categorical

one_hot_train_labels = to_categorical(train_labels)
one_hot_test_labels = to_categorical(test_labels)<jupyter_output><empty_output><jupyter_text>Building the networkIn a stack of Dense layers each layer can only access information present in the output of the previous layer. If one layer drops some information relevant to the classification problem, this information can never be recovered by later layers: each layer can potentially become an information bottleneck.<jupyter_code>from tensorflow.keras import models
from tensorflow.keras import layers

model = models.Sequential()
model.add(layers.Dense(64, activation='relu', input_shape=(data_length,)))
model.add(layers.Dense(64, activation='relu'))<jupyter_output><empty_output><jupyter_text>We end the network with a Dense layer of size 46. This means for each input sample, the network will output a 46-dimensional vector. Each entry in this vector (each dimension) will encode a different output class.<jupyter_code>model.add(layers.Dense(46, activation='softmax'))<jupyter_output><empty_output><jupyter_text>The last layer uses a softmax activation. It means the network will output a probability distribution over the 46 different output classes—for every input sample, the network will produce a 46-dimensional output vector, where output[i] is the probability that the sample belongs to class i. The 46 scores will sum to 1.  Compiling the modelThe best loss function to use in this case is *categorical_crossentropy*. It measures the distance between two probability distributions: here, between the probability distribution output by the network and the true distribution of the labels. By minimizing the distance between these two distributions, we train the network to output something as close as possible to the true labels.<jupyter_code>model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Setting aside a validation setLet’s set apart 1,000 samples in the training data to use as a validation set.<jupyter_code>x_val = x_train[:1000]
partial_x_train = x_train[1000:]

y_val = one_hot_train_labels[:1000]
partial_y_train = one_hot_train_labels[1000:]<jupyter_output><empty_output><jupyter_text>Training the model<jupyter_code>history = model.fit(partial_x_train,
                    partial_y_train,
                    epochs=20,
                    batch_size=512,
                    validation_data=(x_val, y_val))<jupyter_output>Train on 7982 samples, validate on 1000 samples
Epoch 1/20
7982/7982 [==============================] - 2s 188us/step - loss: 2.5032 - acc: 0.5357 - val_loss: 1.7086 - val_acc: 0.6490
Epoch 2/20
7982/7982 [==============================] - 1s 139us/step - loss: 1.4024 - acc: 0.7012 - val_loss: 1.3307 - val_acc: 0.7020
Epoch 3/20
7982/7982 [==============================] - 1s 130us/step - loss: 1.0612 - acc: 0.7642 - val_loss: 1.1501 - val_acc: 0.7500
Epoch 4/20
7982/7982 [==============================] - 1s 129us/step - loss: 0.8369 - acc: 0.8216 - val_loss: 1.0303 - val_acc: 0.7870
Epoch 5/20
7982/7982 [==============================] - 1s 131us/step - loss: 0.6600 - acc: 0.8601 - val_loss: 0.9699 - val_acc: 0.8000
Epoch 6/20
7982/7982 [==============================] - 1s 129us/step - loss: 0.5287 - acc: 0.8884 - val_loss: 0.9336 - val_acc: 0.8090
Epoch 7/20
7982/7982 [==============================] - 1s 135us/step - loss: 0.4233 - acc: 0.9117 - val_loss: 0.9108 - val_acc: 0.8190
[...]<jupyter_text>Plotting the training and validation loss<jupyter_code>import matplotlib.pyplot as plt

loss_values = history.history['loss']
val_loss_values = history.history['val_loss']

epochs = range(1, len(loss_values) + 1)

plt.plot(epochs, loss_values, 'bo', label='Training loss')
plt.plot(epochs, val_loss_values, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>Plotting the training and validation accuracy<jupyter_code>plt.clf()

acc_values = history.history['acc']
val_acc_values = history.history['val_acc']

plt.plot(epochs, acc_values, 'bo', label='Training acc')
plt.plot(epochs, val_acc_values, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>The network begins to overfit after nine epochs. At this point we should train the network from scratch for nine epochs and then evaluate it on the test set.  Retraining the model from scratch<jupyter_code>model = models.Sequential()
model.add(layers.Dense(64, activation='relu', input_shape=(data_length,)))
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(46, activation='softmax'))

model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
model.fit(partial_x_train,
          partial_y_train,
          epochs=9,
          batch_size=512,
          validation_data=(x_val, y_val))
results = model.evaluate(x_test, one_hot_test_labels)<jupyter_output>Train on 7982 samples, validate on 1000 samples
Epoch 1/9
7982/7982 [==============================] - 1s 147us/step - loss: 2.5951 - acc: 0.5088 - val_loss: 1.6864 - val_acc: 0.6280
Epoch 2/9
7982/7982 [==============================] - 1s 103us/step - loss: 1.4005 - acc: 0.7045 - val_loss: 1.3012 - val_acc: 0.7090
Epoch 3/9
7982/7982 [==============================] - 1s 97us/step - loss: 1.0442 - acc: 0.7769 - val_loss: 1.1242 - val_acc: 0.7600
Epoch 4/9
7982/7982 [==============================] - 1s 102us/step - loss: 0.8353 - acc: 0.8178 - val_loss: 1.0345 - val_acc: 0.7860
Epoch 5/9
7982/7982 [==============================] - 1s 100us/step - loss: 0.6636 - acc: 0.8587 - val_loss: 0.9670 - val_acc: 0.8000
Epoch 6/9
7982/7982 [==============================] - 1s 99us/step - loss: 0.5338 - acc: 0.8953 - val_loss: 0.9216 - val_acc: 0.8210
Epoch 7/9
7982/7982 [==============================] - 1s 115us/step - loss: 0.4325 - acc: 0.9107 - val_loss: 0.8898 - val_acc: 0.8220
Epoch 8/9[...]<jupyter_text>Generating predictions on new data<jupyter_code>predictions = model.predict(x_test)<jupyter_output><empty_output><jupyter_text>Each entry in predictions is a vector of length 46.<jupyter_code>predictions[0].shape<jupyter_output><empty_output><jupyter_text>The coefficients in this vector sum to 1.<jupyter_code>np.sum(predictions[0])<jupyter_output><empty_output><jupyter_text>The largest entry is the predicted class. The class with the highest probability.<jupyter_code>np.argmax(predictions[0])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1466730264726926, 'fraction_numerical': 0.06963987598378249, 'mean_word_length': 5.112973760932944, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 63, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '658574', 'license': 'Apache-2.0', 'n_tokens_mistral': 3130, 'n_tokens_neox': 2666, 'n_words': 788, 'path': 'multiclass_classification.ipynb', 'repo_name': 'stanete/notebooks', 'size': '8386'}