<jupyter_start><jupyter_text>Classify different data sets  Basic includes<jupyter_code># Using pandas to load the csv file
import pandas as pd

import numpy as np
import matplotlib.pyplot as plt

from keras import models
from keras import optimizers
from keras import layers 
from keras import callbacks
from keras.utils import to_categorical

# reuters and fashin mnist data set from keras
from keras.datasets import reuters
from keras.datasets import fashion_mnist

# needed to preprocess text
from keras.preprocessing.text import Tokenizer<jupyter_output>Using TensorFlow backend.<jupyter_text>Classify the Fashion Mnist---<jupyter_code>#Get data
(fashion_train_data_raw, fashion_train_labels_raw), (fashion_test_data_raw, fashion_test_labels_raw) = fashion_mnist.load_data()

# These are the real labels, for display purposes only
label_names = [
    'T-shirt/top',
    'Trouser',
    'Pullover',
    'Dress',
    'Coat',
    'Sandal',
    'Shirt',
    'Sneaker',
    'Bag',
    'Ankle boot',
]

# Let's see the size
print(fashion_train_data_raw.shape)


# Now print a sample!
test_index = 10

plt.title("Label: " + label_names[fashion_train_labels_raw[test_index]])
plt.imshow(fashion_train_data_raw[test_index], cmap="gray")<jupyter_output>(60000, 28, 28)<jupyter_text>Prepare data<jupyter_code># Reshaping to 1D array
fashion_train_data = fashion_train_data_raw.reshape((fashion_train_data_raw.shape[0], 28*28))/255
fashion_test_data  = fashion_test_data_raw.reshape((fashion_test_data_raw.shape[0], 28*28))/255

# One hot encodings
fashion_train_labels = to_categorical(fashion_train_labels_raw)
fashion_test_labels  = to_categorical(fashion_test_labels_raw)
# Model Architecture
fashion_network = models.Sequential()
fashion_network.add(layers.Dense(2048, activation='relu', input_shape=(28 * 28,)))
fashion_network.add(layers.Dropout(0.3))
fashion_network.add(layers.Dense(128, activation='relu'))
fashion_network.add(layers.Dropout(0.3))
fashion_network.add(layers.Dense(10, activation='softmax'))

fashion_network.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

fashion_network.summary()
early_stop = callbacks.EarlyStopping(monitor='val_loss', patience=5)
# Training
history = fashion_network.fit(
    fashion_train_data,
    fashion_train_labels,
    batch_size=6000,
    epochs=200,
    validation_split=0.2,
    callbacks=[early_stop], 
    #verbose=2
)
# Evaluate againt test dataset
results = fashion_network.evaluate(fashion_test_data, fashion_test_labels)
print(results)
print(f"Accuracy: {'{0:.2f}'.format(results[1]*100)}%")
# Make Loss and Acc plots
history_dict = history.history
print(history_dict.keys())
acc = history_dict['acc']
val_acc = history_dict['val_acc']
loss = history_dict['loss']
val_loss = history_dict['val_loss']

epochs = range(1, len(acc) + 1)

f, (ax_loss, ax_acc) = plt.subplots(1,2, figsize=(18,8))

ax_loss.plot(epochs, loss, 'bo', label='Training loss')
ax_loss.plot(epochs, val_loss, 'b', label='Validation loss')
ax_loss.set_title('Training and validation loss')
ax_loss.set_xlabel('Epochs')
ax_loss.set_ylabel('Loss')
ax_loss.legend()

ax_acc.plot(epochs, acc, 'bo', label='Training Accuracy')
ax_acc.plot(epochs, val_acc, 'b', label='Validation Accuracy')
ax_acc.set_title('Training and validation accuracy')
ax_acc.set_xlabel('Epochs')
ax_acc.set_ylabel('Accuracy')
ax_acc.legend()

plt.show()<jupyter_output>dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])<jupyter_text>Prediction example<jupyter_code>import random
predict_indexes = [random.randint(0, fashion_test_data.shape[0]) for _ in range(16)]
predict_data = np.array([fashion_test_data[i] for i in predict_indexes])
predictions = fashion_network.predict(predict_data)
predictions = np.argmax(predictions, axis=1)

f, plots = plt.subplots(4,4, figsize=(15,15))

for index, prediction, plot in zip(predict_indexes, predictions, plots.flatten()):
    real = fashion_test_labels_raw[index]
    plot.set_title(f"R: {label_names[real]} - P: {label_names[int(prediction)]}")
    plot.imshow(fashion_test_data_raw[index], cmap='gray')<jupyter_output><empty_output><jupyter_text>Final ThoughtsThis model was faily simple to train, the extensive dataset is certainly a key factor for this. There's a moderately healthy ratio between the dataset size and number of inputs and classes. I started with a model similar to the ones I've seen to be successfull for MNIST since on principle both datasets are very similar, however, I slowly migrated to a model that is heavy on the input and lighter towards the output, I _suppose_ this structure might promote recognition of a lot of patterns, which are then correlated in further layers to finally be narrowed down to the output classes, however, this is just a _possible_ explanation of why this appears to work.I the end, I have a 784>2048>128>10 architecture trained to produce a 89.61% accuracy with the test dataset.  TO DO: Preprocess the data1. Normalize the input data set2. Perform one hot encoding3. Create a train, test, and validation set  TO DO: Define and train a network, then plot the accuracy of the training, validation, and testing1. Use a validation set2. Propose and train a network3. Print the history of the training4. Evaluate with a test set  Classifying newswires---Build a network to classify Reuters newswires into 46 different mutually-exclusive topics.  Load and review the data<jupyter_code># Skip the N most common words that might not mean much
skip_first = 150

# How many words to take
num_words  = 2000

(reuters_train_data_raw, reuters_train_labels_raw), \
    (reuters_test_data_raw, reuters_test_labels_raw) = \
    reuters.load_data(
        num_words=num_words,
        test_split=0.1,
        skip_top=skip_first
    )

print(reuters_train_data_raw.shape)
print(reuters_train_labels_raw.shape)
print(reuters_train_data_raw[0])
print(reuters_train_labels_raw[0])

print(set(reuters_train_labels_raw))<jupyter_output>(10105,)
(10105,)
[2, 2, 2, 2, 2, 2, 447, 2, 2, 207, 270, 2, 2, 2, 2, 369, 186, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 482, 2, 2, 2, 2, 2, 2, 864, 2, 209, 154, 2, 151, 2, 2, 2, 2, 2, 155, 2, 2, 2, 2, 2, 2, 1005, 504, 2, 258, 2, 272, 2, 2, 2, 2, 2, 2, 2, 2, 2, 197, 1245, 2, 2, 2, 2, 209, 2, 2, 2, 2, 2, 2, 2, 2]
3
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45}<jupyter_text>Load the word index to decode the train data.<jupyter_code>word_index = reuters.get_word_index()

reverse_index = {value+3:key for key, value in word_index.items()}
print(len(reverse_index))
## Let's have a look at the first common words, maybe they're useless 
#print(sorted([(k,v) for k,v in reverse_index.items()], key=lambda x:x[0])[:200])

reverse_index[0] = "<PAD>"
reverse_index[1] = "<START>"
reverse_index[2] = "<UNKNOWN>"  # unknown
reverse_index[3] = "<UNUSED>"

# We're only showing the words that will actually make it to the NN
decoded_review = ' '.join([reverse_index.get(i,'?') for i in reuters_train_data_raw[0] if i > 3])

print(decoded_review)
# Tokenize the word sequences into matrices
tokenizer = Tokenizer(num_words=num_words)
reuters_train_data_token = tokenizer.sequences_to_matrix(reuters_train_data_raw, mode='binary')
reuters_test_data_token = tokenizer.sequences_to_matrix(reuters_test_data_raw, mode='binary')
print(reuters_train_data_token.shape)
print(reuters_test_data_token.shape)
# One hot encoding
reuters_one_hot_train_labels = to_categorical(reuters_train_labels_raw)
reuters_one_hot_test_labels = to_categorical(reuters_test_labels_raw)
print(reuters_one_hot_train_labels.shape)
print(reuters_one_hot_test_labels.shape)
# Model Architecture
reuters_model = models.Sequential()
reuters_model.add(layers.Dense(2048, activation='relu', input_dim=num_words))
reuters_model.add(layers.Dropout(0.8))
reuters_model.add(layers.Dense(2048, activation='relu'))
reuters_model.add(layers.Dropout(0.5))
reuters_model.add(layers.Dense(46, activation='softmax'))
reuters_model.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['accuracy'])
reuters_model.summary()
early_stop = callbacks.EarlyStopping(monitor='val_loss', patience=5)
# Training
history = reuters_model.fit(
    reuters_train_data_token,
    reuters_one_hot_train_labels,
    #batch_size=4096,
    epochs=30,
    validation_split=0.1,
    callbacks=[early_stop], 
    verbose=1
)
# Evaluate againt test dataset
results = reuters_model.evaluate(reuters_test_data_token, reuters_one_hot_test_labels)
print(results)
print(f"Accuracy: {'{0:.2f}'.format(results[1]*100)}%")
# Make Loss and Acc plots
history_dict = history.history
print(history_dict.keys())
acc = history_dict['acc']
val_acc = history_dict['val_acc']
loss = history_dict['loss']
val_loss = history_dict['val_loss']

epochs = range(1, len(acc) + 1)

f, (ax_loss, ax_acc) = plt.subplots(1,2, figsize=(18,8))

ax_loss.plot(epochs, loss, 'bo', label='Training loss')
ax_loss.plot(epochs, val_loss, 'b', label='Validation loss')
ax_loss.set_title('Training and validation loss')
ax_loss.set_xlabel('Epochs')
ax_loss.set_ylabel('Loss')
ax_loss.legend()

ax_acc.plot(epochs, acc, 'bo', label='Training Accuracy')
ax_acc.plot(epochs, val_acc, 'b', label='Validation Accuracy')
ax_acc.set_title('Training and validation accuracy')
ax_acc.set_xlabel('Epochs')
ax_acc.set_ylabel('Accuracy')
ax_acc.legend()

plt.show()<jupyter_output>dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])<jupyter_text>Predictions example<jupyter_code>import random
predict_indexes = [random.randint(0, reuters_test_data_raw.shape[0]) for _ in range(5)]
predict_data = np.array([reuters_test_data_token[i] for i in predict_indexes])
predictions = reuters_model.predict(predict_data)
predictions = np.argmax(predictions, axis=1)

ok = 0
for index, prediction in zip(predict_indexes, predictions):
    real = reuters_test_labels_raw[index]
    ok += 1 if real == prediction else 0
    print(f"Prediction:\nReal:{real} - Predicted: {prediction}")
    print(' '.join([reverse_index.get(i,'?') for i in reuters_train_data_raw[index][:30] if i > 3]) + "...")

print(f"\nAccuracy for the samples: {'{0:.2f}'.format(100*ok/float(len(predict_indexes)))}%")<jupyter_output>Prediction:
Real:13 - Predicted: 13
gross domestic product growth likely between head official institute economic...
Prediction:
Real:19 - Predicted: 19
court decision law aimed companies out state businesses high...
Prediction:
Real:3 - Predicted: 3
general completed purchase products department air products chemicals terms disclosed department makes...
Prediction:
Real:13 - Predicted: 13
99 given given nine mths 35 given given...
Prediction:
Real:4 - Predicted: 4
brazilian workers voted strike labour failure government's anti inflation plan...

Accuracy for the samples: 100.00%<jupyter_text>Final ThoughtsThis dataset was notably harder to fine tune and never managed to get it over 80% without significant overfitting. Two of the most prominent reasons I consider significant to this is the size of the dataset (it being relatively small at a total size of 11227 data points) and the ratio of the size to the amount of inputs and outputs (46,  way more compared with the IMDB dataset). To couteract these problems I figured filtering the inputs to discard common and meaningless words, such as 'the', 'he', 'she', 'as', etc., could significantly reduce the useless noice in the input, I was able to improve results by filtering the top 150 words. This approach could be taken further by adding another layer of text processing that can determine what words are really important in terms of conveying content, topic and data, and thus, only feeding this network words that have a stonger correlation with the genre.Given the complexity of the data and the grat number of classes I felt it necessary to try a big network, after trying several combinations 2048 seemed the best number for which the increase in complexity yielded sufficient improvement to be worth it.  TO DO: Preprocess the data1. Normalize the input data set2. Perform one hot encoding3. Create a train, test, and validation set  TO DO: Define and train a network, then plot the accuracy of the training, validation, and testing1. Use a validation set2. Propose and train a network3. Print the history of the training4. Evaluate with a test set  Predicting Student Admissions---Predict student admissions based on three pieces of data:- GRE Scores- GPA Scores- Class rank  Load and visualize the data<jupyter_code>student_data = pd.read_csv("student_data.csv")
print(student_data)<jupyter_output>admit  gre   gpa  rank
0        0  380  3.61     3
1        1  660  3.67     3
2        1  800  4.00     1
3        1  640  3.19     4
4        0  520  2.93     4
5        1  760  3.00     2
6        1  560  2.98     1
7        0  400  3.08     2
8        1  540  3.39     3
9        0  700  3.92     2
10       0  800  4.00     4
11       0  440  3.22     1
12       1  760  4.00     1
13       0  700  3.08     2
14       1  700  4.00     1
15       0  480  3.44     3
16       0  780  3.87     4
17       0  360  2.56     3
18       0  800  3.75     2
19       1  540  3.81     1
20       0  500  3.17     3
21       1  660  3.63     2
22       0  600  2.82     4
23       0  680  3.19     4
24       1  760  3.35     2
25       1  800  3.66     1
26       1  620  3.61     1
27       1  520  3.74     4
28       1  780  3.22     2
29       0  520  3.29     1
..     ...  ...   ...   ...
367      1  540  3.77     2
368      1  680  3.76     3
369      1  680  2.42     1
370      1  620  3.3[...]<jupyter_text>Plot of the GRE and the GPA from the data.<jupyter_code>X = np.array(student_data[["gre","gpa"]])
y = np.array(student_data["admit"])
admitted = X[np.argwhere(y==1)]
rejected = X[np.argwhere(y==0)]
plt.scatter([s[0][0] for s in rejected], [s[0][1] for s in rejected], s = 25, color = 'red', edgecolor = 'k')
plt.scatter([s[0][0] for s in admitted], [s[0][1] for s in admitted], s = 25, color = 'cyan', edgecolor = 'k')
plt.xlabel('Test (GRE)')
plt.ylabel('Grades (GPA)')

plt.show()<jupyter_output><empty_output><jupyter_text>Plot of the data by class rank.<jupyter_code>f, plots = plt.subplots(2, 2, figsize=(20,10))
plots = [plot for sublist in plots for plot in sublist]

for idx, plot in enumerate(plots):
    data_rank = student_data[student_data["rank"]==idx+1]
    plot.set_title("Rank " + str(idx+1))
    X = np.array(data_rank[["gre","gpa"]])
    y = np.array(data_rank["admit"])
    admitted = X[np.argwhere(y==1)]
    rejected = X[np.argwhere(y==0)]
    plot.scatter([s[0][0] for s in rejected], [s[0][1] for s in rejected], s = 25, color = 'red', edgecolor = 'k')
    plot.scatter([s[0][0] for s in admitted], [s[0][1] for s in admitted], s = 25, color = 'cyan', edgecolor = 'k')
    plot.set_xlabel('Test (GRE)')
    plot.set_ylabel('Grades (GPA)')
    
# Normalize both GPA and GRE usig a normal distribution
gpa_mean = student_data['gpa'].mean()
gpa_stdv = student_data['gpa'].std()

gre_mean = student_data['gre'].mean()
gre_stdv = student_data['gre'].std()

student_data['gpa'] -= gpa_mean
student_data['gpa'] /= gpa_stdv

student_data['gre'] -= gre_mean
student_data['gre'] /= gre_stdv
# Create a copy of the array
student_values = student_data.values
# Shuffle the new array
np.random.shuffle(student_values)

# Separate GRE and GPA
s_data = student_values[:,1:3]
# Convert the rank into classes
ranks = to_categorical(student_values[:,3]-1)
# Joing GRE, GPA and rank classes
s_data = np.concatenate((s_data, ranks), axis=1)

# One hot labels
s_labels = to_categorical(student_values[:,0:1])


print(s_data.shape)
print(s_labels.shape)
# Ratio for test dataset separation
test_ratio = 0.20

# Separate train and test datasets
threshold = round(s_data.shape[0] * test_ratio)
s_train_data, s_train_labels = s_data[:s_data.shape[0]-threshold], s_labels[:s_data.shape[0]-threshold]
s_test_data, s_test_labels = s_data[-threshold:], s_labels[-threshold:]

print(s_train_data.shape, s_train_labels.shape)
print(s_test_data.shape, s_test_labels.shape)
# Model Architecture
students_model = models.Sequential()
students_model.add(layers.Dense(4, activation='relu', input_dim=s_data.shape[1]))
students_model.add(layers.Dropout(0.1))
students_model.add(layers.Dense(2, activation='softmax'))
students_model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
students_model.summary()
early_stop = callbacks.EarlyStopping(monitor='val_loss', patience=20)
# Training
history = students_model.fit(
    s_train_data, s_train_labels,
    #batch_size=10000,
    epochs=500,
    validation_split=0.3,
    callbacks=[early_stop], 
    verbose=2
)
# Evaluate againt test dataset
results = students_model.evaluate(s_test_data, s_test_labels)
print(results)
print(f"Accuracy: {'{0:.2f}'.format(results[1]*100)}%")
# Make Loss and Acc plots
history_dict = history.history
print(history_dict.keys())
acc = history_dict['acc']
val_acc = history_dict['val_acc']
loss = history_dict['loss']
val_loss = history_dict['val_loss']

epochs = range(1, len(acc) + 1)

f, (ax_loss, ax_acc) = plt.subplots(1,2, figsize=(18,8))

ax_loss.plot(epochs, loss, 'bo', label='Training loss')
ax_loss.plot(epochs, val_loss, 'b', label='Validation loss')
ax_loss.set_title('Training and validation loss')
ax_loss.set_xlabel('Epochs')
ax_loss.set_ylabel('Loss')
ax_loss.legend()

ax_acc.plot(epochs, acc, 'bo', label='Training Accuracy')
ax_acc.plot(epochs, val_acc, 'b', label='Validation Accuracy')
ax_acc.set_title('Training and validation accuracy')
ax_acc.set_xlabel('Epochs')
ax_acc.set_ylabel('Accuracy')
ax_acc.legend()

plt.show()<jupyter_output>dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])<jupyter_text>TO DO: Preprocess the data1. Normalize the input data set2. Perform one hot encoding3. Create a train, test, and validation set  TO DO: Define and train a network, then plot the accuracy of the training, validation, and testing1. Use a validation set2. Propose and train a network3. Print the history of the training4. Evaluate with a test set  Prediction examples<jupyter_code>import random
predict_indexes = [random.randint(0, s_test_data.shape[0]-1) for _ in range(5)]
predict_data = np.array([s_test_data[i] for i in predict_indexes])
predictions = students_model.predict(predict_data)
predictions = np.argmax(predictions, axis=1)

ok = 0
for index, prediction in zip(predict_indexes, predictions):
    predicition = np.argmax(prediction)
    real = np.argmax(s_test_labels[index])
    ok += 1 if real == prediction else 0
    print(f"Real: {real} - Predicted: {prediction}")
    print(f"GPA: {'{0:.2f}'.format(s_test_data[index][0]*gpa_stdv + gpa_mean)}")
    print(f"GRE: {'{0:.2f}'.format(s_test_data[index][1]*gre_stdv + gre_mean)}")
    print(f"Rank: {np.argmax(s_test_data[index][2:])+1}\n")

print(f"\nAccuracy for the samples: {'{0:.2f}'.format(100*ok/float(len(predict_indexes)))}%")<jupyter_output>Real: 0 - Predicted: 0
GPA: 3.10
GRE: 462.39
Rank: 3

Real: 1 - Predicted: 0
GPA: 3.76
GRE: 638.92
Rank: 2

Real: 0 - Predicted: 0
GPA: 3.10
GRE: 477.61
Rank: 3

Real: 1 - Predicted: 1
GPA: 4.09
GRE: 681.53
Rank: 1

Real: 0 - Predicted: 0
GPA: 3.30
GRE: 322.39
Rank: 2


Accuracy for the samples: 80.00%
{'fraction_non_alphanumeric': 0.12023049369257124, 'fraction_numerical': 0.04900586616830192, 'mean_word_length': 4.932861102556206, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '538792', 'license': 'MIT', 'n_tokens_mistral': 7455, 'n_tokens_neox': 6733, 'n_words': 1917, 'path': 'Keras_assignment.ipynb', 'repo_name': 'DeepLearningVision-2019/a4-keras-classification-franspaco', 'size': '19263'}