<jupyter_start><jupyter_text>**Tutorial on Text as Data**: Deep Learning Source:  [https://github.com/d-insight/code-bank.git](https://github.com/d-insight/code-bank.git)  License: [MIT License](https://opensource.org/licenses/MIT). See open source [license](LICENSE) in the Code Bank repository.  --- For the next section, we will use [Tensorflow Keras](https://www.tensorflow.org/guide/keras/sequential_model), a well known Framework for Deep Learning. Training neural networks, however, if is computationally very expensive and therefore a GPU processor is usually required. You will therefore want to upload this notebook to a virtual macine with access to GPUs. Google provides "Colba" - a free and easy way to get access to a GPU... [Colab](https://colab.research.google.com/) is a tool from Google that allows you to share and use a special (Google) type of Jupyter Notebooks on the cloud. It also allows you to use a computer with GPUs for free. You do not need a Google Account to __see__ a notebook, but you will need a Google account (such as a Gmail account) if you want to actually execute code.To continue with this tutorial, upload the next notebook to Google Colab and continue with it there...**Activate GPU:**The default hardware of Google Colab is CPU. So once you have uploaded this notebook and are ready with CoLab, you need to activate GPU. You can enable GPU by clicking on: "Runtime" > "Change runtime type" > "Hardware accelerator" and then select "gpu". ---  Part 1: Simple Bayes (TFIDF + Bayes)<jupyter_code>import pandas as pd

df = pd.read_csv("https://github.com/dsfm-epfl/data/raw/master/text-as-data/yelp_review.csv")

print(f"Shape: {df.shape}")

df.head(2)<jupyter_output>Shape: (150000, 3)<jupyter_text>Generate the target and split the DataFrame into a `train_df` (80%) and `test_df` (20%).<jupyter_code>df['target'] = (df['stars'] > 4) * 1

from sklearn.model_selection import train_test_split
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)<jupyter_output><empty_output><jupyter_text>Fit a ` TfidfVectorizer` and prepare `X_train`, `X_test`, `Y_train` and `y_test`<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vectorizer = TfidfVectorizer(max_features=500)

X_train = tfidf_vectorizer.fit_transform(train_df['text']).toarray()
X_test = tfidf_vectorizer.transform(test_df['text']).toarray()

y_train = train_df['target'].to_numpy()
y_test = test_df['target'].to_numpy()<jupyter_output><empty_output><jupyter_text>Train a Bayes model and print accuracy. Our next goal will be to beat such baseline (already good).<jupyter_code>from sklearn.naive_bayes import GaussianNB
import numpy as np

gnb = GaussianNB()
y_predicted = gnb.fit(X_train, y_train).predict(X_test)

from sklearn.metrics import accuracy_score
print(accuracy_score(y_test, y_predicted))<jupyter_output>0.7687333333333334<jupyter_text>Part 2: Neural NetworksFor this task we will use [Tensorflow Keras](https://www.tensorflow.org/guide/keras/sequential_model), a very well known Framework for Deep Learning.<jupyter_code>import tensorflow as tf

from tensorflow.keras import layers
from tensorflow.keras import losses
from tensorflow.keras import preprocessing
from tensorflow.keras.preprocessing.text import Tokenizer<jupyter_output><empty_output><jupyter_text>Q1: Our first task consist in creating (again) the `X_train` and `X_test` features. This time, instead of using `TfidfVectorizer` we make use [Tokenizer](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer) directly from Keras.`texts_to_sequences` both tokenize the text and map every token to a number using a dictionary.<jupyter_code>sentences_train = train_df['text']
sentences_test = test_df['text']

tokenizer = Tokenizer(num_words=5000)
tokenizer.fit_on_texts(sentences_train)

X_train = tokenizer.texts_to_sequences(sentences_train)
X_test = tokenizer.texts_to_sequences(sentences_test)

y_train = train_df['target'].to_numpy()
y_test = test_df['target'].to_numpy()

vocab_size = len(tokenizer.word_index) + 1  # Adding 1 because of reserved 0 index
print(f"Vocab size: {vocab_size}")<jupyter_output>Vocab size: 95282<jupyter_text>Q2: Display the first two values of `X_train`. What does they look like? What's the problem we are facing now?<jupyter_code>print(X_train[:2])<jupyter_output>[[104, 42, 20, 1, 1682, 7, 4, 316, 2, 3, 74, 729, 151, 13, 15, 6, 47, 7, 1, 85, 297, 1256, 11, 177, 3, 23, 1, 516, 2682, 997, 17, 25, 73, 44, 516, 18, 87, 22, 2372, 175, 40, 42, 6, 225, 14, 23, 4, 321, 478, 2, 144, 6, 262, 59, 31, 51], [3, 863, 3584, 1588, 75, 3, 142, 4, 320, 544, 7, 1, 297, 1207, 510, 700, 3, 21, 650, 24, 186, 29, 249, 60, 15, 1588, 2, 204, 5, 160, 8, 4, 106, 1, 40, 6, 525, 2, 1, 28, 6, 34, 784, 3, 59, 99, 246, 15, 102, 109, 3, 6, 278, 336, 2, 52, 22, 1819, 15, 102, 11, 12, 260, 358]]<jupyter_text>**Answer**Unlike the TF-IDF matrix, every row of the `X_train` has a different size. We will need to pad every sequences with zeroes so that they have all the same size. Q3: Pad every sequence using a maxlen of 100 and look at the shape of `X_train`<jupyter_code>from tensorflow.keras.preprocessing.sequence import pad_sequences

maxlen = 100

X_train = pad_sequences(X_train, padding='post', maxlen=maxlen)
X_test = pad_sequences(X_test, padding='post', maxlen=maxlen)

print(X_train.shape)<jupyter_output>(120000, 100)<jupyter_text>Q4: We will now define a Neural Network model, we train it and look at the accuracy, did we improved from the baseline?<jupyter_code>import tensorflow

vocab_size = len(tokenizer.word_index) + 1

model = tensorflow.keras.Sequential([
  layers.Embedding(input_dim=vocab_size, output_dim=150, input_length=maxlen),
  layers.Dropout(0.2),
  layers.GlobalAveragePooling1D(),
  layers.Dropout(0.2),
  layers.Dense(1)])

model.summary()
model.compile(
    loss=losses.BinaryCrossentropy(from_logits=True),
    optimizer='adam', 
    metrics=tf.metrics.BinaryAccuracy(threshold=0.0)
)
# fit = start training
history = model.fit(X_train, y_train,
                    epochs=5,
                    verbose=True,
                    validation_data=(X_test, y_test),
                    batch_size=256)

loss, accuracy = model.evaluate(X_train, y_train, verbose=True)

print("Training Accuracy: {:.4f}".format(accuracy))
loss, accuracy = model.evaluate(X_test, y_test, verbose=True)

print("Testing Accuracy:  {:.4f}".format(accuracy))
import matplotlib.pyplot as plt
plt.style.use('ggplot')

def plot_history(history):
    acc = history.history['binary_accuracy']
    val_acc = history.history['val_binary_accuracy']
    loss = history.history['loss']
    val_loss = history.history['val_loss']
    x = range(1, len(acc) + 1)

    plt.figure(figsize=(12, 5))
    plt.subplot(1, 2, 1)
    plt.plot(x, acc, 'b', label='Training acc')
    plt.plot(x, val_acc, 'r', label='Validation acc')
    plt.title('Training and validation accuracy')
    plt.legend()
    plt.subplot(1, 2, 2)
    plt.plot(x, loss, 'b', label='Training loss')
    plt.plot(x, val_loss, 'r', label='Validation loss')
    plt.title('Training and validation loss')
    plt.legend()

plot_history(history)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11997775615181426, 'fraction_numerical': 0.0506047546225497, 'mean_word_length': 5.233968804159446, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 35, 'https://': 8, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '548925', 'license': 'MIT', 'n_tokens_mistral': 2810, 'n_tokens_neox': 2438, 'n_words': 733, 'path': 'tutorials/text-as-data/5-deep-learning-colab.ipynb', 'repo_name': 'dsfm-org/code-bank', 'size': '7193'}