<jupyter_start><jupyter_text>Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.- Author: <NAME>- GitHub Repository: https://github.com/rasbt/deeplearning-models  Bidirectional Multi-layer RNN with LSTM with Own Dataset in CSV Format (AG News) Dataset Description```AG's News Topic Classification DatasetVersion 3, Updated 09/09/2015ORIGINAG is a collection of more than 1 million news articles. News articles have been gathered from more than 2000  news sources by ComeToMyHead in more than 1 year of activity. ComeToMyHead is an academic news search engine which has been running since July, 2004. The dataset is provided by the academic community for research purposes in data mining (clustering, classification, etc), information retrieval (ranking, search, etc), xml, data compression, data streaming, and any other non-commercial activity. For more information, please refer to the link http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html .The AG's news topic classification dataset is constructed by <NAME> (<EMAIL>) from the dataset above. It is used as a text classification benchmark in the following paper: <NAME>, <NAME>, <NAME>. Character-level Convolutional Networks for Text Classification. Advances in Neural Information Processing Systems 28 (NIPS 2015).DESCRIPTIONThe AG's news topic classification dataset is constructed by choosing 4 largest classes from the original corpus. Each class contains 30,000 training samples and 1,900 testing samples. The total number of training samples is 120,000 and testing 7,600.The file classes.txt contains a list of classes corresponding to each label.The files train.csv and test.csv contain all the training samples as comma-sparated values. There are 3 columns in them, corresponding to class index (1 to 4), title and description. The title and description are escaped using double quotes ("), and any internal double quote is escaped by 2 double quotes (""). New lines are escaped by a backslash followed with an "n" character, that is "\n".```<jupyter_code>!pip install -q IPython
!pip install -q ipykernel
!pip install -q watermark
!pip install -q matplotlib
!pip install -q sklearn
!pip install -q pandas
!pip install -q pydot
!pip install -q hiddenlayer
!pip install -q graphviz
!pip install -q gdown
%load_ext watermark
%watermark -a '<NAME>' -v -p torch


import torch
import torch.nn.functional as F
from torchtext import data
from torchtext import datasets
import time
import random
import pandas as pd
import numpy as np

torch.backends.cudnn.deterministic = True<jupyter_output><NAME> 

CPython 3.6.9
IPython 5.5.0

torch 1.5.1+cu101<jupyter_text>General Settings<jupyter_code>RANDOM_SEED = 123
torch.manual_seed(RANDOM_SEED)

VOCABULARY_SIZE = 5000
LEARNING_RATE = 1e-3
BATCH_SIZE = 128
NUM_EPOCHS = 50
DROPOUT = 0.5
DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

EMBEDDING_DIM = 128
BIDIRECTIONAL = True
HIDDEN_DIM = 256
NUM_LAYERS = 2
OUTPUT_DIM = 4<jupyter_output><empty_output><jupyter_text>Dataset The AG News dataset is available from Xiang Zhang's Google Drive folder athttps://drive.google.com/drive/u/0/folders/0Bz8a_Dbh9Qhbfll6bVpmNUtUcFdjYmF2SEpmZUZUcVNiMUw1TWN6RDV3a0JHT3kxLVhVR2MFrom the Google Drive folder, download the file - `ag_news_csv.tar.gz`<jupyter_code>!gdown -O ag_news_csv.tar.gz --id 0Bz8a_Dbh9QhbUDNpeUdjb0wxRms
!tar xvzf  ag_news_csv.tar.gz
!cat ag_news_csv/classes.txt<jupyter_output>World
Sports
Business
Sci/Tech<jupyter_text>Check that the dataset looks okay:<jupyter_code>df = pd.read_csv('ag_news_csv/train.csv', header=None, index_col=None)
df.columns = ['classlabel', 'title', 'content']
df['classlabel'] = df['classlabel']-1
df.head()
np.unique(df['classlabel'].values)
np.bincount(df['classlabel'])
df[['classlabel', 'content']].to_csv('ag_news_csv/train_prepocessed.csv', index=None)
df = pd.read_csv('ag_news_csv/test.csv', header=None, index_col=None)
df.columns = ['classlabel', 'title', 'content']
df['classlabel'] = df['classlabel']-1
df.head()
np.unique(df['classlabel'].values)
np.bincount(df['classlabel'])
df[['classlabel', 'content']].to_csv('ag_news_csv/test_prepocessed.csv', index=None)
del df<jupyter_output><empty_output><jupyter_text>Define the Label and Text field formatters:<jupyter_code>TEXT = data.Field(sequential=True,
                  tokenize='spacy',
                  include_lengths=True) # necessary for packed_padded_sequence

LABEL = data.LabelField(dtype=torch.float)


# If you get an error [E050] Can't find model 'en'
# you need to run the following on your command line:
#  python -m spacy download en<jupyter_output><empty_output><jupyter_text>Process the dataset:<jupyter_code>fields = [('classlabel', LABEL), ('content', TEXT)]

train_dataset = data.TabularDataset(
    path="ag_news_csv/train_prepocessed.csv", format='csv',
    skip_header=True, fields=fields)

test_dataset = data.TabularDataset(
    path="ag_news_csv/test_prepocessed.csv", format='csv',
    skip_header=True, fields=fields)<jupyter_output><empty_output><jupyter_text>Split the training dataset into training and validation:<jupyter_code>train_data, valid_data = train_dataset.split(
    split_ratio=[0.95, 0.05],
    random_state=random.seed(RANDOM_SEED))

print(f'Num Train: {len(train_data)}')
print(f'Num Valid: {len(valid_data)}')<jupyter_output>Num Train: 114000
Num Valid: 6000<jupyter_text>Build the vocabulary based on the top "VOCABULARY_SIZE" words:<jupyter_code>TEXT.build_vocab(train_data,
                 max_size=VOCABULARY_SIZE,
                 vectors='glove.6B.100d',
                 unk_init=torch.Tensor.normal_)

LABEL.build_vocab(train_data)

print(f'Vocabulary size: {len(TEXT.vocab)}')
print(f'Number of classes: {len(LABEL.vocab)}')
list(LABEL.vocab.freqs)[-10:]<jupyter_output><empty_output><jupyter_text>The TEXT.vocab dictionary will contain the word counts and indices. The reason why the number of words is VOCABULARY_SIZE + 2 is that it contains to special tokens for padding and unknown words: `` and ``. Make dataset iterators:<jupyter_code>train_loader, valid_loader, test_loader = data.BucketIterator.splits(
    (train_data, valid_data, test_dataset), 
    batch_size=BATCH_SIZE,
    sort_within_batch=True, # necessary for packed_padded_sequence
    sort_key=lambda x: len(x.content),
    device=DEVICE)<jupyter_output><empty_output><jupyter_text>Testing the iterators (note that the number of rows depends on the longest document in the respective batch):<jupyter_code>print('Train')
for batch in train_loader:
    print(f'Text matrix size: {batch.content[0].size()}')
    print(f'Target vector size: {batch.classlabel.size()}')
    break
    
print('\nValid:')
for batch in valid_loader:
    print(f'Text matrix size: {batch.content[0].size()}')
    print(f'Target vector size: {batch.classlabel.size()}')
    break
    
print('\nTest:')
for batch in test_loader:
    print(f'Text matrix size: {batch.content[0].size()}')
    print(f'Target vector size: {batch.classlabel.size()}')
    break<jupyter_output>Train
Text matrix size: torch.Size([35, 128])
Target vector size: torch.Size([128])

Valid:
Text matrix size: torch.Size([17, 128])
Target vector size: torch.Size([128])

Test:
Text matrix size: torch.Size([16, 128])
Target vector size: torch.Size([128])<jupyter_text>Model<jupyter_code>import torch.nn as nn


class RNN(nn.Module):
    def __init__(self, input_dim, embedding_dim, bidirectional, hidden_dim, num_layers, output_dim, dropout, pad_idx):
        
        super().__init__()
        
        self.embedding = nn.Embedding(input_dim, embedding_dim, padding_idx=pad_idx)
        self.rnn = nn.LSTM(embedding_dim, 
                           hidden_dim,
                           num_layers=num_layers,
                           bidirectional=bidirectional, 
                           dropout=dropout)
        self.fc1 = nn.Linear(hidden_dim * num_layers, 64)
        self.fc2 = nn.Linear(64, output_dim)
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, text, text_length):

        embedded = self.dropout(self.embedding(text))
        packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_length)
        packed_output, (hidden, cell) = self.rnn(packed_embedded)
        # output, output_lengths = nn.utils.rnn.pad_packed_sequence(packed_output)
        hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim=1))
        hidden = self.fc1(hidden)
        hidden = self.dropout(hidden)
        hidden = self.fc2(hidden)
        return hidden
INPUT_DIM = len(TEXT.vocab)

PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]

torch.manual_seed(RANDOM_SEED)
model = RNN(INPUT_DIM, EMBEDDING_DIM, BIDIRECTIONAL, HIDDEN_DIM, NUM_LAYERS, OUTPUT_DIM, DROPOUT, PAD_IDX)
model = model.to(DEVICE)
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
import hiddenlayer as hl
batch = next(iter(train_loader))
hl.build_graph(model, batch.content)<jupyter_output>/usr/local/lib/python3.6/dist-packages/torch/onnx/symbolic_opset9.py:1310: UserWarning: Dropout is a training op and should not be exported in inference mode. Make sure to call eval() on the model, and to export it with param training=False.
  warnings.warn("Dropout is a training op and should not be exported in inference mode. "
/usr/local/lib/python3.6/dist-packages/torch/onnx/symbolic_opset9.py:1586: UserWarning: Exporting a model to ONNX with a batch_size other than 1, with a variable length with LSTM can cause an error when running the ONNX model with a different batch size. Make sure to save the model with a batch size of 1, or define the initial states (h0/c0) as inputs of the model. 
  "or define the initial states (h0/c0) as inputs of the model. ")
/usr/local/lib/python3.6/dist-packages/torch/onnx/symbolic_helper.py:176: UserWarning: ONNX export failed on RNN/GRU/LSTM because dropout in training mode not supported
  warnings.warn("ONNX export failed on " + op + " because " + m[...]<jupyter_text>Training<jupyter_code>def compute_accuracy(model, data_loader, device):
    model.eval()
    correct_pred, num_examples = 0, 0
    with torch.no_grad():
        for batch_idx, batch_data in enumerate(data_loader):
            text, text_lengths = batch_data.content
            logits = model(text, text_lengths)
            _, predicted_labels = torch.max(logits, 1)
            num_examples += batch_data.classlabel.size(0)
            correct_pred += (predicted_labels.long() == batch_data.classlabel.long()).sum()
        return correct_pred.float()/num_examples * 100
start_time = time.time()

for epoch in range(NUM_EPOCHS):
    model.train()
    for batch_idx, batch_data in enumerate(train_loader):
        
        text, text_lengths = batch_data.content
        
        ### FORWARD AND BACK PROP
        logits = model(text, text_lengths)
        cost = F.cross_entropy(logits, batch_data.classlabel.long())
        optimizer.zero_grad()
        
        cost.backward()
        
        ### UPDATE MODEL PARAMETERS
        optimizer.step()
        
        ### LOGGING
        if not batch_idx % 50:
            print (f'Epoch: {epoch+1:03d}/{NUM_EPOCHS:03d} | '
                   f'Batch {batch_idx:03d}/{len(train_loader):03d} | '
                   f'Cost: {cost:.4f}')

    with torch.set_grad_enabled(False):
        print(f'training accuracy: '
              f'{compute_accuracy(model, train_loader, DEVICE):.2f}%'
              f'\nvalid accuracy: '
              f'{compute_accuracy(model, valid_loader, DEVICE):.2f}%')
        
    print(f'Time elapsed: {(time.time() - start_time)/60:.2f} min')
    
print(f'Total Training Time: {(time.time() - start_time)/60:.2f} min')
print(f'Test accuracy: {compute_accuracy(model, test_loader, DEVICE):.2f}%')<jupyter_output>Epoch: 001/050 | Batch 000/891 | Cost: 1.3865
Epoch: 001/050 | Batch 050/891 | Cost: 1.2066
Epoch: 001/050 | Batch 100/891 | Cost: 1.1692
Epoch: 001/050 | Batch 150/891 | Cost: 0.9862
Epoch: 001/050 | Batch 200/891 | Cost: 0.9992
Epoch: 001/050 | Batch 250/891 | Cost: 0.8206
Epoch: 001/050 | Batch 300/891 | Cost: 0.7449
Epoch: 001/050 | Batch 350/891 | Cost: 0.6300
Epoch: 001/050 | Batch 400/891 | Cost: 0.7949
Epoch: 001/050 | Batch 450/891 | Cost: 0.5924
Epoch: 001/050 | Batch 500/891 | Cost: 0.6359
Epoch: 001/050 | Batch 550/891 | Cost: 0.9071
Epoch: 001/050 | Batch 600/891 | Cost: 0.5533
Epoch: 001/050 | Batch 650/891 | Cost: 0.6460
Epoch: 001/050 | Batch 700/891 | Cost: 0.5671
Epoch: 001/050 | Batch 750/891 | Cost: 0.4732
Epoch: 001/050 | Batch 800/891 | Cost: 0.4709
Epoch: 001/050 | Batch 850/891 | Cost: 0.4538
training accuracy: 86.64%
valid accuracy: 85.98%
Time elapsed: 44.74 min
Epoch: 002/050 | Batch 000/891 | Cost: 0.5731
Epoch: 002/050 | Batch 050/891 | Cost: 0.4066
Epoch: [...]<jupyter_text>Evaluation Evaluating on some new text that has been collected from recent news articles and is not part of the training or test sets.<jupyter_code>import spacy
nlp = spacy.load('en')


map_dictionary = {
    0: "World",
    1: "Sports",
    2: "Business",
    3:"Sci/Tech",
}


def predict_class(model, sentence, min_len=4):
    # Somewhat based on
    # https://github.com/bentrevett/pytorch-sentiment-analysis/
    # blob/master/5%20-%20Multi-class%20Sentiment%20Analysis.ipynb
    model.eval()
    tokenized = [tok.text for tok in nlp.tokenizer(sentence)]
    if len(tokenized) < min_len:
        tokenized += ['<pad>'] * (min_len - len(tokenized))
    indexed = [TEXT.vocab.stoi[t] for t in tokenized]
    length = [len(indexed)]
    tensor = torch.LongTensor(indexed).to(DEVICE)
    tensor = tensor.unsqueeze(1)
    length_tensor = torch.LongTensor(length)
    preds = model(tensor, length_tensor)
    preds = torch.softmax(preds, dim=1)
    
    proba, class_label = preds.max(dim=1)
    return proba.item(), class_label.item()
text = """
The windfall follows a tender offer by Z Holdings, which is controlled by SoftBank’s domestic wireless unit, 
for half of Zozo’s shares this month.
"""

proba, pred_label = predict_class(model, text)

print(f'Class Label: {pred_label} -> {map_dictionary[pred_label]}')
print(f'Probability: {proba}')
text = """
EU data regulator issues first-ever sanction of an EU institution, 
against the European parliament over its use of US-based NationBuilder to process voter data 
"""

proba, pred_label = predict_class(model, text)

print(f'Class Label: {pred_label} -> {map_dictionary[pred_label]}')
print(f'Probability: {proba}')
text = """
LG announces CEO <NAME> will be replaced by <NAME>won Dec. 1, amid 2020 
leadership shakeup and LG smartphone division's 18th straight quarterly loss
"""

proba, pred_label = predict_class(model, text)

print(f'Class Label: {pred_label} -> {map_dictionary[pred_label]}')
print(f'Probability: {proba}')
%watermark -iv<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11257565568258238, 'fraction_numerical': 0.04008069939475454, 'mean_word_length': 4.230742173760112, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 61, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '316324', 'license': 'MIT', 'n_tokens_mistral': 5474, 'n_tokens_neox': 4965, 'n_words': 1349, 'path': 'pytorch_ipynb/rnn/rnn_bi_multilayer_lstm_own_csv_agnews.ipynb', 'repo_name': 'DeepSE/deeplearning-models', 'size': '14940'}