<jupyter_start><jupyter_text><jupyter_code>from google.colab import files
uploaded = files.upload()
import json
import numpy as np
import nltk
nltk.download('punkt')
nltk.download('stopwords')
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from torch.utils.data import Dataset, DataLoader

stopwords_list = list(stopwords.words('english'))
stopwords_list.extend(['?', '.', '!',','])
with open('intents.json', 'r') as f:
    intents = json.load(f)<jupyter_output><empty_output><jupyter_text>**Tokenization**It is the process in which we split a string into meaningful units. (e.g. words , characters, numbers)--- **Stemming**Generate the root form of the words.--- **Bag Of Words*** Algorithms **take vectors of numbers as input**, therefore we need to convert documents to fixed-length vectors of numbers.* A simple and effective model for thinking about text documents in machine learning is called the Bag-of-Words Model, or BoW.* This can be done by **assigning each word a unique number**. Then any document we see can be encoded as a fixed-length vector with the length of the vocabulary of known words. The value in each position in the vector could be filled with a count or frequency of each word in the encoded document.* In our case our data look like following :<jupyter_code>def tokenize(sentence):
    return word_tokenize(sentence)

def bag_of_words(tokenized_sentence, words):
    sentence_words = [word for word in tokenized_sentence]

    bag = np.zeros(len(words), dtype=np.float32)
    for idx, w in enumerate(words):
        if w in sentence_words: 
            bag[idx] = 1

    return bag<jupyter_output><empty_output><jupyter_text>**Dataset:*** The Torch Dataset class is basically an abstract class representing the dataset. It allows us to treat the dataset as an object of a class, rather than a set of data and labels.* The main task of the Dataset class is to return a pair of [input, label] every time it is called. We can define functions inside the class to preprocess the data, and return it in the format we require.* The class must contain two main functions:   * __len__(): This is a fuction that returns the length of the dataset.  * __getitem__(): This is a function that returns one training example. **Torch Dataloader:*** The Torch Dataloader not only allows us to **iterate through the dataset in batches**, but also gives us access to inbuilt functions for multiprocessing(allows us to load multiple batches of data in parallel, rather than loading one batch at a time), shuffling, etc.* The torch Dataloader takes a torch Dataset as input, and calls the __getitem__() function from the Dataset class to create a batch of data.<jupyter_code>class ChatDataset(Dataset):
    def __init__(self, X_train, y_train):
        self.n_samples = len(X_train)
        self.x_data = X_train
        self.y_data = y_train

    def __getitem__(self, index):
        return self.x_data[index], self.y_data[index]

    def __len__(self):
        return self.n_samples<jupyter_output><empty_output><jupyter_text>Create Bag Of Words.<jupyter_code>vocabulary = []
tags = []
xy = []

for intent in intents['intents']:
    tag = intent['tag']
    tags.append(tag)
    for pattern in intent['patterns']:
        # 1. Tokenize
        w = tokenize(pattern)
        vocabulary.extend(w)
        xy.append((w, tag))

vocabulary = [w.lower() for w in vocabulary if w not in stopwords_list]

# Remove duplicates and Sort
vocabulary = sorted(set(vocabulary))
tags = sorted(set(tags))

# create training data
X_train = []
y_train = []
for (pattern_sentence, tag) in xy:
    bag = bag_of_words(pattern_sentence, vocabulary)
    X_train.append(bag)

    label = tags.index(tag)
    y_train.append(label)

X_train = np.array(X_train)
y_train = np.array(y_train)<jupyter_output><empty_output><jupyter_text>Create a Dataset<jupyter_code># Hyper-parameters 
num_epochs = 1000
batch_size = 8
learning_rate = 0.001
input_size = len(X_train[0])
hidden_size = 8
output_size = len(tags)
print(input_size, output_size)

dataset = ChatDataset(X_train,y_train)
train_loader = DataLoader(dataset=dataset, batch_size=batch_size, shuffle=True, num_workers=0)<jupyter_output>54 7<jupyter_text>**Model*** Neural networks are made up of layers of neurons, which are the core processing unit of the network. * The basic architecture of a deep learning neural network consists of three main components.  1. **Input Layer** : This is where the training observations are fed.  2. **Hidden Layers** : These are the intermediate layers between the input and output layers. The deep neural network learns about the relationships involved in data in this component.  3. **Output Layer** : This is the layer where the final output is extracted from what’s happening in the previous two layers. In case of classification problems, the output layer will have one of the target classes as output.* We train the model, for which we create a class, NeuralNet. This class in turn inherits from the nn.Module class. The next step is to define the layers of our deep neural network. We start by defining the parameters for the fully connected layers with the __init__() method.* In  forward(self, x) function we define how they interact with each other.<jupyter_code>import torch
import torch.nn as nn

class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(NeuralNet, self).__init__()
        self.l1 = nn.Linear(input_size, hidden_size) 
        self.l2 = nn.Linear(hidden_size, hidden_size) 
        self.l3 = nn.Linear(hidden_size, num_classes)
        self.relu = nn.ReLU()
    
    def forward(self, x):
        out = self.l1(x)
        out = self.relu(out)
        out = self.l2(out)
        out = self.relu(out)
        out = self.l3(out)
        # no activation and no softmax at the end
        return out
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = NeuralNet(input_size, hidden_size, output_size).to(device)<jupyter_output><empty_output><jupyter_text>**Training Phase**:* We’ll use the adam optimizer to optimize the network, and considering that this is a classification problem, we’ll use the cross entropy as loss function.  **Loss Function - CrossEntropyLoss**: **Optimizer**:During neural network training, its weights are randomly initialized initially and then they are updated in each epoch in a manner such that they minimize the loss function.Adam Optimizer uses both momentum and adaptive learning rate for better convergence. **Steps**:1. A forward pass of the input through the model.2. Clearing the last error gradient.3. Calculating the loss for the model output.4. Backpropagating the error through the model.5. Update the model in an effort to reduce loss.<jupyter_code>criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
for epoch in range(num_epochs):
    for (words, labels) in train_loader:
        words = words.to(device)
        labels = labels.to(dtype=torch.long).to(device)
        
        outputs = model(words)
        loss = criterion(outputs, labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
    if (epoch+1) % 100 == 0:
        print (f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')


print(f'final loss: {loss.item():.4f}')

data = {
  "model_state": model.state_dict(),
  "input_size" : input_size,
  "hidden_size": hidden_size,
  "output_size": output_size,
  "vocabulary" : vocabulary,
  "tags"       : tags 
  }

FILE = "data.pth"
torch.save(data, FILE)<jupyter_output>Epoch [100/1000], Loss: 1.4230
Epoch [200/1000], Loss: 0.6202
Epoch [300/1000], Loss: 0.0105
Epoch [400/1000], Loss: 0.6572
Epoch [500/1000], Loss: 0.0386
Epoch [600/1000], Loss: 0.0257
Epoch [700/1000], Loss: 1.3531
Epoch [800/1000], Loss: 0.9231
Epoch [900/1000], Loss: 0.0207
Epoch [1000/1000], Loss: 1.3516
final loss: 1.3516<jupyter_text>**Testing Phase**:<jupyter_code>FILE = "data.pth"
data = torch.load(FILE)

input_size = data["input_size"]
hidden_size = data["hidden_size"]
vocabulary = data["output_size"]
all_words = data['vocabulary']
tags = data['tags']
model_state = data["model_state"]

model = NeuralNet(input_size, hidden_size, output_size).to(device)
model.load_state_dict(model_state)
model.eval()
import random

bot_name = "Greg"
print("Let's chat! (type 'quit' to exit)")
while True:
    # sentence = "do you use credit cards?"
    sentence = input("You: ")
    if sentence == "quit":
        break

    sentence = tokenize(sentence)
    X = bag_of_words(sentence, all_words)
    X = X.reshape(1, X.shape[0])
    X = torch.from_numpy(X).to(device)

    output = model(X)
    _, predicted = torch.max(output, dim=1)

    tag = tags[predicted.item()]

    probs = torch.softmax(output, dim=1)
    prob = probs[0][predicted.item()]
    if prob.item() > 0.75:
        for intent in intents['intents']:
            if tag == intent["tag"]:
                print(f"{bot_name}: {random.choice(intent['responses'])}")
    else:
        print(f"{bot_name}: I do not understand...")<jupyter_output>Let's chat! (type 'quit' to exit)
You: Hello
Greg: I do not understand...
You: quit
{'fraction_non_alphanumeric': 0.1046070460704607, 'fraction_numerical': 0.018970189701897018, 'mean_word_length': 4.111357340720222, 'pattern_counts': {'":': 4, '<': 30, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '215721', 'license': 'MIT', 'n_tokens_mistral': 2989, 'n_tokens_neox': 2821, 'n_words': 1005, 'path': 'test.ipynb', 'repo_name': 'GrigorisPapapostolou/Chat_Bot', 'size': '9225'}