<jupyter_start><jupyter_text>Ref. [https://towardsdatascience.com/build-a-text-generator-web-app-in-under-50-lines-of-python-9b63d47edabb](https://towardsdatascience.com/build-a-text-generator-web-app-in-under-50-lines-of-python-9b63d47edabb) We will be using OpenAI’s GPT-2 as the model and Panel as the web dashboard framework. This guide will be split into two parts. In the first part, we will load our model and write a predictions function. In the second, we will build the web application.Example text generation application. We will be building a simpler variation of this web app.What you will needThis tutorial assumes you already have Python 3.7+ installed and have some understanding of Language Models. Although the steps involved can be done outside of Jupyter, using a jupyter notebook is highly highly recommended.We will be using PyTorch as our Deep Learning library of choice. Within PyTorch, we will use the transformers library to import the pre-trained OpenGPT-2 model. You can install these libraries by individually entering the following commands in your bash:pip install torchpip install transformersFor our web application, we will be utilizing Panel, a great tool for easily creating servable dashboards from either jupyter notebooks or a regular python script. Use the following command to install panel:pip install panelPart 1: Setting up the ModelOpenAI’s GPT is a type of transformer model which has received a lot of buzz about its capabilities to produce human-like text. If you have not experimented with it before, you are likely to come away with the same opinion at the end of this read.Loading the ModelFirst, we need to import the required packages.  use base enviroment<jupyter_code>#pip install transformers
import numpy as np
import torch
import torch.nn.functional as F
from transformers import GPT2Tokenizer, GPT2LMHeadModel
from random import choice<jupyter_output><empty_output><jupyter_text>Next, we will load the OpenGPT2 Tokenizer and the Language Model: (it may take a few minutes to download the pre-trained model if run for the first time)<jupyter_code>tok = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")<jupyter_output>Downloading: 100%|██████████| 0.99M/0.99M [00:00<00:00, 1.34MB/s]
Downloading: 100%|██████████| 446k/446k [00:00<00:00, 638kB/s]  
Downloading: 100%|██████████| 1.29M/1.29M [00:01<00:00, 826kB/s]
Downloading: 100%|██████████| 665/665 [00:00<00:00, 336kB/s]
Downloading: 100%|██████████| 523M/523M [01:55<00:00, 4.76MB/s]<jupyter_text>Predictions FunctionAt this stage, most of the work in already done. Since our model is pre-trained, we don’t need to train it or make any modifications. We simply need to write a function which can input text to the model and generate a prediction.<jupyter_code>def get_pred(text, model, tok, p=0.7):
    input_ids = torch.tensor(tok.encode(text)).unsqueeze(0)
    logits = model(input_ids)[0][:, -1]
    probs = F.softmax(logits, dim=-1).squeeze()
    idxs = torch.argsort(probs, descending=True)
    res, cumsum = [], 0.
    for idx in idxs:
        res.append(idx)
        cumsum += probs[idx]
        if cumsum > p:
            pred_idx = idxs.new_tensor([choice(res)])
            break
    pred = tok.convert_ids_to_tokens(int(pred_idx))
    return tok.convert_tokens_to_string(pred)<jupyter_output><empty_output><jupyter_text>There is a lot happening in this function. So, let’s break it down. First, we are tokenizing and encoding the input text from input_ids. Then, we ask our model to generate a logits vector for the next word/token. After applying softmax and sorting these probabilities in descending order, we have a vector, idxs, which lists the indices of each token in our vocab in order by their respective probabilities.At this stage, we could just pick the token which has the highest probability. However, we want to be able to mix up our results so the same input text can generate a variety of text. To do this, we will add an element of randomness where we choose a random token from a list of the most probable next tokens. This way, we are not selecting the same predicted token each time. To do this, we utilize Nucleus (Top-p) Sampling.We perform this by looping through each probability until the sum of all the probabilities we have looped over is greater than p, an arbitrary number between 0 and 1. All the tokens iterated through until p is exceeded are stored in a list, res. Once, p is exceeded, we choose a token at random from this list. Remember that the list of probabilities that we are looping through contains indices ordered by their probability. Note that if p is higher, more tokens will be included in our list. Vice versa. Therefore, if you want the same result each time, you can set p to 0.Now, let’s test out our pred function a few times:<jupyter_code>get_pred("Artificial Ingeligence", model, tok, p=0.7)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07640859343331982, 'fraction_numerical': 0.027969193352249695, 'mean_word_length': 4.718424101969872, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '558618', 'license': 'MIT', 'n_tokens_mistral': 1533, 'n_tokens_neox': 1401, 'n_words': 657, 'path': 'tarea_45/text_generator.ipynb', 'repo_name': 'zumaia/theEgg', 'size': '4934'}