<jupyter_start><jupyter_text>Creative writing using GPT-2 Text Generation Creative writing can be very fun, yet challenging, especially when you hit that writer's block. In this notebook you will learn how to use AWS Marketplace GPT-2-XL pre-trained model on Amazon SageMaker to generate text based on your prompt to help author prose and poetry.GPT2 ([Generative Pre-trained Transformer 2](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf)) algorithm is an unsupervised transformer language model. Transformer language models take advantage of transformer blocks. These blocks make it possible to process intra-sequence dependencies for all tokens in a sequence at the same time. GPT2 has been developed by [OpenAI](https://openai.com/) and is a powerful generative NLP model that excels in processing long-range dependencies and it is pre-trained on a diverse corpus of text.  Overview:In [step 1](Step-1-Determine-input-prompt-and-visualize-word-dependencies) of this notebook, you will determine an input prompt that will be used to condition the GPT-2 model for text generation. You will also [visualize attention](Step-1.1-Introduction-to-attention) mechanism of GPT-2 model. In [step 2](Step-2-Use-an-ML-model-to-generate-text-based-on-prompt), you will create the model from an AWS Marketplace subscription, and deploy to an Amazon SageMaker endpoint. In [step 3](Step-3-Explore-use-cases-and-model-parameters), you will explore text generation use cases with various model parameter settings. In [step 4](Step-4-Use-Amazon-SageMaker-batch-transform), you will perform inference asynchronously using SageMaker batch transform instead of the endpoint. In [Step 5](Step-5-Next-steps) you will find additional models to explore and experiment with.  Contents:* [Pre-requisites](Pre-requisites)* [Step 1 Determine input prompt and visualize word dependencies](Step-1-Determine-input-prompt-and-visualize-word-dependencies)    * [Step 1.1 Introduction to attention](Step-1.1-Introduction-to-attention)    * [Step 1.2 Specify input prompt and visualize attention mechanism](Step-1.2-Specify-input-prompt-and-visualize-attention-mechanism)* [Step 2 Use an ML model to generate text based on prompt](Step-2-Use-an-ML-model-to-generate-text-based-on-prompt)    * [Step 2.1 Specify model arn from AWS Marketplace subscription](Step-2.1-Specify-model-arn-from-AWS-Marketplace-subscription)    * [Step 2.2 Create model from model package and deploy to endpoint](Step-2.2-Create-model-from-model-package-and-deploy-to-endpoint)* [Step 3 Explore use cases and model parameters](Step-3-Explore-use-cases-and-model-parameters)    * [Step 3.1 Use case 1: Assist writing of prose](Step-3.1-Use-case-1:-Assisted-writing-of-prose)    * [Step 3.2 Use case 2: Autonomous authoring of poem](Step-3.2-Use-case-2:-Autonomous-authoring-of-poem)    * [Step 3.3 Delete Amazon SageMaker endpoint](Step-3.3-Delete-Amazon-SageMaker-endpoint)* [Step 4 Use Amazon SageMaker batch transform](Step-4-Use-Amazon-SageMaker-batch-transform)    * [Step 4.1 Create input file for batch transform job](Step-4.1-Create-input-file-for-batch-transform-job)    * [Step 4.2 Upload file to S3](Step-4.2-Upload-file-to-S3)    * [Step 4.3 Execute the batch transform job](Step-4.3-Execute-the-batch-transform-job)    * [Step 4.4 Visualize output](Step-4.4-Visualize-output)    * [Step 4.5 Delete the model](Step-4.5-Delete-the-model)* [Step 5 Next steps](Step-5-Next-steps)    * [Step 5.1 Additional resources](Step-5.1-Additional-resources)    * [Step 5.2 Cancel AWS Marketplace subscription](Step-5.2-Cancel-AWS-Marketplace-subscription) Usage instructionsYou can run this notebook one cell at a time (By using Shift+Enter for running a cell). **Pre-requisites**This sample notebook requires a subscription to **[GPT-2 XL - Text generation](https://aws.amazon.com/marketplace/pp/prodview-cdujckyfypprg)**, a pre-trained machine learning model package from AWS Marketplace. If your AWS account has not been subscribed to this listing, here is the process you can follow: 1. Open the [listing](https://aws.amazon.com/marketplace/pp/prodview-cdujckyfypprg) from AWS Marketplace1. Read the **Highlights** section and then **product overview** section of the listing.1. View **usage information** and then **additional resources.**1. Note the supported instance types.1. Next, click on **Continue to subscribe.**1. Review **End-user license agreement, support terms**, as well as **pricing information.**1. **"Accept Offer"** button needs to be clicked if your organization agrees with EULA, pricing information as well as support terms.  If **Continue to configuration** button is active, it means your account already has a subscription to this listing. Once you click on **Continue to configuration** button and then choose region, you will see that a Product Arn will appear. This is the model package ARN that you need to specify while creating a deployable model. However, for this notebook, the Model Package ARN has been specified in **src/model_package_arns.py** file and you do not need to specify the same explicitly.2. This notebook requires the IAM role associated with this notebook to have *AmazonSageMakerFullAccess* IAM permission.3. Note: If you want to run this notebook on AWS SageMaker Studio - please use Classic Jupyter mode to be able correctly render visualization. Pick instance type **'ml.t3.large'** or larger. Set kernel to **'conda_python3'**.     **Installing Dependencies**In Step 1 of this notebook, you will use BertViz to visualize Transformer Attention.[BertViz](https://github.com/jessevig/bertviz) is an opensource package for visualizing attention in the Transformer models, including GPT-2. It provides the ability to visualize attention scores and contextual dependencies between tokens. You will take a closer look at the attention mechanism in the next section of the notebook.<jupyter_code># Installing BertViz for attention visualization
import sys
!rm -rf bertviz
!git clone https://github.com/jessevig/bertviz.git
!cd bertviz && {sys.executable} -m pip install -r requirements.txt
#Import necessary libraries
import base64
import json 
import uuid
from sagemaker import ModelPackage
import sagemaker as sage
from sagemaker import get_execution_role
from urllib.parse import urlparse
import boto3
from IPython.display import Image
from PIL import Image as ImageEdit
import urllib.request
import numpy as np
import boto3
from pprint import pprint
from src.model_package_arns import ModelPackageArnProvider
#Importing BertViz packages
from bertviz.bertviz.transformers_neuron_view import GPT2Model, GPT2Tokenizer
from bertviz.bertviz.neuron_view import show<jupyter_output><empty_output><jupyter_text>Initialize JavaScript dependencies:<jupyter_code>%%javascript
require.config({
  paths: {
      d3: '//cdnjs.cloudflare.com/ajax/libs/d3/5.7.0/d3.min',
    jquery: '//ajax.googleapis.com/ajax/libs/jquery/2.0.0/jquery.min',
  }
});<jupyter_output><empty_output><jupyter_text>Step 1 Determine input prompt and visualize word dependencies GPT-2 uses input text to set the initial context for further text generation. The length of an input text can range from few words to a maximum sequence length of 1024 tokens. The longer an initial input, the more subject context is provided to a model. Generally, longer inputs produce a more coherent text output.In this example, let's start with a simple sentence: **"Machine learning is great for humanity. It helps"**. There is only part of the second sentence provided intentionally to explore what options of continuation GPT-2 can generate for us and understand how a GPT-2 model can expand on this context while generating text.  Step 1.1 Introduction to attention[Self-Attention mechanism](https://arxiv.org/abs/1706.03762) is one of the key components for Transformers architectures, including GPT-2. It helps to relate different positions of a specific sequence of tokens in order to compute contextual representation of the sequence.The original paper ([Attention is all you need](https://arxiv.org/abs/1706.03762)) introduces Self-Attention mechanism as follows:>We call our particular attention "Scaled Dot-Product Attention". The input consists of queries and keys of dimension 𝑑𝑘, and values of dimension 𝑑𝑣. We compute the dot products of the query with all keys, divide each by 𝑑𝑘⎯⎯⎯⎯√, and apply a softmax function to obtain the weights on the values.$$                                                                            \mathrm{Attention}(Q, K, V) = \mathrm{softmax}(\frac{QK^T}{\sqrt{d_k}})V               $$   This notebook uses [BertViz](https://github.com/jessevig/bertviz) tool to quickly gain better understanding of practical meaning in addition to the mathematical definition of attention.  Step 1.2 Specify input prompt and visualize attention mechanismYou can experiment with different prompts and see what contextual dependencies exist in your own examples.<jupyter_code>payload = {"input": "Machine learning is great for humanity. It helps"}<jupyter_output><empty_output><jupyter_text>For example, this is how it will look for our initial input:BertViz package uses the attention patterns produced by one or more attention heads in a given transformer layer to provide visualization of words (tokens) in a given text sequence.This example specifically uses a neuron view mode which visualizes the individual neurons in the query and key vectors and shows how they are used to compute attention.The darker blue color reflects a higher score and "stronger" attention connection between the selected token and other tokens in a sequence. You may notice that word "helps" is contextually related to "Machine learning" which is at the beginning of the previous sentence. It is also noticeable there is a connection to the word "humanity" despite it being in a separate sentence. This is just one representation of Layer 8 and Head 8 of a model. Other combinations of layers and heads can surface non-obvious relations between positions in a given sequence of tokens. The next cell will render neuron view visualization. You will be able to point your mouse at the word (token) in your input on the left and it will show you dependencies with other words on the right side.<jupyter_code>sentence = payload['input']
model_type = 'gpt2'
model_version = 'gpt2'
model = GPT2Model.from_pretrained(model_version)
tokenizer = GPT2Tokenizer.from_pretrained(model_version)
show(model, model_type, tokenizer, sentence)<jupyter_output><empty_output><jupyter_text>Try to change layer and head numbers on the widget and hover over to see different dependencies between tokens. Self-attention is one of the most important components of generative NLP models such as GPT, GPT2, GPT3. It makes it possible to achieve  State of The Art performance results in predicting the next word by the given context. Because of the auto-regressive nature of the text generation process, it is possible to generate long stretches of contextually coherent and (many times) close to "written by human quality" paragraphs of text.Note that "words" and "tokens" here are used interchangeably for simplicity. However, one word can consist of several tokens. In this case, attention will be rendered for each part of the word separately. [Byte_Pair Encoding](https://en.wikipedia.org/wiki/Byte_pair_encoding) improves the tokenization process in GPT2. It facilitates proper tokenization based on training dataset specifics.There are multiple layers and multiple attention heads in GPT-2 model architecture. Since the model has been already pre-trained on a large text corpus, different layers and different heads produce attention scores that reflect multiple semantic levels of connections between tokens.**Optional reading**To learn more about GPT architecture, attention, and how these types of visualizations work, please refer to these papers: - [Visualizing Attention in Transformer-Based Language Representation Models](https://arxiv.org/pdf/1904.02679.pdf)- [Improving Language Understanding by Generative Pre-Training](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) - Original introduction of GPT architecture paper- [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf) - GPT2 architecture paper- [Language Models are Few-Shot Learners](https://arxiv.org/pdf/2005.14165.pdf) - GPT3 architecture paper- [Better Language Modelsand Their Implications](https://openai.com/blog/better-language-models) - [OpenAI](https://openai.com)'s original blog post about GPT2- [The Illustrated Transformer](http://jalammar.github.io/illustrated-transformer) - More details about Transformers architecture  Step 2 Use an ML model to generate text based on promptBecause you utilize [GPT-2 XL - Text generation](https://aws.amazon.com/marketplace/pp/prodview-cdujckyfypprg) algorithm from AWS Marketplace - all you need to do to start using it - is to deploy it as an inference endpoint in your account. Alternatively, we can use SageMaker Batch Transformation to run inference on batch payloads. To do that, let's set our payload variable as well as endpoint_name:<jupyter_code># Declare variables
role = get_execution_role()
sagemaker_session = sage.Session()
bucket=sagemaker_session.default_bucket()
endpoint_name='demo-gpt2-endpoint'
client = boto3.client('sagemaker-runtime')<jupyter_output><empty_output><jupyter_text>You are going to specify instance types for real-time inference endpoint as well as batch transformation.You can refer to compatible instance types for this AWS Marketplace offering [here](https://aws.amazon.com/marketplace/pp/prodview-cdujckyfypprg).<jupyter_code>real_time_inference_instance_type='ml.m4.2xlarge'
batch_transform_inference_instance_type='ml.m4.2xlarge'
content_type='application/json'<jupyter_output><empty_output><jupyter_text>Step 2.1 Specify model arn from AWS Marketplace subscription You will use **ModelPackageArnProvider** class to make sure you get the correct ARN in every supported region:<jupyter_code>model_package_arn = ModelPackageArnProvider.get_gpt2_model_package_arn(sagemaker_session.boto_region_name)<jupyter_output><empty_output><jupyter_text>Step 2.2 Create model from model package and deploy to endpoint Finally, you will invoke **model.deploy** API call to create a real-time inference endpoint with GPT2-XL model.This step might take several minutes to run:<jupyter_code>model = ModelPackage(role=role, model_package_arn=model_package_arn)
predictor = model.deploy(1, real_time_inference_instance_type, endpoint_name=endpoint_name)<jupyter_output><empty_output><jupyter_text>Step 3 Explore use cases and model parameters GPT-2 can be used by AI researchers and practitioners to better understand the behaviors, capabilities, biases, and constraints of large-scale generative language models.Also, it can be used in some non-research settings such as:* Writing assistance: Grammar assistance, autocompletion (for normal prose or code)* Creative writing and art: exploring the generation of creative, fictional texts; aiding the creation of poetry and other literary art.* Entertainment: Creation of games, chatbots, and amusing generations. **GPT-2 model package parameters:**You can tweak the following model parameters to influence the serving behavior of the model.* **input**: str (required): Input text* **length**: int = 50: The number of words to generate* **stop_token**: str: Stop text generation if it finds this word (token)* **num_return_sequences**: int = 1: Number of different sequences to generate. All sequences will start from the same input* **temperature**: float = 1.0: temperature of softmax - higher values increase creativity and decrease output coherence* **k**: int = 50: top-k sampling - model will choose from top k most probable words. Lower values eliminate less coherent words* **p**: float = 1.0: top-p nucleus sampling. Should be between 0 and 1 to activate. The alternative to top-k to select a minimum number of candidate words in which cumulative probability exceeds p. Values closer to 1.0 generally provide more coherent outputs.* **repetition_penalty**: float = 1.0: Higher value discourages model from repeating the same token* **seed**: int = None: Set this random seed before making a prediction* **test**: bool = False: If true it will return a sample test response no predictions will be made  Step 3.1 Use case 1: Assisted writing of prose In this first use case, you will see how GPT-2 can assist an author generate ideas for fiction.<jupyter_code># Prompt for fiction
payload = '{"input": "Machine learning is great for humanity. It helps", "length": 50,\
"repetition_penalty": 10,"num_return_sequences": 1}'

response = client.invoke_endpoint(
    EndpointName=endpoint_name, 
    ContentType=content_type,
    Body=payload
    )

output=response["Body"].read()

pprint(json.loads(output))<jupyter_output><empty_output><jupyter_text>In the output above, you will notice that one sequence is generated based on 'input' prompt since 'num_return_sequences' was set to '1'. The length of the sequence is roughly 50 words.Now, let's add an additional parameter, 'stop_token' to the request payload and see its effect.<jupyter_code># Prompt for fiction
payload = '{"input": "Machine learning is great for humanity. It helps build robots.", "length": 50, "stop_token": "robots",\
"repetition_penalty": 10,"num_return_sequences": 1}'

response = client.invoke_endpoint(
    EndpointName=endpoint_name, 
    ContentType=content_type,
    Body=payload
    )

output=response["Body"].read()

pprint(json.loads(output))<jupyter_output><empty_output><jupyter_text>You will notice that no 'output' was generated because the 'stop_token' word 'robots' was included in the 'input'.Now, change the 'input' to not include the 'stop_token' and change the value of 'length' to 100. 'Repetition_penalty' is increased to 10 from default value of 1 to discourage model from generating repetitive text.<jupyter_code># Prompt for fiction
payload = '{"input": "Machine learning is great for humanity. It helps", "length": 100, "stop_token": "robots",\
"repetition_penalty": 10,"num_return_sequences": 1}'

response = client.invoke_endpoint(
    EndpointName=endpoint_name, 
    ContentType=content_type,
    Body=payload
    )

output=response["Body"].read()

pprint(json.loads(output))<jupyter_output><empty_output><jupyter_text>The 'stop_token' parameter has no effect on the output above, since it is not included in the 'input'. The output is more verbose, because we changed the 'length' to 100.  Step 3.2 Use case 2: Autonomous authoring of poem In this second use case, you will see how GPT-2 can author a Shakespear-inspired poem.<jupyter_code># Prompt for poetry
# Sample from Shakespeare's Macbeth
payload = '{"input": "The <NAME>—Worthy to be a rebel, for to that\
The multiplying villanies of nature\
Do swarm upon him—from the western isles\
Of kerns and gallowglasses is supplied;\
And fortune, on his damned quarrel smiling", "length": 100, "repetition_penalty": 1,\
"num_return_sequences": 2, "temperature": 1, "k": 50}'

response = client.invoke_endpoint(
    EndpointName=endpoint_name, 
    ContentType=content_type,
    Body=payload
    )

output=response["Body"].read()

pprint(json.loads(output))<jupyter_output><empty_output><jupyter_text>The 'output' depicts a completely different style of writing, conditioned by the 'input'. This shows how GPT-2 adapts to 'input'. The 'output' now comprises of two sequences, since "num_return_sequences" was set to 2.Increase the values of 'temperature' to 10, and 'k' to 90. The effect should be more creative use of words and vocabulary.<jupyter_code># Prompt for poetry
# Sample from Shakespeare's Macbeth
payload = '{"input": "The <NAME>—Worthy to be a rebel, for to that\
The multiplying villanies of nature\
Do swarm upon him—from the western isles\
Of kerns and gallowglasses is supplied;\
And fortune, on his damned quarrel smiling", "length": 100, "repetition_penalty": 1,\
"num_return_sequences": 2, "temperature": 10, "k": 90}'

response = client.invoke_endpoint(
    EndpointName=endpoint_name, 
    ContentType=content_type,
    Body=payload
    )

output=response["Body"].read()

pprint(json.loads(output))<jupyter_output><empty_output><jupyter_text>Here are a few things you can do to continue experimenting with this model1. Try different prompts representing other genres of writing such as science fiction, non-fiction, essay, novel, etc.2. Play with the parameter settings to see which generates novel, coherent ideas for writing  Step 3.3 Delete Amazon SageMaker endpoint Delete the endpoint when you no longer need it, to avoid getting charged for it.<jupyter_code>sagemaker_session.delete_endpoint(endpoint_name)
sagemaker_session.delete_endpoint_config(endpoint_name)<jupyter_output><empty_output><jupyter_text>Step 4 Use Amazon SageMaker batch transform Use batch transform to run inference when you don't need a persistent endpoint. You would need to specify the following arguments in the job:* Hardware specification (instance count and type).* strategy: determines how records should be batched into each prediction request within the batch transform job. For this job specify 'SingleRecord'. 'MultiRecord' may be used for batching inputs.* assemble_with: Which controls how predictions are output. 'None' does not perform any special processing, 'Line' places each prediction on its own line.* output_path: The S3 location for batch transform to be output. Note, file(s) will be named with '.out' suffixed to the input file(s) names. In this case, the output file will be 'input.json.out'. Note that in this case, multiple batch transform runs will overwrite existing files in the S3 bucket.  Step 4.1 Create input file for batch transform job<jupyter_code>file_name = 'input.json'
%%writefile $file_name

{"input": "The merciless Macdonwald—Worthy to be a rebel, for to that The multiplying villanies of nature Do swarm upon him—from the western isles Of kerns and gallowglasses is supplied; And fortune, on his damned quarrel smiling", "length": 50, "repetition_penalty": 1,"num_return_sequences": 3}<jupyter_output><empty_output><jupyter_text>Step 4.2 Upload file to S3<jupyter_code>#upload the file to S3
transform_input = sagemaker_session.upload_data(file_name, key_prefix=endpoint_name) 
print("Transform input uploaded to " + transform_input)<jupyter_output><empty_output><jupyter_text>Step 4.3 Execute the batch transform job Output from the batch transform job will be saved to S3 bucket in the 'output_path'<jupyter_code>#Run a batch-transform job
transformer = model.transformer(instance_count=1,
                                instance_type='ml.m4.2xlarge',
                                strategy='SingleRecord',
                                assemble_with='Line',
                                output_path='s3://{}/{}/output'.format(bucket,endpoint_name))
transformer.transform(transform_input, content_type=content_type)
transformer.wait()
#output is available on following path
transformer.output_path<jupyter_output><empty_output><jupyter_text>Step 4.4 Visualize output<jupyter_code>from urllib.parse import urlparse

parsed_url = urlparse(transformer.output_path)
bucket_name = parsed_url.netloc
file_key = '{}/{}.out'.format(parsed_url.path[1:], file_name.split("/")[-1])
print(file_key)
s3_client = boto3.client('s3')

response = s3_client.get_object(Bucket = sagemaker_session.default_bucket(), Key = file_key)
output=response["Body"].read()

pprint(json.loads(output))<jupyter_output><empty_output><jupyter_text>Step 4.5 Delete the model<jupyter_code>model.delete_model()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08038598699391651, 'fraction_numerical': 0.012922173274596183, 'mean_word_length': 5.257810448936729, 'pattern_counts': {'":': 31, '<': 79, '<?xml version=': 0, '>': 80, 'https://': 18, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '97052', 'license': 'Apache-2.0', 'n_tokens_mistral': 6757, 'n_tokens_neox': 6354, 'n_words': 2682, 'path': 'aws_marketplace/using_model_packages/creative-writing-using-gpt-2-text-generation/creative-writing-using-gpt-2-text-generation.ipynb', 'repo_name': 'Ali-Parandeh/amazon-sagemaker-examples', 'size': '23863'}