<jupyter_start><jupyter_text>Introduction to JumpStart - Text Generation ---Welcome to Amazon [SageMaker JumpStart](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-jumpstart.html)! You can use JumpStart to solve many Machine Learning tasks through one-click in SageMaker Studio, or through [SageMaker JumpStart API](https://sagemaker.readthedocs.io/en/stable/overview.htmluse-prebuilt-models-with-sagemaker-jumpstart). In this demo notebook, we demonstrate how to use the JumpStart API for Text Generation. Text generation is the task of generating text which appears indistinguishable from the human-written text. It is also sometimes known as "natural language generation". Here, we show how to use state-of-the-art pre-trained GPT models for Text Generation.--- 1. [Set Up](1.-Set-Up)2. [Select a model](2.-Select-a-model)3. [Retrieve JumpStart Artifacts & Deploy an Endpoint](3.-Retrieve-JumpStart-Artifacts-&-Deploy-an-Endpoint)4. [Query endpoint and parse response](4.-Query-endpoint-and-parse-response)5. [Clean up the endpoint](5.-Clean-up-the-endpoint) Note: This notebook was tested on ml.t3.medium instance in Amazon SageMaker Studio with Python 3 (Data Science) kernel and in Amazon SageMaker Notebook instance with conda_python3 kernel.  1. Set Up ---Before executing the notebook, there are some initial steps required for set up. This notebook requires latest version of sagemaker and ipywidgets---<jupyter_code>!pip install sagemaker ipywidgets --upgrade --quiet<jupyter_output><empty_output><jupyter_text>Permissions and environment variables---To host on Amazon SageMaker, we need to set up and authenticate the use of AWS services. Here, we use the execution role associated with the current notebook as the AWS account role with SageMaker access. ---<jupyter_code>import sagemaker, boto3, json
from sagemaker import get_execution_role

aws_role = get_execution_role()
aws_region = boto3.Session().region_name
sess = sagemaker.Session()<jupyter_output><empty_output><jupyter_text>2. Select a model***Here, we download jumpstart model_manifest file from the jumpstart s3 bucket, filter-out all the Text Generation models and select a model for inference. ***<jupyter_code>from ipywidgets import Dropdown

# download JumpStart model_manifest file.
boto3.client("s3").download_file(
    f"jumpstart-cache-prod-{aws_region}", "models_manifest.json", "models_manifest.json"
)
with open("models_manifest.json", "rb") as json_file:
    model_list = json.load(json_file)

# filter-out all the Text Generation models from the manifest list.
text_generation_models = []
for model in model_list:
    model_id = model["model_id"]
    if "-textgeneration-" in model_id and model_id not in text_generation_models:
        text_generation_models.append(model_id)

# display the model-ids in a dropdown to select a model for inference.
model_dropdown = Dropdown(
    options=text_generation_models,
    value="huggingface-textgeneration-gpt2",
    description="Select a model",
    style={"description_width": "initial"},
    layout={"width": "max-content"},
)<jupyter_output><empty_output><jupyter_text>Chose a model for Inference<jupyter_code>display(model_dropdown)
# model_version="*" fetches the latest version of the model
model_id, model_version = model_dropdown.value, "*"<jupyter_output><empty_output><jupyter_text>3. Retrieve JumpStart Artifacts & Deploy an Endpoint***Using JumpStart, we can perform inference on the pre-trained model, even without fine-tuning it first on a new dataset. We start by retrieving the `deploy_image_uri`, `deploy_source_uri`, and `model_uri` for the pre-trained model. To host the pre-trained model, we create an instance of [`sagemaker.model.Model`](https://sagemaker.readthedocs.io/en/stable/api/inference/model.html) and deploy it. This may take a few minutes.***<jupyter_code>from sagemaker import image_uris, model_uris, script_uris, hyperparameters
from sagemaker.model import Model
from sagemaker.predictor import Predictor
from sagemaker.utils import name_from_base


endpoint_name = name_from_base(f"jumpstart-example-infer-{model_id}")

inference_instance_type = "ml.p2.xlarge"

# Retrieve the inference docker container uri. This is the base HuggingFace container image for the default model above.
deploy_image_uri = image_uris.retrieve(
    region=None,
    framework=None,  # automatically inferred from model_id
    image_scope="inference",
    model_id=model_id,
    model_version=model_version,
    instance_type=inference_instance_type,
)

# Retrieve the inference script uri. This includes all dependencies and scripts for model loading, inference handling etc.
deploy_source_uri = script_uris.retrieve(
    model_id=model_id, model_version=model_version, script_scope="inference"
)


# Retrieve the model uri. This includes the pre-trained nvidia-ssd model and parameters.
model_uri = model_uris.retrieve(
    model_id=model_id, model_version=model_version, model_scope="inference"
)


# Create the SageMaker model instance
model = Model(
    image_uri=deploy_image_uri,
    source_dir=deploy_source_uri,
    model_data=model_uri,
    entry_point="inference.py",  # entry point file in source_dir and present in deploy_source_uri
    role=aws_role,
    predictor_cls=Predictor,
    name=endpoint_name,
)

# deploy the Model. Note that we need to pass Predictor class when we deploy model through Model class,
# for being able to run inference through the sagemaker API.
model_predictor = model.deploy(
    initial_instance_count=1,
    instance_type=inference_instance_type,
    predictor_cls=Predictor,
    endpoint_name=endpoint_name,
)<jupyter_output><empty_output><jupyter_text>4. Query endpoint and parse response---Input to the endpoint is any string of text dumped in json and encoded in `utf-8` format. Output of the endpoint is a `json` with generated text.---<jupyter_code>def query(model_predictor, text):
    """Query the model predictor."""

    encoded_text = json.dumps(text).encode("utf-8")

    query_response = model_predictor.predict(
        encoded_text,
        {
            "ContentType": "application/x-text",
            "Accept": "application/json",
        },
    )
    return query_response


def parse_response(query_response):
    """Parse response and return the generated text."""

    model_predictions = json.loads(query_response)
    generated_text = model_predictions["generated_text"]
    return generated_text<jupyter_output><empty_output><jupyter_text>---Below, we  put in some example input text. You can put in any text and the model predicts next words in the sequence. Longer sequences of text can be generated by calling the model repeatedly.---<jupyter_code>newline, bold, unbold = "\n", "\033[1m", "\033[0m"

text1 = "As far as I am concerned, I will"
text2 = "The movie is"

for text in [text1, text2]:
    query_response = query(model_predictor, text)
    generated_text = parse_response(query_response)
    print(f"Input text: {text}{newline}" f"Generated text: {bold}{generated_text}{unbold}{newline}")<jupyter_output><empty_output><jupyter_text>5. Clean up the endpoint<jupyter_code># Delete the SageMaker endpoint
model_predictor.delete_model()
model_predictor.delete_endpoint()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08974004424778761, 'fraction_numerical': 0.0055309734513274336, 'mean_word_length': 5.2299741602067185, 'pattern_counts': {'":': 4, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '261057', 'license': 'MIT-0', 'n_tokens_mistral': 2218, 'n_tokens_neox': 2119, 'n_words': 670, 'path': 'introduction_to_amazon_algorithms/jumpstart_text_generation/Amazon_JumpStart_Text_Generation.ipynb', 'repo_name': 'bolecodex/amazon-sagemaker-best-practice', 'size': '7232'}