<jupyter_start><jupyter_text>How to do it... 1. Import tensorflow<jupyter_code>import tensorflow as tf<jupyter_output><empty_output><jupyter_text>2. Import Transformers library<jupyter_code>from transformers import TFGPT2LMHeadModel, GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2-large")
GPT2 = TFGPT2LMHeadModel.from_pretrained("gpt2-large", pad_token_id=tokenizer.eos_token_id)<jupyter_output><empty_output><jupyter_text>3. Set seed for reproducibility<jupyter_code>SEED = 34
tf.random.set_seed(SEED)

# set maximum number of words in output text
MAX_LEN = 70<jupyter_output><empty_output><jupyter_text>4. Using greedy search, the word with the highest probability is predicted as the next word in the sequence. We can observe how this works in the below code block<jupyter_code>input_sequence = 'There are times when I am really tired of people, but I feel lonely too.'<jupyter_output><empty_output><jupyter_text>Once we have an input sequence, we can encode it and call a decode method:<jupyter_code># Encode context the generation is conditioned on
input_ids = tokenizer.encode(input_sequence, return_tensors='tf')

# generate test until the output length (which includes the context length) reaches 70
greedy_output = GPT2.generate(input_ids, max_length = MAX_LEN)

print("Output:\n" + 100 * '-')
print(tokenizer.decode(greedy_output[0], skip_special_tokens=True))<jupyter_output>Output:
----------------------------------------------------------------------------------------------------
There are times when I am really tired of people, but I feel lonely too. I feel like I'm alone in the world. I feel like I'm alone in my own body. I feel like I'm alone in my own mind. I feel like I'm alone in my own heart. I feel like I'm alone in my own mind<jupyter_text>One can see that this is repetitive and not ideal.  5. Beam search offers a solution, in beam search one tracks the alternative variants, so that more comparisons are possible:<jupyter_code># set return_num_sequences > 1
beam_outputs = GPT2.generate(input_ids, 
                             max_length=MAX_LEN,
                             num_beams = 5,
                             no_repeat_ngram_size = 2,
                             num_return_sequences = 5,
                             early_stopping= True)

print('')
print('Output:\n' + 100 * '-')

# We now have 5 output sequences
for i, beam_output in enumerate(beam_outputs):
    print(f'{i}: {tokenizer.decode(beam_output, skip_special_tokens=True)}')<jupyter_output>Output:
----------------------------------------------------------------------------------------------------
0: There are times when I am really tired of people, but I feel lonely too. I don't know what to do with myself."

"I feel like I can't do anything right now," she said. "I'm so tired."
1: There are times when I am really tired of people, but I feel lonely too. I don't know what to do with myself."

"I feel like I can't do anything right now," she says. "I'm so tired."
2: There are times when I am really tired of people, but I feel lonely too. I don't know what to do with myself."

"I feel like I can't do anything right now," she says. "I'm not sure what I'm supposed to be doing with my life."
3: There are times when I am really tired of people, but I feel lonely too. I don't know what to do with myself."

"I feel like I can't do anything right now," she says. "I'm not sure what I'm supposed to be doing."
4: There are times when I am really tired of people, but I feel lonely to[...]<jupyter_text>This provides a more diverse response, even though the messages tend to be the same. 6. We can now explore sampling - indeterministic decoding. So instead of following a strict path to find the end text with the highest probability, we rather randomly pick the next word by it's conditional probability distribution. A draw back to this is the risk of incoherent ramblings, so a temperature parameter is used to affect the probability mass distribution.<jupyter_code># use temperature to decrease the sensitivity to low probability candidates
sample_output = GPT2.generate(input_ids,
                              do_sample = True,
                              max_length = MAX_LEN,
                              top_k = 0,
                              temperature = 0.2)

print('Output:\n' + 100 * '-')
print(tokenizer.decode(sample_output[0], skip_special_tokens = True))<jupyter_output>Output:
----------------------------------------------------------------------------------------------------
There are times when I am really tired of people, but I feel lonely too. I feel like I'm alone in my own world. I feel like I'm alone in my own life. I feel like I'm alone in my own mind. I feel like I'm alone in my own heart. I feel like I'm alone in my own<jupyter_text>This one can be labelled the "senti" model, because it is clearly in its feels. 7. Observe the differences in raising the temperature now<jupyter_code>sample_output = GPT2.generate(input_ids,
                              do_sample = True,
                              max_length = MAX_LEN,
                              top_k = 0,
                              temperature = 0.8)

print('Output:\n' + 100 * '-')
print(tokenizer.decode(sample_output[0], skip_special_tokens = True))<jupyter_output>Output:
----------------------------------------------------------------------------------------------------
There are times when I am really tired of people, but I feel lonely too. I find it strange how the people around me seem to be always so nice. The only time I feel lonely is when I'm on the road. I can't be alone with my thoughts.

What are some of your favourite things to do in the area<jupyter_text>This is a stark difference, but feels like a train of thought. Beyond temperature, there are more ways to tune the output 8. In Top-K sampling the top k most likely next words are selected and the entire probability mass is shifted to these k words. This removes the low probablity words from occuring altogether<jupyter_code># sample from only top_k most likely words
sample_output = GPT2.generate(input_ids,
                              do_sample = True,
                              max_length = MAX_LEN,
                              top_k = 50)

print('Output:\n' + 100 * '-')
print(tokenizer.decode(sample_output[0], skip_special_tokens = True))<jupyter_output>Output:
----------------------------------------------------------------------------------------------------
There are times when I am really tired of people, but I feel lonely too. I go to a place where you can feel comfortable. It's a place where you can relax. But if you're so tired of going along with the rules, maybe I won't go. You know what? Maybe if I don't go, you won't<jupyter_text>This is beginning to look better.  9. Top-P sampling (nucleus sampling) is similar to Top-K, but instead of choosing the top-k most likely words, we choose the smallest set of words whose total probablity is larger than p, and then the entire probability mass is shifted to the words in this set. In top-p the size of words is dynamic where as in top-k is it static.<jupyter_code># sample from only top_k most likely words
sample_output = GPT2.generate(input_ids,
                              do_sample = True,
                              max_length = MAX_LEN,
                              top_p = 0.8,
                              top_k = 0)

print('Output:\n' + 100 * '-')
print(tokenizer.decode(sample_output[0], skip_special_tokens = True))<jupyter_output>Output:
----------------------------------------------------------------------------------------------------
There are times when I am really tired of people, but I feel lonely too. I feel like I should just be standing there, just sitting there. I know I'm not a danger to anybody. I just feel alone."<jupyter_text>Idk, kinda sus. 10. We can also combine these approaches and really let the model speak<jupyter_code># sample from only top_k most likely words
sample_outputs = GPT2.generate(input_ids,
                              do_sample = True,
                              max_length = 2*MAX_LEN, # to test how long we can generate coherent data
                              top_p = 0.85,
                              top_k = 50,
                              num_return_sequences = 5)

# We now have 5 output sequences
for i, sample_output in enumerate(sample_outputs):
    print(f'{i}: {tokenizer.decode(sample_output, skip_special_tokens=True)}')<jupyter_output>0: There are times when I am really tired of people, but I feel lonely too. I feel like, even though I don't have any friends, I'm still the same person that I am and I have been for 20 years. But I feel lonely sometimes. But I do get to have friends now and then. I still haven't gotten to have a real boyfriend."

While she didn't want to talk about her sex life, Kim also shared her opinions about being single, being single in a relationship, and having a child. "I do feel a lot of guilt," she explained. "But I also feel very happy and lucky and grateful that my life is so interesting."

1: There are times when I am really tired of people, but I feel lonely too. You don't have to be in love with me, you can just be in love with someone else, even if it's a friend."

Her boyfriend, whom she calls "Mr. Big" — his height, size, height and weight are similar to hers — doesn't mind the fact that she's with someone else; he wants her to be happy.

She likes that people don't know he's her bo[...]<jupyter_text>Looking alright, one can reasonably follow the core idea here. 11. We can use prompts directly from OpenAI's GPT-2 website and compare our results with a smaller local model to the full sized GPT-2 model.<jupyter_code>MAX_LEN = 500

prompt1 = 'In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.'

input_ids = tokenizer.encode(prompt1, return_tensors='tf')

sample_outputs = GPT2.generate(input_ids,
                               do_sample = True,
                               max_length = MAX_LEN,
                               top_k = 50,
                               top_p = 0.85)

for i, sample_output in enumerate(sample_outputs):
    print(f'{i}: {tokenizer.decode(sample_output, skip_special_tokens=True)}')<jupyter_output>0: In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.

"We didn't know that there were unicorns in this valley. We don't have any evidence that they speak English. But the people of this valley are extremely friendly and they gave us a lot of information about the local language," said Dr. <NAME>, the lead researcher of the research at the University of British Columbia in Canada.

"The way the unicorn spoke was very different from that of any other animals," he added. "They were very fast, very intelligent. And they were very, very gentle. They were very good with their hooves. They were very gentle. They would go up and hold on to us when we were walking down the street."

The researchers' next challenge was to find the unicorn herd. The valley was in Argentina, which is not very far from Chile, where[...]<jupyter_text>12. It is also possible to generate fake news with this<jupyter_code>MAX_LEN = 500

prompt2 = '<NAME> was caught shoplifting from Abercrombie and Fitch on Hollywood Boulevard today.'

input_ids = tokenizer.encode(prompt2, return_tensors='tf')

sample_outputs = GPT2.generate(input_ids,
                               do_sample = True,
                               max_length = MAX_LEN,
                               top_k = 50,
                               top_p = 0.85)

for i, sample_output in enumerate(sample_outputs):
    print(f'{i}: {tokenizer.decode(sample_output, skip_special_tokens=True)}')<jupyter_output>0: <NAME> was caught shoplifting from Abercrombie and Fitch on Hollywood Boulevard today. She was released on $7,500 bail with conditions of not drinking alcohol, not using drugs, and not committing a crime. (Photo: Courtesy/ABC7)

Hollywood Boulevard was closed off at Westwood Boulevard for more than an hour as police investigated the theft of designer clothing.

Police said a vehicle pulled over near Highland Boulevard and Beverly Boulevard and the suspect was arrested on suspicion of shoplifting a $7,500 watch and $12,000 in other designer clothes.

No arrests have been made in the case.

Hollywood Boulevard was closed for more than an hour due to the arrest of a woman on suspicion of shoplifting. (Photo: Courtesy/ABC7)

The incident happened just before noon at the store, located at 9400 Westwood Boulevard.

The store's hours are 6 a.m. to 11 p.m.<jupyter_text>13. What about some Lord of the Rings<jupyter_code>MAX_LEN = 500

prompt2 = '<NAME> Gimli advanced on the orcs, raising their weapons with a harrowing war cry.'

input_ids = tokenizer.encode(prompt2, return_tensors='tf')

sample_outputs = GPT2.generate(input_ids,
                               do_sample = True,
                               max_length = MAX_LEN,
                               top_k = 50,
                               top_p = 0.85)

for i, sample_output in enumerate(sample_outputs):
    print(f'{i}: {tokenizer.decode(sample_output, skip_special_tokens=True)}')<jupyter_output>0: Legolas and Gimli advanced on the orcs, raising their weapons with a harrowing war cry.

"Let us finish this," shouted Gimli, "And then we will come to Gondor."

And so they did, and Gimli led his brother and the Fellowship to Rivendell, where Frodo and Sam reached their camp on the banks of the river.

It was time for the rest of the Fellowship to join them, and they had gathered in the tent. The rest of the party had gathered around it as well, for they had been ordered to remain in the camp.

"We must come to our home soon," said Elrond. "There are great things to see, and to do."

"Then," said Gimli, "let us meet at the gate, at the Gate of Gondor, so that we may come and go as we will."

"And I?" asked Sam, but Elrond shook his head. "I will not go on this journey."

"Not until I have done so," said Elrond, "because it is better for you than for me."

"Do you love the land of Gondor," said Gimli, "or do you not?"

"I have many memories of Gondor," said Gimli, "but not all of th[...]
{'fraction_non_alphanumeric': 0.10301990391214826, 'fraction_numerical': 0.010501029512697324, 'mean_word_length': 2.6767600302800907, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 52, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '51237', 'license': 'MIT', 'n_tokens_mistral': 4026, 'n_tokens_neox': 3797, 'n_words': 1884, 'path': 'Chapter 10 - Transformers/Text Generation.ipynb', 'repo_name': 'Young-Picasso/TensorFlow2.x_Cookbook', 'size': '14591'}