<jupyter_start><jupyter_text>Finetuning the gpt2 model<jupyter_code>import gpt_2_simple as gpt2
gpt2.download_gpt2(model_name='124M')
gpt2.mount_gdrive()
file_name = 'pride_and_prejudice.txt'
gpt2.copy_file_from_gdrive(file_name)
sess = gpt2.start_tf_sess()

gpt2.finetune(
    sess,
    dataset=file_name,
    model_name='124M',
    steps=1000,
    restore_from='fresh',
    run_name='run1',
    print_every=10,
    sample_every=200,
    save_every=500
)<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:17: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
Loading checkpoint models/124M/model.ckpt
INFO:tensorflow:Restoring parameters from models/124M/model.ckpt<jupyter_text>she will now make her opinions known to the whole of the family, and      to all their friends.      “At a time when so many middle-aged people are moving into      town, when many couples are making fortunes off each other, when      many professions of taste are forming in the society, I am      really anxious to find some time here in the course of the next three      months to write to my dear Elizabeth, to seek her informed opinion      on this happy event, and to recommend it to her husband’s conduct as well      as her own. I often tell people to be cautious when they see      connections of importance here. What is to be done after my death?      To go through the world in such a way as to be forgotten?”      Mr. Bennet replied that he would write again at length to write      very near to the last lines of his letter. Elizabeth cried      out in alarm, and while she remained, a sense of shame had      entered her of her being the less attentive companion she had been      when she first distinguished her acquaintance. Anxiety increased      every moment for the other to return to her senses, and      every opportunity for <NAME> to shine any brighter was given      by the very first letter.<jupyter_code># save model checkpoints to google drive:
gpt2.copy_checkpoint_to_gdrive(run_name='run1')<jupyter_output><empty_output><jupyter_text>You might have to restart colab when you reload/retrain the model (or find another workaround).<jupyter_code># colab restart suggested at this point
%tensorflow_version 1.x
!pip install -q gpt-2-simple
import gpt_2_simple as gpt2
file_name = 'pride_and_prejudice.txt'
# if you restart sometimes not necessary:
# gpt2.mount_gdrive()
# 1. copy checkpoint from google drive:
gpt2.copy_checkpoint_from_gdrive(run_name='run1')

# 2. continue training:
sess = gpt2.start_tf_sess()

gpt2.finetune(
    sess,
    dataset=file_name,
    model_name='124M',
    steps=500,
    restore_from='latest',
    run_name='run1',
    print_every=10,
    overwrite=True,
    sample_every=200,
    save_every=100
)
# copy checkpoints to google drive again:
gpt2.copy_checkpoint_to_gdrive(run_name='run1')<jupyter_output>WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:17: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
Loading checkpoint checkpoint/run1/model-1000
INFO:tensorflow:Restoring parameters from checkpoint/run1/model-1000<jupyter_text>Generating text<jupyter_code># colab restart suggested here
%tensorflow_version 1.x
!pip install -q gpt-2-simple
import gpt_2_simple as gpt2
file_name = 'pride_and_prejudice.txt'
# if you restart, sometimes not necessary:
gpt2.mount_gdrive()

gpt2.copy_checkpoint_from_gdrive(run_name='run1')
sess = gpt2.start_tf_sess()
gpt2.load_gpt2(sess, run_name='run1')<jupyter_output>Mounted at /content/drive
Loading checkpoint checkpoint/run1/model-1500
INFO:tensorflow:Restoring parameters from checkpoint/run1/model-1500<jupyter_text>Please note that you can adjust the length of the returned text (length parameter) in the cell below.<jupyter_code>import datetime
from google.colab import files

gen_file = 'gpt2_gentext_{:%Y%m%d_%H%M%S}.txt'.format(
    datetime.datetime.utcnow()
)

gpt2.generate_to_file(
  sess,
  destination_path=gen_file,
  #length=500,
  temperature=0.7,
  nsamples=100,
  batch_size=20
)
files.download(gen_file)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07669683257918553, 'fraction_numerical': 0.02737556561085973, 'mean_word_length': 4.817105263157894, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '599459', 'license': 'MIT', 'n_tokens_mistral': 1495, 'n_tokens_neox': 1419, 'n_words': 453, 'path': 'chapter10/Writing_a_popular_novel.ipynb', 'repo_name': 'lukecolpo/Artificial-Intelligence-with-Python-Cookbook', 'size': '4424'}