<jupyter_start><jupyter_text>using the 124M model because collabatory has some issues with larger models  Mounting Google Drive<jupyter_code>gpt2.mount_gdrive()
import pandas as pd
import json<jupyter_output><empty_output><jupyter_text>code below takes facebook style `message_[0-9].json` messages downloaded from fb archive and translates them to gpt2 compatible txt files<jupyter_code>df = pd.DataFrame()
for i in range(1, 13):
    f = open("./message_{}.json".format(i))
    messages = json.load(f)
    df = df.append(pd.DataFrame(messages['messages']))
txt =df.apply(lambda x: "{}: {}".format(x['sender_name'],x['content']), axis=1)
#txt = txt.unique()
type(txt)
txt = pd.Series(txt.unique())

import numpy as np
np.savetxt(r'./messages.txt', txt.values, fmt="%s")
file_name = "messages.txt"<jupyter_output><empty_output><jupyter_text>gpt2 tuning- `restore_from`: `fresh` for new, `latest` for last checkpoint    - !!!!!!!for some reason restoring from checkpoints cannot be used with a new tensorflow session???- `sample_every` -> generates a sample every n steps- `print_every` -> steps to print- `learning_rate`     - Actually needs to be tuned for different hyperparams, 1e-4 is the default, but check https://github.com/minimaxir/gpt-2-simpledifferences-between-gpt-2-simple-and-other-text-generation-utilities<jupyter_code>sess = gpt2.start_tf_sess()


gpt2.finetune(sess,
              dataset=file_name,
              model_name='124M',
              steps=1000,
              restore_from='latest',
              run_name='run1',
              print_every=10,
              sample_every=200,
              save_every=100,
              overwrite=True
              )
gpt2.copy_checkpoint_to_gdrive(run_name='run1')
gpt2.copy_checkpoint_from_gdrive(run_name='run1')
sess = gpt2.start_tf_sess()
gpt2.load_gpt2(sess, run_name='run1')
gpt2.generate(sess, run_name='run1')
gpt2.generate(sess,
              length=250,
              temperature=0.7,
              prefix="mana",
              nsamples=5,
              batch_size=5
              )
gen_file = 'gpt2_gentext_{:%Y%m%d_%H%M%S}.txt'.format(datetime.utcnow())

gpt2.generate_to_file(sess,
                      destination_path=gen_file,
                      length=500,
                      temperature=0.7,
                      nsamples=100,
                      batch_size=20
                      )<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12224532224532225, 'fraction_numerical': 0.026611226611226613, 'mean_word_length': 2.9313725490196076, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 15, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '616674', 'license': 'MIT', 'n_tokens_mistral': 828, 'n_tokens_neox': 781, 'n_words': 155, 'path': 'GPT_2_Testing.ipynb', 'repo_name': 'peixian/gpt2-testing', 'size': '2405'}