<jupyter_start><jupyter_text>clone and cd into repo, nshepperd's fork https://github.com/nshepperd/gpt-2<jupyter_code>import tensorflow as tf
tf.__version__
!git clone https://github.com/ncoop57/gpt-2.git ./data/gpt-2
cd data/gpt-2<jupyter_output>/tf/prototypes/gpt-2/data/gpt-2<jupyter_text>Install requirements<jupyter_code>!pip install -r requirements.txt<jupyter_output>Collecting fire>=0.1.3 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/5a/b7/205702f348aab198baecd1d8344a90748cb68f53bdcd1cc30cbc08e47d3e/fire-0.1.3.tar.gz
Collecting regex==2017.4.5 (from -r requirements.txt (line 2))
[?25l  Downloading https://files.pythonhosted.org/packages/36/62/c0c0d762ffd4ffaf39f372eb8561b8d491a11ace5a7884610424a8b40f95/regex-2017.04.05.tar.gz (601kB)
[K     |████████████████████████████████| 604kB 2.3MB/s eta 0:00:01
[?25hCollecting requests==2.21.0 (from -r requirements.txt (line 3))
[?25l  Downloading https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl (57kB)
[K     |████████████████████████████████| 61kB 3.9MB/s eta 0:00:011
[?25hCollecting tqdm==4.31.1 (from -r requirements.txt (line 4))
[?25l  Downloading https://files.pythonhosted.org/packages/6c/4b/c38b5144cf167c4f52288517436ccafefe9dc01b8d1c190e18a6b154cd4a/tqdm-4.3[...]<jupyter_text>Download the model data<jupyter_code>!python download_model.py 345M<jupyter_output>Fetching checkpoint: 1.00kit [00:00, 276kit/s]                                                      
Fetching encoder.json: 1.04Mit [00:00, 1.75Mit/s]                                                   
Fetching hparams.json: 1.00kit [00:00, 222kit/s]                                                    
Fetching model.ckpt.data-00000-of-00001: 1.42Git [05:48, 4.07Mit/s]                                 
Fetching model.ckpt.index: 11.0kit [00:00, 2.14Mit/s]                                               
Fetching model.ckpt.meta: 927kit [00:00, 1.55Mit/s]                                                 
Fetching vocab.bpe: 457kit [00:00, 1.15Mit/s]<jupyter_text>encoding<jupyter_code>!export PYTHONIOENCODING=UTF-8
!apt install wget -y<jupyter_output>Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  wget
0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded.
Need to get 299 kB of archives.
After this operation, 905 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 wget amd64 1.17.1-1ubuntu1.5 [299 kB]
Fetched 299 kB in 0s (1242 kB/s)[33m
debconf: delaying package configuration, since apt-utils is not installed

7[0;23r8[1ASelecting previously unselected package wget.
(Reading database ... 15495 files and directories currently installed.)
Preparing to unpack .../wget_1.17.1-1ubuntu1.5_amd64.deb ...
7[24;0f[42m[30mProgress: [  0%][49m[39m [..........................................................] 87[24;0f[42m[30mProgress: [ 16%][49m[39m [##########................................................] 8Unpacking wget (1.17.1-1ubuntu1.5) ...
7[24;0f[42m[30mP[...]<jupyter_text>Let's get our train on! In this case the file is A Tale of Two Cities (<NAME>) from Project Gutenberg. To change the dataset GPT-2 models will fine-tune on, change this URL to another .txt file, and change corresponding part of the next cell. Note that you can use small datasets if you want but you will have to be sure not to run the fine-tuning for too long or you will overfit badly. Roughly, expect interesting results within minutes to hours in the 1-10s of megabyte ballpark, and below this you may want to stop the run early as fine-tuning can be very fast.<jupyter_code>!wget http://groups.inf.ed.ac.uk/cup/javaGithub/java_projects.tar.gz
!tar xzf java_projects.tar.gz
!ls
!pip install toposort<jupyter_output>Requirement already satisfied: toposort in /usr/local/lib/python3.6/dist-packages (1.5)<jupyter_text>Start training, add --model_name '345M' to use 345 model<jupyter_code>!PYTHONPATH=src ./train.py --dataset ../java_projects --model_name '345M'<jupyter_output>2019-05-28 23:03:05.387967: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz
2019-05-28 23:03:05.388196: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x2086680 executing computations on platform Host. Devices:
2019-05-28 23:03:05.388230: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
2019-05-28 23:03:05.559017: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-05-28 23:03:05.559523: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x2085fa0 executing computations on platform CUDA. Devices:
2019-05-28 23:03:05.559555: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): Tesla T4, Compute Capability 7.5
2019-05-28 23:03:05.559970: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found de[...]<jupyter_text>Save our checkpoints to start training again later<jupyter_code>!cp -r /content/gpt-2/checkpoint/ /content/drive/My\ Drive/<jupyter_output>cp: cannot create directory '/content/drive/My Drive/': No such file or directory<jupyter_text>Load your trained model for use in sampling below (117M or 345M)<jupyter_code>!cp -r /content/gpt-2/checkpoint/run1/* /content/gpt-2/models/117M/
!cp -r /content/gpt-2/checkpoint/run1/* /content/gpt-2/models/345M/<jupyter_output><empty_output><jupyter_text>Generate conditional samples from the model given a prompt you provide -  change top-k hyperparameter if desired (default is 40),  if you're using 345M, add "--model-name 345M"<jupyter_code>!python3 src/interactive_conditional_samples.py --top_k 40 --model_name "345M"<jupyter_output>2019-05-29 01:17:42.237182: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz
2019-05-29 01:17:42.237511: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x33f0680 executing computations on platform Host. Devices:
2019-05-29 01:17:42.237554: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
2019-05-29 01:17:42.432852: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-05-29 01:17:42.433377: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x33effa0 executing computations on platform CUDA. Devices:
2019-05-29 01:17:42.433421: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): Tesla T4, Compute Capability 7.5
2019-05-29 01:17:42.433828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found de[...]<jupyter_text>To check flag descriptions, use:<jupyter_code>!python3 src/interactive_conditional_samples.py -- --help<jupyter_output><empty_output><jupyter_text>Generate unconditional samples from the model,  if you're using 345M, add "--model-name 345M"<jupyter_code>!python3 src/generate_unconditional_samples.py --model_name "345M" | tee /tmp/samples<jupyter_output><empty_output><jupyter_text>To check flag descriptions, use:<jupyter_code>!python3 src/generate_unconditional_samples.py -- --help<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1380359978824775, 'fraction_numerical': 0.11487559555320276, 'mean_word_length': 4.075218267293486, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 47, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '510831', 'license': 'MIT', 'n_tokens_mistral': 3289, 'n_tokens_neox': 2662, 'n_words': 621, 'path': 'main/nbs/poc/gpt-2/fine_tune_gpt-2.ipynb', 'repo_name': 'jason424217/Artificial-Code-Gen', 'size': '7565'}