<jupyter_start><jupyter_text>First we must install Pytorch<jupyter_code>!conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia -y<jupyter_output>Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.8.3
  latest version: 4.10.1

Please update conda by running

    $ conda update -n base -c defaults conda



# All requested packages already installed.<jupyter_text>Lets quickly look at a more naive approach to fine-tuning, one I originally explored.   One easy way to fine-tune small transformers is to use a library called Happytransformer. Happytransformer is a package built on top of the Hugging face transformer library. Using it, you can normally fine-tune small transformers very easiliy. Lets install it.<jupyter_code>!pip install transformers happytransformer<jupyter_output>Requirement already satisfied: transformers in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (4.6.1)
Requirement already satisfied: happytransformer in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (2.2.3)
Requirement already satisfied: huggingface-hub==0.0.8 in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (from transformers) (0.0.8)
Requirement already satisfied: tokenizers<0.11,>=0.10.1 in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (from transformers) (0.10.3)
Requirement already satisfied: numpy>=1.17 in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (from transformers) (1.20.2)
Requirement already satisfied: sacremoses in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (from transformers) (0.0.45)
Requirement already satisfied: requests in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (from transformer[...]<jupyter_text>We will be testing this method by fine-tunning models on Shakespeare on my RTX 3090.<jupyter_code>%env CUDA_VISIBLE_DEVICES=0
from happytransformer import HappyGeneration 
model_to_use = "EleutherAI/gpt-neo-125M"
#1.3B Won't work on 24GB or less cards
# model_to_use = "EleutherAI/gpt-neo-1.3B"

happy_gen = HappyGeneration("GPT-NEO", model_to_use)
happy_gen.train("train.csv")
#this is clean up free up GPU VRAM
try:
    del happy_gen
except:
    pass
import gc 
import torch
gc.collect()
torch.cuda.empty_cache()<jupyter_output><empty_output><jupyter_text>We can see that even the relatively small 125M model, that it takes 10GB of VRAM to fine-tune the model, and the 1.3B parameter model can't fit on 24GB, let alone the 2.7B model.  This was even with a batch size of 1  Is there hope to fine-tune these larger models then with consumer grade hardware then?  Yes, but we need to use a library called DeepSpeed  First we need to clone the DeepSpeed Repo, as we must build some optional items in the package from the source  DeepSpeed is a Deep Learning optimization library by Microsoft that allows researchers to more easily run and train larger models that they otherwise would not be able to.<jupyter_code>!git clone https://github.com/microsoft/DeepSpeed -b v0.4.0<jupyter_output>Cloning into 'DeepSpeed'...
remote: Enumerating objects: 9447, done.[K
remote: Counting objects: 100% (207/207), done.[K
remote: Compressing objects: 100% (151/151), done.[K
remote: Total 9447 (delta 103), reused 109 (delta 53), pack-reused 9240[K
Receiving objects: 100% (9447/9447), 18.52 MiB | 10.32 MiB/s, done.
Resolving deltas: 100% (6412/6412), done.
Note: switching to '2d302d6abb2cfa181f63320da3ed1be45e34ded3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false<jupyter_text>We are now going to run ls to see our folder structure, we should now see DeepSpeed, we are going to cd into that folder<jupyter_code>!ls
%cd DeepSpeed<jupyter_output>/mnt/shared_drive/projects/personal/finetune_vid/DeepSpeed<jupyter_text>If we run ls again, we will see the contents of the DeepSpeed repo<jupyter_code>!ls<jupyter_output>azure		    csrc	       install.sh   requirements  version.txt
bin		    deepspeed	       LICENSE	    SECURITY.md
CODE_OF_CONDUCT.md  DeepSpeedExamples  MANIFEST.in  setup.cfg
CODEOWNERS	    docker	       op_builder   setup.py
CONTRIBUTING.md     docs	       README.md    tests<jupyter_text>We are now going to install DeepSpeed from source using a flag to insure that all the needed ops are installed<jupyter_code>!DS_BUILD_OPS=1 pip install .<jupyter_output>Processing /mnt/shared_drive/projects/personal/finetune_vid/DeepSpeed
[33m  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.[0m
Requirement already satisfied: torch>=1.2 in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (from deepspeed==0.4.0+2d302d6) (1.8.1)
Requirement already satisfied: torchvision>=0.4.0 in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (from deepspeed==0.4.0+2d302d6) (0.9.1)
Requirement already satisfied: tqdm in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (from deepspeed==0.4.0+2d302d6) (4.49.0)
Requirement already satisfied: tensorboardX==1.8 in[...]<jupyter_text>We can now make sure that DeepSpeed is properly installed by running the snippet below.  All of the compatible ops should be installed.  Some are required that like cpu_adam and transformer.  Others may not be, such as async_io.  Personally all were compatible with my system but async.io<jupyter_code>!ds_report<jupyter_output>--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [92m[OKAY][0m
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [92m[YES][0m ...... [92m[OKAY][0m
fused_adam ............. [92m[YES][0m ...... [92m[OKAY][0m
fused_lamb ............. [92m[YES][0m ...... [92m[OKAY][0m
sparse_attn ............ [92m[YES][0m ...... [92m[OKAY][0m
transformer ............ [92m[YES][0m ...... [92m[OKAY][0m
stochastic_transformer . [92m[YES][0m ...... [92m[OKAY][0m
[93m [WARNING] [[...]<jupyter_text>Next we need to download the repo that will be actually finetuning the GPT Neo model using DeepSpeed.  First we need to exit the DeepSpeed repo<jupyter_code>%cd ..
!ls<jupyter_output>DeepSpeed  GPT_Neo_Fine-tune.ipynb  test  train.csv<jupyter_text>Now we clone the finetuning repo<jupyter_code>!git clone https://github.com/Xirider/finetune-gpt2xl<jupyter_output>Cloning into 'finetune-gpt2xl'...
remote: Enumerating objects: 354, done.[K
remote: Counting objects: 100% (50/50), done.[K
remote: Compressing objects: 100% (50/50), done.[K
remote: Total 354 (delta 2), reused 0 (delta 0), pack-reused 304[K
Receiving objects: 100% (354/354), 3.41 MiB | 3.23 MiB/s, done.
Resolving deltas: 100% (221/221), done.<jupyter_text>Now we need enter the pulled repo<jupyter_code>!ls
%cd finetune-gpt2xl<jupyter_output>/mnt/shared_drive/projects/personal/finetune_vid/finetune-gpt2xl<jupyter_text>Lastly, we need to download the datasets library that this repo uses.<jupyter_code>!pip install datasets<jupyter_output>Requirement already satisfied: datasets in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (1.8.0)
Requirement already satisfied: requests>=2.19.0 in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (from datasets) (2.25.1)
Requirement already satisfied: huggingface-hub<0.1.0 in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (from datasets) (0.0.8)
Requirement already satisfied: pyarrow<4.0.0,>=1.0.0 in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (from datasets) (3.0.0)
Requirement already satisfied: importlib-metadata in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (from datasets) (3.10.0)
Requirement already satisfied: xxhash in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (from datasets) (2.0.2)
Requirement already satisfied: numpy>=1.17 in /home/blake/anaconda3/envs/gptneo_finetuned/lib/python3.7/site-packages (from datasets) (1.2[...]<jupyter_text>At this point we are able to finetune GPT Neo(including 2.7B) and other GPT models  For GPT NEO 2.7B parameters, we need a high end machine.  Roughly 70GB of RAM is the minimum required for it, along with roughly 16GB of VRAM. GPT Neo 1.3B and other smaller GPT2 models don't have as high of requirements. This can rented for an ok price from a cloud provider if you dont have a powerful enough machine.  Lets now finetune model with the provided Shakespeare dataset with the example flags<jupyter_code>!deepspeed --num_gpus=1 run_clm.py \
--deepspeed ds_config_gptneo.json \
--model_name_or_path EleutherAI/gpt-neo-1.3B \
--train_file train.csv \
--validation_file validation.csv \
--do_train \
--do_eval \
--fp16 \
--overwrite_cache \
--evaluation_strategy="steps" \
--output_dir finetuned \
--num_train_epochs 1 \
--eval_steps 15 \
--gradient_accumulation_steps 2 \
--per_device_train_batch_size 4 \
--use_fast_tokenizer False \
--learning_rate 5e-06 \
--warmup_steps 10<jupyter_output>[2021-06-14 15:33:34,602] [WARNING] [runner.py:122:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2021-06-14 15:33:34,610] [INFO] [runner.py:360:main] cmd = /home/blake/anaconda3/envs/gptneo_finetuned/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=127.0.0.1 --master_port=29500 run_clm.py --deepspeed ds_config_gptneo.json --model_name_or_path EleutherAI/gpt-neo-1.3B --train_file train.csv --validation_file validation.csv --do_train --do_eval --fp16 --overwrite_cache --evaluation_strategy=steps --output_dir finetuned --num_train_epochs 1 --eval_steps 15 --gradient_accumulation_steps 2 --per_device_train_batch_size 4 --use_fast_tokenizer False --learning_rate 5e-06 --warmup_steps 10
[2021-06-14 15:33:35,151] [INFO] [launch.py:80:main] WORLD INFO DICT: {'localhost': [0]}
[2021-06-14 15:33:35,152] [INFO] [launch.py:89:main] nnodes=1, num_local_procs=1, node_rank=0
[2021-06-14 15:33:35,152] [INFO[...]
{'fraction_non_alphanumeric': 0.13159746657283602, 'fraction_numerical': 0.04864532019704434, 'mean_word_length': 4.082253017434064, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 50, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '623042', 'license': 'Unlicense', 'n_tokens_mistral': 4223, 'n_tokens_neox': 3838, 'n_words': 1107, 'path': 'gpt-neo-test/fine-tuning/fine_tuning.ipynb', 'repo_name': 'Zeno-Paukner/cellarius', 'size': '11368'}