<jupyter_start><jupyter_text>**This notebook combines running training, inference & ensembling for all models. If you would run it like this it would run about 4 days, so you can just copy out the models you want one by one and run them individually! I have uploads of all the weights & features on kaggle if you want to make use of them. Let me know if you have any questions!** 🥶  Data Preparation<jupyter_code># Clone the repo; Get in the data
!cd ../; git clone -b master https://github.com/Muennighoff/vilio.git

# Copy in the data you want to run; For simplification I just copy in all (If you run this on kaggle it might crash)
# Refer to the extraction notebook if you havn't run extraction yet!
!cp -r ../input/hmtsvfeats/* ../vilio/data/
!cp -r ../input/hmtsvfeats/* ../vilio/ernie-vil/data/hm/
# LMDB feats only used for V-Model
!cp -r ../input/hmfeatureszipfin/detectron.lmdb ../vilio/data/

# Copy in the hateful memes data from uploading / downloading it
# If you do not have enough disk space split up the training of models!
# If you downloaded the updated HM data; you just need to copy in everything in the data folder (img, .jsonl's)
# Replace the hatefulmemes below with the name of your uploaded data
!cp -r ../input/hatefulmemes/data/* ../vilio/data/
!cp -r ../input/hatefulmemes/data/* ../vilio/ernie-vil/data/hm/

# Copy in the pretrained models depending on which model you want to run; Again just putting all here
# I have uploaded them all to kaggle so feel free to download from the below public dataets :)


# O
!cp ../input/oscarvglarge/large-vg-labels/ep_20_590000/pytorch_model.bin ../vilio/data/

# U 
!cp ../input/uniterlarge/uniter-large.pt ../vilio/data/

# V
!cp ../input/vbpretrainedfb/model.pth ../vilio/data/


# ES
!cp -r ../input/erniesmall/ ../vilio/ernie-vil/data/
!cp -r ../input/erniesmallvcr/ ../vilio/ernie-vil/data/

# EL
!cp -r ../input/ernielarge/ ../vilio/ernie-vil/data/
!cp -r ../input/ernielargevcr/ ../vilio/ernie-vil/data/<jupyter_output><empty_output><jupyter_text>Run PyTorch Models<jupyter_code># Install the PyTorch requirements
!cd ../vilio; pip install -r requirements.txt
# O
!cd ../vilio; bash bash/training/O/hm_O.sh
# U
!cd ../vilio; bash bash/training/U/hm_U.sh
# V
# Make sure we have the most updated torch with SWA
!pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html
!cd ../vilio; bash bash/training/V/hm_V.sh<jupyter_output><empty_output><jupyter_text>Run PaddlePaddle Models<jupyter_code># Install PaddlePaddle Requirements
!cd ../vilio/ernie-vil; pip install -r requirements.txt
# ES
!cd ../vilio/ernie-vil; bash bash/training/ES/hm_ES.sh
# EL
!cd ../vilio/ernie-vil; bash bash/training/EL/hm_EL.sh<jupyter_output><empty_output><jupyter_text>Combine & Output<jupyter_code># Reinstall normal requirements
!cd ../vilio; pip install -r requirements.txt
!cp -r ../vilio/data/O*/*.csv ../vilio/data
!cp -r ../vilio/data/U*/*.csv ../vilio/data
!cp -r ../vilio/data/V*/*.csv ../vilio/data

!cp -r ../vilio/ernie-vil/data/hm/ES*/*.csv ../vilio/data
!cp -r ../vilio/ernie-vil/data/hm/EL*/*.csv ../vilio/data
!cd ../vilio; bash bash/hm_ens.sh
# Save final csvs
!cp -r ../vilio/data/FIN*.csv ./<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14325323475046212, 'fraction_numerical': 0.006469500924214418, 'mean_word_length': 3.4786206896551723, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '65375', 'license': 'MIT', 'n_tokens_mistral': 1247, 'n_tokens_neox': 1163, 'n_words': 368, 'path': 'notebooks/hm_pipeline.ipynb', 'repo_name': 'yatingtian/vilio', 'size': '3246'}