<jupyter_start><jupyter_text>Setup working directory<jupyter_code>import os
from google.colab import drive 
drive.mount('/content/drive')
WORKING_DIR=os.path.join('/content', 'drive', 'My Drive', 'Colab Notebooks', 'NLU')
os.path.exists(WORKING_DIR)
cd $WORKING_DIR<jupyter_output>/content/drive/My Drive/Colab Notebooks/NLU<jupyter_text>Clone transformers repo and checkout version 2.7.0<jupyter_code>!rm -rf transformers
!git clone https://github.com/huggingface/transformers
cd transformers/
!git checkout -f v2.7.0<jupyter_output>Checking out files: 100% (475/475), done.
Note: checking out 'v2.7.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 6f5a12a5 Release: v2.7.0<jupyter_text>Install packages required for running `run_glue.py`<jupyter_code>!pip install -r ./examples/requirements.txt
!pip install boto3 filelock requests tqdm sentencepiece sacremoses tokenizers<jupyter_output>Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (1.13.4)
Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (3.0.12)
Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (2.23.0)
Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (4.41.1)
Requirement already satisfied: sentencepiece in /usr/local/lib/python3.6/dist-packages (0.1.90)
Requirement already satisfied: sacremoses in /usr/local/lib/python3.6/dist-packages (0.0.43)
Requirement already satisfied: tokenizers in /usr/local/lib/python3.6/dist-packages (0.7.0)
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3) (0.3.3)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3) (0.9.5)
Requirement already satisfied: botocore<1.17.0,>=1.16.4 in /usr/local/lib/python3.6/dist-packages (from boto3) (1.16.4)
Requir[...]<jupyter_text>Download GLUE data<jupyter_code>GLUE_DIR="/tmp/data"
!echo $GLUE_DIR
!python ./utils/download_glue_data.py --help
!python ./utils/download_glue_data.py --data_dir $GLUE_DIR --tasks MRPC<jupyter_output>Processing MRPC...
Local MRPC data not specified, downloading data from https://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_train.txt
	Completed!<jupyter_text>Compute RoBERTa score on MRPC task<jupyter_code>TASK_NAME="MRPC"
!echo $TASK_NAME
!echo $PYTHONPATH
%set_env PYTHONPATH=$WORKING_DIR/transformers/src:/env/python
!echo $PYTHONPATH
MODEL_DIR = os.path.join('/content', 'drive', 'My Drive', 'Colab Notebooks', 'NLU', 'models', 'random_0.15')
os.path.exists(MODEL_DIR)
!python ./examples/run_glue.py \
    --model_type roberta \
    --model_name_or_path /content/drive/My\ Drive/Colab\ Notebooks/NLU/models/random_0.15 \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
    --data_dir $GLUE_DIR/$TASK_NAME \
    --max_seq_length 128 \
    --per_gpu_eval_batch_size=64   \
    --per_gpu_train_batch_size=64   \
    --learning_rate 1e-4 \
    --num_train_epochs 3 \
    --output_dir glue_model \
    --overwrite_output_dir<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12481983280484289, 'fraction_numerical': 0.03603343903142116, 'mean_word_length': 4.697865353037767, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '21718', 'license': 'MIT', 'n_tokens_mistral': 1305, 'n_tokens_neox': 1257, 'n_words': 288, 'path': 'notebooks_GLUE/notebooks_roberta_jumbled_token_discrimination_lr_e-4_prob_0.15/mrpc.ipynb', 'repo_name': 'subhadarship/learning-to-unjumble', 'size': '3469'}