<jupyter_start><jupyter_text>TextAttack End-to-EndThis tutorial provides a broad end-to-end overview of training, evaluating, and attacking a model using TextAttack. [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1cBRUj2l0m8o81vJGGFgO-o_zDLj24M5Y?usp=sharing)[![View Source on GitHub](https://img.shields.io/badge/github-view%20source-black.svg)](https://github.com/QData/TextAttack/blob/master/docs/examples/1_Introduction_and_Transformations.ipynb)  TrainingFirst, we're going to train a model. TextAttack integrates directly with [transformers](https://github.com/huggingface/transformers/) and [nlp](https://github.com/huggingface/nlp) to train any of the `transformers` pre-trained models on datasets from `nlp`. Let's use the SNLI textual entailment dataset: it's relatively short (in word count, at least), and showcases a lot of the features of `textattack train`. Let's take a look at the dataset using `textattack peek-dataset`:<jupyter_code>!textattack peek-dataset --dataset-from-nlp snli<jupyter_output>[34;1mtextattack[0m: Loading [94mnlp[0m dataset [94msnli[0m, split [94mtrain[0m.
[34;1mtextattack[0m: Number of samples: [94m550152[0m
[34;1mtextattack[0m: Number of words per input:
[34;1mtextattack[0m: 	total:   [94m11194676[0m
[34;1mtextattack[0m: 	mean:    [94m20.35[0m
[34;1mtextattack[0m: 	std:     [94m7.01[0m
[34;1mtextattack[0m: 	min:     [94m4[0m
[34;1mtextattack[0m: 	max:     [94m114[0m
[34;1mtextattack[0m: Dataset lowercased: [94mFalse[0m
[34;1mtextattack[0m: First sample:
Premise: A person on a horse jumps over a broken down airplane.
Hypothesis: A person is training his horse for a competition. 

[34;1mtextattack[0m: Last sample:
Premise: A man is surfing in a bodysuit in beautiful blue water.
Hypothesis: On the beautiful blue water there is a man in a bodysuit surfing. 

[34;1mtextattack[0m: Found 4 distinct outputs.
[34;1mtextattack[0m: Most common outputs:
	 0      (183416)
	 2      (183187)
	 1      (182764)
	 -1     (785)<jupyter_text>The dataset looks good! It's not lowercased already, so we'll make sure our model is cased. Looks like there are some missing (-1) labels, so we need to filter those out. The longest input is 114 words, so we can cap our maximum sequence length (`--max-length`) at 128.We'll train [`distilbert-base-cased`](https://huggingface.co/transformers/model_doc/distilbert.html), since it's a relatively small model, and a good example of how we integrate with `transformers`.So we have our command:```bashtextattack train                  \  Train a model with TextAttack    --model distilbert-base-cased \  Using distilbert, cased version, from `transformers`    --dataset snli                \  On the SNLI dataset    --max-length 128              \  With a maximum sequence length of 128    --batch-size 256              \  And a batch size of 256    --epochs 3                    \  For 3 epochs    --allowed-labels 0 1 2           And only allow labels 0, 1, 2 (filter out -1!)```Now let's run it:<jupyter_code>!textattack train --model distilbert-base-cased --dataset snli --max-length 128 --batch-size 256 --epochs 3 --allowed-labels 0 1 2<jupyter_output>[34;1mtextattack[0m: WARNING: TextAttack's model training feature is in beta. Please report any issues on our Github page, https://github.com/QData/TextAttack/issues.
[34;1mtextattack[0m: Writing logs to /p/qdata/jm8wx/research/text_attacks/textattack/outputs/training/distilbert-base-cased-snli-2020-06-24-14:03/log.txt.
[34;1mtextattack[0m: Loading [94mnlp[0m dataset [94msnli[0m, split [94mtrain[0m.
[34;1mtextattack[0m: Loading [94mnlp[0m dataset [94msnli[0m, split [94mvalidation[0m.
[34;1mtextattack[0m: Filtering samples with labels outside of [0, 1, 2].
[34;1mtextattack[0m: Filtered 550152 train samples to 549367 points.
[34;1mtextattack[0m: Filtered 10000 dev samples to 9842 points.
[34;1mtextattack[0m: Loaded dataset. Found: 3 labels: ([0, 1, 2])
[34;1mtextattack[0m: Loading transformers AutoModelForSequenceClassification: distilbert-base-cased
AutoTokenizer using fast tokenizer
[34;1mtextattack[0m: Tokenizing training data. (len: 549367)
[34;1mtext[...]<jupyter_text>EvaluationWe successfully fine-tuned `distilbert-base-cased` for 5 epochs. Now let's evaluate it using `textattack eval`. This is as simple as providing the path to the pretrained model to `--model`, along with the number of evaluation samples. `textattack eval` will automatically load the evaluation data from training:<jupyter_code>!textattack eval --num-examples 1000 --model /p/qdata/jm8wx/research/text_attacks/textattack/outputs/training/distilbert-base-cased-snli-2020-06-24-14:03/<jupyter_output>[34;1mtextattack[0m: Loading transformers AutoModelForSequenceClassification: /p/qdata/jm8wx/research/text_attacks/textattack/outputs/training/distilbert-base-cased-snli-2020-06-24-14:03/
[34;1mtextattack[0m: Loading [94mnlp[0m dataset [94msnli[0m, split [94mvalidation[0m.
Successes 868/1000 ([94m86.80%[0m)<jupyter_text>Awesome -- we were able to train a model up to 86.8% validation-set accuracy– with only a single command!  AttackFinally, let's attack our pre-trained model. We can do this the same way as before (by providing the path to the pretrained model to `--model`). For our attack, let's use the "TextFooler" attack recipe, from the paper ["Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment" (Jin et al, 2019)](https://arxiv.org/abs/1907.11932). We can do this by passing `--recipe textfooler` to `textattack attack`.> *Warning*: We're printing out 1000 examples and, if the attack succeeds, their perturbations. The output of this command is going to be quite long!<jupyter_code>!textattack attack --recipe textfooler --num-examples 1000 --model /p/qdata/jm8wx/research/text_attacks/textattack/outputs/training/distilbert-base-cased-snli-2020-06-24-14:03/<jupyter_output>[34;1mtextattack[0m: Loading transformers AutoModelForSequenceClassification: /p/qdata/jm8wx/research/text_attacks/textattack/outputs/training/distilbert-base-cased-snli-2020-06-24-14:03/
Using /u/jm8wx/.cache/tensorflow-hub to cache modules.
[34;1mtextattack[0m: Goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'> compatible with model DistilBertForSequenceClassification.
Attack(
  (search_method): GreedyWordSwapWIR(
    (wir_method):  unk
  )
  (goal_function):  UntargetedClassification
  (transformation):  WordSwapEmbedding(
    (max_candidates):  50
    (embedding_type):  paragramcf
  )
  (constraints): 
    (0): WordEmbeddingDistance(
        (embedding_type):  paragramcf
        (min_cos_sim):  0.5
        (cased):  False
        (include_unknown_words):  True
      )
    (1): PartOfSpeech(
        (tagger_type):  nltk
        (tagset):  universal
        (allow_verb_noun_swap):  True
      )
    (2): UniversalSen[...]
{'fraction_non_alphanumeric': 0.11981239340534394, 'fraction_numerical': 0.05955088118249005, 'mean_word_length': 4.652208835341366, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 15, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '121652', 'license': 'MIT', 'n_tokens_mistral': 2777, 'n_tokens_neox': 2330, 'n_words': 655, 'path': 'docs/examples/0_End_to_End.ipynb', 'repo_name': 'toriving/TextAttack', 'size': '7036'}