<jupyter_start><jupyter_text>RTE Demo<jupyter_code>import os, sys
from pathlib import Path

if not "cwd" in globals():
    cwd = Path(os.getcwd())
sys.path.insert(0, str(cwd.parents[0]))<jupyter_output><empty_output><jupyter_text>**Recognizing Textual Entailment (RTE)** is a natural language inference task that addresses whether a _premise_ sentence entails a _hypothesis_ sentence, with labels `entailment` (do the facts of the premise imply the facts of the hypothesis?) or `not_entailment`.In this notebook, we'll...* Write over a dozen, simple slicing functions based on an [external error analysis](https://arxiv.org/abs/1904.11544) to initialize our model* Load existing base architecture weights + fine-tune slice heads from pretrained models  Load data and task<jupyter_code>TASK_NAME = "RTE"
BERT_MODEL = "bert-large-cased"

dataloader_config = {
    "batch_size": 8,
    "data_dir": os.environ.get("SUPERGLUEDATA", os.path.join(str(cwd.parents[0]), "data")),
    "splits": ["train", "valid"],
}

trainer_config = {
    "lr": 2e-5,
    "optimizer": "adamax",
    "n_epochs": 15,
    "conter_unit": "epochs",
    "evaluation_freq": 0.25,
}
from dataloaders import get_dataloaders

dataloaders = get_dataloaders(
    task_name=TASK_NAME,
    tokenizer_name=BERT_MODEL,
    **dataloader_config
)
from superglue_tasks import task_funcs

task = task_funcs[TASK_NAME](BERT_MODEL)<jupyter_output>100%|██████████| 1242874899/1242874899 [01:18<00:00, 15835137.96B/s]<jupyter_text>Compare to vanilla model<jupyter_code>from snorkel.mtl.model import MultitaskModel
from snorkel.mtl.trainer import Trainer

vanilla_model = MultitaskModel(tasks=[task], device=-1, dataparallel=False)<jupyter_output><empty_output><jupyter_text>Uncomment the following to train the vanilla BERT model.<jupyter_code># trainer = Trainer(**trainer_config)
# trainer.train_model(model, dataloaders)<jupyter_output><empty_output><jupyter_text>Or, use the one we pretrained for you!<jupyter_code># If you're missing the model, uncomment this line:
# ! wget https://www.dropbox.com/s/t2ri9o0iz765hsn/RTE_bert.pth?dl=0 && mv RTE_bert.pth?dl=0 RTE_bert.pth

vanilla_model.load("RTE_bert.pth")
%%time
vanilla_model.score(dataloaders[1])<jupyter_output>CPU times: user 3min 49s, sys: 23.8 s, total: 4min 13s
Wall time: 1min 3s<jupyter_text>Apply SFsWe emphasize here that our _data slicing_ abstraction follows intuitive, programmer workflows!We rely on error buckets defined by [Kim et. al 2019](https://arxiv.org/pdf/1904.11544.pdf) to define our slices.Then, we apply quick-to-write, heuristics to target each of these buckets. Intuitively, these are slices that were important enough to be measured independently by researchers—so we'd like to write slicing functions to try and improve performance!<jupyter_code>from snorkel.slicing.sf import slicing_function<jupyter_output><empty_output><jupyter_text>Prepositions<jupyter_code>@slicing_function()
def slice_temporal_preposition(example):
    temporal_prepositions = ["after", "before", "past"]
    both_sentences = example.sentence1 + example.sentence2
    return any([p in both_sentences for p in temporal_prepositions])

@slicing_function()
def slice_possessive_preposition(example):
    possessive_prepositions = ["inside of", "with", "within"]
    both_sentences = example.sentence1 + example.sentence2
    return any([p in both_sentences for p in possessive_prepositions])<jupyter_output><empty_output><jupyter_text>Comparatives<jupyter_code>@slicing_function()
def slice_is_comparative(example):
    comparative_words = ["more", "less", "better", "worse", "bigger", "smaller"]
    both_sentences = example.sentence1 + example.sentence2
    return any([p in both_sentences for p in comparative_words])<jupyter_output><empty_output><jupyter_text>Quantification<jupyter_code>@slicing_function()
def slice_is_quantification(example):
    quantification_words = ["all", "some", "none"]
    both_sentences = example.sentence1 + example.sentence2
    return any([p in both_sentences for p in quantification_words])<jupyter_output><empty_output><jupyter_text>Wh-Words<jupyter_code>@slicing_function()
def slice_where(example):
    sentences = example.sentence1 + example.sentence2
    return "where" in sentences

@slicing_function()
def slice_who(example):
    sentences = example.sentence1 + example.sentence2
    return "who" in sentences

@slicing_function()
def slice_what(example):
    sentences = example.sentence1 + example.sentence2
    return "what" in sentences

@slicing_function()
def slice_when(example):
    sentences = example.sentence1 + example.sentence2
    return "when" in sentences<jupyter_output><empty_output><jupyter_text>Coordinating Conjunctions<jupyter_code>@slicing_function()
def slice_and(example):
    sentences = example.sentence1 + example.sentence2
    return "and" in sentences

@slicing_function()
def slice_but(example):
    sentences = example.sentence1 + example.sentence2
    return "but" in sentences

@slicing_function()
def slice_or(example):
    sentences = example.sentence1 + example.sentence2
    return "or" in sentences<jupyter_output><empty_output><jupyter_text>Definite-Indefinite ArticlesMultiple occurences of articles like `the` or `an` might refer to different entities—we try to heuristically capture this here!<jupyter_code>@slicing_function()
def slice_multiple_articles(example):
    sentences = example.sentence1 + example.sentence2
    multiple_indefinite = sum([int(x == "a") for x in sentences.split()]) > 1 \
        or sum([int(x == "an") for x in sentences.split()]) > 1
    multiple_definite = sum([int(x == "the") for x in sentences.split()]) > 1
    return multiple_indefinite or multiple_definite<jupyter_output><empty_output><jupyter_text>(Misc.) Sentence Length<jupyter_code>@slicing_function()
def slice_short_hypothesis(example, thresh=5):
    return len(example.sentence2.split()) < thresh

@slicing_function()
def slice_long_hypothesis(example, thresh=15):
    return len(example.sentence2.split()) > thresh

@slicing_function()
def slice_short_premise(example, thresh=10):
    return len(example.sentence1.split()) < thresh

@slicing_function()
def slice_long_premise(example, thresh=100):
    return len(example.sentence1.split()) > thresh<jupyter_output><empty_output><jupyter_text>Add slices to dataloaders and model<jupyter_code>slicing_functions = [
    slice_temporal_preposition,
    slice_possessive_preposition,
    slice_is_comparative,
    slice_is_quantification,
    slice_where,
    slice_who,
    slice_what,
    slice_when,
    slice_and,
    slice_or,
    slice_but,
    slice_multiple_articles,
    slice_short_hypothesis,
    slice_long_hypothesis,
    slice_short_premise,
    slice_long_premise
]
slice_names = [sf.name for sf in slicing_functions]
from snorkel.slicing.utils import add_slice_labels, convert_to_slice_tasks

# make slices tasks
slice_tasks = convert_to_slice_tasks(task, slice_names)
slice_tasks
from snorkel.slicing.apply import PandasSFApplier
from snorkel.slicing.utils import add_slice_labels
from utils import task_dataset_to_dataframe

applier = PandasSFApplier(slicing_functions)

# add slice labels
for dl in dataloaders:
    df = task_dataset_to_dataframe(dl.dataset)
    S_matrix = applier.apply(df)
    
    # updates dataloaders in place
    add_slice_labels(dl, task, S_matrix, slice_names)
slice_model = MultitaskModel(tasks=slice_tasks, device=-1, dataparallel=False)<jupyter_output><empty_output><jupyter_text>Load from pretrained BERTGiven that the `slice_model` shares the same backbone architecture as the `vanilla_model`, we can simply reload these backbone weights (pretrained on RTE), and then fine-tune the slicing heads! Load previous backbone weights...<jupyter_code>slice_model.load_state_dict(vanilla_model.collect_state_dict())<jupyter_output><empty_output><jupyter_text>And fine-tune the slice heads!<jupyter_code># trainer = Trainer(**trainer_config)
# trainer.train_model(slice_model, dataloaders)<jupyter_output><empty_output><jupyter_text>Or load our pretrained model.<jupyter_code># If you're missing the model, uncomment this line:
# ! wget https://www.dropbox.com/s/18ta5z3tzasba0m/RTE_slice_from_bert.pth?dl=0 && mv RTE_slice_from_bert.pth?dl=0 RTE_slice_from_bert.pth

slice_model.load("RTE_slice_from_bert.pth")
%%time
slice_model.score(dataloaders[1])<jupyter_output>/home/ubuntu/anaconda3/envs/snorkel/lib/python3.6/site-packages/snorkel/slicing/modules/slice_combiner.py:40: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
  for slice_ind_name in slice_ind_op_names
/home/ubuntu/anaconda3/envs/snorkel/lib/python3.6/site-packages/snorkel/slicing/modules/slice_combiner.py:47: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
  for slice_pred_name in slice_pred_op_names
{'fraction_non_alphanumeric': 0.10060701438848921, 'fraction_numerical': 0.017198741007194245, 'mean_word_length': 6.358974358974359, 'pattern_counts': {'":': 8, '<': 69, '<?xml version=': 0, '>': 71, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '417117', 'license': 'Apache-2.0', 'n_tokens_mistral': 3168, 'n_tokens_neox': 3003, 'n_words': 663, 'path': 'tutorials/RTE_slicing_tutorial.ipynb', 'repo_name': 'agentsolaris/xlnn', 'size': '8896'}