<jupyter_start><jupyter_text>To start off, we'll recreate one of the experiments that the Snorkel authors cite and evaluate how well that experiment justifies the use of Snorkel<jupyter_code>%load_ext autoreload
%autoreload 2
%matplotlib inline

import numpy as np
from snorkel import SnorkelSession

session = SnorkelSession()
from snorkel.models import candidate_subclass

ChemicalDisease = candidate_subclass('ChemicalDisease', ['chemical', 'disease'])

train = session.query(ChemicalDisease).filter(ChemicalDisease.split == 0).all()
dev = session.query(ChemicalDisease).filter(ChemicalDisease.split == 1).all()
test = session.query(ChemicalDisease).filter(ChemicalDisease.split == 2).all()

print('Training set:\t{0} candidates'.format(len(train)))
print('Dev set:\t{0} candidates'.format(len(dev)))
print('Test set:\t{0} candidates'.format(len(test)))
from snorkel.annotations import load_marginals
train_marginals = load_marginals(session, split=0)
from snorkel.annotations import load_gold_labels
L_gold_dev = load_gold_labels(session, annotator_name='gold', split=1)
from snorkel.learning.pytorch import LSTM_orig

train_kwargs = {
    'lr':              0.01,
    'embedding_dim':   100,
    'hidden_dim':      100,
    'n_epochs':        20,
    'dropout':         0.5,
    'rebalance':       0.25,
    'print_freq':      5,
}

lstm = LSTM_orig(n_threads=None)
lstm.train(train, train_marginals, X_dev=dev, Y_dev=L_gold_dev, **train_kwargs)
from load_external_annotations import load_external_labels
load_external_labels(session, ChemicalDisease, split=2, annotator='gold')
L_gold_test = load_gold_labels(session, annotator_name='gold', split=2)
L_gold_test
lstm.score(test, L_gold_test)
# find f1 score for a model that predicts "Yes" every time
tp,fp = 0,0
for i in L_gold_test:
    if i == 1:
        tp += 1
    else:
        fp += 1
precision = tp / (tp + fp)
recall = 1
f1 = 2/(1/precision + 1/recall)
print(precision, recall, f1)<jupyter_output>0.3236108154140941 1 0.488981824030883<jupyter_text>Might train this again later and try to get a worse result, but just note that even though the model is better, it's not a significant difference. The "all yes" model is also better than the published result. This means that we're not really learning much other than the fact that we should say yes to most results.<jupyter_code>precision<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10411418975650713, 'fraction_numerical': 0.031066330814441646, 'mean_word_length': 5.1735751295336785, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '128353', 'license': 'Apache-2.0', 'n_tokens_mistral': 852, 'n_tokens_neox': 781, 'n_words': 218, 'path': 'tutorials/cdr/1- Recreate Experiment and compare to baselines.ipynb', 'repo_name': 'cperreault11/snorkel-extraction', 'size': '2382'}