<jupyter_start><jupyter_text>Snorkel Workshop: Extracting Spouse Relations  from the News Part 3: Training the Generative ModelNow, we'll train a model of the LFs to estimate their accuracies. Once the model is trained, we can combine the outputs of the LFs into a single, noise-aware training label set for our extractor. Intuitively, we'll model the LFs by observing how they overlap and conflict with each other.<jupyter_code>%load_ext autoreload
%autoreload 2
%matplotlib inline
import os
import re
import numpy as np

# Connect to the database backend and initalize a Snorkel session
from lib.init import *
from snorkel.models import candidate_subclass
from snorkel.annotations import load_gold_labels

from snorkel.lf_helpers import (
    get_left_tokens, get_right_tokens, get_between_tokens,
    get_text_between, get_tagged_text,
)

# initialize our candidate type definition
Spouse = candidate_subclass('Spouse', ['person1', 'person2'])

# gold (human-labeled) development set labels
L_gold_dev = load_gold_labels(session, annotator_name='gold', split=1)<jupyter_output><empty_output><jupyter_text>I. Loading Labeling Matricies First we'll load our label matrices from notebook 2<jupyter_code>from snorkel.annotations import LabelAnnotator

labeler = LabelAnnotator(lfs=[])
L_train = labeler.load_matrix(session, split=0)
L_dev   = labeler.load_matrix(session, split=1)<jupyter_output><empty_output><jupyter_text>Now we set up and run the hyperparameter search, training our model with different hyperparamters and picking the best model configuration to keep. We'll set the random seed to maintain reproducibility.Note that we are fitting our model's parameters to the training set generated by our labeling functions, while we are picking hyperparamters with respect to score over the development set labels which we created by hand.  II: Unifying supervision  Generative ModelIn data programming, we use a more sophisitcated model to unify our labeling functions. We know that these labeling functions will not be perfect, and some may be quite low-quality, so we will _model_ their accuracies with a generative model, which Snorkel will help us easily apply.This will ultimately produce a single set of **noise-aware training labels**, which we will then use to train an end extraction model in the next notebook.  For more technical details of this overall approach, see our [NIPS 2016 paper](https://arxiv.org/abs/1605.07723).**NOTE: Make sure you've written some of your own LFs in the previous notebook to get a decent score!!!**  1. Training the ModelWhen training the generative model, we'll tune our hyperparamters using a simple grid search. **Parameter Definitions**        epochs     A single pass through all the data in your training set    step_size  The factor by which we update model weights after computing the gradient    decay      The rate our update factor dimishes (decay) over time.<jupyter_code>from snorkel.learning import GenerativeModel
from snorkel.learning import RandomSearch

# use random search to optimize the generative model
param_ranges = {
    'step_size' : [1e-3, 1e-4, 1e-5, 1e-6],
    'decay'     : [0.9, 0.95],
    'epochs'    : [50, 100],
    'reg_param' : [1e-3],
}

model_class_params = {'lf_propensity' : False}

searcher = RandomSearch(GenerativeModel, param_ranges, L_train, n=5, model_class_params=model_class_params)
%time gen_model, run_stats = searcher.fit(L_dev, L_gold_dev)

run_stats<jupyter_output><empty_output><jupyter_text>2. Model AccuraciesThese are the weights learned for each LF<jupyter_code>x = L_dev.lf_stats(session, L_gold_dev)
train_marginals = gen_model.marginals(L_train)<jupyter_output><empty_output><jupyter_text>3. Plotting Marginal ProbabilitiesOne immediate santity check  you can peform using the generative model is to visually examine the distribution of predicted training marginals. Ideally, there should get a bimodal distribution with large seperation between each peaks, as shown below by the far right image. The corresponds to good signal for true and positive class labels. For your first Snorkel application, you'll probably see marginals closer to the far left or middle images. With all mass centered around p=0.5, you probably need to write more LFs got get more overall _coverage_. In the middle image, you have good negative coverage, but not enough positive LFs<jupyter_code>import matplotlib.pyplot as plt
plt.hist(train_marginals, bins=20, range=(0.0, 1.0))
plt.show()<jupyter_output><empty_output><jupyter_text>4. Generative Model Metrics<jupyter_code>dev_marginals = gen_model.marginals(L_dev)
_, _, _, _ = gen_model.error_analysis(session, L_dev, L_gold_dev)<jupyter_output><empty_output><jupyter_text>5. Saving our training labelsFinally, we'll save the `training_marginals`, which are our **"noise-aware training labels"**, so that we can use them in the next tutorial to train our end extraction model:<jupyter_code>from snorkel.annotations import save_marginals
%time save_marginals(session, L_train, train_marginals)<jupyter_output><empty_output><jupyter_text>III. Advanced Generative Model Features A. Structure Learning We may also want to include the dependencies between our LFs when training the generative model. Snorkel makes it easy to do this! `DependencySelector` runs a fast structure learning algorithm over the matrix of LF outputs to identify a set of likely dependencies.<jupyter_code>from snorkel.learning.structure import DependencySelector

MAX_DEPS = 5

ds = DependencySelector()
deps = ds.select(L_train, threshold=0.1)
deps = set(list(deps)[0:min(len(deps), MAX_DEPS)])

print("Using {} dependencies".format(len(deps)))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06699490065060665, 'fraction_numerical': 0.010374538420960084, 'mean_word_length': 5.844765342960288, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '421316', 'license': 'Apache-2.0', 'n_tokens_mistral': 1692, 'n_tokens_neox': 1614, 'n_words': 650, 'path': 'tutorials/workshop/Workshop_3_Generative_Model_Training.ipynb', 'repo_name': 'AndyLc/snorkel', 'size': '5687'}