<jupyter_start><jupyter_text>Snorkel Workshop: Extracting Spouse Relations  from the News Advanced Part 6:  Hyperparameter Tuning via Grid Search<jupyter_code>%load_ext autoreload
%autoreload 2
%matplotlib inline
import os
import numpy as np

# Connect to the database backend and initalize a Snorkel session
from lib.init import *<jupyter_output><empty_output><jupyter_text>We repeat our definition of the `Spouse` `Candidate` subclass, and load the test set:<jupyter_code>Spouse = candidate_subclass('Spouse', ['person1', 'person2'])<jupyter_output><empty_output><jupyter_text>I. Training a `SparseLogisticRegression` Discriminative ModelWe use the training marginals to train a discriminative model that classifies each `Candidate` as a true or false mention. We'll use a random hyperparameter search, evaluated on the development set labels, to find the best hyperparameters for our model. To run a hyperparameter search, we need labels for a development set. If they aren't already available, we can manually create labels using the Viewer.  Feature ExtractionInstead of using a deep learning approach to start, let's look at a standard sparse logistic regression model. First, we need to extract out features. This can take a while, but we only have to do it once!<jupyter_code>from snorkel.annotations import FeatureAnnotator
featurizer = FeatureAnnotator()
F_train = featurizer.load_matrix(session, split=0)
F_dev = featurizer.load_matrix(session, split=1)
F_test = featurizer.load_matrix(session, split=2)

if F_train.size == 0:    
    %time F_train = featurizer.apply(split=0, parallelism=1)
if F_dev.size == 0:     
    %time F_dev  = featurizer.apply_existing(split=1, parallelism=1)
if F_test.size == 0:
    %time F_test = featurizer.apply_existing(split=2, parallelism=1)

print F_train.shape
print F_dev.shape
print F_test.shape<jupyter_output><empty_output><jupyter_text>First, reload the training marginals:<jupyter_code>from snorkel.annotations import load_marginals
train_marginals = load_marginals(session, split=0)
import matplotlib.pyplot as plt
plt.hist(train_marginals, bins=20)
plt.show()<jupyter_output><empty_output><jupyter_text>Load our development data for tuning<jupyter_code>from snorkel.annotations import load_gold_labels

L_gold_dev = load_gold_labels(session, annotator_name='gold', split=1)
L_gold_dev.shape<jupyter_output><empty_output><jupyter_text>The following code performs model selection by tuning our learning algorithm's hyperparamters.<jupyter_code>from snorkel.learning import RandomSearch
from snorkel.learning import SparseLogisticRegression

seed = 1234
num_model_search = 10

# search over this parameter grid
param_grid = {}
param_grid['batch_size'] = [64, 128]
param_grid['lr']         = [1e-4, 1e-3, 1e-2]
param_grid['l1_penalty'] = [1e-6, 1e-4, 1e-2]
param_grid['l2_penalty'] = [1e-6, 1e-4, 1e-2]
param_grid['rebalance']  = [0.0, 0.5]

model_class_params = {
    'seed': seed,
    'n_threads':num_procs
}

model_hyperparams = {
    'n_epochs': 2000,
    'print_freq': 1,
    'dev_ckpt_delay': 0.5,
    'X_dev': F_dev,
    'Y_dev': L_gold_dev
}

searcher = RandomSearch(SparseLogisticRegression, param_grid, F_train, train_marginals,
                        n=num_model_search, seed=seed,
                        model_class_params=model_class_params,
                        model_hyperparams=model_hyperparams)

print "Discriminitive Model Parameter Space (seed={}):".format(seed)
for i, params in enumerate(searcher.search_space()):
    print i, params

disc_model, run_stats = searcher.fit(X_valid=F_dev, Y_valid=L_gold_dev, n_threads=1)
print run_stats<jupyter_output><empty_output><jupyter_text>Examining FeaturesExtracting features allows us to inspect and interperet our learned weights<jupyter_code>from lib.scoring import *
print_top_k_features(session, disc_model, F_train, top_k=25)<jupyter_output><empty_output><jupyter_text>Evaluate on Test Data<jupyter_code>from snorkel.annotations import load_gold_labels
L_gold_test = load_gold_labels(session, annotator_name='gold', split=2)
_, _, _, _ = disc_model.score(session, F_test, L_gold_test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0896351775791254, 'fraction_numerical': 0.015945880647499397, 'mean_word_length': 5.981450252951096, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '419769', 'license': 'Apache-2.0', 'n_tokens_mistral': 1425, 'n_tokens_neox': 1365, 'n_words': 357, 'path': 'tutorials/workshop/Workshop_6_Advanced_Grid_Search.ipynb', 'repo_name': 'dhimmel/snorkel', 'size': '4139'}