<jupyter_start><jupyter_text>Training a Sentiment Analysis LSTM Using Noisy Crowd Labels This is a version of the [crowdsourcing tutorial that uses PySpark](https://github.com/HazyResearch/snorkel/blob/master/tutorials/crowdsourcing/Crowdsourced_Sentiment_Analysis.ipynb) using Pandas instead of SparkSQL.In this tutorial, we'll provide a simple walkthrough of how to use Snorkel to resolve conflicts in a noisy crowdsourced dataset for a sentiment analysis task, and then use these denoised labels to train an LSTM sentiment analysis model which can be applied to new, unseen data to automatically make predictions!1. Creating basic Snorkel objects: `Candidates`, `Contexts`, and `Labels`2. Training the `GenerativeModel` to resolve labeling conflicts3. Training a simple LSTM sentiment analysis model, which can then be used on new, unseen data!Note that this is a simple tutorial meant to give an overview of the mechanics of using Snorkel-- we'll note places where more careful fine-tuning could be done!  Task Detail: Weather Sentiments in TweetsIn this tutorial we focus on the [Weather sentiment](https://www.crowdflower.com/data/weather-sentiment/) task from [Crowdflower](https://www.crowdflower.com/).In this task, contributors were asked to grade the sentiment of a particular tweet relating to the weather. Contributors could choose among the following categories:1. Positive2. Negative3. I can't tell4. Neutral / author is just sharing information5. Tweet not related to weather conditionThe catch is that 20 contributors graded each tweet. Thus, in many cases contributors assigned conflicting sentiment labels to the same tweet. The task comes with two data files (to be found in the `data` directory of the tutorial:1. [weather-non-agg-DFE.csv](data/weather-non-agg-DFE.csv) contains the raw contributor answers for each of the 1,000 tweets.2. [weather-evaluated-agg-DFE.csv](data/weather-evaluated-agg-DFE.csv) contains gold sentiment labels by trusted workers for each of the 1,000 tweets.<jupyter_code>%load_ext autoreload
%autoreload 2
%matplotlib inline
import os
import numpy as np
from snorkel import SnorkelSession
session = SnorkelSession()<jupyter_output>The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload<jupyter_text>Step 1: Preprocessing - Data Loading We load the raw data for our crowdsourcing task (stored in a local csv file) into a dataframe<jupyter_code>import pandas as pd

# Load Raw Crowdsourcing Data
raw_crowd_answers = pd.read_csv("data/weather-non-agg-DFE.csv")

# Load Groundtruth Crowdsourcing Data
gold_crowd_answers = pd.read_csv("data/weather-evaluated-agg-DFE.csv")
# Filter out low-confidence answers
gold_answers = gold_crowd_answers[['tweet_id', 'sentiment', 'tweet_body']][(gold_crowd_answers.correct_category == 'Yes') & (gold_crowd_answers.correct_category_conf == 1)] 

# Keep Only the Tweets with Available Groundtruth
candidate_labeled_tweets = raw_crowd_answers.join(gold_answers.set_index('tweet_id',drop=False),on=['tweet_id'],lsuffix='.raw',rsuffix='.gold',how='inner')
candidate_labeled_tweets = candidate_labeled_tweets[['tweet_id.raw','tweet_body.raw','worker_id','emotion']]
candidate_labeled_tweets.columns = ['tweet_id','tweet_body','worker_id','emotion']<jupyter_output><empty_output><jupyter_text>As mentioned above, contributors can provide conflicting labels for the same tweet:<jupyter_code>candidate_labeled_tweets.sort_values(['worker_id','tweet_id']).head()<jupyter_output><empty_output><jupyter_text>Step 2: Generating Snorkel Objects `Candidates``Candidates` are the core objects in Snorkel representing objects to be classified. We'll use a helper function to create a custom `Candidate` sub-class, `Tweet`, with values representing the possible labels that it can be classified with:<jupyter_code>from snorkel.models import candidate_subclass

values = list(candidate_labeled_tweets.emotion.unique())

Tweet = candidate_subclass('Tweet', ['tweet'], values=values)<jupyter_output><empty_output><jupyter_text>`Contexts`All `Candidate` objects point to one or more `Context` objects, which represent the raw data that they are rooted in. In this case, our candidates will each point to a single `Context` object representing the raw text of the tweet.Once we have defined the `Context` for each `Candidate`, we can commit them to the database. Note that we also split into two sets while doing this:1. **Training set (`split=0`):** The tweets for which we have noisy, conflicting crowd labels; we will resolve these conflicts using the `GenerativeModel` and then use them as training data for the LSTM2. **Test set (`split=1`):** We will pretend that we do not have any crowd labels for this split of the data, and use these to test the LSTM's performance on unseen data<jupyter_code>from snorkel.models import Context, Candidate
from snorkel.contrib.models.text import RawText

# Make sure DB is cleared
session.query(Context).delete()
session.query(Candidate).delete()

# Now we create the candidates with a simple loop
tweet_bodies = candidate_labeled_tweets \
    [["tweet_id", "tweet_body"]] \
    .sort_values("tweet_id") \
    .drop_duplicates()

# Generate and store the tweet candidates to be classified
# Note: We split the tweets in two sets: one for which the crowd 
# labels are not available to Snorkel (test, 10%) and one for which we assume
# crowd labels are obtained (to be used for training, 90%)
total_tweets = len(tweet_bodies)
tweet_list = []
test_split = total_tweets*0.1
for i, t in tweet_bodies.iterrows():
    split = 1 if i <= test_split else 0
    raw_text = RawText(stable_id=t.tweet_id, name=t.tweet_id, text=t.tweet_body)
    tweet = Tweet(tweet=raw_text, split=split)
    tweet_list.append(tweet)
    session.add(tweet)
session.commit()<jupyter_output><empty_output><jupyter_text>`Labels`Next, we'll store the labels for each of the training candidates in a sparse matrix (which will also automatically be saved to the Snorkel database), with one row for each candidate and one column for each crowd worker:<jupyter_code>from snorkel.annotations import LabelAnnotator
from collections import defaultdict

# Extract worker votes
# Cache locally to speed up for this small set
worker_labels = candidate_labeled_tweets[["tweet_id", "worker_id", "emotion"]]
wls = defaultdict(list)
for i, row in worker_labels.iterrows():
    wls[str(row.tweet_id)].append((str(row.worker_id), row.emotion))
    

# Create a label generator
def worker_label_generator(t):
    """A generator over the different (worker_id, label_id) pairs for a Tweet."""
    for worker_id, label in wls[t.tweet.name]:
        yield worker_id, label
        
labeler = LabelAnnotator(label_generator=worker_label_generator)
%time L_train = labeler.apply(split=0)
L_train<jupyter_output>0%|          | 0/629 [00:00<?, ?it/s]<jupyter_text>Finally, we load the ground truth ("gold") labels for both the training and test sets, and store as numpy arrays"<jupyter_code>gold_labels = defaultdict(list)

# Get gold labels in verbose form
verbose_labels = dict([(str(t.tweet_id), t.sentiment) 
                       for i, t in gold_answers[["tweet_id", "sentiment"]].iterrows()])

# Iterate over splits, align with Candidate ordering
for split in range(2):
    cands = session.query(Tweet).filter(Tweet.split == split).order_by(Tweet.id).all() 
    for c in cands:
        # Think this is just an odd way of label encoding between 1 and 5?
        gold_labels[split].append(values.index(verbose_labels[c.tweet.name]) + 1) 
        
train_cand_labels = np.array(gold_labels[0])
test_cand_labels = np.array(gold_labels[1])<jupyter_output><empty_output><jupyter_text>Step 3: Resolving Crowd Conflicts with the Generative ModelUntil now we have converted the raw crowdsourced data into a labeling matrix that can be provided as input to `Snorkel`. We will now show how to:1. Use `Snorkel's` generative model to learn the accuracy of each crowd contributor.2. Use the learned model to estimate a marginal distribution over the domain of possible labels for each task.3. Use the estimated marginal distribution to obtain the maximum a posteriori probability estimate for the label that each task takes.<jupyter_code># Imports
from snorkel.learning.gen_learning import GenerativeModel

# Initialize Snorkel's generative model for
# learning the different worker accuracies.
gen_model = GenerativeModel(lf_propensity=True)
# Train the generative model
gen_model.train(
    L_train,
    reg_type=2,
    reg_param=0.1,
    epochs=30
)<jupyter_output>Inferred cardinality: 5<jupyter_text>Infering the MAP assignment for each taskEach task corresponds to an independent random variable. Thus, we can simply associate each task with the most probably label based on the estimated marginal distribution and get an accuracy score:<jupyter_code>accuracy = gen_model.score(L_train, train_cand_labels)
print("Accuracy: {:.10f}".format(accuracy))<jupyter_output>Accuracy: 0.9952305246<jupyter_text>Majority voteIt seems like we did well- but how well?  Given that this is a fairly simple task--we have 20 contributors per tweet (and most of them are far better than random)--**we expect majority voting to perform extremely well**, so we can check against majority vote:<jupyter_code>from collections import Counter

# Collect the majority vote answer for each tweet
mv = []
for i in range(L_train.shape[0]):
    c = Counter([L_train[i,j] for j in L_train[i].nonzero()[1]])
    mv.append(c.most_common(1)[0][0])
mv = np.array(mv)

# Count the number correct by majority vote
n_correct = np.sum([1 for i in range(L_train.shape[0]) if mv[i] == train_cand_labels[i]])
print ("Accuracy:{}".format(n_correct / float(L_train.shape[0])))
print ("Number incorrect:{}".format(L_train.shape[0] - n_correct))<jupyter_output>Accuracy:0.985691573927
Number incorrect:9<jupyter_text>We see that while majority vote makes 10 errors, the Snorkel model makes only 3!  What about an average crowd worker?  Average human accuracyWe see that the average accuracy of a single crowd worker is in fact much lower:<jupyter_code>accs = []
for j in range(L_train.shape[1]):
    n_correct = np.sum([1 for i in range(L_train.shape[0]) if L_train[i,j] == train_cand_labels[i]])
    acc = n_correct / float(L_train[:,j].nnz)
    accs.append(acc)
print( "Mean Accuracy:{}".format( np.mean(accs)))<jupyter_output>Mean Accuracy:0.729664764868<jupyter_text>Step 4: Training an ML Model with Snorkel for Sentiment Analysis over Unseen TweetsIn the previous step, we saw that Snorkel's generative model can help to denoise crowd labels automatically. However, what happens when we don't have noisy crowd labels for a tweet?In this step, we'll use the estimates of the generative model as _probabilistic training labels_ to train a simple LSTM sentiment analysis model, which takes as input a tweet **for which no crowd labels are available** and predicts its sentiment.First, we get the probabilistic training labels (_training marginals_) which are just the marginal estimates of the generative model:<jupyter_code>train_marginals = gen_model.marginals(L_train)
from snorkel.annotations import save_marginals
save_marginals(session, L_train, train_marginals)<jupyter_output>Saved 629 marginals<jupyter_text>Next, we'll train a simple LSTM:<jupyter_code>from snorkel.learning.tensorflow import TextRNN
train_kwargs = {
    'lr':         0.01,
    'dim':        100,
    'n_epochs':   200,
    'dropout':    0.2,
    'print_freq': 5
}

lstm = TextRNN(seed=1701, cardinality=Tweet.cardinality)
train_cands = session.query(Tweet).filter(Tweet.split == 0).order_by(Tweet.id).all()
lstm.train(train_cands, train_marginals, **train_kwargs)
test_cands = session.query(Tweet).filter(Tweet.split == 1).order_by(Tweet.id).all()
accuracy = lstm.score(test_cands, test_cand_labels)
print("Accuracy: {:.10f}".format(accuracy))<jupyter_output>Accuracy: 0.6666666667
{'fraction_non_alphanumeric': 0.08876933422999328, 'fraction_numerical': 0.012945527908540685, 'mean_word_length': 5.602108768035516, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 45, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '310456', 'license': 'Apache-2.0', 'n_tokens_mistral': 3797, 'n_tokens_neox': 3569, 'n_words': 1265, 'path': 'tutorials/crowdsourcing/Crowdsourced_Sentiment_Analysis-pandas.ipynb', 'repo_name': 'AndyLc/snorkel', 'size': '11896'}