<jupyter_start><jupyter_text>Creating More Training Data Without More Annotating We previously listed 'annotating' more data as one way of improving our model. Since supervised learning requires data, having more data *may* be *potentially* helpful for improving our model. One obvious downside of this is that collecting more training data is time consuming and may not always be very practical. In a GLAM setting we may want to use machine learning to do a task which we wouldn't otherwise have time to do. If we have to spend a lot of time creating our training data, the machine learning approach may also become impractical in terms of resources.  Combining Domain Expertise and Machine Learning The time taken to create training data is one weakness of machine learning for practical tasks. Another potential frustration domain experts may have is that their knowledge isn't always incorporated into the machine learning process. For our use case of trying to identify the genre of a book we may already have a sense of some possible ways in which we could identify whether a book was fiction or non-fiction. For example we may already know that titles for non-fiction books tend to be longer than fiction book titles (cf. 'An account of the mining villages of Wales' vs 'Oliver Twist'). If we create our training data in the usual way by labeling examples of our data with the correct label we might not be able to incorporate this domain knowledge very easily. This might be okay in some cases but we might be able to save time and get better results by trying to leverage what we already know (or can access via domain experts).  Programmatically Generating Training Data One way in which we could do this is by writing a `labelling function` to label titles as being either fiction or non-fiction based on the length of the title - i.e. without any annotation by hand. However, a weakness of this approach is that it deals with averages which might not always be correct. Some non-fiction book titles will be shorter than our threshold, and vice-versa for fiction books. If we could simply determine genre based on the average length of titles, we could have skipped this whole machine learning process and be done already. So our problem is that we have some sense of functions we could use to label our data, but these functions are likely to be wrong some of the time. In this notebook we'll explore how we can use a Python library `Snorkel` to deal with this challenge and try and create additional annotations without doing any annotating by hand.  Generating New Genre Training DataHow will we try to approach this in our particular situation? As a reminder of our broad task, we have a collection of metadata related to the Microsoft Digitised Books collection. The 'genre' field isn't yet fully populated. We have previously used a subset of this data to train a machine learning model. What we want to do is to try and write some labelling functions that will add more labels to the full metadata dataset, so that we can give our models more examples to learn from. If we are able to do this we'll hopefully be able to improve the performance of our model from our previous attempts.  We'll start by doing some installation of our libraries we'll be using in this notebook.<jupyter_code>!pip install snorkel
!pip install fastai --upgrade<jupyter_output>Requirement already satisfied: snorkel in /usr/local/lib/python3.7/dist-packages (0.9.7)
Requirement already satisfied: torch<2.0.0,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from snorkel) (1.9.0+cu111)
Requirement already satisfied: scipy<2.0.0,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from snorkel) (1.4.1)
Requirement already satisfied: munkres>=1.0.6 in /usr/local/lib/python3.7/dist-packages (from snorkel) (1.1.4)
Requirement already satisfied: numpy<1.20.0,>=1.16.5 in /usr/local/lib/python3.7/dist-packages (from snorkel) (1.19.5)
Requirement already satisfied: tqdm<5.0.0,>=4.33.0 in /usr/local/lib/python3.7/dist-packages (from snorkel) (4.62.3)
Requirement already satisfied: scikit-learn<0.25.0,>=0.20.2 in /usr/local/lib/python3.7/dist-packages (from snorkel) (0.22.2.post1)
Requirement already satisfied: networkx<2.4,>=2.2 in /usr/local/lib/python3.7/dist-packages (from snorkel) (2.3)
Requirement already satisfied: pandas<2.0.0,>=1.0.0 in /usr/local/lib/python3.7/d[...]<jupyter_text>Since we already have some training data we can leverage this to help us develop `labelling functions` (more on this below) and to test how well these work.<jupyter_code>import pandas as pd
import numpy as np
dtypes = {
    "BL record ID": "string",
    "Type of resource": "category",
    "Name": "category",
    "Type of name": "category",
    "Country of publication": "category",
    "Place of publication": "category",
    "Genre": "category",
    "Dewey classification": "string",
    "BL record ID for physical resource": "string",
    "annotator_main_language": "category",
    "annotator_summaries_language": "string",
}
df = pd.read_csv("https://raw.githubusercontent.com/Living-with-machines/genre-classification/main/genre_classification_of_bl_books/data/train_valid.csv", dtype=dtypes)
df.head(1)<jupyter_output><empty_output><jupyter_text>We'll use only the data from the training split so we're can continue to use the validation split to compare our results.<jupyter_code>df = df[df.is_valid == False]<jupyter_output><empty_output><jupyter_text>Check how many examples we have to work with<jupyter_code>len(df)<jupyter_output><empty_output><jupyter_text>What is a labeling function?We briefly described a function we could use to label our data using the length of the title. When we use a programmatic approach to creating our training data we can refer to the functions which we use to create our labels as `labeling function`. We'll follow a lot of the approaches outlined in the Snorkel [tutorial](https://www.snorkel.org/use-cases/01-spam-tutorial) in this notebook. They provide this example of a labeling function for the task of identifying if a youtube comment is spam or not:<jupyter_code>from snorkel.labeling import labeling_function


@labeling_function()
def lf_contains_link(x):
    # Return a label of SPAM if "http" in comment text, otherwise ABSTAIN
    return SPAM if "http" in x.text.lower() else ABSTAIN<jupyter_output><empty_output><jupyter_text>There are a few things to note here, but, since we're following a lot of what is covered in the Snorkel tutorial we won't repeat things in too much detail. The first thing we need is to import `labeling_function` from `snorkel` as we use this for declaring our labeling functions. The way in which we create a labeling function will depend on our data, and how we might label it, but in this example we have a simple python function which returns `SPAM` if the text `http` appears in the comment text, if it doesn't it returns `ABSTAIN`. We use a python decorator to indicate that this is a labeling function. If you aren't familiar with Python decorators should just remember that decorators are used to modify the behavior of a function (the one it decorates), just as fairy lights decorate a Christmas tree  and change its behaviour from 'tree' to 'festive ornament'. This will make more sense in the context of Snorkel later on. If you want to dig into decorators further this [article](https://realpython.com/primer-on-python-decorators/) on [Real Python](https://realpython.com/) provides a nice introduction, or, if prefer to watch a video this [youtube tutorial](https://youtu.be/FsAPt_9Bf3U) gives a nice overview too. We can see here that the labeling function makes use of the idea that people often include links in spam comments i.e. "plz checkout my etsy store at http:....". Obviously this won't be correct all the time but fortunately Snorkel has some ways to deal with this.  What Makes a Good Labelling Function?One question we might already have is "what makes a good labeling function?". The short, annoying, answer is that it depends on context. We often have intutions about things that might work because we know the domain or have picked up ideas from working with some of the data already. In our particular example of distinguishing fiction from non-fiction books we may think that some words are likely to indicate whether a book is fiction or non-fiction. We'll start by exploring this.   Important Words?A fairly naive approach to trying to labeling a title as 'fiction' or 'non-fiction' would be to use some keywords. Let's start by finding the most common 50 words. We can use the `Counter` class from the delightful [`collections`](https://docs.python.org/3.3/library/collections.htmlmodule-collections) module to do this.<jupyter_code>from collections import Counter
Counter(" ".join(df["Title"]).split()).most_common(50)<jupyter_output><empty_output><jupyter_text>We can see here that the most common words tend to be [stop words](https://en.wikipedia.org/wiki/Stop_word). Since we want to know which words might be unique to fiction *or* non-fiction we'll look at each of these separately.<jupyter_code>df_fiction = df[df["annotator_genre"] == "Fiction"]
df_non_fiction = df[df["annotator_genre"] == "Non-fiction"]
most_frequent_fiction = Counter(" ".join(df_fiction["Title"]).split()).most_common(50)
most_frequent_fiction
most_frequent_non_fiction = Counter(
    " ".join(df_non_fiction["Title"]).split()
).most_common(50)
most_frequent_non_fiction<jupyter_output><empty_output><jupyter_text>For our indicator words to be most reliable we would rather they didn't appear frequently in both fiction and non-fiction titles. We can use a set to check the values which aren't in both fiction and non-fiction titles.<jupyter_code>set(most_frequent_non_fiction).difference(set(most_frequent_fiction))<jupyter_output><empty_output><jupyter_text>These words are still fairly noisy so we might be wary of using many of them. There are some which make sense intuitively so we'll try some of these out and see how they perform.   Creating our Labelling Functions We'll start by setting some constants for our labels.<jupyter_code>ABSTAIN = -1
FICTION = 0
NON_FICTION = 1<jupyter_output><empty_output><jupyter_text>It is important to note that we set an option for `ABSTAIN`. We often want to add an option to our labeling functions that defers from making a prediction. We usually write our labeling function to try and indicate a label, but usually if the function isn't satisfied doesn't indicate that the other label is correct. Another important part of labeling functions is that we usually want to have many labeling functions, and, since we're not relying on a single function to label our data it's often better for our labeling function to return `ABSTAIN` if our labeling condition isn't met rather than returning another label. This becomes even more important if we have multiple labels. One function we could try to start with is checking if the word "Novel" appears in the title text. You may have noticed that in this particular dataset the word novel often appears as part of the title so this *could* be a useful indicator of fiction titles.  ```{warning}We want to be careful that our labeling functions are specific to our data. In the BL books title metadata that we're trying to label we have noticed that things like `A Novel by...` appear in the title. This may not be the case for other book titles in different catalogues.``` Our first labeling function is basically the same as the spam example above except we look for the word "novel".<jupyter_code>@labeling_function()
def lf_contains_novel(x):
    return FICTION if "novel" in x.Title.lower() else ABSTAIN<jupyter_output><empty_output><jupyter_text>Now we have our labeling function we can apply it to our data. We'll start by doing this only with our validation data since we have the correct labels for this to compare our functions against. There are various ways in which Snorkel can apply our functions to our data. In this notebook we'll stick with an approach designed to work with Pandas dataframes. If we have a larger amount of data to label we might want to explore the use of [dask applifer functions](https://snorkel.readthedocs.io/en/v0.9.6/packages/_autosummary/labeling/snorkel.labeling.apply.dask.DaskLFApplier.html). This uses the [dask](https://dask.org/) library to scale the appliation of labelling functions to very large datasets. We won't need this here but if you are planning to develop this approach with very large collections this could be worth exploring. We put our labelling functions in a list called `LFS`, we then create an applier object and pass in our dataframe.<jupyter_code>from snorkel.labeling import PandasLFApplier

lfs = [lf_contains_novel]

applier = PandasLFApplier(lfs=lfs)
L_train = applier.apply(df=df)<jupyter_output>100%|██████████| 3262/3262 [00:00<00:00, 39484.97it/s]<jupyter_text>We store the output of this in a new variable `L_train`. We can use `LFAnalysis` to get a summary of what our current labeling function is doing.<jupyter_code>from snorkel.labeling import LFAnalysis

LFAnalysis(L=L_train, lfs=lfs).lf_summary()<jupyter_output><empty_output><jupyter_text>We can see a row for our current labeling function, we can also see that at the moment our coverage (i.e. how much of our data is labeled by our function) is very low. At the moment we don't have any overlaps or conflicts since these are relevant only when we have multiple labeling functions.  We have ground truth labels that we can use to evaluate how accurate our labeling function is. To to his we need to use the same labels as Snorkel for ground truth so we'll map our labels to the constants we made above.<jupyter_code>ground_truth = df.annotator_genre.replace({"Fiction": 0, "Non-fiction": 1})<jupyter_output><empty_output><jupyter_text>We can pass our ground truth labels to `lf_empirical_accuracies` to get a sumaary of the peformance of our functions.<jupyter_code>LFAnalysis(L=L_train, lfs=lfs).lf_empirical_accuracies(ground_truth)<jupyter_output><empty_output><jupyter_text>We can see here that our current function is 100% accurate. We shouldn't get too excited about this since our coverage is very low. We'll need to write some more labeling functions to make sure that we have some chance of labeling more of our data than we currently have done.   HeuristicsWe could also use heuristics for our labeling functions. For example the length of the title. I don't have any idea what threshold to use for this. Since we have some labels we can try and identify a sensible threshold. First we'll add a new columns to our DataFrame which contains the length of our titles.<jupyter_code>df["text_len"] = df["Title"].str.len()<jupyter_output><empty_output><jupyter_text>We'll now use a pandas `groupby` to see what the lengths look like for fiction vs non-fiction books. Since it might be useful to have a sense of the distributions we'll use `describe` instead of mean.<jupyter_code>df.groupby(["annotator_genre"])["text_len"].describe()<jupyter_output><empty_output><jupyter_text>Precision vs Recall: What Value to use for our Threshold?We can see various values for mean, min etc. What would be a reasonable value to use for our threshold for a labeling function which labeled a title as 'non-fiction'? This partly comes down to whether we want high coverage (or recall) or high precision. If we choose a threshold that is higher we will label fewer examples, but they will be more likely to be correct. For example, if we use the max value for the length of a non-fiction title `469` most titles will be much shorter than this, so our function will 'abstain' from applying a label, and we would only label a very small number of examples from our data. However, we also won't have many (or any) wrongly-labeled examples since the max value for fiction here is `288`. We need to balance these two aims of coverage and precision. Since we are writing more than one labeling function we probably want to tend towards writing more precise labeling functions rather than aiming for high coverage if this is likely to introduce wrongly labeled examples.  ```{note}As we saw in previous chapters/notebooks we have to be a bit careful in generalizing between what we see in our training and validation data since there may be some distribution drift between our training data (which wasn't a completely randomized sample) and the full data that we want to label. In the error analysis notebook we saw that the performance of our model was worse than it was on validation data. We should keep this in mind when writing a labeling function, since we want our labeling function to work well on new data which doesn't have labels, not just on the data for which we already have labels. ``` We'll use the value of the maximum title length for a fiction book as our threshold. This will *hopefully* give us fairly decent coverage without too many mistakes.<jupyter_code>@labeling_function()
def lf_is_long_title(x):
    return NON_FICTION if x.text_len > 211.0 else ABSTAIN<jupyter_output><empty_output><jupyter_text>We do the same as earlier, including our new labeling function<jupyter_code>lfs = [lf_contains_novel, lf_is_long_title]

applier = PandasLFApplier(lfs=lfs)
L_train = applier.apply(df=df)
LFAnalysis(L=L_train, lfs=lfs).lf_summary()<jupyter_output><empty_output><jupyter_text>We can see our coverage is still fairly low but at the moment we don't have any conflicts. We can keep tweaking our length threshold but for now we'll try a different approach to our labeling function.   Add KeywordsWe already have a labeling function that uses the keyword 'novel' to identify likely fiction books. Since we often want to use keywords the Snorkel tutorial suggests a way we can do this more easily using [keyword lookups](https://www.snorkel.org/use-cases/01-spam-tutoriala-keyword-lfs).<jupyter_code>from snorkel.labeling import LabelingFunction


def keyword_lookup(x, keywords, label):
    if any(word in x.Title.lower() for word in keywords):
        return label
    return ABSTAIN


def make_keyword_lf(keywords, label=FICTION):
    return LabelingFunction(
        name=f"keyword_{keywords[0]}",
        f=keyword_lookup,
        resources=dict(keywords=keywords, label=label),
    )<jupyter_output><empty_output><jupyter_text>We can try two new keyword labels using this more concise approach:<jupyter_code>keyword_tale = make_keyword_lf(keywords=["tale"])
keyword_poem = make_keyword_lf(keywords=["poem"])<jupyter_output><empty_output><jupyter_text>Leveraging Other ModelsSo far we have leveraged some domain knowledge/exploration and our existing labeled data to create our labeling functions. However, we could also utilise other resources to help us label our data. Since we're working with text we should be able to benefit from some existing NLP models to label our data. Snorkel supports this in a few different ways. spaCy is a popular nlp library which supports a range of different models and nlp tasks. Here we're particuarly interested in some of the named entity models supported by spaCy. To work with this library we can use Snorkel's `SpacyPreProcessor`. `Preprocessors` are used in Snorkel to do some preprocessing (hence the name) which is required for our labeling functions. These can be particularly useful if the processing takes some time and might be reused across differnet labelling functions. Let's take a look at an example<jupyter_code>from snorkel.preprocess.nlp import SpacyPreprocessor

# The SpacyPreprocessor parses the text in text_field and
# stores the new enriched representation in doc_field
spacy = SpacyPreprocessor(text_field="Title", doc_field="doc", memoize=True)<jupyter_output><empty_output><jupyter_text>Above we create a `SpacyPreprocessor` which will use our title field and create a new `doc` field. This `doc` refers to the Spacy [`doc`](https://spacy.io/api/doc) container. This can be reused for multiple different labeling functions. We pass in `memoize=True` to cache our results. This means we won't have to wait for the preprocessing to be done multiple times for different labeling functions which reuse the `doc` container.   Using named entities for labeling functions. spaCy has support for named entity recognition. Since these models are already created and can be used directly by us it might be worth seeing if named entities are of any benefit for our particular task. We can again draw from our domain knowledge, intution or guesses (depending on how confident we are) and say that it's likely that we will see more named entities of the `ORG` type in non-fiction titles since these often will be about organizations. We can combine this with a slightly softer threshold for length to label titles as being likely non-fiction. To create this function we replicate closely what we did before except that we pass in our `SpacyPreProcessor` instance to let Snorkel know that this preprocesser is a requirement of this labeling function. Under the hood this will mean that if the preprocessor hasn't been run already this will triger the preprocessing. If we reuse this for another funciton the preprocessing will already have been cached.<jupyter_code>@labeling_function(pre=[spacy])
def has_many_org(x):
    if len(x.doc) > 50 and len([ent.label_ == "ORG" for ent in x.doc.ents]) > 1:
        return NON_FICTION
    else:
        return ABSTAIN<jupyter_output><empty_output><jupyter_text>We might also guess that there will be more location entities in non-fiction titles<jupyter_code>@labeling_function(pre=[spacy])
def has_many_loc(x):
    if len(x.doc) > 50 and len([ent.label_ == "LOC" for ent in x.doc.ents]) > 2:
        return NON_FICTION
    else:
        return ABSTAIN<jupyter_output><empty_output><jupyter_text>Similarly we might also assume that there will be more `GPE` entities for non-fiction<jupyter_code>@labeling_function(pre=[spacy])
def has_many_gpe(x):
    if len(x.doc) > 50 and len([ent.label_ == "GPE" for ent in x.doc.ents]) > 2:
        return NON_FICTION
    else:
        return ABSTAIN<jupyter_output><empty_output><jupyter_text>and law entities...<jupyter_code>@labeling_function(pre=[spacy])
def has_law(x):
    if any([ent.label_ == "LAW" for ent in x.doc.ents]):
        return NON_FICTION
    else:
        return ABSTAIN<jupyter_output><empty_output><jupyter_text>and if it's long and has a date it might be a non-fiction title?<jupyter_code>@labeling_function(pre=[spacy])
def is_long_and_has_date(x):
    if len(x.doc) > 50 and any([ent.label_ == "DATE" for ent in x.doc.ents]):
        return NON_FICTION
    else:
        return ABSTAIN<jupyter_output><empty_output><jupyter_text>or it is long and has a `FAC` entitity<jupyter_code>@labeling_function(pre=[spacy])
def is_long_and_has_fac(x):
    if len(x.doc) > 50 and any([ent.label_ == "FAC" for ent in x.doc.ents]):
        return NON_FICTION
    else:
        return ABSTAIN<jupyter_output><empty_output><jupyter_text>We now have a bunch of labeling functions we'll create a new list containing these and see how they do.<jupyter_code>lfs = [
    lf_contains_novel,
    lf_is_long_title,
    keyword_tale,
    keyword_poem,
    has_many_org,
    has_many_loc,
    has_many_gpe,
    has_law,
    is_long_and_has_date,
]
applier = PandasLFApplier(lfs=lfs)
L_train = applier.apply(df=df)
LFAnalysis(L=L_train, lfs=lfs).lf_summary()<jupyter_output><empty_output><jupyter_text>Again our coverage is quite low but we also don't have too many conflicts. We can check the performance of these functions:<jupyter_code>LFAnalysis(L=L_train, lfs=lfs).lf_empirical_accuracies(ground_truth)<jupyter_output><empty_output><jupyter_text>These are all doing pretty good so we might be okay with lower coverage for now. We also have a resource available to us which should boost our coverage a fair bit: our previously trained model.   Using our previous model In a previously notebook we trained a model which didn't perform terribly. Although we wanted to improve the performance, hence this notebook, it wasn't so disastrous as to be unusable, particularly with the insights we got from the error analysis notebook that if we raise the threshold of confidence for which we accept our models predictions our performance increases quite a bit. We may therefore want to try and incorporate this model as another way of labeling more data. There are various ways in which we can do this, we'll look at one approach below.  We'll start by importing fastai so we can load our previously trained model.<jupyter_code>from fastai.text.all import *<jupyter_output><empty_output><jupyter_text>If you don't have a model saved from notebook you can download one by uncommenting the below cell.<jupyter_code># !wget -O 20210928-model.pkl  https://zenodo.org/record/5245175/files/20210928-model.pkl?download=1
learn = load_learner("20210928-model.pkl")<jupyter_output><empty_output><jupyter_text>We can quickly check our vocab<jupyter_code>learn.dls.vocab[1]<jupyter_output><empty_output><jupyter_text>One way of using this model would be to create a preprocessor that will be used by Snorkel. This will do the setup required to use this model (as we saw with the spaCy example). We can do this by using the `preprocessor` decorator. Our function then calls whatever we need to happen. In this case we store the predicted label and probability.<jupyter_code># from snorkel.preprocess import preprocessor
# @preprocessor(memoize=True)
# def fastai_pred(x):
#     with learn.no_bar():
#         *_, probs = learn.predict(x.title)
#     x.fiction_prob = probs[0]
#     x.non_fiction_prob = probs[1]
#     return x<jupyter_output><empty_output><jupyter_text>In this example we don't want to use this since we then don't benefit from doing our inference in batches. Instead we'll just create some new columns to store our fastai models labels and confidence.<jupyter_code>test_dl = learn.dls.test_dl(df.Title)
preds = learn.get_preds(dl=test_dl)
fiction_prob, non_fiction_prob = np.hsplit(preds[0].numpy(), 2)
fiction_prob
df["fiction_prob"] = fiction_prob
df["non_fiction_prob"] = non_fiction_prob<jupyter_output><empty_output><jupyter_text>We now have some new columns containing the probabilities for our labels from our previously created model. We saw in the previous [](02_error_analysis.ipynb) section that by only using predictions where our model was confident, we could get better results i.e. we only accept suggestions from our model where it is very confident. For example, we could accept a prediction only if it is above 95% confidence. We'll use this in our labelling function to set a threshold at which we accept the previous models predictions. If the model is unsure we don't use it's prediction. This will mean less of our data ends up labelled because some predictions aren't used. However, we will hopefully get *better* predictions because we only use those where our model is confident.<jupyter_code>@labeling_function()
def fastai_fiction_prob_v_high(x):
    return FICTION if x.fiction_prob > 0.97 else ABSTAIN
@labeling_function()
def fastai_non_fiction_prob_v_high(x):
    return NON_FICTION if x.non_fiction_prob > 0.97 else ABSTAIN<jupyter_output><empty_output><jupyter_text>Again we add this to our existing labeling function list and apply it to our data<jupyter_code>lfs += [fastai_fiction_prob_v_high, fastai_non_fiction_prob_v_high]
lfs
applier = PandasLFApplier(lfs=lfs)
L_train = applier.apply(df=df)
LFAnalysis(L=L_train, lfs=lfs).lf_summary()<jupyter_output><empty_output><jupyter_text>We can see that the labelling function which uses our model outputs has a much higher coverage of our data. This should be very helpful in labelling more examples but we want to check that these are correct.<jupyter_code>LFAnalysis(L=L_train, lfs=lfs).lf_empirical_accuracies(ground_truth)<jupyter_output><empty_output><jupyter_text>We can see that our labels all perform pretty well i.e. above 90%. We are also getting much better coverage now that we leverage our existing model.   Creating more training dataSo far we have been using the validation data to develop some potential labeling functions. Now we are fairly satisfied with them let's apply to the full data. We'll quickly look at this process on our current data and then we'll move to the full metadata json file that we use for creating more training data. We use `LabelModel` to fit a model which will be able to take as input all of the predictions from our labelling functions and fit a model which will predict the probability for a label. This model is able to deal with some conflicts between labeling functions and will do much better in most cases than a naive majority vote model i.e. one which just accepts the most often predicted label. The details of this model are beyond the scope of this notebook but if you are interested *Data Programming: Creating Large Training Sets, Quickly* offers a fuller overview of the details of this method{cite:ps}`NIPS2016_6709e8d6`<jupyter_code>from snorkel.labeling.model import LabelModel

label_model = LabelModel(cardinality=2, verbose=True)
label_model.fit(L_train=L_train, n_epochs=500, log_freq=100, seed=123)<jupyter_output><empty_output><jupyter_text>Above we fit our `LabelModel` for 500 epochs. Since we are working with the training set still we can get the score for this model.<jupyter_code>label_model.score(
    L=L_train,
    Y=ground_truth,
    tie_break_policy="abstain",
    metrics=["precision", "recall", "f1"],
)<jupyter_output>WARNING:root:Metrics calculated over data points with non-abstain labels only<jupyter_text>This is looking pretty good and hopefully this performance will be similar for our full data. We'll now load a dataframe that includes all of the BL books metadata.<jupyter_code>df_full = pd.read_csv(
    "https://bl.iro.bl.uk/downloads/e1be1324-8b1a-4712-96a7-783ac209ddef?locale=en",
    dtype=dtypes,
)<jupyter_output><empty_output><jupyter_text>We create a new column `text_len` since we need this for some of our labeling functions.<jupyter_code>df_full["text_len"] = df_full.Title.str.len()<jupyter_output><empty_output><jupyter_text>We also get our fastai model's predictions into new columns. This obviously takes some time since we're now doing inference on a fairly large dataset.<jupyter_code>test_dl = learn.dls.test_dl(df_full.Title)
preds = learn.get_preds(dl=test_dl)
fiction_prob, non_fiction_prob = np.hsplit(preds[0].numpy(), 2)
df_full["fiction_prob"] = fiction_prob
df_full["non_fiction_prob"] = non_fiction_prob<jupyter_output><empty_output><jupyter_text>Now we have all the same columns in place as we had previously we can now apply our labelling functions to all of our data.<jupyter_code>applier = PandasLFApplier(lfs=lfs)
L_train = applier.apply(df=df_full)<jupyter_output>100%|██████████| 52695/52695 [09:27<00:00, 92.85it/s]<jupyter_text>We can check what the coverage, overlaps and conflicts look like<jupyter_code>LFAnalysis(L=L_train, lfs=lfs).lf_summary()<jupyter_output><empty_output><jupyter_text>The coverage is lower than we had previously. This makes sense since we previously used the same data for developing our labeling functions as we used for training our model. It's not suprising our model is more confident about these. If we were being more dilligent we might have held back a different dataset for developing our labelling functions but since we're being a bit pragmatic (lazy) here we won't worry too much about this. We can again fit our model:<jupyter_code>label_model = LabelModel(cardinality=2, verbose=True)
label_model.fit(L_train=L_train, n_epochs=500, log_freq=100, seed=123)<jupyter_output><empty_output><jupyter_text>We now use this model to predict the probabilites from our labelling function outputs<jupyter_code>probs_train = label_model.predict_proba(L_train)<jupyter_output><empty_output><jupyter_text>We currently have predictions for some of our data but not all of it. Since we want only the labelled exampled we use a function from Snorkel to filter out data which our labeling functions didn't annotate.<jupyter_code>from snorkel.labeling import filter_unlabeled_dataframe

df_train_filtered, probs_train_filtered = filter_unlabeled_dataframe(
    X=df_full, y=probs_train, L=L_train
)<jupyter_output><empty_output><jupyter_text>We now have the predicted probabilty for each label. We could work with these probabilities but to keep things simple we'll make these hard predictions i.e. fiction or non-fiction rather than 0.87% fiction. Again Snorkel provides a handy function for doing this.<jupyter_code>from snorkel.utils import probs_to_preds

preds_train_filtered = probs_to_preds(probs=probs_train_filtered)<jupyter_output><empty_output><jupyter_text>Let's see how much data we have now.<jupyter_code>len(preds_train_filtered)<jupyter_output><empty_output><jupyter_text>As a reminder we previosuly had `3262` labeled examples. We can see that we've now gained a lot more examples for relateively little work (especially if we compare how much time it would take to annotate these by hand).<jupyter_code>26566 / 3262<jupyter_output><empty_output><jupyter_text>We'll store out new labels in a label column<jupyter_code>df_train_filtered["snorkel_label"] = preds_train_filtered
df_train_filtered["snorkel_label"]<jupyter_output><empty_output><jupyter_text>Creating our new training dataAs a reminder of what we've done:- we had training data/annotations collected via a zooniverse crowdsourcing task with `2909` labeled examples in our validation set- we had previously used this to train a model that did fairly well - we used our existing training data to generate labeling functions, these leveraged:    - our intuitions about our data    - SpaCy models    - our previous model- we applied these labeling functions to the Microsoft Digitised Books file. Once we excluded examples which weren't labeled by our labeling functions we had `34542` labeled examples we could work with. We now want to get all of this data into a format we can use to train new models with. There are a few things we need to do for this.  Map to our original labelsWe'll map these back to our original fiction and non-fiction labels. This isn't super important but might be more explicit then `1` or `0` for our labels.<jupyter_code>df_train_filtered["snorkel_genre"] = df_train_filtered["snorkel_label"].map(
    {0: "Fiction", 1: "Non-fiction"}
)
df_train_filtered.columns<jupyter_output><empty_output><jupyter_text>Selecting required columns Since we have only been using the title and the label (fiction or non-fiction) to train our models we will just keep these.<jupyter_code>df["annotator_genre"]
df["snorkel_genre"] = df["annotator_genre"]
df_snorkel_train = pd.concat([df, df_train_filtered])
df_snorkel_train["snorkel_genre"].value_counts()<jupyter_output><empty_output><jupyter_text>Prioritising human annotationsWhen we applied our labeling functions across the full Microsoft Digitised Books metadata file we didn't do anything to exclude titles where a human annotator had already provided a label as part of the Zooniverse annotation task. Since we joined the full metadata and the human annotations together we will now have some duplicates. We almost definitely want to prioritise the human annotations over our label function labels. We could use Pandas drop_duplicates and keep the first example (the human annotated one) to deal with this.<jupyter_code>df_snorkel_train.duplicated(subset="Title")
df_snorkel_train = df_snorkel_train.drop_duplicates(subset="Title", keep="first")
df_snorkel_train<jupyter_output><empty_output><jupyter_text>Data leakage Want to exclude data which is in test set so we drop these examples from our training data. Since we care about titles 'leaking' we look up whether any titles in our training data appear in our test data and remove these from the training data.<jupyter_code>df_test = pd.read_csv("test_errors.csv")<jupyter_output><empty_output><jupyter_text>Removing data which is in our test data<jupyter_code>df_snorkel_train = df_snorkel_train[~df_snorkel_train.Title.isin(df_test.title)]
len(df_snorkel_train)<jupyter_output><empty_output><jupyter_text>Creating new splitsWe create some new splits following the same process we used before. We can then use these splits to more accurately compare across models training using this dataset. Since we have kept the test data out of our 'Snorkel dataset' we will also continue to use this test data for final model evaluation.<jupyter_code>from sklearn.model_selection import GroupShuffleSplit
train_inds, valid_ins = next(
    GroupShuffleSplit(n_splits=2, test_size=0.2).split(
        df_snorkel_train, groups=df_snorkel_train["Title"]
    )
)
df_train, df_valid = (
    df_snorkel_train.iloc[train_inds].copy(),
    df_snorkel_train.iloc[valid_ins].copy(),
)
df_train["is_valid"] = False
df_valid["is_valid"] = True
df = pd.concat([df_train, df_valid])
df.snorkel_genre.value_counts()<jupyter_output><empty_output><jupyter_text>We can see we still have a healthy number of examples to train our model on even after dropping titles which appear in our test data<jupyter_code>len(df)<jupyter_output><empty_output><jupyter_text>Saving our new training dataWe'll save our new training data as a csv file.<jupyter_code>df.to_csv("data/snorkel_train.csv", index=False)
df.head(1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06640635447032708, 'fraction_numerical': 0.009012864058195822, 'mean_word_length': 5.3397761953204474, 'pattern_counts': {'":': 13, '<': 242, '<?xml version=': 0, '>': 252, 'https://': 13, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '59', 'dir': 'jupyter-structured-clean-dedup', 'id': '478648', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 11001, 'n_tokens_neox': 10398, 'n_words': 4729, 'path': 'genre_classification_of_bl_books/04_snorkel.ipynb', 'repo_name': 'Living-with-machines/genre-classification', 'size': '37391'}