<jupyter_start><jupyter_text>L5: Information extraction Information extraction (IE) is the task of identifying named entities and semantic relations between these entities in text data. In this lab we will focus on two sub-tasks in IE, **named entity recognition** (identifying mentions of entities) and **entity linking** (matching these mentions to entities in a knowledge base). We start by loading spaCy:<jupyter_code>import spacy

nlp = spacy.load('en_core_web_sm')<jupyter_output><empty_output><jupyter_text>The data that we will be using has been tokenized following the conventions of the [Penn Treebank](ftp://ftp.cis.upenn.edu/pub/treebank/public_html/tokenization.html), and we need to prevent spaCy from using its own tokenizer on top of this. We therefore override spaCy&rsquo;s tokenizer with one that simply splits on space.<jupyter_code>from spacy.tokens import Doc

class WhitespaceTokenizer(object):
    def __init__(self, vocab):
        self.vocab = vocab

    def __call__(self, text):
        return Doc(self.vocab, words=text.split(' '))

nlp.tokenizer = WhitespaceTokenizer(nlp.vocab)<jupyter_output><empty_output><jupyter_text>Data set The main data set for this lab is a collection of news wire articles in which mentions of named entities have been annotated with page names from the [English Wikipedia](https://en.wikipedia.org/wiki/). The next code cell loads the training and the development parts of the data into Pandas data frames.<jupyter_code>import bz2
import csv
import pandas as pd

with bz2.open('ner-train.tsv.bz2', 'rt', encoding='latin_1') as source:
    df_train = pd.read_csv(source, sep='\t', quoting=csv.QUOTE_NONE)

with bz2.open('ner-dev.tsv.bz2', 'rt', encoding='latin_1') as source:
    df_dev = pd.read_csv(source, sep='\t', quoting=csv.QUOTE_NONE)<jupyter_output><empty_output><jupyter_text>Each row in these two data frames corresponds to one mention of a named entity and has five columns:1. a unique identifier for the sentence containing the entity mention2. the pre-tokenized sentence, with tokens separated by spaces3. the start position of the token span containing the entity mention4. the end position of the token span (exclusive, as in Python list indexing)5. the entity label; either a Wikipedia page name or the generic label `--NME--`The following cell prints the first five samples from the training data:<jupyter_code>df_train.head()<jupyter_output><empty_output><jupyter_text>In this sample, we see that the first sentence is annotated with three entity mentions:* the span 0–1 &lsquo;EU&rsquo; is annotated as a mention but only labelled with the generic `--NME--`* the span 2–3 &lsquo;German&rsquo; is annotated with the page [Germany](http://en.wikipedia.org/wiki/Germany)* the span 6–7 &lsquo;British&rsquo; is annotated with the page [United_Kingdom](http://en.wikipedia.org/wiki/United_Kingdom)  Problem 1: Evaluation measures To warm up, we ask you to write code to print the three measures that you will be using for evaluation:<jupyter_code>def evaluation_report(gold, pred):
    """Print precision, recall, and F1 score.
    
    Args:
        gold: The set with the gold-standard values.
        pred: The set with the predicted values.
    
    Returns:
        Nothing, but prints the precision, recall, and F1 values computed
        based on the specified sets.
        
    """
    total_precision, total_recall, total_f1 = 0.0, 0.0, 0.0
    count = 0
    
    for i, (g, p) in enumerate(zip(gold, pred)):   
        recall, precision, f1 = 0, 0, 0
        
        g = set(g)
        p = set(p)

        print(g)
        print(p)
        recall = len(g.intersection(p)) / len(g) * 100
        precision = len(g.intersection(p)) / len(p) * 100
        print(recall)
        print(precision)
        
        if precision != 0.0 and recall != 0.0:
            f1 = 2 * (precision * recall) / (precision + recall)
        
        total_precision += precision
        total_recall += recall
        total_f1 += f1
        count = i + 1
        
    print("Precision: ", total_precision / count)
    print("Recall: ", total_recall / count)
    print("F1 Values: ", total_f1 / count)<jupyter_output><empty_output><jupyter_text>To test your code, you can run the following cell:<jupyter_code>evaluation_report(set(range(3)), set(range(5)))<jupyter_output>recall: 100.0%
precision: 60.0%
f1 score: 74%<jupyter_text>This should give you a precision of 60%, a recall of 100%, and an F1-value of 75%.  Problem 2: Span recognition One of the first tasks that an information extraction system has to solve is to locate and classify (mentions of) named entities, such as persons and organizations. Here we will tackle the simpler task of recognizing **spans** of tokens that contain an entity mention, without the actual entity label.The English language model in spaCy features a full-fledged [named entity recognizer](https://spacy.io/usage/linguistic-featuresnamed-entities) that identifies a variety of entities, and can be updated with new entity types by the user. Your task in this problem is to evaluate the performance of this component when predicting entity spans in the development data.Start by implementing a generator function that yields the gold-standard spans in a given data frame.**Hint:** The Pandas method [`itertuples()`](https://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.DataFrame.itertuples.html) is useful when iterating over the rows in a DataFrame.<jupyter_code>def gold_spans(df):
    """Yield the gold-standard mention spans in a data frame.

    Args:
        df: A data frame.

    Yields:
        The gold-standard mention spans in the specified data frame as
        triples consisting of the sentence id, start position, and end
        position of each span.
    """
    for row in df.itertuples():
        yield (row[1], row[3], row[4])
print(next(gold_spans(df_dev)))
*_, last = gold_spans(df_dev)
print(last)<jupyter_output>('0946-000', 2, 3)
('1161-010', 1, 3)<jupyter_text>To test your code, you can count the spans yielded by your function. When called on the development data, you should get a total of 5,917 unique triples. The first triple and the last triple should be    ('0946-000', 2, 3)    ('1161-010', 1, 3)<jupyter_code>spans_dev_gold = set(gold_spans(df_dev))
print(len(spans_dev_gold))<jupyter_output>5917<jupyter_text>Your next task is to write code that calls spaCy to predict the named entities in the development data, and to evaluate the accuracy of these predictions in terms of precision, recall, and F1. Print these scores using the function that you wrote for Problem&nbsp;1.<jupyter_code># TODO: Write code here to run and evaluate the spaCy NER on the development data
import spacy

def get_predictions(df):
    for row in df.itertuples():
        sentence = row[2]
        sentence_id = row[1]

        doc = nlp(sentence)

        for ent in doc.ents:
            yield sentence_id, ent.start_char, ent.end_char
results_set = set(get_predictions(df_dev))
evaluation_report(spans_dev_gold, results_set)
results_set
spans_dev_gold<jupyter_output><empty_output><jupyter_text>Problem 3: Error analysis As you were able to see in Problem&nbsp;2, the span accuracy of the named entity recognizer is far from perfect. In particular, only slightly more than half of the predicted spans are correct according to the gold standard. Your next task is to analyse this result in more detail.Here is a function that prints the false positives as well as the false negatives spans for a data frame, given a reference set of gold-standard spans and a candidate set of predicted spans.<jupyter_code>from collections import defaultdict

def error_report(df, spans_gold, spans_pred):
    false_pos = defaultdict(list)
    for s, b, e in spans_pred - spans_gold:
        false_pos[s].append((b, e))
    false_neg = defaultdict(list)
    for s, b, e in spans_gold - spans_pred:
        false_neg[s].append((b, e))
    for row in df.drop_duplicates('sentence_id').itertuples():
        if row.sentence_id in false_pos or row.sentence_id in false_neg:
            print('Sentence:', row.sentence)
            for b, e in false_pos[row.sentence_id]:
                print('  FP:', ' '.join(row.sentence.split()[b:e]))
            for b, e in false_neg[row.sentence_id]:
                print('  FN:', ' '.join(row.sentence.split()[b:e]))<jupyter_output><empty_output><jupyter_text>Use this function to inspect and analyse the errors that the automated prediction makes. Can you see any patterns? Base your analysis on the first 500 rows of the training data. Summarize your observations in a short text.<jupyter_code># TODO: Write code here to do your analysis<jupyter_output><empty_output><jupyter_text>*TODO: Write a short text that summarises the errors that you observed* Now, use the insights from your error analysis to improve the automated prediction that you implemented in Problem&nbsp;2. While the best way to do this would be to [update spaCy&rsquo;s NER model](https://spacy.io/usage/linguistic-featuresupdating) using domain-specific training data, for this lab it suffices to write code to post-process the output produced by spaCy. You should be able to improve the F1 score from Problem&nbsp;2 by at last 15 percentage points.<jupyter_code># TODO: Write code here to improve the span prediction from Problem 2<jupyter_output><empty_output><jupyter_text>Show that you achieve the performance goal by reporting the evaluation measures that you implemented in Problem&nbsp;1. Before going on, we ask you to store the outputs of the improved named entity recognizer on the development data in a new data frame. This new frame should have the same layout as the original data frame for the development data that you loaded above, but should contain the *predicted* start and end positions for each token span, rather than the gold positions. As the `label` of each span, you can use the special value `--NME--`.<jupyter_code># TODO: Write code here to store the predicted spans in a new data frame<jupyter_output><empty_output><jupyter_text>Problem 4: Entity linking Now that we have a method for predicting mention spans, we turn to the task of **entity linking**, which amounts to predicting the knowledge base entity that is referenced by a given mention. In our case, for each span we want to predict the Wikipedia page that this mention references.Start by extending the generator function that you implemented in Problem&nbsp;2 to labelled spans.<jupyter_code>def gold_mentions(df):
    """Yield the gold-standard mentions in a data frame.

    Args:
        df: A data frame.

    Yields:
        The gold-standard mention spans in the specified data frame as
        quadruples consisting of the sentence id, start position, end
        position and entity label of each span.
    """
    # TODO: Replace the next line with your own code
    yield None<jupyter_output><empty_output><jupyter_text>A naive baseline for entity linking on our data set is to link each mention span to the Wikipedia page name that we get when we join the tokens in the span by underscores, as is standard in Wikipedia page names. Suppose, for example, that a span contains the two tokens    <NAME>The baseline Wikipedia page name for this span would be    Jimi_HendrixImplement this naive baseline and evaluate its performance. Print the evaluation measures that you implemented in Problem&nbsp;1.     Here and in the remainder of this lab, you should base your entity predictions on the predicted spans that you computed in Problem&nbsp;3.<jupyter_code># TODO: Write code here to implement the baseline<jupyter_output><empty_output><jupyter_text>Problem 5: Extending the training data using the knowledge base State-of-the-art approaches to entity linking exploit information in knowledge bases. In our case, where Wikipedia is the knowledge base, one particularly useful type of information are links to other Wikipedia pages. In particular, we can interpret the anchor texts (the highlighted texts that you click on) as mentions of the entities (pages) that they link to. This allows us to harvest long lists of mention–entity pairings.The following cell loads a data frame summarizing anchor texts and page references harvested from the first paragraphs of the English Wikipedia. The data frame also contains all entity mentions in the training data (but not the development or the test data).<jupyter_code>with bz2.open('kb.tsv.bz2', 'rt') as source:
    df_kb = pd.read_csv(source, sep='\t', quoting=csv.QUOTE_NONE)<jupyter_output><empty_output><jupyter_text>To understand what information is available in this data, the following cell shows the entry for the anchor text `Sweden`.<jupyter_code>df_kb.loc[df_kb.mention == 'Sweden']<jupyter_output><empty_output><jupyter_text>As you can see, each row of the data frame contains a pair $(m, e)$ of a mention $m$ and an entity $e$, as well as the conditional probability $P(e|m)$ for mention $m$ referring to entity $e$. These probabilities were estimated based on the frequencies of mention–entity pairs in the knowledge base. The example shows that the anchor text &lsquo;Sweden&rsquo; is most often used to refer to the entity [Sweden](http://en.wikipedia.org/wiki/Sweden), but in a few cases also to refer to Sweden&rsquo;s national football and ice hockey teams. Note that references are sorted in decreasing order of probability, so that the most probable pairing come first.Implement an entity linking method that resolves each mention to the most probable entity in the data frame. If the mention is not included in the data frame, you can predict the generic label `--NME--`. Print the precision, recall, and F1 of your method using the function that you implemented for Problem&nbsp;1.<jupyter_code># TODO: Write code here to implement the "most probable entity" method.<jupyter_output><empty_output><jupyter_text>Problem 6: Context-sensitive disambiguation Consider the entity mention &lsquo;Lincoln&rsquo;. The most probable entity for this mention turns out to be [Lincoln, Nebraska](http://en.wikipedia.org/Lincoln,_Nebraska); but in pages about American history, we would be better off to predict [Abraham Lincoln](http://en.wikipedia.org/Abraham_Lincoln). This suggests that we should try to disambiguate between different entity references based on the textual context on the page from which the mention was taken. Your task in this last problem is to implement this idea.Set up a dictionary that contains, for each mention $m$ that can refer to more than one entity $e$, a separate Naive Bayes classifier that is trained to predict the correct entity $e$, given the textual context of the mention. As the prior probabilities of the classifier, choose the probabilities $P(e|m)$ that you used in Problem&nbsp;5. To let you estimate the context-specific probabilities, we have compiled a data set with mention contexts:<jupyter_code>with bz2.open('contexts.tsv.bz2') as source:
    df_contexts = pd.read_csv(source, sep='\t', quoting=csv.QUOTE_NONE)<jupyter_output><empty_output><jupyter_text>This data frame contains, for each ambiguous mention $m$ and each knowledge base entity $e$ to which this mention can refer, up to 100 randomly selected contexts in which $m$ is used to refer to $e$. For this data, a **context** is defined as the 5 tokens to the left and the 5 tokens to the right of the mention. Here are a few examples:<jupyter_code>df_contexts.head()<jupyter_output><empty_output><jupyter_text>Note that, in each context, the position of the mention is indicated by the `@` symbol.From this data frame, it is easy to select the data that you need to train the classifiers – the contexts and corresponding entities for all mentions. To illustrate this, the following cell shows how to select all contexts that belong to the mention &lsquo;Lincoln&rsquo;:<jupyter_code>df_contexts.context[df_contexts.mention == 'Lincoln']<jupyter_output><empty_output><jupyter_text>Implement the context-sensitive disambiguation method and evaluate its performance. Here are some more hints that may help you along the way:**Hint 1:** The prior probabilities for a Naive Bayes classifier can be specified using the `class_prior` option. You will have to provide the probabilities in the same order as the alphabetically sorted class (entity) names.**Hint 2:** Not all mentions in the knowledge base are ambiguous, and therefore not all mentions have context data. If a mention has only one possible entity, pick that one. If a mention has no entity at all, predict the `--NME--` label.<jupyter_code># TODO: Write code here to implement the context-sensitive disambiguation method<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07302328365355838, 'fraction_numerical': 0.009516968935176872, 'mean_word_length': 4.463701765860039, 'pattern_counts': {'":': 0, '<': 87, '<?xml version=': 0, '>': 87, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '310270', 'license': 'MIT', 'n_tokens_mistral': 4795, 'n_tokens_neox': 4499, 'n_words': 2115, 'path': 'l5/.ipynb_checkpoints/TM-L5-checkpoint.ipynb', 'repo_name': 'lawrence2269/TDDE16', 'size': '16713'}