<jupyter_start><jupyter_text>Named Entity Recognition of the John Henslow Collection------ What is Named Entity Recognition (NER)?The purpose of **named entity recognition** is to extract information from unstructured text, especially where it's impractical to have humans read and markup a large number of documents.A named entity can be any type of real-world object or meaningful concept that is assigned a name or a [proper name](https://en.wikipedia.org/wiki/Proper_nounProper_names). Typically, named entities can include:- people- organisations- countries- languages- locations- works of art- dates- times- numbers- quantitiesExamples of possible entities: a person, an organisation and a language (modern or ancient):    In practice, you can define your own entities that are relevant to the sort of data you are working with. For example, a corpus of archaeological reports might need additional entities for 'culture', 'material' and 'method' in order to extract useful information within the texts.    Recognising named entities means finding and classifying tokens according to the types of entities you have defined. Named entities can be single tokens, such as 'Berlin' (a place), or they can be spans or contiguous sequences of tokens, like 'The Royal Society' (an organisation) and '<NAME>' (a person).        Once you have a corpus tagged with named entities, you could enrich the data further by linking each entity to a corresponding unique identifier from a knowledge base, such as [VIAF](https://viaf.org/). We will cover this further topic in the [last notebook](5-linking-named-entities.ipynb). ------ NER with Machine Learning using spaCy[spaCy](https://spacy.io) is a free and open-source package that can be used to perform automated named entity recognition.If you're interested in learning how to work with spaCy more broadly for a variety of NLP tasks I recommend the beginner's tutorial [Natural Language Processing with spaCy in Python](https://realpython.com/natural-language-processing-spacy-python/) and, for intermediate programmers, the book [Natural Language Processing with Python and spaCy](https://nostarch.com/NLPPython).spaCy uses a form of machine learning called **convolutional neural networks** (CNNs) for named entity recognition. The nature of these neural networks is out of scope for this workshop, but in general terms a CNN produces a **statistical model** that is used to **predict** what are the most likely named entities in a text. This type of machine learning is described as being **supervised**, which means it must be trained on data that has been correctly labelled with named entities before it can do automated labelling on data it's never seen before.As a statistical technique, the predictions might be wrong; in fact, they often are, and it's usually necessary to re-train and adjust the model until its predictions are better. We will cover training a model in the following notebooks.Fortunately, we don't need to start completely from scratch because spaCy provides a range of [pre-trained language models](https://spacy.io/usage/modelslanguages) for modern languages such as English, Spanish, French and German. For other languages you can train your own and use them with spaCy or re-train one of the existing models.The English models that spaCy offers have been trained on the [OntoNotes corpus](https://catalog.ldc.upenn.edu/LDC2013T19). These models can predict the following entities 'out of the box':TypeDescriptionPERSONPeople, including fictional.NORPNationalities or religious or political groups.FACBuildings, airports, highways, bridges, etc.ORGCompanies, agencies, institutions, etc.GPECountries, cities, states.LOCNon-GPE locations, mountain ranges, bodies of water.PRODUCTObjects, vehicles, foods, etc. (Not services.)EVENTNamed hurricanes, battles, wars, sports events, etc.WORK_OF_ARTTitles of books, songs, etc.LAWNamed documents made into laws.LANGUAGEAny named language.DATEAbsolute or relative dates or periods.TIMETimes smaller than a day.PERCENTPercentage, including ”%“.MONEYMonetary values, including unit.QUANTITYMeasurements, as of weight or distance.ORDINAL“first”, “second”, etc.CARDINALNumerals that do not fall under another type.Table from: https://spacy.io/api/annotationnamed-entities    For more detail about how to use spaCy for NER, see the documentation [Named Entity Recognition 101](https://spacy.io/usage/linguistic-featuresnamed-entities-101).An alternative to spaCy is [Natural Language Toolkit (NLTK)](https://www.nltk.org/), which was the first open-source Python library for NLP, originally released in 2001. It is still a valuable tool for teaching and research and has better support in the community for older and non-Indo-European languages; but spaCy is easier to use and faster, which is why I'm using it here. ------ NER in Practice: A Letter from <NAME>, Jr.,  to <NAME> Included with these Jupyter notebooks are a set of data from the Henslow Correspondence Project (HCP), which locates and transcribes the letters of the botanist and mineralogist, [<NAME>](https://www.darwinproject.ac.uk/john-stevens-henslow) (1796–1861). <NAME> was professor of Botany and Mineralogy at the University of Cambridge, and mentor and friend to <NAME>; they exchanged significant letters throughout their lives.Portrait of <NAME>, by <NAME>, 1851.The latest release of letters can be viewed on [Epsilon](https://epsilon.ac.uk/search?sort=date;f1-collection=John%20Henslow), a collaborative platform for reuniting nineteenth-century letters of science.> **Content warning**: The HCP letters were written during the period of British imperialism, therefore some of the correspondence contains content we now find offensive, for example, `letters_138.xml` contains a racist description. These Jupyter notebooks do not contain or discuss any of this material, but please be aware you may come across it if you browse through the letters independently.You can browse the letters included with these notebooks by clicking this link to the folder: [`data/henslow`](data/henslow/) The letters are included here courtesy of HCP and under [Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/).------ Inspecting the Letter XMLWe start with a letter <NAME>. sent to Henslow in 1831 in which he discusses sending and receiving rare plant specimens, including the rather delightful *Arbutus unedo* (Strawberry tree).Arbutus berries (Palombaggia, Corsica)Christy was a Fellow of the Linnean Society and a non-resident member of the Botanical Society of Edinburgh.---> **EXERCISE**: Open the letter now in your browser by clicking this link: [`letters_152.xml`](data/henslow/letters_152.xml).---Your browser should try to display the XML in a relatively friendly form.As you scroll down the letter, notice the various types of information that are marked up in the ``. Collapse or scroll past the `` to the `` element where the body of the letter itself is marked up. We are interested in the contents of **``**.Christy begins:> *I am ashamed that I have never before acknowledged the receipt of the very valuable packet of specimens you were so kind as to send me. Having now got my plants up from Cheshire I shall lose no time in making up a parcel for you.* ------ Opening the Letter with Beautiful Soup[Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) is a library for getting data out of HTML and XML documents. It makes it easy to search documents for particular information. It's often used for *web scraping*, which involves harvesting data from websites automatically. It can also be used to extract information from plain XML files, which is how we will be using it here.First we import what we need from the library:<jupyter_code>from bs4 import BeautifulSoup<jupyter_output><empty_output><jupyter_text>Then we open the letter using the `open()` function and pass the file into `BeautifulSoup`. This is jokingly called "making the soup".<jupyter_code>with open("data/henslow/letters_152.xml", encoding="utf-8") as file:
    letter = BeautifulSoup(file, "lxml-xml")<jupyter_output><empty_output><jupyter_text>The letter has been opened, parsed by `BeautifulSoup` and stored with the name `letter`. Print out the text of the letter below to check the XML is what you expect.<jupyter_code>#Your code here<jupyter_output><empty_output><jupyter_text>When we inspected the XML before, we saw that the actual content of Christy's letter is in the `` element with the attribute `type` that has the value `"transcription"`. Beautiful Soup allows us to pick out text from a document by element, attribute and/or value. In this case, we will use the **`find()`** method and pass in what we are looking for: an element with the `type` attribute and the value `"transcription"`. We only want the textual content, not the whole element including the tag, so we access the `text` attribute.Feel free to experiment and find out what happens if you miss off the attribute `text`.<jupyter_code>transcription = letter.find(type='transcription').text
len(transcription)<jupyter_output><empty_output><jupyter_text>Can you isolate text from other elements in the letter instead? Refer to the Beautiful Soup documentation on [Searching the tree](https://www.crummy.com/software/BeautifulSoup/bs4/doc/searching-the-tree).<jupyter_code>#Your code here<jupyter_output><empty_output><jupyter_text>Cleaning the TranscriptionOne small point of cleaning we will do now will make a big difference to the quality of NER later on. The style of writing in these letters is to replace the word 'and' with the ampersand (`&`). Experience tells me that spaCy's language model doesn't handle this very well because it is different in style to the texts it was trained on.<jupyter_code>transcription = transcription.replace('& ', 'and ')<jupyter_output><empty_output><jupyter_text>Ideally, we would instead re-train the language model to better understand the use of ampersands. In this notebook, I just want to show you a good example.As you follow along, you may notice more aspects of the text that could or should be cleaned. Do make a note of your thoughts. ------ A Pre-Trained Language ModelWe can now move into the named entity recognition using spaCy. We are going to use a pre-trained English language model provided by spaCy called `'en_core_web_sm'`. The elements of this model's name break down as follows:* `'en'`: the language code, in this case, English* `'core'`: the capabilities for the model - 'core' is general purpose * `'web'`: the type of text the model was trained on* `'sm'`: how large the model is when stored on disk - 'sm' means smallA full list of the [available pre-trained language models](https://spacy.io/usage/modelslanguages) is available.To start, we import and load the pre-trained English language model and give it the name `nlp`, ready to do the work on the transcription. (The name could be anything, but `nlp` is used by convention, and you will see this name used in examples and tutorials.)<jupyter_code>import en_core_web_sm
nlp = en_core_web_sm.load()<jupyter_output><empty_output><jupyter_text>Before we go any further, let's take a moment to examine the metadata of this language model, and remind ourselves of what it consists. We can do this with the `meta` attribute:<jupyter_code>nlp.meta<jupyter_output><empty_output><jupyter_text>Scroll through this metadata and in particular note the `'description'` and `'sources'`, including information on what data the model is trained on, and `'ner'` labels, which should match those listed near the top of this notebook.If you have forgotten already what a label stands for (some of them are rather cryptic!) then you can ask spaCy to give you a human-readable explanation:<jupyter_code>import spacy
spacy.explain('FAC')<jupyter_output><empty_output><jupyter_text>Note also the `'accuracy'` statistics, for your interest. The ones relevant to us are:`'ents_f': 85.5515654809, 'ents_p': 85.8937524646, 'ents_r': 85.212094115`These were created when the model was evaluated and give you an idea of how well it performs. `'ents_p'`, `'ents_r'`, `'ents_f'` stand for "precision", "recall" and "fscore", respectively.**Precision** is the percentage of entities predicted that are actually relevant (_aka_ "positive predictive value"); **recall** is the percentage of relevant entities that were actually predicted (_aka_ "sensitivity"); the **fscore** combines the precision and recall into a single average score.Put rather simplistically, we can say that this model is about 85% accurate. ------ Recognising Named Entities with spaCyNow, we can pass the transcription into the language model and spaCy does the rest. It returns to us a `Doc` object (which we name `document`) that contains all the **tokens** and **annotations** it has created. We print out the document text stored on the attribute `text` just to check that spaCy has correctly parsed the transcription.<jupyter_code>document = nlp(transcription)
document.text<jupyter_output><empty_output><jupyter_text>But what has actually happened? Behind the scenes spaCy has done the following tasks:* tokenizing and lemmatizing* part-of-speech tagging ('tagger')* syntactic dependency parsing ('parser')* named entity recognition ('ner')This series of tasks is called a **processing pipeline** and each individual task is known as a **component**.Of all these components (tasks), the tokenizer is essential and must happen for every pipeline; the rest are optional, depending on what you want to do with your text.In the metadata for this model (`nlp.meta` above), you can see listed the pipeline components:`'pipeline': ['tagger', 'parser', 'ner']`This is the default processing pipeline included with the `'en_core_web_sm'` language model. You can read more about [spaCy Processing Pipelines](https://spacy.io/usage/processing-pipelines).As we only need the `'ner'` component, we could speed up our code by disabling the tagger and the parser like this:`nlp = en_core_web_sm.load(disable=['tagger', 'parser'])`<jupyter_code># Write your code here to compare the performance of the model with and without the unnecessary components
# Hint: you could use the Jupyter magic command `%%time` or `import time` and use `time.time()`<jupyter_output><empty_output><jupyter_text>Now, we just need to inspect the data. All the named entities are available on the `ents` attribute. The following code loops over all the named entities and prints each of them out with the labels that spaCy has predicted.<jupyter_code>for entity in document.ents:
    print(f'{entity.text}: {entity.label_}')<jupyter_output><empty_output><jupyter_text>What do you think of the accuracy of the labelling?  Exporting Named Entities in JSON FormatWe can export these named entities in a JSON format, which can be saved in a file or used as input to other spaCy functions.<jupyter_code># Create a dictionary form of the document data 
doc_dict = document.to_json()
doc_dict
# Get just the named entities item
ents_dict = {key: value for (key, value) in doc_dict.items() if key == 'ents'}
ents_dict
# Import a module for handling json
import json

# Export the named entities dictionary in json format
json.dumps(ents_dict)<jupyter_output><empty_output><jupyter_text>------ Visualising Named Entities with displaCyThe list above could be useful, but it is much easier as a human to understand how accurate the NER has been if we can see where the named entities fall within the context of the document.Fortunately, spaCy provides a nice visualiser called **displacy** that does this for us.<jupyter_code>from spacy import displacy
displacy.render(document, style="ent")<jupyter_output><empty_output><jupyter_text>---> **EXERCISE**: Inspect the visualisation above and make a note of what spaCy has predicted well and what it is has got wrong. Has it mislabelled some tokens? Has it missed some entities out? Has it got some of the spans wrong?--- We can also **save** the results of a visualisation as an HTML file to review later:<jupyter_code># Import a module that helps with filepaths
from pathlib import Path

# Create a filepath for the output file
output_file = Path("output/ent_viz.html")

# Give the document a title for reference
document.user_data["title"] = "Letter from <NAME>, Jr., to <NAME>, 26 February 1831"

# Output the visualisation as HTML
html = displacy.render(document, style='ent', jupyter=False, page=True)

# Write the HTML to the output file
output_file.open("w", encoding="utf-8").write(html)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07461519961519962, 'fraction_numerical': 0.006132756132756133, 'mean_word_length': 5.305155420773313, 'pattern_counts': {'":': 0, '<': 72, '<?xml version=': 0, '>': 76, 'https://': 17, 'lorem ipsum': 0, 'www.': 4, 'xml': 6}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '355792', 'license': 'MIT', 'n_tokens_mistral': 4484, 'n_tokens_neox': 4214, 'n_words': 2216, 'path': '2-named-entity-recognition-of-henslow-data.ipynb', 'repo_name': 'mchesterkadwell/named-entity-recognition', 'size': '16730'}