<jupyter_start><jupyter_text>![UKDS Logo](images/UKDS_Logos_Col_Grey_300dpi.png)  Text-mining: Basics Welcome to the UK Data Service training series on *New Forms of Data for Social Science Research*. This series guides you through some of the most common and valuable new sources of data available for social science research: data collected from websites, social media platorms, text data, conducting simulations (agent based modelling), to name a few. We provide webinars, interactive notebooks containing live programming code, reading lists and more.* To access training materials for the entire series: [Training Materials]* To keep up to date with upcoming and past training events: [Events]* To get in contact with feedback, ideas or to seek assistance: [Help]Dr <NAME> and Dr <NAME> UK Data Service  University of Manchester May 2020 Table of Contents1&nbsp;&nbsp;Introduction2&nbsp;&nbsp;Preliminary NLP (or finishing up the processing)2.1&nbsp;&nbsp;POS - part of speech tagging2.2&nbsp;&nbsp;Named Entity Recogntion and chunking3&nbsp;&nbsp;Counts and (relative) frequency4&nbsp;&nbsp;Similarity5&nbsp;&nbsp;Discovery6&nbsp;&nbsp;Conclusions6.1&nbsp;&nbsp;Further reading There is a table of contents provided here at the top of the notebook, but you can also access this menu at any point by clicking the Table of Contents button on the top toolbar (an icon with four horizontal bars, if unsure hover your mouse over the buttons).   Introduction At the end of the last section, we got held up by the need to do part-of-speech (POS) tagging in order to get a really effective lemmatisation process. POS tagging is actually basic NLP, as opposed to the kind of cleaning and regularising that we were doing in the last section. So why should a NLP process be needed before the preparatory processing is done? Well, it comes done to choices. Not every analysis will need a sophisticated lemmatiser, so those projects may have a nice finish to the processing step and the start of the extraction or NLP step. Others willneed the lemmatiser or name entity recognisers or other advanced preparatory steps. Those projects will have a less clear distinction between preparation for NLP and NLP. But even if the project ends up having a clear distinction between the processes, researchers may find that after they start doing some NLP processes, they need to go back and run different preparatory processes instead of or in addition to the ones they chose earlier. The main takeaway point here is that researchers need to know that developing a text-mining project can be messy, iterative, and complicated. I recommend that you think about each step as elements in a pipeline (or in multiple pipelines). I recommend that you build your own code functions that concatenate the steps, running each one from the output of the previous one. In this way, you get a fresh clean output at the end of the pipeline each time whenever you need one. It also means that everything you apply the pipeline to gets treated in the same way, with each process done in the same order. This helps replicability!  Preliminary NLP (or finishing up the processing) Let's start off by importing and downloading all the things we will need. Run/Shift+Enter.<jupyter_code>import nltk                       # get nltk 
from nltk import word_tokenize    # and some of its key functions
from nltk import sent_tokenize    

!pip install autocorrect          
from autocorrect import Speller   # things we need for spell checking
check = Speller(lang='en')

import re                         # things we need for RegEx corrections
def multiple_replace(dict, text):
  regex = re.compile("(%s)" % "|".join(map(re.escape, dict.keys())))
  return regex.sub(lambda mo: dict[mo.string[mo.start():mo.end()]], text) 

if __name__ == "__main__": 
  dict = {
    "CA" : "California",
    "United Kingdom" : "U.K.",
    "United Kingdom of Great Britain and Northern Ireland" : "U.K.",
    "United Kingdom of Great Britain" : "U.K.",
    "UK" : "U.K.",
    "Privacy Policy" : "noodle soup",}

English_punctuation = "-!\"#$%&()'*+,./:;<=>?@[\]^_`{|}~''“”"      # Things for removing punctuation, stopwords and empty strings
table_punctuation = str.maketrans('','', English_punctuation)   
nltk.download('stopwords')
from nltk.corpus import stopwords
stop_words = set(stopwords.words('english'))

from nltk.corpus import wordnet                    # Finally, things we need for lemmatising!
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer() 
nltk.download('averaged_perceptron_tagger')        # Like a POS-tagger...

print("Succesfully imported necessary modules")    # The print statement is just a bit of encouragement!<jupyter_output>Requirement already satisfied: autocorrect in c:\programdata\anaconda3\lib\site-packages (1.3.0)
Succesfully imported necessary modules<jupyter_text>POS - part of speech tagging Now, let's get back to where we were when we left off last time - with a tokenised corpus on which we need to run a POS tagger. Run/Shift+Enter, as above!<jupyter_code>with open("./data/sample_text.txt", "r") as f:
    corpus = f.read()
    
corpus_words = word_tokenize(corpus)

corpus_lower = [word.lower() for word in corpus_words]

corpus_correct_spell = []
for word in corpus_lower:
    corpus_correct_spell.append(check(word))    

corpus_no_stopwords = []
for word in corpus_correct_spell:
    if word not in stop_words:
        corpus_no_stopwords.append(word)
        
corpus_no_punct = [w.translate(table_punctuation) for w in corpus_no_stopwords]  
corpus_no_space = list(filter(None, corpus_no_punct))      
        
print(corpus_no_space[:100])<jupyter_output>['sample', 'corpus', 'spelling', 'errors', 'numbers', 'written', 'two', 'ways', 'example', '1972', 'ninetysix', 'sample', 'corpus', 'also', 'uses', 'abbreviations', 'sometimes', 'always', 'california', 'spelled', 'also', 'written', 'ca', 'really', 'complicate', 'things', 'another', 'country', 'name', 'written', 'uk', 'uk', 'united', 'kingdom', 'united', 'kingdom', 'great', 'britain', 'united', 'kingdom', 'great', 'britain', 'northern', 'ireland', 'sometimes', 'full', 'names', 'important', 'bunch', 'unrelated', 'text', 'fill', 'space', 'privacy', 'policy', 'privacy', 'policy', 'intended', 'inform', 'policies', 'practices', 'regarding', 'collection', 'use', 'disclosure', 'personal', 'information', 'site', 'sites', 'links', 'privacy', 'policy', 'site', 'define', 'personal', 'information', 'information', 'allows', 'someone', 'identify', 'personally', 'contact', 'including', 'example', 'name', 'address', 'telephone', 'number', 'email', 'address', 'registering', 'us', 'using', 'site', 'expre[...]<jupyter_text>Excellent. Now it is time to tag that corpus with POS-tags. This is pretty easy, as nltk comes with a POS-tagger. Run/Shift+Enter, as you would expect.<jupyter_code>corpus_pos_tagged = nltk.pos_tag(corpus_no_space)        
print(corpus_pos_tagged[:100])<jupyter_output>[('sample', 'NN'), ('corpus', 'NN'), ('spelling', 'VBG'), ('errors', 'NNS'), ('numbers', 'NNS'), ('written', 'VBN'), ('two', 'CD'), ('ways', 'NNS'), ('example', 'NN'), ('1972', 'CD'), ('ninetysix', 'JJ'), ('sample', 'NN'), ('corpus', 'NN'), ('also', 'RB'), ('uses', 'VBZ'), ('abbreviations', 'NNS'), ('sometimes', 'RB'), ('always', 'RB'), ('california', 'NN'), ('spelled', 'VBD'), ('also', 'RB'), ('written', 'VBN'), ('ca', 'MD'), ('really', 'RB'), ('complicate', 'VB'), ('things', 'NNS'), ('another', 'DT'), ('country', 'NN'), ('name', 'NN'), ('written', 'VBN'), ('uk', 'JJ'), ('uk', 'JJ'), ('united', 'JJ'), ('kingdom', 'NN'), ('united', 'JJ'), ('kingdom', 'NN'), ('great', 'JJ'), ('britain', 'NN'), ('united', 'JJ'), ('kingdom', 'NN'), ('great', 'JJ'), ('britain', 'NN'), ('northern', 'JJ'), ('ireland', 'NN'), ('sometimes', 'RB'), ('full', 'JJ'), ('names', 'NNS'), ('important', 'JJ'), ('bunch', 'NN'), ('unrelated', 'JJ'), ('text', 'NN'), ('fill', 'NN'), ('space', 'NN'), ('privacy', 'NN'), ('po[...]<jupyter_text>Excellent. That has successfully added POS tags to all off the words in our corpus. Now, let's try lemmatising again with the POS tags. Despite what seems obvious, the nltk POS tagger does not use the same POS tags that the nltk lemmatize function needs. Why? I have no idea. But to move forward, I need a to define a quick little function called get_wordnet_pos to convert the tag format to the right one. I tell a lie. I did not write this function but copied it off of Stack Overflow. This is not cheating so much as being economical. A HUGE number of the things you want to do or the problems you want to solve will be discussed on Stack Overflow. Just use a popular search engine to find them, read through all the answers, try them out. Having defined the get_wordnet_pos function, the code belowe then creates a new, blank list called corpus_lemmed. After that, the code iterates over corpus_pos_tagged, looking at each word and POS-tag pair, uses the get_wordnet_pos function to convert the POS-tag to the right format, and using that to lemmatize correctly. At the end, the lemmatised word is appended to the new list we created. Go ahead. Run/Shift+Enter. You know you want to!<jupyter_code>def get_wordnet_pos(word):
    """Map POS tag to first character lemmatize() accepts"""
    tag = nltk.pos_tag([word])[0][1][0].upper()
    tag_dict = {"J": wordnet.ADJ,
                "N": wordnet.NOUN,
                "V": wordnet.VERB,
                "R": wordnet.ADV}
    return tag_dict.get(tag, wordnet.NOUN)

corpus_lemmed = []
for pair in corpus_pos_tagged:
    corpus_lemmed.append(lemmatizer.lemmatize(pair[0], get_wordnet_pos(pair[0])))   
print(corpus_lemmed[:100])

#corpus_lemmed_tagged = []    
#for pair in corpus_pos_tagged:
#    corpus_lemmed_tagged.append([(lemmatizer.lemmatize(pair[0], get_wordnet_pos(pair[0])), pair[1])])   
#print(corpus_lemmed_tagged[:100])<jupyter_output>['sample', 'corpus', 'spell', 'error', 'number', 'write', 'two', 'way', 'example', '1972', 'ninetysix', 'sample', 'corpus', 'also', 'us', 'abbreviation', 'sometimes', 'always', 'california', 'spell', 'also', 'write', 'ca', 'really', 'complicate', 'thing', 'another', 'country', 'name', 'write', 'uk', 'uk', 'united', 'kingdom', 'united', 'kingdom', 'great', 'britain', 'united', 'kingdom', 'great', 'britain', 'northern', 'ireland', 'sometimes', 'full', 'name', 'important', 'bunch', 'unrelated', 'text', 'fill', 'space', 'privacy', 'policy', 'privacy', 'policy', 'intend', 'inform', 'policy', 'practice', 'regard', 'collection', 'use', 'disclosure', 'personal', 'information', 'site', 'site', 'link', 'privacy', 'policy', 'site', 'define', 'personal', 'information', 'information', 'allows', 'someone', 'identify', 'personally', 'contact', 'include', 'example', 'name', 'address', 'telephone', 'number', 'email', 'address', 'register', 'u', 'use', 'site', 'expressly', 'consent', 'collection', 'use'[...]<jupyter_text>The output of the above code returns a list with only words but without any POS-tags. If you want to keep the corpus in pairs of word and POS-tag, you will need to activate the second, commented out lines. This means you will need to  remove the '' in front of each line of code starting with 'corpus_lemmed_tagged' and re-run the code.Give it a try!  Named Entity Recogntion and chunking We are really getting somewhere now! Let's try another basic NLP process - CHUNKING!Named Entity Recognition is a specific kind of 'chunk' operation. Chunking operations iterate over a corpus that has been word tokenised and POS-tagged and put it all back together into sentences. Named Entity Recognition does this too, with special attention to building up the noun phrases that capture well-known entities or onganisations. The chunks are returned within sets of nested brackets (both square and round to capture different levels of nesting. So, 'The Cat in the Hat' would come out as (S The/DT (ORGANIZATION Cat/NNP) in/IN the/DT Hat/NNP). The 'S' at the beginning stands for 'sentence' which is the highest level grouping that the chunker can find. the 'Cat' is recognised as the key entity, so is tagged with ORGANIZATION. The 'in the hat' part is captured as belonging to a noun phrase, same as the cat, but it recongsises that this a sentence about a cat, not a sentence about a hat. Clever, eh?Let's try it!<jupyter_code>#importing chunk library from nltk
from nltk import ne_chunk                                    # ne_chunk is 'named entity chunk'. Other chunkers are available.

# NER and other chunkers only work on word tokenised and POS tagged corpora... 
corpus_pos_tagged2 = nltk.pos_tag(corpus_words)
corpus_chunked = ne_chunk(corpus_pos_tagged2)
print(corpus_chunked[57:88])<jupyter_output>[('name', 'NN'), ('is', 'VBZ'), ('written', 'VBN'), ('as', 'IN'), ('the', 'DT'), ('U.K.', 'NNP'), (',', ','), ('the', 'DT'), Tree('ORGANIZATION', [('UK', 'NNP')]), (',', ','), ('the', 'DT'), Tree('ORGANIZATION', [('United', 'NNP'), ('Kingdom', 'NNP')]), (',', ','), ('the', 'DT'), Tree('ORGANIZATION', [('United', 'NNP'), ('Kingdom', 'NNP')]), ('of', 'IN'), Tree('GPE', [('Great', 'NNP'), ('Britain', 'NNP')]), ('and', 'CC'), ('The', 'DT'), Tree('ORGANIZATION', [('United', 'NNP'), ('Kingdom', 'NNP')]), ('of', 'IN'), Tree('GPE', [('Great', 'NNP'), ('Britain', 'NNP')]), ('and', 'CC'), Tree('GPE', [('Northern', 'NNP'), ('Ireland', 'NNP')]), ('becuase', 'NN'), ('sometimes', 'RB'), ('full', 'JJ'), ('names', 'NNS'), ('are', 'VBP'), ('important', 'JJ'), ('.', '.')]<jupyter_text>This time, I specifically only asked for a printout of a key range in the resulting corpus. I wanted to highlight here how the word "Tree" preceeds those ORGANIZATION entities that hang together as multi-word entities. See, for example, how 'United Kingdom', 'Great Britain' and 'Northern Ireland' are each within square brackets to identify them as the multi-word entity captured by the ORGANIZATION tag. You might also have noticed that this chunking function is run on corpus_pos_tagged2, which is simply the corpus_words that has been put through the nltk.pos_tag function. This means that corpus_pos_tagged2 still has its stopwords, punctuation, etc. Why do you think this is? What do you think would happen if you ran the chunking procedure on corpus_pos_tagged which DOES have all the stopwords and punctuation removed?Well, guess what? You can find out by doing that Run/Shift+Enter thing!<jupyter_code>corpus_chunked_extra_processes = ne_chunk(corpus_pos_tagged)
print(corpus_chunked_extra_processes[:100])<jupyter_output>[('sample', 'NN'), ('corpus', 'NN'), ('spelling', 'VBG'), ('errors', 'NNS'), ('numbers', 'NNS'), ('written', 'VBN'), ('two', 'CD'), ('ways', 'NNS'), ('example', 'NN'), ('1972', 'CD'), ('ninetysix', 'JJ'), ('sample', 'NN'), ('corpus', 'NN'), ('also', 'RB'), ('uses', 'VBZ'), ('abbreviations', 'NNS'), ('sometimes', 'RB'), ('always', 'RB'), ('california', 'NN'), ('spelled', 'VBD'), ('also', 'RB'), ('written', 'VBN'), ('ca', 'MD'), ('really', 'RB'), ('complicate', 'VB'), ('things', 'NNS'), ('another', 'DT'), ('country', 'NN'), ('name', 'NN'), ('written', 'VBN'), ('uk', 'JJ'), ('uk', 'JJ'), ('united', 'JJ'), ('kingdom', 'NN'), ('united', 'JJ'), ('kingdom', 'NN'), ('great', 'JJ'), ('britain', 'NN'), ('united', 'JJ'), ('kingdom', 'NN'), ('great', 'JJ'), ('britain', 'NN'), ('northern', 'JJ'), ('ireland', 'NN'), ('sometimes', 'RB'), ('full', 'JJ'), ('names', 'NNS'), ('important', 'JJ'), ('bunch', 'NN'), ('unrelated', 'JJ'), ('text', 'NN'), ('fill', 'NN'), ('space', 'NN'), ('privacy', 'NN'), ('po[...]<jupyter_text>Hmmm. No 'Tree' markers, no 'ORGANIZATION' markers, etc. This is because some chunking processes use some of the stopwords (especially determiners like 'an' and 'the') and punctuation, etc. to be useful in determining appropriate chunks. This may create some challenges for your corpus. For example, if you want to: - Count words, then you probably want to remove stopwords, punctuation, etc. - Identify chunks, like named entities, then you probably want to leave some or all of the stopwords, punctuation, etc.- Count chunks (e.g. count named entities), you probably want to combine the processes in the right order. Good to know!  Counts and (relative) frequency Excellent! Now, you might be surprised, but a very important function of NLP for analysing text boils down to counting things, often words. This is why so much attention in the last section was focussed on making sure all the words that we want to be counted as 'the same word' appeared in the same form while all the words that we want to count as 'different words' appear in different forms. Thus, we want to apply the count functions to a corpus that has had some of that standardisation, consolidation, lemmatised (or at least stemming) processes applied already. First, we import some counting functions, then we apply them to corpus_lemmed. Run/Shift+Enter<jupyter_code>from collections import Counter
corpus_counts = Counter(corpus_lemmed)
print(corpus_counts)<jupyter_output>Counter({'information': 123, 'site': 80, 'use': 74, 'may': 64, 'personal': 58, 'policy': 40, 'u': 37, 'privacy': 34, 'service': 34, 'cooky': 34, 'email': 28, 'collect': 28, 'web': 26, 'address': 22, 'third': 22, 'party': 22, 'provide': 20, 'account': 18, 'visit': 18, 'also': 16, 'beacon': 16, 'company': 16, 'regard': 14, 'include': 14, 'share': 14, 'website': 14, 'track': 14, 'law': 12, 'term': 12, 'social': 12, 'networking': 12, 'browser': 12, 'time': 12, 'analytics': 12, 'receive': 12, 'choice': 12, 'name': 10, 'link': 10, 'request': 10, 'send': 10, 'page': 10, 'user': 10, 'online': 10, 'content': 10, 'promotional': 10, 'communication': 10, 'change': 10, 'http': 9, 'disclosure': 8, 'register': 8, 'give': 8, 'preference': 8, 'indicate': 8, 'connection': 8, 'make': 8, 'please': 8, 'associate': 8, 'keep': 8, 'anonymous': 8, 'affiliate': 8, 'delete': 8, 'product': 8, 'number': 7, 'united': 7, 'collection': 6, 'state': 6, 'password': 6, 'andor': 6, 'fax': 6, 'technology': 6, 'experience':[...]<jupyter_text>Great! You may have noticed that we applied this count function to a list of words rather than word and POS-tag pairs. This is on purpose, but the code could be written so that it only looks at the first item in each word and POS-tag pairs. If you want to try that, go ahead. You may want to refer back to the code block where we defined get_wordnet_pos because the code to create corpus_lemmed_tagged uses indices (in [square brackets]) to refer to only one element within a pair. But, for now, let's have a closer look at the 100 most common words in our corpus by using the 'most_common' function from Counter. Run/Shift+Enter!<jupyter_code>print(corpus_counts.most_common(100))<jupyter_output>[('information', 123), ('site', 80), ('use', 74), ('may', 64), ('personal', 58), ('policy', 40), ('u', 37), ('privacy', 34), ('service', 34), ('cooky', 34), ('email', 28), ('collect', 28), ('web', 26), ('address', 22), ('third', 22), ('party', 22), ('provide', 20), ('account', 18), ('visit', 18), ('also', 16), ('beacon', 16), ('company', 16), ('regard', 14), ('include', 14), ('share', 14), ('website', 14), ('track', 14), ('law', 12), ('term', 12), ('social', 12), ('networking', 12), ('browser', 12), ('time', 12), ('analytics', 12), ('receive', 12), ('choice', 12), ('name', 10), ('link', 10), ('request', 10), ('send', 10), ('page', 10), ('user', 10), ('online', 10), ('content', 10), ('promotional', 10), ('communication', 10), ('change', 10), ('http', 9), ('disclosure', 8), ('register', 8), ('give', 8), ('preference', 8), ('indicate', 8), ('connection', 8), ('make', 8), ('please', 8), ('associate', 8), ('keep', 8), ('anonymous', 8), ('affiliate', 8), ('delete', 8), ('product', 8), ('numb[...]<jupyter_text>Just for comparison, let's find the 100 most common words in 'Emma' by <NAME>. We do need to import the text as a corpus and process it in the same way as we did your corpus so that they can be seen as comparable. Run/Shift+Enter  - but be patient. This is a lot of processes to run.<jupyter_code>import nltk.corpus
emma = nltk.corpus.gutenberg.raw('austen-emma.txt')

emma_words = word_tokenize(emma)

emma_lower = [word.lower() for word in emma_words]

emma_correct_spell = []
for word in emma_lower:
    emma_correct_spell.append(check(word))    

    emma_no_stopwords = []
for word in emma_lower:
    if word not in stop_words:
        emma_no_stopwords.append(word)
        
emma_no_punct = [w.translate(table_punctuation) for w in emma_no_stopwords]  
emma_no_space = list(filter(None, emma_no_punct))              
emma_pos_tagged = nltk.pos_tag(emma_no_space)   

emma_lemmed = []
for pair in emma_pos_tagged:
    emma_lemmed.append(lemmatizer.lemmatize(pair[0], get_wordnet_pos(pair[0])))   
    
emma_counts = Counter(emma_lemmed)
print(emma_counts)<jupyter_output>Counter({'mr': 1855, 'say': 878, 's': 868, 'emma': 861, 'could': 837, 'would': 820, 'miss': 614, 'must': 571, 'well': 557, 'harriet': 501, 'much': 486, 'think': 467, 'thing': 461, 'one': 457, 'know': 451, 'weston': 447, 'every': 435, 'make': 420, 'come': 403, 'go': 384, 'knightley': 383, 'elton': 382, 'little': 361, 'never': 358, 'give': 353, 'see': 349, 'might': 326, 'great': 324, 'good': 314, 'woodhouse': 311, 'time': 307, 'jane': 301, 'look': 301, 'quite': 282, 'thought': 265, 'friend': 259, 'take': 258, 'nothing': 254, 'dear': 243, 'fairfax': 239, 'always': 238, 'churchill': 238, 'man': 236, 'even': 227, 'like': 224, 'soon': 224, 'may': 223, 'wish': 221, 'shall': 219, 'without': 214, 'seem': 212, 'day': 211, 'want': 210, 'frank': 208, 'father': 207, 'first': 207, 'sure': 204, 'indeed': 202, 'young': 198, 'body': 194, 'ever': 193, 'oh': 193, 'talk': 183, 'two': 179, 'though': 177, 'way': 171, 'hope': 164, 'feel': 161, 'hartfield': 160, 'upon': 159, 'walk': 156, 'love': 153, 'really'[...]<jupyter_text>Excellent! Clearly, the words in Emma are very different than those of our sample corpus, and those words that appear in both occur in very different relative frequencies (not least because one is a page of babble and the other is a full novel).To get a better idea, how about we compare the 20 most common words from both corpora. Run/Shift+Enter<jupyter_code>print(corpus_counts.most_common(20))
print(emma_counts.most_common(20))<jupyter_output>[('information', 123), ('site', 80), ('use', 74), ('may', 64), ('personal', 58), ('policy', 40), ('u', 37), ('privacy', 34), ('service', 34), ('cooky', 34), ('collect', 28), ('web', 26), ('address', 22), ('third', 22), ('party', 22), ('provide', 20), ('e-mail', 20), ('account', 18), ('visit', 18), ('also', 16)]
[('--', 3100), ("''", 2452), ('mr', 1855), ('say', 878), ("'s", 866), ('emma', 860), ('could', 837), ('would', 820), ('miss', 614), ('must', 571), ('well', 557), ('harriet', 501), ('much', 486), ('think', 467), ('thing', 461), ('one', 457), ('know', 451), ('weston', 440), ('every', 435), ('make', 420)]<jupyter_text>Ok. Clearly very different. But let's try one more thing for now... Let's count how many times each of these texts use the word 'personal'. We could use any word as the target word, but I happen to know that there is a non-zero result for these two texts for this word. Run/Shift+Enter!!!<jupyter_code>print(corpus_counts['personal'])
print(emma_counts['personal'])<jupyter_output>58
7<jupyter_text>So, despite being much shorter, the sample text corpus uses the word 'personal' over 8 times more often. That sounds very personal. Feel free to choose other words and re-run the code.   Similarity Now, comparing the most common words in two documents is one way to compare how similar they are, but there are more sophisticated ways. spaCy is a relatively new option for text-mining in python, but it is very powerful. First off, we need to download and import a few things. Run/Shift+Enter (you are already so good at this!)<jupyter_code>!pip install spacy -q
import spacy
!python -m spacy download en_core_web_lg -q
from nltk.corpus import webtext<jupyter_output>[+] Download and installation successful
You can now load the model via spacy.load('en_core_web_lg')<jupyter_text>Super. Now, let's load the model via spacy-load, and then test it on a trivial corpus that has only three words. Run/Shift+Enter already!<jupyter_code>nlp = spacy.load('en_core_web_lg')

word_similarity = nlp("troll elf rabbit")


for word1 in word_similarity:
    for word2 in word_similarity:
        print(word1.text, word2.text, word1.similarity(word2))<jupyter_output>troll troll 1.0
troll elf 0.40669888
troll rabbit 0.2995838
elf troll 0.40669888
elf elf 1.0
elf rabbit 0.34079304
rabbit troll 0.2995838
rabbit elf 0.34079304
rabbit rabbit 1.0<jupyter_text>This code does a few things. First, it loads a model of common words in English (this is the 'en_core_web_lg') that has 300 dimension vectors for each word. If that sounds like nonsense, don't worry too much. What it means is that the model has a list of lots of common words in English, each of which comes with a 'scorecard' of how they rank on 300 different features which is a sort of abstract way of capturing the meaning of the word. This is not derived from logical scoring by people, but through an AI sort of analysis of how the words are used in LOTS of text, which finds patterns like:- Is the target word used more often like a noun or a verb? - Is it usually plural (if a noun) or in gerrund ('ing'-form, if a verb)? - Is it frequently preceded by adjectives like 'little' or 'unprecedented' or adverbs like 'always' or 'never'? What comes out of this code is a pair-wise comparison of all the vectors, or scorecards, for the words in our little three-word corpus. This comes out as a number between 0 and 1, with 0 being totally different (or not found in the model) and 1 being a perfect match. Looking at the results, we see that comparing a word to itself (e.g. the first line which has 'dog dog 1.0') scores a 1, or 100 percent match. Not surprising. We also see that 'dog' and 'cat' are a pretty good match at 0.8. Both words are likely to be used in similar ways. For example, both would fit equally well into a sentence like "I really want to get a pet (dog/cat), but I just don't spend enough time at home to take care of it properly."We also see that 'banana' is closer to 'cat' than to 'dog', but not by much. Presumably, bananas are more likely to sit around like cats than to run around like dogs? No idea. Feel free to edit the little three-word corpus and re-run the similarity test. Try 'puppy' to see if it is closer to 'dog' than 'dog' is to 'cat'? Try adding 'apple'? Or 'unprecedented'? Or anything else? Of course, comparing individual words is all well and good, but what you probably want to compare is one text to another. To do that, first we need to prepare a few texts to do some comparing. You have already seen our sample corpus and 'Emma' by <NAME>, but let's also add 'Persuasion' by <NAME> and a selection of text from another nltk.corpus of texts from the web. The specific text is called 'firefox'. All of these texts need to be put through the nlp function we created from spaCy so that it creates a document vector. Document vectors are much like word vectors in that they score the document on a large number of dimensions. However, instead of coming packaged with spaCy, they are created from the text that you pass to spaCy. Which is what we are doing now.Run/Shift+Enter below!<jupyter_code>SimEmma = nlp(nltk.corpus.gutenberg.raw('austen-emma.txt'))
SimPers = nlp(nltk.corpus.gutenberg.raw('austen-persuasion.txt'))
SimFire = nlp(nltk.corpus.webtext.raw('firefox.txt'))
SimCorp = nlp(corpus)<jupyter_output><empty_output><jupyter_text>You can, of course, have a peek at the contents of 'Persuasion' or 'firefox' if you like. You probably know how to do that with a print command, but maybe you want to run some of the other operations on the text too. Or you can plow on ahead and Run/Shift+Enter to run the document vector similarity comparisons below.<jupyter_code>print(SimEmma.similarity(SimPers))
#print(SimEmma.similarity(SimCeas))
print(SimEmma.similarity(SimFire))
print(SimEmma.similarity(SimCorp))<jupyter_output>0.9986067112131936
0.8631958218382987
0.9142934448845454<jupyter_text>Each of these compares 'Emma' to one of the other texts (the ones in parentheses at the end). Are you surprised by the results? Feel free to try comparing the other texts to each other, rather than just to 'Emma'.   Discovery Now, for the final bit of NLP that we cover here, let's talk about discovery. This is about identifying patterns that reveal relationships and applying it more widely to discover additional relationships. Let's start by importing a few things that we need. Run/Shift+Enter.<jupyter_code>import re 
import string 
import nltk 
import spacy 
import pandas as pd 
import numpy as np 
import math 
from tqdm import tqdm 

from spacy.matcher import Matcher 
from spacy.tokens import Span 
from spacy import displacy 

pd.set_option('display.max_colwidth', 200)<jupyter_output><empty_output><jupyter_text>Now, let's take a look at 'Emma'. We start by tokenising the raw text into sentences, then we create a list of all sentences that contain the sub-string "like a", then we create run our list through our nlp function from spaCy. Run/Shift+Enter.<jupyter_code># sample text 
emma_sentences = nltk.sent_tokenize(nltk.corpus.gutenberg.raw('austen-emma.txt'))
emma_such_as =""
for sentence in emma_sentences:
    if "like a " in sentence:
        emma_such_as += sentence
                
# create a spaCy object 
doc = nlp(emma_such_as)<jupyter_output>His appearance was very neat, and he looked like a sensible young man,
but his person had no other advantage; and when he came to be
contrasted with gentlemen, she thought he must lose all the ground
he had gained in Harriet's inclination.--
Yet he would be so anxious for her being perfectly warm, would be
so interested about her father, and so delighted with Mrs. Weston;
and at last would begin admiring her drawings with so much zeal
and so little knowledge as seemed terribly like a would-be lover,
and made it some effort with her to preserve her good manners.When she looked at the hedges, she thought the elder at
least must soon be coming out; and when she turned round to Harriet,
she saw something like a look of spring, a tender smile even there.He begged to be shewn the house which his father had lived in so long,
and which had been the home of his father's father; and on recollecting
that an old woman who had nursed him was still living, walked in quest
of her cottage from one end[...]<jupyter_text>Now, let's take a closer look at context around those instances of "like a". To do that, we use some spaCy functions that print the word, print its role in the sentence, and print its POS-tag. This lets us see if we can find any patterns in the word roles or POS-tags that might help us understand the patterns relating to "like a". Run/Shift+Enter.<jupyter_code># print token, dependency, POS tag 
for tok in doc: 
  print(tok.text, "-->",tok.dep_,"-->", tok.pos_)<jupyter_output>His --> poss --> DET
appearance --> nsubj --> NOUN
was --> ROOT --> AUX
very --> advmod --> ADV
neat --> acomp --> ADJ
, --> punct --> PUNCT
and --> cc --> CCONJ
he --> nsubj --> PRON
looked --> conj --> VERB
like --> prep --> SCONJ
a --> det --> DET
sensible --> amod --> ADJ
young --> amod --> ADJ
man --> pobj --> NOUN
, --> punct --> PUNCT

 -->  --> SPACE
but --> cc --> CCONJ
his --> poss --> DET
person --> nsubj --> NOUN
had --> ROOT --> AUX
no --> det --> DET
other --> amod --> ADJ
advantage --> dobj --> NOUN
; --> punct --> PUNCT
and --> cc --> CCONJ
when --> advmod --> ADV
he --> nsubj --> PRON
came --> advcl --> VERB
to --> aux --> PART
be --> auxpass --> AUX

 -->  --> SPACE
contrasted --> conj --> VERB
with --> prep --> ADP
gentlemen --> pobj --> NOUN
, --> punct --> PUNCT
she --> nsubj --> PRON
thought --> ROOT --> VERB
he --> nsubj --> PRON
must --> aux --> VERB
lose --> ccomp --> VERB
all --> predet --> DET
the --> det --> DET
ground --> dobj --> NOUN

 -->  --> SPACE
he -[...]<jupyter_text>It seems like a good start would be to define a pattern with "like a" followed by a noun. So first, we define that pattern. Run/Shift+Enter.<jupyter_code>#define the pattern 
pattern = [{'LOWER': 'like'}, 
           {'LOWER': 'a'}, 
           {'POS': 'NOUN'}]<jupyter_output><empty_output><jupyter_text>Next, we run a function called Matcher over the text that returns all of the substrings that match the pattern. Run/Shift+Enter.<jupyter_code># Matcher class object 
matcher = Matcher(nlp.vocab) 
matcher.add("matching_1", None, pattern) 

matches = matcher(doc) 
for match_id, start, end in matches:
    string_id = nlp.vocab.strings[match_id]  # Get string representation
    span = doc[start:end]  # The matched span
    print(span.text)<jupyter_output>like a look
like a merit
like a gentleman.--
like a job
like a job
like a woman
like a bride
like a brother
like a daughter<jupyter_text>As you would expect, we get a list of substrings that match the pattern we have defined. Let's create a little more ambitious pattern. This time, we want to capture verbs followed by "like a" followed by up to three optional modifiers (adverbs and adjectives) and finally followed by a noun. Run/Shift+Enter.<jupyter_code># Matcher class object
matcher = Matcher(nlp.vocab)

#redefine the pattern
pattern2 = [{'POS':'VERB'},
           {'LOWER': 'like'},
           {'LOWER': 'a'},
           {'DEP':'amod', 'OP':"?"},
           {'DEP':'amod', 'OP':"?"},
           {'DEP':'amod', 'OP':"?"},
           {'POS': 'NOUN'}]

matcher.add("matching_1", None, pattern2)
matches = matcher(doc)

for match_id, start, end in matches:
    string_id = nlp.vocab.strings[match_id]  # Get string representation
    span = doc[start:end]  # The matched span
    print(span.text)<jupyter_output>looked like a sensible young man
argued like a young man
appear like a bride
seemed like a perfect cure
enters like a brother
writes like a sensible man
{'fraction_non_alphanumeric': 0.14664352532799907, 'fraction_numerical': 0.02294522562805201, 'mean_word_length': 4.796248934356351, 'pattern_counts': {'":': 5, '<': 73, '<?xml version=': 0, '>': 163, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '73993', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 11376, 'n_tokens_neox': 10277, 'n_words': 4469, 'path': 'T-M_Basic_Extraction.ipynb', 'repo_name': 'UKDataServiceOpen/TM_R', 'size': '34028'}