<jupyter_start><jupyter_text>![UKDS Logo](images/UKDS_Logos_Col_Grey_300dpi.png)  Text-mining: Basics in R Welcome to the UK Data Service training series on New Forms of Data for Social Science Research. This series guides you through some of the most common and valuable new sources of data available for social science research: data collected from websites, social media platorms, text data, conducting simulations (agent based modelling), to name a few. We provide webinars, interactive notebooks containing live programming code, reading lists and more.To access training materials for the entire series: [Training Materials]To keep up to date with upcoming and past training events: [Events]To get in contact with feedback, ideas or to seek assistance: [Help]Dr <NAME>UK Data ServiceUniversity of ManchesterNovember 2020 Table of Contents1  Introduction2  Retrieval3  Processing3.1  Tokenisation3.2  Standardising3.2.1  Remove uppercase letters3.2.2  Spelling correction3.2.3  RegEx replacements3.3  Removing irrelevancies3.3.1  Remove punctuation3.3.2  Stopwords3.4  Consolidation3.4.1  Stemming words3.4.2  Lemmatisation4  Conclusions4.1  Further reading There is a table of contents provided here at the top of the notebook, but you can also access this menu at any point by clicking the Table of Contents button on the top toolbar (an icon with four horizontal bars, if unsure hover your mouse over the buttons).   Introduction This is the first in a series of jupyter notebooks on text-mining that cover basic preparation processes, common natural language processing tasks, and some more advanced natural language tasks. These interactive code-along notebooks use python as a programming language, but introduce various packages related to text-mining and text processing. Most of those tasks could be done in other packages, so please be aware that the options demonstrated here are not the only way, or even the best way, to accomplish a text-mining task. For more information on what jupyter notebooks are or how to interact with them, follow [THESE LINKS] (insert links please).   Retrieval The first step in text-mining, or any form of data-mining, is retrieving a data set to work with. Within text-mining, or any language analysis context, one data set is usually referred to as 'a corpus' while multiple data sets are referred to as 'corpora'. 'Corpus' is a latin-root word and therefore has a funny plural. For text-mining, a corpus can be:- a set of tweets, - the full text of an 18th centrury novel,- the contents of a page in the dictionary, - minutes of local council meetings, - random gibberish letters and numbers, or- just about anything else in text format. Retrieval is a very important step, but it is not the focus of this particular training series. If you are interested in creating a corpus from internet data, then you may want to check out previous the NFoD training series that covers Web-scraping (available as recordings of webinars or as a code-along jupyter notebook like this one) and API's (also as recording or jupyter notebook). Both of these demonstrate and discuss ways to get data from the internet that you could use to build a corpus. Instead, for the purposes of this session, we will assume that you already have a corpus to analyse. This is easy for us to assume, because we have provided a sample text file that we can use as a corpus for these exercises. First, let's check that it is there. To do that, click in the code cell below and hit the 'Run' button at the top of this page or by holding down the 'Shift' key and hitting the 'Enter' key. For the rest of this notebook, I will use 'Run/Shift+Enter' as short hand for 'click in the code cell below and hit the 'Run' button at the top of this page or by hold down the 'Shift' key while hitting the 'Enter' key'.<jupyter_code># It is good practice to always start by importing the modules and packages you will need. 

library(dplyr)
library(readr)
library(stringr)
library(coreNLP)
initCoreNLP()

print(list.files(path = "./data", pattern = "*"))      # create list of all .txt files in the data folder<jupyter_output><empty_output><jupyter_text>Great! We have imported some useful libraries from the tidyverse package.  We have also had a look in the ./data folder and found a file that we can use for practicing some text-mining. Now we need to load that sample_text file into a variable and have a look at it. Time to Run/Shift+Enter again!<jupyter_code># Open the "sample_text" file and read (import) its contents to a variable called "corpus"

corpus <- read_file("./data/sample_text.txt") 
print(corpus)<jupyter_output><empty_output><jupyter_text>Hmm. Not excellent literature, but it will do for our purposes. A quick look tells us that there are capital letters, contractions, punctuation, numbers as digits, numbers written out, abbreviations, and other things that, as humans, we know are equivalent but that computers do not know about. Before we go further, it helps to know what kind of variable corpus is. Run/Shift+Enter the next code block to find out!S<jupyter_code>typeof(corpus)<jupyter_output><empty_output><jupyter_text>This tells us that 'corpus' is a string or 'character' variable, which is a good start for text-mining. Congratulations! We are done with the retreival portion of this process. The rest won't be quite so straightforward because next up... Processing. Processing is about cleaning, correcting, standardizing and formatting the raw data returned from the retrieval process.   Processing The string we have as our corpus is a good starting point, but it is not perfect. It has a bunch of errors and punctuation which need to be corrected. But even worse, it is 'one long thing' when statistical analysis typically requires 'lots of short things'.So, clearly, we have a few steps to go through with our raw text.* Tokenisation, (or splitting text into various kinds of 'short things' that can be statistically analysed).* Standardising the next (including converting uppercase to lower, correcting spelling, find-and-replace operations to remove abbreviations, etc.).   * Removing irrelevancies (anything from punctuation to stopwords like 'the' or 'to' that are unhelpful for many kinds of analysis).   * Consolidating (including stemming and lemmatisation that strip words back to their 'root').* Basic NLP (that put some of the small things back together into logically useful medium things, like multi-word noun or verb phrases and proper names).In practice, most text-mining work will require that any given corpus undergo multiple steps, but the exact steps and the exact order of steps depends on the desired analysis to be done. Thus, some of the examples that follow will use the raw text corpus as an input to the process while others use a processed corpus as an input.As a side note, it is good practice to create new variables whenever you manipulate an existing variable rather than write over the original. This means that you keep the original and can go back to it anytime you need to if you want to try a different manipulation or correct an error. You will see how this works as we progress through the processing steps.  Tokenisation Our first step is to cut our 'one big thing' into tokens, or 'lots of little things'. As an example, one project I worked involved downloading a file with hundreds of recorded chess games, which I then divided into individual text files with one game each. The games had a very standard format, with every game ending with either '1-0', '0-1' or '1/2-1/2'. Thus, I was able to use regular expressions (covered in more detail later) to iterate over the file, selecting everyithing until it found an instance of '1-0', '0-1' or '1/2-1/2', at which point it would cut what it had selected, write it to a blank file, save it, and start iterating over the original file again.Other options that might make more sense with other kinds of files would be to to cut and write from the large file to new files after a specified number of lines or characters.Whether you have one big file or many smaller ones, most text-mining work will also want to divide the corpus into what are known as 'tokens'. These 'tokens' are the unit of analysis, which might be chapters, sections, paragraphs, sentences, words, or something else.Since we have one file already loaded as a corpus, we can skip the right to tokenising that text into sentences and words. Both options are functions available through the ntlk package that we imported earlier. These are both useful tokens in their own way, so we will see how to produce both kinds.We start by dividing our corpus into words, splitting the string into substrings whenever 'word_tokenize' detects a word.Let's try that. But this time, let's just have a look at the first 100 things it finds instead of the entire text. Run/Shift+Enter.S<jupyter_code># code to tokenise into sentences and/or words<jupyter_output><empty_output><jupyter_text>Let's have a look.We can see that corpus_words is a list of strings. We know it is a list because it starts and ends with square brackets and we know the things in that list are strings because they are surrounded by single quotes.We can also see that puctuation marks are counted as tokens in that list. For example, the full stop at the end of the first sentence appears as its own token because word_tokenize knows that it does not count as part of the previous word. Interestingly, 'U.K.' is all one token, despite having full stops in. Clever stuff, this tokenisation function!Word_tokenize is a useful function if you want to take a 'bag of words' approach to text-mining. This reduces a lot of the contextual information within the original corpus because it ignores how the words were used or in what order they originally appeared, making it easy to count how often each word occurrs. There is a surprising amount of insight to be gained here, but it does mean that 'building' in the next two sentences will be counted as the "same" word."He is building a diorama for a school project." where 'building' is a verb"The building is a clear example of brutalist architecture." where 'building' is a nounThere are other kinds of analyses that you could do if you want verb-building and noun-building to be counted as different words. That usually starts with tokenising differently, for example into sentences rather than words. Let's see what that looks like by running the same basic analysis again, but this time with sentence-token things instead of word-token things.Do that funky Run/Shift+Enter thing!  Remove uppercase letters Remove uppercase lettersIf we want to focus on the 'bag of words' approach, we don't really care about uppercase or lowercase distinctions. For example, we want 'Privacy' to count as the same word as 'privacy', rather than as two different words.We can remove all uppercase letters with a built in python command on corpus_words. Do this in the next code cell, again returning just the first 100 items instead of the whole thing.Do the Run/Shift+Enter thing.<jupyter_code>lower_string <- tolower(my_string)<jupyter_output><empty_output><jupyter_text>Corpus_sentences is also a list of strings (starts and ends with square brackets, each item is surrounded by single quotes).This time, the full stops at the end of each sentence are included within the sentence token, which makes sense.Moving forward, some of the next steps make more sense to do on the word-tokens while others on sentence-tokens.  Spelling correction Everybody loves spelling... RIGHT?!?Fortunately, there are several decent spellchecking packages written for python. They are not automatically installed and ready to import in the same way that the 'os' or 'nltk' packages were, but we just need to install the packages and import the functions we need through an installer called 'pip'. You will see 'pip' in the next code block, but since this is in jupyter notebook rather than directly in a python shell, we need to put a '!' in front of the 'pip' function. Don't worry too much about that now, I just mention it here in case you find it interesting to know.The next code cell:installs the 'autocorrect' package,imports the Speller function, andcreates a one-word command that specifies that the Speller function should use English language.Run/Shift+Enter, as per usual.<jupyter_code># code to correct spelling<jupyter_output><empty_output><jupyter_text>Super. Creating that one-word command saves us some time, which is maybe less important here but is a good skill to be aware of if you are working on text-mining every day for weeks on end. Always be on the look out for good ways to save time.Moving on, we need to iterate over our corpus, checking and correcting each token. This is easy to do if you start with a new, empty list (I called mine 'corpus_correct_spell'). As I work through corpus_words, one token at a time, we append (which is just fancy for 'add to the end') the corrected word to our new blank list.Then, as usual, we have a quick look at the first 100 entries in the new 'corpus_correct_spell'.Run/Shift+Enter. You know how to do it. Don't worry if it takes a while... Checking the spelling on each word is not a cakewalk.<jupyter_code># more code<jupyter_output><empty_output><jupyter_text>How did it do? Well, this spell-checker replaced 'haz' with 'had' rather than 'has'. That is ok, I guess. No automatic spelling correction programme will get it 100% right 100% of the time. Maybe your project has specific research questions that won't work with this decision.In that case, you would have to check out some other spell-checkers like textblob or pyspellchecker. You might even want to custom build or adapt your own spell-checker, especially if you were working with very non-standard text, like comment boards that use a bunch of slang, common typos, or specific terms.But take a moment here and consider the following questions...Can you apply this spell-checker to corpus_sentences rather than corpus_words? If you are not sure what happens, try it out by copying, editing and re-running the above code block.Should you have appled this spell-checker to corpus_lower rather than corpus_words? What difference would it make? Again, try it out if you are not sure.Next up, specific replacements with RegEx!S  RegEx replacements RegEx stands for REGular EXpressions, which is probably familiar to you as the basis for how find-and-replace works in text documents. I mentioned this above when I talked about cutting up a large file into smaller files whenever the computer iterating over the large file found one of three specific combinations of numbers and symbols.But RegEx is actually stronger than that because you can use it to identify combinations of letters, numbers, symbols, spaces and more, some of which can be repeated more than once or can be optional. I won't go into RegEx too much more here, because that is a whole set of lessons on its own. But here are a couple of examples that you might find useful in a text like ours where we know that there are mixtures of numbers written as numbers, numbers spelled out, geographic abbreviations and more.As you might expect, do the Run/Shift+Enter thing.<jupyter_code># regex code<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.04401569653368215, 'fraction_numerical': 0.004905166775670372, 'mean_word_length': 4.692851824274014, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '211262', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3801, 'n_tokens_neox': 3623, 'n_words': 2408, 'path': 'T-M_Basic_Processing_R.ipynb', 'repo_name': 'UKDataServiceOpen/TM_R', 'size': '15297'}