<jupyter_start><jupyter_text>Counting words and phrases I've harvested [a gigabyte of OCRd text](https://glam-workbench.github.io/trove-books/ocrd-text-from-trove-books-and-ephemera) from Trove's digitised books and shared it through Cloudstor. Here we'll explore [*Australian Plain Cookery by a Practical Cook*](https://nla.gov.au/nla.obj-579917051) from 1882. However, you could change the `text_file` value below to point to any of the other books on Cloudstor. There's a complete list [in this CSV file](https://github.com/GLAM-Workbench/trove-books/blob/master/trove_digitised_books_with_ocr.csv).<jupyter_code>import requests
from textblob import TextBlob
import re
from collections import Counter
from nltk.corpus import stopwords
from wordcloud import WordCloud
import nltk
nltk.download('stopwords')
nltk.download('punkt')
CLOUDSTOR_URL = 'https://cloudstor.aarnet.edu.au/plus/s/ugiw3gdijSKaoTL'
text_file = 'australian-plain-cookery-by-a-practical-cook-nla.obj-579917051.txt'
stop_words = stopwords.words('english')
# Get the text file from Cloudstor
response = requests.get(f'{CLOUDSTOR_URL}/download?files={text_file}')
text = response.text<jupyter_output><empty_output><jupyter_text>Counting wordsOne way of getting a sense of what a piece of text is about is to look at the frequencies with which words appear. You don't need any special software to do basic word counts. You can just split the text into individual words (called tokens) using a regular expression – in the case below, `\w+` looks for groups of alphanumeric characters, separating words from punctuation and spaces. The you can use `Counter` to find the frequency of each word and `.most_common()` to rank them.<jupyter_code>words = re.findall(r'\w+', text.lower())
Counter(words).most_common(10)<jupyter_output><empty_output><jupyter_text>Try changing the number in the brackets of .most_common(). When you've finished reemember to run the cell again using Shift+Enter! No suprise that the most common words are things like 'the' and 'of'. To try and focus on words that are distinctive of this particular text we can filter our list of words to remove the most common words in English – we call these 'stopwords'. In the cells above we imported a list of stopwords from the NLTK package. Here we'll drop any words that appear in the stopwords and recount.<jupyter_code>filtered_words = [w for w in words if w not in stop_words]
Counter(filtered_words).most_common(10)<jupyter_output><empty_output><jupyter_text>That looks a bit more interesting.  Displaying a word cloudWord clouds are a familiar way of visualising word frequencies in a text. The Wordcloud package makes it easy to generate them. Note that Wordcloud takes care of all the tokenisation, counting, and stopword removal for us – we just give it the text and display the results.<jupyter_code># First we make the wordcloud
# See how we feed the text variable into the generate function?
wc = WordCloud(width=600, height=300).generate(text)

# Display the wordcloud
wc.to_image()<jupyter_output><empty_output><jupyter_text>You might notice that the wordcloud includes some two word phrases (called bigrams) such as 'an hour'. That's the packages default setting, but we can exclude them by setting `collocations=False`.<jupyter_code>wc = WordCloud(width=600, height=300, collocations=False).generate(text)

# Display the wordcloud
wc.to_image()<jupyter_output><empty_output><jupyter_text>Word and n-gram frequenciesTo go further in analysing texts, you'll probably want to use a specialised package. There's NLTK, Spacey, and many others. Here we'll use TextBlob, which is probably the simplest to use and understand.Finding word frequencies with TextBlob is as simple as accessing `blob.word_counts`. However, we probably want to do a bit of filtering to remove some of the noise introduced by OCR. As before, we can also remove common stop words like 'a', 'the', and 'and'.The cell below only keeps words that are longer than two characters (`len(w) > 2`), contain only alphabetical characters (`re.match(r'^[a-zA-Z \-]*$', w)`) and are not in the standard list of stop words.Once we have our list of words/frequencies we'll use `Counter` to get the 10 most common words.<jupyter_code># First we load our text into TextBlob
blob = TextBlob(text)
# The we get the word counts, filter them, and display the most common
words = {w: c for w, c in dict(blob.word_counts).items() if len(w) > 2 and re.match(r'^[a-zA-Z \-]*$', w) and w not in stop_words}
Counter(words).most_common(10)<jupyter_output><empty_output><jupyter_text>If you compare this list with the one above, you'll see they're very similar, but not eaxctly the same. Why? Most likely it's because of differences in the way words are extracted from the text. Above we used a very simple regular expression to identify words, but TextBlob uses a [more sophisticated](https://textblob.readthedocs.io/en/dev/_modules/textblob/tokenizers.html) set of rules. TextBlob will also break our text up into multiple word phrases – *n*-grams, where 'n' represents the number of words. We saw some bigrams in the wordcloud above, but let's find more. To get a list of bigrams we just use `blob.ngrams(2)`. We can then use `Counter` to find the most common.<jupyter_code>blob.ngrams(2)[:10]<jupyter_output><empty_output><jupyter_text>We can then join the separate parts of the ngram together (using `.join`) and use Counter to find the most common phrases.<jupyter_code>Counter([' '.join(l) for l in blob.ngrams(2)]).most_common(10)<jupyter_output><empty_output><jupyter_text>We can try trigrams as well!<jupyter_code>Counter([' '.join(l) for l in blob.ngrams(3)]).most_common(10)<jupyter_output><empty_output><jupyter_text>The most common trigrams seem to relate to quantities – not so surprising in a recipe book! Let's focus in on this by looking at 4-grams (4 word phrases) that start with some of the common trigrams.<jupyter_code>Counter([' '.join(l) for l in blob.ngrams(4) if ' '.join(l).startswith('a pound of')]).most_common(10)
Counter([' '.join(l) for l in blob.ngrams(4) if ' '.join(l).startswith('a pint of')]).most_common(10)
Counter([' '.join(l) for l in blob.ngrams(4) if ' '.join(l).startswith('a tablespoonful of')]).most_common(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09325681492109039, 'fraction_numerical': 0.010521281683405069, 'mean_word_length': 5.555903866248694, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 43, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '341286', 'license': 'MIT', 'n_tokens_mistral': 1971, 'n_tokens_neox': 1858, 'n_words': 774, 'path': 'counting-words-in-text.ipynb', 'repo_name': 'GLAM-Workbench/trove-books', 'size': '6273'}