<jupyter_start><jupyter_text>![Callysto.ca Banner](https://github.com/callysto/curriculum-notebooks/blob/master/callysto-notebook-banner-top.jpg?raw=true)  Most Used English WordsIf you are learning to speak or read English, it is good to know which words are the most used.We will check where words on a word list made by a reading teacher named [<NAME>](https://books.google.ca/books/about/The_Reading_Teacher_s_Book_Of_Lists.html?id=XYidAAAAMAAJ&redir_esc=y) are on a word list from all [Google Books between the years 1800 and 2019](https://books.google.com/ngrams) that [someone made for us](https://github.com/hackerb9/gwordlist).We will explain it using only the [1000 most common words in English](https://www.splasho.com/upgoer5) plus the word [list](https://www.thefreedictionary.com/list). StartingPress the `▶▶` at the top to make this all work.<jupyter_code>import pandas as pd
fry = pd.read_csv('data/fry-sight-words.csv')
ngrams = pd.read_table('https://raw.githubusercontent.com/hackerb9/gwordlist/master/frequency-alpha-gcide.txt', delim_whitespace=True)
words = pd.merge(fry, ngrams, on='WORD', how='left')
words.drop(['COUNT','PERCENT','CUMULATIVE'], axis=1, inplace=True)
words<jupyter_output><empty_output><jupyter_text>We see there are some words, like "workers" that show `NaN`. This means they are not in the Google list.<jupyter_code>words[words['#RANKING'].isnull()]<jupyter_output><empty_output><jupyter_text>Important Parts of WordsThere are 101 words from the Fry list that are not in the Google list. But we can check if the important part of a word is in the Google list.<jupyter_code>important_part = 'worker'

row = ngrams[ngrams['WORD']==important_part].index[0]
print('The word "'+important_part+'" is number',int(ngrams.at[row, '#RANKING']),'in the Google list.')<jupyter_output><empty_output><jupyter_text>So we need to find the important parts of those words and see if they are in the Google list.<jupyter_code>import nltk
from nltk.stem.porter import PorterStemmer
important_part_finder = PorterStemmer()

important_parts = []
for row in words.fillna(value='fix').itertuples():
    if row._3 == 'fix':
        important_parts.append(important_part_finder.stem(row.WORD))
    else:
        important_parts.append(row.WORD)
fry['ORIGINAL_WORD'] = fry['WORD']
fry['WORD'] = important_parts
words = pd.merge(fry, ngrams, on='WORD', how='left')
words.drop(['COUNT','PERCENT','CUMULATIVE'], axis=1, inplace=True)
print('Now there are',len(words[words['#RANKING'].isnull()]),'words not found.')<jupyter_output><empty_output><jupyter_text>We can look at the words that are not found.<jupyter_code>words[words['#RANKING'].isnull()]<jupyter_output><empty_output><jupyter_text>Taking Out WordsOf those 57 words, most are [contractions](https://www.thefreedictionary.com/contraction) (like `we'll`), [nationalities](https://www.thefreedictionary.com/nationality) (like `Greek`), or places (like `Washington`). We could go through and change them, but instead we will just take them out.<jupyter_code>words.dropna(inplace=True)
words<jupyter_output><empty_output><jupyter_text>Coloured LinesNow we can see how well the `RANKING` order from Google matches up with the `LIST` order from Fry.<jupyter_code>from plotly.subplots import make_subplots
import plotly.graph_objects as go
fig = make_subplots(rows=10, cols=1)
for fry_list in words['LIST'].unique():
    df = words[words['LIST']==fry_list]
    fig.append_trace(go.Scatter(x=df.index,y=df['#RANKING'],name=str(fry_list),hovertext=df['WORD']),row=int(fry_list/100),col=1)
fig.update_layout(height=1200,legend_title_text='Fry List',title_text='Google List and Fry List Match (shorter is better match)')
fig.update_yaxes(title='Google').update_xaxes(title='Fry', row=10)
fig.show()<jupyter_output><empty_output><jupyter_text>We see that there are some words like `spell` and `numeral` that are used often in the Fry list but not so much in the Google list. This may be because the Fry list is from books for Grades 3 to 9, and the Google list is from as many books as they could scan.It is possible to see the words that are not a good match between lists because they are later in the Google list.<jupyter_code>google_max = 3000
words[words['#RANKING']>google_max]<jupyter_output><empty_output><jupyter_text>We can make the coloured lines again without those words.<jupyter_code>google_max = 3000
fig = make_subplots(rows=10, cols=1)
for fry_list in words['LIST'].unique():
    df = words[(words['LIST']==fry_list) & (words['#RANKING']<google_max)]
    fig.append_trace(go.Scatter(x=df.index,y=df['#RANKING'],name=str(fry_list),hovertext=df['WORD']),row=int(fry_list/100),col=1)
fig.update_layout(height=1200,legend_title_text='Fry List',title_text='Google List (under '+str(google_max)+') and Fry List Match')
fig.update_yaxes(title='Google').update_xaxes(title='Fry', row=10)
fig.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1333197223356472, 'fraction_numerical': 0.012249897917517355, 'mean_word_length': 6.965853658536585, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 9, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '460281', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 1734, 'n_tokens_neox': 1643, 'n_words': 425, 'path': 'Languages/MostUsedEnglishWords/most-used-english-words.ipynb', 'repo_name': 'callysto/curriculum-notebooks', 'size': '4902'}