<jupyter_start><jupyter_text>Implementing a Spell Checker after transcriptions Testing multiple functions to count spelling mistakes in UGC- For now we decided not to use this. The google transcriptions do not give an accurate enough representation of the children's writing to get an accurate count of spelling errrs. Seems to add in extra errors for bad handwriting<jupyter_code># Imports
import pandas as pd
import pkg_resources
from symspellpy import SymSpell, Verbosity
from spellchecker import SpellChecker
import string
# Load transribed stories df
df = pd.read_csv('transcribed_stories.csv')
df.head()
# Check spell check for the first entry
input_term = df['Transcribed Text'][0]<jupyter_output><empty_output><jupyter_text>Symspell package- Slow<jupyter_code># Load dictionary
sym_spell = SymSpell(max_dictionary_edit_distance=6, prefix_length=7, count_threshold= 15)
dictionary_path = pkg_resources.resource_filename(
    "symspellpy", "frequency_dictionary_en_82_765.txt")
bigram_path = pkg_resources.resource_filename(
    "symspellpy", "frequency_bigramdictionary_en_243_342.txt")

sym_spell.load_dictionary(dictionary_path, term_index=0, count_index=1)
sym_spell.load_bigram_dictionary(bigram_path, term_index=0, count_index=2)
# Run transcribed text through Symspell to get spelling suggestions back
suggestions = sym_spell.lookup_compound(input_term, max_edit_distance=6, ignore_term_with_digits= True, ignore_non_words= True)

suggestions
for suggestion in suggestions:
    print('------')
    print(suggestion)<jupyter_output>------
page i 3132 once there was a little cheetah and the cheetah had a best friend lion the cheetahs name was paws and the lions name was dylan they always played with each other after they went hunting dylan went to play with paws after hunting with pack and paws wont to play after hunting with his mon they always met up near the same rock art the same lake they would talk and have water fights here is what the talked about your parents and my pack might have a fight said dylan we both don't want that to happen we might have to fight against each other said paws after a few minutes it was time for them bale to their family next morning they saw there family having a fight dylan family won and plus mon was here at their us all he meeting time they talked about it my mon got here its not fair said paws i think the best way to fix this is to tell are family we are friend said dylan i think your right said paws so after there talk time and water fight they went back to there family and [...]<jupyter_text>Python Spellchecker<jupyter_code>def spell_check(text):
    # Initialize spellchecker
    spell = SpellChecker()

    # Strip the punctuation
    exclude = set(string.punctuation)
    text = ''.join(ch for ch in text if ch not in exclude)

    # Split text into list of words
    text = text.split()

    # Find misspelled words
    misspelled = spell.unknown(text)

    # Get corrections
    for word in misspelled:
        # Get the most likely correct spelling
        print(f'{word}: {spell.correction(word)}')
input_term = df['Transcribed Text'][4]
df['Transcribed Text'][4]
# Return identified typo and the suggested correction
spell_check(input_term)<jupyter_output>kayla: layla
recimals: decimals
hotice: notice
withiber: wither
goining: joining
3rd: ord
ezzzzz: ezzzzz
arobot: robot
suid: said
shes: she
soom: room
godess: goddess
asock: sock
mtlava: lava
wampice: vampire
ster: step
kindergarden: kindergarten
anella: nella
writting: writing
oht: out
cof: of
beltness: boldness
didnt: dint
begula: betula
elementry: elementary
vocab: vocal
stadies: studies
mrsklapen: mrsklapen
begular: regular
plete: plate
emania: mania
{'fraction_non_alphanumeric': 0.058743909041689225, 'fraction_numerical': 0.007579859231185706, 'mean_word_length': 5.178929765886288, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '246687', 'license': 'MIT', 'n_tokens_mistral': 1120, 'n_tokens_neox': 1086, 'n_words': 460, 'path': 'notebooks/count_spelling_errors.ipynb', 'repo_name': 'stimsonc/scribble-stadium-ds', 'size': '3694'}