<jupyter_start><jupyter_text>Text NormalizationVocabulary large.A lot of repeated characters so effiency matters.Langauge is ambiguous so system can get confused.Tries to remove redundant or unnecessary information.  Remove Tags"This is the best restaurant ever!""This is the **best** restaurant ever!"> BeautifulSoup/lxml> Regular ExpressionsRegex cannot detect and remove all HTML because HTML is not a regular language.   Accented Characters.> Niño Brown vs Nino Brown> Niño -> Nino ?> Preserve unicode? Turn it into ASCII equivalent?> \Sum vs E  Expand Contractions> I'm going home -> I am going home> I [ain't] [gonna] do it -> I [am not] [going to] do it  Special Characters> I am sooo Happy 😁!! -> I am so happy !  Stemming vs Lemmatization Stem- Remove prefixes and suffixes from words and keeping only the base form.- base form of the word does not have to be a vocabulary word.> running -> run> thowing -> throw> carpet -> carp    <<> [dis]continue -> continue> [a]sham[ed] -> sham> begin[ing] -> beginPorter Stemmer Snowball Stemmer- Fast- Simple- Pretty accurate- May have tragic mistakes Lemmatization- Converts a word to its root form based on its context and part of speech.- requires the part of speech and a dictionary.- Extremely accurate- Orders of magnitude slower than Stemming- Requires extra information (dictionary and p-o-s)  Removing Stop Words- Useless words> 'a', 'the', 'as'> URLS- functional words to help sentence structure but do not help with meaning  Multilingual cleaningCan you detect the language?Options:1. Words can be translated into a common language2. Remove non-"english" text <- english centric appraoch could be problematic3. Use Universal ModelHandling multple language requires care.Other language have stopwords and other langauge specific features that must be managed.<jupyter_code>import spacy
import spacy.lang
print(spacy.lang.en.stop_words.STOP_WORDS)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07235142118863049, 'fraction_numerical': 0.0015503875968992248, 'mean_word_length': 4.813813813813814, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '99553', 'license': 'MIT', 'n_tokens_mistral': 527, 'n_tokens_neox': 501, 'n_words': 261, 'path': 'notebooks/lecture15.ipynb', 'repo_name': 'cegme/cs5293sp22', 'size': '1935'}