<jupyter_start><jupyter_text>StemmingStemming is a *text normalization* method used in NLP to simplify text before it is processed by a model. When stemming break the final few characters of a word in order to find a common form of the word. If we take the following sentence:<jupyter_code>txt = "I am amazed by how amazingly amazing you are"<jupyter_output><empty_output><jupyter_text>We use different forms of the word **amaze** a total of *three* times. Each of these different forms is called an *'inflection'*, which is the modification of a word to slightly adjust the meaning or context of the word. When we tokenize this text we produce three different tokens for each inflection of happy, which is okay but in many applications this level of granularity in the semantic meaning of the word is not required and can damage model performance.*Later, when we get to using more complex, sophisticated models (eg BERT), we will use different methods that maintain the inflection of each word - but it is important to understand stemming as it was a very important part of text preprocessing for a very long time, and still relevant to many applications.*To apply stemming we will be using the NLTK package, which provides several different stemmers, we will test the `PorterStemmer` and `LancasterStemmer`.<jupyter_code>words_to_stem = ['happy', 'happiest', 'happier', 'cactus', 'cactii', 'elephant', 'elephants', 'amazed', 'amazing', 'amazingly', 'cement', 'owed', 'maximum']

from nltk.stem import PorterStemmer, LancasterStemmer

porter = PorterStemmer()
lancaster = LancasterStemmer()

stemmed = [(porter.stem(word), lancaster.stem(word)) for word in words_to_stem]

print("Porter | Lancaster")
for stem in stemmed:
    print(f"{stem[0]} | {stem[1]}")<jupyter_output>Porter | Lancaster
happi | happy
happiest | happiest
happier | happy
cactu | cact
cactii | cacti
eleph | eleph
eleph | eleph
amaz | amaz
amaz | amaz
amazingli | amaz
cement | cem
owe | ow
maximum | maxim
{'fraction_non_alphanumeric': 0.07587253414264036, 'fraction_numerical': 0.0010116337885685382, 'mean_word_length': 5.380645161290323, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '260069', 'license': 'MIT', 'n_tokens_mistral': 594, 'n_tokens_neox': 559, 'n_words': 269, 'path': 'course/preprocessing/02_stemming.ipynb', 'repo_name': 'sarvex/transformers-nlp', 'size': '1977'}