<jupyter_start><jupyter_text>![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/nlu/blob/master/examples/colab/component_examples/text_pre_processing_and_cleaning/NLU_stemmer_example.ipynb) Stemming with NLU Stemming returns the base form, the so called stem / root or base word of every token in the input data.    I. e. 'He was hungry' becomes 'He wa hungri'Stemming works by applying a heuristic process that strips and mutates suffixes on  words. 1. Install Java and NLU<jupyter_code>!wget https://setup.johnsnowlabs.com/nlu/colab.sh -O - | bash
  

import nlu<jupyter_output>--2022-04-15 02:32:33--  https://setup.johnsnowlabs.com/nlu/colab.sh
Resolving setup.johnsnowlabs.com (setup.johnsnowlabs.com)... 172.16.58.3
Connecting to setup.johnsnowlabs.com (setup.johnsnowlabs.com)|172.16.58.3|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/scripts/colab_setup.sh [following]
--2022-04-15 02:32:33--  https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/scripts/colab_setup.sh
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 192.168.127.12, 172.16.31.10, 192.168.3.11, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|192.168.127.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1665 (1.6K) [text/plain]
Saving to: ‘STDOUT’

-                   100%[===================>]   1.63K  --.-KB/s    in 0s      

2022-04-15 02:32:34 (31.6 MB/s) - written to stdout [1665/1665]

Installing  NLU [...]<jupyter_text>2. Load Model and stemm sample string<jupyter_code>import nlu
pipe = nlu.load('en.stem')
pipe.predict('He was suprised by the diversity of NLU')<jupyter_output>sentence_detector_dl download started this may take some time.
Approximate size to download 354.6 KB
[OK!]<jupyter_text>3. Get one row per stemmed token by setting outputlevel to token.    This lets us compare what the original token was and what it was stemmed to to.<jupyter_code>pipe.predict('He was suprised by the diversity of NLU', output_level='token')<jupyter_output><empty_output><jupyter_text>4. Checkout the Stemm models NLU has to offer for other languages than English!<jupyter_code>nlu.print_all_model_kinds_for_action('stem')<jupyter_output>For language <en> NLU provides the following Models : 
nlu.load('en.stem') returns Spark NLP model stemmer<jupyter_text>4.1 Let's try German stemming!<jupyter_code>nlu.load('de.stem').predict("Er war von der Vielfältigkeit des NLU Packets begeistert",output_level='token')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12904398400581607, 'fraction_numerical': 0.05234460196292257, 'mean_word_length': 4.931034482758621, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 20, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 6, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '352844', 'license': 'Apache-2.0', 'n_tokens_mistral': 1089, 'n_tokens_neox': 961, 'n_words': 228, 'path': 'nlu/colab/component_examples/text_pre_processing_and_cleaning/NLU_stemmer_example.ipynb', 'repo_name': 'iamvarol/spark-nlp-workshop', 'size': '2765'}