<jupyter_start><jupyter_text>![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/nlu/blob/master/examples/colab/component_examples/text_pre_processing_and_cleaning/NLU_lemmatization.ipynb) Lemmatization with NLU Lemmatizing returns the base form, the so called lemma of every token in the input data.    I. e. 'He was hungry' becomes 'He be hungry'The Lemmatizer works by operating on a dictionary and taking context into account. This lets the Lemmatizer dervie a different base word for for a word in two different contexts which depends on the Part of Speech tags. This is the main difference  to Stemming, which solves the same problem by applying a heuristic process that removes the end of words. 1. Install Java and NLU<jupyter_code>!wget https://setup.johnsnowlabs.com/nlu/colab.sh -O - | bash
  

import nlu<jupyter_output>--2021-05-01 23:18:27--  https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/scripts/colab_setup.sh
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 192.168.3.11, 172.16.58.3, 192.168.3.11, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|192.168.3.11|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1671 (1.6K) [text/plain]
Saving to: ‘STDOUT’


-                     0%[                    ]       0  --.-KB/s               
-                   100%[===================>]   1.63K  --.-KB/s    in 0s      

2021-05-01 23:18:27 (35.6 MB/s) - written to stdout [1671/1671]

Installing  NLU 3.0.0 with  PySpark 3.0.2 and Spark NLP 3.0.1 for Google Colab ...
[K     |████████████████████████████████| 204.8MB 73kB/s 
[K     |████████████████████████████████| 153kB 50.6MB/s 
[K     |████████████████████████████████| 204kB 20.4MB/s 
[K     |████████████████████████████████| 204kB 52.5MB/s 
[?25h  Building wheel[...]<jupyter_text>2. Load Model and lemmatize sample string<jupyter_code>import nlu
pipe = nlu.load('en.lemma')
pipe.predict('He was suprised by the diversity of NLU')<jupyter_output>lemma_antbnc download started this may take some time.
Approximate size to download 907.6 KB
[OK!]
sentence_detector_dl download started this may take some time.
Approximate size to download 354.6 KB
[OK!]<jupyter_text>3. Get one row per lemmatized token by setting outputlevel to token.    This lets us compare what the original token was and what it was lemmatized to to.<jupyter_code>pipe.predict('He was suprised by the diversity of NLU', output_level='token')<jupyter_output><empty_output><jupyter_text>4. Checkout the Lemma models NLU has to offer for other languages than English!<jupyter_code>nlu.print_all_model_kinds_for_action('lemma')<jupyter_output>For language <nl> NLU provides the following Models : 
nlu.load('nl.lemma') returns Spark NLP model lemma
For language <en> NLU provides the following Models : 
nlu.load('en.lemma') returns Spark NLP model lemma_antbnc
nlu.load('en.lemma.antbnc') returns Spark NLP model lemma_antbnc
For language <fr> NLU provides the following Models : 
nlu.load('fr.lemma') returns Spark NLP model lemma
For language <de> NLU provides the following Models : 
nlu.load('de.lemma') returns Spark NLP model lemma
For language <it> NLU provides the following Models : 
nlu.load('it.lemma') returns Spark NLP model lemma_dxc
nlu.load('it.lemma.dxc') returns Spark NLP model lemma_dxc
For language <nb> NLU provides the following Models : 
nlu.load('nb.lemma') returns Spark NLP model lemma
For language <pl> NLU provides the following Models : 
nlu.load('pl.lemma') returns Spark NLP model lemma
For language <pt> NLU provides the following Models : 
nlu.load('pt.lemma') returns Spark NLP model lemma
For language <ru>[...]<jupyter_text>4.1 Let's try German lematization!<jupyter_code>nlu.load('de.lemma').predict("Er war von der Vielfältigkeit des NLU Packets begeistert",output_level='token')<jupyter_output>lemma download started this may take some time.
Approximate size to download 4 MB
[OK!]
sentence_detector_dl download started this may take some time.
Approximate size to download 354.6 KB
[OK!]
{'fraction_non_alphanumeric': 0.13434969032872796, 'fraction_numerical': 0.035493091948546926, 'mean_word_length': 4.404118404118404, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 27, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '524336', 'license': 'Apache-2.0', 'n_tokens_mistral': 1622, 'n_tokens_neox': 1391, 'n_words': 433, 'path': 'nlu/colab/component_examples/text_pre_processing_and_cleaning/NLU_lemmatization.ipynb', 'repo_name': 'fcivardi/spark-nlp-workshop', 'size': '4211'}