<jupyter_start><jupyter_text>![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/nlu/blob/master/examples/colab/component_examples/text_pre_processing_and_cleaning/NLU_stopwords_removal_example.ipynb) Stopwords removal with NLU Stopwords refer to the most common words in a language.       I. e.  'the, is, at, which,on' are stopwords which will be removed. 1. Install Java and NLU<jupyter_code>!wget https://setup.johnsnowlabs.com/nlu/colab.sh -O - | bash
  

import nlu<jupyter_output>--2021-05-01 23:22:51--  https://raw.githubusercontent.com/JohnSnowLabs/nlu/master/scripts/colab_setup.sh
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 192.168.127.12, 172.16.58.3, 172.16.31.10, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|192.168.127.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1671 (1.6K) [text/plain]
Saving to: ‘STDOUT’


-                     0%[                    ]       0  --.-KB/s               
-                   100%[===================>]   1.63K  --.-KB/s    in 0s      

2021-05-01 23:22:52 (33.4 MB/s) - written to stdout [1671/1671]

Installing  NLU 3.0.0 with  PySpark 3.0.2 and Spark NLP 3.0.1 for Google Colab ...
[K     |████████████████████████████████| 204.8MB 76kB/s 
[K     |████████████████████████████████| 153kB 43.2MB/s 
[K     |████████████████████████████████| 204kB 17.9MB/s 
[K     |████████████████████████████████| 204kB 43.4MB/s 
[?25h  Building wheel[...]<jupyter_text>2. Load Model and remove stopwords from sample string<jupyter_code>import nlu
pipe = nlu.load('stopwords')
pipe.predict('He was suprised by the diversity of NLU')<jupyter_output>stopwords_en download started this may take some time.
Approximate size to download 2.9 KB
[OK!]
sentence_detector_dl download started this may take some time.
Approximate size to download 354.6 KB
[OK!]<jupyter_text>4. Checkout the stopword models NLU has to offer for other languages than English!<jupyter_code>nlu.print_all_model_kinds_for_action('stopwords')<jupyter_output>For language <en> NLU provides the following Models : 
nlu.load('en.stopwords') returns Spark NLP model stopwords_en
For language <fr> NLU provides the following Models : 
nlu.load('fr.stopwords') returns Spark NLP model stopwords_fr
For language <de> NLU provides the following Models : 
nlu.load('de.stopwords') returns Spark NLP model stopwords_de
For language <it> NLU provides the following Models : 
nlu.load('it.stopwords') returns Spark NLP model stopwords_it
For language <pl> NLU provides the following Models : 
nlu.load('pl.stopwords') returns Spark NLP model stopwords_pl
For language <pt> NLU provides the following Models : 
nlu.load('pt.stopwords') returns Spark NLP model stopwords_pt
For language <ru> NLU provides the following Models : 
nlu.load('ru.stopwords') returns Spark NLP model stopwords_ru
For language <af> NLU provides the following Models : 
nlu.load('af.stopwords') returns Spark NLP model stopwords_af
For language <ar> NLU provides the following Models : 
nlu.load([...]<jupyter_text>4.1 Let's try German stopword removal!<jupyter_code>nlu.load('stopwords').predict("Er war von der Vielfältigkeit des NLU Packets begeistert",output_level='token')<jupyter_output>stopwords_en download started this may take some time.
Approximate size to download 2.9 KB
[OK!]
sentence_detector_dl download started this may take some time.
Approximate size to download 354.6 KB
[OK!]
{'fraction_non_alphanumeric': 0.14722222222222223, 'fraction_numerical': 0.041944444444444444, 'mean_word_length': 4.39880059970015, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 23, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '428475', 'license': 'Apache-2.0', 'n_tokens_mistral': 1447, 'n_tokens_neox': 1232, 'n_words': 333, 'path': 'nlu/colab/component_examples/text_pre_processing_and_cleaning/NLU_stopwords_removal_example.ipynb', 'repo_name': 'fcivardi/spark-nlp-workshop', 'size': '3609'}