<commit_before>---
layout: model
title: Bhojpuri Lemmatizer
author: <NAME>
name: lemma
date: 2021-01-18
tags: [bh, bho, open_source, lemmatizer]
article_header:
  type: cover
use_language_switcher: "Python-Scala-Java"
---

## Description

This model uses context and language knowledge to assign all forms and inflections of a word to a single root. This enables the pipeline to treat the past and present tense of a verb, for example, as the same word instead of two completely different words. The lemmatizer takes into consideration the context surrounding a word to determine which root is correct when the word form alone is ambiguous.

{:.btn-box}
<button class="button button-orange" disabled>Live Demo</button>
[Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Public/3.SparkNLP_Pretrained_Models.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/lemma_bh_2.7.0_2.4_1610989221391.zip){:.button.button-orange.button-orange-trans.arr.button-icon}

## How to use



<div class="tabs-box" markdown="1">
{% include programmingLanguageSelectScalaPythonNLU.html %}
```python
document_assembler = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")

tokenizer = Tokenizer()\
    .setInputCols(["document"]) \
    .setOutputCol("token")

lemmatizer = LemmatizerModel.pretrained("lemma", "bh") \
        .setInputCols(["token"]) \
        .setOutputCol("lemma")
<commit_msg>Added correct live demo and colab links lemma_bh<commit_after>---
layout: model
title: Bhojpuri Lemmatizer
author: <NAME>
name: lemma
date: 2021-01-18
tags: [bh, bho, open_source, lemmatizer]
article_header:
  type: cover
use_language_switcher: "Python-Scala-Java"
---

## Description

This model uses context and language knowledge to assign all forms and inflections of a word to a single root. This enables the pipeline to treat the past and present tense of a verb, for example, as the same word instead of two completely different words. The lemmatizer takes into consideration the context surrounding a word to determine which root is correct when the word form alone is ambiguous.

{:.btn-box}
[Live Demo](https://demo.johnsnowlabs.com/public/TEXT_PREPROCESSING/){:.button.button-orange}
[Open in Colab](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/streamlit_notebooks/TEXT_PREPROCESSING.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/lemma_bh_2.7.0_2.4_1610989221391.zip){:.button.button-orange.button-orange-trans.arr.button-icon}

## How to use



<div class="tabs-box" markdown="1">
{% include programmingLanguageSelectScalaPythonNLU.html %}
```python
document_assembler = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")

tokenizer = Tokenizer()\
    .setInputCols(["document"]) \
    .setOutputCol("token")

lemmatizer = LemmatizerModel.pretrained("lemma", "bh") \
        .setInputCols(["token"]) \
        .setOutputCol("lemma")

{'fraction_non_alphanumeric': 0.12245537112433448, 'fraction_numerical': 0.017851550266207328, 'mean_word_length': 4.970093457943925, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2548591', 'n_tokens_mistral': 1082, 'n_tokens_neox': 1011, 'n_words': 252}