<commit_before>---
layout: model
title: Polish Lemmatizer
author: <NAME>
name: lemma
date: 2020-05-03 18:18:00 +0800
tags: [lemmatizer, pl]
article_header:
  type: cover
use_language_switcher: "Python-Scala-Java"
---

{:.h2_title}
## Description
This model uses context and language knowledge to assign all forms and inflections of a word to a single root. This enables the pipeline to treat the past and present tense of a verb, for example, as the same word instead of two completely different words. The lemmatizer takes into consideration the context surrounding a word to determine which root is correct when the word form alone is ambiguous.

{:.btn-box}
<button class="button button-orange" disabled>Live Demo</button>
[Open in Colab](https://githubtocolab.com/JohnSnowLabs/spark-nlp-workshop/blob/b2eb08610dd49d5b15077cc499a94b4ec1e8b861/jupyter/annotation/english/model-downloader/Create%20custom%20pipeline%20-%20NerDL.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/lemma_pl_2.5.0_2.4_1588518491035.zip){:.button.button-orange.button-orange-trans.arr.button-icon}

{:.h2_title}
## How to use

<div class="tabs-box" markdown="1">

{% include programmingLanguageSelectScalaPython.html %}

```python
...
lemmatizer = LemmatizerModel.pretrained("lemma", "pl") \
        .setInputCols(["token"]) \
        .setOutputCol("lemma")
nlp_pipeline = Pipeline(stages=[document_assembler, tokenizer, lemmatizer])
light_pipeline = LightPipeline(nlp_pipeline.fit(spark.createDataFrame([['']]).toDF("text")))
results = light_pipeline.fullAnnotate("Oprócz bycia królem północy, <NAME> jest angielskim lekarzem i liderem w rozwoju anestezjologii i higieny medycznej.")
```

```scala
...
val lemmatizer = LemmatizerModel.pretrained("lemma", "pl")
        .setInputCols(Array("token"))
        .setOutputCol("lemma")
val pipeline = new Pipeline().setStages(Array(document_assembler, tokenizer, lemmatizer))
val result = pipeline.fit(Seq.empty["Oprócz bycia królem północy, <NAME> jest angielskim lekarzem i liderem w rozwoju anestezjologii i higieny medycznej."].toDS.toDF("text")).transform(data)
```
</div>

{:.h2_title}
## Results

```bash
[Row(annotatorType='token', begin=0, end=5, result='oprócz', metadata={'sentence': '0'}, embeddings=[]),
Row(annotatorType='token', begin=7, end=11, result='bycia', metadata={'sentence': '0'}, embeddings=[]),
Row(annotatorType='token', begin=13, end=18, result='król', metadata={'sentence': '0'}, embeddings=[]),
Row(annotatorType='token', begin=20, end=26, result='północ', metadata={'sentence': '0'}, embeddings=[]),
Row(annotatorType='token', begin=27, end=27, result=',', metadata={'sentence': '0'}, embeddings=[]),
...]
```

{:.model-param}
## Model Information

{:.table-model}
|---|---|
|Model Name:|lemma|
|Type:|lemmatizer|
|Compatibility:|Spark NLP 2.5.0+|
|Edition:|Official|
|Input labels:|[token]|
|Output labels:|[lemma]|
|Language:|pl|
|Case sensitive:|false|
|License:|Open Source|

{:.h2_title}
## Data Source
The model is imported from [https://universaldependencies.org](https://universaldependencies.org)<commit_msg>[skip_ci] new metadata lines added to 31.file 2020-05-03-lemma_pl.md<commit_after>---
layout: model
title: Polish Lemmatizer
author: <NAME>
name: lemma
date: 2020-05-03 18:18:00 +0800
task: Lemmatization
language: pl
edition: Spark NLP 2.5.0
tags: [lemmatizer, pl]
article_header:
  type: cover
use_language_switcher: "Python-Scala-Java"
---

{:.h2_title}
## Description
This model uses context and language knowledge to assign all forms and inflections of a word to a single root. This enables the pipeline to treat the past and present tense of a verb, for example, as the same word instead of two completely different words. The lemmatizer takes into consideration the context surrounding a word to determine which root is correct when the word form alone is ambiguous.

{:.btn-box}
<button class="button button-orange" disabled>Live Demo</button>
[Open in Colab](https://githubtocolab.com/JohnSnowLabs/spark-nlp-workshop/blob/b2eb08610dd49d5b15077cc499a94b4ec1e8b861/jupyter/annotation/english/model-downloader/Create%20custom%20pipeline%20-%20NerDL.ipynb){:.button.button-orange.button-orange-trans.co.button-icon}
[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/lemma_pl_2.5.0_2.4_1588518491035.zip){:.button.button-orange.button-orange-trans.arr.button-icon}

{:.h2_title}
## How to use

<div class="tabs-box" markdown="1">

{% include programmingLanguageSelectScalaPython.html %}

```python
...
lemmatizer = LemmatizerModel.pretrained("lemma", "pl") \
        .setInputCols(["token"]) \
        .setOutputCol("lemma")
nlp_pipeline = Pipeline(stages=[document_assembler, tokenizer, lemmatizer])
light_pipeline = LightPipeline(nlp_pipeline.fit(spark.createDataFrame([['']]).toDF("text")))
results = light_pipeline.fullAnnotate("Oprócz bycia królem północy, <NAME> jest angielskim lekarzem i liderem w rozwoju anestezjologii i higieny medycznej.")
```

```scala
...
val lemmatizer = LemmatizerModel.pretrained("lemma", "pl")
        .setInputCols(Array("token"))
        .setOutputCol("lemma")
val pipeline = new Pipeline().setStages(Array(document_assembler, tokenizer, lemmatizer))
val result = pipeline.fit(Seq.empty["Oprócz bycia królem północy, <NAME> jest angielskim lekarzem i liderem w rozwoju anestezjologii i higieny medycznej."].toDS.toDF("text")).transform(data)
```
</div>

{:.h2_title}
## Results

```bash
[Row(annotatorType='token', begin=0, end=5, result='oprócz', metadata={'sentence': '0'}, embeddings=[]),
Row(annotatorType='token', begin=7, end=11, result='bycia', metadata={'sentence': '0'}, embeddings=[]),
Row(annotatorType='token', begin=13, end=18, result='król', metadata={'sentence': '0'}, embeddings=[]),
Row(annotatorType='token', begin=20, end=26, result='północ', metadata={'sentence': '0'}, embeddings=[]),
Row(annotatorType='token', begin=27, end=27, result=',', metadata={'sentence': '0'}, embeddings=[]),
...]
```

{:.model-param}
## Model Information

{:.table-model}
|---|---|
|Model Name:|lemma|
|Type:|lemmatizer|
|Compatibility:|Spark NLP 2.5.0+|
|Edition:|Official|
|Input labels:|[token]|
|Output labels:|[lemma]|
|Language:|pl|
|Case sensitive:|false|
|License:|Open Source|

{:.h2_title}
## Data Source
The model is imported from [https://universaldependencies.org](https://universaldependencies.org)
{'fraction_non_alphanumeric': 0.16213697779158254, 'fraction_numerical': 0.03307967075632862, 'mean_word_length': 6.3098751418842225, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5880511', 'n_tokens_mistral': 2366, 'n_tokens_neox': 2225, 'n_words': 494}