task_categories:feature-extraction size_categories:1K<n<10K language:tg license:apache-2.0 text_segmentaion nlp tg tajik sentence_segmentation region:us
---
license: apache-2.0
task_categories:
- feature-extraction
language:
- tg
tags:
- text_segmentaion
- nlp
- tg
- tajik
- sentence_segmentation
pretty_name: Tajik sentence-wise text segmentation
size_categories:
- 1K<n<10K
---

This dataset contains texts in Tajik language with sentence annotations. It can be used to train and evaluate sentence-wise text segmentation algorithms.
The dataset contains more than 100 short and long texts and more than 3000 annotated sentences. The texts were carefully selected from different catergories 
such as news, articles, novels, classical texts, poetry, and religious texts. It deliberately contains more of "hard" passages where splitting them by period "." characters would result in bad segmentation.

No preprocessing is done except reducing consecutive whitespaces and linebreaks to singles. The texts are sometimes poorly formatted just as they are copied and pasted from the web. This could make the training algorithm robust to noises.
