task_categories:token-classification language:fa region:us
Language recognition has been significantly advanced in recent years by means of modern machine learning methods such as deep learning 
and benchmarks with rich annotations. However, research is still limited in low-resource formal languages. This consists of a significant 
gap in describing the colloquial language especially for low-resourced ones such as Persian. In order to target this gap for low resource languages, 
we propose a “Large Scale Colloquial Persian Dataset” (LSCP). LSCP is hierarchically organized in a semantic taxonomy that focuses on 
multi-task informal Persian language understanding as a comprehensive problem. This encompasses the recognition of multiple semantic aspects in the human-level sentences, 
which naturally captures from the real-world sentences. We believe that further investigations and processing, as well as the application of novel algorithms and methods, 
can strengthen enriching computerized understanding and processing of low resource languages. The proposed corpus consists of 120M sentences resulted from 27M tweets 
annotated with parsing tree, part-of-speech tags, sentiment polarity and translation in five different languages.
@inproceedings{abdi-khojasteh-etal-2020-lscp,
    title = "{LSCP}: Enhanced Large Scale Colloquial {P}ersian Language Understanding",
    author = "Abdi Khojasteh, Hadi  and
      Ansari, Ebrahim  and
      Bohlouli, Mahdi",
    booktitle = "Proceedings of the Twelfth Language Resources and Evaluation Conference",
    month = may,
    year = "2020",
    address = "Marseille, France",
    publisher = "European Language Resources Association",
    url = "https://aclanthology.org/2020.lrec-1.776",
    pages = "6323--6327",
    abstract = "Language recognition has been significantly advanced in recent years by means of modern machine learning methods such as deep learning and benchmarks with rich annotations. However, research is still limited in low-resource formal languages. This consists of a significant gap in describing the colloquial language especially for low-resourced ones such as Persian. In order to target this gap for low resource languages, we propose a {``}Large Scale Colloquial Persian Dataset{''} (LSCP). LSCP is hierarchically organized in a semantic taxonomy that focuses on multi-task informal Persian language understanding as a comprehensive problem. This encompasses the recognition of multiple semantic aspects in the human-level sentences, which naturally captures from the real-world sentences. We believe that further investigations and processing, as well as the application of novel algorithms and methods, can strengthen enriching computerized understanding and processing of low resource languages. The proposed corpus consists of 120M sentences resulted from 27M tweets annotated with parsing tree, part-of-speech tags, sentiment polarity and translation in five different languages.",
    language = "English",
    ISBN = "979-10-95546-34-4",
}
---
task_categories:
- token-classification
language:
- fa
pretty_name: LSCP Dataset (500k samples version)
---

This is a 500 thousand sample version of the original [LSCP dataset](https://iasbs.ac.ir/~ansari/lscp/) that only contains the text and part-of-speech tags and is used for sequence labeling.

### Citation
```bibtex
@InProceedings{abdikhojasteh:2020:LREC,
  author = {Abdi Khojasteh, Hadi  and  Ansari, Ebrahim  and  Bohlouli, Mahdi},
  title  = {LSCP: Enhanced Large Scale Colloquial Persian Language Understanding},
  booktitle = {Proceedings of the Twelfth International Conference on Language Resources and Evaluation (LREC 2020)},
  year      = {2020}
  address   = {Marseille, France},
  publisher = {European Language Resources Association}
  pages  = {6323--6327},
  url    = {https://www.aclweb.org/anthology/2020.lrec-1.776}
}
```