<commit_before><!-- ---
layout: page
title: Research
permalink: /research/
---
## Indian Language Datasets

As a part of my RnD project at [IIT Bombay](http://www.iitb.ac.in/), I am releasing the dataset I used to train my neural network language models. These have been mined from Wikipedia and I hope this will help further research in language modelling for morphologically rich languages. The folder also contains the original PTB dataset.

* Malayalam (denoted by `ml`)
* Tamil (denoted by `ta`)
* Kannada (denoted by `kn`)
* Telugu (denoted by `te`)
* Hindi (denoted by `hi`)
* PTB (denoted by `ptb`)

All these datasets are compatible with [SRILM](http://www.speech.sri.com/projects/srilm/). Files marked with `unk` have replaced all singletons with `<unk>` tokens. Files marked with `char` are character versions. All datasets have a `train`, `valid` and `test` file.

You will find the dataset [here](https://drive.google.com/file/d/0B5Y_SiDYwIObaE52dmZ0YVFXckU/view?usp=sharing).
 --><commit_msg>Removing research page<commit_after>---
layout: page
title: Research
permalink: /research/
---
<!-- ## Indian Language Datasets

As a part of my RnD project at [IIT Bombay](http://www.iitb.ac.in/), I am releasing the dataset I used to train my neural network language models. These have been mined from Wikipedia and I hope this will help further research in language modelling for morphologically rich languages. The folder also contains the original PTB dataset.

* Malayalam (denoted by `ml`)
* Tamil (denoted by `ta`)
* Kannada (denoted by `kn`)
* Telugu (denoted by `te`)
* Hindi (denoted by `hi`)
* PTB (denoted by `ptb`)

All these datasets are compatible with [SRILM](http://www.speech.sri.com/projects/srilm/). Files marked with `unk` have replaced all singletons with `<unk>` tokens. Files marked with `char` are character versions. All datasets have a `train`, `valid` and `test` file.

You will find the dataset [here](https://drive.google.com/file/d/0B5Y_SiDYwIObaE52dmZ0YVFXckU/view?usp=sharing).
 -->
{'fraction_non_alphanumeric': 0.11479465611083622, 'fraction_numerical': 0.004948045522018802, 'mean_word_length': 4.479674796747967, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1112729', 'n_tokens_mistral': 642, 'n_tokens_neox': 621, 'n_words': 256}