<commit_before># LSTM with residual connection
A model to generate sanskrit text and learn reprenstations of sankrit aksharas(unit of graphemic symbols in the Indian writing system). 

### Hypothesis: 
Sanskrit is a finely structured language. Minor variations in aksharas change the meaning of the words. Composition and translation of Sanskrit texts is still a challenging  task due to the semantic perplexity in its beautiful and rich verses. Refer to the following examples-
#### Pronouns-
| विभक्ति – Cases | एकवचन – Singular | द्विवचन – Dual | बहुवचन – Plular |
|:--------------:|:----------------:|:-------------:|:--------------:|
|       1.       |       रामः       |      रामौ     |      रामाः     |
|       2.       |        रामं       |      रामौ     |      रामान्     |
|       3.       |       रामेण       |    रामाभ्यां    |      रामैः      |

Here look at rama – अत्र  ***रामं***  पश्य 
&nbsp;
Here look at two ramas – अत्र  ***रामौ***  पश्य 
&nbsp; 
in a rama - ***रामे*** 

#### Sandhi-
***सः पश्यति → स पश्यति 
&nbsp;
< saḥ paśyati → sa paśyati >
&nbsp;
He sees.***


Residual connection[[1]](#resnet) in the model will aid in learning simple aksharas mappings, whereas LSTM[[2]](#lstm) cells will learn complex relationships. 

*Note: This model was trained on Nvidia GeForce GT 730 using Chapter-7 of Mahabharata ~ 677972 characters. If you've more computation resources, you can use bigger training corpus.*

<p align="center">
<img src="model.png">
</p>

### Requirements:
* Thenao
* Keras

### Usage:
`python reslstm.py`
#### To use pretrained weights:
* Remove layers weight intializations.
* Uncomment line 139 in reslstm.py and add your weights.
* Fine-tuning: load given weights and add parameter *trainable = False* for layers which doesn't require fine-tuning.

#### Future Work:
* The residual connection in this model, merges input vectors (binary  vectors) and dense layer features. One can implement element wise addition to check for improvemnts in feature representation with more data. 
* Comapare Character level CNN and ResLSTM for Sanskrit language modeling tasks.
* Apply stateful LSTM and compare.



####Citations:
<a id="resnet">
[1]<NAME>., <NAME>., <NAME>., & <NAME>. (2016). Identity mappings in deep residual networks. arXiv preprint arXiv:1603.05027.

<a id="lstm">
[2]<NAME>., & <NAME>. (2005). Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5), 602-610.


*Please feel free to contact me at ishak269[at]gmail.com to report any issues.*<commit_msg>typo fix<commit_after># LSTM with residual connection
A model to generate sanskrit text and learn reprenstations of sankrit aksharas(unit of graphemic symbols in the Indian writing system). 

### Hypothesis: 
Sanskrit is a finely structured language. Minor variations in aksharas change the meaning of the words. Composition and translation of Sanskrit texts is still a challenging  task due to the semantic perplexity in its beautiful and rich verses. Refer to the following examples-
#### Pronouns-
| विभक्ति – Cases | एकवचन – Singular | द्विवचन – Dual | बहुवचन – Plular |
|:--------------:|:----------------:|:-------------:|:--------------:|
|       1.       |       रामः       |      रामौ     |      रामाः     |
|       2.       |        रामं       |      रामौ     |      रामान्     |
|       3.       |       रामेण       |    रामाभ्यां    |      रामैः      |

Here look at rama – अत्र  ***रामं***  पश्य 
&nbsp;
Here look at two ramas – अत्र  ***रामौ***  पश्य 
&nbsp; 
in a rama - ***रामे*** 

#### Sandhi-
***सः पश्यति → स पश्यति 
&nbsp;
< saḥ paśyati → sa paśyati >
&nbsp;
He sees.***


Residual connection[[1]](#resnet) in the model will aid in learning simple aksharas mappings, whereas LSTM[[2]](#lstm) cells will learn complex relationships. 

*Note: This model was trained on Nvidia GeForce GT 730 using Chapter-7 of Mahabharata ~ 677972 characters. If you've more computation resources, you can use bigger training corpus.*

<p align="center">
<img src="model.png">
</p>

### Requirements:
* Thenao
* Keras

### Usage:
`python reslstm.py`
#### To use pretrained weights:
* Remove layers weight intializations.
* Uncomment line 139 in reslstm.py and add your weights.
* Fine-tuning: load given weights and add parameter *trainable = False* for layers which doesn't require fine-tuning.

#### Future Work:
* The residual connection in this model, merges input vectors (binary  vectors) and dense layer features. One can implement element wise addition to check for improvemnts in feature representation with more data. 
* Comapare Character level CNN and ResLSTM for Sanskrit language modeling tasks.
* Apply stateful LSTM and compare.



####Citations:
<a id="resnet">
[1]<NAME>., <NAME>., <NAME>., & <NAME>. (2016). Identity mappings in deep residual networks. arXiv preprint arXiv:1603.05027.

<a id="lstm">
[2]<NAME>., & <NAME>. (2005). Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5), 602-610.


*Please feel free to contact me at <EMAIL>9[at]gmail.com to report any issues.*
{'fraction_non_alphanumeric': 0.1403235236795946, 'fraction_numerical': 0.018709803157279284, 'mean_word_length': 3.138709677419355, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5944615', 'n_tokens_mistral': 1792, 'n_tokens_neox': 1655, 'n_words': 638}