<commit_before>*Work in progress*

# Labelizer

This code build a LSTM Neural Network to classify scientific abstract into a set of labels.

## Requirements

*TODO: reduce the amount of required packages and specify it per file*

This code is written in Python and require a consequent amount of additional package, mostly included in distribution such as [Anaconda](http://continuum.io/downloads) or [Canopy](https://www.enthought.com/products/canopy/).
Additionally, you need to install:
`TextBlob` for fast lemmatization
`Keras` for neural network implementation (based on theano)
`Seaborn` for visualization
You can get this packages through pip (prepend sudo if required):
```bash
pip install textblob
pip install textblob-aptagger
pip install keras
pip install seaborn
```

If you encounter `ImportError: No module named packages` while installing textblob-aptagger with anaconda:
In `/home/username/anaconda/lib/python2.7/site-packages/textblob_aptagger/taggers.py` change line 10:
`from textblob.packages import nltk`
to
`import nltk`

## Usage

Try the notebook 'Complete_workflow_V2' for a step by step process

*TODO*

## License

MIT<commit_msg>correction<commit_after>*Work in progress*

# Labelizer

This code build a LSTM Neural Network to classify scientific abstract into a set of labels.

## Requirements

*TODO: reduce the amount of required packages and specify it per file*

This code is written in Python and require packages mostly included in distribution such as [Anaconda](http://continuum.io/downloads) or [Canopy](https://www.enthought.com/products/canopy/).
Additionally, you need to install:
`TextBlob` for fast lemmatization
`Keras` for neural network implementation (based on theano)
`Seaborn` for visualization
You can get this packages through pip (prepend sudo if required):
```bash
pip install textblob
pip install textblob-aptagger
pip install keras
pip install seaborn
```

If you encounter `ImportError: No module named packages` while installing textblob-aptagger with anaconda:
In `/home/username/anaconda/lib/python2.7/site-packages/textblob_aptagger/taggers.py` change line 10:
`from textblob.packages import nltk`
to
`import nltk`

## Usage

Try the notebook 'Complete_workflow_V2' for a step by step process

*TODO*

## License

MIT
{'fraction_non_alphanumeric': 0.07597716293368467, 'fraction_numerical': 0.004391743522178305, 'mean_word_length': 5.363128491620111, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2383006', 'n_tokens_mistral': 685, 'n_tokens_neox': 649, 'n_words': 272}