<commit_before>Polyglot
========

Polyglot is a language detector written in Python 2.7.

It's heavily inspired in <a href="https://github.com/github/linguist"> Linguist </a> by GitHub.

This project was born as a learning experience with the Python language. It's meant to be taken seriously but, still, is far from what Linguist is capable of.

We still have a lot to work on.

### Usage

Running the program on its on `polyglot/` folder:

    $ python daemon.py ../polyglot/
    
    83.3% of the files were recognized. Recognized files:
    ** Python: 67.0%
    ** YAML: 17.0%
    16.7% of the files were of unknown type. Unknown extensions:
    'pyc'
  
### Language Detection

### Installation

For the moment there's no installation *per se*. For now, just run the script to get the output.

It the future there will probably exist one.

### Contributing

For now, contributions will only be accepted for the `languages.yml` file because the project is still under heavy development and, specially, lacking a definitive direction.

We want to understand where we want to go first, and go there, before we can have other people on board.

### Todo

- Make sure the core functionalities are robust and well implemented.
- Create an analyser to classify languages that can be tricky.<commit_msg>Updated README to newest changes<commit_after>Polyglot
========

Polyglot is a language detector written in Python 2.7.

It's heavily inspired in <a href="https://github.com/github/linguist"> Linguist </a> by GitHub.

This project was born as a learning experience with the Python language. It's meant to be taken seriously but, still, is far from what Linguist is capable of.

We still have a lot to work on.

### Usage

Running the program on its on `polyglot/` folder without the debug option:

    $ python polyglot.py ../polyglot/
    
    ** Python: 75.0%
    	polyglot\polyglot\polyglot.py
    	polyglot\polyglot\test.py
    	polyglot\polyglot\__init__.py
    ** YAML: 25.0%
    	polyglot\polyglot\languages.yml

With the debug option:

    $ python polyglot.py ../polyglot/
    
    ** Python: 75.0%
    	polyglot\polyglot\polyglot.py
    	polyglot\polyglot\test.py
    	polyglot\polyglot\__init__.py
    ** YAML: 25.0%
    	polyglot\polyglot\languages.yml
    Unknown extensions:
    'txt', 'pyc'

  
### Language Detection

### Installation

For the moment there's no installation *per se*. For now, just run the script to get the output.

It the future there will probably exist one.

### Contributing

For now, contributions will only be accepted for the `languages.yml` file because the project is still under heavy development and, specially, lacking a definitive direction.

We want to understand where we want to go first, and go there, before we can have other people on board.

### Todo

- Make sure the core functionalities are robust and well implemented.
- Create an analyser to classify languages that can be tricky.
{'fraction_non_alphanumeric': 0.08137555328566565, 'fraction_numerical': 0.009533537623425264, 'mean_word_length': 3.471841704718417, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '545671', 'n_tokens_mistral': 900, 'n_tokens_neox': 849, 'n_words': 387}