<commit_before># AGROVOC autocoder

This Open-Ag auto-classification model is a product of 
[Foundation Center](http://foundationcenter.org/). It was developed
by <NAME> (<EMAIL>) and <NAME> 
(<EMAIL>). This project uses Python 3.5.x in order to handle all
UTF-8 encoding issues. 

Training data for this model was obtained from 
[Food and Agriculture Organization of the United Nations](http://agris.fao.org/agris-search/index.do).

The predictions are available for free via the `/text/ag_classification` REST API endpoint
at [apibeta.foundationcenter.org](https://apibeta.foundationcenter.org/docs/v2.0/documentation.html#/README).
If you wish to host the model locally, the pre-trained models can be 
[downloaded](https://s3.amazonaws.com/fc-public/svm/open_ag_models.zip), and 
should be stored in `src\model\clf_data\`. They can be served using the Flask
server included in this project.

## Install

This project contains the scraping code used for extracting training data
from FAO AGROVOC. If you would like to be able to run the scraping script
then please do the following in a virtual environment (note: this is not
supported in Windows):

On Ubuntu:

    ./bootstrap.sh
    
This will install Anaconda with Python 3, which includes dependent
libraries such as scikit-learn. <commit_msg>Update README to reflect production API endpoint.<commit_after># AGROVOC autocoder

This Open-Ag auto-classification model is a product of 
[Foundation Center](http://foundationcenter.org/). It was developed
by <NAME> (<EMAIL>) and <NAME> 
(<EMAIL>). This project uses Python 3.5.x in order to handle all
UTF-8 encoding issues. 

Training data for this model was obtained from 
[Food and Agriculture Organization of the United Nations](http://agris.fao.org/agris-search/index.do).

The predictions are available as a free service through the Foundation Center's API via the 
`/text/ag_classification` REST API endpoint at [api.foundationcenter.org](https://api.foundationcenter.org).

If you wish to host the model locally, the pre-trained models can be 
[downloaded](https://s3.amazonaws.com/fc-public/svm/open_ag_models.zip), and 
should be stored in `src\model\clf_data\`. They can be served using the Flask
server included in this project.

## Install

This project contains the scraping code used for extracting training data
from FAO AGROVOC. If you would like to be able to run the scraping script
then please do the following in a virtual environment (note: this is not
supported in Windows):

On Ubuntu:

    ./bootstrap.sh
    
This will install Anaconda with Python 3, which includes dependent
libraries such as scikit-learn. 
{'fraction_non_alphanumeric': 0.08004552352048558, 'fraction_numerical': 0.004552352048558422, 'mean_word_length': 4.437113402061856, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2384413', 'n_tokens_mistral': 774, 'n_tokens_neox': 733, 'n_words': 327}