<commit_before>auto-tagger
===========

Text classifier that suggests tags for Stackoverflow questions.

## Requirements, Assumptions

This software has been tested on a relatively clean Mac OS (10.9.3), with the following components installed:

* Python 2.7
* Node.js/npm (0.10/1.4)

If not available, the easiest way to obtain these is using [Homebrew](http://brew.sh/).

For the front-end, we'll need the [Bower package manager](http://bower.io/), which can be installed with `npm install -g bower`.

## Additional Installation

The project depends on several Python and front-end packages, as well as a locally running Solr server. These can be obtained as follows:

1. `pip install -r requirements.txt` -- you may want to install these packages into a Python virtualenv for this project,
2. `cd src/main/python/static/` and `bower install` will download all front-end dependencies into a `bower_components` directory,
3. `brew install solr` will install the Solr server. If not using Homebrew, Solr can be installed manually from [Apache](http://lucene.apache.org/solr/);

## Getting Stackoverflow questions into Solr

The project contains a script for downloading questions from the Stackoverflow api. Running 

```
python get-so-data.py
```

from `src/main/python` will download the questions into a file `question_data.json`. Currently, the download will consist of 250k questions and may take up to 1 hr. To modify the download size, you can change the value of `questions_per_tag` in the script.

Once the download is finished, it's time to start Solr. Solr requires a directory for reading its configuration and writing its data. In this case, it should be the absolute path pointing to the `src/main/solr-config` directory inside of this project. As such, starting Solr should look as 

```
solr [absolute_path_to_auto_tagger_project]/src/main/solr-config
```

Confirm that Solr is running by checking http://localhost:8983/solr/.<commit_msg>Modified solr startup instructions to reflect changes in latest stable version<commit_after>auto-tagger
===========

Text classifier that suggests tags for Stackoverflow questions.

## Requirements, Assumptions

This software has been tested on a relatively clean Mac OS (10.9.3), with the following components installed:

* Python 2.7
* Node.js/npm (0.10/1.4)

If not available, the easiest way to obtain these is using [Homebrew](http://brew.sh/).

For the front-end, we'll need the [Bower package manager](http://bower.io/), which can be installed with `npm install -g bower`.

## Additional Installation

The project depends on several Python and front-end packages, as well as a locally running Solr server. These can be obtained as follows:

1. `pip install -r requirements.txt` -- you may want to install these packages into a Python virtualenv for this project,
2. `cd src/main/python/static/` and `bower install` will download all front-end dependencies into a `bower_components` directory,
3. `brew install solr` will install the Solr server. If not using Homebrew, Solr can be installed manually from [Apache](http://lucene.apache.org/solr/);

## Getting Stackoverflow questions into Solr

The project contains a script for downloading questions from the Stackoverflow api. Running 

```
python get-so-data.py
```

from `src/main/python` will download the questions into a file `question_data.json`. Currently, the download will consist of 250k questions and may take up to 1 hr. To modify the download size, you can change the value of `questions_per_tag` in the script.

Once the download is finished, it's time to start Solr. Solr requires a directory for reading its configuration and writing its data. In this case, it should be the absolute path pointing to the `src/main/solr-config` directory inside of this project. As such, starting Solr should look as 

```
solr start -s [absolute_path_to_auto_tagger_project]/src/main/solr-config
```
or, for solr versions < 4.9,
```
solr [absolute_path_to_auto_tagger_project]/src/main/solr-config
```

Confirm that Solr is running by checking http://localhost:8983/solr/.
{'fraction_non_alphanumeric': 0.08447600391772772, 'fraction_numerical': 0.011263467189030362, 'mean_word_length': 4.542740841248304, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1129158', 'n_tokens_mistral': 1155, 'n_tokens_neox': 1099, 'n_words': 534}