<commit_before># SciRate 3

A rewrite of [<NAME>'s](http://dabacon.org) SciRate in Ruby on Rails, previously developed by <NAME>, [<NAME>](http://intractable.ca/bill/) and [<NAME>](http://www.mit.edu/~aram/). Currently being expanded upon by [Draftable](https://draftable.com/).

The older production site is deployed at [https://scirate3.herokuapp.com/](https://scirate3.herokuapp.com/). A more recent testing version can be found at [http://scirate-dev.draftable.com/](https://scirate-dev.draftable.com/).

## Contributing

We encourage contributions!

* You can submit a bug report [here](https://github.com/draftable/scirate3/issues).

* You can contribute to the code by sending a pull request on Github to the [canonical repository](https://github.com/draftable/scirate3).

* You can talk about scirate on our [mailing list](https://groups.google.com/forum/?fromgroups=#!forum/scirate) and about scirate development on the [development mailing list](https://groups.google.com/forum/?fromgroups=#!forum/scirate-dev).

## Setting up for development

Scirate is based on [Ruby 2.1.0+](http://rvm.io/) and [Rails 4](http://rubyonrails.org/). Under Ubuntu 12.04 (our current deployment environment) the following native packages are needed:

```shell
sudo apt-get install postgresql libpq-dev libmysqlclient-dev libxml2-dev libxslt-dev nodejs
```

Otherwise, development should be platform agnostic.

```shell
git clone <EMAIL>:draftable/scirate3
cd scirate3
bundle install
cp config/database.yml.example config/database.yml
```

Edit config/database.yml and enter your auth details for the development database.

```shell
rake db:setup
rake arxiv:feed_import
rails server
```

You should now have a working local copy of SciRate! However, you'll also want some papers to fiddle with.

## Populating the database

```shell
rake arxiv:oai_update
```

When run for the first time, this will download paper metadata from the last day. Subsequent calls will download all metadata since the last time. The production server runs this task every day to keep the database in sync.

## Enabling search

Due to the size of the arxiv database, we use a separate specialized search server. You'll need to install [Sphinx 2.1.4+](http://sphinxsearch.com/downloads/release/). Then run the rake tasks to build the initial search index and start the server:

```shell
rake ts:index
rake ts:start
```

## Testing

There is a fairly comprehensive series of unit and integration tests in `spec`. Running `rspec` in the top-level directory will attempt all of them.<commit_msg>Sphinxsearch doesn't seem to be optional for dev<commit_after># SciRate 3

A rewrite of [Dave Bacon's](http://dabacon.org) SciRate in Ruby on Rails, previously developed by <NAME>, [<NAME>](http://intractable.ca/bill/) and [<NAME>](http://www.mit.edu/~aram/). Currently being expanded upon by [Draftable](https://draftable.com/).

The older production site is deployed at [https://scirate3.herokuapp.com/](https://scirate3.herokuapp.com/). A more recent testing version can be found at [http://scirate-dev.draftable.com/](https://scirate-dev.draftable.com/).

## Contributing

We encourage contributions!

* You can submit a bug report [here](https://github.com/draftable/scirate3/issues).

* You can contribute to the code by sending a pull request on Github to the [canonical repository](https://github.com/draftable/scirate3).

* You can talk about scirate on our [mailing list](https://groups.google.com/forum/?fromgroups=#!forum/scirate) and about scirate development on the [development mailing list](https://groups.google.com/forum/?fromgroups=#!forum/scirate-dev).

## Setting up for development

Scirate is based on [Ruby 2.1.0+](http://rvm.io/) and [Rails 4](http://rubyonrails.org/). Under Ubuntu 12.04 (our current deployment environment) the following native packages are needed:

```shell
sudo apt-get install postgresql libpq-dev libxml2-dev libxslt-dev nodejs libodbc1 libmysqlclient-dev
```

You will also need to download and install [sphinxsearch 2.1.4](http://sphinxsearch.com/downloads/release/). Otherwise, development should be platform agnostic.

```shell
git clone <EMAIL>:draftable/scirate3
cd scirate3
bundle install
cp config/database.yml.example config/database.yml
```

Edit config/database.yml and enter your auth details for the development database. Then initialize the database and sphinx, and download the basic feed layout:

```shell
rake db:setup
rake ts:generate
rake arxiv:feed_import
rails server
```

You should now have a working local copy of SciRate! However, you'll also want some papers to fiddle with.

## Populating the database

```shell
rake arxiv:oai_update
```

When run for the first time, this will download paper metadata from the last day. Subsequent calls will download all metadata since the last time. The production server runs this task every day to keep the database in sync.

## Testing

There is a fairly comprehensive series of unit and integration tests in `spec`. Running `rspec` in the top-level directory will attempt all of them.
{'fraction_non_alphanumeric': 0.10695927243969948, 'fraction_numerical': 0.00771055753262159, 'mean_word_length': 4.79495990836197, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 16, 'lorem ipsum': 0, 'www.': 2, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6461304', 'n_tokens_mistral': 1559, 'n_tokens_neox': 1436, 'n_words': 573}