<commit_before>= BigIndex

A Rails plugin that drops into models and provides indexing functionality. Uses an adapter/repository pattern inspired by Datamapper to abstract the actual indexer used in the background, and exposes the model to a simple indexing API.

This should be used in conjunction with BigRecord in order to provide a more complete ORM.

== Supported search servers

* Solr
* Sphinx (planned)

== Installation

(1) Install the Solr search server into your Rails application with the bigindex-solr package [http://github.com/openplaces/bigindex-solr]. Alternatively, if you want to setup your own Solr server, you'll need to read the [Solr schema] section below and take note of the Solr url to connect to.

(2) In your Rails application, add Bigindex as a gem to your config/environment.rb file:

  config.gem "bigindex", :source => "http://gemcutter.org"

and run the following rake task to install all the gems listed for your Rails app:

  [sudo] rake gems:install

(3) Add the following line to the bottom of your RAILS_ROOT/Rakefile

  require 'bigindex/tasks'

=== NOTE: Skip steps (4) and (5) if you're using the bigindex-solr package.

(4) Create a Bigindex config file for your Rails application

  rake bigindex:generate_config

(5) Modify the config file config/bigindex.yml[.sample] to correspond to your Solr server.


== Getting Started

Modify your Ruby class/model similarly to the following:

  class Model < BigRecord::Base
    include BigIndex::Resource                    # 1. Include the BigIndex::Resource module into your model.

    column :name, :string
    column :description, :text

    index :name,  :string                         # 2. Define each attribute you want to index along with its type.
    index :description                            # Defaults to type :text
  end

BigIndex will then override the default Model.find() method to pass through the indexer first. Model.find() will also accept the option {:bypass_index => true}, which bypasses the indexed #find method and dispatches it to the original Model.find() method, e.g. Model.find(:all, :bypass_index => true). Alternatively, you can use Model.find_without_index(:all) for the same functionality.

== Rebuilding your index

Once you have your models setup with Bigindex, you will need to rebuild the index. To do so, run the rake task:

  rake bigindex:rebuild_index<commit_msg>Small update to README<commit_after>= BigIndex

A Rails plugin that drops into models and provides indexing functionality. Uses an adapter/repository pattern inspired by Datamapper to abstract the actual indexer used in the background, and exposes the model to a simple indexing API.

This should be used in conjunction with BigRecord in order to provide a more complete ORM.

== Supported search servers

* Solr
* Sphinx (planned)

== Installation

(1) Install the Solr search server into your Rails application with the bigindex-solr package [http://github.com/openplaces/bigindex-solr]. Alternatively, if you want to setup your own Solr server, you'll need to read the [Solr schema] section below and take note of the Solr url to connect to.

(2) In your Rails application, add Bigindex as a gem to your config/environment.rb file:

  config.gem "bigindex", :source => "http://gemcutter.org"

and run the following rake task to install all the gems listed for your Rails app:

  [sudo] rake gems:install

(3) Add the following line to the bottom of your RAILS_ROOT/Rakefile

  require 'big_index/tasks'

=== NOTE: Skip steps (4) and (5) if you're using the bigindex-solr package.

(4) Create a Bigindex config file for your Rails application

  rake bigindex:generate_config

(5) Modify the config file config/bigindex.yml[.sample] to correspond to your Solr server.


== Getting Started

Modify your Ruby class/model similarly to the following:

  class Model < BigRecord::Base
    include BigIndex::Resource                    # 1. Include the BigIndex::Resource module into your model.

    column :name, :string
    column :description, :text

    index :name,  :string                         # 2. Define each attribute you want to index along with its type.
    index :description                            # Defaults to type :text
  end

BigIndex will then override the default Model.find() method to pass through the indexer first. Model.find() will also accept the option {:bypass_index => true}, which bypasses the indexed #find method and dispatches it to the original Model.find() method, e.g. Model.find(:all, :bypass_index => true). Alternatively, you can use Model.find_without_index(:all) for the same functionality.

== Rebuilding your index

Once you have your models setup with Bigindex, you will need to rebuild the index. To do so, run the rake task:

  rake bigindex:rebuild_index
{'fraction_non_alphanumeric': 0.06580591420241566, 'fraction_numerical': 0.003748438150770512, 'mean_word_length': 3.7366863905325443, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '443882', 'n_tokens_mistral': 1332, 'n_tokens_neox': 1245, 'n_words': 638}