<commit_before>Search
======

Search is built with Haystack on top of a Whoosh search backend. Whoosh is an easy to deploy and highly portale file-based search backend written in Python.<commit_msg>Documentation for search.<commit_after>Search
======

Overview
--------

Open Budget has a simple feature for end users to discover content via keyword entry. Several text fields across the project models are included in search and it is trivial to add more as required. A single search resuls page show the matching results for a query, no matter what the source model of the result, making it simple for common users to get anywhere they need to go in the app.

Open Budget search is implemented through Haystack, a popular Django extension that provides a common interface to several "search backends". For the backend, we use Whoos


Search is built with Haystack on top of a Whoosh search backend. Whoosh is an easy to deploy and highly portale file-based search backend written in Python.

Configuration
-------------

Global
~~~~~~

The main configuration for Open Budget search is done via the project settings:

https://github.com/hasadna/omuni-budget/blob/develop/openbudget/settings/base.py

Here we define the search backend via HAYSTACK_CONNECTIONS, and we also define the scehdules for updating and rebuilding the search index, via CELERYBEAT_SCHEDULE.

Indexes
~~~~~~~

Configuration of data for the search index is per Haystack requirements. Any app that has a search_indexes.py in it's package has models that are registered for search.

Please refer to the Haystack documentation for information on how to make these configurations:

http://django-haystack.readthedocs.org/en/latest/

At present, the following models are indexed:

* Budgets
* Actuals
* Budget Items
* Actual Items
* Entities

Dependencies
------------

Open Budget search depends on the following 3rd party packages:

* Haystack
* Whoosh
* Celery

Haystack
~~~~~~~~

https://github.com/toastdriven/django-haystack

Haystack is a 3rd party Django app that provides a common interface to a number of search backends. This makes it easy to integrate search into an app, and swap out the search engine with any supported backend, or even add a custom backend.

http://django-haystack.readthedocs.org/en/latest/

How to import::

    from haystack import indexes

Example implementation:

https://github.com/hasadna/omuni-budget/blob/develop/openbudget/apps/budgets/search_indexes.py

Whoosh
~~~~~~

https://bitbucket.org/mchaput/whoosh/wiki/Home

Whoosh is a file-based search engine written in Python. It is lightweight and easy to use, making it a great solution for simple search tasks.

In Open Budget, Whoosh is completely behind-the-scenes, and only ever accessed by Haystack.

http://whoosh.readthedocs.org/en/latest/

Celery
~~~~~~

https://github.com/celery/celery

Celery is a very popular python package for task queues.

Celery also has a crontab implementation, making it an ideal replacement for "cron-like" scheduled tasks in Python apps (which is relvant to us here, in oru search implementation).

We use Celery in other Open Budget use cases, but for search, we use Celery Beat to update and rebuild the search indexes at regular intervals.

http://docs.celeryproject.org/en/latest/index.html

How to import::

    from celery.schedules import crontab

Example implementation:

https://github.com/hasadna/omuni-budget/blob/develop/openbudget/settings/base.py

https://github.com/hasadna/omuni-budget/blob/develop/openbudget/commons/tasks.py

Project Code
------------

There is no specific custom code for this feature. All the code in the codebase simply follows settings for Haystack and Celery.
{'fraction_non_alphanumeric': 0.0746955345060893, 'fraction_numerical': 0.0005412719891745603, 'mean_word_length': 4.191011235955056, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7295241', 'n_tokens_mistral': 1014, 'n_tokens_neox': 976, 'n_words': 466}