<commit_before>shellinford
===========

.. image:: https://badge.fury.io/py/shellinford.png
    :target: http://badge.fury.io/py/shellinford

Shellinford is an implementation of a Wavelet Matrix/Tree succinct data structure for document retrieval.

Based on `shellinford`_ C++ library.

.. _shellinford: https://code.google.com/p/shellinford/

Installation
============

::

 $ pip install shellinford


Usage
=====

Create a new FM-index instance
-----------------------------

::

 >>> import shellinford
 >>> fm = shellinford.FMIndex()


- shellinford.Shellinford([use_wavelet_tree=True, filename=None])

  - When given a filename, Shellinford loads FM-index data from the file


Build FM-index
-----------------------------

::

 >>> fm.build(['Milky Holmes', '<NAME>', 'Milky'], 'milky.fm')

- build([docs, filename])

  - When given a filename, Shellinford stores FM-index data to the file


Search word from FM-index
---------------------------------

::

 >>> for doc in fm.search('Milky'):
 >>>     print 'doc_id:', doc.doc_id
 >>>     print 'count:', doc.count
 >>>     print 'text:', doc.text
 doc_id:    0
 count: 1
 text:  Milky Holmes
 doc_id:    2
 count: 1
 text:  Milky

 >>> for doc in fm.search(['Milky', 'Holmes']):
 >>>     print 'doc_id:', doc.doc_id
 >>>     print 'count:', doc.count
 >>>     print 'text:', doc.text
 doc_id:    1
 count: 1
 text:  Milky Holmes

- search(query)

  - Giving Sequence data (list, tuple, etc.) enables to "AND" search
  - NOTE: The search function is available after FM-index is built or loaded


Add a document
---------------------------------

::

 >>> fm.push_back('Baritsu')

- push_back(doc)

  - NOTE: A document added by this method is not available to search until build


Read FM-index from a binary file
---------------------------------

::

 >>> fm.read('milky_holmes.fm')

- read(path)


Write FM-index binary to a file
---------------------------------

::

 >>> fm.write('milky_holmes.fm')

- write(path)


License
=========
- Wrapper code is licensed under the New BSD License.
- Bundled `shellinford`_ C++ library (c) 2012 echizen_tm is licensed under the New BSD License.

<commit_msg>Add travis badge<commit_after>shellinford
===========

.. image:: https://badge.fury.io/py/shellinford.png
    :target: http://badge.fury.io/py/shellinford
.. image:: https://travis-ci.org/ikegami-yukino/shellinford-python.svg?branch=master
    :target: https://travis-ci.org/ikegami-yukino/shellinford-python

Shellinford is an implementation of a Wavelet Matrix/Tree succinct data structure for document retrieval.

Based on `shellinford`_ C++ library.

.. _shellinford: https://code.google.com/p/shellinford/

Installation
============

::

 $ pip install shellinford


Usage
=====

Create a new FM-index instance
-----------------------------

::

 >>> import shellinford
 >>> fm = shellinford.FMIndex()


- shellinford.Shellinford([use_wavelet_tree=True, filename=None])

  - When given a filename, Shellinford loads FM-index data from the file


Build FM-index
-----------------------------

::

 >>> fm.build(['Milky Holmes', '<NAME>" Shellingford', 'Milky'], 'milky.fm')

- build([docs, filename])

  - When given a filename, Shellinford stores FM-index data to the file


Search word from FM-index
---------------------------------

::

 >>> for doc in fm.search('Milky'):
 >>>     print 'doc_id:', doc.doc_id
 >>>     print 'count:', doc.count
 >>>     print 'text:', doc.text
 doc_id:    0
 count: 1
 text:  <NAME>
 doc_id:    2
 count: 1
 text:  Milky

 >>> for doc in fm.search(['Milky', 'Holmes']):
 >>>     print 'doc_id:', doc.doc_id
 >>>     print 'count:', doc.count
 >>>     print 'text:', doc.text
 doc_id:    1
 count: 1
 text:  <NAME>

- search(query)

  - Giving Sequence data (list, tuple, etc.) enables to "AND" search
  - NOTE: The search function is available after FM-index is built or loaded


Add a document
---------------------------------

::

 >>> fm.push_back('Baritsu')

- push_back(doc)

  - NOTE: A document added by this method is not available to search until build


Read FM-index from a binary file
---------------------------------

::

 >>> fm.read('milky_holmes.fm')

- read(path)


Write FM-index binary to a file
---------------------------------

::

 >>> fm.write('milky_holmes.fm')

- write(path)


License
=========
- Wrapper code is licensed under the New BSD License.
- Bundled `shellinford`_ C++ library (c) 2012 echizen_tm is licensed under the New BSD License.


{'fraction_non_alphanumeric': 0.22291713200804109, 'fraction_numerical': 0.004467277194549922, 'mean_word_length': 2.455246913580247, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 91, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4836235', 'n_tokens_mistral': 1460, 'n_tokens_neox': 1376, 'n_words': 417}