<commit_before>SQLAlchemy-Searchable
=====================


SQLAlchemy-Searchable provides FullText search capabilities for SQLAlchemy models. Currently it only supports PostgreSQL.

QuickStart
----------

1. Import and call make_searchable function.
1. Define TSVectorType columns to your SQLAlchemy declarative model.


First let's define a simple Article model. This model has three fields: id, name and content.
We want the name and content to be fulltext indexed, hence we put them inside the definition of TSVectorType.
::

    import sqlalchemy as sa
    from sqlalchemy.ext.declarative import declarative_base
    from sqlalchemy_searchable import make_searchable
    from sqlalchemy_utils.types import TSVectorType


    Base = declarative_base()

    make_searchable()


    class Article(Base):
        __tablename__ = 'article'

        id = sa.Column(sa.Integer, primary_key=True)
        name = sa.Column(sa.Unicode(255))
        content = sa.Column(sa.UnicodeText)
        search_vector = sa.Column(TSVectorType('name', 'content'))


Now lets create some dummy data.
::


    engine = create_engine('postgres://localhost/sqlalchemy_searchable_test')
    Base.metadata.create_all(engine)

    Session = sessionmaker(bind=engine)
    session = Session()

    article1 = Article(name=u'First article', content=u'This is the first article')
    article2 = Article(name=u'Second article', content=u'This is the second article')

    session.add(article1)
    session.add(article2)
    session.commit()


After we've created the articles, we can search trhough them.
::


    from sqlalchemy_searchable import search


    query = session.query(Article)

    query = search(query, 'first')

    print query.first().name
    >>> First article


Multiple search vectors per class
=================================


::


    class Article(Base):
        __tablename__ = 'article'

        id = sa.Column(sa.Integer, primary_key=True)
        name = sa.Column(sa.Unicode(255))
        content = sa.Column(sa.UnicodeText)
        description = sa.Column(sa.UnicodeText)
        simple_search_vector = sa.Column(TSVectorType('name'))

        fat_search_vector = sa.Column(
            TSVectorType('name', 'content', 'desription')
        )



Search query operators
======================

As of version 0.3.0 SQLAlchemy-Searchable comes with built in search query parser. The search query parser is capable of parsing human readable search queries into PostgreSQL search query syntax.


Basic operators
---------------

AND operator
^^^^^^^^^^^^

Example: Search for articles containing 'star' and 'wars'

The default operator is 'and', hence the following queries are essentially the same:

::

    query = search(query, 'star wars')
    query2 = search(query, 'star and wars')
    assert query == query2

OR operator
^^^^^^^^^^^

Searching for articles containing 'star' or 'wars'

::


    query = search(query, 'star or wars')


Negation operator
^^^^^^^^^^^^^^^^^

SQLAlchemy-Searchable search query parser supports negation operator. By default the negation operator is '-'.

Example: Searching for article containing 'star' but not 'wars'

::


    query = search(query, 'star or -wars')



Using parenthesis
-----------------

1. Searching for articles containing 'star' and 'wars' or 'luke'

::


    query = search(query '(star wars) or luke')



Special cases
-------------


Hyphens between words
^^^^^^^^^^^^^^^^^^^^^

SQLAlchemy-Searchable is smart enough to not convert hyphens between words to negation operators. Instead, it simply converts all hyphens between words to spaces.

Hence the following search queries are essentially the same:

::


    query = search(query, 'star wars')
    query2 = search(query, 'star-wars')


Emails as search terms
^^^^^^^^^^^^^^^^^^^^^^

PostgreSQL tsvectors handle recognize email strings in a way that they don't get split into multiple tsvector terms. SQLAlchemy-Searchable handles email search terms the same way:

::

    # single search term used: '<EMAIL>'
    query = search(query, u'<EMAIL>')

    # not a valid email, split into three search terms:
    # 'john', 'fastmonkeys' and 'com'
    query = search(query, u'<EMAIL>)


Internals
---------

If you wish to use only the query parser this can be achieved by invoking `parse_search_query` function. This function parses human readable search query into PostgreSQL specific format.

::


    parse_search_query('(star wars) or luke')<commit_msg>Fixed typo<commit_after>SQLAlchemy-Searchable
=====================


SQLAlchemy-Searchable provides FullText search capabilities for SQLAlchemy models. Currently it only supports PostgreSQL.

QuickStart
----------

1. Import and call make_searchable function.
1. Define TSVectorType columns to your SQLAlchemy declarative model.


First let's define a simple Article model. This model has three fields: id, name and content.
We want the name and content to be fulltext indexed, hence we put them inside the definition of TSVectorType.
::

    import sqlalchemy as sa
    from sqlalchemy.ext.declarative import declarative_base
    from sqlalchemy_searchable import make_searchable
    from sqlalchemy_utils.types import TSVectorType


    Base = declarative_base()

    make_searchable()


    class Article(Base):
        __tablename__ = 'article'

        id = sa.Column(sa.Integer, primary_key=True)
        name = sa.Column(sa.Unicode(255))
        content = sa.Column(sa.UnicodeText)
        search_vector = sa.Column(TSVectorType('name', 'content'))


Now lets create some dummy data.
::


    engine = create_engine('postgres://localhost/sqlalchemy_searchable_test')
    Base.metadata.create_all(engine)

    Session = sessionmaker(bind=engine)
    session = Session()

    article1 = Article(name=u'First article', content=u'This is the first article')
    article2 = Article(name=u'Second article', content=u'This is the second article')

    session.add(article1)
    session.add(article2)
    session.commit()


After we've created the articles, we can search trhough them.
::


    from sqlalchemy_searchable import search


    query = session.query(Article)

    query = search(query, 'first')

    print query.first().name
    >>> First article


Multiple search vectors per class
=================================


::


    class Article(Base):
        __tablename__ = 'article'

        id = sa.Column(sa.Integer, primary_key=True)
        name = sa.Column(sa.Unicode(255))
        content = sa.Column(sa.UnicodeText)
        description = sa.Column(sa.UnicodeText)
        simple_search_vector = sa.Column(TSVectorType('name'))

        fat_search_vector = sa.Column(
            TSVectorType('name', 'content', 'desription')
        )



Search query operators
======================

As of version 0.3.0 SQLAlchemy-Searchable comes with built in search query parser. The search query parser is capable of parsing human readable search queries into PostgreSQL search query syntax.


Basic operators
---------------

AND operator
^^^^^^^^^^^^

Example: Search for articles containing 'star' and 'wars'

The default operator is 'and', hence the following queries are essentially the same:

::

    query = search(query, 'star wars')
    query2 = search(query, 'star and wars')
    assert query == query2

OR operator
^^^^^^^^^^^

Searching for articles containing 'star' or 'wars'

::


    query = search(query, 'star or wars')


Negation operator
^^^^^^^^^^^^^^^^^

SQLAlchemy-Searchable search query parser supports negation operator. By default the negation operator is '-'.

Example: Searching for article containing 'star' but not 'wars'

::


    query = search(query, 'star or -wars')



Using parenthesis
-----------------

1. Searching for articles containing 'star' and 'wars' or 'luke'

::


    query = search(query '(star wars) or luke')



Special cases
-------------


Hyphens between words
^^^^^^^^^^^^^^^^^^^^^

SQLAlchemy-Searchable is smart enough to not convert hyphens between words to negation operators. Instead, it simply converts all hyphens between words to spaces.

Hence the following search queries are essentially the same:

::


    query = search(query, 'star wars')
    query2 = search(query, 'star-wars')


Emails as search terms
^^^^^^^^^^^^^^^^^^^^^^

PostgreSQL tsvectors handle email strings in a way that they don't get split into multiple tsvector terms. SQLAlchemy-Searchable handles email search terms the same way:

::

    # single search term used: '<EMAIL>'
    query = search(query, u'<EMAIL>')

    # not a valid email, split into three search terms:
    # 'john', 'fastmonkeys' and 'com'
    query = search(query, u'<EMAIL>)


Internals
---------

If you wish to use only the query parser this can be achieved by invoking `parse_search_query` function. This function parses human readable search query into PostgreSQL specific format.

::


    parse_search_query('(star wars) or luke')
{'fraction_non_alphanumeric': 0.12619074302364675, 'fraction_numerical': 0.004258657402218985, 'mean_word_length': 3.583461736004109, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1594172', 'n_tokens_mistral': 2592, 'n_tokens_neox': 2417, 'n_words': 877}