<commit_before>======
pysolr
======

``pysolr`` is a lightweight Python client for `Apache Solr`_. It provides an
interface that queries the server and returns results based on the query.

.. _`Apache Solr`: http://lucene.apache.org/solr/

Status
======

.. image:: https://secure.travis-ci.org/django-haystack/pysolr.png
   :target: https://secure.travis-ci.org/django-haystack/pysolr

`Changelog <https://github.com/django-haystack/pysolr/blob/master/CHANGELOG.rst>`_

Features
========

* Basic operations such as selecting, updating & deleting.
* Index optimization.
* `"More Like This" <http://wiki.apache.org/solr/MoreLikeThis>`_ support (if set up in Solr).
* `Spelling correction <http://wiki.apache.org/solr/SpellCheckComponent>`_ (if set up in Solr).
* Timeout support.
* SolrCloud awareness

Requirements
============

* Python 2.7 - 3.7
* Requests 2.9.1+
* **Optional** - ``simplejson``
* **Optional** - ``kazoo`` for SolrCloud mode

Installation
============

pysolr is on PyPI:

.. code-block:: console

   $ pip install pysolr

Or if you want to install directly from the repository:

.. code-block:: console

    $ python setup.py install

Usage
=====

Basic usage looks like:

.. code-block:: python

    # If on Python 2.X
    from __future__ import print_function

    import pysolr

    # Create a client instance. The timeout and authentication options are not required.
    solr = pysolr.Solr('http://localhost:8983/solr/', always_commit=True, [timeout=10], [auth=<type of authentication>])

    # Note that auto_commit defaults to False for performance. You can set
    # `auto_commit=True` to have commands always update the index immediately, make
    # an update call with `commit=True`, or use Solr's `autoCommit` / `commitWithin`
    # to have your data be committed following a particular policy.

    # Do a health check.
    solr.ping()

    # How you'd index data.
    solr.add([
        {
            "id": "doc_1",
            "title": "A test document",
        },
        {
            "id": "doc_2",
            "title": "The Banana: Tasty or Dangerous?",
            "_doc": [
                { "id": "child_doc_1", "title": "peel" },
                { "id": "child_doc_2", "title": "seed" },
            ]
        },
    ])

    # You can index a parent/child document relationship by
    # associating a list of child documents with the special key '_doc'. This
    # is helpful for queries that join together conditions on children and parent
    # documents.

    # Later, searching is easy. In the simple case, just a plain Lucene-style
    # query is fine.
    results = solr.search('bananas')

    # The ``Results`` object stores total results found, by default the top
    # ten most relevant results and any additional data like
    # facets/highlighting/spelling/etc.
    print("Saw {0} result(s).".format(len(results)))

    # Just loop over it to access the results.
    for result in results:
        print("The title is '{0}'.".format(result['title']))

    # For a more advanced query, say involving highlighting, you can pass
    # additional options to Solr.
    results = solr.search('bananas', **{
        'hl': 'true',
        'hl.fragsize': 10,
    })

    # You can also perform More Like This searches, if your Solr is configured
    # correctly.
    similar = solr.more_like_this(q='id:doc_2', mltfl='text')

    # Finally, you can delete either individual documents,
    solr.delete(id='doc_1')

    # also in batches...
    solr.delete(id=['doc_1', 'doc_2'])

    # ...or all documents.
    solr.delete(q='*:*')

.. code-block:: python

    # For SolrCloud mode, initialize your Solr like this:

    zookeeper = pysolr.ZooKeeper("zkhost1:2181,zkhost2:2181,zkhost3:2181")
    solr = pysolr.SolrCloud(zookeeper, "collection1", auth=<type of authentication>)


Multicore Index
~~~~~~~~~~~~~~~
<commit_msg>Show an example of how to use cursor traversal<commit_after>======
pysolr
======

``pysolr`` is a lightweight Python client for `Apache Solr`_. It provides an
interface that queries the server and returns results based on the query.

.. _`Apache Solr`: http://lucene.apache.org/solr/

Status
======

.. image:: https://secure.travis-ci.org/django-haystack/pysolr.png
   :target: https://secure.travis-ci.org/django-haystack/pysolr

`Changelog <https://github.com/django-haystack/pysolr/blob/master/CHANGELOG.rst>`_

Features
========

* Basic operations such as selecting, updating & deleting.
* Index optimization.
* `"More Like This" <http://wiki.apache.org/solr/MoreLikeThis>`_ support (if set up in Solr).
* `Spelling correction <http://wiki.apache.org/solr/SpellCheckComponent>`_ (if set up in Solr).
* Timeout support.
* SolrCloud awareness

Requirements
============

* Python 2.7 - 3.7
* Requests 2.9.1+
* **Optional** - ``simplejson``
* **Optional** - ``kazoo`` for SolrCloud mode

Installation
============

pysolr is on PyPI:

.. code-block:: console

   $ pip install pysolr

Or if you want to install directly from the repository:

.. code-block:: console

    $ python setup.py install

Usage
=====

Basic usage looks like:

.. code-block:: python

    # If on Python 2.X
    from __future__ import print_function

    import pysolr

    # Create a client instance. The timeout and authentication options are not required.
    solr = pysolr.Solr('http://localhost:8983/solr/', always_commit=True, [timeout=10], [auth=<type of authentication>])

    # Note that auto_commit defaults to False for performance. You can set
    # `auto_commit=True` to have commands always update the index immediately, make
    # an update call with `commit=True`, or use Solr's `autoCommit` / `commitWithin`
    # to have your data be committed following a particular policy.

    # Do a health check.
    solr.ping()

    # How you'd index data.
    solr.add([
        {
            "id": "doc_1",
            "title": "A test document",
        },
        {
            "id": "doc_2",
            "title": "The Banana: Tasty or Dangerous?",
            "_doc": [
                { "id": "child_doc_1", "title": "peel" },
                { "id": "child_doc_2", "title": "seed" },
            ]
        },
    ])

    # You can index a parent/child document relationship by
    # associating a list of child documents with the special key '_doc'. This
    # is helpful for queries that join together conditions on children and parent
    # documents.

    # Later, searching is easy. In the simple case, just a plain Lucene-style
    # query is fine.
    results = solr.search('bananas')

    # The ``Results`` object stores total results found, by default the top
    # ten most relevant results and any additional data like
    # facets/highlighting/spelling/etc.
    print("Saw {0} result(s).".format(len(results)))

    # Just loop over it to access the results.
    for result in results:
        print("The title is '{0}'.".format(result['title']))

    # For a more advanced query, say involving highlighting, you can pass
    # additional options to Solr.
    results = solr.search('bananas', **{
        'hl': 'true',
        'hl.fragsize': 10,
    })

    # Traverse a cursor using its iterator:
    for doc in solr.search('*:*',fl='id',cursorMark='*'):
        print(doc['id'])

    # You can also perform More Like This searches, if your Solr is configured
    # correctly.
    similar = solr.more_like_this(q='id:doc_2', mltfl='text')

    # Finally, you can delete either individual documents,
    solr.delete(id='doc_1')

    # also in batches...
    solr.delete(id=['doc_1', 'doc_2'])

    # ...or all documents.
    solr.delete(q='*:*')

.. code-block:: python

    # For SolrCloud mode, initialize your Solr like this:

    zookeeper = pysolr.ZooKeeper("zkhost1:2181,zkhost2:2181,zkhost3:2181")
    solr = pysolr.SolrCloud(zookeeper, "collection1", auth=<type of authentication>)


Multicore Index
~~~~~~~~~~~~~~~

{'fraction_non_alphanumeric': 0.13708349170150766, 'fraction_numerical': 0.010642341315089319, 'mean_word_length': 3.025497195308516, 'pattern_counts': {'":': 18, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5465614', 'n_tokens_mistral': 2544, 'n_tokens_neox': 2422, 'n_words': 781}