<commit_before>========================
python-ldap_paged_search
========================

Summary
=======

ldap_paged_search is a python library to easily perform LDAP queries with more
than 1000 results, or to break down queries into smaller result sets to reduce
server loads.

Many LDAP servers will not return more than 1000 results unless paged requests
are used.  The existing python ldap library does support pageing, but requires
some not very intuitive coding to perform it.  This library is simply a wrapper
for that code.

Its interface is also slightly easier to perform queries than the LDAP library
it wraps.

Requirements
============

* Tested on python 2.8
* Default python library includes ldap library

Installation
============

Via pip or easy_install
-----------------------

.. code:: bash
    $ sudo pip install ldap_paged_search   # If you prefer PIP
    $ sudo easy_install ldap_paged_search  # If you prefer easy_install

Manual installation
-------------------

.. code:: bash
    $ git clone https://github.com/neoCrimeLabs/python-ldap_paged_search.git
    $ cd python-ldap_paged_search
    $ sudo python setup.py install

Usage
=====

Initial setup
-------------

.. code:: python

    from ldappagedsearch import LdapPagedSearch

    # Required values
    url             = 'ldap://your.ldap.server'
    username        = 'username'      # for anything but active directory
    username        = 'domain\\user'  # for active directory
    password        = '<PASSWORD>'

    baseDN          = 'dc=company,dc=com'
    searchFilter    = '(&(objectCategory=user))'

    # Optional values
    maxPages        = 0     # 0 = everything
    pageSize        = 1000  # How many records per page
                            # Usual max is 1000; check your LDAP server docs

Defining a callback method
--------------------------

.. code:: python

    # Using a callback method to process results uses less memory on large queries
    # Not using a callback search() will return all results as a single list

    def myCallback(dn,record):
        print dn, record

Query using 'with'
------------------

.. code:: python

    # maxPages, pageSize, and callback are all OPTIONAL

    with LdapPagedSearch(url, username, password, maxPages=2, pageSize=2 ) as l:
        results = l.search(baseDN, searchFilter, callback = myCallback)


Alternative query method
------------------------

.. code:: python

    # maxPages, pageSize, and callback are all OPTIONAL

    l = LdapPagedSearch(url, username, password, maxPages=2, pageSize=2 )
    results = l.search(baseDN, searchFilter, callback = myCallback)
    

Results format
--------------

.. code:: python

    # If you don't set a callback, your results will be returned as follows

    [
        ('DistinctName1',
            {
                'FieldName':    ['value1', 'value2'],
                'AnotherField': ['value'], }),
        ('DistinctName2',
            {
                'FieldName':    ['value1', 'value2'],
                'AnotherField': ['value'], }),
        ...
    ]

    
<commit_msg>initial commit<commit_after>========================
python-ldap_paged_search
========================

Summary
=======

ldap_paged_search is a python library to easily perform LDAP queries with more
than 1000 results, or to break down queries into smaller result sets to reduce
server loads.

Many LDAP servers will not return more than 1000 results unless paged requests
are used.  The existing python ldap library does support pageing, but requires
some not very intuitive coding to perform it.  This library is simply a wrapper
for that code.

Its interface is also slightly easier to perform queries than the LDAP library
it wraps.

Requirements
============

* Tested on python 2.8
* Default python library includes ldap library

Installation
============

Via pip or easy_install
-----------------------

.. code:: bash

    $ sudo pip install ldap_paged_search   # If you prefer PIP
    $ sudo easy_install ldap_paged_search  # If you prefer easy_install

Manual installation
-------------------

.. code:: bash

    $ git clone https://github.com/neoCrimeLabs/python-ldap_paged_search.git
    $ cd python-ldap_paged_search
    $ sudo python setup.py install

Usage
=====

Initial setup
-------------

.. code:: python

    from ldappagedsearch import LdapPagedSearch

    # Required values
    url             = 'ldap://your.ldap.server'
    username        = 'username'      # for anything but active directory
    username        = 'domain\\user'  # for active directory
    password        = '<PASSWORD>'

    baseDN          = 'dc=company,dc=com'
    searchFilter    = '(&(objectCategory=user))'

    # Optional values
    maxPages        = 0     # 0 = everything
    pageSize        = 1000  # How many records per page
                            # Usual max is 1000; check your LDAP server docs

Defining a callback method
--------------------------

.. code:: python

    # Using a callback method to process results uses less memory on large queries
    # Not using a callback search() will return all results as a single list

    def myCallback(dn,record):
        print dn, record

Query using 'with'
------------------

.. code:: python

    # maxPages, pageSize, and callback are all OPTIONAL

    with LdapPagedSearch(url, username, password, maxPages=2, pageSize=2 ) as l:
        results = l.search(baseDN, searchFilter, callback = myCallback)


Alternative query method
------------------------

.. code:: python

    # maxPages, pageSize, and callback are all OPTIONAL

    l = LdapPagedSearch(url, username, password, maxPages=2, pageSize=2 )
    results = l.search(baseDN, searchFilter, callback = myCallback)
    

Results format
--------------

.. code:: python

    # If you don't set a callback, your results will be returned as follows

    [
        ('DistinctName1',
            {
                'FieldName':    ['value1', 'value2'],
                'AnotherField': ['value'], }),
        ('DistinctName2',
            {
                'FieldName':    ['value1', 'value2'],
                'AnotherField': ['value'], }),
        ...
    ]

    

{'fraction_non_alphanumeric': 0.1421385688787928, 'fraction_numerical': 0.00973551841635567, 'mean_word_length': 2.2962566844919787, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1712222', 'n_tokens_mistral': 1710, 'n_tokens_neox': 1605, 'n_words': 603}