<commit_before>OAI-PMH Harvest
===============

9th May 2013 (2013-05-09)


Contents
--------

- `Description`_
- `Author(s)`_
- `Latest Version`_
  - `Roadmap`_
- `Documentation`_
- `Requirements / Dependencies`_
- `Installation`_
- `Documentation`_
- `Roadmap`_
- `Bugs, Feature requests etc.`_
- `Copyright And Licensing`_
- `Examples`_


Description
-----------

A harvester to collect records from an OAI-PMH enabled provider.

The harvester can be used to carry out one-time harvesting of all records from
a particular OAI-PMH provider by giving its base URL. It can also be used for
selective harvesting, e.g. to harvest only records updated after, or before 
specified dates.

To assist in regular harvesting from one or more OAI-PMH providers, there's
a provider registry. It is possible to associate a short memorable name for a
provider with its base URLs, destination directory for harvested records, and
the format (metadataPrefix) in which records should be harvested. The registry
will also record the date and time of the most recent harvest, and
automatically add this to subsequent requests in order to avoid repeatedly
harvesting unmodified records.

This could be used in conjunction with a scheduler (e.g. CRON) to maintain a
reasonably up-to-date copy of the record in one or more providers. `Examples`_
of how to accomplish these tasks are available below. 


Author(s)
---------

<NAME> <<EMAIL>> at the `University of Liverpool`_ 


Latest Version
--------------


This is a pre-release repository of the code. There are no discrete version
numbers as yet. Source code is under version control and available from:
http://github.com/bloomonkey/oai-harvest


Roadmap
~~~~~~~

**Version 1.0 – April 2013**

- Harvest from a given URL

- Selective harvesting by modification date

- Facilitate regular harvesting from 1 or more providers

  - Register parameters - e.g. URL, Schema (metadataPrefix), target directory -
    for regular harvesting from a named provider
    
  - Specify provider using registered name when harvesting, or harvest from all
    registered providers using the reserved name `all` 
    
  - Harvest only records modified since last harvest (unless overridden)


**Version 1.1 - May 2013**

- Sensible handling of deletions

- Selective harvesting by set

- Limit harvesting to an arbitrary number of records (e.g. to avoid throttling 
  / blacklisting by provider server)


Documentation
-------------

All executable commands are self documenting, i.e. you can get help on how to
use them with the ``-h`` or ``--help`` option.

At this time the only additional documentation that exists can be found in this
README file!


Requirements / Dependencies
---------------------------

- Python_ >= 2.6
- pyoai_
- lxml_
- sqlite3_


Installation
------------

Users
~~~~~

::

    pip install git+http://github.com/bloomonkey/oai-harvest.git#egg=oaiharvest


Developers
~~~~~~~~~~

I recommend that you use virtualenv_ to isolate your development environment
from system Python_ and any packages that may be installed there.

1. In GitHub_, fork the repository

2. Clone your fork::

       git clone git@github.com:<username>/oai-harvest.git

3. Install dependencies::

       pip install -r requirements.txt

4. Install in develop / editable mode::

       pip install -e .


Bugs, Feature requests etc.
---------------------------

Bug reports and feature requests can be submitted to the GitHub issue tracker:
http://github.com/bloomonkey/oai-harvest/issues

If you'd like to contribute code, patches etc. please email the author, or
submit a pull request on GitHub.


Copyright And Licensing
-----------------------

Copyright (c) `University of Liverpool`_, 2013

See `LICENSE.rst <LICENSE.rst>`_ for licensing details.


Examples
--------

Harvesting records from an OAI-PMH provider URL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

All records
'''''''''''

::

   oai-harvest http://example.com/oai


Records modified since a certain date
'''''''''''''''''''''''''''''''''''''

::

   oai-harvest --from 2013-01-01 http://example.com/oai


Getting help on all available options
'''''''''''''''''''''''''''''''''''''

::

   oai-harvest --help


OAI-PMH Provider Registry
~~~~~~~~~~~~~~~~~~~~~~~~~

Adding a provider
'''''''''''''''''

::

   oai-reg add provider1 http://example.com/oai/1

<commit_msg>docs: added example of harvesting a set<commit_after>OAI-PMH Harvest
===============

9th May 2013 (2013-05-09)


Contents
--------

- `Description`_
- `Author(s)`_
- `Latest Version`_
  - `Roadmap`_
- `Documentation`_
- `Requirements / Dependencies`_
- `Installation`_
- `Documentation`_
- `Roadmap`_
- `Bugs, Feature requests etc.`_
- `Copyright And Licensing`_
- `Examples`_


Description
-----------

A harvester to collect records from an OAI-PMH enabled provider.

The harvester can be used to carry out one-time harvesting of all records from
a particular OAI-PMH provider by giving its base URL. It can also be used for
selective harvesting, e.g. to harvest only records updated after, or before 
specified dates.

To assist in regular harvesting from one or more OAI-PMH providers, there's
a provider registry. It is possible to associate a short memorable name for a
provider with its base URLs, destination directory for harvested records, and
the format (metadataPrefix) in which records should be harvested. The registry
will also record the date and time of the most recent harvest, and
automatically add this to subsequent requests in order to avoid repeatedly
harvesting unmodified records.

This could be used in conjunction with a scheduler (e.g. CRON) to maintain a
reasonably up-to-date copy of the record in one or more providers. `Examples`_
of how to accomplish these tasks are available below. 


Author(s)
---------

<NAME> <<EMAIL>> at the `University of Liverpool`_ 


Latest Version
--------------


This is a pre-release repository of the code. There are no discrete version
numbers as yet. Source code is under version control and available from:
http://github.com/bloomonkey/oai-harvest


Roadmap
~~~~~~~

**Version 1.0 – April 2013**

- Harvest from a given URL

- Selective harvesting by modification date

- Facilitate regular harvesting from 1 or more providers

  - Register parameters - e.g. URL, Schema (metadataPrefix), target directory -
    for regular harvesting from a named provider
    
  - Specify provider using registered name when harvesting, or harvest from all
    registered providers using the reserved name `all` 
    
  - Harvest only records modified since last harvest (unless overridden)


**Version 1.1 - May 2013**

- Sensible handling of deletions

- Selective harvesting by set

- Limit harvesting to an arbitrary number of records (e.g. to avoid throttling 
  / blacklisting by provider server)


Documentation
-------------

All executable commands are self documenting, i.e. you can get help on how to
use them with the ``-h`` or ``--help`` option.

At this time the only additional documentation that exists can be found in this
README file!


Requirements / Dependencies
---------------------------

- Python_ >= 2.6
- pyoai_
- lxml_
- sqlite3_


Installation
------------

Users
~~~~~

::

    pip install git+http://github.com/bloomonkey/oai-harvest.git#egg=oaiharvest


Developers
~~~~~~~~~~

I recommend that you use virtualenv_ to isolate your development environment
from system Python_ and any packages that may be installed there.

1. In GitHub_, fork the repository

2. Clone your fork::

       git clone <EMAIL>:<username>/oai-harvest.git

3. Install dependencies::

       pip install -r requirements.txt

4. Install in develop / editable mode::

       pip install -e .


Bugs, Feature requests etc.
---------------------------

Bug reports and feature requests can be submitted to the GitHub issue tracker:
http://github.com/bloomonkey/oai-harvest/issues

If you'd like to contribute code, patches etc. please email the author, or
submit a pull request on GitHub.


Copyright And Licensing
-----------------------

Copyright (c) `University of Liverpool`_, 2013

See `LICENSE.rst <LICENSE.rst>`_ for licensing details.


Examples
--------

Harvesting records from an OAI-PMH provider URL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

All records
'''''''''''

::

   oai-harvest http://example.com/oai


Records modified since a certain date
'''''''''''''''''''''''''''''''''''''

::

   oai-harvest --from 2013-01-01 http://example.com/oai


Records from a named set
''''''''''''''''''''''''

::

   oai-harvest --set "some:set" http://example.com/oai


Getting help on all available options
'''''''''''''''''''''''''''''''''''''

::

   oai-harvest --help


OAI-PMH Provider Registry
~~~~~~~~~~~~~~~~~~~~~~~~~

Adding a provider
'''''''''''''''''

::

   oai-reg add provider1 http://example.com/oai/1


{'fraction_non_alphanumeric': 0.14652345958168456, 'fraction_numerical': 0.010627473148671567, 'mean_word_length': 3.3214460185637518, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '281025', 'n_tokens_mistral': 2692, 'n_tokens_neox': 2445, 'n_words': 1043}