<commit_before>A python interface to archive.org
---------------------------------

.. image:: https://travis-ci.org/jjjake/internetarchive.svg
    :target: https://travis-ci.org/jjjake/internetarchive

.. image:: https://img.shields.io/pypi/dm/internetarchive.svg
    :target: https://pypi.python.org/pypi/internetarchive

This package installs a CLI tool named ``ia`` for using archive.org from the command-line.
It also installs the ``internetarchive`` python module for programatic access to archive.org.
Please report all bugs and issues on `Github <https://github.com/jjjake/ia-wrapper/issues>`__.

.. contents:: Table of Contents:


Installation
~~~~~~~~~~~~

You can install this module via pip:

``pip install internetarchive``

Alternatively, you can install a few extra dependencies to help speed things up a bit:

``pip install "internetarchive[speedups]"``

This will install `ujson <https://pypi.python.org/pypi/ujson>`__ for faster JSON parsing,
and `gevent <https://pypi.python.org/pypi/gevent>`__ for concurrent downloads.

If you want to install this module globally on your system instead of inside a ``virtualenv``, use sudo:

``sudo pip install internetarchive``


Configuring
~~~~~~~~~~~
You can configure both the ``ia`` command-line tool and the Python interface from the command-line:

.. code:: bash

    $ ia configure

You will be prompted to enter your Archive.org login credentials. If authorization is successful a config file will be saved
on your computer that contains your Archive.org S3 keys for uploading and modifying metadata.


Command-Line Usage
------------------
Help is available by typing ``ia --help``. You can also get help on a command: ``ia <command> --help``.
Available subcommands are ``configure``, ``metadata``, ``upload``, ``download``, ``search``, ``delete``, ``list``, and ``catalog``.


Downloading
~~~~~~~~~~~

To download the entire `TripDown1905 <https://archive.org/details/TripDown1905>`__ item:

.. code:: bash

    $ ia download TripDown1905

``ia download`` usage examples:

.. code:: bash

    #download just the mp4 files using ``--glob``
    $ ia download TripDown1905 --glob='*.mp4'

    #download all the mp4 files using ``--formats``:
    $ ia download TripDown1905 --format='512Kb MPEG4'

    #download multiple formats from an item:
    $ ia download TripDown1905 --format='512Kb MPEG4' --format='Ogg Video'

    #list all the formats in an item:
    $ ia metadata --formats TripDown1905

    #download a single file from an item:
    $ ia download TripDown1905 TripDown1905_512kb.mp4

    #download multiple files from an item:
    $ ia download TripDown1905 TripDown1905_512kb.mp4 TripDown1905.ogv


Uploading
~~~~~~~~~

You can use the provided ``ia`` command-line tool to upload items. After `configuring ia <https://github.com/jjjake/internetarchive#configuring>`__,
you can upload files like so:

.. code:: bash

    #upload files:
    $ ia upload <identifier> file1 file2 --metadata="title:foo" --metadata="blah:arg"

    #upload from `stdin`:
    $ curl http://dumps.wikimedia.org/kywiki/20130927/kywiki-20130927-pages-logging.xml.gz |
      ia upload <identifier> - --remote-name=kywiki-20130927-pages-logging.xml.gz --metadata="title:Uploaded from stdin."

Metadata
~~~~~~~~

You can use the ``ia`` command-line tool to download item metadata in JSON format:

.. code:: bash

    $ ia metadata TripDown1905

You can also modify metadata after `configuring ia <https://github.com/jjjake/internetarchive#configuring>`__.

.. code:: bash

    $ ia metadata <identifier> --modify="foo:bar" --modify="baz:foooo"

Searching
~~~~~~~~~

You can search using the provided ``ia`` command-line script:

.. code:: bash

    $ ia search 'subject:"market street" collection:prelinger'


Parallel Downloading
~~~~~~~~~~~~~~~~~~~~

If you have the GNU ``parallel`` tool intalled, then you can combine ``ia search`` and ``ia metadata`` to quickly retrieve data for many items in parallel:

.. code:: bash

    $ ia search 'subject:"collection:prelinger' --itemlist | parallel -j10 'ia metadata {} > {}_meta.json'



Python module usage
-------------------

Below is brief overview of the ``internetarchive`` Python library.
Please refer to the `API documentation <http://ia-wrapper.readthedocs.org/en/latest/>`__ for more specific details.

Downloading from Python
~~~~~~~~~~~~~~~~~~~~~~~

The Internet Archive stores data in
`items <http://blog.archive.org/2011/03/31/how-archive-org-items-are-structured/>`__.
You can query the archive using an item identifier:

.. code:: python

    >>> from internetarchive import get_item
    >>> item = get_item('stairs')
    >>> print(item.metadata)

Items contains files. You can download the entire item:<commit_msg>fixed example in readme even more.<commit_after>A python interface to archive.org
---------------------------------

.. image:: https://travis-ci.org/jjjake/internetarchive.svg
    :target: https://travis-ci.org/jjjake/internetarchive

.. image:: https://img.shields.io/pypi/dm/internetarchive.svg
    :target: https://pypi.python.org/pypi/internetarchive

This package installs a CLI tool named ``ia`` for using archive.org from the command-line.
It also installs the ``internetarchive`` python module for programatic access to archive.org.
Please report all bugs and issues on `Github <https://github.com/jjjake/ia-wrapper/issues>`__.

.. contents:: Table of Contents:


Installation
~~~~~~~~~~~~

You can install this module via pip:

``pip install internetarchive``

Alternatively, you can install a few extra dependencies to help speed things up a bit:

``pip install "internetarchive[speedups]"``

This will install `ujson <https://pypi.python.org/pypi/ujson>`__ for faster JSON parsing,
and `gevent <https://pypi.python.org/pypi/gevent>`__ for concurrent downloads.

If you want to install this module globally on your system instead of inside a ``virtualenv``, use sudo:

``sudo pip install internetarchive``


Configuring
~~~~~~~~~~~
You can configure both the ``ia`` command-line tool and the Python interface from the command-line:

.. code:: bash

    $ ia configure

You will be prompted to enter your Archive.org login credentials. If authorization is successful a config file will be saved
on your computer that contains your Archive.org S3 keys for uploading and modifying metadata.


Command-Line Usage
------------------
Help is available by typing ``ia --help``. You can also get help on a command: ``ia <command> --help``.
Available subcommands are ``configure``, ``metadata``, ``upload``, ``download``, ``search``, ``delete``, ``list``, and ``catalog``.


Downloading
~~~~~~~~~~~

To download the entire `TripDown1905 <https://archive.org/details/TripDown1905>`__ item:

.. code:: bash

    $ ia download TripDown1905

``ia download`` usage examples:

.. code:: bash

    #download just the mp4 files using ``--glob``
    $ ia download TripDown1905 --glob='*.mp4'

    #download all the mp4 files using ``--formats``:
    $ ia download TripDown1905 --format='512Kb MPEG4'

    #download multiple formats from an item:
    $ ia download TripDown1905 --format='512Kb MPEG4' --format='Ogg Video'

    #list all the formats in an item:
    $ ia metadata --formats TripDown1905

    #download a single file from an item:
    $ ia download TripDown1905 TripDown1905_512kb.mp4

    #download multiple files from an item:
    $ ia download TripDown1905 TripDown1905_512kb.mp4 TripDown1905.ogv


Uploading
~~~~~~~~~

You can use the provided ``ia`` command-line tool to upload items. After `configuring ia <https://github.com/jjjake/internetarchive#configuring>`__,
you can upload files like so:

.. code:: bash

    #upload files:
    $ ia upload <identifier> file1 file2 --metadata="title:foo" --metadata="blah:arg"

    #upload from `stdin`:
    $ curl http://dumps.wikimedia.org/kywiki/20130927/kywiki-20130927-pages-logging.xml.gz |
      ia upload <identifier> - --remote-name=kywiki-20130927-pages-logging.xml.gz --metadata="title:Uploaded from stdin."

Metadata
~~~~~~~~

You can use the ``ia`` command-line tool to download item metadata in JSON format:

.. code:: bash

    $ ia metadata TripDown1905

You can also modify metadata after `configuring ia <https://github.com/jjjake/internetarchive#configuring>`__.

.. code:: bash

    $ ia metadata <identifier> --modify="foo:bar" --modify="baz:foooo"

Searching
~~~~~~~~~

You can search using the provided ``ia`` command-line script:

.. code:: bash

    $ ia search 'subject:"market street" collection:prelinger'


Parallel Downloading
~~~~~~~~~~~~~~~~~~~~

If you have the GNU ``parallel`` tool intalled, then you can combine ``ia search`` and ``ia metadata`` to quickly retrieve data for many items in parallel:

.. code:: bash

    $ ia search 'collection:prelinger' --itemlist | parallel -j10 'ia metadata {} > {}_meta.json'



Python module usage
-------------------

Below is brief overview of the ``internetarchive`` Python library.
Please refer to the `API documentation <http://ia-wrapper.readthedocs.org/en/latest/>`__ for more specific details.

Downloading from Python
~~~~~~~~~~~~~~~~~~~~~~~

The Internet Archive stores data in
`items <http://blog.archive.org/2011/03/31/how-archive-org-items-are-structured/>`__.
You can query the archive using an item identifier:

.. code:: python

    >>> from internetarchive import get_item
    >>> item = get_item('stairs')
    >>> print(item.metadata)

Items contains files. You can download the entire item:
{'fraction_non_alphanumeric': 0.14595222997252166, 'fraction_numerical': 0.022828154724159798, 'mean_word_length': 3.830525778458397, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 47, 'https://': 20, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 40, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1135113', 'n_tokens_mistral': 2989, 'n_tokens_neox': 2710, 'n_words': 984}