<commit_before>====================
django-postgres-copy
====================

Quickly move comma-delimited data in and out of a Django model using PostgreSQL's COPY command.


Why and what for?
-----------------

`The people <http://www.californiacivicdata.org/about/>`_ who made this library are data journalists. We are often downloading, cleaning and analyzing new data.

That means we write a load of loaders. You can usually do this by looping through each row and saving it to the database using the Django's ORM `create method <https://docs.djangoproject.com/en/1.10/ref/models/querysets/#django.db.models.query.QuerySet.create>`_.

.. code-block:: python

    import csv
    from myapp.models import MyModel

    data = csv.DictReader(open("./data.csv"))
    for row in data:
        MyModel.objects.create(name=row['NAME'], number=row['NUMBER'])

But if you have a big CSV, Django will rack up database queries and it can take a long time to finish.

Lucky for us, PostgreSQL has a built-in tool called `COPY <http://www.postgresql.org/docs/9.4/static/sql-copy.html>`_ that will hammer data into the database with one quick query.

This package tries to make using COPY as easy any other database routine supported by Django. It is largely based on the design of the `LayerMapping <https://docs.djangoproject.com/en/1.8/ref/contrib/gis/layermapping/>`_ utility for importing geospatial data.

.. code-block:: python

    from myapp.models import MyModel
    from postgres_copy import CopyMapping

    c = CopyMapping(
        MyModel,
        "./data.csv",
        dict(name='NAME', number='NUMBER')
    )
    c.save()


Installation
------------

The package can be installed from the Python Package Index with `pip`.

.. code-block:: bash

    $ pip install django-postgres-copy

You will of course have to have Django, PostgreSQL and an adapter between the two (like psycopg2) already installed to put this library to use.

An example
----------
<commit_msg>Rewrite the top of the docs<commit_after>====================
django-postgres-copy
====================

Quickly move comma-delimited data in and out of a Django model using PostgreSQL's COPY command.


Why and what for?
-----------------

`The people <http://www.californiacivicdata.org/about/>`_ who made this library are data journalists. We are often downloading, cleaning and analyzing new data.

That means we write a load of loaders. In the past we did this by looping through each row and saving it to the database using the Django's ORM `create method <https://docs.djangoproject.com/en/dev/ref/models/querysets/#django.db.models.query.QuerySet.create>`_.

.. code-block:: python

    import csv
    from myapp.models import MyModel

    data = csv.DictReader(open("./data.csv"))
    for row in data:
        MyModel.objects.create(name=row['NAME'], number=row['NUMBER'])

That works, but if you have a big file as Django racks up a database query for each row it can take a long time to get all the data in the database.

Lucky for us, PostgreSQL has a built-in tool called `COPY <http://www.postgresql.org/docs/9.4/static/sql-copy.html>`_ that can hammer data in and out the database with one quick query.

This package tries to make using COPY as easy any other database routine supported by Django. It is implemented by a custom `model manager<https://docs.djangoproject.com/en/dev/topics/db/managers/>`_.

Here's how it imports a CSV to a database table.

.. code-block:: python

    from myapp.models import MyModel

    MyModel.objects.from_csv(
        "./data.csv",
        dict(name='NAME', number='NUMBER')
    )

And here's how it exports a database table to a CSV.

.. code-block:: python

    from myapp.models import MyModel

    MyModel.objects.to_csv("./data.csv")


Installation
------------

The package can be installed from the Python Package Index with `pip`.

.. code-block:: bash

    $ pip install django-postgres-copy

You will of course have to have Django, PostgreSQL and an adapter between the two (like psycopg2) already installed to put this library to use.

An example
----------

{'fraction_non_alphanumeric': 0.12194525904203324, 'fraction_numerical': 0.002688172043010753, 'mean_word_length': 3.3496280552603612, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '54997', 'n_tokens_mistral': 1208, 'n_tokens_neox': 1140, 'n_words': 482}