<commit_before># Data Pusher

[![Build Status](https://travis-ci.org/okfn/datapusher.png)](https://travis-ci.org/okfn/datapusher)

__WORK IN PROGRESS__ - Expect release in mid 2013.

A service that migrates data to the CKAN datastore. Built on the [CKAN Service Provider](https://github.com/okfn/ckan-service-provider) and [Data Converters](https://github.com/okfn/dataconverters).

## API

Post the following data to `/job`

```json
{
    "apikey": "my-secret-key",
    "job_type": "push_to_datastore",
    "result_url": "https://www.ckan.org/datapusher/callback",
    "metadata': {
        "ckan_url": "http://www.ckan.org/",
        "resource_id": "3b2987d2-e0e8-413c-92f0-7f9bfe148adc"
    }
}
```

## Deployment

The datapusher is a standard flask app so you can choose your preferred [way of deployment](http://flask.pocoo.org/docs/deploying/). in the following we will explain a set up with postgres, nginx and gunicorn.

This is work in progress!<commit_msg>Link to service provider docs<commit_after># Data Pusher

[![Build Status](https://travis-ci.org/okfn/datapusher.png)](https://travis-ci.org/okfn/datapusher)

__WORK IN PROGRESS__ - Expect release in mid 2013.

A service that migrates data to the CKAN datastore. Built on the [CKAN Service Provider](https://github.com/okfn/ckan-service-provider) and [Data Converters](https://github.com/okfn/dataconverters).

## API

Post the following data to `/job`

```json
{
    "apikey": "my-secret-key",
    "job_type": "push_to_datastore",
    "result_url": "https://www.ckan.org/datapusher/callback",
    "metadata': {
        "ckan_url": "http://www.ckan.org/",
        "resource_id": "3b2987d2-e0e8-413c-92f0-7f9bfe148adc"
    }
}
```

Read more about the API at http://ckan-service-provider.readthedocs.org/en/latest/

## Deployment

The datapusher is a standard flask app so you can choose your preferred [way of deployment](http://flask.pocoo.org/docs/deploying/). in the following we will explain a set up with postgres, nginx and gunicorn.

This is work in progress!
{'fraction_non_alphanumeric': 0.140748031496063, 'fraction_numerical': 0.022637795275590553, 'mean_word_length': 3.9585365853658536, 'pattern_counts': {'":': 10, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 10, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6754497', 'n_tokens_mistral': 761, 'n_tokens_neox': 732, 'n_words': 182}