<commit_before>**Django Data Wizard** is an interactive tool for mapping structured data (e.g. Excel, XML) into a normalized database structure via [wq.io] and the [Django REST Framework].  Django Data Wizard allows novice users to map spreadsheet columns to serializer fields (and cell values to foreign keys) on-the-fly during the import process.  This reduces the need for preset spreadsheet formats, which most data import solutions require.

<img width="33%"
     alt="Column Choices"
     src="https://raw.githubusercontent.com/wq/django-data-wizard/master/images/02-columns.png">
<img width="33%"
     alt="Auto Import - Progress Bar"
     src="https://raw.githubusercontent.com/wq/django-data-wizard/master/images/08-data75.png">
<img width="33%"
     alt="Imported Records"
     src="https://raw.githubusercontent.com/wq/django-data-wizard/master/images/10-records.png">

By default, Django Data Wizard supports any format supported by [wq.io] (Excel, CSV, JSON, and XML).  Additional formats can be integrating by creating a [custom wq.io class] and then registering it with the wizard.  For example, the [Climata Viewer] uses Django Data Wizard to import data from [climata]'s wq.io-based web service client.

The Data Wizard supports straightforward one-to-one mappings from spreadsheet columns to database fields, as well as more complex scenarios like [natural keys] and [Entity-Attribute-Value] (or "wide") table mappings.  It was originally developed for use with the [ERAV data model][ERAV] provided by [vera].

[![Latest PyPI Release](https://img.shields.io/pypi/v/data-wizard.svg)](https://pypi.python.org/pypi/data-wizard)
[![Release Notes](https://img.shields.io/github/release/wq/django-data-wizard.svg)](https://github.com/wq/django-data-wizard/releases)
[![License](https://img.shields.io/pypi/l/data-wizard.svg)](https://wq.io/license)
[![GitHub Stars](https://img.shields.io/github/stars/wq/django-data-wizard.svg)](https://github.com/wq/django-data-wizard/stargazers)
[![GitHub Forks](https://img.shields.io/github/forks/wq/django-data-wizard.svg)](https://github.com/wq/django-data-wizard/network)
[![GitHub Issues](https://img.shields.io/github/issues/wq/django-data-wizard.svg)](https://github.com/wq/django-data-wizard/issues)

[![Travis Build Status](https://img.shields.io/travis/wq/django-data-wizard.svg)](https://travis-ci.org/wq/django-data-wizard)
[![Python Support](https://img.shields.io/pypi/pyversions/data-wizard.svg)](https://pypi.python.org/pypi/data-wizard)
[![Django Support](https://img.shields.io/badge/Django-1.8%2C%201.9%2C%201.10%2C%201.11-blue.svg)](https://pypi.python.org/pypi/data-wizard)

# Usage

Django Data Wizard provides a JSON (and HTML) API for specifying a data set to import, selecting a serializer, mapping the data columns and identifiers, and (asynchronously) importing the data into the database.

## Installation

```bash
# Recommended: create virtual environment
# python3 -m venv venv
# . venv/bin/activate

pip install data-wizard
```

See <https://github.com/wq/django-data-wizard> to report any issues.

## Celery

Django Data Wizard requires [Celery] to handle asynchronous tasks, and is usually used with [Redis] as the memory store.  These should be configured first or the REST API may not work.  Once Redis is installed, you should be able to add the following to your project settings:
```python
# myproject/settings.py
CELERY_RESULT_BACKEND = BROKER_URL = 'redis://localhost:6379/1'
```

Then, define a celery app:
```python
# myproject/celery.py
from __future__ import absolute_import
from celery import Celery
from django.conf import settings
app = Celery('myproject')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
```

And reference it from your project:

```python
# myproject/__init__.py
from .celery import app as celery_app
```

Finally, make sure that celery is running in the background.  You can test with the following command, then move to a more stable configuration (e.g. [daemonization]).

```bash
# celery.sh
export DJANGO_SETTINGS_MODULE=myproject.settings
celery -A myproject worker -l info
```

Note that any time you change your serializer registration, you should reload celery in addition to restarting the Django WSGI  instance.

## Serializer Registration

As of version 1.0, Data Wizard relies on serializer classes as the primary means of detecting model fields.  These are subclasses of Django REST Framework's [ModelSerializer class].  You can register serializers by creating a `wizard.py` in your app directory (analagous to Django's `admin.py` and wq.db's `rest.py`).

```python
# myapp/wizard.py
from rest_framework import serializers
from data_wizard import registry
from .models import TimeSeries

class TimeSeriesSerializer(serializers.ModelSerializer):
    class Meta:
        model = TimeSeries
        fields = '__all__'

registry.register("Time Series", TimeSeriesSerializer)
```

At least one serializer should be registered in order to use the wizard.  Note the use of a human-friendly serializer label when registering.  This name should be unique throughout the project, but can be changed later on without breaking existing data.  (The class path is used as the actual identifier behind the scenes.)

## REST API

The Data Wizard REST API provides the following capabilities.  If you are using wq.db, the wizard will automatically register itself with the router.  Otherwise, be sure to include `data_wizard.urls` in your URL configuration:

```python<commit_msg>loader notes<commit_after>**Django Data Wizard** is an interactive tool for mapping structured data (e.g. Excel, XML) into a normalized database structure via [wq.io] and the [Django REST Framework].  Django Data Wizard allows novice users to map spreadsheet columns to serializer fields (and cell values to foreign keys) on-the-fly during the import process.  This reduces the need for preset spreadsheet formats, which most data import solutions require.

<img width="33%"
     alt="Column Choices"
     src="https://raw.githubusercontent.com/wq/django-data-wizard/master/images/02-columns.png">
<img width="33%"
     alt="Auto Import - Progress Bar"
     src="https://raw.githubusercontent.com/wq/django-data-wizard/master/images/08-data75.png">
<img width="33%"
     alt="Imported Records"
     src="https://raw.githubusercontent.com/wq/django-data-wizard/master/images/10-records.png">

By default, Django Data Wizard supports any format supported by [wq.io] (Excel, CSV, JSON, and XML).  Additional formats can be integrating by creating a [custom wq.io class] and then registering it with the wizard.  For example, the [Climata Viewer] uses Django Data Wizard to import data from [climata]'s wq.io-based web service client.

The Data Wizard supports straightforward one-to-one mappings from spreadsheet columns to database fields, as well as more complex scenarios like [natural keys] and [Entity-Attribute-Value] (or "wide") table mappings.  It was originally developed for use with the [ERAV data model][ERAV] provided by [vera].

[![Latest PyPI Release](https://img.shields.io/pypi/v/data-wizard.svg)](https://pypi.python.org/pypi/data-wizard)
[![Release Notes](https://img.shields.io/github/release/wq/django-data-wizard.svg)](https://github.com/wq/django-data-wizard/releases)
[![License](https://img.shields.io/pypi/l/data-wizard.svg)](https://wq.io/license)
[![GitHub Stars](https://img.shields.io/github/stars/wq/django-data-wizard.svg)](https://github.com/wq/django-data-wizard/stargazers)
[![GitHub Forks](https://img.shields.io/github/forks/wq/django-data-wizard.svg)](https://github.com/wq/django-data-wizard/network)
[![GitHub Issues](https://img.shields.io/github/issues/wq/django-data-wizard.svg)](https://github.com/wq/django-data-wizard/issues)

[![Travis Build Status](https://img.shields.io/travis/wq/django-data-wizard.svg)](https://travis-ci.org/wq/django-data-wizard)
[![Python Support](https://img.shields.io/pypi/pyversions/data-wizard.svg)](https://pypi.python.org/pypi/data-wizard)
[![Django Support](https://img.shields.io/badge/Django-1.8%2C%201.9%2C%201.10%2C%201.11-blue.svg)](https://pypi.python.org/pypi/data-wizard)

# Usage

Django Data Wizard provides a JSON (and HTML) API for specifying a data set to import (by referencing a previously-uploaded file), selecting a serializer, mapping the data columns and identifiers, and (asynchronously) importing the data into the database.

## Installation

```bash
# Recommended: create virtual environment
# python3 -m venv venv
# . venv/bin/activate

pip install data-wizard
```

See <https://github.com/wq/django-data-wizard> to report any issues.

## Initial Configuration

### Celery

Django Data Wizard requires [Celery] to handle asynchronous tasks, and is usually used with [Redis] as the memory store.  These should be configured first or the REST API may not work.  Once Redis is installed, you should be able to add the following to your project settings:
```python
# myproject/settings.py
CELERY_RESULT_BACKEND = BROKER_URL = 'redis://localhost:6379/1'
```

Then, define a celery app:
```python
# myproject/celery.py
from __future__ import absolute_import
from celery import Celery
from django.conf import settings
app = Celery('myproject')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
```

And reference it from your project:

```python
# myproject/__init__.py
from .celery import app as celery_app
```

Finally, make sure that celery is running in the background.  You can test with the following command, then move to a more stable configuration (e.g. [daemonization]).

```bash
# celery.sh
export DJANGO_SETTINGS_MODULE=myproject.settings
celery -A myproject worker -l info
```

Note that any time you change your serializer registration, you should reload celery in addition to restarting the Django WSGI  instance.

### Data Loader

To use Data Wizard with the default configuration, your project should provide a Django model with a `FileField` named `file`.  You will want to provide your users with the ability to upload spreadsheets and other files to this model, for example by registering it with [wq.db].  The actual import process can then be triggered after the file is uploaded, for example by adding a form to the detail page ([see below](#new-run)).

```python
# myapp/models.py
from django.db import models

class FileModel(models.Model):
    file = models.FileField(upload_to='spreadsheets')
    # ...
```

```python
# myapp/rest.py
from wq.db import rest
from .models import FileModel

rest.router.register_model(
    FileModel,
    fields="__all__",
)
```

#### Custom FileField Name
You can use a model with a different `FileField` name by extending `data_wizard.loaders.FileLoader` and setting `DATA_WIZARD_LOADER`:

```python
# myapp/models.py
from django.db import models

class FileModel(models.Model):
    spreadsheet = models.FileField(upload_to='spreadsheets')
```

```python
# myapp/loaders.py
from data_wizard import loaders
class FileLoader(loaders.FileLoader):
    file_attr = 'spreadsheet'
```

```python
# myapp/settings.py
DATA_WIZARD_LOADER = 'myapp.loaders.FileLoader'
```

#### Custom Data Source
You can also customize the loader to load data from a [custom wq.io class].  For example, the [Climata Viewer] uses [climata] classes to load data directly from third-party webservices.  To do this, extend `data_wizard.loaders.BaseLoader` with a custom `load_io()` function that returns the data from wq.io:

```python
# myapp/models.py
class WebSource(models.Model):
    url = models.URLField()
```

```myapp/loaders.py
from data_wizard import loaders

class UrlLoader(loaders.BaseLoader):
    def load_io(self):
        source = self.run.content_object
        from wq.io import load_url # or e.g. JsonNetIO
        return load_url(source.url)
```

```python
# myapp/settings.py
DATA_WIZARD_LOADER = 'myapp.loaders.UrlLoader'
```

Note that there still should be a custom model to define the parameters for the IO class, even though there is no file being stored.

### Serializer Registration

Data Wizard uses wq.io classes (via the loader) to determine the *source columns* present on the spreadsheet or other data source.  It uses Django REST Framework's [ModelSerializer class] to determine the *destination fields* on the database model.  You can register serializers by creating a `wizard.py` in your app directory (analogous to Django's `admin.py` and wq.db's `rest.py`).  Multiple serializers can be registered with the wizard to support multiple import configurations and destination models.  

```python
# myapp/wizard.py
from rest_framework import serializers
from data_wizard import registry
from .models import TimeSeries

class TimeSeriesSerializer(serializers.ModelSerializer):
    class Meta:
        model = TimeSeries
        fields = '__all__'

registry.register("Time Series", TimeSeriesSerializer)
```

At least one serializer should be registered in order to use the wizard.  Note the use of a human-friendly serializer label when registering.  This name should be unique throughout the project, but can be changed later on without breaking existing data.  (The class path is used as the actual identifier behind the scenes.)

## Run-Time Usage (REST API)

The Data Wizard REST API provides the following capabilities.  If you are using wq.db, the wizard will automatically register itself with the router.  Otherwise, be sure to include `data_wizard.urls` in your URL configuration:

```python
{'fraction_non_alphanumeric': 0.10763555883437846, 'fraction_numerical': 0.005901881224640354, 'mean_word_length': 4.722245673279865, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 44, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1102455', 'n_tokens_mistral': 4240, 'n_tokens_neox': 3986, 'n_words': 1393}