<commit_before># mixport [![Build Status](https://travis-ci.org/erik/mixport.png?branch=master)](https://travis-ci.org/erik/mixport)

`mixport` utilizes Mixpanel's
[data export API](https://mixpanel.com/docs/api-documentation/exporting-raw-data-you-inserted-into-mixpanel)
to download event data and make it far easier to consume and store.

Mixpanel exposes this data as a bunch of JSON blobs in the form `{"event":
"EventName", "properties": {...}}`, where the `properties` map contains all of
the arbitrary dimensions attached to the data point. There isn't a defined
schema, property keys and event types can come and go, data types can
change. Such is JSON.

This does however make storing the data in a SQL database somewhat of a chore,
since you can really only try to guess the valid types (I wrote and maintained
this application as the precursor to `mixport`, believe me, this isn't fun or
very logical).

`mixport` is really only concerned with the export portion of the problem, but
can do some streaming transforms on the data to make it more digestible.

## Building

*Requires Go >= 1.1 to compile.*

Using `go get`:

```bash
$ go get github.com/erik/mixport
```

## Usage

First, let's set up a configuration file:

```bash
$ cp mixport.conf{.example,}
$ $EDITOR mixport.conf
```

There are several comments in the example configuration which should give a
decent explanation of what's required and available to configure.

Let's start off by just downloading yesterday's data for all the products
specified in the configuration:

```bash
$ ./mixport
```

That's it! If your configuration isn't in `.` or is named something other than
`mixport.conf`, you need to specify it with `-c path/to/config`.

If you want to just export a specific subset of the products defined in the
configuration, the `--products` flags will be useful:<commit_msg>Update README to reflect Go 1.3+ dependency<commit_after># mixport [![Build Status](https://travis-ci.org/erik/mixport.png?branch=master)](https://travis-ci.org/erik/mixport)

`mixport` utilizes Mixpanel's
[data export API](https://mixpanel.com/docs/api-documentation/exporting-raw-data-you-inserted-into-mixpanel)
to download event data and make it far easier to consume and store.

Mixpanel exposes this data as a bunch of JSON blobs in the form `{"event":
"EventName", "properties": {...}}`, where the `properties` map contains all of
the arbitrary dimensions attached to the data point. There isn't a defined
schema, property keys and event types can come and go, data types can
change. Such is JSON.

This does however make storing the data in a SQL database somewhat of a chore,
since you can really only try to guess the valid types (I wrote and maintained
this application as the precursor to `mixport`, believe me, this isn't fun or
very logical).

`mixport` is really only concerned with the export portion of the problem, but
can do some streaming transforms on the data to make it more digestible.

## Building

*Requires Go >= 1.3 to compile.*

Using `go get`:

```bash
$ go get github.com/erik/mixport
```

## Usage

First, let's set up a configuration file:

```bash
$ cp mixport.conf{.example,}
$ $EDITOR mixport.conf
```

There are several comments in the example configuration which should give a
decent explanation of what's required and available to configure.

Let's start off by just downloading yesterday's data for all the products
specified in the configuration:

```bash
$ ./mixport
```

That's it! If your configuration isn't in `.` or is named something other than
`mixport.conf`, you need to specify it with `-c path/to/config`.

If you want to just export a specific subset of the products defined in the
configuration, the `--products` flags will be useful:
{'fraction_non_alphanumeric': 0.09076348104644955, 'fraction_numerical': 0.001601708489054992, 'mean_word_length': 4.510294117647059, 'pattern_counts': {'":': 4, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '584268', 'n_tokens_mistral': 1086, 'n_tokens_neox': 1020, 'n_words': 515}