<commit_before>![datadez.png](./docs/datadez.png)

### Pandas dataframe inspection, filtering, balancing

The main goal of this package is to make your life easier if you want to:

- Inspect a dataset and compute metrics about its columns content (auto type inference: numeric, mono-label or multi-label).
- Filter the dataset one some criteria (minimum label occurrence, empty example).
- Balance the dataset (TODO) in order to get better performance while training ML or NN models.

### Requirements

- Python 2.7
- Numpy and Pandas

### Do some tests

Just clone this repository, and execute:

    python -m tests.main
    
This will execute a test sample, for you to get what's going on.<commit_msg>[Test] Travis status in README.md<commit_after>![datadez.png](./docs/datadez.png)

### Pandas dataframe inspection, filtering, balancing [![Build Status][travis-badge]][travis-link] [![MIT License][license-badge]](LICENSE)

The main goal of this package is to make your life easier if you want to:

- Inspect a dataset and compute metrics about its columns content (auto type inference: numeric, mono-label or multi-label).
- Filter the dataset one some criteria (minimum label occurrence, empty example).
- Balance the dataset (TODO) in order to get better performance while training ML or NN models.

### Requirements

- Python 2.7
- Numpy and Pandas

### Do some tests

Just clone this repository, and execute:

    python -m tests.main
    
This will execute a test sample, for you to get what's going on.

[travis-badge]:    https://travis-ci.org/dezounet/datadez.svg?branch=master
[travis-link]:     https://travis-ci.org/dezounet/datadez
[license-badge]:   https://img.shields.io/badge/license-MIT-007EC7.svg
{'fraction_non_alphanumeric': 0.09720605355064028, 'fraction_numerical': 0.004656577415599534, 'mean_word_length': 4.1467065868263475, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6358061', 'n_tokens_mistral': 524, 'n_tokens_neox': 496, 'n_words': 207}