<commit_before># pytextql

A python-based clone of the textql tool.

## Usage
    pytextql [--source <source>...] [-s <query>] [options]

## Options
    --source=<path>     The source file(s) to load, or '-' for STDIN. You
                        can specify any number of sources.
    --no-header         The source files do not contain headers in the first
                        row.
    --named-tables      Use the filenames (without extension) of source files
                        as the table name.
    --db=<path>         Store the resulting SQLite3 database at <path>
                        instead of using a temporary file. A good idea if
                        working on large (100mb+) files.
    --delimiter=<,>     The delimiter used by the source file(s).
                        [default: ,]
    --encoding=<utf8>   The file encoding. If you're getting unicode errors
                        on a file exported from excel, try "cp1250".
                        [default: utf8]
    --chunk-size=<n>    The number of rows to insert per SQLite3 transaction.
                        [default: 50000]
    --sql=<q>, -s       Run an SQL query against the database.

## Testing

Tests are run using [nose][].

```
pip install nose
python setup.py nosetests
```

### Sample Data

Sample data is generated using [FNG][] and is available as `tests/sample_*.csv`.

[nose]: http://nose.readthedocs.org
[FNG]: http://www.fakenamegenerator.com/<commit_msg>Add a couple of examples.<commit_after># pytextql

A python-based clone of the textql tool.

## Installing

    pip install pytextql

## Usage

    pytextql [--source <source>...] [-s <query>] [options]

## Options

    --source=<path>     The source file(s) to load, or '-' for STDIN. You
                        can specify any number of sources.
    --no-header         The source files do not contain headers in the first
                        row.
    --named-tables      Use the filenames (without extension) of source files
                        as the table name.
    --db=<path>         Store the resulting SQLite3 database at <path>
                        instead of using a temporary file. A good idea if
                        working on large (100mb+) files.
    --delimiter=<,>     The delimiter used by the source file(s).
                        [default: ,]
    --encoding=<utf8>   The file encoding. If you're getting unicode errors
                        on a file exported from excel, try "cp1250".
                        [default: utf8]
    --chunk-size=<n>    The number of rows to insert per SQLite3 transaction.
                        [default: 50000]
    --sql=<q>, -s       Run an SQL query against the database.

## Examples

Lets play with one of the samples including with the pytextql. By default,
pytextql treats all data as UTF-8, but you can specify the encoding with
`--encoding=` if you have somethinge exotic.

    pytextql --source tests/sample_small_unicode.csv -s "SELECT DISTINCT(Gender), COUNT(Gender) FROM tbl0 GROUP BY Gender"
    city,COUNT(City)
    Halifax,1
    Val Brillant,1
    Vancouver,1
    Zenon Park,1
    ΑΓΙΟΣ ΓΕΩΡΓΙΟΣ ΚΕΛΟΚΕ∆ΑΡΩΝ,1
    ΑΓΙΟΣ ΕΠΙΦΑΝΕΙΟΣ ΣΟΛΕΑΣ,1
    ΛΑΡΝΑΚΑ,1
    ΠΟΜΟΣ,1
    ΤΑΛΑ,1
    ∆ΙΚΩΜΟ ΚΑΤΩ,1

If you're working with extremely large source files (hundreds of thousands to
millions of rows), you can use the `--db=<path>` option to store the results
and use them over and over again.

    pytextql --source my_really_large_file.csv --db=testing.db
    pytextql --db=testing.db -s "SELECT * FROM tbl0 LIMIT 1"
    ...
    pytextql --db=testing.db -s "SELECT COUNT(id) FROM tbl0;"
    ...

## Testing

Tests are run using [nose][].

    pip install nose
    python setup.py nosetests

### Sample Data

Sample data is generated using [FNG][] and is available as `tests/sample_*.csv`.

[nose]: http://nose.readthedocs.org
[FNG]: http://www.fakenamegenerator.com/
{'fraction_non_alphanumeric': 0.09688139059304704, 'fraction_numerical': 0.012014314928425357, 'mean_word_length': 2.0309837335398915, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3557747', 'n_tokens_mistral': 1242, 'n_tokens_neox': 1209, 'n_words': 439}