<commit_before># DBTruck

The current database import process is like a toilet pipe.  The pipe
easily gets clogged if your data is a bit dirty.

DBTruck turns the import process into a dump truck.  Just throw data
into it, it'll just work.  You can clean it up later!

It currently only works for postgresql.  It assumes that PostgreSQL
is installed and running on your machine, and uses `psql` to load data.

# Usage

## Help me

    python dbtruck.py -h

## Load datafile.txt

    python dbtruck.py datafile.txt tablename dbname<commit_msg>added readme<commit_after># DBTruck

The current database import process is like a toilet pipe.  The pipe
easily gets clogged if your data is a bit dirty.

DBTruck turns the import process into a dump truck.  Just throw data
into it, it'll just work.  You can clean it up later!

It currently only works for postgresql.  It assumes that PostgreSQL
is installed and running on your machine, and uses `psql` to load data.

# Usage

## Help me

    python dbtruck.py -h

## Load datafile.txt

    python dbtruck.py data/testfile.txt tablename dbname
{'fraction_non_alphanumeric': 0.04871323529411765, 'fraction_numerical': 0.0, 'mean_word_length': 3.6340425531914895, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5023949', 'n_tokens_mistral': 337, 'n_tokens_neox': 320, 'n_words': 166}