<commit_before># ImportLite: Import CSV files into SQLite databases

ImportLite is a command line utility for importing CSV files into SQLite. It can both create tables from scratch and load data into existing tables. It also offers advanced functionality, including value pre-processing and data normalization.

<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
##Table of Contents

- [ImportLite: Import CSV files into SQLite databases](#importlite-import-csv-files-into-sqlite-databases)
  - [Installation & Notes](#installation-&-notes)
  - [Prereq: Set Up Your Database](#prereq-set-up-your-database)
  - [Simple Usage](#simple-usage)
    - [Create a Table and Import Data](#create-a-table-and-import-data)
    - [Import Data to an Existing Table](#import-data-to-an-existing-table)
  - [Advanced Usage: Using Schema Files](#advanced-usage-using-schema-files)
    - [Run ImportLite with a Schema File](#run-importlite-with-a-schema-file)
    - [Specify Column Datatypes](#specify-column-datatypes)
    - [Define Additional Columns](#define-additional-columns)
    - [Change Column Names](#change-column-names)
    - [Ignore Duplicate Rows](#ignore-duplicate-rows)
    - [Pre-process Incoming Values](#pre-process-incoming-values)
    - [Normalize Incoming Data](#normalize-incoming-data)
- [Reference](#reference)
  - [Command-Line Arguments](#command-line-arguments)
  - [Schema Files](#schema-files)
    - [Columns](#columns)
    - [Tables](#tables)
    - [Foreign Keys](#foreign-keys)
- [Tests](#tests)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->


## Installation & Notes

* Requires Python 2.7 or 3.2+
* Tested with SQLite v. 3.7
* Compatible with UTF-8 encoded files.


Install from the command line with pip, Python's package manager:

    $ pip install importlite


If you don't have pip, you can also install from source:

    $ git clone https://github.com/KatrinaE/importlite importlite
    $ cd importlite
    $ python setup.py install

After installation, run ImportLite from the command line. To see options, use the `--help` switch.

    $ importlite --help<commit_msg>Add --pre flag to installation instructions<commit_after># ImportLite: Import CSV files into SQLite databases

ImportLite is a command line utility for importing CSV files into SQLite. It can both create tables from scratch and load data into existing tables. It also offers advanced functionality, including value pre-processing and data normalization.

<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
##Table of Contents

- [ImportLite: Import CSV files into SQLite databases](#importlite-import-csv-files-into-sqlite-databases)
  - [Installation & Notes](#installation-&-notes)
  - [Prereq: Set Up Your Database](#prereq-set-up-your-database)
  - [Simple Usage](#simple-usage)
    - [Create a Table and Import Data](#create-a-table-and-import-data)
    - [Import Data to an Existing Table](#import-data-to-an-existing-table)
  - [Advanced Usage: Using Schema Files](#advanced-usage-using-schema-files)
    - [Run ImportLite with a Schema File](#run-importlite-with-a-schema-file)
    - [Specify Column Datatypes](#specify-column-datatypes)
    - [Define Additional Columns](#define-additional-columns)
    - [Change Column Names](#change-column-names)
    - [Ignore Duplicate Rows](#ignore-duplicate-rows)
    - [Pre-process Incoming Values](#pre-process-incoming-values)
    - [Normalize Incoming Data](#normalize-incoming-data)
- [Reference](#reference)
  - [Command-Line Arguments](#command-line-arguments)
  - [Schema Files](#schema-files)
    - [Columns](#columns)
    - [Tables](#tables)
    - [Foreign Keys](#foreign-keys)
- [Tests](#tests)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->


## Installation & Notes

* Requires Python 2.7 or 3.2+
* Tested with SQLite v. 3.7
* Compatible with UTF-8 encoded files.


Install from the command line with pip, Python's package manager. The `--pre` flag is necessary because importlite is still at pre-release stage; the latest version is 0.1.0.

    $ pip install --pre importlite


If you don't have pip, you can also install from source:

    $ git clone https://github.com/KatrinaE/importlite importlite
    $ cd importlite
    $ python setup.py install

After installation, run ImportLite from the command line. To see options, use the `--help` switch.

    $ importlite --help
{'fraction_non_alphanumeric': 0.11561071195297191, 'fraction_numerical': 0.003701284563466144, 'mean_word_length': 3.882040382571732, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5743932', 'n_tokens_mistral': 1420, 'n_tokens_neox': 1291, 'n_words': 450}