<commit_before># csv-analyze
Perform analytics on CSV files in order to better understand the kinds of data they represent

## Purpose
There is a treasure-trove of information currently available on the internet that is just
waiting for someone to come along and do something useful with it. Frequently this data will
find its way into a CSV format, but it may not have been properly sanitized for general use.
This program seeks to:

  * Perform analyses of CSV files to find non-uniformities in the contained data
  * Provide guidance on the kind of data in each column, in order to guide future
    reserialization of the contained information

### Checks

  - [ ] Missing columns
  - [ ] Empty cells
  - More TBD

### Data Types

  - [x] Integer
    - [x] uint8
    - [x] uint16
    - [x] uint32
    - [x] uint64
    - [x] int8
    - [x] int16
    - [x] int32
    - [x] int64
  - [x] Floating Point
    - [x] float32
    - [x] float64
  - [ ] Timestamps
    - Formats TBD, but probably everything in the `time` package plus an Apache log format
  - [ ] Boolean
    - [ ] Y/N
    - [ ] yes/no
    - [ ] true/false
    - [ ] t/f
    - [ ] 1/0
  - [ ] Strings
    - [ ] Alpha
    - [ ] Alphanumeric
    - [ ] Extended

### License
Copyright 2017 <NAME> <<EMAIL>>

Licensed under the Apache License, Version 2.0 (the "License");<commit_msg>Now supports timestamps<commit_after># csv-analyze
Perform analytics on CSV files in order to better understand the kinds of data they represent

## Purpose
There is a treasure-trove of information currently available on the internet that is just
waiting for someone to come along and do something useful with it. Frequently this data will
find its way into a CSV format, but it may not have been properly sanitized for general use.
This program seeks to:

  * Perform analyses of CSV files to find non-uniformities in the contained data
  * Provide guidance on the kind of data in each column, in order to guide future
    reserialization of the contained information

### Checks

  - [ ] Missing columns
  - [ ] Empty cells
  - More TBD

### Data Types

  - [x] Integer
    - [x] uint8
    - [x] uint16
    - [x] uint32
    - [x] uint64
    - [x] int8
    - [x] int16
    - [x] int32
    - [x] int64
  - [x] Floating Point
    - [x] float32
    - [x] float64
  - [x] Timestamps
    - Everything in the `time` package plus an Apache log format
  - [ ] Boolean
    - [ ] Y/N
    - [ ] yes/no
    - [ ] true/false
    - [ ] t/f
    - [ ] 1/0
  - [ ] Strings
    - [ ] Alpha
    - [ ] Alphanumeric
    - [ ] Extended

### License
Copyright 2017 <NAME> <<EMAIL>>

Licensed under the Apache License, Version 2.0 (the "License");
{'fraction_non_alphanumeric': 0.09196696696696696, 'fraction_numerical': 0.01951951951951952, 'mean_word_length': 2.515831134564644, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6972473', 'n_tokens_mistral': 879, 'n_tokens_neox': 820, 'n_words': 350}