<commit_before># Crump

A parser for [the Virginia State Corporation Commission's business entity records](https://www.scc.virginia.gov/clk/purch.aspx), which are provided as a single, enormous fixed-width file. Named for <NAME>, the first member of the State Corporation Commission.

[The file layout](https://www.scc.virginia.gov/clk/files/layout_be.pdf) is documented on the SCC's website as a PDF generated from an ASCII file. (Also available as [not-entirely-formatted ASCII](record_layouts.txt).)

[A sample data file](http://s3.amazonaws.com/data.openva.com/corporations/2014-04-30.zip) is available.

## To Do

* [x] Create a tool that will break the file up into its 9 constituent files, which have been rudely concatenated into a single file. This is probably best done in sed or AWK, and can be done based on the length of each line, since each table has, luckily, a unique column count.</del>
* [x] [Create a machine-readable mapping of column positions to column names, by file](http://github.com/openva/crump/issues/4).
* [ ] [Figure out what "tables" are and how those come into play](https://github.com/openva/crump/issues/18). (Those are listed in file 1.) I'm pretty sure that the tables are lookup tables, e.g., to map `WV` to `West Virginia`, among many other less-obvious things.
* [x] Create a tool that will turn each of the 9 data files into CSV and JSON. (The former may well be the output from the AWK recipes.)
* [ ] Script the retrieval of the file (updated weekly, shortly after 1:00 AM on Wednesdays) and storage of it at a stable URL, plus archiving of past versions.
* [ ] Create an importer that will load this data into Elasticsearch.
* [ ] Create a website to search for, retrieve, display records.<commit_msg>To do items: Create JSON and website<commit_after># Crump

A parser for [the Virginia State Corporation Commission's business entity records](https://www.scc.virginia.gov/clk/purch.aspx), which are provided as a single, enormous fixed-width file. Named for <NAME>, the first member of the State Corporation Commission.

[The file layout](https://www.scc.virginia.gov/clk/files/layout_be.pdf) is documented on the SCC's website as a PDF generated from an ASCII file. (Also available as [not-entirely-formatted ASCII](record_layouts.txt).)

[A sample data file](http://s3.amazonaws.com/data.openva.com/corporations/2014-04-30.zip) is available.

## To Do

* [x] Create a tool that will break the file up into its 9 constituent files, which have been rudely concatenated into a single file. This is probably best done in sed or AWK, and can be done based on the length of each line, since each table has, luckily, a unique column count.</del>
* [x] [Create a machine-readable mapping of column positions to column names, by file](http://github.com/openva/crump/issues/4).
* [ ] [Figure out what "tables" are and how those come into play](https://github.com/openva/crump/issues/18). (Those are listed in file 1.) I'm pretty sure that the tables are lookup tables, e.g., to map `WV` to `West Virginia`, among many other less-obvious things.
* [x] Create a tool that will turn each of the 9 data files into CSV. (The former may well be the output from the AWK recipes.)
* [ ] [Turn CSV into JSON](https://github.com/openva/crump/issues/19).
* [ ] Set up a website to house the data file, this tool, and the data that it produces
* [ ] Script the retrieval of the file (updated weekly, shortly after 1:00 AM on Wednesdays) and storage of it at a stable URL, plus archiving of past versions.
* [ ] Create an importer that will load this data into Elasticsearch.
* [ ] Create a website to search for, retrieve, display records.
{'fraction_non_alphanumeric': 0.09445809445809446, 'fraction_numerical': 0.010374010374010374, 'mean_word_length': 4.249283667621777, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 7, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '549798', 'n_tokens_mistral': 1086, 'n_tokens_neox': 1021, 'n_words': 506}