<commit_before>Implementation Notes
====================

Implementation Steps
--------------------

* aggregations following each dimension
* dimension auto-detect by variation of column values
* dimension auto-detect by typical data, e.g. timestamp, geo-coord, city/county-name? (need dictionary)

* start with a single table
  * [done] keep columns with meta data, from load/format
  * [done] keep content of columns
  * derive meta data from content (date or id etc)
  * know some facts about columns (number of unique values), which is sort of statistical data
  * find columns that always change together e.g. "zip and city" or "city and country"
  * predict fact or dimension and which dimension - tbd
  * sort columns of dimensions
  * [done] aggregate fact/HBase primary column name idea, use <table>_<smallest dimension>_<next> and so on.

* have a table with a foreign key
  * [done] expand columns of foreign table into table
  * [done] mark new columns as meta data from original table

* table with more (simple) foreign keys
  * recursively

* m2n
  * tbd

* how to export the de-normalised data?
  This is the goal. needs to be pre-aggregated and sorted by different aggregators counts descending
  if any groups are found, then use each group


Design
------

A model of the tables and their relations is created during load.
The model contains classes like `Table`, `Column` and so on. Often there is extra
data or meta-data for the classes as well, e.g. `TableData` or `ColumnMetaData`. In case of
(simple) analytical functions there is an extra class for that, e.g. `ColumnAnalytics`.
After the model has been loaded, the references are defined and the chosen table is
exploded/flattened into a single table.


Sakila
------

To start exploring de-normalising relations I need a simple schema with a few relations.
The schema needs to have some data and should be freely accessible. The Sakila example
database was originally developed for MySQL as part of the documentation.
See https://code.google.com/p/sakila-sample-database-ports/

* Simple Table: `County` only 1 data column.
* Foreign Keys: `Address` -> `City` -> `Country`
* m2n: `film_category`

Resources
---------

* http://hstack.org/hbasecon-low-latency-olap-with-hbase/
* http://msdn.microsoft.com/en-us/library/cc505841.aspx<commit_msg>markdown formatting<commit_after>Implementation Notes
====================

Implementation Steps
--------------------

* aggregations following each dimension
* dimension auto-detect by variation of column values
* dimension auto-detect by typical data, e.g. time-stamp, geo-coordinates, city/county name? (need dictionary)

* start with a single table
  * [done] keep columns with meta data, from load/format
  * [done] keep content of columns
  * derive meta data from content (date or id etc)
  * know some facts about columns (number of unique values), which is sort of statistical data
  * find columns that always change together e.g. "zip and city" or "city and country"
  * predict fact or dimension and which dimension - tbd
  * sort columns of dimensions
  * [done] aggregate fact/HBase primary column name idea, use <table>_<smallest dimension>_<next> and so on.


* have a table with a foreign key
  * [done] expand columns of foreign table into table
  * [done] mark new columns as meta data from original table


* table with more (simple) foreign keys
  * recursively


* m2n
  * tbd


* how to export the de-normalised data?
  This is the goal. needs to be pre-aggregated and sorted by different aggregators counts descending
  if any groups are found, then use each group


Design
------

A model of the tables and their relations is created during load.
The model contains classes like `Table`, `Column` and so on. Often there is extra
data or meta-data for the classes as well, e.g. `TableData` or `ColumnMetaData`. In case of
(simple) analytical functions there is an extra class for that, e.g. `ColumnAnalytics`.
After the model has been loaded, the references are defined and the chosen table is
exploded/flattened into a single table.


Sakila
------

To start exploring de-normalising relations I need a simple schema with a few relations.
The schema needs to have some data and should be freely accessible. The Sakila example
database was originally developed for MySQL as part of the documentation.
See https://code.google.com/p/sakila-sample-database-ports/

* Simple Table: `County` only 1 data column.
* Foreign Keys: `Address` -> `City` -> `Country`
* m2n: `film_category`

For testing purpose the SQLite version of Sakila (`sqlite-sakila.sq`) has been added to `src\test\resources``
and is loaded using JDBC in the tests.


Resources
---------

* http://hstack.org/hbasecon-low-latency-olap-with-hbase/
* http://msdn.microsoft.com/en-us/library/cc505841.aspx
{'fraction_non_alphanumeric': 0.09521829521829522, 'fraction_numerical': 0.0037422037422037424, 'mean_word_length': 3.7823061630218686, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 13, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6751306', 'n_tokens_mistral': 1340, 'n_tokens_neox': 1296, 'n_words': 633}