<commit_before># OmniDataHouse

Utilities for OMNILab data warehouse.<commit_msg>Add readme<commit_after># OmniDataHouse

Utilities for OMNILab data warehouse.


## Architecture

OMNILab data warehouse is designed with three layers:

* Layer0: Original raw data from multiple sources.

* Layer1: Independent wide tables for each source after simple ETLing.

* Layer2: A bunch of small tables (models) after combining multiple data at Layer1 to fit different applications.

In most scenarios, data administrators hold the access right to Layer0 and Layer1, and data users have accesses to
the small tables at Layer2 to meet their requirements.

The data users can also contribute new data models to Layer2 when they develop a new type of table from application. In this process, other
data sources may be involved to generate the new model. At this time, the user should contact admin to add new data sources to Layer0 or
Layer1.


## Utilities

To add...
{'fraction_non_alphanumeric': 0.03878406708595388, 'fraction_numerical': 0.010482180293501049, 'mean_word_length': 4.617647058823529, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7671524', 'n_tokens_mistral': 254, 'n_tokens_neox': 237, 'n_words': 140}