<commit_before>CDV - Community Data Validator
==============================


About
-----


CDV adds the ability of creating validation tests on the Pentaho BA server for the
purpose of verifying both the integrity of the server itself and also the data being
used by the server.


Motivation
----------


Very often we want that several questions related to the data would be answered before we noticed it: 

* Do we have up to date data?
* Can we trust the data?
* How long did the queries take to run?
* Do we have wrong data? (duplicated users in community)
* Do we have a big number of 'unknowns'?(tk=1 in DW)
* Do we have peaks or valleys in the data? (due to double process or no process)
* Is the data stalled? (eg: number of twitter followers not updating)
* Did the data format change?
* We need a way to handleknow effects (eg: Christmas dip)
* We need to correlate independent datasources
* Be able to validade big chnks of reprocessing
* Do we have clearly wrong rows in resultset? (eg: a `<null>` line there)
* etc

So we decide to build CDV - a data validator that periodically do a set of tests that answer all the above questions.

You can see the [Original RFC for CDV](http://pedroalves-bi.blogspot.com/2012/05/cdv-request-for-comments.html). Note
that not every feature has been implemented


User interface
--------------<commit_msg>- Fixed typos in docs<commit_after>CDV - Community Data Validator
==============================


About
-----


CDV adds the ability of creating validation tests on the Pentaho BA server for the
purpose of verifying both the integrity of the server itself and also the data being
used by the server.


Motivation
----------


Very often we want that several questions related to the data would be answered before we noticed it: 

* Do we have up to date data?
* Can we trust the data?
* How long did the queries take to run?
* Do we have wrong data? (duplicated users in community)
* Do we have a big number of 'unknowns'?(tk=1 in DW)
* Do we have peaks or valleys in the data? (due to double process or no process)
* Is the data stalled? (eg: number of twitter followers not updating)
* Did the data format change?
* We need a way to handle know effects (eg: Christmas dip)
* We need to correlate independent datasources
* Be able to validade big chunks of reprocessing
* Do we have clearly wrong rows in resultset? (eg: a `<null>` line there)
* etc

So we decide to build CDV - a data validator that periodically do a set of tests that answer all the above questions.

You can see the [Original RFC for CDV](http://pedroalves-bi.blogspot.com/2012/05/cdv-request-for-comments.html). Note
that not every feature has been implemented


User interface
--------------
{'fraction_non_alphanumeric': 0.09522058823529411, 'fraction_numerical': 0.005147058823529412, 'mean_word_length': 3.7157712305026, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6753038', 'n_tokens_mistral': 760, 'n_tokens_neox': 702, 'n_words': 419}