<jupyter_start><jupyter_text>Why Data Quality - Data driven oraganizations/products- Wrong desicion / Bad product- Its all about trust  Data Quality Effect - Is the data correct ?- If so to what time ? ![](dashboard.png) - great to find violations as early as possible- productivity and integrity are the same thing data stuff ![](identify_the_problem.png) ![](pull.png) ![](push.png)  Trying to maintain data systems that are untested, undocumented and unstable is nearly impossible  Data Quality characteristics - **Accuracy**: for whatever data described, it needs to be accurate.- **Relevancy**: the data should meet the requirements for the intended use.- **Completeness**: the data should not have missing values or miss data records.- **Timeliness**: the data should be up to date.- **Consistency**:the data should have the data format as expected and can be cross reference-able with the same results.  Data Quality tools - [Great Expectations](https://github.com/great-expectations/great_expectations)- [Deequ](https://github.com/awslabs/deequ)- [Tensorflow Data Validation](https://www.tensorflow.org/tfx/data_validation/get_started)  Great Expectations Native<jupyter_code>import pandas as pd
import great_expectations as ge
import great_expectations.jupyter_ux
import json

from datetime import datetime
from sklearn.model_selection import train_test_split
df = ge.read_csv("data/titanic.csv")
train, test = train_test_split(df, test_size=0.3)<jupyter_output><empty_output><jupyter_text>expectations are assertions about data<jupyter_code>train.expect_column_values_to_be_between("Age", 0,80)
train.expect_column_values_to_be_in_set('Survived', [1, 0])
train.expect_column_mean_to_be_between("Age", 20,40)
train.expect_column_values_to_match_regex('Name', '[A-Z][a-z]+(?: \([A-Z][a-z]+\))?, ', mostly=.95)
train.expect_column_values_to_be_in_set("Sex", ["male", "female"])

results = train.validate() 
if train.validate()["SUCCESS"]:
    ...
results
my_expectations = train.get_expectation_suite()
test.validate(expectation_suite=my_expectations)<jupyter_output>2020-10-21T09:11:59+0300 - INFO - 	5 expectation(s) included in expectation_suite. result_format settings filtered.<jupyter_text>Great Expectations Validation in Your pipeline<jupyter_code># ! great_expectations init
context = ge.data_context.DataContext()
context.list_expectation_suite_names()
expectation_suite_name = " "
batch_kwargs = {'path': "https://github.com/plotly/datasets/raw/master/titanic.csv",
                'datasource': "titanic"}
batch = context.get_batch(batch_kwargs, my_expectations)
batch.head()
run_id = {"run_name": "First validation", "run_time": datetime.now()}
results = context.run_validation_operator("action_list_operator", 
                                          assets_to_validate=[batch], 
                                          run_id=run_id)
results<jupyter_output>2020-10-21T11:51:14+0300 - INFO - 	5 expectation(s) included in expectation_suite.<jupyter_text>Test as data documentations  your docs are your tests and your  tests are your docs<jupyter_code>context.build_data_docs()
context.open_data_docs()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1105830164765526, 'fraction_numerical': 0.016159695817490494, 'mean_word_length': 5.442857142857143, 'pattern_counts': {'":': 2, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '183029', 'license': 'MIT', 'n_tokens_mistral': 996, 'n_tokens_neox': 936, 'n_words': 246, 'path': 'Lectures/data_quality/Great Expectations.ipynb', 'repo_name': 'eyaltrabelsi/Monty-Hall-Problem', 'size': '3156'}