<jupyter_start><jupyter_text>Tensorflow data validation demoLet's use the Tensorflow data validation to examine our data. Install the packageThe official pypi page can be found here: https://pypi.org/project/tensorflow-data-validation/ Note, It only works in python 3.7 and python 3.8pip install tensorflow-data-validationThere are also bugs in the package dependencies' management, the above pip install will install an outdated numpy, pandas package. But it requires a newer version. So after running the above pip install, you need to run```shellpip install numpy --upgradepip install pandas --upgrade```<jupyter_code>import tensorflow as tf
import tensorflow_data_validation as tfdv
import os
import tempfile, urllib, zipfile
import pandas as pd

print('TF version:', tf.__version__)
print('TFDV version:', tfdv.version.__version__)
work_dir="../../data"
# Read train data
train_data_path=f"{work_dir}/adult_train.csv"
train_data=pd.read_csv(train_data_path, sep=",", na_values=["?"])
train_data.head()
# Read eval data

eval_data_path=f"{work_dir}/adult_eval.csv"

eval_data=pd.read_csv(eval_data_path, sep=",", na_values=["?"])
eval_data.head()
# read serving data

serving_data_path=f"{work_dir}/adult_serving.csv"
serving_data=pd.read_csv(serving_data_path, sep=",", na_values=["?"])
serving_data.head()<jupyter_output><empty_output><jupyter_text>Step1 Data profilingFirst we'll use [tfdv.generate_statistics_from_dataframe](https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/generate_statistics_from_dataframe) to understand our training data.<jupyter_code>train_stats = tfdv.generate_statistics_from_dataframe(dataframe=train_data)<jupyter_output><empty_output><jupyter_text>Now let's use [tfdv.visualize_statistics](https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/visualize_statistics).<jupyter_code>tfdv.visualize_statistics(train_stats)<jupyter_output><empty_output><jupyter_text>Step2: Infer a schemaIn TFDV, the definition of **schema** is different from database or dataframe. It describes not only the column name, column datatypes, it also describes the characteristics of your data such as presence/abscence of data, expected range of values, etc.In general, TFDV uses **conservative heuristics** to infer stable data properties from the statistics in order to avoid overfitting the schema to the specific dataset. **It is strongly advised to review the inferred schema and refine it as needed**, to capture any domain knowledge about the data that TFDV’s heuristics might have missed.If we want to compare with solutions such as Great Expectations, TDDA. The stats(TFDV) are profiler(TDDA), the schema(TFDV) is validation rules(TDDA).<jupyter_code>schema = tfdv.infer_schema(statistics=train_stats)
tfdv.display_schema(schema=schema)<jupyter_output><empty_output><jupyter_text>Check evaluation data for errorsSo far we've only been looking at the training data.  It's important that our evaluation data is consistent with our training data, including that it uses the same schema.  It's also important that the evaluation data includes examples of roughly the same ranges of values for our numerical features as our training data, so that our coverage of the loss surface during evaluation is roughly the same as during training.<jupyter_code># Compute stats for evaluation data
eval_stats = tfdv.generate_statistics_from_dataframe(dataframe=eval_data)
# Compare evaluation data with training data
tfdv.visualize_statistics(lhs_statistics=eval_stats, rhs_statistics=train_stats,
                          lhs_name='EVAL_DATASET', rhs_name='TRAIN_DATASET')<jupyter_output><empty_output><jupyter_text>Step 4. Check for evaluation anomaliesDoes our evaluation dataset match the schema from our training dataset?  This is especially important for categorical features, where we want to identify the range of acceptable values.Key Point: What would happen if we tried to evaluate using data with categorical feature values that were not in our training dataset?  What about numeric features that are outside the ranges in our training dataset?<jupyter_code># Check eval data for errors by validating the eval data stats using the previously inferred schema.
anomalies = tfdv.validate_statistics(statistics=eval_stats, schema=schema)
tfdv.display_anomalies(anomalies)<jupyter_output><empty_output><jupyter_text>Fix evaluation anomalies in the schemaIn the eval data, the column native-country contains Holand-Netherlands, that does not exist in training data. We can fix the anomaly by adding this value in the schema.<jupyter_code># Add new value to the domain of feature workclass.
nativecountry_type_domain = tfdv.get_domain(schema, 'native-country')
nativecountry_type_domain.value.append('Holand-Netherlands')


# Validate eval stats after updating the schema 
updated_anomalies = tfdv.validate_statistics(statistics=eval_stats, schema=schema)
tfdv.display_anomalies(updated_anomalies)<jupyter_output><empty_output><jupyter_text>We also split off a 'serving' dataset for this example, so we should check that too.  By default all datasets in a pipeline should use the same schema, but there are often exceptions. For example, in supervised learning we need to include labels in our dataset, but when we serve the model for inference the labels will not be included. In some cases introducing slight schema variations is necessary.**Environments** can be used to express such requirements. In particular, features in schema can be associated with a set of environments using `default_environment`, `in_environment` and `not_in_environment`.For example, in this dataset the `income` feature is included as the label for training, but it's missing in the serving data. Without environment specified, it will show up as an anomaly.<jupyter_code>serving_stats = tfdv.generate_statistics_from_dataframe(dataframe=serving_data)
serving_anomalies = tfdv.validate_statistics(serving_stats, schema)

tfdv.display_anomalies(serving_anomalies)
# Add new value to the domain of feature workclass.
workclass_type_domain = tfdv.get_domain(schema, 'workclass')
workclass_type_domain.value.append('public')

serving_anomalies = tfdv.validate_statistics(serving_stats, schema)
tfdv.display_anomalies(serving_anomalies)<jupyter_output><empty_output><jupyter_text>Now we just have the `income` feature (which is our label) showing up as an anomaly ('Column dropped').  Of course we don't expect to have labels in our serving data, so let's tell TFDV to ignore that.<jupyter_code># All features are by default in both TRAINING and SERVING environments.
schema.default_environment.append('TRAINING')
schema.default_environment.append('SERVING')

# Specify that 'tips' feature is not in SERVING environment.
tfdv.get_feature(schema, 'income').not_in_environment.append('SERVING')

serving_anomalies_with_env = tfdv.validate_statistics(serving_stats, schema, environment='SERVING')

tfdv.display_anomalies(serving_anomalies_with_env)<jupyter_output><empty_output><jupyter_text>Freeze the schemaNow that the schema has been reviewed and curated, we will store it in a file to reflect its "frozen" state. TFDV uses the **protobuf library**, which is becoming a unified method to manipulate your static data (datastructure, transformation scheme, frozen models…).<jupyter_code>from tensorflow.python.lib.io import file_io
from google.protobuf import text_format

output_dir="../data"
schema_file = os.path.join(output_dir, 'schema.pbtxt')
tfdv.write_schema_text(schema, schema_file)

!cat {schema_file}<jupyter_output>feature {
  name: "age"
  type: INT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "workclass"
  value_count {
    min: 1
    max: 1
  }
  type: BYTES
  domain: "workclass"
  presence {
    min_count: 1
  }
}
feature {
  name: "fnlwgt"
  type: INT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "education"
  type: BYTES
  domain: "education"
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "education-num"
  type: INT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "marital-status"
  type: BYTES
  domain: "marital-status"
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "occupation"
  value_count {
    min: 1
    max: 1
  }
  type: BY[...]
{'fraction_non_alphanumeric': 0.07904583723105706, 'fraction_numerical': 0.0037418147801683817, 'mean_word_length': 5.401946107784431, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '206916', 'license': 'Apache-2.0', 'n_tokens_mistral': 2676, 'n_tokens_neox': 2551, 'n_words': 855, 'path': 'demo/04.TFDV/TFDV_demo.ipynb', 'repo_name': 'pengfei99/DataQualityAndValidation', 'size': '8552'}