<jupyter_start><jupyter_text>- [Github](https://github.com/tensorflow/tfx/blob/master/docs/tutorials/data_validation/chicago_taxi.ipynb)- Python2에서 진행    - Python3에서도 되긴 하는데, 몇 기능이 안될듯(Apache Beam이 아직 파이썬2만 지원)<jupyter_code>from __future__ import print_function
import sys, os
import tempfile, urllib, zipfile
# Confirm that we're using Python 2
assert sys.version_info.major is 2, 'Oops, not running Python 2'

# Set up some globals for our file paths
BASE_DIR = tempfile.mkdtemp()
DATA_DIR = os.path.join(BASE_DIR, 'data')
OUTPUT_DIR = os.path.join(BASE_DIR, 'chicago_taxi_output')
TRAIN_DATA = os.path.join(DATA_DIR, 'train', 'data.csv')
EVAL_DATA = os.path.join(DATA_DIR, 'eval', 'data.csv')
SERVING_DATA = os.path.join(DATA_DIR, 'serving', 'data.csv')

# Download the zip file from GCP and unzip it
zip, headers = urllib.urlretrieve('https://storage.googleapis.com/tfx-colab-datasets/chicago_data.zip')
zipfile.ZipFile(zip).extractall(BASE_DIR)
zipfile.ZipFile(zip).close()

print("Here's what we downloaded:")
!ls -lR {os.path.join(BASE_DIR, 'data')}
!pip2 install -q tensorflow_data_validation
import tensorflow_data_validation as tfdv

print('TFDV version: {}'.format(tfdv.version.__version__))<jupyter_output>[33mDEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.[0m<jupyter_text>Compute and visualize statistics- [tfdv.generate_statistics_from_csv](https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/generate_statistics_from_csv)로 데이터 분포 생성- 많은 데이터일 경우 내부적으로 Apache Beam을 사용해 병렬처리- Beam의 PTransform과 결합 가능<jupyter_code>train_stats = tfdv.generate_statistics_from_csv(data_location=TRAIN_DATA)<jupyter_output>WARNING:root:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.<jupyter_text>- [tfdv.visualize_statistics](https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/visualize_statistics)를 사용해 시각화, 내부적으론 [Facets](https://pair-code.github.io/facets/)을 사용한다 함- numeric, categorical feature들을 나눔<jupyter_code>tfdv.visualize_statistics(train_stats)<jupyter_output><empty_output><jupyter_text>Infer a scahema- 데이터를 통해 스키마 추론- [tfdv.infer_schema](https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/infer_schema)- [tfdv.display_schema](https://www.tensorflow.org/tfx/data_validation/api_docs/python/tfdv/display_schema)<jupyter_code>schema = tfdv.infer_schema(statistics=train_stats)
tfdv.display_schema(schema=schema)<jupyter_output><empty_output><jupyter_text>평가 데이터 에러 체크- train, validation에서 다른 데이터들이 있음- 캐글할 때 유용할듯<jupyter_code># Compute stats for evaluation data
eval_stats = tfdv.generate_statistics_from_csv(data_location=EVAL_DATA)

# Compare evaluation data with training data
tfdv.visualize_statistics(lhs_statistics=eval_stats, rhs_statistics=train_stats,
                          lhs_name='EVAL_DATASET', rhs_name='TRAIN_DATASET')<jupyter_output><empty_output><jupyter_text>Check for evaluation anomalies- train 데이터엔 없었는데 validation에 생긴 데이터 있는지 확인<jupyter_code># Check eval data for errors by validating the eval data stats using the previously inferred schema.
anomalies = tfdv.validate_statistics(statistics=eval_stats, schema=schema)
tfdv.display_anomalies(anomalies)<jupyter_output><empty_output><jupyter_text>Fix evaluation anomalies in the schema- 수정<jupyter_code># Relax the minimum fraction of values that must come from the domain for feature company.
company = tfdv.get_feature(schema, 'company')
company.distribution_constraints.min_domain_mass = 0.9

# Add new value to the domain of feature payment_type.
payment_type_domain = tfdv.get_domain(schema, 'payment_type')
payment_type_domain.value.append('Prcard')

# Validate eval stats after updating the schema 
updated_anomalies = tfdv.validate_statistics(eval_stats, schema)
tfdv.display_anomalies(updated_anomalies)<jupyter_output><empty_output><jupyter_text>Schema Environments- serving할 때도 스키마 체크해야 함- Environments can be used to express such requirements. In particular, features in schema can be associated with a set of environments using default_environment, in_environment and not_in_environment.<jupyter_code>serving_stats = tfdv.generate_statistics_from_csv(SERVING_DATA)
serving_anomalies = tfdv.validate_statistics(serving_stats, schema)

tfdv.display_anomalies(serving_anomalies)<jupyter_output><empty_output><jupyter_text>- Int value가 있음 => Float으로 수정<jupyter_code>options = tfdv.StatsOptions(schema=schema, infer_type_from_schema=True)
serving_stats = tfdv.generate_statistics_from_csv(SERVING_DATA, stats_options=options)
serving_anomalies = tfdv.validate_statistics(serving_stats, schema)

tfdv.display_anomalies(serving_anomalies)
# All features are by default in both TRAINING and SERVING environments.
schema.default_environment.append('TRAINING')
schema.default_environment.append('SERVING')

# Specify that 'tips' feature is not in SERVING environment.
tfdv.get_feature(schema, 'tips').not_in_environment.append('SERVING')

serving_anomalies_with_env = tfdv.validate_statistics(
    serving_stats, schema, environment='SERVING')

tfdv.display_anomalies(serving_anomalies_with_env)<jupyter_output><empty_output><jupyter_text>Check for drift and skew- Drift    - Drift detection is supported for categorical features and between consecutive spans of data (i.e., between span N and span N+1), such as between different days of training data. We express drift in terms of [L-infinity distance](https://en.wikipedia.org/wiki/Chebyshev_distance), and you can set the threshold distance so that you receive warnings when the drift is higher than is acceptable. Setting the correct distance is typically an iterative process requiring domain knowledge and experimentation. - Skew    - Schema Skew        - 같은 스키마를 가지지 않을 때    - Feature Skew        - Feature 생성 로직이 변경될 때    - Dsitribution Skew        - Train, Serving시 데이터 분포가 다를 경우<jupyter_code># Add skew comparator for 'payment_type' feature.
payment_type = tfdv.get_feature(schema, 'payment_type')
payment_type.skew_comparator.infinity_norm.threshold = 0.01

# Add drift comparator for 'company' feature.
company=tfdv.get_feature(schema, 'company')
company.drift_comparator.infinity_norm.threshold = 0.001

skew_anomalies = tfdv.validate_statistics(train_stats, schema,
                                          previous_statistics=eval_stats,
                                          serving_statistics=serving_stats)

tfdv.display_anomalies(skew_anomalies)<jupyter_output><empty_output><jupyter_text>Freeze the schema- 스키마 저장<jupyter_code>from tensorflow.python.lib.io import file_io
from google.protobuf import text_format

file_io.recursive_create_dir(OUTPUT_DIR)
schema_file = os.path.join(OUTPUT_DIR, 'schema.pbtxt')
tfdv.write_schema_text(schema, schema_file)

!cat {schema_file}<jupyter_output>feature {
  name: "fare"
  type: FLOAT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "trip_start_hour"
  type: INT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: 1
    }
  }
}
feature {
  name: "pickup_census_tract"
  type: BYTES
  presence {
    min_count: 0
  }
}
feature {
  name: "dropoff_census_tract"
  value_count {
    min: 1
    max: 1
  }
  type: FLOAT
  presence {
    min_count: 1
  }
}
feature {
  name: "company"
  value_count {
    min: 1
    max: 1
  }
  type: BYTES
  domain: "company"
  presence {
    min_count: 1
  }
  distribution_constraints {
    min_domain_mass: 0.9
  }
  drift_comparator {
    infinity_norm {
      threshold: 0.001
    }
  }
}
feature {
  name: "trip_start_timestamp"
  type: INT
  presence {
    min_fraction: 1.0
    min_count: 1
  }
  shape {
    dim {
      size: [...]
{'fraction_non_alphanumeric': 0.0902558931047523, 'fraction_numerical': 0.0071851758477246945, 'mean_word_length': 4.903273809523809, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 43, 'https://': 8, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '86591', 'license': 'MIT', 'n_tokens_mistral': 2979, 'n_tokens_neox': 2960, 'n_words': 610, 'path': 'Tensorflow-Extended/TFDV(data validation) example.ipynb', 'repo_name': 'zzsza/TIL', 'size': '7933'}