<commit_before># Data Validation Tooling
The goal of this tool is to allow easy comparison and validation between different tables.  The repository supports several types of comparison
	- Count Validation: Total counts and other aggregates match between source and destination
	- Partitioned Count: Partitioned counts and other aggregations between source and destination
		- Grouped values ie SELECT updated_at::DATE, COUNT(1) matches for each date
	- Stretch: Custom queries of any kind match between source and destination
	- Stretch: Random sampling of data matches between source and destination


## Example

client = example_client.ExampleClient()
builder = query_builder.QueryBuilder.build_count_validator()

query = builder.render_query(client, "schema_name", "table_name", partition_column=None, partition_column_type=None)
print(query)
TODO more<commit_msg>adding more details and example to readme<commit_after># Data Validation Tooling
The goal of this tool is to allow easy comparison and validation between different tables.  The repository supports several types of comparison
	- Count Validation: Total counts and other aggregates match between source and destination
	- Partitioned Count: Partitioned counts and other aggregations between source and destination
		- Grouped values ie SELECT updated_at::DATE, COUNT(1) matches for each date
	- Stretch: Custom queries of any kind match between source and destination
	- Stretch: Random sampling of data matches between source and destination


## Source Configuration
Every source type requires its own configuration for connectivity.  Below is an example of the expected configuration for each source type.

### BigQuery
{
    "source_type": "BigQuery",
    "project_id": os.environ["PROJECT_ID"],
    "schema_name": "bigquery-public-data.new_york_citibike",
    "table_name": "citibike_trips"
}

## Example of Comparing BigQuery to BigQuery
```
from data_validation import data_validation
from data_validation.data_sources import data_client, example_client, bigquery
from data_validation.query_builder import query_builder

inp_config = {"source_type": "BigQuery",
              "project_id": os.environ["PROJECT_ID"],
              "schema_name": "bigquery-public-data.new_york_citibike",
              "table_name": "citibike_trips"}
out_config = {"source_type": "BigQuery",
              "project_id": os.environ["PROJECT_ID"],
              "schema_name": "bigquery-public-data.new_york_citibike",
              "table_name": "citibike_trips"}

builder = query_builder.QueryBuilder.build_count_validator()
data_validation.process_data(builder, inp_config, out_config)
```
{'fraction_non_alphanumeric': 0.07145557655954632, 'fraction_numerical': 0.0007561436672967864, 'mean_word_length': 5.041095890410959, 'pattern_counts': {'":': 12, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2382882', 'n_tokens_mistral': 699, 'n_tokens_neox': 681, 'n_words': 254}