<jupyter_start><jupyter_text>Automatic Suggestion of ConstraintsIn our experience, a major hurdle in data validation is that someone needs to come up with the actual constraints to apply on the data. This can be very difficult for large, real-world datasets, especially if they are very complex and contain information from a lot of different sources. We build so-called constraint suggestion functionality into deequ to assist users in finding reasonable constraints for their data.Our constraint suggestion first [profiles the data](./data_profiling_example.ipynb) and then applies a set of heuristic rules to suggest constraints. In the following, we give a concrete example on how to have constraints suggested for your data.<jupyter_code>from pyspark.sql import SparkSession, Row, DataFrame
import json
import pandas as pd
import sagemaker_pyspark

import pydeequ

classpath = ":".join(sagemaker_pyspark.classpath_jars())

spark = (SparkSession
    .builder
    .config("spark.driver.extraClassPath", classpath)
    .config("spark.jars.packages", pydeequ.deequ_maven_coord)
    .config("spark.jars.excludes", pydeequ.f2j_maven_coord)
    .getOrCreate())<jupyter_output><empty_output><jupyter_text>Let's first generate some example data:<jupyter_code>df = spark.sparkContext.parallelize([
    Row(productName="thingA", totalNumber="13.0", status="IN_TRANSIT", valuable="true"),
    Row(productName="thingA", totalNumber="5", status="DELAYED", valuable="false"),
    Row(productName="thingB", totalNumber=None, status="DELAYED", valuable=None),
    Row(productName="thingC", totalNumber=None, status="IN_TRANSIT", valuable="false"),
    Row(productName="thingD", totalNumber="1.0", status="DELAYED", valuable="true"),
    Row(productName="thingC", totalNumber="7.0", status="UNKNOWN", valuable=None),
    Row(productName="thingC", totalNumber="20", status="UNKNOWN", valuable=None),
    Row(productName="thingE", totalNumber="20", status="DELAYED", valuable="false"),
    Row(productName="thingA", totalNumber="13.0", status="IN_TRANSIT", valuable="true"),
    Row(productName="thingA", totalNumber="5", status="DELAYED", valuable="false"),
    Row(productName="thingB", totalNumber=None, status="DELAYED", valuable=None),
    Row(productName="thingC", totalNumber=None, status="IN_TRANSIT", valuable="false"),
    Row(productName="thingD", totalNumber="1.0", status="DELAYED", valuable="true"),
    Row(productName="thingC", totalNumber="17.0", status="UNKNOWN", valuable=None),
    Row(productName="thingC", totalNumber="22", status="UNKNOWN", valuable=None),
    Row(productName="thingE", totalNumber="23", status="DELAYED", valuable="false")]).toDF()<jupyter_output><empty_output><jupyter_text>Now, we ask PyDeequ to compute constraint suggestions for us on the data. It will profile the data and then apply the set of rules specified in `addConstraintRules()` to suggest constraints.<jupyter_code>from pydeequ.suggestions import *

suggestionResult = ConstraintSuggestionRunner(spark) \
             .onData(df) \
             .addConstraintRule(DEFAULT()) \
             .run()<jupyter_output><empty_output><jupyter_text>We can now investigate the constraints that deequ suggested. We get a textual description and the corresponding Python code for each suggested constraint. Note that the constraint suggestion is based on heuristic rules and assumes that the data it is shown is 'static' and correct, which might often not be the case in the real world. Therefore the suggestions should always be manually reviewed before being applied in real deployments.<jupyter_code>for sugg in suggestionResult['constraint_suggestions']:
    print(f"Constraint suggestion for \'{sugg['column_name']}\': {sugg['description']}")
    print(f"The corresponding Python code is: {sugg['code_for_constraint']}\n")<jupyter_output>Constraint suggestion for 'valuable': 'valuable' has less than 62% missing values
The corresponding Python code is: .hasCompleteness("valuable", lambda x: x >= 0.38, "It should be above 0.38!")

Constraint suggestion for 'valuable': 'valuable' has type Boolean
The corresponding Python code is: .hasDataType("valuable", ConstrainableDataTypes.Boolean)

Constraint suggestion for 'totalNumber': 'totalNumber' has no negative values
The corresponding Python code is: .isNonNegative("totalNumber")

Constraint suggestion for 'totalNumber': 'totalNumber' has less than 47% missing values
The corresponding Python code is: .hasCompleteness("totalNumber", lambda x: x >= 0.53, "It should be above 0.53!")

Constraint suggestion for 'totalNumber': 'totalNumber' has type Fractional
The corresponding Python code is: .hasDataType("totalNumber", ConstrainableDataTypes.Fractional)

Constraint suggestion for 'productName': 'productName' has value range 'thingC', 'thingA', 'thingB', 'thingE', 'thingD'
The cor[...]
{'fraction_non_alphanumeric': 0.11541649399088272, 'fraction_numerical': 0.00870285951098218, 'mean_word_length': 6.460587326120557, 'pattern_counts': {'":': 1, '<': 16, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '29406', 'license': 'Apache-2.0', 'n_tokens_mistral': 1365, 'n_tokens_neox': 1327, 'n_words': 432, 'path': 'tutorials/constraint_suggestion_example.ipynb', 'repo_name': 'nfx/python-deequ', 'size': '4826'}