<jupyter_start><jupyter_text>We'll work with a data set of customer preferences on trains, available [here](http://vincentarelbundock.github.io/Rdatasets/doc/Ecdat/Train.html). This is a staticdataset and isn't being updated, but you could imagine that each month the Dutch authoritiesupload a new month's worth of data. We can start by making some very basic assertions, that the dataset is the correct shape, and that a few columns are the correct dtypes. Assertions are made as decorators to functions that return a DataFrame.<jupyter_code>import pandas as pd
import engarde.decorators as ed

pd.set_option('display.max_rows', 10)

dtypes = dict(
    price1=int,
    price2=int,
    time1=int,
    time2=int,
    change1=int,
    change2=int,
    comfort1=int,
    comfort2=int
)

@ed.is_shape((-1, 11))
@ed.has_dtypes(items=dtypes)
def unload():
    url = "http://vincentarelbundock.github.io/Rdatasets/csv/Ecdat/Train.csv"
    trains = pd.read_csv(url, index_col=0)
    return trains
df = unload()
df.head()<jupyter_output><empty_output><jupyter_text>Notice two things: we only specified the dtypes for some of the columns, and we don't care about the length of the DataFrame (just its width), so we passed -1 for the first dimension of the shape. Since people are rational, their first choice is surely going to be better in *at least* one way than their second choice. This is fundamental to our analysis later on, so we'll explicilty state it in our code, and check it in our data.<jupyter_code>def rational(df):
    """
    Check that at least one criteria is better.
    """
    r = ((df.price1 < df.price2) | (df.time1 < df.time2) |
         (df.change1 < df.change2) | (df.comfort1 > df.comfort2))
    return r

@ed.is_shape((-1, 11))
@ed.has_dtypes(items=dtypes)
@ed.verify_all(rational)
def unload():
    url = "http://vincentarelbundock.github.io/Rdatasets/csv/Ecdat/Train.csv"
    trains = pd.read_csv(url, index_col=0)
    return trains
df = unload()<jupyter_output><empty_output><jupyter_text>OK, so apparently people aren't rational... We'll fix this problem by ignoring those people (why change your mind when you can change the data?).<jupyter_code>@ed.verify_all(rational)
def drop_silly_people(df):
    r = ((df.price1 < df.price2) | (df.time1 < df.time2) |
         (df.change1 < df.change2) | (df.comfort1 > df.comfort2))
    return df[r]


@ed.is_shape((-1, 11))
@ed.has_dtypes(items=dtypes)
def unload():
    url = "http://vincentarelbundock.github.io/Rdatasets/csv/Ecdat/Train.csv"
    trains = pd.read_csv(url, index_col=0)
    return trains

def main():
    df = (unload()
          .pipe(drop_silly_people)
          )
    return df
df = main()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11312384473197781, 'fraction_numerical': 0.014417744916820702, 'mean_word_length': 3.983425414364641, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '502608', 'license': 'MIT', 'n_tokens_mistral': 938, 'n_tokens_neox': 915, 'n_words': 279, 'path': 'examples/Trains.ipynb', 'repo_name': 'ericmjl/engarde', 'size': '2705'}