<jupyter_start><jupyter_text>Demo of faster_df_apply`pandas.apply` is a lot slower than it should be. See discussion [here](https://stackoverflow.com/questions/54432583/when-should-i-ever-want-to-use-pandas-apply-in-my-code).This notebooks demonstrates the `faster_df_apply` function, which speeds things up dramatically.<jupyter_code>import ticdat
import time
import pandas as pd
pandat = ticdat.PanDatFactory(table_data = [[], ["SoldCount", "StrengthFactor", "ItemCount"]])
pandat.set_data_type("table_data", "SoldCount", min=0, max=50, inclusive_min=True, 
                     inclusive_max=True, nullable=False)
dat = pandat.csv.create_pan_dat("pandat_data")
dat.table_data["SoldCount"].max(), dat.table_data["SoldCount"].min()
def faster_df_apply(df, func):
    cols = list(df.columns)
    data, index = [], []
    for row in df.itertuples(index=True):
        row_dict = {f:v for f,v in zip(cols, row[1:])}
        data.append(func(row_dict))
        index.append(row[0])
    return pd.Series(data, index=index)
def bad_row(row):
    val = row["SoldCount"]
    try:
        if 0 <= val <= 50:
            return False
    except:
        return True
    return True<jupyter_output><empty_output><jupyter_text>Here are three different ways to find the rows with a bad "SoldCount", each with very different run times. * Using the `bad_row` function with `pandas.DataFrame`. * Using the `bad_row` function with `faster_df_apply`. * Using `PanDatFactory.find_data_type_failures`.<jupyter_code>%timeit dat.table_data.apply(bad_row, axis=1)
%timeit faster_df_apply(dat.table_data, bad_row)
%timeit pandat.find_data_type_failures(dat)<jupyter_output>806 ms ± 10.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)<jupyter_text>ConclusionThe default implementation of `apply` appears to be slow for silly reasons. It can be sigmnificantly improved by a more thoughtful pure Python rewrite. With this commit, we have woven `faster_df_apply` into `ticdat`, thus addressing issue [7](https://github.com/ticdat/ticdat/issues/7). That said, `PanDatFactory.find_data_type_failures` has a slower version of `bad_row`, and thus is somewhat slower than the `faster_df_apply(dat.table_data, bad_row)` call. There may or may-not be room for run time improvement, but closing issue 7 for now.<jupyter_code>import cProfile
#cProfile.run("pandat.find_data_type_failures(dat)", sort=2)
#cProfile.run("faster_df_apply(dat.table_data, bad_row)", sort=2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11624139327663022, 'fraction_numerical': 0.012150668286755772, 'mean_word_length': 5.098765432098766, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 12, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '195794', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 857, 'n_tokens_neox': 817, 'n_words': 218, 'path': 'examples/expert_section/notebooks/issue_7_run_time_fix_internal_demo.ipynb', 'repo_name': 'adampkehoe/ticdat', 'size': '2469'}