<jupyter_start><jupyter_text>Slicing with pandas, gurobipy.tuplelist, and O(n) slicing Run a little python script that sets up the performance comparisons.<jupyter_code>run prep_for_different_slicings.py<jupyter_output><empty_output><jupyter_text>The slicing will be over small, medium, and large tables.<jupyter_code>[len(getattr(td, "childTable")) for td in (smallTd, medTd, bigTd)]<jupyter_output><empty_output><jupyter_text>We will run three series of four tests each.Each series tests 1. slicing with `.sloc` and `pandas` 1. slicing with `gurobipy.tuplelist` 1. slicing with `ticdat.Slicer` (with the `gurobipy` enhancement disabled) 1. O(n) slicing First, we see that with a small table (1,200) rows, the `pandas` slicing is only somewhat faster than the O(n) slicing, while `Slicer` slicing is quite a bit faster and `tuplelist` faster still.<jupyter_code>%timeit checkChildDfLen(smallChildDf, *smallChk)
%timeit checkTupleListLen(smallSmartTupleList, *smallChk)
%timeit checkSlicerLen(smallSlicer, *smallChk)
%timeit checkTupleListLen(smallDumbTupleList, *smallChk)<jupyter_output>100 loops, best of 3: 5.71 ms per loop<jupyter_text>Next we see that with a table of 31,800 rows, `pandas` slicing is now ~100 faster than O(n) slicing (but `tuplelist` and `Slicer` are still the fastest by far).<jupyter_code>%timeit checkChildDfLen(medChildDf, *medChk)
%timeit checkTupleListLen(medSmartTupleList, *medChk)
%timeit checkSlicerLen(medSlicer, *medChk)
%timeit checkTupleListLen(medDumbTupleList, *medChk)<jupyter_output>10 loops, best of 3: 164 ms per loop<jupyter_text>Finally, we see that with a table of 270,000 rows, `pandas` slicing is ~1000X faster than O(n) slicing. Here, `tuplelist` is blindingly fast - nearly as much an improvement shows over `pandas` as `pandas` shows over O(n). `Slicer` again comes in a respectably close second.<jupyter_code>%timeit checkChildDfLen(bigChildDf, *bigChk)
%timeit checkTupleListLen(bigSmartTupleList, *bigChk)
%timeit checkSlicerLen(bigSlicer, *bigChk)
%timeit checkTupleListLen(bigDumbTupleList, *bigChk)<jupyter_output>1 loops, best of 3: 1.42 s per loop<jupyter_text>Bottom line? `pandas` isn't really designed with "iterating over indicies and slicing" in mind, so it isn't the absolutely fastest way to write this sort of code. However, `pandas` also doesn't implement naive O(n) slicing. For most instances, the `.sloc` approach to slicing will be fast enough. In general, so long as you use the optimal big-O subroutines, the time to solve a MIP or LP model will be larger than the time to formulate the model.  However, in those instances where the slicing is the bottleneck operation, `gurobipy.tuplelist` or `ticdat.Slicer` can be used, or the model building code can be refactored to be more pandonic.    Addendum There was a request to check `sum` as well as `len`. Here the results vindicate `pandas`, in as much as all three "smart" strategies are roughly equivalent.<jupyter_code>%timeit checkChildDfSum(bigChildDf, *bigChk)
%timeit checkTupleListSum(bigSmartTupleList, bigTd, *bigChk)
%timeit checkSlicerSum(bigSlicer, bigTd, *bigChk)
%timeit checkTupleListSum(bigDumbTupleList, bigTd, *bigChk)<jupyter_output>1 loops, best of 3: 1.42 s per loop
{'fraction_non_alphanumeric': 0.08686176836861768, 'fraction_numerical': 0.015255292652552927, 'mean_word_length': 6.65, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '342262', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 1151, 'n_tokens_neox': 1093, 'n_words': 383, 'path': 'examples/expert_section/notebooks/different_slicings.ipynb', 'repo_name': 'adampkehoe/ticdat', 'size': '3212'}