<jupyter_start><jupyter_text>8. Say No to `apply` with `groupby`  What is `apply` with `groupby`Typically when calling `groupby`, you will use one of the standard aggregation functions that Pandas already has built such as sum, min, median, etc... Occasionally, you might need to write your own customized aggregation (or non-aggregation) function. To do so require you use `apply`.Previously, we learned that `apply` can lead to very poor performance. This is especially true when using it with `groupby`. Let's see this in action with a problem that might lead you to think we absolutely need to use `apply`.  Challenge Using the college dataset, for each state, find the percentage of the total state population made up by the 5 largest colleges of that state.<jupyter_code>import pandas as pd
pd.set_option('display.max_columns', 100)
college = pd.read_csv('data/college.csv')
college.head()<jupyter_output><empty_output><jupyter_text>A naive solutionWe can define a custom aggregation function to sort each state's population and then find the top 5 values and compute the percent of the total like this.<jupyter_code>def func1(s):
    s = s.sort_values(ascending=False)
    top5_total = s.iloc[:5].sum()
    total = s.sum()
    return top5_total / total
df = college.groupby('stabbr')['ugds'].apply(func1)
df.head()<jupyter_output><empty_output><jupyter_text>Do as much as you can outside of the custom functionAs a general rule, you should attempt to do as many operations outside the custom operation as possible. For instance, we can sort all the values just once instead of within each group. Let's compare this performance.<jupyter_code>df = college.sort_values('ugds', ascending=False)
def func2(s):
    top5_total = s.iloc[:5].sum()
    total = s.sum()
    return top5_total / total
df.groupby('stabbr')['ugds'].apply(func2).head()<jupyter_output><empty_output><jupyter_text>Time performanceNice performance boost.<jupyter_code>%timeit -n 5 college.groupby('stabbr')['ugds'].apply(func1)
%%timeit -n 5 
df = college.sort_values('ugds', ascending=False)
df.groupby('stabbr')['ugds'].apply(func2)<jupyter_output><empty_output><jupyter_text>Remove function completelyYou can actually completely avoid using `apply` by first selecting the top 5 values for each state.<jupyter_code>df = college.sort_values('ugds', ascending=False)
df_top5 = df.groupby('stabbr').head()<jupyter_output><empty_output><jupyter_text>We can then find the totals for each DataFrame and divide the result.<jupyter_code>top5_total = df_top5.groupby('stabbr').agg({'ugds': 'sum'})
total = df.groupby('stabbr').agg({'ugds': 'sum'})
df_final = top5_total / total
df_final.head()<jupyter_output><empty_output><jupyter_text>Performance is now about 6x as fast as before.<jupyter_code>%%timeit -n 5 

df = college.sort_values('ugds', ascending=False)
df_top5 = df.groupby('stabbr').head()
top5_total = df_top5.groupby('stabbr').agg({'ugds': 'sum'})
total = df.groupby('stabbr').agg({'ugds': 'sum'})
df_final = top5_total / total<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11561264822134387, 'fraction_numerical': 0.010210803689064558, 'mean_word_length': 6.062790697674418, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '335538', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 993, 'n_tokens_neox': 955, 'n_words': 308, 'path': '8. Say No to apply with groupby.ipynb', 'repo_name': 'MathMachado/Minimally-Sufficient-Pandas', 'size': '3036'}