Q:

Pandas groupby object unique count performance

I have a large dataset of transaction data which looks like:
| cust_no | acct_no | trans_id | product_id | ..... |
I try several way to count how many unique accounts per customer, and how many unique products customer buy etc.

Method 1.a

transaction_df[['cust_no','acct_no']].groupby('cust_no')['acct_no'].nunique()
which runs average 91.5ms

Method 1.b

transaction_df.groupby('cust_no')['acct_no'].nunique()
which runs average 85.5ms

Method 2.a

transaction_df[['cust_no','acct_no']].groupby(['cust_no','acct_no']).size().groupby('cust_no').size()
which runs 61.5ms

Method 2.a

transaction_df.groupby(['cust_no','acct_no']).size().groupby('cust_no').size()
which runs 55.3ms
I have two question:

why is the the DataFrame after slicing run slower, i.e. transaction_df[['cust_no','acct_no']] is slower than just transaction_df?
why .nunique() method is much slower than just stack up two groupby?

A:

1) Slicing requires memory assignment and/or a copy of the object depending on the operation.  Here you're creating a new DataFrame before starting your operations.
2) nunique is going to either implement logic for or directly call a set, which runs in O(N) time.  size will run O(1)
Knowing prior structural information about your dataset can help you optimize function selection as you're experimenting with here.  Read into https://en.wikipedia.org/wiki/Time_complexity if you're interested  

