I am working on learning how to do frequency analysis of Server Fault question tags to see if there is any useful data that I can glean from them. I'm storing the raw data in Bitbucket for global access, so this code will use the same dataset I am using, it's about 30Mb.import pandas as pddebiandf = pd.read_csv(https://goo.gl/zLDPw6)debiandf[CreationDate] = pd.to_datetime(debiandf[CreationDate], format=%Y-%m-%d %H:%M:%S)debiandf = debiandf.set_index([CreationDate])tag_df = pd.DataFrame(index=debiandf.index, data=debiandf[Tags])tag_df = tag_df.reset_index().drop_duplicates(subset='CreationDate', keep='last').set_index('CreationDate')x = tag_df[Tags].str.extractall(r'\<(.*?)\>').unstack()x.columns = x.columns.droplevel(0)# column names signify the index location of the tag when extracted.# i.e. with <ubuntu><networking><tag3> you would have [ubuntu,networking,tag3]x.rename(columns={0: 1, 1: 2, 2: 3, 3: 4, 4: 5}, inplace=True)x1 = x.groupby(x.index.year)[1].apply(lambda grp: grp.value_counts().head(5))x2 = x.groupby(x.index.year)[2].apply(lambda grp: grp.value_counts().head(5))x3 = x.groupby(x.index.year)[3].apply(lambda grp: grp.value_counts().head(5))x4 = x.groupby(x.index.year)[4].apply(lambda grp: grp.value_counts().head(5))x5 = x.groupby(x.index.year)[5].apply(lambda grp: grp.value_counts().head(5))x6 = pd.concat([x1,x2,x3], axis=1)x6 = x6.reset_index()x6.rename(columns={level_0: Year, level_1: Tag}, inplace=True)print x6I'm new to using pandas and I'm learning how to do data analysis on my own so I can produce original content for reddit. How can I simplify my x.groupby lines so I can get the top 5 value counts from every column in my x DataFrame?I'm new to pandas, so if you could explain why it would be done that way, that would be really helpful. Thanks!
Simplify DataFrame operations in Python
python;pandas;data mining