I'm currently working on a project with a partner where we analyze large datasets of past sporting events. There are approximately 30,000 events per year and we have historical data for five years. Thus, we're looking at analyzing 150,000 events.I wrote our first version in Python with a heavy reliance on Pandas (great tool, by the way). However, each time I add a factor or calculation to the process it slows down by quite a bit. Simply put, it can read all the various data files and data queries we need at a rate of 25-events-per-second.The problem is once I start working with the data, it drops fast. If I add a simple calculation, it drops to 24... and each calculation after that it drops again.With just 10-12 factors calculated from the data (we're not talking complex loops or anything crazy), it's now down to 6-events-per-second. ... and we're barely scratching the surface. At this rate, it'll take this thing days to work through all 150k events!I've optimized our data queries, flat file reads etc. Those really aren't the issue. I can live with doing 15-20 events per second. However, I can't live with such drastic slowdowns each time a new factor calculation is added.I've read a lot where Python as an interpreted language is slow etc., but my question to the experts out there - what language should this project be done in?EXAMPLE...One of the key DataFrames I am using through Pandas is fairly large. 350 rows x 70 columns, as an example. Even when trying to simply multiply one column's value by another... data['value'] = data['col1'] * data['col2']...I see what I could consider significant drop in performance. It's very puzzling, and very frustrating. I understand this is a large set of data, but I can't believe this is anything so crazy that Python would be slowing to a crawl.If I just read the data files and do no calculations on them at all, it reads 67 events in 2.807 seconds. If I add one simple calculation where I execute code like the example above, it slows to 2.877 seconds. Based on our research, we're needed to add upwards of 100 calculations on the data.... so 7 seconds worth of slowdown? Seems too hard to believe.
Is Python suitable for a statistical modeling application looking over thousands of past events?
programming languages;python;performance
I work on a platform that uses Python for testing and executing market data.For just the equity pricing there are roughly ~98,000 events for each equityper year, so when dealing with an individual stock have a similar profile forour data sets.(The aforementioned platform is Quantopian, https://quantopian.com.)We also use pandas for the majority of our data structures, and though it is tough to really compare without also comparing the machines used, the number of calculations done per event... we are able to process a few orders of magnitude more eventsper second; so there may not be reasons to give up on pandas, yet.(The backtester used by our site is opensourced, https://github.com/quantopian/ziplineif you'd like to see how we use pandas throughout; of course, we still have bottlenecks to hammer out, too!))From what I can see with how you are using your data set, this tip may help.If possible, avoid writing your calculations back into your dataset DataFrame.A corollary to that is, if possible, avoid indexing in inner loops.While powerful and expressive, the indexing of a DataFrame has a performance penalty; much like accessing into a Python dictionary will create a lookup penalty.Writing the data into the DataFrame will be more expensive.So if your program can just use the Series that is returned by multiplying thosetwo columns, you may see some benefit from assigning that Series to its own variableinstead of putting it back into the DataFrame.To illustrate, below is some excerpts from an IPython notebook that step by stepgets the pandas DataFrame values closer and closer to using a raw np.ndarray.This following is not intended to be a cookbook, but to show where both writing and indexing into the DataFrame can hurt performance.Also, I wouldn't be surprised if there is something in the pandas API that I am unaware of, which makes assignment of the column much more performant, which would invalidate my advice.        In [1]:# Create a DataFrame with 3 columns and 350 random values each.data = pd.DataFrame(randn(350, 3), columns=['col1', 'col2', 'value'])In [2]:# Time original example%timeit data['value'] = data['col1'] * data['col2']10000 loops, best of 3: 170 s per loopIn [3]:# Time without writing the example back into the DataFrame# The value variable will be a pd.Series%timeit value = data['col1'] * data['col2']10000 loops, best of 3: 43.9 s per loop# Extract the pd.Series before inner loop col1 = data['col1']col2 = data['col2']In [25]:# Time without writing the example back into the DataFrame# The value variable will be a pd.Series%timeit value = col1 * col210000 loops, best of 3: 32.3 s per loopIn [4]:# Time with using the underlying numpy arrays.# The value variable will be a np.ndarray%timeit value = data['col1'].values * data['col2'].values100000 loops, best of 3: 15.9 s per loopIn [5]:# Extract the numpy arrays before inner loop col1_v = data['col1'].valuescol2_v = data['col2'].valuesIn [6]:# Time just the multiplication of two np.ndarrays %timeit col1_v * col2_v1000000 loops, best of 3: 1.5 s per loop