<jupyter_start><jupyter_text>Numpy and Pandas Performance Comparison[<NAME>](http://gouthamanbalaraman.com) Pandas and Numpy are two packages that are core to a lot of data analysis. In this post I will compare the performance of numpy and pandas. tl;dr:- `numpy` consumes less memory compared to `pandas`- `numpy` generally performs better than `pandas` for 50K rows or less- `pandas` generally performs better than `numpy` for 500K rows or more- for 50K to 500K rows, it is a toss up between `pandas` and `numpy` depending on the kind of operation<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
plt.style.use("seaborn-pastel")
%matplotlib inline
import seaborn.apionly as sns
import numpy as np
from timeit import timeit 
import sys
iris = sns.load_dataset('iris')
data = pd.concat([iris]*100000)
data_rec = data.to_records()
print (len(data), len(data_rec))<jupyter_output>15000000 15000000<jupyter_text>Here I have loaded the `iris` dataset and replicated it so as to have 15MM rows of data. The space requirement for 15MM rows of data in a `pandas dataframe` is more than twice that of a `numpy recarray`.<jupyter_code>MB = 1024*1024
print("Pandas %d MB " % (sys.getsizeof(data)/MB))
print("Numpy %d MB " % (sys.getsizeof(data_rec)/MB))<jupyter_output>Pandas 1506 MB 
Numpy 686 MB<jupyter_text>A snippet of the data shown below.<jupyter_code>data.head()
# <!-- collapse=True -->
def perf(inp, statement, grid=None):
    length = len(inp) 
    gap = int(length/5)
    #grid = np.array([int(x) for x in np.logspace(np.log10(gap), np.log10(length+1) , 5)])
    if grid is None:
        grid = np.array([10000, 100000, 1000000, 5000000, 10000000])
    num = 100
    time = []
    data = {'pd': pd, 'np': np}
    for i in grid:
        if isinstance(inp, pd.DataFrame):
            sel = inp.iloc[:i]
            data['data'] = sel
        else:
            sel = inp[:i]
            data['data_rec'] = sel
        t = timeit(stmt=statement, globals=data, number=num)
        time.append(t/num)
    return grid, np.array(time)

def bench(pd_inp, pd_stmt, np_inp, np_stmt, title="", grid=None):
    g,v1 = perf(pd_inp, pd_stmt, grid)
    g,v2 = perf(np_inp, np_stmt, grid)
    fig, ax = plt.subplots()
    ax.loglog()
    ax.plot(g, v1, label="pandas",marker="o", lw=2)
    ax.plot(g, v2, label="numpy", marker="v", lw=2)
    ax.set_xticks(g)
    plt.legend(loc=2)
    plt.xlabel("Number of Records")
    plt.ylabel("Time (s)")
    plt.grid(True)
    plt.xlim(min(g)/2,max(g)*2)
    plt.title(title)<jupyter_output><empty_output><jupyter_text>In this post, performance metrics for a few different categories are compared between `numpy` and `pandas`:- operations on a column of data, such as mean or applying a vectorised function- operations on a filtered column of data- vector operations on a column or filtered column  Operations on a Column Here some performance metrics with operations on one column of data. The operations involved in here include fetching a view, and a reduction operation such as `mean`, vectorised `log` or a string based `unique` operation. All these are `O(n)` calculations. The mean calculation is orders of magnitude faster in `numpy` compared to `pandas` for array sizes of 100K or less. For sizes larger than 100K `pandas` maintains a lead over `numpy`.<jupyter_code>bench(data, "data.loc[:, 'sepal_length'].mean()", 
      data_rec, "np.mean(data_rec.sepal_length)",
     title="Mean on Unfiltered Column")<jupyter_output><empty_output><jupyter_text>Below, the vectorized `log` operation is faster in `numpy` for sizes less than 100K but pandas costs about the same for sizes larger than 100K.<jupyter_code>bench(data, "np.log(data.loc[:, 'sepal_length'])",
      data_rec, "np.log(data_rec.sepal_length)",
     title="Vectorised log on Unfiltered Column")<jupyter_output><empty_output><jupyter_text>The one differentiating aspect about the test below is that the column `species` is of string type. The operation demonstrated is a `unique` calculation. We observe that the `unique` calculation is roughly an order of magnitude faster in pandas for sizes larger than 1K rows.<jupyter_code>bench(data, "data.loc[:,'species'].unique()",
      data_rec, "np.unique(data_rec.species)", 
      grid=np.array([100, 1000, 10000, 100000, 1000000]),
     title="Unique on Unfiltered String Column")<jupyter_output><empty_output><jupyter_text>Operations on a Filtered Column Below we perform the same tests as above, except that the column is not a full view, but is instead a filtered view. The filters are simple filters with an arithmetic bool comparison for the first two and a string comparison for the third below. Below, `mean` is calculated for a filtered column `sepal_length`. Here performance of `pandas` is better for row sizes larger than 10K. In the `mean` on unfiltered column shown above, `pandas` performed better for 1MM or more. Just having selection operations has shifted performance chart in favor of `pandas` for even smaller number of records.<jupyter_code>bench(data, "data.loc[(data.sepal_width>3) & \
             (data.petal_length<1.5), 'sepal_length'].mean()",
      data_rec, "np.mean(data_rec[(data_rec.sepal_width>3) & \
                 (data_rec.petal_length<1.5)].sepal_length)",
      grid=np.array([1000, 10000, 100000, 1000000]),
     title="Mean on Filtered Column")<jupyter_output><empty_output><jupyter_text>For vectorised `log` operation on a unfiltered column shown above, `numpy` performed better than `pandas` for number of records less than 100K while the performance was comparable for the two for sizes larger than 100K. But the moment you introduce a filter on a column, `pandas` starts to show an edge over `numpy` for number of records larger than 10K.<jupyter_code>bench(data, "np.log(data.loc[(data.sepal_width>3) & \
            (data.petal_length<1.5), 'sepal_length'])",
      data_rec, "np.log(data_rec[(data_rec.sepal_width>3) & \
                 (data_rec.petal_length<1.5)].sepal_length)",
      grid=np.array([1000, 10000, 100000, 1000000]),
     title="Vectorised log on Filtered Column")<jupyter_output><empty_output><jupyter_text>Here is another example of a `mean` reduction on a column but with a string filter. We see a similar behavior where `numpy` performs significantly better at small sizes and `pandas` takes a gentle lead for larger number of records.<jupyter_code>bench(data, "data[data.species=='setosa'].sepal_length.mean()",
      data_rec, "np.mean(data_rec[data_rec.species=='setosa'].sepal_length)",
      grid=np.array([1000, 10000, 100000, 1000000]),
     title="Mean on (String) Filtered Column")<jupyter_output><empty_output><jupyter_text>Vectorized Operation on a Column In this last section, we do vectorised arithmetic using multiple columns. This involves creating a view and vectorised math on these views. Even when there is no filter, `pandas` has a slight edge over `numpy` for large number of records. For smaller than 100K records, `numpy` performs significantly better.<jupyter_code>bench(data, "data.petal_length * data.sepal_length + \
      data.petal_width * data.sepal_width",
      data_rec, "data_rec.petal_length*data_rec.sepal_length + \
      data_rec.petal_width * data_rec.sepal_width",
     title="Vectorised Math on Unfiltered Columns")<jupyter_output><empty_output><jupyter_text>In the following figure, the filter involves vectorised arithmetic operation, and `mean` reduction is computed on the filtered column. The presence of a filter makes `pandas` significantly faster for sizes larger than 100K, while `numpy` maitains a lead for smaller than 10K number of records.<jupyter_code>bench(data, "data.loc[data.sepal_width * data.petal_length > \
             data.sepal_length, 'sepal_length'].mean()",
      data_rec, "np.mean(data_rec[data_rec.sepal_width * data_rec.petal_length \
                 > data_rec.sepal_length].sepal_length)",
     title="Vectorised Math in Filtering Columns",
     grid=np.array([100, 1000, 10000, 100000, 1000000]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10639621702339472, 'fraction_numerical': 0.03272772523643604, 'mean_word_length': 4.516129032258065, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 51, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '17350', 'license': 'Artistic-2.0', 'n_tokens_mistral': 2768, 'n_tokens_neox': 2507, 'n_words': 858, 'path': 'content/extra/notebooks/pandas_vs_numpy.ipynb', 'repo_name': 'gouthambs/karuth-source', 'size': '8047'}