<jupyter_start><jupyter_text>ExpressionsExpressions are only evaluated when needed by vaex, and save you memory.<jupyter_code>np.sqrt(df.x**2 + df.y**2)<jupyter_output><empty_output><jupyter_text>Virtual columnsExpression can be added to a DataFrame to create a virtual column. A virtual column can be treated the same as a normal column, except it does not use up RAM.<jupyter_code>df['r'] = np.sqrt(df.x**2 + df.y**2)
df[['x', 'y', 'r']]
df.r.mean()<jupyter_output><empty_output><jupyter_text>JIT (Just in time) compilationIf an expression becomes to show, try optimizing it with numba, or Pythran<jupyter_code>df['r_normal'] = np.sqrt(df.x**2 + df.y**2)
df['r_jit'] = np.sqrt(df.x**2 + df.y**2).jit_numba()
%%timeit -n3 -r10
df.mean(df.r_normal)
%%timeit -n3 -r10
df.mean(df.r_jit)<jupyter_output><empty_output><jupyter_text>MaterializeOr, if you have plenty of RAM, materialize the column.<jupyter_code>df_m = df.materialize('r')
%%timeit -n3 -r10
df_m.mean(df.r)<jupyter_output><empty_output><jupyter_text>FilteringFiltering makes no copy of the data, ideal when exploring your 1TB dataset.<jupyter_code>df_filtered = df[df.x > 0]
df_filtered[['x', 'y', 'r']]<jupyter_output><empty_output><jupyter_text>SelectionsAll statistical functions can take 1 or more selections as arguments. Multiple selections allow for multiple computations in 1 pass over the data.<jupyter_code>df.mean(df.x, selection=[df.x < 0, df.x > 0])<jupyter_output><empty_output><jupyter_text>Data cleansingEven fillna does not use memory, try different values without wasting time or RAM.<jupyter_code>df_fillna_0 = df.fillna(value=0, column_names=['x'])
df_fillna_3 = df.fillna(value=3, column_names=['x'])
df_fillna_5 = df.fillna(value=5, column_names=['x'])<jupyter_output><empty_output><jupyter_text>N-d statisticsAll statistical methods can be computed on N-dimensional regular grids.<jupyter_code>df.mean(df.x, binby=df.y, limits=[-10, 10], shape=20)<jupyter_output><empty_output><jupyter_text>VisualizationThe N-d statistics are the basis for many of the build-in visualizations.<jupyter_code>df.plot1d(df.x, limits=[-10, 10]);
df.plot(df.x, df.y, limits=[-10, 10]);<jupyter_output><empty_output><jupyter_text>Interactive vizBased on ipywidgets / bqplot, you can even do interactive visualization*Note that (since we are on mybinder) we only use 100.000 rows, instead of 150.000.000 or >1.000.000.000 rows. Download it from https://docs.vaex.io/en/latest/datasets.html if you want to try it out on your local computer.*<jupyter_code># the first 100,000 rows 
df_taxi = vaex.open('./nyc_taxi_2015_100k.arrow')
df_taxi.plot_widget(df_taxi.dropoff_longitude, df_taxi.dropoff_latitude, shape=400,
                    f='log1p', controls_selection=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13115942028985508, 'fraction_numerical': 0.031159420289855074, 'mean_word_length': 6.564383561643836, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 44, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '470331', 'license': 'MIT', 'n_tokens_mistral': 1075, 'n_tokens_neox': 1004, 'n_words': 242, 'path': 'medium-out-of-core/article_snippets.ipynb', 'repo_name': 'kunsen-an/vaex-examples', 'size': '2760'}