<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbsphinx-toctree": {
     "hidden": true
    }
   },
   "source": [
    "[Installation](installing.rst)\n",
    "[Tutorial](tutorial.ipynb)\n",
    "[Examples](examples.rst)\n",
    "[API](api.rst)\n",
    "[Machine learning](ml.rst)\n",
    "[Datasets](datasets.rst)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/html"
   },
   "source": [
    "<style>\n",
    "pre {\n",
    " white-space: pre-wrap !important;\n",
    "}\n",
    ".table-striped > tbody > tr:nth-of-type(odd) {\n",
    "    background-color: #f9f9f9;\n",
    "}\n",
    ".table-striped > tbody > tr:nth-of-type(even) {\n",
    "    background-color: white;\n",
    "}\n",
    ".table-striped td, .table-striped th, .table-striped tr {\n",
    "    border: 1px solid black;\n",
    "    border-collapse: collapse;\n",
    "    margin: 1em 2em;\n",
    "}\n",
    ".rendered_html td, .rendered_html th {\n",
    "    text-align: left;\n",
    "    vertical-align: middle;\n",
    "    padding: 4px;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Vaex?\n",
    "\n",
    "Vaex is a python library for lazy **Out-of-Core DataFrames** (similar to Pandas), to visualize and explore big tabular datasets. It can calculate *statistics* such as mean, sum, count, standard deviation etc, on an *N-dimensional grid* up to **a billion** ($10^9$) objects/rows **per second**. Visualization is done using **histograms**, **density plots** and **3d volume rendering**, allowing interactive exploration of big data. Vaex uses memory mapping, zero memory copy policy and lazy computations for best performance (no memory wasted)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why vaex\n",
    " \n",
    " * **Performance:** Works with huge tabular data, process $\\gt 10^9$ rows/second\n",
    " * **Lazy / Virtual columns:** compute on the fly, without wasting ram\n",
    " * **Memory efficient** no memory copies when doing filtering/selections/subsets.\n",
    " * **Visualization:** directly supported, a one-liner is often enough.\n",
    " * **User friendly API:** You will only need to deal with the DataFrame object, and tab completion + docstring will help you out: `ds.mean<tab>`, feels very similar to Pandas.\n",
    " * **Lean:** separated into multiple packages\n",
    "    * `vaex-core`: DataFrame and core algorithms, takes numpy arrays as input columns.\n",
    "    * `vaex-hdf5`: Provides memory mapped numpy arrays to a DataFrame.\n",
    "    * `vaex-arrow`: [Arrow](https://arrow.apache.org/) support for cross language data sharing.\n",
    "    * `vaex-viz`: Visualization based on matplotlib.\n",
    "    * `vaex-jupyter`: Interactive visualization based on Jupyter widgets / ipywidgets, bqplot, ipyvolume and ipyleaflet.\n",
    "    * `vaex-astro`: Astronomy related transformations and FITS file support.\n",
    "    * `vaex-server`: Provides a server to access a DataFrame remotely.\n",
    "    * `vaex-distributed`: (Proof of concept) combined multiple servers / cluster into a single DataFrame for distributed computations.\n",
    "    * `vaex-qt`: Program written using Qt GUI.\n",
    "    * `vaex`: meta package that installs all of the above.\n",
    "    * `vaex-ml`: [Machine learning](ml.ipynb)\n",
    "\n",
    " * **Jupyter integration**: vaex-jupyter will give you interactive visualization and selection in the Jupyter notebook and Jupyter lab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "Using conda: \n",
    "\n",
    " * `conda install -c conda-forge vaex`\n",
    "\n",
    "Using pip:\n",
    "\n",
    " * `pip install --upgrade vaex`\n",
    "  \n",
    "Or read the [detailed instructions](installing.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting started\n",
    "\n",
    "We assuming you have installed vaex, and are running a [Jupyter notebook server](https://jupyter.readthedocs.io/en/latest/running.html). We start by importing vaex and ask it to give us sample example dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/maartenbreddels/miniconda3/envs/working/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "import vaex\n",
    "df = vaex.example()  # open the example dataset provided with vaex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead, you can [download some larger datasets](datasets.rst), or [read in your csv file](api.rst#vaex.from_csv)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>#                                  </th><th>x           </th><th>y           </th><th>z           </th><th>vx         </th><th>vy         </th><th>vz         </th><th>E              </th><th>L                 </th><th>Lz                 </th><th>FeH                </th></tr>\n",<commit_msg>docs: remove warning<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbsphinx-toctree": {
     "hidden": true
    }
   },
   "source": [
    "[Installation](installing.rst)\n",
    "[Tutorial](tutorial.ipynb)\n",
    "[Examples](examples.rst)\n",
    "[API](api.rst)\n",
    "[Machine learning](ml.rst)\n",
    "[Datasets](datasets.rst)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/html"
   },
   "source": [
    "<style>\n",
    "pre {\n",
    " white-space: pre-wrap !important;\n",
    "}\n",
    ".table-striped > tbody > tr:nth-of-type(odd) {\n",
    "    background-color: #f9f9f9;\n",
    "}\n",
    ".table-striped > tbody > tr:nth-of-type(even) {\n",
    "    background-color: white;\n",
    "}\n",
    ".table-striped td, .table-striped th, .table-striped tr {\n",
    "    border: 1px solid black;\n",
    "    border-collapse: collapse;\n",
    "    margin: 1em 2em;\n",
    "}\n",
    ".rendered_html td, .rendered_html th {\n",
    "    text-align: left;\n",
    "    vertical-align: middle;\n",
    "    padding: 4px;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Vaex?\n",
    "\n",
    "Vaex is a python library for lazy **Out-of-Core DataFrames** (similar to Pandas), to visualize and explore big tabular datasets. It can calculate *statistics* such as mean, sum, count, standard deviation etc, on an *N-dimensional grid* up to **a billion** ($10^9$) objects/rows **per second**. Visualization is done using **histograms**, **density plots** and **3d volume rendering**, allowing interactive exploration of big data. Vaex uses memory mapping, zero memory copy policy and lazy computations for best performance (no memory wasted)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why vaex\n",
    " \n",
    " * **Performance:** Works with huge tabular data, process $\\gt 10^9$ rows/second\n",
    " * **Lazy / Virtual columns:** compute on the fly, without wasting ram\n",
    " * **Memory efficient** no memory copies when doing filtering/selections/subsets.\n",
    " * **Visualization:** directly supported, a one-liner is often enough.\n",
    " * **User friendly API:** You will only need to deal with the DataFrame object, and tab completion + docstring will help you out: `ds.mean<tab>`, feels very similar to Pandas.\n",
    " * **Lean:** separated into multiple packages\n",
    "    * `vaex-core`: DataFrame and core algorithms, takes numpy arrays as input columns.\n",
    "    * `vaex-hdf5`: Provides memory mapped numpy arrays to a DataFrame.\n",
    "    * `vaex-arrow`: [Arrow](https://arrow.apache.org/) support for cross language data sharing.\n",
    "    * `vaex-viz`: Visualization based on matplotlib.\n",
    "    * `vaex-jupyter`: Interactive visualization based on Jupyter widgets / ipywidgets, bqplot, ipyvolume and ipyleaflet.\n",
    "    * `vaex-astro`: Astronomy related transformations and FITS file support.\n",
    "    * `vaex-server`: Provides a server to access a DataFrame remotely.\n",
    "    * `vaex-distributed`: (Proof of concept) combined multiple servers / cluster into a single DataFrame for distributed computations.\n",
    "    * `vaex-qt`: Program written using Qt GUI.\n",
    "    * `vaex`: meta package that installs all of the above.\n",
    "    * `vaex-ml`: [Machine learning](ml.ipynb)\n",
    "\n",
    " * **Jupyter integration**: vaex-jupyter will give you interactive visualization and selection in the Jupyter notebook and Jupyter lab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "Using conda: \n",
    "\n",
    " * `conda install -c conda-forge vaex`\n",
    "\n",
    "Using pip:\n",
    "\n",
    " * `pip install --upgrade vaex`\n",
    "  \n",
    "Or read the [detailed instructions](installing.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting started\n",
    "\n",
    "We assuming you have installed vaex, and are running a [Jupyter notebook server](https://jupyter.readthedocs.io/en/latest/running.html). We start by importing vaex and ask it to give us sample example dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
   ],
   "source": [
    "import vaex\n",
    "df = vaex.example()  # open the example dataset provided with vaex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead, you can [download some larger datasets](datasets.rst), or [read in your csv file](api.rst#vaex.from_csv)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>#                                  </th><th>x           </th><th>y           </th><th>z           </th><th>vx         </th><th>vy         </th><th>vz         </th><th>E              </th><th>L                 </th><th>Lz                 </th><th>FeH                </th></tr>\n",
{'fraction_non_alphanumeric': 0.17768479776847979, 'fraction_numerical': 0.003905160390516039, 'mean_word_length': 2.57936772046589, 'pattern_counts': {'":': 75, '<': 61, '<?xml version=': 0, '>': 69, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1841834', 'n_tokens_mistral': 3727, 'n_tokens_neox': 3585, 'n_words': 968}