<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extending Python with Compiled Code\n",
    "\n",
    "In a previous session, we talked about ways to get performance out of Python using vectorization strategies within NumPy.  Today we're going to talk about approaches which involve **interfacing Python to external compiled code**, or **converting Python code to compilable code**.  There are several different approaches.  We'll give examples of solving a problem by several means:\n",
    "\n",
    "- [Ctypes](http://docs.python.org/2/library/ctypes.html): the C-linking utilities included in the Python standard library\n",
    "- [F2Py](http://cens.ioc.ee/projects/f2py2e/usersguide/): the Fortran to Python utility which is bundled with NumPy\n",
    "- [Cython](http://cython.org/): a module that allows conversion of Python and Python-like code to C\n",
    "- [Numba](http://numba.pydata.org/): a Python to LLVM bytecode converter\n",
    "\n",
    "There is no way to go into depth on all of these, but I mainly just want to show some examples so that you know the types of problems they solve.  At the above links, you'll find more in-depth documentation and tutorials."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are other options we won't talk about here, because they're not as useful for our purposes:\n",
    "\n",
    "- [Python C API](http://docs.python.org/2/c-api/): Python is implemented in C, and you can actually write C code which will compile to a Python module!  This is extremely low-level, and I would only recommend it if you enjoy pain (see <NAME>'s [blog post](http://dan.iel.fm/posts/python-c-extensions/) for a concise-as-possible introduction)\n",
    "- [SWIG](http://www.swig.org/): the Simplified Wrapper Interface Generator, can generate wrappers from a variety of low-level languages to a variety of high-level languages.  It's used heavily by LSST and other projects.\n",
    "- [Weave](http://docs.scipy.org/doc/scipy/reference/tutorial/weave.html): included in SciPy, weave is a method of putting C snippets within a Python program.  It's largely been superseded by Cython in practice.\n",
    "- [PyPy](http://pypy.org/): a JIT compiler for Python written in Python.  PyPy doesn't support the C backend which most scientific tools rely on, so it's not extremely useful for scientific computing.\n",
    "\n",
    "The optimal approach depends on the desired use-case.  We'll go through a few examples, from lowest to highest level:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [<commit_msg>add CFFI mention to Extending Python<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extending Python with Compiled Code\n",
    "\n",
    "In a previous session, we talked about ways to get performance out of Python using vectorization strategies within NumPy.  Today we're going to talk about approaches which involve **interfacing Python to external compiled code**, or **converting Python code to compilable code**.  There are several different approaches.  We'll give examples of solving a problem by several means:\n",
    "\n",
    "- [Ctypes](http://docs.python.org/2/library/ctypes.html): the C-linking utilities included in the Python standard library\n",
    "- [F2Py](http://cens.ioc.ee/projects/f2py2e/usersguide/): the Fortran to Python utility which is bundled with NumPy\n",
    "- [Cython](http://cython.org/): a module that allows conversion of Python and Python-like code to C\n",
    "- [Numba](http://numba.pydata.org/): a Python to LLVM bytecode converter\n",
    "\n",
    "There is no way to go into depth on all of these, but I mainly just want to show some examples so that you know the types of problems they solve.  At the above links, you'll find more in-depth documentation and tutorials."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Note on CFFI and PyPy\n",
    "\n",
    "A newer option that I've not personally used (but is probably the way of the future for wrapping compiled code in Cython) is the [C Foreign Function Interface (CFFI)](https://cffi.readthedocs.org/). One huge advantage of this is that it works with [PyPy](http://pypy.org/), where the other options don't.\n",
    "\n",
    "[PyPy](http://pypy.org/) is a JIT compiler for Python written in Python.  PyPy doesn't support the C backend which most scientific tools rely on, so it's not extremely useful for scientific computing. If the functionality of numpy, scipy, etc. is rewritten using the CFFI in the future, though, PyPy may become a competetive option."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Options\n",
    "\n",
    "There are other options we won't talk about here, because they have mostly been superseded by the above tools:\n",
    "\n",
    "- [Python C API](http://docs.python.org/2/c-api/): Python is implemented in C, and you can actually write C code which will compile to a Python module!  This is extremely low-level, and I would only recommend it if you enjoy pain (see <NAME>'s [blog post](http://dan.iel.fm/posts/python-c-extensions/) for a concise-as-possible introduction)\n",
    "- [SWIG](http://www.swig.org/): the Simplified Wrapper Interface Generator, can generate wrappers from a variety of low-level languages to a variety of high-level languages.  It's used heavily by LSST and other projects.\n",
    "- [Weave](http://docs.scipy.org/doc/scipy/reference/tutorial/weave.html): included in SciPy, weave is a method of putting C snippets within a Python program.  It's largely been superseded by Cython in practice.\n",
    "\n",
    "The optimal approach depends on the desired use-case.  We'll go through a few examples, from lowest to highest level:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
{'fraction_non_alphanumeric': 0.11870082035827892, 'fraction_numerical': 0.0016742005692281935, 'mean_word_length': 3.5603053435114504, 'pattern_counts': {'":': 23, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2842016', 'n_tokens_mistral': 1775, 'n_tokens_neox': 1711, 'n_words': 740}