<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `numpy.arange()` with floats considered harmful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The examples in this notebook are taken from http://quantumwise.com/forum/index.php?topic=110.0#.VIVgyIctjRZ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We often need series of regularly spaced numbers. [`numpy.arange()`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.arange.html) is great for that, as long as the step size is an integer.\n",
    "\n",
    "If the step size is a floating-point number, however, the result might not be what you expect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## But when I last tried it worked as expected ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[`numpy.arange()`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.arange.html) is NumPy's extension of Python's built-in `range()` function.\n",
    "The latter only works with integers, which are also no problem for the former:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(7)  # defaults: start=0, step=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(1, 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(1, 7, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(1, 7, 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `start` value is always part of the returned range, the `stop` value is never part of it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contrary to `range()`, `numpy.arange()` also works with floating-point numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(0.4, 1.1, 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also here, as expected, the `stop` value is not part of the range."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## So what's the problem?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The problem is that some numbers (many, actually) cannot be represented exactly as binary floating-point numbers. Doing calculations with floating-point numbers will often lead to rounding errors. And exactly those rounding errors may lead to unexpected results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(0.5, 1.1, 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow, now `stop` is suddenly included in the range! We didn't expect that!\n",
    "\n",
    "That's the reason why care has to be taken when using `np.arange()` with floating-point numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## OK, I see the problem now, but what's the solution?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, there isn't a single solution for all use cases, it depends on what you need.\n",
    "\n",
    "If you know the first and the last value that should be in your range and the total number of values, you should use `numpy.linspace()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.linspace(0.5, 1.1, 7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In contrast to `np.arange()`, the resulting range includes the last value.\n",
    "\n",
    "If you want it to exclude the last value, you can use the `endpoint` argument (and reduce the number of values by 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.linspace(0.5, 1.1, 6, endpoint=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you, however, want to create a range with a given *spacing*, you'd have to manually calculate the *number of values* before being able to use `numpy.linspace()`.\n",
    "\n",
    "It would be great if there were a feature of either `numpy.linspace()` or `numpy.arange()` that would allow specifying a *spacing* but would also avoid the above-mentioned problem.\n",
    "This was discussed years ago on the NumPy mailing list. Sadly, nobody could come up with a good solution.\n",
    "\n",
    "So for now, you're stuck with adding or subtracting small amounts to/from the `stop` value to get the desired behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "step = 0.1\n",
    "np.arange(0.5, 1.1 - step*0.5, step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This yields the expected result, but it only works if know that the distance between `start` and `stop` is (approximately) an integer multiple of the spacing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert np.isclose((1.1 - 0.5) % step, 0.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to include the `stop` value, you can do something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(0.5, 1.1 + step*0.5, step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, you have to make sure that the spacing fits to the distance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert np.isclose((1.1 - 0.5) % step, 0.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Can't this be done automatically?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, let's try ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def myrange(start, stop, step=1):\n",
    "    # Don't use this function, see below!\n",
    "    return np.arange(start, stop - step*0.5, step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myrange(0.5, 1.1, 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, the problematic case from above works. But let's change the `stop` value a little bit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myrange(0.5, 1.11, 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we would expect `1.1` to be part of the range, because it's clearly smaller than `1.11`. So let's modify our function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def myrange(start, stop, step=1):\n",
    "    if np.isclose((stop - start) % step, 0.0):\n",
    "        stop -= step * 0.5\n",
    "    return np.arange(start, stop, step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myrange(0.5, 1.11, 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, now `1.1` is part of the range, as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myrange(0.5, 1.1, 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous example still works. Good."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This isn't a very elegant solution, though. The behavior depends on the tolerances used in `numpy.isclose()`, namely `rtol` and `atol`. Let's add those to our function!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def myrange(start, stop, step=1, dtype=None, **kwargs):\n",
    "    if np.isclose((stop - start) % step, 0.0, **kwargs):\n",<commit_msg>Add some links to the np.arange() notebook<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `numpy.arange()` with floats considered harmful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few examples used in this notebook are taken from http://quantumwise.com/forum/index.php?topic=110.0#.VIVgyIctjRZ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We often need series of regularly spaced numbers. [`numpy.arange()`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.arange.html) is great for that, as long as the step size is an integer.\n",
    "\n",
    "If the step size is a floating-point number, however, the result might not be what you expect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## But when I last tried it worked as expected ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[`numpy.arange()`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.arange.html) is NumPy's extension of Python's built-in [`range()`](https://docs.python.org/3/library/functions.html#func-range) function.\n",
    "The latter only works with integers, which are also no problem for the former:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(7)  # defaults: start=0, step=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(1, 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(1, 7, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(1, 7, 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `start` value is always part of the returned range, the `stop` value is never part of it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contrary to `range()`, `numpy.arange()` also works with floating-point numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(0.4, 1.1, 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also here, as expected, the `stop` value is not part of the range."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## So what's the problem?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The problem is that some numbers (many, actually) cannot be represented exactly as binary floating-point numbers. Doing calculations with floating-point numbers will often lead to rounding errors. And exactly those rounding errors may lead to unexpected results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(0.5, 1.1, 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow, now `stop` is suddenly included in the range! We didn't expect that!\n",
    "\n",
    "That's the reason why care has to be taken when using `np.arange()` with floating-point numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## OK, I see the problem now, but what's the solution?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, there isn't a single solution for all use cases, it depends on what you need.\n",
    "\n",
    "If you know the first and the last value that should be in your range and the total number of values, you should use [`numpy.linspace()`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.linspace.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.linspace(0.5, 1.1, 7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In contrast to `np.arange()`, the resulting range includes the last value.\n",
    "\n",
    "If you want it to exclude the last value, you can use the `endpoint` argument (and reduce the number of values by 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.linspace(0.5, 1.1, 6, endpoint=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you, however, want to create a range with a given *spacing*, you'd have to manually calculate the *number of values* before being able to use `numpy.linspace()`.\n",
    "\n",
    "It would be great if there were a feature of either `numpy.linspace()` or `numpy.arange()` that would allow specifying a *spacing* but would also avoid the above-mentioned problem.\n",
    "This was discussed years ago on the NumPy mailing list. Sadly, nobody could come up with a good solution.\n",
    "\n",
    "So for now, you're stuck with adding or subtracting small amounts to/from the `stop` value to get the desired behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "step = 0.1\n",
    "np.arange(0.5, 1.1 - step*0.5, step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This yields the expected result, but it only works if know that the distance between `start` and `stop` is (approximately) an integer multiple of the spacing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert np.isclose((1.1 - 0.5) % step, 0.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to include the `stop` value, you can do something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(0.5, 1.1 + step*0.5, step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, you have to make sure that the spacing fits to the distance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert np.isclose((1.1 - 0.5) % step, 0.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Can't this be done automatically?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, let's try ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def myrange(start, stop, step=1):\n",
    "    # Don't use this function, see below!\n",
    "    return np.arange(start, stop - step*0.5, step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myrange(0.5, 1.1, 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, the problematic case from above works. But let's change the `stop` value a little bit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myrange(0.5, 1.11, 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we would expect `1.1` to be part of the range, because it's clearly smaller than `1.11`. So let's modify our function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def myrange(start, stop, step=1):\n",
    "    if np.isclose((stop - start) % step, 0.0):\n",
    "        stop -= step * 0.5\n",
    "    return np.arange(start, stop, step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myrange(0.5, 1.11, 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, now `1.1` is part of the range, as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myrange(0.5, 1.1, 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous example still works. Good."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This isn't a very elegant solution, though. The behavior depends on the tolerances used in [`numpy.isclose()`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.isclose.html), namely `rtol` and `atol`. Let's add those to our function!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def myrange(start, stop, step=1, dtype=None, **kwargs):\n",
    "    if np.isclose((stop - start) % step, 0.0, **kwargs):\n",
{'fraction_non_alphanumeric': 0.19426997245179065, 'fraction_numerical': 0.011955922865013775, 'mean_word_length': 2.7020191719355497, 'pattern_counts': {'":': 352, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5140940', 'n_tokens_mistral': 6730, 'n_tokens_neox': 6462, 'n_words': 1739}