<commit_before>{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Problem 4.2. Simple Statistics II.\n",
      "\n",
      "This problem is a continuation of problem 4.1. Recall that you wrote a function named `get_stats()` that takes a list and returns a tuple of minimum, maximum, mean, and median. To use this function, you have to convert your IPython notebook to a regular `.py` file. One way to do this is to use the IPython `%%script%%` magic function; open up a new notebook, and in an IPython notebook cell, type (assuming the filename of your IPython notebook from Problem 4.1 is `stats.ipynb`):\n",
      "\n",
      "    %%bash\n",
      "    ipython3 nbconvert --to python stats.ipynb\n",
      "\n",
      "and press <kbd>shift</kbd> + <kbd>enter</kbd>. This will create a Python script\n",
      "named `stats.py`. We will import this as a module in `stats2.ipynb`:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from stats import get_stats"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We will use the function `get_stats()` to compute basic statistics of a number of columns from the arline performance dataset we downloaded in week 2. Namely, we will use the following columns:\n",
      "\n",
      "- Column 15, \"ArrDelay\": arrival delay, in minutes,\n",
      "- Column 16, \"DepDelay\": departure delay, in minutes, and\n",
      "- Column 19, \"Distance\": distance, in miles.\n",
      "\n",
      "To extract these columns from the CSV file,\n",
      "\n",
      "- Write a function named `get_column(filename, n, header = True)` that reads the `n`-th column from a file and returns a list of integers.\n",
      "\n",
      "- You may assume that the column is made of integers.\n",
      "\n",
      "- We will also use the optional argument `header` because the first line of our file lists the names of the columns, but we might want to turn this off to handle a file that doesn't have a header.\n",
      "\n",
      "- Use a combination of `with` statement and `open()` function to open `filename` in the `get_column()` function.\n",
      "  \n",
      "  Tip: When I tried to use `open()` to read `2001.csv`, I had the following error:\n",
      "  \n",
      "        'utf-8' codec can't decode byte 0xe4 in position 343: invalid continuation byte\n",
      "        \n",
      "  You can avoid this error by using `encoding='latin-1'` option in `open()`.\n",
      "  \n",
      "- Skip the first line if the `header` parameter is `True`; do not skip if it's `False`.\n",
      "\n",
      "- Some columns have missing values `'NA'`, and you need a way to handle these\n",
      "  missing values. If the `n`-th column is missing, you should **not** include\n",
      "  that column in `result`; that is, skip all rows with `'NA'`.\n",
      "  As a result, lists returned from different\n",
      "  columns may have different lengths."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def get_column(filename, n, header=True):\n",
      "    '''\n",
      "    Returns a list from reading the specified column in the CSV file.\n",
      "\n",
      "    Parameters\n",
      "    __________\n",
      "    filename(str): Input file name in Comma Separated Values (CSV) format\n",
      "    n(int): Column number. The first column starts at 0. The column must be\n",
      "            a list of integers.\n",
      "    header(bool): If True, the first line of file is column names.\n",
      "                  Default: True.\n",
      "\n",
      "    Examples\n",
      "    ________\n",
      "    >>> get_column('/data/airline/2001.csv', 14)[:10]\n",
      "    [-3, 4, 23, 10, 20, -3, -10, -12, -9, -1]\n",
      "    >>> get_column('/data/airline/2001.csv', 15)[-10:]\n",
      "    [-4, -5, -8, 4, -7, 4, 8, -4, -4, 9]\n",
      "    '''\n",
      "    result = []\n",
      "    \n",
      "    # your code goes here\n",
      "    \n",
      "    return result"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We also want to print out the results in a nicely formatted manner.\n",
      "\n",
      "- The `print_stats(input_list, title=None)` function is already written for you.\n",
      "  You don't need to write this function.\n",
      "\n",
      "It takes a list of integers and prints out the basic statistics."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def print_stats(input_list, title=None):\n",
      "    '''\n",
      "    Computes minimum, maximum, mean, and median using get_stats function from\n",
      "      stats module, and prints them out in a nice format.\n",
      "\n",
      "    Parameters:\n",
      "      input_list(list): a list representing a column\n",
      "      title(str): Optional. If given, title is printed out before the stats.\n",
      "\n",
      "    Examples:\n",
      "    >>> print_stats(list(range(50)))\n",
      "    Minimum: 0\n",
      "    Maximum: 49\n",
      "    Mean: 24.5\n",
      "    Median: 24.5\n",
      "    >>> print_stats(list(range(100)), title = 'Stats!')\n",
      "    Stats!\n",
      "    Minimum: 0\n",
      "    Maximum: 99\n",
      "    Mean: 49.5\n",
      "    Median: 49.5\n",
      "    '''\n",
      "    if title is not None:\n",
      "        print(title)\n",
      "        \n",
      "    minimum, maximum, mean, median = get_stats(input_list)\n",
      "    print('Minimum: {0}\\n'\n",
      "          'Maximum: {1}\\n'\n",
      "          'Mean: {2:.2f}\\n'\n",
      "          'Median: {3:.2f}'.format(minimum, maximum, mean, median))\n",
      "    return None"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "When you run the following cell, you should get\n",
      "\n",
      "    Arrival delay, in minutes.\n",
      "    Minimum: -1116\n",
      "    Maximum: 1688\n",
      "    Mean: 5.38\n",
      "    Median: -2.00"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#warning: this could take a while.\n",
      "filename = '/data/airline/2001.csv' # 2001 airline on-time performance dataset\n",
      "\n",
      "arr_delay = get_column(filename, 14)\n",
      "print_stats(arr_delay, \"Arrival delay, in minutes.\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "When you run the following cell, you should get\n",
      "\n",
      "    Departure delay, in minutes.\n",
      "    Minimum: -75\n",
      "    Maximum: 1692\n",
      "    Mean: 8.03\n",
      "    Median: 0.00"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#warning: this could take a while.\n",
      "dep_delay = get_column(filename, 15)\n",
      "print_stats(dep_delay, \"Departure delay, in minutes.\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "When you run the following cell, you should get\n",<commit_msg>Fix error in sample outputs.<commit_after>{
 "metadata": {
  "name": "",
  "signature": "sha256:8cf6b34e2909addb4f8378318e63137f55a1d3baa0ca6e83eac81dc7893db8a5"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Problem 4.2. Simple Statistics II.\n",
      "\n",
      "This problem is a continuation of problem 4.1. Recall that you wrote a function named `get_stats()` that takes a list and returns a tuple of minimum, maximum, mean, and median. To use this function, you have to convert your IPython notebook to a regular `.py` file. One way to do this is to use the IPython `%%script%%` magic function; open up a new notebook, and in an IPython notebook cell, type (assuming the filename of your IPython notebook from Problem 4.1 is `stats.ipynb`):\n",
      "\n",
      "    %%bash\n",
      "    ipython3 nbconvert --to python stats.ipynb\n",
      "\n",
      "and press <kbd>shift</kbd> + <kbd>enter</kbd>. This will create a Python script\n",
      "named `stats.py`. We will import this as a module in `stats2.ipynb`:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from stats import get_stats"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We will use the function `get_stats()` to compute basic statistics of a number of columns from the arline performance dataset we downloaded in week 2. Namely, we will use the following columns:\n",
      "\n",
      "- Column 15, \"ArrDelay\": arrival delay, in minutes,\n",
      "- Column 16, \"DepDelay\": departure delay, in minutes, and\n",
      "- Column 19, \"Distance\": distance, in miles.\n",
      "\n",
      "To extract these columns from the CSV file,\n",
      "\n",
      "- Write a function named `get_column(filename, n, header = True)` that reads the `n`-th column from a file and returns a list of integers.\n",
      "\n",
      "- You may assume that the column is made of integers.\n",
      "\n",
      "- We will also use the optional argument `header` because the first line of our file lists the names of the columns, but we might want to turn this off to handle a file that doesn't have a header.\n",
      "\n",
      "- Use a combination of `with` statement and `open()` function to open `filename` in the `get_column()` function.\n",
      "  \n",
      "  Tip: When I tried to use `open()` to read `2001.csv`, I had the following error:\n",
      "  \n",
      "        'utf-8' codec can't decode byte 0xe4 in position 343: invalid continuation byte\n",
      "        \n",
      "  You can avoid this error by using `encoding='latin-1'` option in `open()`.\n",
      "  \n",
      "- Skip the first line if the `header` parameter is `True`; do not skip if it's `False`.\n",
      "\n",
      "- Some columns have missing values `'NA'`, and you need a way to handle these\n",
      "  missing values. If the `n`-th column is missing, you should **not** include\n",
      "  that column in `result`; that is, skip all rows with `'NA'`.\n",
      "  As a result, lists returned from different\n",
      "  columns may have different lengths."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def get_column(filename, n, header=True):\n",
      "    '''\n",
      "    Returns a list from reading the specified column in the CSV file.\n",
      "\n",
      "    Parameters\n",
      "    __________\n",
      "    filename(str): Input file name in Comma Separated Values (CSV) format\n",
      "    n(int): Column number. The first column starts at 0. The column must be\n",
      "            a list of integers.\n",
      "    header(bool): If True, the first line of file is column names.\n",
      "                  Default: True.\n",
      "\n",
      "    Examples\n",
      "    ________\n",
      "    >>> get_column('/data/airline/2001.csv', 14)[:10]\n",
      "    [-3, 4, 23, 10, 20, -3, -10, -12, -9, -1]\n",
      "    >>> get_column('/data/airline/2001.csv', 15)[-10:]\n",
      "    [-4, -5, -8, 4, -7, 4, 8, -4, -4, 9]\n",
      "    '''\n",
      "    result = []\n",
      "    \n",
      "    # your code goes here\n",
      "    \n",
      "    return result"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We also want to print out the results in a nicely formatted manner.\n",
      "\n",
      "- The `print_stats(input_list, title=None)` function is already written for you.\n",
      "  You don't need to write this function.\n",
      "\n",
      "It takes a list of integers and prints out the basic statistics."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def print_stats(input_list, title=None):\n",
      "    '''\n",
      "    Computes minimum, maximum, mean, and median using get_stats function from\n",
      "      stats module, and prints them out in a nice format.\n",
      "\n",
      "    Parameters:\n",
      "      input_list(list): a list representing a column\n",
      "      title(str): Optional. If given, title is printed out before the stats.\n",
      "\n",
      "    Examples:\n",
      "    >>> print_stats(list(range(50)))\n",
      "    Minimum: 0\n",
      "    Maximum: 49\n",
      "    Mean: 24.5\n",
      "    Median: 24.5\n",
      "    >>> print_stats(list(range(100)), title = 'Stats!')\n",
      "    Stats!\n",
      "    Minimum: 0\n",
      "    Maximum: 99\n",
      "    Mean: 49.5\n",
      "    Median: 49.5\n",
      "    '''\n",
      "    if title is not None:\n",
      "        print(title)\n",
      "        \n",
      "    minimum, maximum, mean, median = get_stats(input_list)\n",
      "    print('Minimum: {0}\\n'\n",
      "          'Maximum: {1}\\n'\n",
      "          'Mean: {2:.2f}\\n'\n",
      "          'Median: {3:.2f}'.format(minimum, maximum, mean, median))\n",
      "    return None"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "When you run the following cell, you should get\n",
      "\n",
      "    Arrival delay, in minutes.\n",
      "    Minimum: -1116\n",
      "    Maximum: 1688\n",
      "    Mean: 5.53\n",
      "    Median: -2.00"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#warning: this could take a while.\n",
      "filename = '/data/airline/2001.csv' # 2001 airline on-time performance dataset\n",
      "\n",
      "arr_delay = get_column(filename, 14)\n",
      "print_stats(arr_delay, \"Arrival delay, in minutes.\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "When you run the following cell, you should get\n",
      "\n",
      "    Departure delay, in minutes.\n",
      "    Minimum: -204\n",
      "    Maximum: 1692\n",
      "    Mean: 8.15\n",
      "    Median: 0.00"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#warning: this could take a while.\n",
      "dep_delay = get_column(filename, 15)\n",
      "print_stats(dep_delay, \"Departure delay, in minutes.\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "When you run the following cell, you should get\n",
{'fraction_non_alphanumeric': 0.16727557411273486, 'fraction_numerical': 0.02055062630480167, 'mean_word_length': 1.985781067393845, 'pattern_counts': {'":': 115, '<': 11, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2929458', 'n_tokens_mistral': 5262, 'n_tokens_neox': 4950, 'n_words': 1571}