<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to Code\n",
    "\n",
    "Now that we've studied a problem-solving process and learned the common programming patterns using pseudocode, it's finally time to express ourselves using actual Python programming language syntax. Keep in mind that, to implement any program, we should follow the problem-solving process and write things out in pseudocode first. Then, coding is a simple matter of translating pseudocode to Python.\n",
    "\n",
    "Let's review our computation model.  Our basic raw ingredient is data (numbers or text strings) that lives on our disk typically (or SSDs nowadays). Note: we might have to go get that data with code; see MSAN692. The disk is very large but cannot serve up data fast enough for the processor, which is many orders of magnitude faster than the disk. Consequently, our first act in an analytics program is often to load some data from the disk into temporary memory. The memory is faster than the disk but smaller and disappears when the power goes off or your program terminates. The processor is still much faster than the memory but we have lots of tricks to increase the speed of communication between the processor and memory (e.g., caches).\n",
    "\n",
    "The processor is where all of the action occurs because it is the entity that executes the statements in a program. The operations in a program boil down to one of these fundamental instructions within the processor:\n",
    "\n",
    "* load small chunks of data from memory into the CPU\n",
    "* perform arithmetic computations on data in the CPU\n",
    "* store small chunks of data back to memory\n",
    "* conditionally perform computations\n",
    "* repeat operations\n",
    "\n",
    "In [Model of Computation](computation.md), we studied pseudocode that maps to one or more of these fundamental instructions. We saw how some of the [higher-level programming patterns](patterns.md) map down to pseudocode chosen from these fundamental instructions. We finished up by looking at some of the [low-level programming patterns](combinations.md) that combine fundamental instructions to do useful things like filtering and image processing.\n",
    "\n",
    "The act of translating a pseudocode operation into Python code involves choosing the right Python construct, just like programming involves choosing the right pattern to solve a piece of a real-world problem.  Then, it's a matter of shoehorning our free-form pseudocode into the straitjacket of programming language syntax. Before we get to those details, however, let's look at the big picture and a general strategy for writing programs.\n",
    "\n",
    "## Coding is similar to writing a paper\n",
    "\n",
    "Writing and executing a program are remarkably similar to writing and reading a paper or report. Just as with our program work plan, we begin writing a paper by clearly identifying a thesis or problem statement. Analogous to identifying input-output pairs, we might identify the target audience and what we hope readers will come away with after reading the paper. With this in mind, we should write an outline of the paper, which corresponds to identifying the processing steps in the program work plan. Sections and chapters in a paper might correspond to functions and packages in the programming world.\n",
    "\n",
    "When reading a paper, we read the sections and paragraphs in order, like a processor executes a program. The paper text can ask the reader to jump temporarily to a figure or different section and return. This is analogous to a program calling a function and returning, possibly with some information.  When reading a paper, we might also encounter conditional sections, such as \"*If you've studied quantum physics, you can skip this section*.\"  There can even be loops in a paper, such as \"*Now's a good time to reread the background section on linear algebra*.\"\n",
    "\n",
    "The point is that, if you've been taught how to properly write a paper, the process of writing code should feel very familiar. To simplify the process of learning to code in Python, we're going to restrict ourselves to a subset of the language and follow a few templates that will help us organize our programs.\n",
    "\n",
    "## A small introductory Python subset\n",
    "\n",
    "While I was in graduate school, I worked in Paris for six months (building an industrial robot control language). A friend, who didn't speak French, came over to work as well and got a tutor.  The tutor started him out with just the present tense, four verbs, and a few key nouns like *café* and *croissant*.   Moreover, the tutor gave him simple sentence templates in the French equivalent of these English templates:\n",
    "\n",
    "`_______ go _______.`\n",
    "\n",
    "and\n",
    "\n",
    "`I am _________.`\n",
    "\n",
    "that he could fill in with subjects and objects (nouns, pronouns).\n",
    "\n",
    "That's also an excellent approach for learning to code in a programming language.  We're going to start out playing around in a small sandbox, picking a simple subset of python that lets us do some interesting things. \n",
    "\n",
    "The \"nouns\" in this subset are numbers like `34` and `3.4`, strings like `parrt`, and lists of nouns like `[3,1.5,4]`.  We can name these values using *variables* just as we use names like Mary to refer to a specific human being. The \"verbs\", which act on nouns, are arithmetic operators like `cost + tax`, relational operators like `quantity<10`, and some built-in functions like `len(names)`.  We'll also use some sentence templates for conditional statements and loops. Finally, we'll also need the ability to pull in (`import`) code written by other programmers to help us out. It's like opening a cookbook that lets us refer to existing recipes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Values and variables\n",
    "\n",
    "The atomic elements in python, so to speak, are numbers and strings of text. We distinguish between integer and real numbers, which we call floating-point numbers, because computers represent the two internally in a different way. Here's where the *data type* comes back into play. Numbers like `34`, `-9`, and `0` are said to have type `int` whereas `3.14159` and `0.123` are type `float`. These values are called int or float *literals*.  Strings of text are just a sequence of characters in single quotes (there are more quoting options but we can ignore for now) and have type `string`.  For example, `'parrt'` and `'May 25, 1999'`. Note that the string representation of `'207'` is very different than the integer `207`. The former is a list with three characters and the latter is a numeric value that we could, say, multiply by 10.\n",
    "\n",
    "Let's look at our first python program!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hi\n"
     ]
    }
   ],
   "source": [
    "print 'hi'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That code is a kind of *statement* that instructs the computer to print the specified value to the computer screen (the console). So here is our first statement template:\n",
    "\n",
    "`print ___________`\n",
    "\n",
    "We can fill that hole with any kind of expression; right now, we only know about values (the simplest expressions) so we can do things like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "34\n",
      "3.14159\n"
     ]
    }
   ],
   "source": [
    "print 34\n",
    "print 3.14159"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the order of execution. The processor is executing one statement after the other.\n",
    "\n",
    "Instead of printing values to the screen, let's store values into memory through variable assignment statements. The statement template looks like:\n",
    "\n",
    "*variablename* `= __________`\n",
    "\n",
    "For example, we can store the value one into a variable called `count` and then reference that variable to load the data back from memory for use by a print statement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "count = 1\n",
    "print count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, the sequence matters. Putting the `print` before the assignment will cause an error because `count` is not defined as a variable until after the assignment.\n",
    "\n",
    "To see how things are stored in memory, let's look at three assignments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "count = 1\n",
    "name = 'iPhone'\n",
    "price = 699.99"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualize the state of computer memory after executing that program using [pythontutor.com](https://goo.gl/5kLND1). It shows a snapshot of memory like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/assign-python-tutor.png\" style=\"width: 100\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(fyi, the \"Global frame\" holds what we call *global variables*. For now, everything will be globally visible and so we can just call them variables.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [<commit_msg>improve notes<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to Code\n",
    "\n",
    "Now that we've studied a problem-solving process and learned the common programming patterns using pseudocode, it's finally time to express ourselves using actual Python programming language syntax. Keep in mind that, to implement any program, we should follow the problem-solving process and write things out in pseudocode first. Then, coding is a simple matter of translating pseudocode to Python.\n",
    "\n",
    "Let's review our computation model.  Our basic raw ingredient is data (numbers or text strings) that lives on our disk typically (or SSDs nowadays). Note: we might have to go get that data with code; see MSAN692. The disk is very large but cannot serve up data fast enough for the processor, which is many orders of magnitude faster than the disk. Consequently, our first act in an analytics program is often to load some data from the disk into temporary memory. The memory is faster than the disk but smaller and disappears when the power goes off or your program terminates. The processor is still much faster than the memory but we have lots of tricks to increase the speed of communication between the processor and memory (e.g., caches).\n",
    "\n",
    "The processor is where all of the action occurs because it is the entity that executes the statements in a program. The operations in a program boil down to one of these fundamental instructions within the processor:\n",
    "\n",
    "* load small chunks of data from memory into the CPU\n",
    "* perform arithmetic computations on data in the CPU\n",
    "* store small chunks of data back to memory\n",
    "* conditionally perform computations\n",
    "* repeat operations\n",
    "\n",
    "In [Model of Computation](computation.md), we studied pseudocode that maps to one or more of these fundamental instructions. We saw how some of the [higher-level programming patterns](patterns.md) map down to pseudocode chosen from these fundamental instructions. We finished up by looking at some of the [low-level programming patterns](combinations.md) that combine fundamental instructions to do useful things like filtering and image processing.\n",
    "\n",
    "The act of translating a pseudocode operation into Python code involves choosing the right Python construct, just like programming involves choosing the right pattern to solve a piece of a real-world problem.  Then, it's a matter of shoehorning our free-form pseudocode into the straitjacket of programming language syntax. Before we get to those details, however, let's look at the big picture and a general strategy for writing programs.\n",
    "\n",
    "## Coding is similar to writing a paper\n",
    "\n",
    "Writing and executing a program are remarkably similar to writing and reading a paper or report. Just as with our program work plan, we begin writing a paper by clearly identifying a thesis or problem statement. Analogous to identifying input-output pairs, we might identify the target audience and what we hope readers will come away with after reading the paper. With this in mind, we should write an outline of the paper, which corresponds to identifying the processing steps in the program work plan. Sections and chapters in a paper might correspond to functions and packages in the programming world.\n",
    "\n",
    "When reading a paper, we read the sections and paragraphs in order, like a processor executes a program. The paper text can ask the reader to jump temporarily to a figure or different section and return. This is analogous to a program calling a function and returning, possibly with some information.  When reading a paper, we might also encounter conditional sections, such as \"*If you've studied quantum physics, you can skip this section*.\"  There can even be loops in a paper, such as \"*Now's a good time to reread the background section on linear algebra*.\"\n",
    "\n",
    "The point is that, if you've been taught how to properly write a paper, the process of writing code should feel very familiar. To simplify the process of learning to code in Python, we're going to restrict ourselves to a subset of the language and follow a few templates that will help us organize our programs.\n",
    "\n",
    "## A small introductory Python subset\n",
    "\n",
    "While I was in graduate school, I worked in Paris for six months (building an industrial robot control language). A friend, who didn't speak French, came over to work as well and got a tutor.  The tutor started him out with just the present tense, four verbs, and a few key nouns like *café* and *croissant*.   Moreover, the tutor gave him simple sentence templates in the French equivalent of these English templates:\n",
    "\n",
    "`_______ go _______.`\n",
    "\n",
    "and\n",
    "\n",
    "`I am _________.`\n",
    "\n",
    "that he could fill in with subjects and objects (nouns, pronouns).\n",
    "\n",
    "That's also an excellent approach for learning to code in a programming language.  We're going to start out playing around in a small sandbox, picking a simple subset of python that lets us do some interesting things. \n",
    "\n",
    "The \"nouns\" in this subset are numbers like `34` and `3.4`, strings like `parrt`, and lists of nouns like `[3,1.5,4]`.  We can name these values using *variables* just as we use names like Mary to refer to a specific human being. The \"verbs\", which act on nouns, are arithmetic operators like `cost + tax`, relational operators like `quantity<10`, and some built-in functions like `len(names)`.  We'll also use some sentence templates for conditional statements and loops. Finally, we'll also need the ability to pull in (`import`) code written by other programmers to help us out. It's like opening a cookbook that lets us refer to existing recipes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Values and variables\n",
    "\n",
    "The atomic elements in python, so to speak, are numbers and strings of text. We distinguish between integer and real numbers, which we call floating-point numbers, because computers represent the two internally in a different way. Here's where the *data type* comes back into play. Numbers like `34`, `-9`, and `0` are said to have type `int` whereas `3.14159` and `0.123` are type `float`. These values are called int or float *literals*.  Strings of text are just a sequence of characters in single quotes (there are more quoting options but we can ignore for now) and have type `string`.  For example, `'parrt'` and `'May 25, 1999'`. Note that the string representation of `'207'` is very different than the integer `207`. The former is a list with three characters and the latter is a numeric value that we could, say, multiply by 10.\n",
    "\n",
    "Let's look at our first python program!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hi\n"
     ]
    }
   ],
   "source": [
    "print 'hi'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That code is a kind of *statement* that instructs the computer to print the specified value to the computer screen (the console). So here is our first statement template:\n",
    "\n",
    "`print ___________`\n",
    "\n",
    "We can fill that hole with any kind of expression; right now, we only know about values (the simplest expressions) so we can do things like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "34\n",
      "3.14159\n"
     ]
    }
   ],
   "source": [
    "print 34\n",
    "print 3.14159"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the order of execution. The processor is executing one statement after the other.\n",
    "\n",
    "Instead of printing values to the screen, let's store values into memory through variable assignment statements. The statement template looks like:\n",
    "\n",
    "*variablename* `= __________`\n",
    "\n",
    "For example, we can store the value one into a variable called `count` and then reference that variable to load the data back from memory for use by a print statement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "count = 1\n",
    "print count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, the sequence matters. Putting the `print` before the assignment will cause an error because `count` is not defined as a variable until after the assignment.\n",
    "\n",
    "To see how things are stored in memory, let's look at three assignments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "count = 1\n",
    "name = 'iPhone'\n",
    "price = 699.99"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualize the state of computer memory after executing that program using [pythontutor.com](https://goo.gl/5kLND1). It shows a snapshot of memory like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(fyi, the \"Global frame\" holds what we call *global variables*. For now, everything will be globally visible and so we can just call them variables.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
{'fraction_non_alphanumeric': 0.10000993640699524, 'fraction_numerical': 0.007601351351351352, 'mean_word_length': 3.3232388316151202, 'pattern_counts': {'":': 124, '<': 6, '<?xml version=': 0, '>': 4, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 6, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3992605', 'n_tokens_mistral': 5663, 'n_tokens_neox': 5374, 'n_words': 2773}