<jupyter_start><jupyter_text>First Python Notebook: Scripting your way to the storyBy [<NAME>](http://palewi.re/who-is-ben-welsh/)A step-by-step guide to analyzing data with Python and the Jupyter Notebook.This tutorial will teach you how to use computer programming tools to analyze data by exploring contributors to campaigns for and again Proposition 64, a ballot measure asking California voters to decide if recreational marijuana should be legalized.This guide was developed by <NAME> for a [Oct. 2, 2016, "watchdog workshop" organized by Investigative Reporters and Editors](http://ire.org/events-and-training/event/2819/2841/) at San Diego State University's school of journalism. The class is designed for beginners who have zero Python experience.  Prelude: Prequisites Before you can begin, your computer needs the following tools installed and working to participate.1. A [command-line interface](https://en.wikipedia.org/wiki/Command-line_interface) to interact with your computer2. Version 2.7 of the [Python](http://python.org/download/releases/2.7.6/) programming language3. The [pip](https://pip.pypa.io/en/latest/installing.html) package manager and [virtualenv](http://www.virtualenv.org/en/latest/) environment manager for Python  Command-line interfaceUnless something is wrong with your computer, there should be a way to open a window that lets you type in commands. Different operating systems give this tool slightly different names, but they all have some form of it, and there are alternative programs you can install as well.On Windows you can find the command-line interface by opening the "command prompt." Here are instructions for [Windows 10](http://www.howtogeek.com/235101/10-ways-to-open-the-command-prompt-in-windows-10/) and for [Windows 8](http://windows.microsoft.com/en-us/windows/command-prompt-faq1TC=windows-8) and [earlier versions](http://windows.microsoft.com/en-us/windows-vista/open-a-command-prompt-window). On Apple computers, you open the ["Terminal" application](http://blog.teamtreehouse.com/introduction-to-the-mac-os-x-command-line). Ubuntu Linux comes with a program of the [same name](http://askubuntu.com/questions/38162/what-is-a-terminal-and-how-do-i-open-and-use-it).  PythonIf you are using Mac OSX or a common flavor of Linux, Python version 2.7 is probably already installed and you can test to see what version, if any, is already available by typing the following into your terminal.```bashpython -V```Even if you find it already on your machine, Mac users should install it separately by following [these instructions](http://docs.python-guide.org/en/latest/starting/install/osx/) offered by The Hitchhikers Guide to Python. Windows people can find a similar guide [here](http://docs.python-guide.org/en/latest/starting/install/win/) which will have them try downloading and installing Python from [here](https://www.python.org/downloads/release/python-2712/).  pip and virtualenvThe [pip package manager](https://pip.pypa.io/en/latest/) makes it easy to install open-source libraries that expand what you're able to do with Python. Later, we will use it to install everything needed to create a working web application.If you don't have it already, you can get pip by following [these instructions](https://phttps://pip.pypa.io/en/latest/ip.pypa.io/en/latest/installing.html). In Windows, it's necessary to make sure that the Python ``Scripts`` directory is available on your system's ``PATH`` so it can be called from anywhere on the command line. [This screencast](http://showmedo.com/videotutorials/video?name=960000&fromSeriesID=96) can help.Verify pip is installed with the following.```bashpip -V```The [virtualenv environment manager](http://www.virtualenv.org/en/latest/) makes it possible to create an isolated corner of your computer where all the different tools you use to build an application are sealed off.It might not be obvious why you need this, but it quickly becomes important when you need to juggle different toolsfor different projects on one computer. By developing your applications inside separate virtualenv environments, you can use different versions of the same third-party Python libraries without a conflict. You can also more easily recreate your project on another machine, handy when you want to copy your code to a server that publishes pages on the Internet.You can check if virtualenv is installed with the following.```bashvirtualenv --version```If you don't have it, install it with pip.```bashpip install virtualenv If you're on a Mac or Linux and get an error saying you lack permissions, try again as a superuser.sudo pip install virtualenv```If that doesn't work, [try following this advice](http://virtualenv.readthedocs.org/en/latest/installation.html).  Act 1: Hello Jupyter Notebook Start by creating a new development environment with virtualenv in your terminal. Name it after our application.```bashvirtualenv first-python-notebook``` Jump into the directory it created.```bashcd first-python-notebook``` Turn on the new virtualenv, which will instruct your terminal to only use those libraries installedinside its sealed space. You only need to create the virtualenv once, but you'll need to repeat these"activation" steps each time you return to working on this project.```bash In Linux or Mac OSX try this.... bin/activate In Windows it might take something more like...cd Scriptsactivatecd ..``` Use ``pip`` on the command line to install [Jupyter Notebook](http://jupyter.org/), an open-source tool for writing and sharing Python scripts.```bashpip install jupyter``` Start up the notebook from your terminal.```bashjupyter notebook``` That will open up a new tab in your default web browser that looks something like this:![](http://jupyter.readthedocs.io/en/latest/_images/tryjupyter_file.png) Click the "New" button in the upper right and create a new Python 2 notebook. Now you're all setup and ready to start writing code.  Act 2: Hello Python You are now ready to roll within the Jupyter Notebook's framework for writing Python. Don't stress. There's nothing too fancy about it. You can start by just doing a little simple math. Type the following into the first box, then hit the play button in the toolbox (or hit SHIFT+ENTER on your keyboard).<jupyter_code>2+2<jupyter_output><empty_output><jupyter_text>There. You've just written your first Python code. You've entered two integers (the 2's) and added them together using the plus sign operator. Not so bad, right? Next, let's introduce one of the basics of computer programming, a variable.Variables are like containers that hold different types of data so you can go back and refer to them later. They’re fundamental to programming in any language, and you’ll use them all the time when you're writing Python.Move down to the next box. Now let's put that number two into our first variable.<jupyter_code>san = 2<jupyter_output><empty_output><jupyter_text>In this case, we’ve created a variable called ``san`` and assigned it the integer value 2.In Python, variable assignment is done with the = sign. On the left is the name of the variable you want to create (it can be anything) and on the right is the value that you want to assign to that variable.If we use the ``print`` command on the variable, Python will output its contents to the terminal because that value is stored in the variable. Let's try it.<jupyter_code>print san<jupyter_output>2<jupyter_text>We can do the same thing again with a different variable name<jupyter_code>diego = 2<jupyter_output><empty_output><jupyter_text>Then add those two together the same way we added the numbers at the top.<jupyter_code>san + diego<jupyter_output><empty_output><jupyter_text>Variables can contain many different kinds of data types. There are integers, strings, floating point numbers (decimals), lists and dictionaries.<jupyter_code>string = "Hello"
decimal = 1.2
list_of_strings = ["a", "b", "c", "d"]
list_of_integers = [1, 2, 3, 4]
list_of_whatever = ["a", 2, "c", 4]
my_phonebook = {'Mom': '713-555-5555', 'Chinese Takeout': '573-555-5555'}<jupyter_output><empty_output><jupyter_text>Playing with data we invent can be fun, but it's a long way from investigative journalism. Now's the time for us to get our hands on some real data and get some real work done. Your assignment: Proposition 64. The use and sale of marijuana for recreational purposes is illegal in California. [Proposition 64](http://www.oag.ca.gov/system/files/initiatives/pdfs/15-0103%20%28Marijuana%29_1.pdf), scheduled to appear on the November 2016 ballot, asked voters if it ought to be legalized. A "yes" vote would support legalization. A "no" vote would oppose it. A similar measure, [Proposition 19](http://articles.latimes.com/print/2010/nov/03/local/la-me-pot-20101103-1), was defeated in 2010.[According to California's Secretary of State](http://www.sos.ca.gov/campaign-lobbying/cal-access-resources/measure-contributions/marijuana-legalization-initiative-statute/), more than $16 million was been raised to campaign in support of Prop. 64 as of September 20. Just over 2 million was been raised to oppose it.Your mission, should you choose to accept it, is to download a list of campaign contributors and figure out the biggest donors both for and against the measure.[Click here](https://docs.google.com/spreadsheets/d/1Zsxlq01Wqu9D1qLLesjA7aGwclYTGeNsPY4Ax_jwtYI/pub?gid=0&single=true&output=csv) to download the file as a list of comma-separated values. This is known as a CSV file. It is the most common way you will find data published online. Save the file with the name ``first-python-notebook.csv`` in the same directory where you made this notebook.  Python can read files using the built-in ``open`` function. You feed two things into it: 1) The path to the file; 2) What type of operation you'd like it to execute on the file. "r" stands for read.<jupyter_code>data_file = open("./first-python-notebook.csv", "r")<jupyter_output><empty_output><jupyter_text>Print that variable and you see that ``open`` has created a file "object" that offers a number of different ways to interact with the contents of the file.<jupyter_code>print data_file<jupyter_output><open file './first-python-notebook.csv', mode 'r' at 0x7fd6ce25b4b0><jupyter_text>One thing a file object can do is read in all of the data from the file. Let's do that next and store the contents in a new variable.<jupyter_code>data = data_file.read()
print data<jupyter_output>FILING_ID,COMMITTEE_ID,COMMITTEE_NAME,COMMITTEE_POSITION,AMEND_ID,FIRST_NAME,LAST_NAME,CITY,STATE,ZIPCODE,EMPLOYER,OCCUPATION,DATE,AMOUNT
1680042,1343793,"Californians for Responsible Marijuana Reform, Sponsored by Drug Policy Action, Yes on Prop. 64",SUPPORT,0,,Drug Policy Action Committee to Tax and Regulate Marijuana,Sacramento,CA,95815,,,1/3/2012 12:00:00 AM,10975.5
1937009,1343793,"Californians for Responsible Marijuana Reform, Sponsored by Drug Policy Action, Yes on Prop. 64",SUPPORT,0,Luke,Sinquefield,Pacific Palisades,CA,90272,Luke Sinquefield,Realtor,7/25/2014 12:00:00 AM,15000
2038717,1343793,"Californians for Responsible Marijuana Reform, Sponsored by Drug Policy Action, Yes on Prop. 64",SUPPORT,1,Susan,Podolsky,Modesto,CA,95350,Ontario Base Hospital Group,Physician,3/21/2016 12:00:00 AM,100
2038717,1343793,"Californians for Responsible Marijuana Reform, Sponsored by Drug Policy Action, Yes on Prop. 64",SUPPORT,1,Philip,Davis,Salinas,CA,93901,Monterey County Office of Ed[...]<jupyter_text>That's all good, but the data is printing out as one big long string. If we're going to do some real analysis, we need Python to recognize and respect the structure of our data, in the way an Excel spreadsheet would.To do that, we're going to need something smarter than ``open``. We're going to need something like pandas.  Act 3: Hello pandas  Lucky for us, Python already has tools filled with functions to do pretty much anything you’d ever want to do with a programming language: navigate the web, parse data, interact with a database, run fancy statistics, build a pretty website and so much more.Some of those tools are included a toolbox that comes with the language, known as the standard library. Others have been built by members of Python's developer community and need to be downloaded and installed from the web.For this exercise, we're going to install and use [pandas](http://pandas.pydata.org/), a tool developed by a financial investment firm that has become the leading open-source tool for accessing and analyzing data.There are several others we could use instead (like [agate](http://agate.readthedocs.io/en/1.4.0/)) but we're picking pandas here because it's the most popular and powerful. We'll install pandas the same way we installed the Jupyter Notebook earlier: Our friend ``pip``. Save your notebook, switch to your window/command prompt and hit CTRL-C. That will kill your notebook and return you to the command line. There we'll install pandas.```bashpip install pandas``` Now let's restart our notebook and get back to work.```bashjupyter notebook``` Use the next open box to import pandas into our script, so we can use all its fancy methods here in our script.<jupyter_code>import pandas<jupyter_output><empty_output><jupyter_text>Opening our CSV isn't any harder than with ``open``, you just need to know the right trick to make it work.<jupyter_code>pandas.read_csv("./first-python-notebook.csv")<jupyter_output><empty_output><jupyter_text>Great. Now let's do it again and assign it to a variable this time<jupyter_code>table = pandas.read_csv("./first-python-notebook.csv")<jupyter_output><empty_output><jupyter_text>Now let's see what that returns when we print it.<jupyter_code>table.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 318 entries, 0 to 317
Data columns (total 14 columns):
FILING_ID             318 non-null int64
COMMITTEE_ID          318 non-null int64
COMMITTEE_NAME        318 non-null object
COMMITTEE_POSITION    318 non-null object
AMEND_ID              318 non-null int64
FIRST_NAME            269 non-null object
LAST_NAME             318 non-null object
CITY                  318 non-null object
STATE                 318 non-null object
ZIPCODE               318 non-null object
EMPLOYER              257 non-null object
OCCUPATION            265 non-null object
DATE                  318 non-null object
AMOUNT                318 non-null float64
dtypes: float64(1), int64(3), object(10)
memory usage: 34.9+ KB<jupyter_text>Here's how you can see the first few rows<jupyter_code>table.head()<jupyter_output><empty_output><jupyter_text>How many rows are there? Here's how to find out.<jupyter_code>print len(table)<jupyter_output>318<jupyter_text>Even with that simple question and answer, we've begun the process of interviewing our data.In some ways, your database is no different from a human source. Getting a good story requires careful, thorough questioning.In the next section we will move ahead by conducting an interview with pandas to pursue our quest of finding out the biggest donors to Proposition 64.  Act 4: Hello analysis Let's start with something easy. What are the ten biggest contributions? That will require a sort using the column with the money in it.<jupyter_code>table.sort_values("AMOUNT")<jupyter_output><empty_output><jupyter_text>We've got it sorted the wrong way. Let's reverse it.<jupyter_code>table.sort_values("AMOUNT", ascending=False)<jupyter_output><empty_output><jupyter_text>Now let's limit it to the top 10.<jupyter_code>table.sort_values("AMOUNT", ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>What is the total sum of contributions that have been reported? First, let's get our hands on the column with our numbers in it. In pandas you can do that like so.<jupyter_code>table['AMOUNT']<jupyter_output><empty_output><jupyter_text>Now adding it up is this easy.<jupyter_code>table['AMOUNT'].sum()<jupyter_output><empty_output><jupyter_text>There's our big total. Why is it lower than the ones I quoted above? That's because campaigns are only required to report the names of donors over $200, so our data is missing all of the donors who gave smaller amounts of money.The overall totals are reported elsewhere in lump sums and cannot be replicated by adding up the individual contributions. Understanding this is crucial to understanding not just this data, but all campaign finance data, which typically has this limitation.  Filtering Adding up a big total is all well and good. But we're aiming for something more nuanced. We want to separate the money for the proposition from the money against it. To do that, we'll need to learn how to filter. First let's look at the column we're going to filter by<jupyter_code>table['COMMITTEE_POSITION']<jupyter_output><empty_output><jupyter_text>Now let's filter using that column using pandas oddball method<jupyter_code>table[table['COMMITTEE_POSITION'] == 'SUPPORT']<jupyter_output><empty_output><jupyter_text>Stick that in a variable<jupyter_code>support_table = table[table['COMMITTEE_POSITION'] == 'SUPPORT']<jupyter_output><empty_output><jupyter_text>So now we can ask: How many contributions does the supporting side have?<jupyter_code>print len(support_table)<jupyter_output>280<jupyter_text>Next: What are the 10 biggest supporting contributions?<jupyter_code>support_table.sort_values("AMOUNT", ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>Now let's ask the same questions of the opposing side.<jupyter_code>oppose_table = table[table['COMMITTEE_POSITION'] == 'OPPOSE']
print len(oppose_table)
oppose_table.sort_values("AMOUNT", ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>How about the sum total of contributions for each?<jupyter_code>support_table['AMOUNT'].sum()
oppose_table['AMOUNT'].sum()<jupyter_output><empty_output><jupyter_text>Grouping One thing we noticed as we explored the data is that there are a lot of different committees. A natural question follows: Which ones have raised the most money?To figure that out, we'll need to group the data by that column and sum up the amount column for each. Here's how pandas does that.<jupyter_code>table.groupby("COMMITTEE_NAME")['AMOUNT'].sum()<jupyter_output><empty_output><jupyter_text>Wow. That's pretty ugly. Why? Because pandas is weird. To convert a raw dump like that into a clean table (known in pandas slang as a "dataframe") you can add the ``reset_index`` method to the end of your code. Don't ask why.<jupyter_code>table.groupby("COMMITTEE_NAME")['AMOUNT'].sum().reset_index()<jupyter_output><empty_output><jupyter_text>Now let's sort it by size<jupyter_code>table.groupby("COMMITTEE_NAME")['AMOUNT'].sum().reset_index().sort_values("AMOUNT", ascending=False)<jupyter_output><empty_output><jupyter_text>Okay. Committees are good. But what about something a little more interesting. **Who** has given the most money?To do that, we'll group by the two name columns at the same time.<jupyter_code>table.groupby(["FIRST_NAME", "LAST_NAME"])['AMOUNT'].sum().reset_index().sort_values("AMOUNT", ascending=False)<jupyter_output><empty_output><jupyter_text>But which side where they are? Add in the position column to see that too.<jupyter_code>table.groupby([
    "FIRST_NAME",
    "LAST_NAME",
    "COMMITTEE_POSITION"
])['AMOUNT'].sum().reset_index().sort_values("AMOUNT", ascending=False)<jupyter_output><empty_output><jupyter_text>Pretty cool, right? Now now that we've got this interesting list of people, let's see if we can make a chart out of it.  Act 5: Hello viz Python has a number of charting tools that can work hand in hand with pandas. The most popular is [matplotlib](http://matplotlib.org/). It isn't the prettiest thing in the world, but it offers some reasonably straightfoward tools for making quick charts. And, best of all, it can display right here in our Jupyter Notebook. Before we start, we'll need to make sure matplotlib is installed. Return to your terminal and try installing it with our buddy pip, as we installed other things before. ```bash    pip install matplotlib``` Once you've got it in here, you can import it just as we would anything else. Though by adding the optional ``as`` option at the end we can create a shorter alias for accessing its tools.<jupyter_code>import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Before we'll get started, let's run one more trick to configure matplotlib to show its charts in our notebook.<jupyter_code>%matplotlib inline<jupyter_output><empty_output><jupyter_text>Now let's save the data we want to chart into a variable<jupyter_code>top_supporters = support_table.groupby(
    ["FIRST_NAME", "LAST_NAME"]
)['AMOUNT'].sum().reset_index().sort_values("AMOUNT", ascending=False).head(10)
top_supporters<jupyter_output><empty_output><jupyter_text>Making a quick bar chart is as easy as this.<jupyter_code>top_supporters['AMOUNT'].plot.bar()
top_supporters['AMOUNT'].plot.barh()<jupyter_output><empty_output><jupyter_text>It's really those first five that are the most interesting, so let's trim our chart.<jupyter_code>top_supporters.head(5)['AMOUNT'].plot.barh()<jupyter_output><empty_output><jupyter_text>What are those y axis labels? Those are the row number (pandas calls them indexes) of each row. We don't want that. We want the names.<jupyter_code>chart = top_supporters.head(5)['AMOUNT'].plot.barh()
chart.set_yticklabels(top_supporters['LAST_NAME'])<jupyter_output><empty_output><jupyter_text>Okay, but what if I want to combine the first and last name?First, make a new column. First let's look at what we have now.<jupyter_code>top_supporters.head(5)<jupyter_output><empty_output><jupyter_text>In plain old Python, we created a string at the start of our less. Remember this?<jupyter_code>print string<jupyter_output>Hello<jupyter_text>Combining strings can be as easy as addition.<jupyter_code>print string + "World"<jupyter_output>HelloWorld<jupyter_text>And if we want to get a space in there yet we can do something like:<jupyter_code>print string + " " + "World"<jupyter_output>Hello World<jupyter_text>And guess what we can do the same thing with two columns in our table, and use a pandas trick that will apply it to every row.<jupyter_code>top_supporters['FULL_NAME'] = top_supporters['FIRST_NAME'] + " " + top_supporters['LAST_NAME']<jupyter_output><empty_output><jupyter_text>Now let's see the results<jupyter_code>top_supporters.head()<jupyter_output><empty_output><jupyter_text>Now let's chart that.<jupyter_code>chart = top_supporters.head(5)['AMOUNT'].plot.barh()
chart.set_yticklabels(top_supporters['FULL_NAME'])<jupyter_output><empty_output><jupyter_text>That's all well and good, but this chart is pretty ugly. If you wanted to hand this data off to your graphics department, or try your hand at a simple chart yourself using something like [Chartbuilder](https://quartz.github.io/Chartbuilder/), you'd need to export this data into a spreadsheet. It's this easy.<jupyter_code>top_supporters.head(5).to_csv("top_supporters.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08511927956011603, 'fraction_numerical': 0.01753474477204832, 'mean_word_length': 5.575007116424708, 'pattern_counts': {'":': 0, '<': 180, '<?xml version=': 0, '>': 180, 'https://': 8, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '46', 'dir': 'jupyter-structured-clean-dedup', 'id': '362843', 'license': 'MIT', 'n_tokens_mistral': 7093, 'n_tokens_neox': 6521, 'n_words': 2685, 'path': 'tutorials/first-python-notebook.ipynb', 'repo_name': 'california-civic-data-coalition/python-calacess-notebooks', 'size': '23103'}