<jupyter_start><jupyter_text>Jupyter Notebook ExampleCredit: This is slightly modified from examples used in the FSCI 2017 Computational Reproducibility Day (https://osf.io/sbnz7/), which was created by <NAME> and <NAME> from the Center for Open Science.   Setting up the notebook Lets get startedThe notebook is built up from separate editable areas, or cells.A new notebook contains a single *code* cell.Add a line of code and execute it by:* *clicking the run button*, or* click in the cell, and press shift-return<jupyter_code>print('hello world')<jupyter_output><empty_output><jupyter_text>Navigating and Selecting CellsTo select a cell, click on it. The selected cell will be surrounded by a box with the left hand side highlighted.Move the selection focus to the cell above/below using the keyboard up/down arrow keys.Additionally select adjacent cells using SHIFT-UP ARROW or SHIFT-DOWN ARROW. Managing Cells - Add, Delete, ReorderAdd a new cell to the notebook by:* click the + button on the toolbar* Insert -> Insert Cell Above or ESC-A* Insert -> Insert Cell Below or ESC-BDelete a cell by selecting it and:* click the scissors button on the toolbar* Edit -> Delete cells or ESC-XUndelete the last deleted cell:* Edit -> Undo Delete cells or ESC-ZEach cell has a cell history associated with it. Use CMD-Z to step back through previous cell contents.Reorder cells by:* moving them up and down the notebook using the up and down arrows on the toolbar* Edit -> Move Cell Up or Edit -> Move Cell Down* cutting and pasting them:    * Edit - >Cut or Edit->Paste Cells Above or Edit->Paste Cells Below on the toolbarYou can also copy selected cells from the toolbar, Edit -> Copy Cells or ESC-C. More resources[More stuff you can do in notebooks](https://towardsdatascience.com/how-to-effortlessly-optimize-jupyter-notebooks-e864162a06ee)  About Libraries in PythonLets use our first code cell to import a library. A library in Python contains a set of tools (called functions) that perform tasks on our data. Importing a library is like getting a piece of lab equipment out of a storage locker and setting it up on the bench for use in a project. Once a library is imported, it can be used or called to perform many tasks.Python doesn’t load all of the libraries available to it by default. We have to add an import statement to our code in order to use library functions. To import a library, we use the syntax `import libraryName`. If we want to give the library a nickname to shorten the command, we can add `as nickNameHere`. An example of importing the Pandas library using the common nickname `pd` is below.**`import`** `pandas` **`as`** `pd` The Pandas LibraryOne of the best options for working with tabular data in Python is the Python Data Analysis Library (a.k.a. Pandas). The Pandas library is built on top of the NumPy package (another Python library). Pandas provides data structures, produces high quality plots with matplotlib, and integrates nicely with other libraries that use NumPy arrays. Those familiar with spreadsheets should become comfortable with Pandas data structures.<jupyter_code>import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>Each time we call a function that’s in a library, we use the syntax `LibraryName.FunctionName`. Adding the library name with a `.` before the function name tells Python where to find the function. In the example above, we have imported Pandas as `pd`. This means we don’t have to type out `pandas` each time we call a Pandas function.See this free [Pandas cheat sheet](https://www.datacamp.com/community/blog/python-pandas-cheat-sheet) from DataCamp for the most common Pandas commands.   MarkdownWe're seen how we can have coding cells and show their output below them, but what about that plain language I mentioned? We can added another type of cell, a Markdown cell that contains narrative text. Markdown is a popular markup language that is a superset of HTML. To learn more, see [Jupyter's Markdown guide](http://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Working%20With%20Markdown%20Cells.html) or revisit the [Reproducible Research lesson on Markdown](https://github.com/Reproducible-Science-Curriculum/introduction-RR-Jupyter/blob/master/notebooks/Navigating%20the%20notebook%20-%20instructor%20script.ipynb).Lets add a markdown cell above our library imports. Do to this:* Change the cell type using the drop down list in the toolbar, or by using the ESC-M keyboard shortcut.* To "open" or select a markdown cell for editing, double click the cell.* View the rendered markdown by running the cell:Markdown cells can contain:* headings    Prefix a line of text in a markdown cell by one or more  signs, followed by a space, to specify the level of the heading required. Heading 1 Heading 2... Heading 6  Getting data into the notebookWe will begin by locating and reading our data which are in a table format as a tab-delimited file. We will use Pandas’ `read_csv` function to pull the file directly into a `DataFrame`. What’s a `DataFrame`?A `DataFrame` is a 2-dimensional data structure that can store in columns data of different types (including characters, integers, floating point values, factors and more). It is similar to a spreadsheet or a SQL table or data.frame in R. A `DataFrame` always has an index (0-based). An index refers to the position of an element in the data structure.Note that we use `pd.read_csv`, not just `read_csv` or `pandas.read_csv`, because we imported Pandas as `pd`.In our original file, the columns in the data set are separated by a TAB. We need to tell the `read_csv` function in Pandas that that is the delimiter with `sep = ‘\t’`.<jupyter_code>url = "https://osf.io/z274d/download"
#You can also read your table in from a file directory
gapminder = pd.read_csv(url, sep = "\t")<jupyter_output><empty_output><jupyter_text>The first thing to do when loading data into the notebook is to actually "look" at it.  How many rows and columns are there?  What types of variables are in it and what values can they take?There are usually too many rows to print to the screen.  By default, when you type the name of the `DataFrame` and run a cell, Pandas knows to not print the whole thing.  Instead, you will see the first and last few rows with dots in between.  A neater way to see a preview of the dataset is the `head()` method.  Calling `dataset.head()` will display the first 5 rows of the data.  You can specify how many rows you want to see as an argument, like `dataset.head(10)`.  The `tail()` method does the same with the last rows of the `DataFrame`.<jupyter_code>gapminder.head()<jupyter_output><empty_output><jupyter_text>Sometimes the table has too many columns to print on screen. Calling `df.columns.values` will print all the column names in an array.<jupyter_code>gapminder.columns.values<jupyter_output><empty_output><jupyter_text>Assess the structure and cleanliness How many rows and columns are in the data?We often want to know how many rows and columns are in the data -- what is the "shape" of the `DataFrame`. Shape is an attribute of the `DataFrame`. Pandas has a convenient way for getting that information by using `DataFrame.shape`  (using `DataFrame` here as a generic name for your `DataFrame`). This returns a tuple (immutable values separated by commas) representing the dimensions of the `DataFrame` (rows, columns).To get the shape of the gapminder `DataFrame`:<jupyter_code>gapminder.shape<jupyter_output><empty_output><jupyter_text>We can learn even more about our `DataFrame`. The `info()` method gives a few useful pieces of information, including the shape of the `DataFrame`, the variable type of each column, and the amount of memory stored.The output from `info()` displayed below shows that the fields ‘year’ and ‘pop’ (population) are represented as ‘float’ (that is: numbers with a decimal point). This is not appropriate: year and population should be integers or whole numbers. We can change the data-type with the function `astype()`. The code for `astype()` is shown below; however, we will change the data types later in this lesson.<jupyter_code>gapminder.info()<jupyter_output><empty_output><jupyter_text>The `describe()` method will take the numeric columns and provide a summary of their values. This is useful for getting a sense of the ranges of values and seeing if there are any unusual or suspicious numbers.<jupyter_code>gapminder.describe()<jupyter_output><empty_output><jupyter_text>The `DataFrame` function `describe()` just blindly looks at all numeric variables. We wouldn't actually want to take the mean year. Additionally, we obtain ‘NaN’ values for our quartiles. This suggests we might have missing data which we can (and will) deal with shortly when we begin to clean our data.For now, let's pull out only the columns that are truly continuous numbers (i.e. ignore the description for ‘year’). This is a preview of selecting columns from the data; we'll talk more about how to do it later in the lesson.<jupyter_code>gapminder[['pop', 'lifeexp', 'gdppercap']].describe()<jupyter_output><empty_output><jupyter_text>Data cleaning  Referencing objects vs copying objectsBefore we get started with cleaning our data, let's practice good data hygiene by first creating a copy of our original data set. Often, you want to leave the original data untouched.  To protect your original, you can make a copy of your data (and save it to a new `DataFrame` variable) before operating on the data or a subset of the data.  This will ensure that a new version of the original data is created and your original is preserved. Why this is importantSuppose you take a subset of your `DataFrame` and store it in a new variable, like `gapminder_early = gapminder[gapminder['year'] < 1970]`.  Doing this does not actually create a new object. Instead, you have just given a name to that subset of the original data: `gapminder_early`. This subset still points to the original rows of `gapminder`.  Any changes you make to the new `DataFrame` `gapminder_early` will appear in the corresponding rows of your original `gapminder` `DataFrame` too.<jupyter_code>gapminder = pd.read_csv(url, sep = "\t")
gapminder_copy = gapminder.copy()
gapminder_copy.head()<jupyter_output><empty_output><jupyter_text>Handling Missing DataMissing data (often denoted as 'NaN'- not a number- in Pandas, or as 'null') is an important issue to handle because Pandas cannot compute on rows or columns with missing data. 'NaN' or 'null' does not mean the value at that position is zero, it means that there is no information at that position. Ignoring missing data doesn't make it go away. There are different ways of dealing with it which include:* analyzing only the available data (i.e. ignore the missing data)* input the missing data with replacement values and treating these as though they were observed* input the missing data and account for the fact that these were inputed with uncertainty (ex: create a new boolean variable so you know that these values were not actually observed)* use statistical models to allow for missing data--make assumptions about their relationships with the available data as necessaryFor our purposes with the dirty gapminder data set, we know our missing data is excess (and unnecessary) and we are going to choose to analyze only the available data. To do this, we will simply remove rows with missing values.This is incredibly easy to do because Pandas allows you to either remove all instances with null data or replace them with a particular value.`df = df.dropna()` drops rows with any column having NA/null data.  `df = df.fillna(value)` replaces all NA/null data with the argument `value`.For more fine-grained control of which rows (or columns) to drop, you can use `how` or `thresh`. These are more advanced topics and are not covered in this lesson; you are encouraged to explore them on your own.<jupyter_code>gapminder_copy = gapminder_copy.dropna()
gapminder_copy.head()<jupyter_output><empty_output><jupyter_text>Changing Data TypesWe can change the data-type with the function `astype()`. The code for `astype()` is shown below.  SubsettingWe can subset (or slice) by giving the numbers of the rows you want to see between square brackets.*REMINDER:* Python uses 0-based indexing. This means that the first element in an object is located at position 0. this is different from other tools like R and Matlab that index elements within objects starting at 1.<jupyter_code>gapminder_copy[0:15]
#Select the first 15 rows
gapminder_copy[:15]
#Select the last 10 rows
gapminder_copy[-10:]<jupyter_output><empty_output><jupyter_text>Subsetting can also be done by selecting for a particular column or for a particular value in a column; for instance select the rows that have ‘africa’ in the column ‘continent. Note the double equal sign: single equal signs are used in Python to assign something to a variable. The double equal sign is a comparison: the variable to the left has to be exactly equal to the string to the right.<jupyter_code>#Select for a particular column
gapminder_copy['year']

#this syntax, calling the column as an attribute, gives you the same output
gapminder_copy.year<jupyter_output><empty_output><jupyter_text>Summarize and plotSummaries (but can’t *say* statistics…)* Sort data* Can make note about using numpy functions, dif between `DataFrame` and `array`Good Plots for the data/variable typePlots * of subsets, * single variables* pairs of variables* Matplotlib syntax (w/ Seaborn for defaults (prettier, package also good for more analysis later...))Exploring is often iterative - summarize, plot, summarize, plot, etc. - sometimes it branches…  Summarizing dataRemember that the `info()` method gives a few useful pieces of information, including the shape of the `DataFrame`, the variable type of each column, and the amount of memory stored. We can see many of our changes (continent and country columns instead of region, higher number of rows, etc.) reflected in the output of the `info()` method.<jupyter_code>gapminder_copy.info()<jupyter_output><empty_output><jupyter_text>We also saw above that the `describe()` method will take the numeric columns and give a summary of their values. We have to remember that we changed the column names and this time it shouldn't have NaNs.<jupyter_code>gapminder_copy[['pop', 'lifeexp', 'gdppercap']].describe()<jupyter_output><empty_output><jupyter_text>More summariesWhat if we just want a single value, like the mean of the population? We can call mean on a single column this way:<jupyter_code>gapminder_copy['pop'].mean()<jupyter_output><empty_output><jupyter_text>What if we want to know the mean population by _continent_? Then we need to use the Pandas `groupby()` method and tell it which column we want to group by.<jupyter_code>gapminder_copy[['continent', 'pop']].groupby(by='continent').mean()<jupyter_output><empty_output><jupyter_text>What if we wanted a new `DataFrame` that just contained these summaries? This could be a table in a report, for example.<jupyter_code>continent_mean_pop = gapminder_copy[['continent', 'pop']].groupby(by='continent').mean()
continent_mean_pop = continent_mean_pop.rename(columns = {'pop':'meanpop'})
continent_row_ct = gapminder_copy[['continent', 'country']].groupby(by='continent').count()
continent_row_ct = continent_row_ct.rename(columns = {'country':'nrows'})
continent_median_pop = gapminder_copy[['continent', 'pop']].groupby(by='continent').median()
continent_median_pop = continent_median_pop.rename(columns = {'pop':'medianpop'})
gapminder_summs = pd.concat([continent_row_ct,continent_mean_pop,continent_median_pop], axis=1)
gapminder_summs = gapminder_summs.rename(columns = {'y':'year'})
gapminder_summs<jupyter_output><empty_output><jupyter_text>Visualization with `matplotlib`[matplotlib](http://matplotlib.org) is Python's main visualization library. It provides a range of tools for constructing plots and numerous high-level plotting libraries (e.g., [Seaborn](http://seaborn.pydata.org)) are built with matplotlib in mind. When we were in the early stages of setting up our analysis, we loaded these libraries like so:<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
sns.set()<jupyter_output><empty_output><jupyter_text>*Consider the above three commands to be essential practice for plotting (asessential as **`import`** `pandas` **`as`** `pd` is for data munging).*Now, let's turn to data visualization. In order to get a feel for the propertiesof the data set we are working with, data visualization is key. While, we willfocus only on the essentials of how to properly construct plots in univariateand bivariate settings here, it's worth noting that both matplotlib and Seabornsupport a diversity of plots: [matplotlib gallery](http://matplotlib.org/gallery.html), [Seaborngallery](http://seaborn.pydata.org/examples/). ---  Single variables* __Histograms__ - provide a quick way of visualizing the distribution of numerical  data, or the frequencies of observations for categorical variables.<jupyter_code>#import numpy as npa
plt.hist(gapminder_copy['lifeexp'])
plt.xlabel('lifeexp')
plt.ylabel('count')<jupyter_output><empty_output><jupyter_text>Hmmm, something does look right.  Let's check the data.<jupyter_code>gapminder_copy['lifeexp']
gapminder_copy['lifeexp'].describe()<jupyter_output><empty_output><jupyter_text>Ok, it seems that there's a 999999 value that was used to indicate **something**, and it's throwing off the histogram. So let's remove rows that don't have a life expentency between 0 and 100, and then try again<jupyter_code>gapminder_clean = gapminder_copy[gapminder_copy['lifeexp'].between(0, 100)]
gapminder_clean['lifeexp'].describe()
plt.hist(gapminder_clean['lifeexp'])
plt.xlabel('lifeexp')
plt.ylabel('count')<jupyter_output><empty_output><jupyter_text>* __Boxplots__ - provide a way of comparing the summary measures (e.g., max, min,  quartiles) across variables in a data set. Boxplots can be particularly useful with larger data sets.---<jupyter_code>sns.boxplot(x='year', y='lifeexp', data = gapminder_clean)
plt.xlabel('year')
plt.ylabel('lifeexp')<jupyter_output><empty_output><jupyter_text>Pairs of variables* __Scatterplots__ - visualization of relationships across two variables...<jupyter_code># example plot goes here

plt.scatter(gapminder_clean['gdppercap'], gapminder_clean['lifeexp'])
plt.xlabel('gdppercap')
plt.ylabel('lifeexp')
plt.scatter(gapminder_clean['gdppercap'], gapminder_clean['lifeexp'])
plt.xscale('log')
plt.xlabel('gdppercap')
plt.ylabel('lifeexp')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07491158503911692, 'fraction_numerical': 0.003911692208766478, 'mean_word_length': 5.482459187217784, 'pattern_counts': {'":': 0, '<': 100, '<?xml version=': 0, '>': 109, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '251873', 'license': 'MIT', 'n_tokens_mistral': 5305, 'n_tokens_neox': 5039, 'n_words': 2464, 'path': 'Notebook_Demonstration.ipynb', 'repo_name': 'danielskatz/repro-fdtd1d', 'size': '18682'}