<jupyter_start><jupyter_text>This notebook will be a short review of key concepts in python. The goal of this notebook is to jog your memory and refresh concepts.   Table of contents* Jupyter notebook* Libraries* Plotting* Pandas DataFrame manipulation* Unit testing* Randomness and reproducibility* Bonus: list comprehension  Jupyter notebookStraight from the [Juptyer website](http://jupyter.org/): "The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more."  To run code in a cell, you can press the run tab, or press control + enter. The 'Kernel' tab is quite useful. If you find your code is stuck running (maybe you wrote an infinite loop), you can go to 'Kernel' -> 'Interrupt' to force quit. A few useful keyboard shortcuts:* Run cell, select below: shift + enter* Run cell: ctrl + enter* Run cell, insert below: option + enterBy pressing the 'esc' key, you enter command mode (the colored border around the currently selected cell should change from green to blue). Once in command mode, you can use these shortcuts:* Insert cell above: a* Insert cell below: b* Copy cell: c* Paste cell: v* Delete selected cell(s): d d* Change selected cell to markdown: m* Change selected cell to code: yTo exit command mode, click anywhere in a cell or press enter.And of course, don't forget the ever useful:* Save file: command + s  LibrariesThere are a few libraries that you will use almost always:* Numpy* Pandas* Matplotlib or* SeabornThe key points to remember are how to import these libraries and their standard import names, as well as their main uses. Numpy (from the [Numpy website](http://www.numpy.org/)):  NumPy is the fundamental package for scientific computing with Python. It contains among other things:* a powerful N-dimensional array object* sophisticated (broadcasting) functions* tools for integrating C/C++ and Fortran code* useful linear algebra, Fourier transform, and random number capabilitiesPandas (from the [Pandas website](https://pandas.pydata.org/)):  high-performance, easy-to-use data structures and data analysis tools for the Python programming languageBoth the Matplotlib and Seaborn libraries are for creating graphs.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Plotting<jupyter_code># Load in the data set
tips_data = sns.load_dataset("tips")<jupyter_output><empty_output><jupyter_text>Plot a histogram of the tips<jupyter_code># with seaborn
sns.distplot(tips_data["tip"], kde = False, bins=10).set_title("Histogram of Total Tip")
plt.show()
# with matplotlib
plt.hist(tips_data['tip'], bins=10)
plt.title("Histogram of Total Tip")
plt.show()<jupyter_output><empty_output><jupyter_text>Create a boxplot of the total bill amounts<jupyter_code># with seaborn
sns.boxplot(tips_data["total_bill"]).set_title("Box plot of the Total Bill")
plt.show()
# with matplotlib
plt.boxplot(tips_data["total_bill"])
plt.title("Box plot of the Total Bill")
plt.show()<jupyter_output><empty_output><jupyter_text>Pandas DataFrame manipulation<jupyter_code># Import NHANES 2015-2016 data
df = pd.read_csv("nhanes_2015_2016.csv")
# look at top 3 rows
df.head(3)<jupyter_output><empty_output><jupyter_text>Pick columns by name<jupyter_code>df['SEQN'].head()<jupyter_output><empty_output><jupyter_text>Pick columns and rows by index name<jupyter_code>df.loc[[0, 1], ['SEQN', 'RIAGENDR']]<jupyter_output><empty_output><jupyter_text>Pick columns and rows by index location<jupyter_code>df.iloc[[1,2], [0,5]]<jupyter_output><empty_output><jupyter_text>Unit testingThis is the idea that you should run complicated code on a simple test case that you know that outcome of. If your code outputs something you did not expect, then you know there is an error somewhere that must be fixed. When working with large datasets, it is easy to get reasonable output that is actually measuring something different than you wanted.   ExamplePerhaps you want to take the mean of the first row.<jupyter_code>df = pd.DataFrame({'col1':[1, 2, 3], 'col2':[3, 4, 5]})
df
df.mean()[0]<jupyter_output><empty_output><jupyter_text>This looks correct, but lets on a DataFrame that doesn't have the same mean for the first row and the first column.<jupyter_code>df = pd.DataFrame({'col1':[1, 2, 3], 'col2':[6, 7, 8]})
df
df.mean()[0]<jupyter_output><empty_output><jupyter_text>Looks like this is actually taking the mean of the first row. Doing a simple test, we found an error that would have been much harder to spot had our DataFrame been 100,000 rows and 300 columns.<jupyter_code># Use the argument 'axis=1' to specify that we are taking the mean of the columns. 
# Use 'axis=0' (which as we can see is the default) to take to mean of the rows
df.mean(axis=1)[0]<jupyter_output><empty_output><jupyter_text>Randomness and reproducibilityIn Python, we refer to randomness as the ability to generate data, strings, or, more generally, numbers at random.However, when conducting analysis it is important to consider reproducibility. If we are creating random data, how can we enable reproducible analysis?We do this by utilizing pseudo-random number generators (PRNGs). PRNGs start with a random number, known as the seed, and then use an algorithm to generate a psuedo-random sequence based on it.This means that we can replicate the output of a random number generator in python simply by knowing which seed was used.We can showcase this by using the functions in the python library random.<jupyter_code>import random
random.seed(1234)
random.random()
random.seed(1234)
random.random()<jupyter_output><empty_output><jupyter_text>The random library includes standard distributions that may come in handy<jupyter_code># Uniform distribution
random.uniform(25,50)
mu = 0

sigma = 1

random.normalvariate(mu, sigma)<jupyter_output><empty_output><jupyter_text>List comprehensionList comprehensions allow you to easy create lists. They follow the format:```my_list = [expression(i) for i in input list]```For example, if you wanted to plot the sin curve from -$\pi$ to $\pi$:<jupyter_code>x = np.linspace(-np.pi, np.pi, 100) # create a list of 100 equally spaced points between -pi and pi
x
# Here's our list comprehension. For each point in x, we want y=sin(x)
y = [np.sin(value) for value in x] 
# let looks at just the first 5 elements of y
y[:5]
# It doesn't really matter what word you use to represent a value in the input list, 
# the following will built the same y
y = [np.sin(i) for i in x]
y[:5]
plt.plot(x,y)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08476568238577935, 'fraction_numerical': 0.012046422800058763, 'mean_word_length': 5.774129353233831, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 58, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '74671', 'license': 'MIT', 'n_tokens_mistral': 2082, 'n_tokens_neox': 1964, 'n_words': 870, 'path': 'Inferential_Statistical_Analysis_with_Python-master/week1/course1review.ipynb', 'repo_name': 'rezapci/UofM_Statistics_with_Python_Specialization', 'size': '6807'}