<jupyter_start><jupyter_text>Numpy + Pandas I  Numpypandas is built atop `NumPy`, historically and in the actual library, so it's helpful to have a basic understanding of it.  ndarrayThe core of numpy is the `ndarray`, N-dimensional array. These are singly-typed, fixed-length data containers.NumPy also provides many convenient and fast methods implemented on the `ndarray`.<jupyter_code>import numpy as np
# per convention
x = np.array([1, 2, 3])
x
x.dtype
y = np.array([[True, False], [False, True]])
y
z = np.array([1., 2, 3])
z
s = np.array(['hello', 1])
s
x = np.array([1, 2, 3])
x.shape
y = np.array([[1., 2., 3.], [4., 5., 6.]])
print(y)
print(y.shape)<jupyter_output>[[ 1.  2.  3.]
 [ 4.  5.  6.]]
(2, 3)<jupyter_text>dtypesUnlike python lists, NumPy arrays care about the type of data stored within.The full list of NumPy dtypes can be found in the [NumPy documentation](http://docs.scipy.org/doc/numpy/user/basics.types.html).![dtypes](http://docs.scipy.org/doc/numpy/_images/dtype-hierarchy.png) We sacrifice the convinience of mixing bools and ints and floats within an array for much better performance.However, an unexpected `dtype` change will probably bite you at some point in the future. The two biggest things to remember are- Missing values (NaN) cast integer or boolean arrays to floats- NumPy arrays only have a single dtype for every element- the object dtype is the fallbackYou'll want to avoid object dtypes. It's typically slow.<jupyter_code>np.nan
np.nan * 1.
type(np.nan)<jupyter_output><empty_output><jupyter_text>Vectorization_dtypes_ and _vectorization_ are part of what make NumPy fast<jupyter_code>x = np.random.randint(0, 10, 10)
y = np.random.randint(0, 10, 10)
x, y
[x[i] + y[i] for i in range(10)]
x + y
a = np.random.rand(100000)
b = np.random.rand(100000)
%timeit [a[i] + b[i] for i in range(a.shape[0])]
%timeit a + b<jupyter_output>The slowest run took 19.52 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 3: 107 µs per loop<jupyter_text>1ms = 1000$\mu$s  Broadcasting It's super cool and super useful. The one-line explanation is that when doing elementwise operations, things expand to the "correct" shape.<jupyter_code>x = np.arange(5)
x<jupyter_output><empty_output><jupyter_text>Adding scalars<jupyter_code>x + 1
y = np.random.uniform(size=(2, 5))
y
y + 1<jupyter_output><empty_output><jupyter_text>Adding other arrays<jupyter_code>print(x)
print(y)
x + y
z = x + y
print(z.dtype, z.shape)
a = np.ones((5,2))
print(a)
b = np.ones((2,5))
print(b)
a + b
print(b)
print(b.T)
a + b.T<jupyter_output><empty_output><jupyter_text>Element-wise operations- Arithmetic: `+`, `-`, `*`, `/`, `**`- Comparisons: `==`, `!=`, ``, `=`<jupyter_code>x * y
x**2
x < y
x < 2
x < 2<jupyter_output><empty_output><jupyter_text>Array methods- `.sum()`, `.mean()`, `.min()`, `.max()`, `.prod()`, `.cumsum()`, etc- `np.exp`, `np.log`, `np.sin`, `np.sqrt` etc<jupyter_code>y
y.sum()
y.sum(axis=0)
y.sum(axis=1)
y.prod()
y.mean()
print(y.min(), y.max())
print(x)
print(x.cumsum())
print(y)
print(y.cumsum(axis=0))
z = np.arange(1,6)
print(z)
print(np.log(z))
np.exp(z)<jupyter_output><empty_output><jupyter_text>Numpy indexing and slicingWorks similarly to list indexing, but with multiple dimensions<jupyter_code>x = np.arange(50)
x
y = x.reshape((5,10))
y
y[0]
y[1]
y[10]
y[1][0]
y[1,0]
y[0:,0]
y[0,0:]
y[2:4,4:8]
y[2:4,4:8:2]
y[-1,]
z = np.arange(60).reshape((3,4,5))
z
z[1,1,1]
z[1,:,1]<jupyter_output><empty_output><jupyter_text>Boolean indexingIf you pass in an array of boolean values of a compatible shape, you can index that way, too.<jupyter_code>x = np.arange(5)
x
x[np.array([False,True,False,True,False])]
even = (x % 2 == 0)
even
x[even]
lt3 = x < 3
lt3
x[lt3]<jupyter_output><empty_output><jupyter_text>**pandas** is a Python package providing fast, flexible, and expressive data structures designed to work with both *relational* and *labeled* data.  It is a fundamental high-level building block for doing practical, real world data analysis in Python.   Why pandas?Numpy is great. But it lacks a few things that are conducive to doing statisitcal analysis. By building on top of Numpy, pandas provides: - labeled arrays - heterogenous data types within a table - "better" missing data handling - convenient methods (`groupby`, `rolling`, `resample`) - more data types (Categorical, Datetime) - indexing handles data alignment - built in functionality for reading/writing many kinds of files<jupyter_code>from IPython.core.display import HTML
HTML("<iframe src=http://pandas.pydata.org width=1024 height=500></iframe>")<jupyter_output><empty_output><jupyter_text>Pandas Data Structures  SeriesA **Series** is a single vector of data with an *index* that labels each element in the vector<jupyter_code>import pandas as pd
counts = pd.Series([632, 1638, 569, 115])
counts<jupyter_output><empty_output><jupyter_text>If an index is not specified, a default sequence of integers is assigned as the index. A numpy array comprises the values of the `Series`, while the index is a pandas `Index` object.<jupyter_code>counts.values
counts.index<jupyter_output><empty_output><jupyter_text>We can assign meaningful labels to the index, if they are available:<jupyter_code>bacteria = pd.Series([632, 1638, 569, 115], 
    index=['Firmicutes', 'Proteobacteria', 'Actinobacteria', 'Bacteroidetes'])
bacteria<jupyter_output><empty_output><jupyter_text>These labels can be used to refer to the values in the `Series`.<jupyter_code>bacteria['Actinobacteria']
bacteria['Firmicutes':'Actinobacteria']<jupyter_output><empty_output><jupyter_text>Notice that the indexing operation preserved the association between the values and the corresponding indices.We can still use positional indexing if we wish.<jupyter_code>bacteria[0]<jupyter_output><empty_output><jupyter_text>Or boolean arrays/lists<jupyter_code>bac = [name.endswith('bacteria') for name in bacteria.index]
bac
bacteria[bac]<jupyter_output><empty_output><jupyter_text>We can give both the array of values and the index meaningful labels themselves:<jupyter_code>bacteria.name = 'counts'
bacteria.index.name = 'phylum'
bacteria<jupyter_output><empty_output><jupyter_text>We can also filter according to the values in the `Series`:<jupyter_code>bacteria[bacteria > 1000]
bacteria > 1000<jupyter_output><empty_output><jupyter_text>A `Series` can be thought of as an ordered key-value store. In fact, we can create one from a `dict`:<jupyter_code>bacteria_dict = {'Firmicutes': 632, 
                 'Proteobacteria': 1638, 
                 'Actinobacteria': 569, 
                 'Bacteroidetes': 115}
pd.Series(bacteria_dict)<jupyter_output><empty_output><jupyter_text>Notice that the `Series` is created in key-sorted order.If we pass a custom index to `Series`, it will select the corresponding values from the dict, and treat indices without corrsponding values as missing. Pandas uses the `NaN` (not a number) type for missing values.<jupyter_code>bacteria2 = pd.Series(bacteria_dict, index=['Cyanobacteria','Firmicutes','Proteobacteria','Actinobacteria'])
bacteria2
bacteria2.isnull()<jupyter_output><empty_output><jupyter_text>Critically, the labels are used to **align data** when used in operations with other Series objects:<jupyter_code>bacteria + bacteria2<jupyter_output><empty_output><jupyter_text>DataFrame Inevitably, we want to be able to store, view and manipulate data that is *multivariate*, where for every index there are multiple fields or columns of data, often of varying data type. A `DataFrame` is a tabular data structure, encapsulating multiple series like columns in a spreadsheet. Data are stored internally as a 2-dimensional object, but the `DataFrame` allows us to represent and manipulate higher-dimensional data.<jupyter_code>data = pd.DataFrame({'value':[632, 1638, 569, 115, 433, 1130, 754, 555],
                     'patient':[1, 1, 1, 1, 2, 2, 2, 2],
                     'phylum':['Firmicutes', 'Proteobacteria', 'Actinobacteria', 
                               'Bacteroidetes', 'Firmicutes', 'Proteobacteria', 
                               'Actinobacteria', 'Bacteroidetes']})
data<jupyter_output><empty_output><jupyter_text>Notice the `DataFrame` is sorted by column name. We can change the order by indexing them in the order we desire:<jupyter_code>data[['phylum','value','patient']]<jupyter_output><empty_output><jupyter_text>A `DataFrame` has a second index, representing the columns:<jupyter_code>data.columns<jupyter_output><empty_output><jupyter_text>If we wish to access columns, we can do so either by dict-like indexing or by attribute:<jupyter_code>data['value']
data.value
type(data.value)
type(data[['value']])<jupyter_output><empty_output><jupyter_text>Notice this is different than with `Series`, where dict-like indexing retrieved a particular element (row).   Indexing `DataFrame`sIf we want access to a row in a `DataFrame`, we index its `loc` or `iloc` attribute.<jupyter_code>np.random.seed(0)
data = pd.DataFrame({'v1': np.random.rand(4), 
                     'v2': np.random.rand(4)}, 
                    index=['w','x','y','z'])
data.head()<jupyter_output><empty_output><jupyter_text>`.loc`To access something by its _label_, use `.loc[]`:<jupyter_code>data.loc['y']
data.loc[['x','y']]
data.loc[['x','y'],'v1']
data.loc[['x','y'],'v2']
data.loc['w':'y',['v1','v2']]<jupyter_output><empty_output><jupyter_text>`.iloc`Whereas if we want to access something by its row/column number, use `.iloc[]`:<jupyter_code>data.iloc[2]
data.iloc[:2]
data.iloc[:2,0]
data.iloc[0,:]<jupyter_output><empty_output><jupyter_text>Boolean indexing`.loc` _also_ supports boolean indices<jupyter_code>data.v1 > .6
data.loc[data.v1 > .6]
data.loc[(data.v1 > .6) & (data.v2 < .5)]
data.loc[(data.v1 > .6) & (data.v2 < .5), 'v2']<jupyter_output><empty_output><jupyter_text>`.ix`If you want to live dangerously, you can use `.ix[]`, which lets you mix integers and labels but does some guesswork... Why am I even telling you this?   👴<jupyter_code>data.ix['w']
data.ix[0]
data.ix['w', 0]
data.ix[2:, 'v2']
data.ix[data.index == 'w', 'v2']<jupyter_output><empty_output><jupyter_text>.query() - A convenient way to use SQL-like syntax for selecting - But unfortunately does not allow assignment... - 😎 - More info in [the docs](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.query.html)<jupyter_code>data.query('v1 < .6')
data.query('v1 < .6 & v2 > .7')<jupyter_output><empty_output><jupyter_text>Operating on `DataFrame`s Operations on `DataFrame`s are very similar to operations on `Series`, including the ability to automatically align axes.<jupyter_code>df1 = pd.DataFrame.from_dict({'USA': {'lat': 37.1, 'lon': 95.7},
                              'CAN': {'lat': 56.1, 'lon': 106.3},
                              'MEX': {'lat': 23.6, 'lon': 102.6}}, orient='index')
df1
df2 = pd.DataFrame.from_dict({'USA': {'temp': 70},
                              'CAN': {'temp': 60},
                              'MEX': {'temp': 50}}, orient='index')
df2
df1['lat'] / df2['temp']<jupyter_output><empty_output><jupyter_text>Do we have to sort values?<jupyter_code>df1 = pd.DataFrame.from_dict({'USA': {'lat': 37.1, 'lon': 95.7},
                              'CAN': {'lat': 56.1, 'lon': 106.3},
                              'MEX': {'lat': 23.6, 'lon': 102.6}}, orient='index')
df2 = pd.DataFrame.from_dict({'CAN': {'temp': 60},
                              'USA': {'temp': 70},
                              'MEX': {'temp': 50}}, orient='index')
df1['lat'] / df2['temp']<jupyter_output><empty_output><jupyter_text>What about missing values?<jupyter_code>df3 = pd.DataFrame.from_dict({'USA': {'temp': 70},
                              'CAN': {'temp': 60},
                              'MEX': {'temp': 50},
                              'GRL': {'temp': 40}}, orient='index')
df3
df1['lat'] / df3['temp']<jupyter_output><empty_output><jupyter_text>Adding to and changing `DataFrame`s<jupyter_code>data = pd.DataFrame.from_dict({0: {'patient': 1, 'phylum': 'Firmicutes', 'value': 632},
                               1: {'patient': 1, 'phylum': 'Proteobacteria', 'value': 1638},
                               2: {'patient': 1, 'phylum': 'Actinobacteria', 'value': 569},
                               3: {'patient': 1, 'phylum': 'Bacteroidetes', 'value': 115},
                               4: {'patient': 2, 'phylum': 'Firmicutes', 'value': 433},
                               5: {'patient': 2, 'phylum': 'Proteobacteria', 'value': 1130},
                               6: {'patient': 2, 'phylum': 'Actinobacteria', 'value': 754},
                               7: {'patient': 2, 'phylum': 'Bacteroidetes', 'value': 555}}, orient='index')
data<jupyter_output><empty_output><jupyter_text>Its important to note that the Series returned when a DataFrame is indexed is merely a **view** on the DataFrame, and not a copy of the data itself. So you must be cautious when manipulating this data:<jupyter_code>vals = data.value
vals
vals[5] = 0
vals
data
vals = data.value.copy()
vals[5] = 1000
data<jupyter_output><empty_output><jupyter_text>We can modify existing values by assignment:<jupyter_code>data.loc[3, 'value'] = 14
data
data.iloc[0, 1] = 2
data
data['patient'] = data['patient'] + 1
data<jupyter_output><empty_output><jupyter_text>And we can also create new variables by assignment:<jupyter_code>data['year'] = 2013
data
data['age'] = np.repeat([5,10],4)
data
data['iq'] = np.arange(5)<jupyter_output><empty_output><jupyter_text>We can use attribute indexing of columns to assign values:<jupyter_code>data.year = 2014
data<jupyter_output><empty_output><jupyter_text>But not to add new columns:<jupyter_code>data.treatment = 1
data
data.treatment<jupyter_output><empty_output><jupyter_text>Specifying a `Series` as a new columns cause its values to be added according to the `DataFrame`'s index:<jupyter_code>treatment = pd.Series([0]*4 + [1]*2)
treatment
data['treatment'] = treatment
data<jupyter_output><empty_output><jupyter_text>`.eval()`- Another new(ish) method similar to `.query()`- Takes advantage of some tricks to speed things up- See [the docs](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.eval.html) for more info<jupyter_code>data = pd.DataFrame.from_dict({0: {'patient': 1, 'phylum': 'Firmicutes', 'value': 632},
                               1: {'patient': 1, 'phylum': 'Proteobacteria', 'value': 1638},
                               2: {'patient': 1, 'phylum': 'Actinobacteria', 'value': 569},
                               3: {'patient': 1, 'phylum': 'Bacteroidetes', 'value': 115},
                               4: {'patient': 2, 'phylum': 'Firmicutes', 'value': 433},
                               5: {'patient': 2, 'phylum': 'Proteobacteria', 'value': 1130},
                               6: {'patient': 2, 'phylum': 'Actinobacteria', 'value': 754},
                               7: {'patient': 2, 'phylum': 'Bacteroidetes', 'value': 555}}, orient='index')
data
data.eval('patient + value')
data.eval('newvar = patient + value', inplace=True)
data<jupyter_output><empty_output><jupyter_text>Other Python data structures (ones without an index) need to be the same length as the `DataFrame`:<jupyter_code>month = ['Jan', 'Feb', 'Mar', 'Apr']
data['month'] = month
data['month'] = ['Jan']*len(data)
data<jupyter_output><empty_output><jupyter_text>We can use `del` to remove columns, in the same way `dict` entries can be removed:<jupyter_code>del data['month']
data<jupyter_output><empty_output><jupyter_text>Or `.drop()` can be used:<jupyter_code>data.drop?
data['month'] = ['Jan']*len(data)
data.drop('month', axis=1, inplace=True)
data<jupyter_output><empty_output><jupyter_text>We can extract the underlying data as a simple `ndarray` by accessing the `values` attribute:<jupyter_code>data.values<jupyter_output><empty_output><jupyter_text>Notice that because of the mix of string and integer (and `NaN`) values, the dtype of the array is `object`. The dtype will automatically be chosen to be as general as needed to accomodate all the columns.<jupyter_code>df = pd.DataFrame({'foo': [1,2,3], 'bar':[0.4, -1.0, 4.5]})
df.values
df.values.dtype<jupyter_output><empty_output><jupyter_text>Pandas uses a custom data structure to represent the indices of `Series` and `DataFrame`s.<jupyter_code>data.index<jupyter_output><empty_output><jupyter_text>Index objects are immutable:<jupyter_code>data.index[0] = 15<jupyter_output><empty_output><jupyter_text>This is so that Index objects can be shared between data structures without fear that they will be changed.<jupyter_code>bacteria2.index = bacteria.index
bacteria2<jupyter_output><empty_output><jupyter_text>Importing data Pandas provides a convenient set of functions for importing tabular data in a number of formats directly into a `DataFrame` object. These functions include a slew of options to perform type inference, indexing, parsing, iterating and cleaning automatically as data are imported. Let's start with some more bacteria data, stored in csv format.<jupyter_code>!head -n10 data/microbiome.csv<jupyter_output>Taxon,Patient,Tissue,Stool
Firmicutes,1,632,305
Firmicutes,2,136,4182
Firmicutes,3,1174,703
Firmicutes,4,408,3946
Firmicutes,5,831,8605
Firmicutes,6,693,50
Firmicutes,7,718,717
Firmicutes,8,173,33
Firmicutes,9,228,80<jupyter_text>This table can be read into a DataFrame using `read_csv`:<jupyter_code>mb = pd.read_csv("data/microbiome.csv")
mb.shape
mb.head()
mb.tail()<jupyter_output><empty_output><jupyter_text>Notice that `read_csv` automatically considered the first row in the file to be a header row.We can override default behavior by customizing some the arguments, like `header`, `names` or `index_col`.<jupyter_code>pd.read_csv("data/microbiome.csv", header=None).head()<jupyter_output><empty_output><jupyter_text>`read_csv` is just a convenience function for `read_table`, since csv is such a common format:<jupyter_code>mb = pd.read_table("data/microbiome.csv", sep=',')
mb.head()<jupyter_output><empty_output><jupyter_text>The `sep` argument can be customized as needed to accomodate arbitrary separators. For example, we can use a regular expression to define a variable amount of whitespace, which is unfortunately very common in some data formats:         sep='\s+' For a more useful index, we can specify the first two columns, which together provide a unique index to the data.<jupyter_code>mb = pd.read_csv("data/microbiome.csv", index_col=['Taxon','Patient'])
mb.head()<jupyter_output><empty_output><jupyter_text>This is called a *hierarchical* index, which we will revisit later. If we have sections of data that we do not wish to import (for example, known bad data), we can populate the `skiprows` argument:<jupyter_code>pd.read_csv("data/microbiome.csv", skiprows=[3,4,6]).head()<jupyter_output><empty_output><jupyter_text>Conversely, if we only want to import a small number of rows from, say, a very large data file we can use `nrows`:<jupyter_code>pd.read_csv("data/microbiome.csv", nrows=4)<jupyter_output><empty_output><jupyter_text>Alternately, if we want to process our data in reasonable chunks, the `chunksize` argument will return an iterable object that can be employed in a data processing loop. For example, our microbiome data are organized by bacterial phylum, with 15 patients represented in each:<jupyter_code>data_chunks = pd.read_csv("data/microbiome.csv", chunksize=15)

mean_tissue = {chunk.Taxon[0]: chunk.Tissue.mean() for chunk in data_chunks}
    
mean_tissue<jupyter_output><empty_output><jupyter_text>Most real-world data is incomplete, with values missing due to incomplete observation, data entry or transcription error, or other reasons. Pandas will automatically recognize and parse common missing data indicators, including `NA` and `NULL`.<jupyter_code>!head -n12 data/microbiome_missing.csv
pd.read_csv("data/microbiome_missing.csv").head(12)<jupyter_output><empty_output><jupyter_text>Above, Pandas recognized `NA` and an empty field as missing data.<jupyter_code>pd.isnull(pd.read_csv("data/microbiome_missing.csv")).head(12)<jupyter_output><empty_output><jupyter_text>Unfortunately, there will sometimes be inconsistency with the conventions for missing data. In this example, there is a question mark "?" and a large negative number where there should have been a positive integer. We can specify additional symbols with the `na_values` argument:<jupyter_code>pd.read_csv("data/microbiome_missing.csv", na_values=['?', -99999]).head(12)<jupyter_output><empty_output><jupyter_text>These can be specified on a column-wise basis using an appropriate dict as the argument for `na_values`.  Microsoft ExcelSince so much scientific data ends up in Excel spreadsheets, Pandas' ability to directly import Excel spreadsheets is valuable. This support is contingent on having one or two dependencies (depending on what version of Excel file is being imported) installed: `xlrd` and `openpyxl` (these may be installed with either `pip` or `easy_install`). Importing Excel data to Pandas is a two-step process. First, we create an `ExcelFile` object using the path of the file:<jupyter_code>mb_file = pd.ExcelFile('data/microbiome/MID1.xls')
mb_file<jupyter_output><empty_output><jupyter_text>Then, since modern spreadsheets consist of one or more "sheets", we parse the sheet with the data of interest:<jupyter_code>mb1 = mb_file.parse("Sheet 1", header=None)
mb1.columns = ["Taxon", "Count"]
mb1.head()<jupyter_output><empty_output><jupyter_text>There is also a `read_excel` convenience function in Pandas that combines these steps into a single call:<jupyter_code>mb2 = pd.read_excel('data/microbiome/MID2.xls', sheetname='Sheet 1', header=None)
mb2.columns = ['Taxon', 'Count']
mb2.head()<jupyter_output><empty_output><jupyter_text>Saving data<jupyter_code>data
data.to_csv?
data.to_csv('data/test.csv')
!cat data/test.csv
data.to_csv('data/test.csv', index=False)
!cat data/test.csv
data.to_csv('data/test.csv', index=False, header=False)
!cat data/test.csv<jupyter_output>Firmicutes,1,632,633
Proteobacteria,1,1638,1639
Actinobacteria,1,569,570
Bacteroidetes,1,115,116
Firmicutes,2,433,435
Proteobacteria,2,1130,1132
Actinobacteria,2,754,756
Bacteroidetes,2,555,557
{'fraction_non_alphanumeric': 0.1335308908045977, 'fraction_numerical': 0.028017241379310345, 'mean_word_length': 4.993810548977395, 'pattern_counts': {'":': 0, '<': 271, '<?xml version=': 0, '>': 270, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '66', 'dir': 'jupyter-structured-clean-dedup', 'id': '244290', 'license': 'CC-BY-3.0', 'n_tokens_mistral': 8060, 'n_tokens_neox': 7442, 'n_words': 2123, 'path': 'Lecture 3/Lecture 3 - Numpy + Pandas I (Complete).ipynb', 'repo_name': 'iEvidently/ihme-python-course', 'size': '22272'}