<jupyter_start><jupyter_text>Pandas - Data Analysis Library for Python - Part 1 BasicsOriginal by <NAME>         Modified by <NAME> (<EMAIL>)`Pandas` is one of the most important libraries available for data analysts. It is extremely valuable when processing time-series output data<jupyter_code>%matplotlib inline
from IPython.core.display import Image
Image(filename='pandaslogo.jpg')<jupyter_output><empty_output><jupyter_text>Pandas video tutorialWes McKinney created the Pandas library and this notebook was used in a live tutorial which can be found on youtube:<jupyter_code>from IPython.display import YouTubeVideo
YouTubeVideo("w26x-z-BdWQ")
from IPython.core.display import Image
Image(filename='pandasbook.jpg')<jupyter_output><empty_output><jupyter_text>The first step in any analysis is to load the necessary libraries -- in this case `pandas` and `numpy`<jupyter_code>from pandas import *
import pandas
import numpy as np



# plt.rc('figure', figsize=(10, 6))
# pandas.set_option('notebook_repr_html',False)<jupyter_output><empty_output><jupyter_text>Pandas Series Object: 1 dimensional data container======This object is a data container for vectors -- incorporating an index and string search functions<jupyter_code>s = Series(np.random.randn(5))
s
labels = ['a', 'b', 'c', 'd', 'e']
s = Series(np.random.randn(5), index = labels)
s
'b' in s
s['b']
s
mapping = s.to_dict()
mapping
s = Series(mapping)
s
s[:3]
s.index<jupyter_output><empty_output><jupyter_text>DataFrame: 2D collection of Series==================================<jupyter_code>df = DataFrame({'a': np.random.randn(6),
                'b': ['foo', 'bar'] * 3,
                'c': np.random.randn(6)})
df.info()
df.index
df
df = DataFrame({'a': np.random.randn(6),
                'b': ['foo', 'bar'] * 3,
                'c': np.random.randn(6)},
               index = date_range('1/1/2000', periods=6))
df
df = DataFrame({'a': np.random.randn(6),
                'b': ['foo', 'bar'] * 3,
                'c': np.random.randn(6)},
               columns=['a', 'b', 'c', 'd'])
df<jupyter_output><empty_output><jupyter_text>Creation from nested dicts--------------------------These arise naturally in Python code<jupyter_code>data = {}
for col in ['foo', 'bar', 'baz']:
    for row in ['a', 'b', 'c', 'd']:
        data.setdefault(col, {})[row] = np.random.randn()
data
DataFrame(data)<jupyter_output><empty_output><jupyter_text>Data alignment==============<jupyter_code>close_px = read_csv('stock_data.csv', index_col=0, parse_dates=True)
close_px
s1 = close_px['AAPL'][-20:]
s2 = close_px['AAPL'][-25:-10]
s1
s2
s1 + s2
df = close_px.iloc[-10:, :3]
df
b, c  = s1.align(s2, join='inner')
b
c
b, c  = s1.align(s2, join='outer')
b
b, c  = s1.align(s2, join='right')

df = close_px.ix[-10:, ['AAPL', 'IBM', 'MSFT']]
df
df2 = df.ix[::2, ['IBM', 'MSFT']]
df2
df + df2
b, c = df.align(df2, join='inner')<jupyter_output><empty_output><jupyter_text>Truncation - clipping a datetime indexed object<jupyter_code>df
df.truncate(before='2011-10-05')
df.truncate(before='2011-10-05',after='2011-10-12')<jupyter_output><empty_output><jupyter_text>Resampling - Useful time series aggregation<jupyter_code>df
df.resample('M').mean()
df.resample('5D').mean()
df.resample('5D').max()<jupyter_output><empty_output><jupyter_text>Missing Data - Filling in the Gaps<jupyter_code>dfgaps = df.resample('D').mean()
dfgaps
dfgaps.dropna()
dfgaps.fillna(method = 'bfill')<jupyter_output><empty_output><jupyter_text>Transposing: no copy if all columns are same type-------------------------------------------------<jupyter_code>df[:5].T<jupyter_output><empty_output><jupyter_text>Columns can be any type-----------------------<jupyter_code>n = 10
foo = DataFrame(index=range(n))
foo['floats'] = np.random.randn(n)
foo['ints'] = np.arange(n)
foo['strings'] = ['foo', 'bar'] * (n / 2)
foo['bools'] = foo['floats'] > 0
foo['objects'] = date_range('1/1/2000', periods=n)
foo
foo.dtypes<jupyter_output><empty_output><jupyter_text>N.B. transposing is not roundtrippable in this case (column-oriented data structure)<jupyter_code>foo.T.T
foo.T.T.dtypes<jupyter_output><empty_output><jupyter_text>Function applicationYou can apply arbitrary functions to the rows or columns of a DataFrame<jupyter_code>df.apply(np.mean)
df.apply(np.mean, axis=1)<jupyter_output><empty_output><jupyter_text>You can get as fancy as you want<jupyter_code>close_px
df.apply(lambda x: x.max() - x.min()) # np.ptp
np.log(close_px)<jupyter_output><empty_output><jupyter_text>PlottingSome basic plotting integration with matplotlib in Series / DataFrame<jupyter_code>close_px[['AAPL', 'IBM', 'MSFT', 'XOM']].plot();<jupyter_output><empty_output><jupyter_text>Hierarchical indexing---------------------<jupyter_code>index = MultiIndex(levels=[['foo', 'bar', 'baz', 'qux'],
                                   ['one', 'two', 'three']],
                           labels=[[0, 0, 0, 1, 1, 2, 2, 3, 3, 3],
                                   [0, 1, 2, 0, 1, 1, 2, 0, 1, 2]])
hdf = DataFrame(np.random.randn(10, 3), index=index,
                columns=['A', 'B', 'C'])
hdf
hdf.loc['foo']
hdf.loc['foo'] = 0
hdf
hdf.loc['foo', 'three']<jupyter_output><empty_output><jupyter_text>Stacking and unstacking-----------------------<jupyter_code>tuples = zip(*[['bar', 'bar', 'baz', 'baz',
                'foo', 'foo', 'qux', 'qux'],
               ['one', 'two', 'one', 'two',
                'one', 'two', 'one', 'two']])
index = MultiIndex.from_tuples(tuples)
columns = MultiIndex.from_tuples([('A', 'cat'), ('B', 'dog'),
                                  ('B', 'cat'), ('A', 'dog')])
df = DataFrame(np.random.randn(8, 4), index=index, columns=columns)
df
df2 = df.iloc[[0, 1, 2, 4, 5, 7]]
df2
df.unstack()['B']<jupyter_output><empty_output><jupyter_text>GroupBy<jupyter_code>df = DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
                       'foo', 'bar', 'foo', 'foo'],
                'B' : ['one', 'one', 'two', 'three',
                       'two', 'two', 'one', 'three'],
                'C' : np.random.randn(8),
                'D' : np.random.randn(8)})
df
for key, group in df.groupby('A'):
    print key
    print group
df.groupby('A')['C'].describe().T
df.groupby('A').mean()
for key, group in df.groupby('A'):
    print key
    print group
df.groupby(['A', 'B']).mean()
df.groupby(['A', 'B'], as_index=False).mean()<jupyter_output><empty_output><jupyter_text>GroupBy with hierarchical indexing----------------------------------<jupyter_code>tuples = zip(*[['bar', 'bar', 'baz', 'baz',
                'foo', 'foo', 'qux', 'qux'],
               ['one', 'two', 'one', 'two',
                'one', 'two', 'one', 'two']])
index = MultiIndex.from_tuples(tuples)
columns = MultiIndex.from_tuples([('A', 'cat'), ('B', 'dog'),
                                  ('B', 'cat'), ('A', 'dog')])
df = DataFrame(np.random.randn(8, 4), index=index, columns=columns)
df
df.groupby(level=0, axis=0).mean()
df.stack()
df.stack().mean(1).unstack()
# could also have done
df.groupby(level=1, axis=1).mean()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.21058606499219526, 'fraction_numerical': 0.01944089683553285, 'mean_word_length': 3.7653820148749153, 'pattern_counts': {'":': 0, '<': 84, '<?xml version=': 0, '>': 85, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '62933', 'license': 'CC-BY-3.0', 'n_tokens_mistral': 2694, 'n_tokens_neox': 2562, 'n_words': 550, 'path': '1_PandasTutorial/Pandas_DataAnalysisLibrary.ipynb', 'repo_name': 'buds-lab/python-for-building-analysts', 'size': '7081'}