<jupyter_start><jupyter_text>Pandas TutorialSoftware Carpentry, EITN, Paris, November 20th, 2015<NAME>forked from the tutorial at EuroScipy 2015 by <NAME> (Ghent University, Belgium)Licensed under [CC BY 4.0 Creative Commons](http://creativecommons.org/licenses/by/4.0/)  Content of this talk- Why do you need pandas?- Basic introduction to the data structures- Guided tour through some of the pandas features with two case studies: **movie database** and a **case study about air quality**If you want to follow along, this is a notebook that you can view or run yourself:- All materials (notebook, data): https://github.com/btel/2015_eitn_swc_pandas- You need `pandas` >= 0.15.2 (easy solution is using Anaconda) Some imports:<jupyter_code>%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

pd.options.display.max_rows = 8<jupyter_output><empty_output><jupyter_text>Let's start with a showcase  Case study: air quality in EuropeAirBase (The European Air quality dataBase): hourly measurements of all air quality monitoring stations from EuropeStarting from these hourly data for different stations:<jupyter_code>data = pd.read_csv('data/airbase_data.csv', index_col=0, parse_dates=True, na_values='-9999')
data<jupyter_output><empty_output><jupyter_text>to answering questions about this data in a few lines of code:**Does the air pollution show a decreasing trend over the years?**<jupyter_code>data['1999':].resample('A').plot(ylim=[0,100])<jupyter_output><empty_output><jupyter_text>**How many exceedances of the limit values?**<jupyter_code>exceedances = data > 200
exceedances = exceedances.groupby(exceedances.index.year).sum()
ax = exceedances.loc[2005:].plot(kind='bar')
ax.axhline(18, color='k', linestyle='--')<jupyter_output><empty_output><jupyter_text>**What is the difference in diurnal profile between weekdays and weekend?**<jupyter_code>data['weekday'] = data.index.weekday
data['weekend'] = data['weekday'].isin([5, 6])
data_weekend = data.groupby(['weekend', data.index.hour])['FR04012'].mean().unstack(level=0)
data_weekend.plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11650943396226415, 'fraction_numerical': 0.025, 'mean_word_length': 7.126436781609195, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '213056', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 715, 'n_tokens_neox': 661, 'n_words': 199, 'path': '01 - Introduction.ipynb', 'repo_name': 'StphTphsn/2015_eitn_swc_pandas', 'size': '2146'}