<jupyter_start><jupyter_text>Interactive Financial Analytics  with Python & IPython **Tutorial with Examples based on the VSTOXX Volatility Index** Dr. <NAME>Continuum Analytics Europe GmbH<EMAIL>For Python Quants &ndash; 14. March 2014 You find the presentation and the IPython Notebook here:    * http://www.hilpisch.com/YH_FPQ_Volatility_Tutorial.html* http://www.hilpisch.com/YH_FPQ_Volatility_Tutorial.ipynb  About Me A brief bio:* Managing Director Europe of Continuum Analytics Inc.* Founder of Visixion GmbH – The Python Quants* Lecturer Mathematical Finance at Saarland University* Focus on Financial Industry and Financial Analytics* Book "**Derivatives Analytics with Python**" (2013)* Book "**Python for Finance**" O'Reilly (2014)* Dr.rer.pol in Mathematical Finance* Graduate in Business Administration* Martial Arts Practitioner and FanSee www.hilpisch.com. Python for Finance (O'Reilly Shop)  Python for Analytics This tutorial **focuses** on * Python as a general purpose financial analytics environment * interactive analytics examples * prototyping-like Python usage    It does **not** address such important issues like * architectural issues regarding hardware and software * development processes, testing, documentation and production * real world problem modeling A fundamental Python stack for interactive data analytics and visualization should at least contain the following libraries tools: * **Python** &ndash; the Python interpreter itself * **NumPy** &ndash; high performance, flexible array structures and operations * **SciPy** &ndash; collection of scientific modules and functions (e.g. for regression, optimization, integration) * **pandas** &ndash; time series and panel data analysis and I/O  * **PyTables** &ndash; hierarchical, high performance database (e.g. for out-of-memory analytics) * **matplotlib** &ndash; 2d and 3d visualization * **IPython** &ndash; interactive data analytics, visualization, publishing    It is best to use e.g. a Python distribution like **Anaconda** to ensure consistency of libraries.  First Financial Analytics Example We need to make a couple of imports for what is to come.<jupyter_code>import numpy as np
import pandas as pd
import pandas.io.data as pdd
from urllib import urlretrieve
%matplotlib inline<jupyter_output><empty_output><jupyter_text>The convenience function _**DataReader**_ makes it easy to read historical stock price data from Yahoo! Finance (http://finance.yahoo.com).<jupyter_code>try:    
    index = pdd.DataReader('^GDAXI', data_source='yahoo', start='2007/3/30')
      # e.g. the EURO STOXX 50 ticker symbol -- ^SX5E
except:
    index = pd.read_csv('dax.txt', index_col=0, parse_dates=True)
index.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 1775 entries, 2007-03-30 00:00:00 to 2014-03-11 00:00:00
Data columns (total 6 columns):
Open         1775 non-null float64
High         1775 non-null float64
Low          1775 non-null float64
Close        1775 non-null float64
Volume       1775 non-null int64
Adj Close    1775 non-null float64
dtypes: float64(5), int64(1)<jupyter_text>pandas strength is the handling of **indexed/labeled/structured data**, like times series data.<jupyter_code>index.tail()<jupyter_output><empty_output><jupyter_text>pandas makes it easy to implement **vectorized operations**, like calculating log-returns over whole time series.<jupyter_code>index['Returns'] = np.log(index['Close'] / index['Close'].shift(1))<jupyter_output><empty_output><jupyter_text>In addition, pandas makes **plotting** quite simple and compact.<jupyter_code>index[['Close', 'Returns']].plot(subplots=True, style='b', figsize=(8, 5))<jupyter_output><empty_output><jupyter_text>We now want to check **how annual volatility changes over time**.<jupyter_code>index['Mov_Vol'] = pd.rolling_std(index['Returns'], window=252) * np.sqrt(252)<jupyter_output><empty_output><jupyter_text>Obviously, the annual volatility **changes significantly** over time.<jupyter_code>index[['Close', 'Returns', 'Mov_Vol']].plot(subplots=True, style='b', figsize=(8, 5))<jupyter_output><empty_output><jupyter_text>Exercise Trend-based investment strategy with the EURO STOXX 50 index:* 2 trends 42d & 252d * long, short, cash positions* no transaction costsSignal generation:* invest (go long) when the 42d trend is more than 100 points above the 252d trend* sell (go short) when the 42d trend is more than 20 points below the 252d trend* invest in cash (no interest) when neither of both is true  Historical Correlation between EURO STOXX 50 and VSTOXX It is a stylized fact that stock indexes and related volatility indexes are **highly negatively correlated**. The following example analyzes this stylized fact based on the EURO STOXX 50 stock index and the VSTOXX volatility index using Ordinary Least-Squares regession (OLS). First, we collect historical data for both the EURO STOXX 50 stock and the VSTOXX volatility index.<jupyter_code>import pandas as pd
import datetime as dt
from urllib import urlretrieve
try:
    es_url = 'http://www.stoxx.com/download/historical_values/hbrbcpe.txt'
    vs_url = 'http://www.stoxx.com/download/historical_values/h_vstoxx.txt'
    urlretrieve(es_url, 'es.txt')
    urlretrieve(vs_url, 'vs.txt')
except:
    pass<jupyter_output><empty_output><jupyter_text>The **EURO STOXX 50 data** is not yet in the right format. Some house cleaning is necessary (I).<jupyter_code>lines = open('es.txt').readlines()  # reads the whole file line-by-line
lines[:5]  # header not well formatted<jupyter_output><empty_output><jupyter_text>The **EURO STOXX 50 data** is not yet in the right format. Some house cleaning is necessary (II).<jupyter_code>lines[3883:3890]  # from 27.12.2001 additional semi-colon<jupyter_output><empty_output><jupyter_text>The **EURO STOXX 50 data** is not yet in the right format. Some house cleaning is necessary (III).<jupyter_code>lines = open('es.txt').readlines()  # reads the whole file line-by-line
new_file = open('es50.txt', 'w')  # opens a new file
new_file.writelines('date' + lines[3][:-1].replace(' ', '') + ';DEL' + lines[3][-1])
    # writes the corrected third line (additional column name)
    # of the orginal file as first line of new file
new_file.writelines(lines[4:])  # writes the remaining lines of the orginal file<jupyter_output><empty_output><jupyter_text>The **EURO STOXX 50 data** is not yet in the right format. Some house cleaning is necessary (IV).<jupyter_code>list(open('es50.txt'))[:5]  # opens the new file for inspection<jupyter_output><empty_output><jupyter_text>Now, the data can be safely read into a DataFrame object.<jupyter_code>es = pd.read_csv('es50.txt', index_col=0, parse_dates=True, sep=';', dayfirst=True)
del es['DEL']  # delete the helper column
es.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 6997 entries, 1986-12-31 00:00:00 to 2014-02-18 00:00:00
Data columns (total 8 columns):
SX5P    6997 non-null float64
SX5E    6997 non-null float64
SXXP    6997 non-null float64
SXXE    6997 non-null object
SXXF    6996 non-null float64
SXXA    6996 non-null float64
DK5F    6996 non-null float64
DKXF    6996 non-null float64
dtypes: float64(7), object(1)<jupyter_text>The VSTOXX data can be read without touching the raw data.<jupyter_code>vs = pd.read_csv('vs.txt', index_col=0, header=2, parse_dates=True, sep=',', dayfirst=True)

# you can alternatively read from the Web source directly
# without saving the csv file to disk:
# vs = pd.read_csv(vs_url, index_col=0, header=2,
#                  parse_dates=True, sep=',', dayfirst=True)<jupyter_output><empty_output><jupyter_text>We now **merge the data** for further analysis.<jupyter_code>import datetime as dt
data = pd.DataFrame({'EUROSTOXX' :
            es['SX5E'][es.index > dt.datetime(1999, 12, 31)]})
data = data.join(pd.DataFrame({'VSTOXX' :
            vs['V2TX'][vs.index > dt.datetime(1999, 12, 31)]}))
data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 3622 entries, 2000-01-03 00:00:00 to 2014-02-18 00:00:00
Data columns (total 2 columns):
EUROSTOXX    3622 non-null float64
VSTOXX       3600 non-null float64
dtypes: float64(2)<jupyter_text>Let's inspect the two time series.<jupyter_code>data.head()<jupyter_output><empty_output><jupyter_text>A **picture** can tell almost the complete story.<jupyter_code>data.plot(subplots=True, grid=True, style='b', figsize=(10, 5))<jupyter_output><empty_output><jupyter_text>We now generate **log returns** for both time series.<jupyter_code>rets = np.log(data / data.shift(1)) 
rets.head()<jupyter_output><empty_output><jupyter_text>To this new data set, also stored in a DataFrame object, we apply **OLS**.<jupyter_code>xdat = rets['EUROSTOXX']
ydat = rets['VSTOXX']
model = pd.ols(y=ydat, x=xdat)
model<jupyter_output><empty_output><jupyter_text>Again, we want to see how our results look graphically.<jupyter_code>import matplotlib.pyplot as plt
plt.plot(xdat, ydat, 'r.')
ax = plt.axis()  # grab axis values
x = np.linspace(ax[0], ax[1] + 0.01)
plt.plot(x, model.beta[1] + model.beta[0] * x, 'b', lw=2)
plt.grid(True)
plt.axis('tight')<jupyter_output><empty_output><jupyter_text>Let us see if we can identify **systematics over time**. And indeed, during the crisis 2007/2008 (yellow dots) volatility has been more pronounced than more recently (red dots).<jupyter_code>import matplotlib as mpl
mpl_dates = mpl.dates.date2num(rets.index)
plt.figure(figsize=(8, 4))
plt.scatter(rets['EUROSTOXX'], rets['VSTOXX'], c=mpl_dates, marker='o')
plt.grid(True)
plt.xlabel('EUROSTOXX')
plt.ylabel('VSTOXX')
plt.colorbar(ticks=mpl.dates.DayLocator(interval=250),
          format=mpl.dates.DateFormatter('%d %b %y'))<jupyter_output><empty_output><jupyter_text>Exercise We want to test whether the EURO STOXX 50 and/or the VSTOXX returns are **normally distributed** or not (e.g. if they might have fat tails). We want to do a* graphical illustration (using _**qqplot**_ of _**statsmodels.api**_) and a* statistical test (using _**normaltest**_ of _**scipy.stats**_)Add on: plot a histogram of the log return frequencies and compare that to a normal distribution with same mean and variance (using e.g. _**norm.pdf**_ from _**scipy.stats**_)  Constant Proportion VSTOXX Investment There has been a number of studies which have illustrated that **constant proportion investments in volatility derivatives** &ndash; given a diversified equity portfolio &ndash; might improve investment performance considerably. See, for instance, the studyThe Benefits of Volatility Derivatives in Equity Portfolio ManagementWe now want to replicate (in a simplified fashion) what you can flexibly test here on the basis of two backtesting applications for **VSTOXX-based investment strategies**:Two Assets BacktestingFour Assets Backtesting The strategy we are going to implement and test is characterized as follows:* An investor has total wealth of say 100,000 EUR* He invests, say, 70% of that into a diversified equity portfolio* The remainder, i.e. 30%, is invested in the VSTOXX index directly* Through (daily) trading the investor keeps the proportions constant* No transaction costs apply, all assets are infinitely divisible We already have the necessary data available. However, we want to drop 'NaN' values and want to **normalize the index values**.<jupyter_code>data = data.dropna()
data = data / data.ix[0] * 100
data.head()<jupyter_output><empty_output><jupyter_text>First, the **initial invest**.<jupyter_code>invest = 100
cratio = 0.3
data['Equity'] = (1 - cratio) * invest / data['EUROSTOXX'][0]
data['Volatility'] = cratio * invest / data['VSTOXX'][0]<jupyter_output><empty_output><jupyter_text>This can already be considered an **static** investment strategy.<jupyter_code>data['Static'] = (data['Equity'] * data['EUROSTOXX']
                + data['Volatility'] * data['VSTOXX'])
data[['EUROSTOXX', 'Static']].plot(figsize=(10, 5))<jupyter_output><empty_output><jupyter_text>Second, the dynamic strategy with **daily adjustments** to keep the value ratio constant.<jupyter_code>for i in range(1, len(data)):
    evalue = data['Equity'][i - 1] * data['EUROSTOXX'][i]
      # value of equity position
    vvalue = data['Volatility'][i - 1] * data['VSTOXX'][i]
      # value of volatility position
    tvalue = evalue + vvalue
      # total wealth 
    data['Equity'][i] = (1 - cratio) * tvalue / data['EUROSTOXX'][i]
      # re-allocation of total wealth to equity ...
    data['Volatility'][i] = cratio * tvalue / data['VSTOXX'][i]
      # ... and volatility position<jupyter_output><empty_output><jupyter_text>Third, the **total wealth position**.<jupyter_code>data['Dynamic'] = (data['Equity'] * data['EUROSTOXX']
                + data['Volatility'] * data['VSTOXX'])
data.head()<jupyter_output><empty_output><jupyter_text>A brief check if the **ratios are indeed constant**.<jupyter_code>(data['Volatility'] * data['VSTOXX'] / data['Dynamic'])[:5]
(data['Equity'] * data['EUROSTOXX'] / data['Dynamic'])[:5]<jupyter_output><empty_output><jupyter_text>Let us inspect the performance of the strategy.<jupyter_code>data[['EUROSTOXX', 'Dynamic']].plot(figsize=(10, 5))<jupyter_output><empty_output><jupyter_text>Exercise Write a Python function which allows for an **arbitrary but constant ratio** to be invested in the VSTOXX index and which returns net performance values (in percent) for the constant proportion VSTOXX strategy.Add on: find the ratio to be invested in the VSTOXX that gives the maximum performance.  Analyzing High Frequency Data Using standard Python functionality and pandas, the code that follows reads **intraday, high-frequency data** from a Web source, plots it and resamples it.<jupyter_code>try:
    url = 'http://hopey.netfonds.no/posdump.php?'
    url += 'date=%s%s%s&paper=AAPL.O&csv_format=csv' % ('2014', '03', '12')
    # you may have to adjust the date since only recent dates are available
    urlretrieve(url, 'aapl.csv')
except:
    pass
AAPL = pd.read_csv('aapl.csv', index_col=0, header=0, parse_dates=True)
AAPL.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 9637 entries, 2014-03-12 09:00:01 to 2014-03-12 19:21:50
Data columns (total 6 columns):
bid                  9637 non-null float64
bid_depth            9637 non-null int64
bid_depth_total      9637 non-null int64
offer                9637 non-null float64
offer_depth          9637 non-null int64
offer_depth_total    9637 non-null int64
dtypes: float64(2), int64(4)<jupyter_text>The **intraday evolution** of the Apple stock price.<jupyter_code>AAPL['bid'].plot()
AAPL = AAPL[AAPL.index > dt.datetime(2014, 3, 12, 10, 0, 0)]
  # only data later than 10am at that day<jupyter_output><empty_output><jupyter_text>A **resampling of the data** is easily accomplished with pandas.<jupyter_code># this resamples the record frequency to 5 minutes, using mean as aggregation rule
AAPL_5min = AAPL.resample(rule='5min', how='mean').fillna(method='ffill')
AAPL_5min.head()<jupyter_output><empty_output><jupyter_text>Let's have a graphical look at the new data set.<jupyter_code>AAPL_5min['bid'].plot()<jupyter_output><empty_output><jupyter_text>With pandas you can easily apply **custom functions** to time series data.<jupyter_code>AAPL_5min['bid'].apply(lambda x: 2 * 530 - x).plot()
  # this mirrors the stock price development at<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11499641857133555, 'fraction_numerical': 0.03294914371296477, 'mean_word_length': 5.540885860306644, 'pattern_counts': {'":': 0, '<': 135, '<?xml version=': 0, '>': 138, 'https://': 0, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '298996', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 5322, 'n_tokens_neox': 4874, 'n_words': 1557, 'path': 'example/test.ipynb', 'repo_name': 'scollis/scipy_2015', 'size': '15398'}