<jupyter_start><jupyter_text>How to use CNN with time series data The regular measurements of time series result in a similar grid-like data structure as for the image data we have focused on so far. As a result, we can use CNN architectures for univariate and multivariate time series. In the latter case, we consider different time series as channels, similar to the different color signals.  Imports & Settings<jupyter_code>%matplotlib inline

import sys
from time import time
from pathlib import Path

import numpy as np
import pandas as pd
from scipy.stats import spearmanr
from sklearn.feature_selection import mutual_info_regression

import tensorflow as tf
tf.autograph.set_verbosity(0, True)
from tensorflow.keras.models import Sequential
from tensorflow.keras import regularizers
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.layers import (Dense, 
                                     Flatten, 
                                     Conv1D, 
                                     MaxPooling1D, 
                                     Dropout, 
                                     BatchNormalization)

import matplotlib.pyplot as plt
import seaborn as sns
gpu_devices = tf.config.experimental.list_physical_devices('GPU')
if gpu_devices:
    print('Using GPU')
    tf.config.experimental.set_memory_growth(gpu_devices[0], True)
else:
    print('Using CPU')
sys.path.insert(1, Path(sys.path[0], '..').as_posix())
from utils import MultipleTimeSeriesCV, format_time
np.random.seed(1)
tf.random.set_seed(1)
sns.set_style('whitegrid')
results_path = Path('results', 'time_series')
if not results_path.exists():
    results_path.mkdir(parents=True)<jupyter_output><empty_output><jupyter_text>Prepare Data<jupyter_code>prices = (pd.read_hdf('../data/assets.h5', 'quandl/wiki/prices')
          .adj_close
          .unstack().loc['2000':])
prices.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 4706 entries, 2000-01-03 to 2018-03-27
Columns: 3199 entries, A to ZUMZ
dtypes: float64(3199)
memory usage: 114.9 MB<jupyter_text>Compute monthly returns<jupyter_code>returns = (prices
           .resample('M')
           .last()
           .pct_change()
           .dropna(how='all')
           .loc['2000': '2017']
           .dropna(axis=1)
           .sort_index(ascending=False))

# remove outliers likely representing data errors
returns = returns.where(returns<1).dropna(axis=1)
returns.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 215 entries, 2017-12-31 to 2000-02-29
Freq: -1M
Columns: 1511 entries, A to ZQK
dtypes: float64(1511)
memory usage: 2.5 MB<jupyter_text>Create model data<jupyter_code>n = len(returns)
nlags = 12
lags = list(range(1, nlags + 1))
cnn_data = []
for i in range(n-nlags-1):
    df = returns.iloc[i:i+nlags+1]        # select outcome and lags
    date = df.index.max()                 # use outcome date
    cnn_data.append(df.reset_index(drop=True)  # append transposed series
                    .transpose()
                    .assign(date=date)
                    .set_index('date', append=True)
                    .sort_index(1, ascending=True))

cnn_data = (pd.concat(cnn_data)
            .rename(columns={0: 'label'})
            .sort_index())

cnn_data.info(null_counts=True)<jupyter_output><class 'pandas.core.frame.DataFrame'>
MultiIndex: 305222 entries, ('A', Timestamp('2001-03-31 00:00:00')) to ('ZQK', Timestamp('2017-12-31 00:00:00'))
Data columns (total 13 columns):
 #   Column  Non-Null Count   Dtype  
---  ------  --------------   -----  
 0   label   305222 non-null  float64
 1   1       305222 non-null  float64
 2   2       305222 non-null  float64
 3   3       305222 non-null  float64
 4   4       305222 non-null  float64
 5   5       305222 non-null  float64
 6   6       305222 non-null  float64
 7   7       305222 non-null  float64
 8   8       305222 non-null  float64
 9   9       305222 non-null  float64
 10  10      305222 non-null  float64
 11  11      305222 non-null  float64
 12  12      305222 non-null  float64
dtypes: float64(13)
memory usage: 31.5+ MB<jupyter_text>Evaluate features  Mutual Information<jupyter_code># mi = mutual_info_regression(X=cnn_data.drop('label', axis=1), y=cnn_data.label)
# mi = pd.Series(mi, index=cnn_data.drop('label', axis=1).columns)<jupyter_output><empty_output><jupyter_text>Information Coefficient<jupyter_code># ic = {}
# for lag in lags:
#     ic[lag] = spearmanr(cnn_data.label, cnn_data[lag])
# ic = pd.DataFrame(ic, index=['IC', 'p-value']).T
# ax = ic.plot.bar(rot=0, figsize=(14, 4),
#                  ylim=(-0.05, .05),
#                  title='Feature Evaluation')
# ax.set_xlabel('Lag')
# sns.despine()
# plt.tight_layout()
# plt.savefig(results_path / 'cnn_ts1d_feature_ic', dpi=300)<jupyter_output><empty_output><jupyter_text>Plot Metrics<jupyter_code># metrics = pd.concat([mi.to_frame('Mutual Information'),
#                      ic.IC.to_frame('Information Coefficient')], axis=1)
# ax = metrics.plot.bar(figsize=(12, 4), rot=0)
# ax.set_xlabel('Lag')
# sns.despine()
# plt.tight_layout()
# plt.savefig(results_path / 'ts1d_metrics', dpi=300)<jupyter_output><empty_output><jupyter_text>CNN  Model Architecture We design a simple one-layer CNN that uses one-dimensional convolutions combined with max pooling to learn time series patterns:<jupyter_code>def get_model(filters=32, kernel_size=5, pool_size=2):
    model = Sequential([Conv1D(filters=filters,
                               kernel_size=kernel_size,
                               activation='relu',
                               padding='causal',
                               input_shape=input_shape,
                               use_bias=True,
                               kernel_regularizer=regularizers.l1_l2(l1=1e-5,
                                                                     l2=1e-5)),
                        MaxPooling1D(pool_size=pool_size),
                        Flatten(),
                        BatchNormalization(),
                        Dense(1, activation='linear')])
    model.compile(loss='mse',
                  optimizer='Adam')
    return model<jupyter_output><empty_output><jupyter_text>Set up CV<jupyter_code>cv = MultipleTimeSeriesCV(n_splits=12 * 3,
                          train_period_length=12 * 5,
                          test_period_length=1,
                          lookahead=1)
input_shape = nlags, 1<jupyter_output><empty_output><jupyter_text>Train Model<jupyter_code>def get_train_valid_data(X, y, train_idx, test_idx):
    x_train, y_train = X.iloc[train_idx, :], y.iloc[train_idx]
    x_val, y_val = X.iloc[test_idx, :], y.iloc[test_idx]
    m = X.shape[1]
    return (x_train.values.reshape(-1, m, 1), y_train,
            x_val.values.reshape(-1, m, 1), y_val)
batch_size = 64
epochs = 100
filters = 32
kernel_size = 4
pool_size = 4
get_model(filters=filters,
          kernel_size=kernel_size, 
          pool_size=pool_size).summary()
result = {}
start = time()
for fold, (train_idx, test_idx) in enumerate(cv.split(cnn_data)):
    X_train, y_train, X_val, y_val = get_train_valid_data(cnn_data
                                                          .drop('label', axis=1)
                                                          .sort_index(ascending=False),
                                                          cnn_data.label,
                                                          train_idx,
                                                          test_idx)
    test_date = y_val.index.get_level_values('date').max()
    model = get_model(filters=filters,
                      kernel_size=kernel_size,
                      pool_size=pool_size)
    
    best_ic = -np.inf
    stop = 0
    for epoch in range(50):         
        training = model.fit(X_train, y_train,
                             batch_size=batch_size,
                             validation_data=(X_val, y_val),
                             epochs=epoch + 1,
                             initial_epoch=epoch,
                             verbose=0,
                             shuffle=True)
        predicted = model.predict(X_val).squeeze()
        ic, p_val_ = spearmanr(predicted, y_val)
        if ic > best_ic:
            best_ic = ic
            p_val = p_val_
            stop = 0
        else:
            stop += 1
        if stop == 10:
            break
    
    nrounds = epoch + 1 - stop
    result[test_date] = [nrounds, best_ic, p_val]
    df = pd.DataFrame(result, index=['epochs', 'IC', 'p-value']).T
    msg = f'{fold + 1:02d} | {format_time(time()-start)} | {nrounds:3.0f} | '
    print(msg + f'{best_ic*100:5.2} ({p_val:7.2%}) | {df.IC.mean()*100:5.2}')<jupyter_output>01 | 00:00:41 |   3 |   1.9 ( 45.76%) |   1.9
02 | 00:01:46 |  11 |   2.4 ( 34.95%) |   2.2
03 | 00:03:04 |  15 |   1.4 ( 59.61%) |   1.9
04 | 00:03:38 |   1 |  -1.6 ( 52.22%) |   1.0
05 | 00:04:13 |   1 |   3.0 ( 24.90%) |   1.4
06 | 00:04:46 |   1 |   4.3 (  9.63%) |   1.9
07 | 00:05:55 |  14 |   4.2 ( 10.18%) |   2.2
08 | 00:06:38 |  17 |   2.4 ( 35.67%) |   2.2
09 | 00:06:58 |   2 |   2.5 ( 32.65%) |   2.3
10 | 00:07:22 |   4 |   2.5 ( 33.57%) |   2.3
11 | 00:07:41 |   1 |  0.96 ( 70.92%) |   2.2
12 | 00:08:02 |   1 |   4.7 (  6.93%) |   2.4
13 | 00:08:24 |   2 |   3.6 ( 16.38%) |   2.5
14 | 00:08:55 |   7 |   5.0 (  5.07%) |   2.7
15 | 00:09:14 |   1 |   2.9 ( 26.00%) |   2.7
16 | 00:10:08 |  23 |   2.4 ( 34.92%) |   2.7
17 | 00:10:51 |   8 |   3.3 ( 20.00%) |   2.7
18 | 00:11:28 |   2 |   2.8 ( 28.12%) |   2.7
19 | 00:13:05 |  22 |   4.6 (  7.32%) |   2.8
20 | 00:13:39 |   1 |   2.0 ( 44.75%) |   2.8
21 | 00:14:14 |   1 |   3.3 ( 20.26%) |   2.8
22 | 00:14:56 |   4 | -0.51 ( 84.3[...]<jupyter_text>Evaluate Results<jupyter_code>metrics = pd.DataFrame(result, index=['epochs', 'IC', 'p-value']).T
ax = metrics.IC.plot(figsize=(12, 4), 
                     label='Information Coefficient',
                     title='Validation Performance',
                    ylim=(0, .08))
metrics.IC.expanding().mean().plot(ax=ax, label='Cumulative Average')
plt.legend()
sns.despine()
plt.tight_layout()
plt.savefig(results_path / 'cnn_ts1d_ic', dpi=300);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1290732948429396, 'fraction_numerical': 0.07349055680594971, 'mean_word_length': 2.129210042865891, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '538420', 'license': 'MIT', 'n_tokens_mistral': 3963, 'n_tokens_neox': 3503, 'n_words': 741, 'path': '18_convolutional_neural_nets/04_time_series_prediction.ipynb', 'repo_name': 'driscolljt/Machine-Learning-for-Algorithmic-Trading-Second-Edition_Original', 'size': '10219'}