<jupyter_start><jupyter_text>We'll try to:- import the data- preprocess for a tensorflow model (following time series forecasting tutorial)- complete the first part of the modelling (up to a simple single-window model)<jupyter_code>import os
import datetime

import IPython
import IPython.display
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import tensorflow as tf

mpl.rcParams['figure.figsize'] = (8, 6)
mpl.rcParams['axes.grid'] = False
train_path = '/home/joyvan/cfm/train_dc2020.h5'
df = pd.read_hdf(train_path)
df.head()

df = df.loc[df['stock_id'] == 387 ]
# make a datetime (to improve?)
def mk_dt(df,num):
    df_dt= df['day_id'] + df[(num,'tod')]/1e11
    return df_dt<jupyter_output><empty_output><jupyter_text>We'd like to predict the next _source_id_ of trades. We have trades with a time of day, and a day ID for each day. the dataset runs over a whole year.For one row in the dataset we have the whole time window encapsulated in the ten trades, whose columns are ```sh(x,'source_id'), (x,'price'), (x,'qty'), (x,'tod')```We're going to isolate these into a reduced dataset and discard the rest, as the OB info will require significantly more preprocessing work to include in any model *note; we have one row of 6 order books for the latest trade (the label trade) and nothing for the rest of the window... so LSTM is not suited there*<jupyter_code>for j in range(10):
    df[(j,'dt')] = mk_dt(df,j)
    
df[(4,'dt')].head()
_cols_sid= [(0,'source_id'),(1,'source_id'),(2,'source_id'),(3,'source_id'),(4,'source_id'),(5,'source_id'),(6,'source_id'),(7,'source_id'),(8,'source_id'),(9,'source_id')]
_cols_price=[(0,'price'),(1,'price'),(2,'price'),(3,'price'),(4,'price'),(5,'price'),(6,'price'),(7,'price'),(8,'price'),(9,'price')]
_cols_qty=[(0,'qty'),(1,'qty'),(2,'qty'),(3,'qty'),(4,'qty'),(5,'qty'),(6,'qty'),(7,'qty'),(8,'qty'),(9,'qty')] 
_cols_tod=[(0,'tod'),(1,'tod'),(2,'tod'),(3,'tod'),(4,'tod'),(5,'tod'),(6,'tod'),(7,'tod'),(8,'tod'),(9,'tod')]


#def mk_ds()
plot_features=df[[_cols_sid[0],_cols_price[0],_cols_qty[0]]]
plot_features.index = df[(0,'dt')].sort_values()
_ = plot_features.plot(subplots=True)
plot_features.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.19787516600265603, 'fraction_numerical': 0.027003098716246128, 'mean_word_length': 7.100358422939068, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '469874', 'license': 'MIT', 'n_tokens_mistral': 872, 'n_tokens_neox': 842, 'n_words': 213, 'path': 'towards-lstm.ipynb', 'repo_name': 'nfilipov/notebooks', 'size': '2259'}