<jupyter_start><jupyter_text>Timeseries Forecasting This notebook explains how to use `tsfresh` in time series foreacasting.Make sure you also read through the [documentation](https://tsfresh.readthedocs.io/en/latest/text/forecasting.html) to learn more on this feature.We will use the stock price of Apple for this.In this notebook we will only showcase how to work with a single time series at a time (one stock).There exist another notebook in the `advanced` folder, which treats several stocks at the same time.Basically the same - but a bit more complex when it comes to pandas multi-indexing.<jupyter_code>%matplotlib inline

import numpy as np
import pandas as pd
import matplotlib.pylab as plt

from tsfresh import extract_features, select_features
from tsfresh.utilities.dataframe_functions import roll_time_series, make_forecasting_frame
from tsfresh.utilities.dataframe_functions import impute

try:
    import pandas_datareader.data as web
except ImportError:
    print("You need to install the pandas_datareader. Run pip install pandas_datareader.")

from sklearn.ensemble import AdaBoostRegressor<jupyter_output><empty_output><jupyter_text>Reading the data We download the data from "stooq" and only store the High value.Please note: this notebook is for showcasing `tsfresh`s feature extraction - not to predict stock market prices :-)<jupyter_code>df = web.DataReader("AAPL", 'stooq')["High"]
df.head()
plt.figure(figsize=(15, 6))
df.plot(ax=plt.gca())
plt.show()<jupyter_output><empty_output><jupyter_text>We want to make the time dependency a bit clearer and add an identifier to each of the stock values (in this notebook we only have Google though).<jupyter_code>df_melted = pd.DataFrame({"high": df.copy()})
df_melted["date"] = df_melted.index
df_melted["Symbols"] = "AAPL"

df_melted.head()<jupyter_output><empty_output><jupyter_text>Create training data sample Forecasting typically involves the following steps:* take all data up to today* do feature extraction (e.g. by running `extract_features`)* run a prediction model (e.g. a regressor, see below)* us the result as the forecast for tomorrow In training however, we need multiple examples to train.If we would only use the time series until today (and wait for the value of tomorrow to have a target), we would only have a single training example.Therefore we use a trick: we replay the history.Imagine you have a cut-out window sliding over your data.At each time step $t$, you treat the data as it would be today. You extract the features with everything you know until today (which is all data until and including $t$).The target for the features until time $t$ is the time value of time $t + 1$ (which you already know, because everything has already happened).The process of window-sliding is implemented in the function `roll_time_series`.Our window size will be 20 (we look at max 20 days in the past) and we disregard all windows which are shorter than 5 days.<jupyter_code>df_rolled = roll_time_series(df_melted, column_id="Symbols", column_sort="date",
                             max_timeshift=20, min_timeshift=5)
df_rolled.head()<jupyter_output><empty_output><jupyter_text>The resulting dataframe now consists of these "windows" stamped out of the original dataframe.For example all data with the `id = (AAPL, 2020-07-14 00:00:00)` comes from the original data of stock `AAPL` including the last 20 days until `2020-07-14`:<jupyter_code>df_rolled[df_rolled["id"] == ("AAPL", pd.to_datetime("2020-07-14"))]
df_melted[(df_melted["date"] <= pd.to_datetime("2020-07-14")) & 
          (df_melted["date"] >= pd.to_datetime("2020-06-15")) & 
          (df_melted["Symbols"] == "AAPL")]<jupyter_output><empty_output><jupyter_text>If you now group by the new `id` column, each of the groups will be a certain stock symbol until and including the data until a certain day (and including the last 20 days in the past).Whereas we started with 1259 data samples:<jupyter_code>len(df_melted)<jupyter_output><empty_output><jupyter_text>we now have 1254 unique windows (identified by stock symbol and ending date):<jupyter_code>df_rolled["id"].nunique()<jupyter_output><empty_output><jupyter_text>We "lost" 5 windows, as we required to have a minimum history of more than 5 days.<jupyter_code>df_rolled.groupby("id").size().agg([np.min, np.max])<jupyter_output><empty_output><jupyter_text>The process is also shown in this image (please note that the window size is smaller for better visibility):   Extract Features The rolled (windowed) data sample is now in the correct format to use it for `tsfresh`s feature extraction.As normal, features will be extracted using all data for a given `id`, which is in our case all data of a given window and a given id (one colored box in the graph above).If the feature extraction returns a row with the index `(AAPL, 2020-07-14 00:00:00)`, you know it has been calculated using the `AAPL` data up and including `2020-07-14` (and 20 days of history).<jupyter_code>X = extract_features(df_rolled.drop("Symbols", axis=1), 
                     column_id="id", column_sort="date", column_value="high", 
                     impute_function=impute, show_warnings=False)
X.head()<jupyter_output><empty_output><jupyter_text>We make the data a bit easier to work with by removing the tuple-index<jupyter_code>X = X.set_index(X.index.map(lambda x: x[1]), drop=True)
X.index.name = "last_date"
X.head()<jupyter_output><empty_output><jupyter_text>Our `(AAPL, 2020-07-14 00:00:00)` is also in the data again:<jupyter_code>X.loc['2020-07-14']<jupyter_output><empty_output><jupyter_text>Just to repeat: the features in this row were only calculated using the time series values of `AAPL` up to and including `2015-07-14` and the last 20 days.  Prediction We can now use the extracted features to train a regressor.But what will be our targets?The target for the row `2020-07-13` is the value on the next timestep (that would be `2020-07-14` in this case).So all we need to do is go back to our original dataframe and take the stock value of tomorrow.This is done with `shift`:<jupyter_code>y = df_melted.set_index("date").sort_index().high.shift(-1)<jupyter_output><empty_output><jupyter_text>Quick consistency test:<jupyter_code>y["2020-07-13"], df["2020-07-14"].iloc[0]<jupyter_output><empty_output><jupyter_text>However, we need to be a bit careful here: `X` is missing the first 5 dates (as our minimum window size was 5) and `y` is missing the last date (as there is nothing to predict on today).So lets make sure we have a consistent view on the data.<jupyter_code>y = y[y.index.isin(X.index)]
X = X[X.index.isin(y.index)]<jupyter_output><empty_output><jupyter_text>We can now train normal AdaBoostRegressors to predict the next time step .Let's split the data into a training and testing sample (but make sure to keep temporal consistency).We take everything until 2019 as train data an the rest as test:<jupyter_code>X[:"2018"]
X_train = X[:"2018"]
X_test = X["2019":]

y_train = y[:"2018"]
y_test = y["2019":]<jupyter_output><empty_output><jupyter_text>and do feature selection before training<jupyter_code>X_train_selected = select_features(X_train, y_train)
ada = AdaBoostRegressor()

ada.fit(X_train_selected, y_train)<jupyter_output><empty_output><jupyter_text>Now lets check how good our prediction is:<jupyter_code>X_test_selected = X_test[X_train_selected.columns]

y_pred = pd.Series(ada.predict(X_test_selected), index=X_test_selected.index)<jupyter_output><empty_output><jupyter_text>The prediction is for the next day, so for drawing we need to shift 1 step back:<jupyter_code>plt.figure(figsize=(15, 6))

y.plot(ax=plt.gca())
y_pred.plot(ax=plt.gca(), legend=None, marker=".")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09518882565959648, 'fraction_numerical': 0.025090532850491464, 'mean_word_length': 5.660637381567614, 'pattern_counts': {'":': 3, '<': 74, '<?xml version=': 0, '>': 74, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '87713', 'license': 'MIT', 'n_tokens_mistral': 2583, 'n_tokens_neox': 2386, 'n_words': 877, 'path': 'notebooks/examples/05 Timeseries Forecasting.ipynb', 'repo_name': 'alxfmpl/tsfresh', 'size': '7732'}