<jupyter_start><jupyter_text>Part 2 - Dealing with Timeseries as a non-timeseries problem<jupyter_code>import pandas as pd 
from sklearn.linear_model import LinearRegression
from matplotlib import pyplot as plt 
from sklearn.metrics import r2_score
from statsmodels.graphics.tsaplots import plot_pacf
from sklearn.ensemble import RandomForestRegressor
% matplotlib inline 

import warnings
warnings.filterwarnings(action="ignore")
warnings.filterwarnings(action="ignore", module="scipy", message="^internal gelsd")

import utils<jupyter_output><empty_output><jupyter_text>![](https://i.imgflip.com/2acblw.jpg) Let's get our airlines data<jupyter_code>airlines = utils.load_airline_data()<jupyter_output><empty_output><jupyter_text>Ok Neo, picture this. We take the next period's data, and make it the target.<jupyter_code>airlines_as_dataframe = pd.DataFrame(airlines)
airlines_as_dataframe['target'] = airlines.shift(-1)<jupyter_output><empty_output><jupyter_text>Now we take a few previous periods, and make them features.<jupyter_code>airlines_as_dataframe['1 period before'] = airlines.shift(1)
airlines_as_dataframe['2 periods before'] = airlines.shift(2)<jupyter_output><empty_output><jupyter_text>Behold.<jupyter_code>airlines_as_dataframe.tail(3)<jupyter_output><empty_output><jupyter_text>With this, we can train on the previous days, using them as features, and predict the target.  The idea is elegant, and sounds simple. There will however be a lot of weird Pandas to get the data ready. Remember Neo: > This section is completely optional.This is your last chance. ![](https://i.imgflip.com/2akbei.jpg) ------- Remember. All I'm offering you is the Truth.  We will be making some functions along the way, or this will get out of control fast. Firstly, a function to build our target, by getting the next period:<jupyter_code>def build_target(series_, number_of_periods_ahead):
    """ 
    takes a series, turned it into a dataframe, and adds a new column called target
    This column is the input series, lagged number_of_periods_ahead into the future
    """
    
    # make a copy 
    series_ = series_.copy()
    series_.name = 'observed_values'
    
    # make a dataframe from the series
    df_ = pd.DataFrame(series_)
    
    # the target column will be the input series, lagged into the future
    df_['target'] = series_.shift(-number_of_periods_ahead)
    return df_<jupyter_output><empty_output><jupyter_text>What does this do to our `airlines` data?<jupyter_code>airlines_with_target = build_target(airlines, number_of_periods_ahead=1)

airlines_with_target.tail()<jupyter_output><empty_output><jupyter_text>Quite simple, the target becomes the value of the next period. The last day, of course, does not have a target.  Now, let's build some more features manually, by taking the differences between consecutive days. Maybe this helps to give the model an idea of how things are changing:<jupyter_code>def build_some_features(df_, num_diffs, num_periods_lagged): 
    """
    Builds some features by calculating differences between periods  
    """
    # make a copy 
    df_ = df_.copy()
    
    # for a few values, get the diffs 
    for i in range(1, num_diffs+1):
        # make a new feature, with the diffs in the observed values column
        df_['diffed_%s' % str(i)] = df_['observed_values'].diff(i)
        
    # for a few values, get the lags  
    for i in range(1, num_periods_lagged+1):
        # make a new feature, with the diffs in the observed values column
        df_['lagged_%s' % str(i)] = df_['observed_values'].shift(i)
        
    return df_
airlines_with_some_hand_made_features = build_some_features(airlines_with_target, 2, 2)
airlines_with_some_hand_made_features.tail()<jupyter_output><empty_output><jupyter_text>So let's see what this mean... > If I take `observed values` on day `1960-10-31`     (461.0)   > ... and subtract `observed values` on day `1960-08-31` (606.0)  > I get `diffed_2` for day `1960-10-31`               (-145.0)> If I take `observed values` on day `1960-10-31`     (461.0)   > And move it to `1960-11-30`  > I've created a `lagged_1` feature for day  `1960-11-30`  (461.0)    Does that make sense? There are obviously a lot more features we can hand-engineer, but let's start with the super basic stuff first. Next up, let's separate that last day (which we can't use for training), and separate the features from the target:<jupyter_code>def separate_last_day(df_):
    
    """
    takes a dataset which has the target and features built 
    and separates it into the last day
    """
    # take the last period 
    last_period = df_.iloc[-1]
    
    # the last period is now a series, so it's name will be the timestamp
    training_data = df_.loc[df_.index < last_period.name]

    return last_period, training_data<jupyter_output><empty_output><jupyter_text>Does that work?<jupyter_code>last_period, training_data = separate_last_day(airlines_with_some_hand_made_features)<jupyter_output><empty_output><jupyter_text>What is our last period?<jupyter_code>last_period<jupyter_output><empty_output><jupyter_text>And the training data?<jupyter_code>training_data.tail(3)<jupyter_output><empty_output><jupyter_text>Excellent, stops right before our last period. One reminder, it will still have missing data (creating our features made some missing data at the start), but we'll just get rid of it when the time comes:<jupyter_code>training_data.head(3)<jupyter_output><empty_output><jupyter_text>Lastly, let's make a method for separating the features and targets, so we get our complete train test split:<jupyter_code>def separate_train_and_test_set(last_period_, training_data_, target='target'): 
    
    """ 
    separates training and test set (clue was in the name, really... )
    Ok, we were lazy and left the target hardcoded as 'target'. Shame on us. 
    """
    
    # anything that isn't a target is a feature 
    features = [feature for feature in training_data_.columns if feature != target]
    
    # adding a sneaky little dropna to avoid the missing data problem above 
    X_train = training_data_.dropna()[features]
    y_train = training_data_.dropna()[target]
    
    X_last_period = last_period_[features]
    
    return X_train, y_train, X_last_period
X_train, y_train, X_last_period = separate_train_and_test_set(last_period, training_data, target='target')
X_last_period<jupyter_output><empty_output><jupyter_text>Let's take a look at our outputs:<jupyter_code>print(X_train.tail(2), end='\n\n')
print(y_train.tail(2), end='\n\n')
print(X_last_period.tail(2))<jupyter_output>observed_values  diffed_1  diffed_2  lagged_1  lagged_2
Month                                                              
1960-09-30            508.0     -98.0    -114.0     606.0     622.0
1960-10-31            461.0     -47.0    -145.0     508.0     606.0

Month
1960-09-30    461.0
1960-10-31    390.0
Freq: M, Name: target, dtype: float64

lagged_1    461.0
lagged_2    508.0
Name: 1960-11-30 00:00:00, dtype: float64<jupyter_text>Great. Now let's make a utility function to put all this together:<jupyter_code>def prepare_for_prediction(series_, number_of_periods_ahead, num_diffs, num_periods_lagged):
    
    """ 
    Wrapper to go from the original series to X_train, y_train, X_last_period 
    
    """
    
    # build the target 
    data_with_target = build_target(series_, 
                                    number_of_periods_ahead)
    
    # build the features 
    data_with_target_and_features = build_some_features(data_with_target, 
                                                             num_diffs=num_diffs, 
                                                        num_periods_lagged=num_periods_lagged)
    # separate train and test data 
    last_period, training_data = separate_last_day(data_with_target_and_features)

    # separate X_train, y_train, and X_test 
    X_train, y_train, X_last_period = separate_train_and_test_set(last_period, 
                                                           training_data, 
                                                           target='target')
    
    # return ALL OF THE THINGS! (well, actually just the ones we need)
    return X_train, y_train, X_last_period<jupyter_output><empty_output><jupyter_text>Did that work?<jupyter_code>X_train, y_train, X_last_period = prepare_for_prediction(airlines, 
                                                         number_of_periods_ahead=1, 
                                                         num_diffs=3, 
                                                        num_periods_lagged=3)
# this is just to see X train and y train side by side 
pd.concat([X_train, y_train], axis=1).tail()
# what about our last period X? 
X_last_period<jupyter_output><empty_output><jupyter_text>Huzzah! Now we can treat this as a normal regression problem (kind of). Let's try to predict the next day, using a super-basic [vanilla](https://youtu.be/rog8ou-ZepE?t=58s) [Linear Regression](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html):<jupyter_code># what's the dumbest, simplest model I can think of? 
lr = LinearRegression()
lr.fit(X_train, y_train);<jupyter_output><empty_output><jupyter_text>So what would this model predict for our next day?<jupyter_code>lr.predict(X_last_period.values.reshape(1, -1))<jupyter_output><empty_output><jupyter_text>Note: we will be predicting a single observation, hence the annoying `.values.reshape` stuff. The API changes a bit when predicting a single point. If you forget, Pandas will tell you what to do, so don't worry too much about it) How did our model make its prediction?<jupyter_code>def explain_linear_regression(lr, features):
    
    betas = lr.coef_
    print('Regression: \n(%0.3f * %s)' % (betas[0], features[0]))
    for i in range(1, len(betas)): 
        print('+ (%0.3f * %s)' % (betas[i], features[i]))

explain_linear_regression(lr, X_train.columns)<jupyter_output>Regression: 
(0.461 * observed_values)
+ (0.503 * diffed_1)
+ (0.204 * diffed_2)
+ (0.181 * diffed_3)
+ (-0.042 * lagged_1)
+ (0.258 * lagged_2)
+ (0.280 * lagged_3)<jupyter_text>----  ![](https://i.imgflip.com/2akdo8.jpg) ---- At this point, we have two options. We can either:1. take the prediction we've just made, feed it as truth into the model, and predict again. 2. Just train the model to predict two days in advance. Given that we aren't insane, we'll choose 2.<jupyter_code>X_train, y_train, X_last_period = prepare_for_prediction(airlines, 
                                                         number_of_periods_ahead=2, 
                                                         num_diffs=3, 
                                                        num_periods_lagged=3)
pd.concat([X_train, y_train], axis=1).tail()<jupyter_output><empty_output><jupyter_text>Quick note: we now have data until `1960-09-30`, we used to have 1 more period. However because we are predicting two periods in advance, we don't have data for the last two days. Makes sense? You will also notice that `X_last_period` is `1960-11-30`, two periods after the last entry in the training data:<jupyter_code>X_last_period<jupyter_output><empty_output><jupyter_text>Great! Fit it!<jupyter_code># weeeee 
lr = LinearRegression()
lr.fit(X_train, y_train);
lr.predict(X_last_period.values.reshape(1, -1))<jupyter_output><empty_output><jupyter_text>And yes, that does indeed mean that you can use whatever model you want.  ![](https://i.imgflip.com/2akf0s.jpg) But beware, fitting the models and particularly validating them by hand is not particularly trivial.  Predicting as many periods as we like  (I told you to take the blue pill, Neo.)<jupyter_code>def predict_period_n(series_, model, number_of_periods_ahead, num_diffs, num_periods_lagged): 
    
        X_train, y_train, X_last_period = prepare_for_prediction(series_, 
                                                             number_of_periods_ahead, 
                                                             num_diffs, 
                                                            num_periods_lagged)
        
        model.fit(X_train, y_train);
        return model.predict(X_last_period.values.reshape(1, -1))
def predict_n_periods(series_, n_periods, model, num_diffs, num_periods_lagged): 
    predictions = []

    for period_ahead in range(1, n_periods+1):
        pred = predict_period_n(series_=series_, 
                                model=model, 
                                number_of_periods_ahead=period_ahead, 
                                num_diffs=num_diffs,
                                num_periods_lagged=num_periods_lagged)
        
        predictions.append(pred[0])
        
    return predictions<jupyter_output><empty_output><jupyter_text>Let's predict a few periods ahead:<jupyter_code>predict_n_periods(series_=airlines, 
                  n_periods=2, 
                  model=LinearRegression(), 
                  num_diffs=3, 
                  num_periods_lagged=3)<jupyter_output><empty_output><jupyter_text>How did we do?  Actually we have no idea. We don't actually have the future to know what the answer should be. Let's get a quick split _(you'll get better ways to do this in Part 3 - workflows)_<jupyter_code>split_date = '1960-02-29'
train = airlines.loc[airlines.index < split_date]
test = airlines.loc[airlines.index >= split_date].iloc[0:3]<jupyter_output><empty_output><jupyter_text>Did it work?<jupyter_code>train.tail(2)
test<jupyter_output><empty_output><jupyter_text>Yep. Now, let's train on `train`, and predict 4 days of `test`<jupyter_code>predictions = predict_n_periods(series_=train, 
                  n_periods=3, 
                  model=LinearRegression(), 
                  num_diffs=3, 
                  num_periods_lagged=3)

print('Predictions: %s'  % predictions)

print('\nHow wrong were we on each day?')
print(test - predictions)<jupyter_output>Predictions: [418.7765068623498, 421.668065081549, 405.70519573607316]

How wrong were we on each day?
Month
1960-02-29   -27.776507
1960-03-31    -2.668065
1960-04-30    55.294804
Freq: M, Name: passengers_thousands, dtype: float64<jupyter_text>Not amazing... How about more features?<jupyter_code>predictions = predict_n_periods(series_=train, 
                  n_periods=3, 
                  model=LinearRegression(), 
                  num_diffs=7, 
                  num_periods_lagged=7)

print('Predictions: %s'  % predictions)

print('\nHow wrong were we on each day?')
print(test - predictions)<jupyter_output>Predictions: [415.6171997346689, 423.21634069801127, 415.83073167649667]

How wrong were we on each day?
Month
1960-02-29   -24.617200
1960-03-31    -4.216341
1960-04-30    45.169268
Freq: M, Name: passengers_thousands, dtype: float64<jupyter_text>Meh. Different model?<jupyter_code>predictions = predict_n_periods(series_=train, 
                  n_periods=3, 
                  model=RandomForestRegressor(n_estimators=100, max_depth=3, min_samples_split=4), 
                  num_diffs=7, 
                  num_periods_lagged=7)

print('Predictions: %s'  % predictions)

print('\nHow wrong were we on each day?')
print(test - predictions)<jupyter_output>Predictions: [425.38838998426763, 424.73036836782273, 397.71502562247014]

How wrong were we on each day?
Month
1960-02-29   -34.388390
1960-03-31    -5.730368
1960-04-30    63.284974
Freq: M, Name: passengers_thousands, dtype: float64
{'fraction_non_alphanumeric': 0.0902221648152932, 'fraction_numerical': 0.039653836218031514, 'mean_word_length': 3.217047930283224, 'pattern_counts': {'":': 0, '<': 118, '<?xml version=': 0, '>': 124, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 29, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '324110', 'license': 'MIT', 'n_tokens_mistral': 5365, 'n_tokens_neox': 4859, 'n_words': 1318, 'path': 'learning_notebooks/Part 2 - Dealing with Timeseries as a non-timeseries problem .ipynb', 'repo_name': 'AlexGomesDS/batch2-BLU03', 'size': '15484'}