<jupyter_start><jupyter_text>Sliding Windows  Temporal Correlations**We have show how our time series is almost periodical**<jupyter_code>nab.plot_series(data.iloc[:3*48], figsize=figsize)<jupyter_output><empty_output><jupyter_text>* ...But the period is not the only temporal correlation!* There are patterns (down, up, stable) even between _nearby time points_  Temporal Correlations**Another way to see that: let's check again the autocorrelation plot**<jupyter_code>nab.plot_autocorrelation(data, max_lag=50, figsize=figsize)<jupyter_output><empty_output><jupyter_text>* The correlation is strong up to 4-5 lags  Temporal Correlations**These correlations are _a source of information_*** They could be exploited to improve our estimated probabilities* ...But our models so far make _no use_ of them**If we want to take advantage of them:*** We need to feed _sequences_ to our estimators* ...Rather than individual observations**Our previous approach for combining multiple observations is _not enough_:*** It was based on assuming i.i.d. observations* ...And here we want to exploit dependencies!  Sliding Window**A common approach consist in using a _sliding window_*** We choose a _window length $w$_, i.e. the length of each sub-sequence* We place the "window" at the beginning of the series* ...We extract the corresponding observations* Then, we move the forward by a certain _stride_ and we repeat  Sliding Window**The result is a table**Let $m$ be the number of examples and $w$ be the window length|               | $\bf s_0$ | $\bf s_1$    | $\bf \ldots$ | $\bf s_{w-1}$ || ------------- | --------- | ------------ | ------------ | ------------- | | $\bf t_{w-1}$ | $x_0$     | $x_1$        | $\ldots$     | $x_{w-1}$     || $\bf t_{w}$   | $x_1$     | $x_2$        | $\ldots$     | $x_{w}$       || $\bf t_{w+1}$ | $x_2$     | $x_3$        | $\ldots$     | $x_{w+1}$     || $\bf \vdots$  | $\vdots$  | $\vdots$     | $\vdots$     | $\vdots$      || $\bf t_{m-1}$ | $x_{m-w}$ | $x_{m-w+1}$  | $\vdots$     | $x_{m-1}$     |* The first window includes observations from $x_0$ to $x_{w-1}$* The second from $x_1$ to $x_{w}$ and so on* $t_i$ is the _time window index_ (where it was applied)* $s_j$ is the _position_ of an observation _within a window_  Sliding Window in pandas**pandas provides a sliding window _iterator_**```pythonDataFrame.rolling(window, ...)```<jupyter_code>wlen = 48
for i, w in enumerate(data['value'].rolling(wlen)):
    print(w)
    if i == 2: break # We print the first three windows<jupyter_output>timestamp
2014-07-01    0.357028
Name: value, dtype: float64
timestamp
2014-07-01 00:00:00    0.357028
2014-07-01 00:30:00    0.267573
Name: value, dtype: float64
timestamp
2014-07-01 00:00:00    0.357028
2014-07-01 00:30:00    0.267573
2014-07-01 01:00:00    0.204458
Name: value, dtype: float64<jupyter_text>Notice how the first windows are not full (shorter than `wlen`)  Sliding Window in pandas**We can build our dataset using the `rolling` iterator*** We discard the first `wlen-1` (incomplete) applications* Then we store each window in a list* Finally we wrap everything in a `DataFrame`<jupyter_code>%%time
rows = []
for i, w in enumerate(data['value'].rolling(wlen)):
    if i >= wlen-1: rows.append(w.values)

wdata_index = data.index[wlen-1:]
cols = range(wlen)
wdata = pd.DataFrame(index=wdata_index, columns=cols, data=rows)<jupyter_output>CPU times: user 1.35 s, sys: 3.1 ms, total: 1.35 s
Wall time: 1.36 s<jupyter_text>* The `values` field allows access to the `Series` content as a numpy array* We use it to _discard the index_* ...Since the series for multiple iterations have inconsistent indexes  Sliding Window in pandas**This method works, but _it's a bit slow_*** We are building our table by rows...* ...But it is usually _faster to do it by columns_!* After all, there are usually _fewer columns than rows_**Let us look again at our table:**|               | $\bf s_0$ | $\bf s_1$    | $\bf \ldots$ | $\bf s_{w-1}$ || ------------- | --------- | ------------ | ------------ | ------------- | | $\bf t_{w-1}$ | $x_0$     | $x_1$        | $\ldots$     | $x_{w-1}$     || $\bf t_{w}$   | $x_1$     | $x_2$        | $\ldots$     | $x_{w}$       || $\bf t_{w+1}$ | $x_2$     | $x_3$        | $\ldots$     | $x_{w+1}$     || $\bf \vdots$  | $\vdots$  | $\vdots$     | $\vdots$     | $\vdots$      || $\bf t_{m-1}$ | $x_{m-w}$ | $x_{m-w+1}$  | $\vdots$     | $x_{m-1}$     |  Sliding Window in pandas**We can build the columns by _slicing the original `DataFrame`_**<jupyter_code>m = len(data)
c0 = data.iloc[0:m-wlen+1]   # first column
c1 = data.iloc[1:m-wlen+1+1] # second column
print(c0.iloc[0:3])
print(c1.iloc[0:3])<jupyter_output>value
timestamp                    
2014-07-01 00:00:00  0.357028
2014-07-01 00:30:00  0.267573
2014-07-01 01:00:00  0.204458
                        value
timestamp                    
2014-07-01 00:30:00  0.267573
2014-07-01 01:00:00  0.204458
2014-07-01 01:30:00  0.153294<jupyter_text>* `iloc` in pandas allows to address a `DataFrame` by _position_  Sliding Window in pandas**Now we collect all columns in a list and we _stack them_**<jupyter_code>lc = [data.iloc[i:m-wlen+i+1].values for i in range(0, wlen)]
lc = np.hstack(lc)
wdata = pd.DataFrame(index=wdata_index, columns=cols, data=lc)
wdata.head()<jupyter_output><empty_output><jupyter_text>Sliding Window in pandas**We can wrap this approach in a function:**```pythondef sliding_window_1D(data, wlen):    m = len(data)    lc = [data.iloc[i:m-wlen+i+1] for i in range(0, wlen)]    wdata = np.hstack(lc)    wdata = pd.DataFrame(index=data.index[wlen-1:],            data=wdata, columns=range(wlen))    return wdata```<jupyter_code>wdata = nab.sliding_window_1D(data, wlen=wlen)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1622700451232211, 'fraction_numerical': 0.05744533148212426, 'mean_word_length': 3.2688888888888887, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '368230', 'license': 'MIT', 'n_tokens_mistral': 2209, 'n_tokens_neox': 1990, 'n_words': 626, 'path': 'notebooks/1. Sliding Windows.ipynb', 'repo_name': 'lompabo/aiiti-course-2021-02', 'size': '5762'}