<jupyter_start><jupyter_text>DCAE: Deep Convolutional Autoencoder> This notebook encodes the data using the autoencoder described in [TimeCluster](https://link.springer.com/article/10.1007/s00371-019-01673-y).<jupyter_code>try:
    from tsai.all import *
except:
    from tsai.all import * # TODO: Weird error when loading tsai!from tchub.all import *
import wandb
wandb_api = wandb.Api()
from fastcore.all import *
from fastai.callback.wandb import WandbCallback
from tchub.all import *<jupyter_output><empty_output><jupyter_text>For the experiment tracking and hyperparameter we will use the tool **Weights & Biases**. Before running this notebook, make sure you have the `$WANDB_API_KEY` environment varibale defined with your API_KEY (run in a terminal `echo $WANDB_API_KEY` to see it). If not, run in a terminal `wandb login [API_KEY]`. You can see your API_KEY [here](https://wandb.ai/authorize) or in the settings of your W&B account.  Notebook configuration<jupyter_code>config = AttrDict(
    use_wandb = True, # Whether to use or not wandb for experiment tracking
    wandb_group = None, # Whether to group this run in a wandb group
    wandb_entity = 'pacmel',
    wandb_project = 'tchub',
    train_artifact = 'pacmel/tchub/ABP:v0', # complete name
    valid_artifact = None, # complete name (None for random validation set)
    valid_size = 0.1,  # In case valid_artifact is None, this will set the percentage of random items to go to val
    w = 210, # window size for the sliding window
    stride = 1, # n datapoints the window is moved ahead along the sequence in the sliding window
    delta = 60, # Size of the autoencoder bottleneck layer (for now use the same as w)
    nfs = [90,60,30], #The number of filters in the last conv layer must be equal to the product of pool sizes
    kss = [10,5,5],
    output_filter_size = 10,
    pool_szs = [2, 3, 5], # The product of pool sizes must be a divisor of the window size
    batch_size = 64,
    epochs = 50,
    top_k = 3 # Number of elements to analyse for the top losses
)
run = wandb.init(entity = config.wandb_entity,
                      project=config.wandb_project,
                      group=config.wandb_group,
                      job_type='encoder_DCAE',
                      allow_val_change=True,
                      mode='online' if config.use_wandb else 'disabled',
                      config=config,
                      resume=False)
# Botch to use artifacts offline
artifacts_gettr = run.use_artifact if config.use_wandb else wandb_api.artifact<jupyter_output><empty_output><jupyter_text>Loading the datasetsTo load the dataset we will download a specific dataset artifact from the collection of artifactsstored in the weights and biases (wandb) project associated to this experiment.  Sliding window features Define a continuous multivariate time-series data $D$ of dimension $d$ with $n$ time-steps, $D = X_1,X_2,\dots,X_n$ , where each $X_i = \{x_i^1,\dots,x_i^d\}$ . Let $w$ be the window width, $s$ the stride, and $t$ the start time of a sliding window in the data.Define a new matrix $Z_k$ where each row is a vector of size $w$ of data extracted from the $k^{th}$ dimension.\begin{aligned}&Z_k(w,s,t)\\&\quad =\begin{bmatrix} x_{t}^k&\quad x_{t+1}^k&\quad \dots&\quad x_{t+w-1}^k \\ x_{t+s}^k&\quad x_{t+s+1}^k&\quad \dots&\quad x_{t+s+w-1}^k \\ \vdots&\quad \vdots&\quad \ddots&\quad \vdots \\ x_{t+(r-1)s}^k&\quad x_{t+(r-1)s+1}^k&\quad \dots&\quad x_{t+(r-1)s+w-1}^k \end{bmatrix} \end{aligned}where $r$ is the number of desired rows, and $t+(r-1)s+w-1 \le n$ $Z$ is a $w \times s \times t$ matrix. The first step consists in slicing the original multivariate time series into slices of shape ($w \times d$), as shown in this figure from the paper. The hyperparameters of this sliding window approach are given values by default here. If the value has been already set previously, that means this notebook is being called from a wandb sweep, and we must use the value that the sweep is bringing.<jupyter_code>sw = SlidingWindow(window_len=config.w, stride=config.stride, get_y=[])
train_artifact = artifacts_gettr(config.train_artifact)
df_train = train_artifact.to_df()
X_train, _ = sw(df_train)
df_train.shape, X_train.shape
if config.valid_artifact:
    valid_artifact = artifacts_gettr(config.valid_artifact)
    df_val = valid_artifact.to_df()
    X_valid, _ = sw(df_val)
    df_val.shape, X_valid.shape<jupyter_output><empty_output><jupyter_text>Extract important features from the multivariate time series data through Deep Convolutional Autoencoders Deep Convolutional Auto Encoders (DCAE) is a powerful method for learning high-level and mid-level abstractions from low-level raw data. It has the ability to extract features from complex and large time-series in an unsupervised manner. This is useful to overcome the complexity of multivariate time-series.Compared to the conventional auto-encoder, DCAE has fewer parameters than the conventional auto-encoder which means less training time. Also, DCAE uses local information to reconstruct the signal while conventional auto-encoders utilize fully connected layers to globally do the reconstruction. DCAE is an unsupervised model for representation learning which maps inputs into a new representation space. It has two main parts which are the encoding part that is used to project the data into a set of feature spaces and the decoding part that reconstructs the original data. The latent space representation is the space where the data lie in the bottleneck layers.The loss function of the DCAE is defined as the error between the input and the output. DCAE aims to find a code for each input by minimizing the mean squared error (MSE) between its input (original data) and output (reconstructed data). The MSE is used which assists to minimize the loss; thus, the network is forced to learn a low-dimensional representation of the input.We will implement the DCAE of the paper [TimeCluster](https://link.springer.com/article/10.1007/s00371-019-01673-y), whose architecture is shown in the table below:![](https://i.imgur.com/3EjuAfQ.png) Note that, in the paper, the input shape is $60 \times 3$, due to multivariate time series has 3 variables and the window size is 60. Generally, the size of the input/output of the autoencoder will depend on the shape of each slice obtained in the previos step. The number of latent features to be discovered is $60$ in the table above, but we can consider this as a free hyperparameter $\delta$. Also, according to the paper: "*The number of feature maps, size of filter and depth of the model are set based on the reconstruction error on validation set.*". Thus, we must provide flexibility in the creation of the DCAE in terms of these hyperparameters.º In case you are not using a config file, you can also uncomment the following cell and define the hyperparameters in the fly<jupyter_code>all_equal([len(x) for x in [config.nfs, config.kss, config.pool_szs]], 
          np.repeat(len(config.nfs), 3))<jupyter_output><empty_output><jupyter_text>Create the modelThe implementation of the DCAE is done using Keras.<jupyter_code>m = DCAE_torch(c_in=X_train.shape[1], seq_len=config.w, delta=config.delta, 
               pool_szs=config.pool_szs, nfs=config.nfs)
print(m)
foo = torch.rand(1, X_train.shape[1], config.w)
foo.shape, m(foo).shape<jupyter_output>DCAE_torch(
  (downsample): Sequential(
    (0): SameConv1d(
      (conv1d_same): Conv1d(1, 90, kernel_size=(10,), stride=(1,))
    )
    (1): MaxPool1d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (2): Conv1d(90, 60, kernel_size=(5,), stride=(1,), padding=(2,))
    (3): MaxPool1d(kernel_size=3, stride=3, padding=0, dilation=1, ceil_mode=False)
    (4): Conv1d(60, 30, kernel_size=(5,), stride=(1,), padding=(2,))
    (5): MaxPool1d(kernel_size=5, stride=5, padding=0, dilation=1, ceil_mode=False)
  )
  (bottleneck): Sequential(
    (flatten): Flatten(start_dim=1, end_dim=-1)
    (latent_in): Linear(in_features=210, out_features=60, bias=True)
    (latent_out): Linear(in_features=60, out_features=210, bias=True)
    (reshape): Reshape(bs, 30, 7)
  )
  (upsample): Sequential(
    (0): Conv1d(30, 30, kernel_size=(5,), stride=(1,), padding=(2,))
    (1): Upsample(scale_factor=5.0, mode=nearest)
    (2): Conv1d(30, 60, kernel_size=(5,), stride=(1,), padding=(2,))
    ([...]<jupyter_text>Train the model with fastai Learner class, to abstract from Pytorch's training loop<jupyter_code>if config.valid_artifact:
    X, y, splits  = combine_split_data(xs=[X_train, X_valid], ys=[X_train, X_valid])
else:
    X = X_train
    y = X_train
    splits = get_splits(np.arange(len(X)), valid_size=config.valid_size)
splits
tfms = [ToFloat(), ToFloat()]
batch_tfms = [TSStandardize(by_sample=True)]
dls = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms)
dls.dataset[0][0].show(), ToTSTensor()(dls.dataset[0][1]).show()
m = DCAE_torch(c_in=X_train.shape[1], seq_len=config.w, delta=config.delta, 
               pool_szs=config.pool_szs, nfs=config.nfs)
learn = Learner(dls=dls, model=m, loss_func=nn.MSELoss(), opt_func=Adam, 
                cbs=[WandbCallback(log_preds=False)])
lr_valley, lr_steep = learn.lr_find(suggest_funcs=[valley, steep])
learn.fit_one_cycle(config.epochs, lr_max=lr_valley)
learn.plot_metrics()<jupyter_output><empty_output><jupyter_text>To track the performance of this model fit, go to the project dashboard in Weights & Biases. The link is provided at the beginning of this notebook, after the execution of the function `wandb.init()''  Finally, log the learner to be used by the next notebook in the pipeline<jupyter_code>aux_learn = learn.export_and_get()
if config.use_wandb: 
    ar = ReferenceArtifact(aux_learn, f'dcae', type='learner', metadata=dict(run.config))
    run.log_artifact(ar, aliases=f'run-{run.project}-{run.id}')<jupyter_output><empty_output><jupyter_text>Baseline models (To rewrite)Calculate baseline models taking into account that the best prediction is the average and median value of each of the windows  Validate the autoencoderLet's validate the autoencoder quality visually:  Get the best and the worst k predictions using the autoencoder<jupyter_code>interp = Interpretation.from_learner(learn)
top_losses = interp.top_losses(3)
top_losses
for i in top_losses.indices: dls.dataset[i][0].show()
run.finish()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10079754011722879, 'fraction_numerical': 0.017584318247333527, 'mean_word_length': 4.613807982740021, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 33, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '584580', 'license': 'Apache-2.0', 'n_tokens_mistral': 3439, 'n_tokens_neox': 3268, 'n_words': 1156, 'path': 'nbs_pipeline/02a_encoder_DCAE.ipynb', 'repo_name': 'vrodriguezf/deepvats', 'size': '10407'}