Clinical data such as Intensive Care Unit (ICU) measurements comprise of multi-variate, time-series observations coming from sensor measurements, test results, and subjective assessments. Song et al. introduced Transformers to predict various variables of clinical interest from the MIMIC-III benchmark dataset [1]}, named Simply Attend and Diagnose or SAnD [2]}. Preprocessed data is passed to an input embedding layer, which uses 1-D convolution for mapping inputs into \(d\)  dimensional vectors (\(d\)  \(>\)  number of variables in the multi-variate time-series data). After the addition of hard-coded positional encoding to the embedded data, it is passed through the attention module. The multi-head attention layer uses restricted self-attention to introduce causality, i.e., perform computations using information earlier than the current time. The output of the stacked attention modules passes on to the `dense interpolation layer'. This layer, together with positional encoding, is used to capture the temporal structure of clinical data. A linear layer and softmax/sigmoid are used as the final layers for classification. The proposed Transformer model was evaluated on all MIMIC-III benchmark tasks and was reported to perform better than RNNs.
