RealiT. We follow the same architectural design for RealiT. As an encoding model, we employ a 6-layer Transformer encoder [1]}, a hidden size of 512, an intermediate size of 2048 and 8 attention heads. During training, we use a dropout regularization of 0.1. For encoding the positions of tokens, we employ relative position encoding [2]} as we found that this performed better. A comparison of Transformer with and without relative position encoding can be found in Table REF  ("RealiT - without fine-tuning on real bug fixes" vs "Transformer").
