Inspired by the success of delta rule (Munkhdalai et al., 2019; Schlag et al., 2020, 2021), we have also incorporated it into our Infini-attention. The delta rule attempts a slightly improved memory update by first retrieving existing value entries and subtracting them from the new values before applying the associative bindings as new update.