Our first contribution is therefore an iterative framework, where characters predicted in the previous iteration provide clues through a feedback loop [1]} to enhance performance in the subsequent iterations. This is fundamentally different to current state-of-the-arts [2]}, [3]}, [4]}, most of which adapt a feed-forward framework consisting of three-components (feature-extracting backbone, bidirectional-RNN encoder, and attention-based recurrent decoder).
Despite the attention mechanism, its single-pass nature still dictates wrong predictions, thereby leaving no chance for the model to recover.
To this end, our iterative design enables the revision of incorrect intermediate predictions in its subsequent steps, via a novel cross-modal (i.e., text prediction to image feature-maps) feedback mechanism.
The key to our success lies with how feedback is progressed at each iteration.
A naive solution might be to apply an independent spelling correction network [5]}, [6]} chained serially to a basic text recognition model. Apart from not being end-to-end trainable, this also ignores the intermediate visual features from the recognition network, ultimately bisecting the feedback loop.
We on the other hand advocate that earlier word predictions (text labels) should be fed back cross-modal to the main text recognition network and directly modulate the visual feature maps at the next iteration. That is, the feedback module triggers a mapping from the discrete predicated label space, to a continuous space of affine transformation parameters (akin to [7]}) which are consequently used to condition visual features (hence closing the feedback loop).
