Simply knowing how feedback works is not enough – we still need to devise what information should be fed back to give a model its best shot at rectifying itself. For this, we resort to distilling knowledge from textual error distributions collected from state-of-the-art text recognition models – this is akin to humans who use prior experiences to help them to make corrections. For example, `\(hello\) ' might be wrongly predicted as `\(nello\) ' or `\(bello\) ' due to partial structural similarity of `\(h\) ' with `\(n\) ' or `\(b\) '. By distilling such error distributions into the feedback module (during training only), the model will gain knowledge of correct character associations. Our second contribution is therefore designing the feedback module via a conditional variational auto-encoder (CVAE) [1]} that learns from such error distributions. More specifically, we augment the vanilla CVAE with an auxiliary decoder that tries to directly reconstruct the correct word, given any incorrect prediction at each iteration. Note that deterministic alternatives such as typical feedback networks [2]}, [3]} or spelling correction (prediction refining) networks, [4]}, [5]} would not work well since they do not model the uncertainty among multiple erroneous alternatives, dictating a variational formulation like ours.
