Network Design: Keeping text rectification and recognition networks similar to Shi [1]}, we implement our framework using open sourced codes [2]}, [3]} in PyTorch [4]}. A bi-directional LSTM of hidden size 256 is used in designing \(F_{enc}\)  of our feedback module which accepts a one layer MLP embedded 128-dimensional representation of discrete character sequences. For both posterior \(F_{post}\)  and prior \(F_{prior}\)  networks, we use 2 layer MLPs with tanh non-linearity. The latent variable \(z\)  has a size of 256. The auxiliary decoder \(F_{aux}\)  is a one layer LSTM decoder whose initial hidden state is initialized by applying a FC-layer on the concatenated representation of \(Y_{enc}\)  and sampled \(z\) . The parameter prediction network \(F_{\mathcal {P}}\)  is a convolutional decoder network inspired from [5]}. The first layer is a fully-connected layer implemented through a 1x1 convolution that maps to a tensor of size equal to the last CNN layer of \(\mathbf {T}\)  via a reshaping operation. Thereafter we introduce a sequence of residual decoder blocks which upsamples feature maps to higher spatial dimensions in the reversed order of down-sampling followed by \(\mathbf {T}\) . In other words, if a ResNet encoder block in \(\mathbf {T}\)  halves the height of feature map, the corresponding ResNet decoder block [5]} in \(\mathbf {F}_{\mathcal {P}}\)  will double it. This strategy essentially formulates intermediate feature maps of a decoder ResNet block (\(\mathbf {F}_\mathcal {P}\) ), whose spatial size is similar to the corresponding layer in the encoder ResNet block (\(\mathbf {T}\) ), thus predicting required affine transformation parameters of similar dimensions as well. Since we need to predict both \(\Gamma \)  and \(\Omega \) , we double the number of convolutional filters in the last layer of \(\mathbf {F}_\mathcal {P}\)  and split the output channel-wise to obtain \(\Gamma \)  and \(\Omega \)  individually.
