Recently, the hybrid convolutional neural network hidden Markov model (CNN-HMM) has been introduced for offline handwritten Chinese text recognition (HCTR) and has achieved state-of-the-art performance. In a CNN-HMM system, a handwritten text line is modeled by a series of cascading HMMs, each representing one character, and the posterior distributions of HMM states are calculated by CNN. However, modeling each of the large vocabulary of Chinese characters with a uniform and fixed number of hidden states requires high memory and computational costs and makes the tens of thousands of HMM state classes confusing. Another key issue of CNN-HMM for HCTR is the diversified writing style, which leads to model strain and a significant performance decline for specific writers. To address these issues, we propose a writer-aware CNN based on parsimonious HMM (WCNN-PHMM). Validated on the ICDAR 2013 competition of CASIA-HWDB database, the more compact WCNN-PHMM of a 7360-class vocabulary can achieve a relative character error rate (CER) reduction of 16.6 over the conventional CNN-HMM without considering language modeling. Moreover, the state-tying results of PHMM explicitly show the information sharing among similar characters and the confusion reduction of tied state classes. Finally, we visualize the learned writer codes and demonstrate the strong relationship with the writing styles of different writers. To the best of our knowledge, WCNN-PHMM yields the best results on the ICDAR 2013 competition set, demonstrating its power when enlarging the size of the character vocabulary.
This study is comprehensively extended from our previous conference papers @cite_46 @cite_40 with the following new contributions: 1) the proposed PHMM is introduced with more technical details and verified for a more promising CNN-HMM, rather than the DNN-HMM in @cite_46 ; 2) we present a novel unsupervised adaptation strategy with writer codes and adaptation layers to guide the convolutional layers in CNN-HMM, rather than using the fully connected layers in DNN-HMM @cite_40 ; 3) WCNN-PHMM perfectly combines the two techniques to yield a compact and high-performance model; and 4) all experiments are redesigned to verify the effectiveness of WCNN-PHMM, and detailed analyses are described to give the readers a deep understanding of our approach.
@cite_40 @cite_46
2573871018 2887688727
Recently, we propose deep neural network based hidden Markov models (DNN-HMMs) for offline handwritten Chinese text recognition. In this study, we design a novel writer code based adaptation on top of the DNN-HMM to further improve the accuracy via a customized recognizer. The writer adaptation is implemented by incorporating the new layers with the original input or hidden layers of the writer-independent DNN. These new layers are driven by the so-called writer code, which guides and adapts the DNN-based recognizer with the writer information. In the training stage, the writer-aware layers are jointly learned with the conventional DNN layers in an alternative manner. In the recognition stage, with the initial recognition results from the first-pass decoding with the writer-independent DNN, an unsupervised adaptation is performed to generate the writer code via the cross-entropy criterion for the subsequent second-pass decoding. The experiments on the most challenging task of ICDAR 2013 Chinese handwriting competition show that our proposed adaptation approach can achieve consistent and significant improvements of recognition accuracy over a highperformance writer-independent DNN-HMM based recognizer across all 60 writers, yielding a relative character error rate reduction of 23.62 in average. Recently, hidden Markov models (HMMs) have achieved promising results for offline handwritten Chinese text recognition. However, due to the large vocabulary of Chinese characters with each modeled by a uniform and fixed number of hidden states, a high demand of memory and computation is required. In this study, to address this issue, we present parsimonious HMMs via the state tying which can fully utilize the similarities among different Chinese characters. Two-step algorithm with the data-driven question-set is adopted to generate the tied-state pool using the likelihood measure. The proposed parsimonious HMMs with both Gaussian mixture models (GMMs) and deep neural networks (DNNs) as the emission distributions not only lead to a compact model but also improve the recognition accuracy via the data sharing for the tied states and the confusion decreasing among state classes. Tested on ICDAR-2013 competition database, in the best configured case, the new parsimonious DNN-HMM can yield a relative character error rate (CER) reduction of 6.2 , 25 reduction of model size and 60 reduction of decoding time over the conventional DNN-HMM. In the compact setting case of average 1-state HMM, our parsimonious DNN-HMM significantly outperforms the conventional DNN-HMM with a relative CER reduction of 35.5 .