DNN-based acoustic model BIBREF0
EXTRACTIVE
Unanswerable
NONE
We adopt a DNN-based acoustic model BIBREF0 with 11 hidden layers and the alignment used to train the model is derived from a HMM-GMM model trained with SAT criterion. In a conventional Gaussian Mixture Model - Hidden Markov Model (GMM-HMM) acoustic model, the state emission log-likelihood of the observation feature vector $o_t$ for certain tied state $s_j$ of HMMs at time $t$ is computed as
1910.05603
Introduction </s> In form al Ġspeech Ġis Ġdifferent Ġfrom Ġformal Ġspeech , Ġespecially Ġin ĠVietnamese Ġdue Ġto Ġmany Ġconj un ctive Ġwords Ġin Ġthis Ġlanguage . ĠBuilding Ġan ĠAS R Ġmodel Ġto Ġhandle Ġsuch Ġkind Ġof Ġspeech Ġis Ġparticularly Ġdifficult Ġdue Ġto Ġthe Ġlack Ġof Ġtraining Ġdata Ġand Ġalso Ġcost Ġfor Ġdata Ġcollection . ĠThere Ġare Ġtwo Ġcomponents Ġof Ġan ĠAS R Ġsystem Ġthat Ġcontribute Ġthe Ġmost Ġto Ġthe Ġaccuracy Ġof Ġit , Ġan Ġacoustic Ġmodel Ġand Ġa Ġlanguage Ġmodel . ĠWhile Ġcollecting Ġdata Ġfor Ġacoustic Ġmodel Ġis Ġtime - consuming Ġand Ġcostly , Ġlanguage Ġmodel Ġdata Ġis Ġmuch Ġeasier Ġto Ġcollect . </s> The Ġlanguage Ġmodel Ġtraining Ġfor Ġthe ĠAutomatic ĠSpeech ĠRecogn ition Ġ( AS R ) Ġsystem Ġusually Ġbased Ġon Ġcorpus Ġcrawled Ġon Ġformal Ġtext , Ġso Ġthat Ġsome Ġconj un ctive Ġwords Ġwhich Ġoften Ġused Ġin Ġconversation Ġwill Ġbe Ġmissed Ġout , Ġleading Ġto Ġthe Ġsystem Ġis Ġgetting Ġbiased Ġto Ġwriting - style Ġspeech . </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġour Ġattempt Ġto Ġmitigate Ġthe Ġproblems Ġusing Ġa Ġlarge Ġscale Ġdata Ġset Ġand Ġa Ġlanguage Ġmodel Ġcombination Ġtechnique Ġthat Ġonly Ġrequire Ġa Ġsmall Ġamount Ġof Ġconversation Ġdata Ġbut Ġcan Ġstill Ġhandle Ġvery Ġwell Ġconversation Ġspeech . </s> System ĠDescription </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour ĠAS R Ġsystem , Ġwhich Ġconsists Ġof Ġ2 Ġmain Ġcomponents , Ġan Ġacoustic Ġmodel Ġwhich Ġmodels Ġthe Ġcorrelation Ġbetween Ġphon em es Ġand Ġspeech Ġsignal ; Ġand Ġa Ġlanguage Ġmodel Ġwhich Ġguides Ġthe Ġsearch Ġalgorithm Ġthroughout Ġinference Ġprocess . </s> System ĠDescription Ġ: :: ĠAc oustic ĠModel </s> We Ġadopt Ġa ĠD NN - based Ġacoustic Ġmodel ĠB IB REF 0 Ġwith Ġ11 Ġhidden Ġlayers Ġand Ġthe Ġalignment Ġused Ġto Ġtrain Ġthe Ġmodel Ġis Ġderived Ġfrom Ġa ĠH MM - G MM Ġmodel Ġtrained Ġwith ĠSAT Ġcriterion . ĠIn Ġa Ġconventional ĠGa ussian ĠM ixture ĠModel Ġ- ĠHidden ĠMark ov ĠModel Ġ( G MM - H MM ) Ġacoustic Ġmodel , Ġthe Ġstate Ġemission Ġlog - like lihood Ġof Ġthe Ġobservation Ġfeature Ġvector Ġ$ o _ t $ Ġfor Ġcertain Ġtied Ġstate Ġ$ s _ j $ Ġof ĠHM Ms Ġat Ġtime Ġ$ t $ Ġis Ġcomputed Ġas </s> where ĠM Ġis Ġthe Ġnumber Ġof ĠGa ussian Ġm ixtures Ġin Ġthe ĠGM M Ġfor Ġstate Ġ$ j $ Ġand Ġ$\ pi Ġ_ { j m } $ Ġis Ġthe Ġmixing Ġweight . ĠAs Ġthe Ġoutputs Ġfrom ĠD NN s Ġrepresent Ġthe Ġstate Ġposter iors Ġ$ p ( s _ j | \ math bf Ġ{ o } _ t ) $, Ġa ĠD NN - H MM Ġhybrid Ġsystem Ġuses Ġpseudo Ġlog - like lihood Ġas Ġthe Ġstate Ġemissions Ġthat Ġis Ġcomputed Ġas </s> where Ġthe Ġstate Ġpri ors Ġlog Ġ$ p ( s _ j Ġ) $ Ġcan Ġbe Ġestimated Ġusing Ġthe Ġstate Ġalign ments Ġon Ġthe Ġtraining Ġspeech Ġdata . </s> System ĠDescription Ġ: :: ĠLanguage ĠModel </s> Our Ġlanguage Ġmodel Ġtraining Ġpipeline Ġis Ġdescribed Ġin ĠFigure ĠFIG REF 6 . ĠFirst , Ġwe Ġcollect Ġand Ġclean Ġlarge Ġamount Ġof Ġtext Ġdata Ġfrom Ġvarious Ġsources Ġincluding Ġnews , Ġmanual Ġlabeled Ġconversation Ġvideo . ĠThen , Ġthe Ġcollected Ġdata Ġis Ġcategorized Ġinto Ġdomains . ĠThis Ġis Ġan Ġimportant Ġstep Ġas Ġthe ĠAS R Ġperformance Ġis Ġhighly Ġdepends Ġon Ġthe Ġspeech Ġdomain . ĠAfter Ġthat , Ġthe Ġtext Ġis Ġfed Ġinto Ġa Ġdata Ġcleaning Ġpipeline Ġto Ġclean Ġbad Ġtone Ġmarks , Ġnormal izing Ġnumbers Ġand Ġdates . </s> For Ġeach Ġdomain Ġtext Ġdata , Ġwe Ġtrain Ġan Ġn - gram Ġlanguage Ġmodel BI B REF 1 Ġthat Ġis Ġoptimized Ġfor Ġthat Ġdomain . ĠAs Ġthe Ġresults , Ġwe Ġhave Ġmore Ġthan Ġ10 Ġlanguage Ġmodels . ĠThese Ġlanguage Ġmodels Ġare Ġcombined Ġbased Ġon Ġperplex ity Ġcalculated Ġon Ġa Ġsmall Ġtext Ġof Ġa Ġdomain Ġthat Ġwe Ġwant Ġto Ġoptimize Ġfor . </s> In Ġour Ġsystem , Ġthe Ġlanguage Ġmodel Ġis Ġused Ġin Ġ2 Ġpass - dec oding . ĠIn Ġthe Ġfirst Ġpass , Ġthe Ġlanguage Ġmodel Ġis Ġcombined Ġwith Ġacoustic Ġand Ġlex icon Ġmodel Ġto Ġform Ġa Ġfull Ġdecoding Ġgraph . ĠIn Ġthis Ġstage , Ġthe Ġlanguage Ġmodel Ġis Ġtypically Ġsmall Ġin Ġsize Ġby Ġutilizing Ġpr uning Ġmethod . ĠIn Ġthe Ġsecond Ġstage , Ġwe Ġuse Ġa Ġun - pr uned Ġlanguage Ġmodel Ġto Ġresc ore Ġdec oded Ġlatt ices . </s> Corp us ĠDescription Ġ: :: ĠSpeech Ġdata </s> Our Ġspeech Ġcorpus Ġconsists Ġof Ġapproximately Ġ3000 Ġhours Ġof Ġspeech Ġdata Ġincluding Ġvarious Ġdomains Ġand Ġspeaking Ġstyles . ĠThe Ġdata Ġis Ġaugmented Ġwith Ġnoise Ġand Ġroom Ġimpulse Ġrespond Ġto Ġincrease Ġthe Ġquantity Ġand Ġprevent Ġover - fitting . </s> Corp us ĠDescription Ġ: :: ĠText Ġdata </s> To Ġtrain Ġn - gram Ġlanguage Ġmodels Ġthat Ġare Ġrobust Ġto Ġvarious Ġdomains Ġand , Ġwe Ġcollect Ġcorpus Ġfrom Ġmany Ġresources , Ġmainly Ġis Ġcome Ġfrom Ġnewspaper Ġsite Ġ( like Ġd ant ri , Ġv nex press , .. ), Ġlaw Ġdocument Ġand Ġsome Ġcrawled Ġrepository . ĠIn Ġtotal , Ġmore Ġthan Ġ50 GB Ġof Ġtext Ġsplit Ġto Ġseparate Ġsubjects Ġwas Ġused Ġto Ġtrain Ġn - gram Ġlanguage Ġmodels . ĠTable ĠT AB REF 10 Ġshows Ġthe Ġstatistic Ġof Ġthe Ġcollected Ġdata . </s> Exper iments </s> There Ġare Ġtwo Ġdifferent Ġtesting Ġsets Ġfrom ĠV L SP Ġ2018 Ġand ĠV L SP Ġ2019 . ĠIn Ġgeneral , Ġthe Ġdata Ġof Ġthis Ġyear Ġis Ġmore Ġcomplex Ġthan Ġthe Ġlast Ġyear Ġone , Ġso Ġthere Ġis Ġa Ġbig Ġgap Ġin Ġresults Ġbetween Ġtwo Ġof Ġthem . ĠThe Ġexperiments Ġare Ġconducted Ġusing Ġthe ĠK aldi Ġspeech Ġrecognition Ġtool kit BI B REF 2 . </s> Exper iments Ġ: :: ĠEvaluation Ġdata </s> Testing Ġdata ĠV L SP Ġ2018 : ĠThis Ġdataset Ġcontains Ġrelatively Ġ5 Ġhours Ġof Ġshort Ġaudio Ġvoices Ġwith Ġ7 96 Ġsamples . </s> Testing Ġdata ĠV L SP Ġ2019 : ĠThe Ġtesting Ġset Ġin Ġ2019 Ġis Ġquite Ġharder Ġcompare Ġwith Ġthe Ġset Ġin Ġ2018 . ĠThere Ġare Ġmore Ġthan Ġ17 Ġhours Ġwith Ġ16 , 214 Ġshort Ġaudio Ġsamples . ĠThe Ġset Ġis Ġdifficult Ġbecause Ġaudio Ġcontains Ġmore Ġnoise Ġand Ġhaving Ġmore Ġinformal Ġspeaking Ġstyle . </s> Exper iments Ġ: :: ĠSingle Ġsystem Ġevaluation </s> Table ĠT AB REF 15 Ġshows Ġthe Ġresults Ġon Ġthe ĠV L SP Ġ2019 Ġtest Ġset Ġwith Ġtwo Ġlanguage Ġmodels . ĠVarious Ġlanguage Ġmodel Ġweights Ġwere Ġalso Ġused Ġto Ġfind Ġthe Ġoptimal Ġvalue Ġfor Ġthe Ġtest Ġset . ĠAs Ġwe Ġcan Ġsee , Ġthe Ġconversation Ġlanguage Ġmodel Ġwith Ġthe Ġweight Ġof Ġ8 Ġyielded Ġthe Ġbest Ġsingle Ġsystem Ġof Ġ15 . 47 % ĠW ER . </s> Exper iments Ġ: :: ĠSystem Ġcombination </s> To Ġfurther Ġimprove Ġthe Ġperformance , Ġwe Ġadopt Ġsystem Ġcombination Ġon Ġthe Ġdecoding Ġlatt ice Ġlevel . ĠBy Ġcombining Ġsystems , Ġwe Ġcan Ġtake Ġadvantage Ġof Ġthe Ġstrength Ġof Ġeach Ġmodel Ġthat Ġis Ġoptimized Ġfor Ġdifferent Ġdomains . ĠThe Ġresults Ġfor Ġ2 Ġtest Ġsets Ġis Ġshowed Ġon ĠTable ĠT AB REF 17 Ġand ĠT AB REF 18 . </s> As Ġwe Ġcan Ġsee , Ġfor Ġboth Ġtest Ġsets , Ġsystem Ġcombination Ġsignificantly Ġreduce Ġthe ĠW ER . ĠThe Ġbest Ġresult Ġfor Ġv l sp 2018 Ġof Ġ4 . 85 % ĠW ER Ġis Ġobtained Ġby Ġthe Ġcombination Ġweights Ġ0 . 6 : 0 . 4 Ġwhere Ġ0 . 6 Ġis Ġgiven Ġto Ġthe Ġgeneral Ġlanguage Ġmodel Ġand Ġ0 . 4 Ġis Ġgiven Ġto Ġthe Ġconversation Ġone . ĠOn Ġthe Ġv l sp 2019 Ġset , Ġthe Ġratio Ġis Ġchange Ġslightly Ġby Ġ0 . 7 : 0 . 3 Ġto Ġdeliver Ġthe Ġbest Ġresult Ġof Ġ15 . 09 %. </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġour ĠAS R Ġsystem Ġparticipated Ġin ĠV L SP Ġ2019 Ġchallenge Ġthat Ġincorporates Ġa Ġlanguage Ġmodel Ġcombination Ġtechnique Ġto Ġhandle Ġconversation Ġspeech Ġwith Ġsmall Ġamount Ġof Ġtext Ġdata Ġrequired . ĠThe Ġmethod Ġdemonstrated Ġthat Ġit Ġcan Ġhelp Ġto Ġreduce ĠW ER Ġby Ġ3 % Ġon Ġthe ĠV L SP Ġ2019 Ġchallenge .
Introduction Informal speech is different from formal speech, especially in Vietnamese due to many conjunctive words in this language. Building an ASR model to handle such kind of speech is particularly difficult due to the lack of training data and also cost for data collection. There are two components of an ASR system that contribute the most to the accuracy of it, an acoustic model and a language model. While collecting data for acoustic model is time-consuming and costly, language model data is much easier to collect. The language model training for the Automatic Speech Recognition (ASR) system usually based on corpus crawled on formal text, so that some conjunctive words which often used in conversation will be missed out, leading to the system is getting biased to writing-style speech. In this paper, we present our attempt to mitigate the problems using a large scale data set and a language model combination technique that only require a small amount of conversation data but can still handle very well conversation speech. System Description In this section, we describe our ASR system, which consists of 2 main components, an acoustic model which models the correlation between phonemes and speech signal; and a language model which guides the search algorithm throughout inference process. System Description ::: Acoustic Model We adopt a DNN-based acoustic model BIBREF0 with 11 hidden layers and the alignment used to train the model is derived from a HMM-GMM model trained with SAT criterion. In a conventional Gaussian Mixture Model - Hidden Markov Model (GMM-HMM) acoustic model, the state emission log-likelihood of the observation feature vector $o_t$ for certain tied state $s_j$ of HMMs at time $t$ is computed as where M is the number of Gaussian mixtures in the GMM for state $j$ and $\pi _{jm}$ is the mixing weight. As the outputs from DNNs represent the state posteriors $p(s_j|\mathbf {o}_t)$, a DNN-HMM hybrid system uses pseudo log-likelihood as the state emissions that is computed as where the state priors log $p(s_j )$ can be estimated using the state alignments on the training speech data. System Description ::: Language Model Our language model training pipeline is described in Figure FIGREF6. First, we collect and clean large amount of text data from various sources including news, manual labeled conversation video. Then, the collected data is categorized into domains. This is an important step as the ASR performance is highly depends on the speech domain. After that, the text is fed into a data cleaning pipeline to clean bad tone marks, normalizing numbers and dates. For each domain text data, we train an n-gram language modelBIBREF1 that is optimized for that domain. As the results, we have more than 10 language models. These language models are combined based on perplexity calculated on a small text of a domain that we want to optimize for. In our system, the language model is used in 2 pass-decoding. In the first pass, the language model is combined with acoustic and lexicon model to form a full decoding graph. In this stage, the language model is typically small in size by utilizing pruning method. In the second stage, we use a un-pruned language model to rescore decoded lattices. Corpus Description ::: Speech data Our speech corpus consists of approximately 3000 hours of speech data including various domains and speaking styles. The data is augmented with noise and room impulse respond to increase the quantity and prevent over-fitting. Corpus Description ::: Text data To train n-gram language models that are robust to various domains and, we collect corpus from many resources, mainly is come from newspaper site (like dantri, vnexpress,..), law document and some crawled repository. In total, more than 50GB of text split to separate subjects was used to train n-gram language models. Table TABREF10 shows the statistic of the collected data. Experiments There are two different testing sets from VLSP 2018 and VLSP 2019. In general, the data of this year is more complex than the last year one, so there is a big gap in results between two of them. The experiments are conducted using the Kaldi speech recognition toolkitBIBREF2. Experiments ::: Evaluation data Testing data VLSP 2018: This dataset contains relatively 5 hours of short audio voices with 796 samples. Testing data VLSP 2019: The testing set in 2019 is quite harder compare with the set in 2018. There are more than 17 hours with 16,214 short audio samples. The set is difficult because audio contains more noise and having more informal speaking style. Experiments ::: Single system evaluation Table TABREF15 shows the results on the VLSP 2019 test set with two language models. Various language model weights were also used to find the optimal value for the test set. As we can see, the conversation language model with the weight of 8 yielded the best single system of 15.47% WER. Experiments ::: System combination To further improve the performance, we adopt system combination on the decoding lattice level. By combining systems, we can take advantage of the strength of each model that is optimized for different domains. The results for 2 test sets is showed on Table TABREF17 and TABREF18. As we can see, for both test sets, system combination significantly reduce the WER. The best result for vlsp2018 of 4.85% WER is obtained by the combination weights 0.6:0.4 where 0.6 is given to the general language model and 0.4 is given to the conversation one. On the vlsp2019 set, the ratio is change slightly by 0.7:0.3 to deliver the best result of 15.09%. Conclusion In this paper, we presented our ASR system participated in VLSP 2019 challenge that incorporates a language model combination technique to handle conversation speech with small amount of text data required. The method demonstrated that it can help to reduce WER by 3% on the VLSP 2019 challenge.
What are the deep learning architectures used in the task?
53f74250948015c394e7b8438a2041fdeb330911
What Ġare Ġthe Ġdeep Ġlearning Ġarchitectures Ġused Ġin Ġthe Ġtask ?
D NN - based Ġacoustic Ġmodel ĠB IB REF 0
<s> What Ġare Ġthe Ġdeep Ġlearning Ġarchitectures Ġused Ġin Ġthe Ġtask ? </s> Introduction </s> In form al Ġspeech Ġis Ġdifferent Ġfrom Ġformal Ġspeech , Ġespecially Ġin ĠVietnamese Ġdue Ġto Ġmany Ġconj un ctive Ġwords Ġin Ġthis Ġlanguage . ĠBuilding Ġan ĠAS R Ġmodel Ġto Ġhandle Ġsuch Ġkind Ġof Ġspeech Ġis Ġparticularly Ġdifficult Ġdue Ġto Ġthe Ġlack Ġof Ġtraining Ġdata Ġand Ġalso Ġcost Ġfor Ġdata Ġcollection . ĠThere Ġare Ġtwo Ġcomponents Ġof Ġan ĠAS R Ġsystem Ġthat Ġcontribute Ġthe Ġmost Ġto Ġthe Ġaccuracy Ġof Ġit , Ġan Ġacoustic Ġmodel Ġand Ġa Ġlanguage Ġmodel . ĠWhile Ġcollecting Ġdata Ġfor Ġacoustic Ġmodel Ġis Ġtime - consuming Ġand Ġcostly , Ġlanguage Ġmodel Ġdata Ġis Ġmuch Ġeasier Ġto Ġcollect . </s> The Ġlanguage Ġmodel Ġtraining Ġfor Ġthe ĠAutomatic ĠSpeech ĠRecogn ition Ġ( AS R ) Ġsystem Ġusually Ġbased Ġon Ġcorpus Ġcrawled Ġon Ġformal Ġtext , Ġso Ġthat Ġsome Ġconj un ctive Ġwords Ġwhich Ġoften Ġused Ġin Ġconversation Ġwill Ġbe Ġmissed Ġout , Ġleading Ġto Ġthe Ġsystem Ġis Ġgetting Ġbiased Ġto Ġwriting - style Ġspeech . </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġour Ġattempt Ġto Ġmitigate Ġthe Ġproblems Ġusing Ġa Ġlarge Ġscale Ġdata Ġset Ġand Ġa Ġlanguage Ġmodel Ġcombination Ġtechnique Ġthat Ġonly Ġrequire Ġa Ġsmall Ġamount Ġof Ġconversation Ġdata Ġbut Ġcan Ġstill Ġhandle Ġvery Ġwell Ġconversation Ġspeech . </s> System ĠDescription </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġour ĠAS R Ġsystem , Ġwhich Ġconsists Ġof Ġ2 Ġmain Ġcomponents , Ġan Ġacoustic Ġmodel Ġwhich Ġmodels Ġthe Ġcorrelation Ġbetween Ġphon em es Ġand Ġspeech Ġsignal ; Ġand Ġa Ġlanguage Ġmodel Ġwhich Ġguides Ġthe Ġsearch Ġalgorithm Ġthroughout Ġinference Ġprocess . </s> System ĠDescription Ġ: :: ĠAc oustic ĠModel </s> We Ġadopt Ġa ĠD NN - based Ġacoustic Ġmodel ĠB IB REF 0 Ġwith Ġ11 Ġhidden Ġlayers Ġand Ġthe Ġalignment Ġused Ġto Ġtrain Ġthe Ġmodel Ġis Ġderived Ġfrom Ġa ĠH MM - G MM Ġmodel Ġtrained Ġwith ĠSAT Ġcriterion . ĠIn Ġa Ġconventional ĠGa ussian ĠM ixture ĠModel Ġ- ĠHidden ĠMark ov ĠModel Ġ( G MM - H MM ) Ġacoustic Ġmodel , Ġthe Ġstate Ġemission Ġlog - like lihood Ġof Ġthe Ġobservation Ġfeature Ġvector Ġ$ o _ t $ Ġfor Ġcertain Ġtied Ġstate Ġ$ s _ j $ Ġof ĠHM Ms Ġat Ġtime Ġ$ t $ Ġis Ġcomputed Ġas </s> where ĠM Ġis Ġthe Ġnumber Ġof ĠGa ussian Ġm ixtures Ġin Ġthe ĠGM M Ġfor Ġstate Ġ$ j $ Ġand Ġ$\ pi Ġ_ { j m } $ Ġis Ġthe Ġmixing Ġweight . ĠAs Ġthe Ġoutputs Ġfrom ĠD NN s Ġrepresent Ġthe Ġstate Ġposter iors Ġ$ p ( s _ j | \ math bf Ġ{ o } _ t ) $, Ġa ĠD NN - H MM Ġhybrid Ġsystem Ġuses Ġpseudo Ġlog - like lihood Ġas Ġthe Ġstate Ġemissions Ġthat Ġis Ġcomputed Ġas </s> where Ġthe Ġstate Ġpri ors Ġlog Ġ$ p ( s _ j Ġ) $ Ġcan Ġbe Ġestimated Ġusing Ġthe Ġstate Ġalign ments Ġon Ġthe Ġtraining Ġspeech Ġdata . </s> System ĠDescription Ġ: :: ĠLanguage ĠModel </s> Our Ġlanguage Ġmodel Ġtraining Ġpipeline Ġis Ġdescribed Ġin ĠFigure ĠFIG REF 6 . ĠFirst , Ġwe Ġcollect Ġand Ġclean Ġlarge Ġamount Ġof Ġtext Ġdata Ġfrom Ġvarious Ġsources Ġincluding Ġnews , Ġmanual Ġlabeled Ġconversation Ġvideo . ĠThen , Ġthe Ġcollected Ġdata Ġis Ġcategorized Ġinto Ġdomains . ĠThis Ġis Ġan Ġimportant Ġstep Ġas Ġthe ĠAS R Ġperformance Ġis Ġhighly Ġdepends Ġon Ġthe Ġspeech Ġdomain . ĠAfter Ġthat , Ġthe Ġtext Ġis Ġfed Ġinto Ġa Ġdata Ġcleaning Ġpipeline Ġto Ġclean Ġbad Ġtone Ġmarks , Ġnormal izing Ġnumbers Ġand Ġdates . </s> For Ġeach Ġdomain Ġtext Ġdata , Ġwe Ġtrain Ġan Ġn - gram Ġlanguage Ġmodel BI B REF 1 Ġthat Ġis Ġoptimized Ġfor Ġthat Ġdomain . ĠAs Ġthe Ġresults , Ġwe Ġhave Ġmore Ġthan Ġ10 Ġlanguage Ġmodels . ĠThese Ġlanguage Ġmodels Ġare Ġcombined Ġbased Ġon Ġperplex ity Ġcalculated Ġon Ġa Ġsmall Ġtext Ġof Ġa Ġdomain Ġthat Ġwe Ġwant Ġto Ġoptimize Ġfor . </s> In Ġour Ġsystem , Ġthe Ġlanguage Ġmodel Ġis Ġused Ġin Ġ2 Ġpass - dec oding . ĠIn Ġthe Ġfirst Ġpass , Ġthe Ġlanguage Ġmodel Ġis Ġcombined Ġwith Ġacoustic Ġand Ġlex icon Ġmodel Ġto Ġform Ġa Ġfull Ġdecoding Ġgraph . ĠIn Ġthis Ġstage , Ġthe Ġlanguage Ġmodel Ġis Ġtypically Ġsmall Ġin Ġsize Ġby Ġutilizing Ġpr uning Ġmethod . ĠIn Ġthe Ġsecond Ġstage , Ġwe Ġuse Ġa Ġun - pr uned Ġlanguage Ġmodel Ġto Ġresc ore Ġdec oded Ġlatt ices . </s> Corp us ĠDescription Ġ: :: ĠSpeech Ġdata </s> Our Ġspeech Ġcorpus Ġconsists Ġof Ġapproximately Ġ3000 Ġhours Ġof Ġspeech Ġdata Ġincluding Ġvarious Ġdomains Ġand Ġspeaking Ġstyles . ĠThe Ġdata Ġis Ġaugmented Ġwith Ġnoise Ġand Ġroom Ġimpulse Ġrespond Ġto Ġincrease Ġthe Ġquantity Ġand Ġprevent Ġover - fitting . </s> Corp us ĠDescription Ġ: :: ĠText Ġdata </s> To Ġtrain Ġn - gram Ġlanguage Ġmodels Ġthat Ġare Ġrobust Ġto Ġvarious Ġdomains Ġand , Ġwe Ġcollect Ġcorpus Ġfrom Ġmany Ġresources , Ġmainly Ġis Ġcome Ġfrom Ġnewspaper Ġsite Ġ( like Ġd ant ri , Ġv nex press , .. ), Ġlaw Ġdocument Ġand Ġsome Ġcrawled Ġrepository . ĠIn Ġtotal , Ġmore Ġthan Ġ50 GB Ġof Ġtext Ġsplit Ġto Ġseparate Ġsubjects Ġwas Ġused Ġto Ġtrain Ġn - gram Ġlanguage Ġmodels . ĠTable ĠT AB REF 10 Ġshows Ġthe Ġstatistic Ġof Ġthe Ġcollected Ġdata . </s> Exper iments </s> There Ġare Ġtwo Ġdifferent Ġtesting Ġsets Ġfrom ĠV L SP Ġ2018 Ġand ĠV L SP Ġ2019 . ĠIn Ġgeneral , Ġthe Ġdata Ġof Ġthis Ġyear Ġis Ġmore Ġcomplex Ġthan Ġthe Ġlast Ġyear Ġone , Ġso Ġthere Ġis Ġa Ġbig Ġgap Ġin Ġresults Ġbetween Ġtwo Ġof Ġthem . ĠThe Ġexperiments Ġare Ġconducted Ġusing Ġthe ĠK aldi Ġspeech Ġrecognition Ġtool kit BI B REF 2 . </s> Exper iments Ġ: :: ĠEvaluation Ġdata </s> Testing Ġdata ĠV L SP Ġ2018 : ĠThis Ġdataset Ġcontains Ġrelatively Ġ5 Ġhours Ġof Ġshort Ġaudio Ġvoices Ġwith Ġ7 96 Ġsamples . </s> Testing Ġdata ĠV L SP Ġ2019 : ĠThe Ġtesting Ġset Ġin Ġ2019 Ġis Ġquite Ġharder Ġcompare Ġwith Ġthe Ġset Ġin Ġ2018 . ĠThere Ġare Ġmore Ġthan Ġ17 Ġhours Ġwith Ġ16 , 214 Ġshort Ġaudio Ġsamples . ĠThe Ġset Ġis Ġdifficult Ġbecause Ġaudio Ġcontains Ġmore Ġnoise Ġand Ġhaving Ġmore Ġinformal Ġspeaking Ġstyle . </s> Exper iments Ġ: :: ĠSingle Ġsystem Ġevaluation </s> Table ĠT AB REF 15 Ġshows Ġthe Ġresults Ġon Ġthe ĠV L SP Ġ2019 Ġtest Ġset Ġwith Ġtwo Ġlanguage Ġmodels . ĠVarious Ġlanguage Ġmodel Ġweights Ġwere Ġalso Ġused Ġto Ġfind Ġthe Ġoptimal Ġvalue Ġfor Ġthe Ġtest Ġset . ĠAs Ġwe Ġcan Ġsee , Ġthe Ġconversation Ġlanguage Ġmodel Ġwith Ġthe Ġweight Ġof Ġ8 Ġyielded Ġthe Ġbest Ġsingle Ġsystem Ġof Ġ15 . 47 % ĠW ER . </s> Exper iments Ġ: :: ĠSystem Ġcombination </s> To Ġfurther Ġimprove Ġthe Ġperformance , Ġwe Ġadopt Ġsystem Ġcombination Ġon Ġthe Ġdecoding Ġlatt ice Ġlevel . ĠBy Ġcombining Ġsystems , Ġwe Ġcan Ġtake Ġadvantage Ġof Ġthe Ġstrength Ġof Ġeach Ġmodel Ġthat Ġis Ġoptimized Ġfor Ġdifferent Ġdomains . ĠThe Ġresults Ġfor Ġ2 Ġtest Ġsets Ġis Ġshowed Ġon ĠTable ĠT AB REF 17 Ġand ĠT AB REF 18 . </s> As Ġwe Ġcan Ġsee , Ġfor Ġboth Ġtest Ġsets , Ġsystem Ġcombination Ġsignificantly Ġreduce Ġthe ĠW ER . ĠThe Ġbest Ġresult Ġfor Ġv l sp 2018 Ġof Ġ4 . 85 % ĠW ER Ġis Ġobtained Ġby Ġthe Ġcombination Ġweights Ġ0 . 6 : 0 . 4 Ġwhere Ġ0 . 6 Ġis Ġgiven Ġto Ġthe Ġgeneral Ġlanguage Ġmodel Ġand Ġ0 . 4 Ġis Ġgiven Ġto Ġthe Ġconversation Ġone . ĠOn Ġthe Ġv l sp 2019 Ġset , Ġthe Ġratio Ġis Ġchange Ġslightly Ġby Ġ0 . 7 : 0 . 3 Ġto Ġdeliver Ġthe Ġbest Ġresult Ġof Ġ15 . 09 %. </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġour ĠAS R Ġsystem Ġparticipated Ġin ĠV L SP Ġ2019 Ġchallenge Ġthat Ġincorporates Ġa Ġlanguage Ġmodel Ġcombination Ġtechnique Ġto Ġhandle Ġconversation Ġspeech Ġwith Ġsmall Ġamount Ġof Ġtext Ġdata Ġrequired . ĠThe Ġmethod Ġdemonstrated Ġthat Ġit Ġcan Ġhelp Ġto Ġreduce ĠW ER Ġby Ġ3 % Ġon Ġthe ĠV L SP Ġ2019 Ġchallenge .