Building competitive direct acoustics-to-word models for English conversational speech recognition

Abstract: Direct acoustics-to-word (A2W) models in the end-to-end paradigm have received increasing attention compared to conventional sub-word based automatic speech recognition models using phones, characters, or context-dependent hidden Markov model states. This is because A2W models recognize words from speech without any decoder, pronunciation lexicon, or externally-trained language model, making training and decoding with such models simple. Prior work has shown that A2W models require orders of magnitude more training data in order to perform comparably to conventional models. Our work also showed this accuracy gap when using the English Switchboard-Fisher data set. This paper describes a recipe to train an A2W model that closes this gap and is at-par with state-of-the-art sub-word based models. We achieve a word error rate of 8.8%/13.9% on the Hub5-2000 Switchboard/CallHome test sets without any decoder or language model. We find that model initialization, training data order, and regularization have the most impact on the A2W model performance. Next, we present a joint word-character A2W model that learns to first spell the word and then recognize it. This model provides a rich output to the user instead of simple word hypotheses, making it especially useful in the case of words unseen or rarely-seen during training.

\section{Introduction}
\label{sec:intro}
Conventional sub-word based automatic speech recognition (ASR) typically involves three models - acoustic model (AM), pronunciation model and decision tree (PMT), and
language model (LM)~\cite{jelinek1997statistical}. The AM computes the probability $p_\text{AM}(\mathbf{A}|\mathbf{s})$ of the acoustics
$\mathbf{A}$ given the sub-word units $\mathbf{s}$. The PMT models the probability $p_\text{PMT}(\mathbf{w}|\mathbf{s})$ of the word sequence
$\mathbf{w}$ given the sub-word unit sequence $\mathbf{s}$. The LM acts as a prior $p_\text{LM}(\mathbf{w})$ on the
word sequence $\mathbf{w}$. Hence, finding the most likely sequence of words given the acoustics $\mathbf{A}$ becomes
a maximum aposteriori optimization problem over the following probability density function:
\begin{align}
     p_\text{ASR}(\mathbf{w}|\mathbf{A}) &\propto p_\text{LM}(\mathbf{w}) p(\mathbf{A}|\mathbf{w}) \\
                 &= p_\text{LM}(\mathbf{w}) \sum_{\mathbf{s}} p_\text{AM}(\mathbf{A}|\mathbf{s},\mathbf{w}) p_\text{PMT}(\mathbf{s}|\mathbf{w}) \\
                 &\approx p_\text{LM}(\mathbf{w}) \sum_{\mathbf{s}} p_\text{AM}(\mathbf{A}|\mathbf{s}) p_\text{PMT}(\mathbf{s}|\mathbf{w}) \;.
\end{align}
While both the AM and LM in modern ASR systems use deep neural networks (DNNs) and their variants~\cite{hinton2012deep}, the PMT is
usually based on decision trees and finite state transducers.
Training the AM requires alignments between the acoustics and sub-word units, and several
iterations of model training and re-alignment. Recent end-to-end (E2E) models have obviated the need for
aligning the sub-word units to the acoustics. Popular E2E models include recurrent neural networks (RNNs)
trained with the connectionist temporal classification (CTC) loss function~\cite{graves2006connectionist, graves2014towards, zweig2016advances, sak2015fast, miao2015eesen, miao2016empirical, hannun2014deep, amodei2016deep} and the attention-based encoder-decoder RNNs~\cite{bahdanau2016end, bahdanau2014neural, maas2015lexicon, chan2016listen, chan2016latent}.These approaches are not truly E2E because they still use sub-word units, and hence require a decoder and separately-trained
LM to perform well.

In contrast, the recently-proposed direct acoustics-to-word (A2W) models~\cite{soltau2016neural,audhkhasi2017direct} train a single RNN to directly optimize $p_\text{ASR}(\mathbf{w}|\mathbf{A})$.
This eliminates the need for sub-word units, pronunciation model, decision tree, decoder, and externally-trained LM, which significantly simplifies
the training and decoding process. However, prior research on A2W models has shown
that such models require several orders of magnitude more training data when compared with conventional
sub-word based models. This is because the A2W
models need sufficient acoustic training examples per word to train well.
For example,~\cite{soltau2016neural} used more than 125,000 hours of speech to train an A2W model with a vocabulary of nearly 100,000 words that matched the performance of a state-of-the-art CD state-based CTC model. Our prior work~\cite{audhkhasi2017direct} explored A2W models for the well-known English Switchboard task and presented
a few initialization techniques to effectively train such models with only 2000 hours of data. However, we still observed a gap of around 3-4\% absolute in WER between the Switchboard phone CTC and the A2W models on the Hub5-2000 evaluation set.

This paper further improves the state-of-the-art in A2W models for English conversational speech recognition. We
present a training recipe that achieves WER of 8.8\%/13.9\% on the Switchboard/CallHome subsets of the Hub5-2000 evaluation set, compared to our previous best result of 13.0\%/18.8\%~\cite{audhkhasi2017direct}. These new results are at par with several state-of-the-art models that use
sub-word units, a decoder, and a LM. We quantify the gains made by each ingredient of our training recipe and conclude that model initialization, training data order, and regularization are the most important factors.

Next, we turn our attention to the issue of data sparsity while training A2W models. The conventional solution to this problem uses a sub-word unit-based model that
needs a decoder and LM during testing. As an alternative, we propose the spell and recognize (SAR) CTC model that
learns to first spell the word into its character sequence and then recognize it. Not only does this model retain
all advantages of a direct A2W model, it also provides rich hypotheses to the user which are readable
especially in the case of unseen or rarely-seen words. We illustrate the benefits of this model for out-of-vocabulary (OOV) words.

The next section discusses the baseline A2W model~\cite{audhkhasi2017direct}. Section~\ref{sec:recipe} discusses the proposed training recipe, an analysis of the impact of individual ingredients, and the results. Section~\ref{sec:sar}
presents our SAR model that jointly models words and characters. The paper concludes in Section~\ref{sec:concl} with a summary of findings.

\section{Baseline Acoustics-to-Word Model}\label{sec:baseline}

\subsection{CTC Loss}\label{sec:ctc_loss}
Conventional losses used for training neural networks, such as cross-entropy, require a one-to-one mapping (or alignment)
between the rows of the $T\times D$ input feature vector matrix $\mathbf{A}$ and length-$L$ output label sequence $\mathbf{y}$. The connectionist temporal
classification (CTC) loss relaxes this requirement by considering all possible alignments. It introduces a special \emph{blank symbol} $\phi$ that expands the length-$L$ target label sequence $\mathbf{y}$
to multiple length-$T$ label sequences $\tilde{\mathbf{y}}$ containing $\phi$, such that $\tilde{\mathbf{y}}$ maps
to $\mathbf{y}$ after removal of all repeating symbols and $\phi$. The CTC loss is then
\begin{align}
	p(\mathbf{y}|\mathbf{A}) &= \sum_{\tilde{\mathbf{y}}} p(\tilde{\mathbf{y}}|\mathbf{A}) = \sum_{\tilde{y}\in\Omega(\mathbf{y})} \prod_{t=1}^T p(\tilde{y}_t|\mathbf{a}_t) \;,
\end{align}
where the set $\Omega(\mathbf{y})$, is the set of CTC paths for $\mathbf{y}$, and $\mathbf{a}_t$/$y_t$ denote the $t^\text{th}$ elements of the sequences.
A forward-backward algorithm efficiently computes the above loss function and its
gradient, which is then back-propagated through the neural network~\cite{graves2006connectionist}. The next section describes the baseline A2W model~\cite{audhkhasi2017direct}.

\subsection{Baseline A2W Model}\label{sec:base_a2w}
We used two standard training data sets for our experiments. The ``300-hour'' set contained 262 hours of segmented speech
from the Switchboard-1 audio with transcripts provided by Mississippi State University. The ``2000-hour'' set contained
an additional 1698 hours from the Fisher data collection and 15 hours from CallHome audio.

We extracted 40-dimensional logMel filterbank features over 25 ms frames every 10 ms from the input speech signal.
We used stacking+decimation~\cite{sak2015fast}, where we stacked two successive frames and dropped every alternate frame
during training. This resulted in a stream of 80-dimensional acoustic feature vectors at half the frame rate of the original stream. The baseline models also used 100-dimensional i-vectors~\cite{saon2017english} for each speaker, resulting in 180-dimensional
acoustic feature vectors.

The baseline A2W model consisted of a 5-layer bidirectional LSTM (BLSTM) RNN with a 180-dimensional input and 320-dimensional hidden layers in the forward and backward directions. We picked words with at least 5 occurrences in the
training data in the vocabulary. This resulted in a 10,000-word output layer for the 300-hour A2W system and a 25,000-output layer for the 2000-hour system. As noted in~\cite{audhkhasi2017direct}, initialization is crucial to training
an A2W model. Thus, we initialized the A2W BLSTM with the BLSTM from a trained phone CTC model, and the final linear
layer using word embeddings trained using GloVe~\cite{pennington2014glove}.

Table~\ref{tab:base_results} gives the WERs of the baseline A2W and phone CTC models reported in~\cite{audhkhasi2017direct} on the Hub5-2000 Switchboard (SWB) and
CallHome (CH) test sets. We performed the decoding of the A2W models via simple peak-picking over the output word posterior distribution, and removing repetitions and blank symbols. The phone CTC model used a full decoding graph and a LM. We observe that the 2000-hour A2W model lags behind the phone CTC model by 3.4\%/2.8\% absolute WER on SWB/CH, and the gap is much bigger for the 300-hour models. We next discuss our new training recipe.

\begin{table}[ht]
\caption{This table shows the Switchboard/CallHome WERs for our baseline 300/2000-hour A2W and phone CTC models from~\cite{audhkhasi2017direct}.}
\begin{center}
\begin{tabular}{|c|c|c|c|c|} \hline
\emph{Data (hrs)} &\emph{AM} & \emph{LM} & \emph{SWB} & \emph{CH} \\ \hline\hline
300 & A2W & - & 20.8 & 30.4 \\ \hline
300 & Phone CTC & Small & 14.5 & 25.1 \\ \hline\hline
2000 & A2W & - & 13.0 & 18.8 \\ \hline
2000 & Phone CTC & Big &  9.6 & 16.0 \\ \hline
\end{tabular}
\end{center}
\label{tab:base_results}
\end{table}
\vspace{-20pt}

\section{Updated Training Recipe}\label{sec:recipe}
Our prior experience with training A2W models led us to conclude that model initialization and regularization
are important aspects of training such models. One key reason for this is the fact that A2W models attempt
to solve the difficult problem of directly recognizing words from acoustics with a single neural network.
Hence, our previously-proposed strategy of initializing the A2W BLSTM with the phone CTC BLSTM and the final
linear layer with word embeddings gave WER gains. In this work, we started by exploring several
other strategies in the same spirit. All our new experiments were conducted in PyTorch~\cite{pytorch} with the following changes compared to~\cite{audhkhasi2017direct}:
\begin{itemize}
	\item We included delta and delta-delta coefficients because they slightly improved the WER. Hence, the total acoustic vector was
of size 340 after stacking+decimation and appending the 100-dimensional i-vectors.
	\item We initialized all matrices to samples of a uniform distribution over $(-\epsilon,\epsilon)$
	where $\epsilon$ is the inverse of the $\sqrt{\text{fan-in}}$ or the size of the input vector~\cite{glorot2010understanding}. This takes the dimensionality of the input vectors at each layer in account, which improved convergence compared to our old
	strategy of using $\epsilon = 0.01$.
	\item In place of new-bob annealing, we kept a fixed learning rate for the first 10 epochs and decayed it by $\sqrt{0.5}$ every epoch.
\end{itemize}

\subsection{Training Data Order}\label{sec:data_order}
Training data order is an important consideration for sequence-to-sequence models such as E2E ASR systems
because such models operate on the entire input and output sequences. All training sequences have to be padded to the length of the longest sequence in the batch in order to do GPU tensor operations.
Random sequence order during batch creation is not memory-efficient because batches will contain a larger range of
sequence lengths, which will lead to more wasteful padding on average. Hence, sequences have to be sorted
before batch creation.

We compare the impact of sorting input acoustic sequences in order of ascending and descending length in
Table~\ref{tab:recipe_increment}. Our results show that ascending order gives significantly
better WER than sorting in descending order. The intuition behind this result is that
shorter sequences are easier to train on initially, which enables the network to reach a better point in the parameter space.  This can be regarded as an instance of curriculum learning~\cite{Bengio2009}.

\begin{table}[ht]
\caption{This table shows the impact of various training strategies on the 300-hour A2W model.}
\begin{center}
\begin{tabular}{|l|c|c|} \hline
\emph{Model} & \emph{SWB} & \emph{CH} \\ \hline\hline
Descending Order & 23.0 & 30.7 \\ \hline
Ascending Order & 18.3 & 28.1 \\ \hline
+Momentum & 18.0 & 27.4 \\ \hline
+Dropout & 17.4 & 26.9 \\ \hline
+Output Projection & 16.9 & 26.3 \\ \hline
+Phone BLSTM Init. & 14.9 & 23.8 \\ \hline
{\bf +Bigger Model} & {\bf 14.6} & {\bf 23.6} \\ \hline\hline
Previous best~\cite{audhkhasi2017direct} & 20.8 & 30.4 \\ \hline
\end{tabular}
\end{center}
\label{tab:recipe_increment}
\end{table}
\vspace{-10pt}

\subsection{Momentum and Dropout}\label{sec:dropout}
We also experimented with Nesterov momentum-based stochastic gradient descent (SGD), which has been shown
to give better convergence compared to simple SGD on several tasks. We use the following parameter updates:
\begin{align}
	\mathbf{v}_n &= \rho \mathbf{v}_{n-1} + \lambda \nabla f(\Theta_{n-1}+\rho v_{n-1}) \\
	\Theta_{n} &= \Theta_{n-1} - \mathbf{v}_n \;,
\end{align}
where $\mathbf{v}_n$ is the \emph{velocity} or a running weighted-sum of the gradient of
the loss function $f$. The constant $\rho$ is usually set to $0.9$ and $\lambda$ is the learning rate, set to $0.01$ in our experiments. We also experimented with a dropout of 0.25 in order to prevent over-fitting. Table~\ref{tab:recipe_increment} shows that both momentum and dropout improve the WER.

\subsection{Output Projection Layer}\label{sec:proj}
In contrast with phone or character-based CTC models, A2W models have a large output size equal to the
size of the vocabulary. Prior research~\cite{sainath2013low} has shown that decomposing the output linear layer
of size $V \times D$ into two layers of sizes $V \times d$ and $d \times D$ with $d < D$ speeds-up
model training due to reduced number of parameters. We experimented with a projection layer of size 256 and found that it speeds-up training by a factor of 1.2x and also slightly improves the WER, which
we attribute to a reduction in over-fitting.

\subsection{Phone BLSTM Initialization and Bigger Model}\label{sec:bigger}
We finally initialized our model with the phone CTC BLSTM which gave improvements in our previous work~\cite{audhkhasi2017direct}. As expected, this initialization lowered the WER, despite the presence of
all the above useful strategies. With dropout in place, we trained a bigger 6-layer model with 512-dimensional
BLSTM and saw slight gains in WER.

\subsection{Final Model and Results}\label{sec:results}
We initialized the 2000-hour A2W model with the best 300-hour A2W model and used the same recipe for training. Table~\ref{tab:2k_wers} shows the WER of the resulting system along with our previous best A2W
WER and several other published results. We obtained a significant improvement of 4.2\%/4.9\% absolute
WER compared to our previous result. We also see that our direct A2W is at par with most hybrid CD
state-based and E2E models, while utilizing no decoder or LM. As noted in~\cite{saon2017english}, the CallHome test set
is more challenging than Switchboard because 36 out of 40 speakers in the latter appear in the
training set. The results on the CallHome
test set are especially good, where our A2W model matches the best result obtained using a
hybrid BLSTM~\cite{saon2017english} that used exactly the same acoustic features\footnotemark.\footnotetext{Adding additional FMLLR
features gives a WER of 7.2\%/12.7\%~\cite{saon2017english}.}

\begin{table}[ht]
\caption{This table shows the WER of our current and previous-best A2W model trained on the 2000-hour Switchboard+Fisher set. We have also included several other published results for comparison. Results with $^*$ use data augmentation on the 2000-hour training set.}
\begin{center}
\begin{tabular}{|l|c|c|c|c|} \hline
\emph{Model} & \emph{Output} & \emph{Decoder/} & \emph{SWB} & \emph{CH} \\
& \emph{Units} & \emph{LM} & & \\ \hline\hline
BLSTM+LF MMI~\cite{povey2016purely} & CD state & Y/Y & 8.5 & 15.3 \\ \hline
LACE+LF MMI~\cite{xiong2016achieving} & CD state & Y/Y & 8.3 & 14.8 \\ \hline
Dilated Conv.~\cite{sercu2016dense} & CD state & Y/Y & 7.7 & 14.5 \\ \hline
BLSTM~\cite{saon2017english} & CD state & Y/Y & 7.7 & 13.9 \\ \hline
Iterated CTC~\cite{xiong2016microsoft} & Char & Y/Y & 11.3 & 18.7 \\ \hline
CTC$^*$~\cite{liu2017gram} & Char & Y/Y & 9.0 & 17.7 \\ \hline
Gram-CTC$^*$~\cite{liu2017gram} & Char N-gm & Y/Y & 7.9 & 15.8 \\ \hline
CTC+Gram-CTC$^*$~\cite{liu2017gram} & Char N-gm & Y/Y & 7.3 & 14.7 \\ \hline
RNN-T$^*$~\cite{battenberg2017exploring} & Char & Y/N & 8.5 & 16.4 \\ \hline
Attention$^*$~\cite{battenberg2017exploring} & Char & Y/N & 8.6 & 17.8 \\ \hline\hline
CTC A2W~\cite{audhkhasi2017direct} & Word & N/N & 13.0 & 18.8 \\ \hline
{\bf CTC A2W (current)} & {\bf Word} & {\bf N/N} & {\bf 8.8} & {\bf 13.9} \\ \hline
\end{tabular}
\end{center}
\label{tab:2k_wers}
\end{table}

We also note that our A2W model uses a
vocabulary of 25,000 words which has an OOV rate of 0.5\%/0.8\% on the SWB/CH test sets. All the other
models used much bigger vocabularies and hence did not suffer from OOV-induced errors.

\subsection{Ablation Study}\label{sec:ablation}
In order to understand the impact of individual components of our recipe, we conducted an ablation study
on our best 300-hour A2W model. We removed each component of the recipe while keeping others fixed, trained
a model, and decoded the test sets. Figure~\ref{fig:ablation_results} shows the results of this experiment. We observed that changing the training data order from ascending to descending order of length
resulted in the biggest drop in performance. The second biggest factor was dropout - excluding it leads to
over-training because the heldout loss rises after epoch 10. Choosing a smaller 5-layer model
instead of 6-layer led to the next largest drop in WER. Finally, as expected, excluding the projection layer
had the least impact on WER.

\begin{figure}[htbp]
\begin{center}
\includegraphics[width=0.4\textwidth]{ablation_results.png}
\caption{This figure shows the results of the ablation experiment on our best 300-hour A2W model. The numbers in parentheses in the legend give the SWB/CH WERs.}
\label{fig:ablation_results}
\end{center}
\end{figure}

Despite strong results, the A2W model does not give any meaningful output to the user in case of OOV words
but simply emits an \emph{``UNK''} tag. This is not a big problem for the Switchboard task because the OOV rates for the SWB/CH test sets are 0.5\%/0.8\% on the 25,000 word vocabulary. But other tasks
might be affected by the limited vocabulary. As a solution, the next section discusses our joint word-character model that aims to provide the user with a richer output that is especially useful for unseen or rarely-seen words.

\vspace{-5pt}
\section{Spell and Recognize Model}\label{sec:sar}
The advantage of the A2W model is that it directly emits word hypotheses by forward-passing acoustic
features through a RNN without needing a decoder or externally-trained LM. However, its vocabulary is fixed and OOV words cannot be recognized
by the system. Furthermore, words infrequently seen in the training data are not recognized well by the
network due to insufficient training examples. Prior approaches to dealing with the above limitations completely rely on sub-word units. This
includes work on character models~\cite{xiong2016microsoft, hannun2014deep} and N-grams~\cite{liu2017gram}, RNN-Transducer~\cite{battenberg2017exploring}, RNN-Aligner~\cite{sak2017recurrent}.

In contrast, our approach is to have the best of both worlds by combining the ease of decoding a A2W model
with the flexibility of recognizing unseen/rarely-seen words with a character-based model. One natural
candidate is a multi-task learning (MTL) model containing a shared lower network, and two output networks
corresponding to the two tasks - recognizing words and characters. However, such an MTL network is not
suitable for our purpose because the recognized word and character sequences for an input speech utterance
are not guaranteed to be synchronized in time. This is because the CTC loss
does not impose time-alignment on the output sequence.

The proposed spell and recognize (SAR) model circumvents this alignment problem by presenting training
examples that contain both words and characters. This allows us to continue to leverage an A2W framework without resorting to more complex graph-based decoding methodologies employed in, for example, word-fragment based systems~\cite{siohan2005fast,bhuvana2009fast,rastrow2009towards}. Consider the output word sequence ``THE CAT IS BLACK''
The SAR model
uses the following target sequence:

{\footnotesize
{\color{red}b-t h e-e} {\color{blue}THE} {\color{red}b-c a e-t} {\color{blue}CAT} {\color{red}b-i e-s} {\color{blue}IS} {\color{red}b-b l a c e-k} {\color{blue}BLACK}}

where lowercase alphabets are the character targets, and b-/e- denote special prefixes for word beginning and end.
Hence, the model is trained to first spell the word and then recognize it. In contrast with a MTL model, the
SAR model has a single softmax over words+characters in the output layer.

\subsection{Choice of Character Set}
We experimented with two character sets for the SAR model. The first one is the simple character set consisting
of a total of 41 symbols - alphabets a-z, digits 0-9, whitespace \_, and other punctuations. The second
character set is the one used in~\cite{zweig2016advances}, and includes separate character variants depending on position
in a word - beginning, middle, and end. It also includes special symbols for repeated characters, e.g.
a separate symbol for \emph{ll}. The intuition behind this character set is that its symbols capture
more context as compared to the simple set, and also disambiguate legitimate character repetitions from
double peaks emitted by the CTC model. We observed that the performance with the latter character set is slightly better than using simple characters. Hence, we present results only for this case.

\subsection{Experiments and Results}
We restricted ourselves to the 300-hour set for experiments on the SAR model because it uses a 10,000 word vocabulary, leading to a higher OOV
rate than the 2000-hour set and also contains several rare words. We trained a 6-layer BLSTM with joint
word and character targets after preparing the output training sequences as described in the previous section.
We initialized the SAR BLSTM using the A2W BLSTM. The training recipe
was the same as for the A2W model presented previously. The SAR model permits three decodes:
\begin{itemize}
	\item {\bf Word}: Use only word predictions, similar to the A2W model.
	\item {\bf Characters}: Use only character predictions, and combine them into words using the word-begin characters.
	\item {\bf Switched}: Use the character predictions up to the previous word when the model predicts an ``UNK'' symbol, and use the word prediction everywhere else.
\end{itemize}

\vspace{-10pt}
\begin{table}[ht]
\caption{This table shows performance of the 300-hour SAR model for three types of decodes.}
\begin{center}
\begin{tabular}{|l|c|c|} \hline
\emph{Decode} & \emph{SWB} & \emph{CH} \\ \hline\hline
Word & 14.5 & 23.9 \\ \hline
Characters & 18.9 & 30.9 \\ \hline
Switched & 14.4 & 24.0 \\ \hline
\end{tabular}
\end{center}
\label{tab:sar_results}
\end{table}
\vspace{-10pt}

We observe that the SAR model gives comparable performance to the baseline A2W model, but additionally gives
meaningful output for OOVs, as illustrated by the following test set examples:

\vspace{5pt}
{\scriptsize \noindent\fbox{%
    \parbox{0.45\textwidth}{%
        \emph{REF:} SUCH AS LIKE (\%HESITATION) THE {\bf MURDERING} OF A COP OR\\
        \emph{HYP: }{\color{red}b-s u c e-h} {\color{blue}SUCH} {\color{red}\_ b-a e-s} {\color{blue}AS} {\color{red}\_ b-l i k e-e} {\color{blue}LIKE} {\color{red}\_ b-t h e-e} {\color{blue}THE} {\color{red}\_ b-u e-u} {\color{blue}UH} {\color{red}\bf\_ b-m u r d e r i n e-g} {\color{blue}\bf UNK} {\color{red}\_ b-o e-f} {\color{blue}OF} {\color{red}\_ b-a} {\color{blue}A} {\color{red}\_ b-s o e-p} {\color{blue}COP} {\color{red}\_ b-o e-r} {\color{blue}OR}\\
        \noindent\rule{8cm}{0.4pt} \\
        \emph{REF:} THAT IS RIGHT WE ARE WE ARE {\bf FURTHERING} HIGHER\\
        \emph{HYP:} {\color{red}b-t h a e-s} {\color{blue}THAT'S} {\color{red}\_ b-r i g h e-t} {\color{blue} RIGHT} {\color{red}\_ b-r h} {\color{blue}I RIGHT} {\color{red}\_ b-w e ' r e-e} {\color{blue}WE'RE} {\color{red}\_ b-w e ' r e-e} {\color{blue}WE'RE} {\color{red}\bf\_ b-f u r t h e r i n e-g} {\bf\color{blue}UNK} {\color{red}\_ b-h i g h e e-r} {\color{blue}HIGHER}\\
        \noindent\rule{8cm}{0.4pt} \\
        \emph{REF:} BUT SOMETIMES LIKE I JUST HAD TO DO THIS SUMMARY OF THIS ONE YOU KNOW THESE {\bf SCHOLARLY JOURNALS} AND STUFF\\
        \emph{HYP:} {\color{red}b-b u e-t} {\color{blue}BUT} {\color{red}\_ b-s o m e t i m e e-s} {\color{blue}SOMETIMES} {\color{red}\_ b-l i k e-e} {\color{blue}LIKE} {\color{red}\_ b-i I b-j u s e-t} {\color{blue}JUST} {\color{red}\_ b-h a e-d} {\color{blue}HAD} {\color{red}\_ b-t e-o} {\color{blue}TO} {\color{red}\_ b-d e-o} {\color{blue}DO} {\color{red}\_ b-t h e-e} {\color{blue}THIS} {\color{red}\_ b-s u mm e r e-y} {\color{blue}SUMMARY} {\color{red}\_ b-t h i e-s} {\color{blue}THIS} {\color{red}\_ b-o n e-e} {\color{blue}ONE} {\color{red}\_ b-y o e-u} {\color{blue}YOU} {\color{red}\_ b-k n o e-w} {\color{blue}KNOW} {\color{red}\_ b-t h e s e-e} {\color{blue}THESE} {\color{red}\bf\_ b-c o l a r l e-y} {\color{blue}\bf UNK} {\color{red}\bf\_ b-j o u r n a l e-s} {\color{blue}\bf UNK} {\color{red}\_ b-a n e-d} {\color{blue}AND} {\color{red}\_ b-s t u e-2f} {\color{blue}STUFF}
    }%
}
}
\vspace{5pt}

The words in bold are OOVs. We observe that the SAR model emits the UNK tag in these cases, but the characters preceding it contain the spelling of the word. In some cases, this spelling is incorrect, e.g. "SCHOLARLY $\rightarrow$ COLARLY", but still is more meaningful to the user than the UNK tag. Future research will try to fix these errors using data-driven methods.

\section{Conclusion}\label{sec:concl}
Conventional wisdom and prior research suggests that direct acoustic-to-word (A2W) models require orders
of magnitude more data than sub-word unit-based models to perform competitively. This paper presents a recipe
to train a A2W model on the 2000-hour Switchboard+Fisher data set that performs at-par with several state-of-the-art hybrid and end-to-end models using sub-word units. We conclude that data order, model initialization, and regularization are crucial to obtaining a competitive A2W model with a WER of 8.8\%/13.9\% on the Switchboard/CallHome subsets of the Hub5-2000 test set. Next, we present a spell and recognize (SAR) model that learns to first spell a word and then recognize it. The proposed SAR model gives a rich and readable output
to the user while maintaining the training/decoding simplicity and performance of a A2W model. We show
some examples illustrating the SAR model's benefit for utterances containing OOV words.

\bibliographystyle{IEEEbib}

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE); Machine Learning (stat.ML)


Measurements of the heavy-flavour nuclear modification factor in p--Pb collisions at $\sqrt{s_{\rm NN}}$ = 5.02 TeV with ALICE at the LHC

Abstract: The heavy-flavour nuclear modification factor $R_{\rm pPb}$ has been measured with the ALICE detector in p--Pb collisions at the nucleon--nucleon center of mass energy $\sqrt{s_{\rm NN}}$ = 5.02 TeV at the CERN LHC in a wide rapidity and transverse momentum range, as well as in several decay channels. $R_{\rm pPb}$ is consistent with unity within uncertainties at mid-rapidity and forward rapidity. In the backward region a slight enhancement of the yield of heavy-flavour decay muons is found in the region $2<p_{\rm T}<4$ GeV/$c$. The results are described within uncertainties by theoretical calculations that include initial-state effects. The measurements add experimental evidence that the suppression of heavy-flavour production observed at high $p_{\rm T}$ in central Pb--Pb collisions with respect to pp collisions is due to a medium effect induced by the interaction of heavy quarks with the partonic matter.

\section{Introduction}
\vspace{-0.7em}
\label{intro}
Heavy quarks (charm and beauty) are essential probes of the properties of the medium created in heavy-ion
collisions, since they are produced in the early stage of hadronic collisions via
scattering processes with large momentum transfer. Heavy-flavour production has been
measured via semi-electronic and semi-muonic decays, as well as fully reconstructed
D mesons, in pp, p--Pb and Pb--Pb collisions with ALICE~\cite{alice} at the LHC.
In central Pb--Pb collisions at $\sqrt{s_{\rm NN}}$ = 2.76 TeV,
a strong suppression of high transverse momentum ($p_{\rm T}$)
D mesons and electrons (muons) from heavy-flavour hadron decays was observed at
mid- (forward) rapidity~\cite{dmesonPbPb, ePbPb, muonPbPb}.
This suppression is interpreted as an effect of parton energy loss in the
medium created in heavy-ion collisions. However, a quantitative understanding
of the Pb-Pb results requires cold nuclear matter (CNM) effects to be taken into account,
which can be accessed by studying p--Pb collisions~\cite{dmesonpPb, jpsipPb} assuming that
a hot and dense extended system is not formed in such collisions.
CNM effects are studied using the nuclear modification factor $R_{\rm pPb}$, defined as
\begin{equation}
\vspace{-0.4em}
R_{\rm pPb}=\frac{1}{\langle T_{\rm pPb}\rangle}\times\frac{\rm d\it N_{\rm pPb}/ \rm d\it p_{\rm T}}{\rm d\it \sigma_{\rm pp}/\rm d\it p_{\rm T}}
           =\frac{1}{\rm A}\times\frac{\rm d\it \sigma_{\rm pPb}/ \rm d\it p_{\rm T}}{\rm d\it \sigma_{\rm pp}/\rm d\it p_{\rm T}}
\end{equation}
where $\langle T_{\rm AA} \rangle$ is the average nuclear overlap function
estimated through the Glauber model,
which gives $\langle T_{\rm AA} \rangle = {\rm 0.0983\pm0.0035} \rm mb^{-1}$~\cite{identpPb};
$\rm d\it N_{\rm pPb}/\rm d\it p_{\rm T}$ ($\rm d\it \sigma_{\rm pPb}/\rm d\it p_{\rm T}$) is
the $p_{\rm T}$-differential yield (cross section) in p--Pb collisions and
$\rm d\it \sigma_{\rm pp}/\rm d\it p_{\rm T}$ is
the $p_{\rm T}$-differential cross section in pp collisions;
A is the mass number of the Pb nucleus.
The value of $R_{\rm pPb}$ is unity in absence of nuclear effects.

\vspace{-0.5em}
\section{Heavy-flavour measurements with ALICE}
\vspace{-0.7em}
ALICE is the dedicated heavy-ion experiment at the LHC.
It allows to investigate heavy-flavour production in several decay channels over a
wide rapidity and transverse momentum range.
The ALICE detector consists of a set of central barrel detectors ($\mid\eta_{\rm lab}\mid< 0.9$),
a muon spectrometer ($-4.0<\eta_{\rm lab}<-2.5$) and global detectors
for triggering and event characterization purposes.
At mid-rapidity, the Inner Tracking System (ITS) and
the Time Projection Chamber (TPC) provide track reconstruction
down to very low transverse momentum ($\sim$100 MeV/$c$)
with a momentum resolution better than 4$\%$ for $p_{\rm T}<20$ GeV/$c$,
as well as good impact parameter
(distance of closest approach of the track to the primary interaction vertex)
resolution~\cite{alice, cbperform}.
D mesons are reconstructed via their hadronic decay channels
and electrons from semileptonic decays of charm and$/$or beauty hadrons are measured.
At forward rapidity, muons from
heavy-flavour hadron decays can be measured, since the ALICE muon spectrometer allows to identify muons
by requiring that a track reconstructed in the tracking system is matched with
the corresponding track candidate in the muon trigger system.

D mesons are reconstructed
from their hadronic decay channels
${\rm D^{0}}$ $\rightarrow$ ${\rm K^{-}\pi^{+}}$ (BR = 3.88$\%$),
${\rm D^{+}}$ $\rightarrow$ ${\rm K^{-}\pi^{+}\pi^{+}}$ (BR = 9.13$\%$),
${\rm D^{*+}}$ $\rightarrow$ ${\rm D^{0}\pi^{+}}$ $\rightarrow$ ${\rm K^{-}\pi^{+}\pi^{+}}$ (BR = 67.7$\%$)
and ${\rm D^{+}_{s}}$ $\rightarrow$ ${\rm \phi\pi^{+}}$ $\rightarrow$ ${\rm K^{+}K^{-}\pi^{+}}$ (BR = 2.28$\%$).
The selection of D meson
candidates against the large combinatorial background is based on the
reconstruction of decay vertices displaced by a few hundred $\mu m$ from the
interaction vertex, exploiting the $c\rm \tau$ of
${\rm D^{0}}$, ${\rm D^{+}}$ and ${\rm D^{+}_{s}}$, which is
about $\rm 123-300~\mu m$ depending on the D-meson species.
In order to further enhance the ratio between the D-meson
signal and the combinatorial background,
the measurements of the particle time-of-flight from the collision point to the Time Of Flight (TOF) detector
and of the specific energy loss in the TPC gas are used to identify
$\rm K^{\pm}$ and $\pi^{\pm}$~\cite{dmesonhp}.
The measurement is performed in the rapidity interval $-0.96<y_{\rm cms}<0.04$
(which is specific for p-Pb collisions) over a wide transverse momentum range.

The electrons are identified with TPC and TOF at low $p_{\rm T}$,
as well as the TPC and Electromagnetic Calorimeter (EMCAL) at high $p_{\rm T}$.
The measurement of electrons from heavy-flavour hadron decays at mid-rapidity
($-1.06<y_{\rm cms}<0.14$)
requires the subtraction of contributions from several background sources from the inclusive
electron distribution. The dominant contribution is from photon conversions
in the detector material and Dalitz decays of light neutral mesons
($\pi^{\rm 0}$ and $\eta$, mainly).
These contributions are statistically subtracted by using
the cocktail
(i.e. a calculation of the background based mainly on measured
$p_{\rm T}$-differential cross sections of the main electron sources)
and invariant mass
(i.e. the measurement of electrons from photon conversions and Dalitz
decays via low-mass electron-positron pairs) methods~\cite{ePbPb}.
With the high spatial resolution of the track impact parameter measurement,
one can isolate the beauty-decay contribution
thanks to the much larger lifetime of beauty hadrons ($c\rm \tau \approx {\rm 500\mu m}$)
compared to that of charm hadrons and other background sources. The electrons from their
semileptonic decays have, consequently, a larger average impact parameter with respect
to the primary interaction vertex.

Muons from heavy-flavour hadron decays have been measured at forward
(i.e. proton-beam direction, $2.5<y_{\rm cms}<3.53$) and backward
(i.e. Pb-beam direction, $-4<y_{\rm cms}<-2.96$) rapidity, respectively,
by analyzing data collected with different beam configurations.
The main source of background in the $p_{\rm T}$-differential
inclusive spectrum consists of muons from light hadron decays
($\pi^{\pm}$ and $\rm K^{\pm}$, mainly).
This contribution dominates the muon yield for
$p_{\rm T}<2$ GeV/$c$ and prevents a measurement of muons from heavy-flavour
decays at low $p_{\rm T}$.
At larger $p_{\rm T}$ the contamination is estimated via
Monte-Carlo simulations, as well as with a data-driven method based
on the extrapolation of charged hadron
yields~\cite{cbhadron} measured at mid-rapidity with ALICE to forward rapidity.

\vspace{-0.5em}
\section{Results}
\vspace{-0.7em}
The nuclear modification factors, $R_{\rm pPb}$, of ${\rm D^{0}}$, ${\rm D^{+}}$, ${\rm D^{*+}}$
and ${\rm D^{+}_{s}}$ mesons are consistent with each other, and they are compatible with unity
within uncertainties~\cite{dmesonpPb}.
The pp reference cross section at $\sqrt{s}$ = 5.02 TeV is
obtained by a pQCD-based energy scaling~\cite{enscal,FONLL} of the $p_{\rm T}$-differential cross
sections measured at $\sqrt{s}$ = 7 TeV~\cite{dmesonpp}.
Figure~\ref{fig:DMeson} (left) presents the average $R_{\rm pPb}$
of prompt ${\rm D^{0}}$, ${\rm D^{+}}$ and ${\rm D^{*+}}$ in $1<p_{\rm T}<24$ GeV/$c$.
The $R_{\rm pPb}$ can
be described by means of next-to-leading-order (NLO) pQCD~\cite{MNR} calculations, including the
EPS09~\cite{EPS09} nuclear modification of the CTEQ6M~\cite{CTEQ6M} Parton Distribution Functions (PDF)
and calculations based on the Color Glass Condensate (CGC~\cite{CGC}) initial-state prescription.
The data are also well described by calculations which include
energy loss in cold nuclear matter,
nuclear shadowing and $k_{\rm T}$-broadening~\cite{kTBroad}.
As demonstrated in Figure~\ref{fig:DMeson} (right),
cold nuclear matter effects are small for $p_{\rm T}\gtrsim$3 GeV/$c$.
This indicates that the suppression of D mesons observed in the 20$\%$ most
central Pb--Pb collisions at $\sqrt{s_{\rm NN}}$ = 2.76 TeV~\cite{dmesonPbPb} results from
an effect due to the presence of the hot and dense medium.

The $R_{\rm pPb}$ of electrons from heavy-flavour hadron decays in p--Pb collisions
at $\sqrt{s_{\rm NN}}$ = 5.02 TeV is shown in Figure~\ref{fig:Electron} (left).
The pp reference is obtained following a strategy similar to the one used for the D-meson analysis.
The $R_{\rm pPb}$ is compatible with unity, showing that cold nuclear matter effects are small.
The measurement is compared with a pQCD calculation with the EPS09~\cite{EPS09} NLO
parameterization of nuclear PDFs. Within uncertainties,
the data can be described by the model predictions.
Figure~\ref{fig:Electron} (right) displays the measured $R_{\rm pPb}$ of electrons from
heavy-flavour hadron decays together with that of electrons from beauty-hadron decays.
The $R_{\rm pPb}$ of electrons from beauty-hadron decays is consistent with unity within
uncertainties, and it is also compatible with that of electrons from heavy-flavour hadron decays.
The data indicate that the suppression of electrons from heavy-flavour hadron decays observed in the 10$\%$ most
central Pb--Pb collisions at $\sqrt{s_{\rm NN}}$ = 2.76 TeV~\cite{ePbPb} results from
a hot medium effect.
\begin{figure}[htbp]
\vspace{-1.1em}
\begin{center}
\includegraphics*[width=0.38\textwidth]{2014-May-15-pPbWithModels}
\hspace{1.0em}
\includegraphics*[width=0.38\textwidth]{2014-May-15-pPbAndPbPb}
\end{center}
\vspace{-1.7em}
\caption{Left: Average $R_{\rm pPb}$ of prompt ${\rm D^{0}}$, ${\rm D^{+}}$ and
        ${\rm D^{*+}}$ mesons as a function of $p_{\rm T}$ compared to model
	calculations~\cite{MNR, EPS09, CTEQ6M, CGC, kTBroad};
        Right: Average $R_{\rm pPb}$ of prompt ${\rm D^{0}}$, ${\rm D^{+}}$ and
        ${\rm D^{*+}}$ mesons as a function of $p_{\rm T}$ compared to the prompt D-meson $R_{\rm AA}$ in central
	(0-20$\%$) and semi-peripheral (40-80$\%$) Pb--Pb collisions at $\sqrt{s_{\rm NN}}$ = 2.76 TeV~\cite{ePbPb}.}
\label{fig:DMeson}
\end{figure}

\begin{figure}[htbp]
\vspace{-1.7em}
\begin{center}
\includegraphics*[width=0.40\textwidth]{2014-May-14-rPa_minbias_shadow}
\includegraphics*[width=0.40\textwidth]{2014-May-15-FigRpPb_derived}
\end{center}
\vspace{-1.7em}
\caption{Left: $R_{\rm pPb}$ of electrons from heavy-flavour hadron decays
        as a function of $p_{\rm T}$ compared to model calculations~\cite{FONLL, EPS09};
        Right: $R_{\rm pPb}$ of electrons from heavy-flavour hadron decays
        compared to $R_{\rm pPb}$ of electrons from beauty-hadron decays.}
\label{fig:Electron}
\end{figure}

The nuclear modification factor of muons from heavy-flavour hadron
decays, shown in Figure~\ref{fig:Muon} (left), is measured in the center
of mass rapidity domains $2.5<y_{\rm cms}<3.53$ (forward rapidity) and
$-4<y_{\rm cms}<-2.96$ (backward rapidity).
The pp reference cross section of heavy-flavour hadron decay muons at $\sqrt{s}$ = 5.02 TeV is
obtained via a pQCD-based energy scaling procedure~\cite{FONLL}
(i.e. calculated taking as input the $p_{\rm T}$-differential cross
sections measured at $\sqrt{s}$ = 7 TeV~\cite{muonpp}), performed at forward and backward
rapidity, respectively.
This procedure is used to obtain the $p_{\rm T}$-differential cross section up to 12 GeV/$c$.
In order to measure the $R_{\rm pPb}$ in a wider transverse
momentum range, the $p_{\rm T}$-differential cross section in pp collisions at $\sqrt{s}$ = 5.02 TeV was
extrapolated to higher $p_{\rm T}$
using the spectrum predicted by FONLL scaled to match the obtained pp reference in the range $6<p_{\rm T}<12$ GeV/$c$.
The $R_{\rm pPb}$ measured at forward rapidity is consistent with
unity within uncertainties. $R_{\rm pPb}$ at backward rapidity
is slightly larger than unity in the range $2<p_{\rm T}<4$ GeV/$c$
and close to unity at higher $p_{\rm T}$.
This supports the previous observation that the suppression measured in the 10$\%$ most central Pb--Pb
collisions at $\sqrt{s_{\rm NN}}$ = 2.76 TeV~\cite{muonPbPb} results from hot and dense medium effect.

Another way to quantify cold nuclear matter effects is to study the forward-to-backward ratio
defined as the cross section measured at forward rapidity with respect to the one at backward rapidity,
The main advantage of using such a ratio is that the pp reference and the nuclear overlap
function cancel. The drawback of this approach is the limited statistics,
because the common $y_{\rm cms}$ interval covered at both forward and backward rapidity
is small ($\sim 0.57$ units).
\begin{equation}
R_{\rm FB}(2.96<\mid y_{\rm cms}\mid<3.54)=\frac{\rm d\it \sigma^{\rm forward}/  \rm d\it p_{\rm T}{(\rm 2.96}<y_{\rm cms}{\rm <3.54)}}
                                                {\rm d\it \sigma^{\rm backward}/ \rm d\it p_{\rm T}{(\rm -3.54}<y_{\rm cms}{\rm <-2.96)}}
\end{equation}
The measured $R_{\rm FB}$ is presented in Figure~\ref{fig:Muon} (right).
It is systematically smaller than unity in the range $2<p_{\rm T}<4$ GeV/$c$ and close to unity at higher $p_{\rm T}$.
This is well reproduced by a pQCD calculation including the EPS09~\cite{EPS09} NLO
parameterization of nuclear PDFs~\cite{MNR} in the range $2<p_{\rm T}<16$ GeV/$c$.
The $R_{\rm pPb}$ measured at backward rapidity is slightly underestimated by
such theoretical calculations at low $p_{\rm T}$.
\begin{figure}[htbp]
\vspace{-1.1em}
\begin{center}
\includegraphics*[width=0.45\textwidth]{2014-May-15-RpA_RAp_RAA}
\includegraphics*[width=0.45\textwidth]{2014-May-15-RFB_Results}
\end{center}
\vspace{-1.7em}
\caption{Left: $R_{\rm pPb}$ of muons from heavy-flavour hadron decays
        as a function of $p_{\rm T}$ at forward rapidity ($2.5<y_{\rm cms}<3.53$) and
        backward rapidity ($-4<y_{\rm cms}<-2.96$), compared to the $R_{\rm AA}$ for
        muons from heavy-flavour hadron decays in the 10$\%$ most central Pb--Pb
        collisions at $\sqrt{s_{\rm NN}}$ = 2.76 TeV~\cite{muonPbPb};
        Right: Forward-to-backward ratio $R_{\rm FB}$ of muons from heavy-flavour hadron decays
        compared to a model calculation~\cite{MNR, EPS09}.}
\label{fig:Muon}
\end{figure}

\vspace{-1.0em}
\section{Conclusions}
\vspace{-0.7em}
The nuclear modification factor of heavy-flavour production has been measured
in a wide rapidity and transverse momentum range, as well as in several
decay channels, in p--Pb collisions at
$\sqrt{s_{\rm NN}}$ = 5.02 TeV with ALICE. The $R_{\rm pPb}$ results provide evidence
that cold nuclear matter effects are small, showing that the strong suppression observed in central
Pb--Pb collisions at $\sqrt{s_{\rm NN}}$ = 2.76 TeV is a final state effect due to in-medium parton energy loss.

\vspace{-0.5em}
\section*{Acknowledgments}
\vspace{-0.7em}
{\small
This work is supported partly by the Chinese Ministry of Science and Technology
973 grant 2013CB837803, the NSFC Grant 11375071 and IRG11221504,
CCNU Key grant CCNU13F026, and Key Laboratory QLPL2014P0109.
}

\vspace{-0.5em}

High Energy Physics - Experiment (hep-ex); Nuclear Experiment (nucl-ex)