where we used the notation \(\tau _{kl}=P(\phi _t=l|\phi _{t-1}=k)\) , \(k,l\in \lbrace 0,1\rbrace \) . These can then be vectorized for the whole layer as \(\rho _0\) , \(\tau _{11}\)  and \(\tau _{01}\)  \(\in [0,1]^H\) . In order to express the remaining HMM parameters related to the set of distributions \(B(x_t)\) , we can assume that the likelihood of observing \(x_t\)  given the current state of the hidden features \(\phi _t\) , can be described using a distribution from the exponential family. As we will see in Section REF , only the ratio of these distributions will be necessary to compute. As demonstrated by Garner and Tong [1]} drawing from Bridle [2]}, this ratio of likelihood \(r_t\)  can then be expressed as
\(r_t:=\frac{p(x_t|\,\lnot \phi _t\,)}{p(x_t|\,\phi _t\,)}=\exp \Big [-W^T\,x_t - b \Big ] \, .\) 
