The Halting Policy Network, on the other hand, samples its actions and so its training is more intensive, though we follow the standard policy gradient method and use the REINFORCE algorithm [1]} to estimate the gradient with which we update the network's parameters.
To balance between earliness and accuracy, the parameters of the Halting Policy Network are updated with respect to two goals: Make \(\tau \)  small, and make \(\hat{y}\)  accurate.
Following the state-of-the-art [2]}, we achieve smooth optimization by rewarding accurate predictions and penalizing the cumulative probability of Waiting over each episode.
Thus the loss function for optimizing the halting policy network is computed as:
\(\mathcal {L}_\text{early} =&-\mathbb {E}\left[\sum _{t^\prime }\log \pi _\text{stop}(a_{t^\prime }|h_{t^\prime })\left[\sum _{j=t^\prime }(r_{t^\prime }-b_{t^\prime })\right]\right]\\&-\mathbb {E}\left[\sum _{t^\prime }\log \pi _\text{hop}(a_{t^\prime }|h_{t^\prime })\left[\sum _{j=t^\prime }(r_{t^\prime }-b_{t^\prime })\right]\right]\\&- \lambda \sum _{t^\prime }\log \pi _\text{stop}(a_{t^\prime }=\text{Stop}|h_{t^\prime }),\) 
