The previous approach is a continuation of using local vocabulary cache for language modelling, as proposed by [1]}. The effect of specific history size is examined as well. The usage of cache can be interpreted as using a small local dynamic knowledge base that is being updated after every prediction. This is motivated by the fact that especially rare words tend to occur more probably than by overall uniform distribution, given that they appeared in recent history. The LSTM [2]} hidden state size is again 1024-dimensional, although this time, it is not used for retrieval but only by the aggregator. For the fusion, the authors propose two methods: (1) linear interpolation, as seen in the previous language model and (2) joint softmax over the artefact and the language model output distribution.
