Various techniques have been proposed to produce fixed-length continuous word vectors based on characters or morphemes [1]}, [2]}, [3]}, [4]}.
An effort to apply such techniques to NMT, parallel to ours, has found no significant improvement over word-based approaches [5]}.
One technical difference from our work is that the attention mechanism still operates on the level of words in the model by 2015arXiv151104586L, and that the representation of each word is fixed-length.
We expect that the attention mechanism benefits from our variable-length representation: the network can learn to place attention on different subword units at each step.
Recall our introductory example Abwasserbehandlungsanlange, for which a subword segmentation avoids the information bottleneck of a fixed-length representation.
