The seq2seq (or encoder-decoder) is an important standard architecture for BlenderBot and other conversational ai [1]}.
The Transformer, by [2]}, is often used as the underlying architecture for it, though the lstm, by hochreiter1997long, may also be used.
Generally, the encoder-decoder conditions on the encoding of the prompts and responses up to the last time-step for it to generate the next token as response [3]}, [4]}.
The sequence of tokens is run through the encoder stack's embedding layer, which then compresses it in the dense feature layer into fixed-length feature vector.
A sequence of tokens is then produced by the decoder after they are passed from the encoder layer.
The Softmax function is then used to normalized this, such that the token with the highest probability is the output.
