Before we have discussed the limitations of the Dot Product decoder.
An interesting questions is whether the HadamardMLP decoder can replace the Dot Product decoder by approximating it.
If the MLP decoder can learn a dot product easily, it is safe to use MLP decoder instead of the dot product ones in most cases.
There are similar problems actively studied in machine learning.
Existing work imply that the difficulty scales polynomial with dimensionality \(d\)  and \(1/\epsilon \)  in theory [1]}, [2]}, [3]}.
This motivates us to investigate the question empirically.
<FIGURE><FIGURE>