While many of the feature transformation mechanisms described so far have been linear (i.e. parameterized as an affine map and bias term) there have been several recent efforts to learn quadratic feature modifications using a self attention module. This is the primary contribution of [1]} and [2]} which were published simultaneously. Svoboda et al. [3]} combine this module with the contrastive representation learning of [4]}.
