On Neural Architecture Inductive Biases for Relational Tasks
Current deep learning approaches have shown good in-distribution
generalization performance, but struggle with out-of-distribution
generalization. This is especially true in the case of tasks involving abstract
relations like recognizing rules in sequences, as we find in many intelligence
tests. Recent work has explored how forcing relational representations to
remain distinct from sensory representations, as it seems to be the case in the
brain, can help artificial systems. Building on this work, we further explore
and formalize the advantages afforded by 'partitioned' representations of
relations and sensory details, and how this inductive bias can help recompose
learned relational structure in newly encountered settings. We introduce a
simple architecture based on similarity scores which we name Compositional
Relational Network (CoRelNet). Using this model, we investigate a series of
inductive biases that ensure abstract relations are learned and represented
distinctly from sensory data, and explore their effects on out-of-distribution
generalization for a series of relational psychophysics tasks. We find that
simple architectural choices can outperform existing models in
out-of-distribution generalization. Together, these results show that
partitioning relational representations from other information streams may be a
simple way to augment existing network architectures' robustness when
performing out-of-distribution relational computations.