Impossibility of Collective Intelligence
Democratization of AI involves training and deploying machine learning models
across heterogeneous and potentially massive environments. Diversity of data
opens up a number of possibilities to advance AI systems, but also introduces
pressing concerns such as privacy, security, and equity that require special
attention. This work shows that it is theoretically impossible to design a
rational learning algorithm that has the ability to successfully learn across
heterogeneous environments, which we decoratively call collective intelligence
(CI). By representing learning algorithms as choice correspondences over a
hypothesis space, we are able to axiomatize them with essential properties.
Unfortunately, the only feasible algorithm compatible with all of the axioms is
the standard empirical risk minimization (ERM) which learns arbitrarily from a
single environment. Our impossibility result reveals informational
incomparability between environments as one of the foremost obstacles for
researchers who design novel algorithms that learn from multiple environments,
which sheds light on prerequisites for success in critical areas of machine
learning such as out-of-distribution generalization, federated learning,
algorithmic fairness, and multi-modal learning.