Fully Privacy-Preserving Federated Representation Learning via Secure Embedding Aggregation
We consider a federated representation learning framework, where with the
assistance of a central server, a group of $N$ distributed clients train
collaboratively over their private data, for the representations (or
embeddings) of a set of entities (e.g., users in a social network). Under this
framework, for the key step of aggregating local embeddings trained at the
clients in a private manner, we develop a secure embedding aggregation protocol
named SecEA, which provides information-theoretical privacy guarantees for the
set of entities and the corresponding embeddings at each client
$simultaneously$, against a curious server and up to $T < N/2$ colluding
clients. As the first step of SecEA, the federated learning system performs a
private entity union, for each client to learn all the entities in the system
without knowing which entities belong to which clients. In each aggregation
round, the local embeddings are secretly shared among the clients using
Lagrange interpolation, and then each client constructs coded queries to
retrieve the aggregated embeddings for the intended entities. We perform
comprehensive experiments on various representation learning tasks to evaluate
the utility and efficiency of SecEA, and empirically demonstrate that compared
with embedding aggregation protocols without (or with weaker) privacy
guarantees, SecEA incurs negligible performance loss (within 5%); and the
additional computation latency of SecEA diminishes for training deeper models
on larger datasets.