The key privacy concern for users is whether model updates reveal too much about the data on which they were calculated. Although [1]} discuss that “model updates are more focused on the learning task at hand than is the raw data (i.e. they contain strictly no additional information about the user, and typically significantly less, compared to the raw data)", scenarios can be constructed in which the model updates themselves can be inverted to recover their input user information [2]}, [3]}. Simple knowledge of the shared model state and model update can be sufficient for such an attack [4]}, [5]}. These inversion attacks are particularly fruitful if a user's model update is based on a single data point or only a small batch.
Accordingly, a strong defense against these attacks is aggregation. The user only reports model updates aggregated over a significant number of local data points, and data from multiple users can be combined with secure aggregation protocols [6]} before being passed to the server. This ability to aggregate user updates while maintaining their utility is thought to be the main source of security in federated learning. Averaging raw local data in similar amounts would make it unusable for training, but model updates can be effectively aggregated.
