Compression Methods –
The communication-efficient global model update methods can be categorized into two groups: structured update and sketched update ([1]}).
The structured update indicates the methods that enforce a pre-defined fixed structure of the local updates, such as low-rank approximation and random mask methods.
The sketched update indicates the methods that post-process the local updates via compression, sparsification, or quantization.
Both directions are well studied and have shown successful results ([2]}, [3]}, [4]}, [5]}, [6]}, [7]}, [8]}, [9]}, [10]}).
The common principle behind these methods is that the local updates can be replaced with a different data representation with a smaller size.
