Visual Transformer Meets CutMix for Improved Accuracy, Communication Efficiency, and Data Privacy in Split Learning
This article seeks for a distributed learning solution for the visual
transformer (ViT) architectures. Compared to convolutional neural network (CNN)
architectures, ViTs often have larger model sizes, and are computationally
expensive, making federated learning (FL) ill-suited. Split learning (SL) can
detour this problem by splitting a model and communicating the hidden
representations at the split-layer, also known as smashed data.
Notwithstanding, the smashed data of ViT are as large as and as similar as the
input data, negating the communication efficiency of SL while violating data
privacy. To resolve these issues, we propose a new form of CutSmashed data by
randomly punching and compressing the original smashed data. Leveraging this,
we develop a novel SL framework for ViT, coined CutMixSL, communicating
CutSmashed data. CutMixSL not only reduces communication costs and privacy
leakage, but also inherently involves the CutMix data augmentation, improving
accuracy and scalability. Simulations corroborate that CutMixSL outperforms
baselines such as parallelized SL and SplitFed that integrates FL with SL.