Models and Datasets.
To evaluate the performance of SALO and its speedup ratio compared to GPU and CPU, we use two representative models, Longformer [1]} and ViL [2]} as the benchmark. Longformer is a variant of Transformer designed for long document in natural language processing. We choose the most commonly used implentation from Transformers [3]} by Huggingface. ViL is a variant of ViT [4]} extracting multi-scale features from images with sparse hybrid mechanisms. We use the implementation open sourced by the author. As for the hyper parameters, we follow the settings of Longformer-Base-4096 and ViL-Medium-Wide. Note that ViL consists of 4 stages with an attention layer in each stage. But we only test the performance on the first 2 stages that involve hybrid attention mechanism. We use IMDB and Hyperpartisan to evaluate the accuracy of Longformer and ImageNet-1K to evaluate the accuracy of ViL.
