As the pruning literature develops, the pruning criteria also evolves to better reflect final efficiency. The early phase of the field focuses on maximum parameter removal in seek for minimum representations of the pretrained model. This leads to a flourish of approaches that rank neurons effectively to measure their importance [1]}, [2]}. As each neuron/filter possesses intrinsically different computation, following works explore proxy to enhance redundancy removal, FLOPs being one of the most widely adopted metrics [3]}, [4]}, [5]} to reflect how many multiplication and addition computes needed for the model.
Hovewer, for models with very similar FLOPs, their latency can vary significantly [6]}.
Recently, more and more works start directly working on reducing latency [7]}, [8]}. However, not much was done in the field of GPU friendly pruning methods due to non-trivial latency-architecture trade-off.
For example, as recently observed in [9]}, GPU usually imposes staircase-shaped latency patterns for convolutional operators with varying channels, which inevitably occur per varying pruning rate, see the latency surface in Fig. REF .
This imposes a constraint that pruning needs to be done in groups to achieve latency improvement. Moreover, getting the exact look-up table of layers under different pruning configuration will benefit maximizing performance while minimizing latency.
<FIGURE>