Structured pruning via scaling factors is a technique where the output of a structure is multiplied to a scaling factor. In online structured pruning, the pruning is modelled as a task of learning this scaling factor associated to each filter [1]}, [2]}. This scaling factor acts as an on/off switch which in its off state mimics the elimination of the filter. Regularization is usually applied to these scaling factors to prune the corresponding structures from the network [1]}, [2]}. Several schemes have been proposed to update the values of these scaling factors during training. For e.g. differentiable Markov Chains has been used to find prunable filters where each node of the chain is the state of a filter either on or off [5]}. Similarly, if a noise across layers is represented as a Markov Chain, then its posterior can be used to reflect inter-layer dependency which act as a measure for pruning[6]}. Rather than pruning the network to the maximum possible limit without noticeable loss in accuracy, pruning can also be performed to achieve a certain budget. In such cases knowledge distillation loss [7]} with budget aware regularizers have been shown to perform the desired amount of compression [1]}.
