We investigate learning to probabilistically bypass computations in a network architecture. Our approach is motivated by AIG, where layers are conditionally executed depending on their inputs, and the network is trained against a target bypass rate using a per-layer loss. We propose a per-batch loss function, and describe strategies for handling probabilistic bypass during inference as well as training. Per-batch loss allows the network additional flexibility. In particular, a form of mode collapse becomes plausible, where some layers are nearly always bypassed and some almost never; such a configuration is strongly discouraged by AIG's per-layer loss. We explore several inference-time strategies, including the natural MAP approach. With data-dependent bypass, we demonstrate improved performance over AIG. With data-independent bypass, as in stochastic depth, we observe mode collapse and effectively prune layers. We demonstrate our techniques on ResNet-50 and ResNet-101 for ImageNet , where our techniques produce improved accuracy (.15--.41 in precision@1) with substantially less computation (bypassing 25--40 of the layers).
During train time, the activation rate, a hyperparameter set by the user, determines how frequently each individual gate should be open and the target loss is added to the classification loss, where activation loss is L2 of the difference between the target rate and current rate. Our work differs from @cite_31 several ways. We reformulate the loss to a per-batch loss and consider both data-dependent and data-independent layer bypass. Data-independent per-batch loss results in the network trying to remove enough layers to reach the target rate while retaining accuracy (similar to pruning techniques), while data-dependent per-batch loss has more flexibility to utilize layers.
@cite_31
2011266042
Using a multi-tier Poisson model, this paper studies the performance gain of successive interference cancellation (SIC) in the downlink of K-tier heterogeneous cellular networks (HCNs). For each tier, a fraction of base stations (BSs) is non-accessible. By using a framework based on the marked path loss process with fading and calculating the equivalent access probability, we analytically characterize the coverage probability, i.e., the probability of successfully connecting to at least one accessible BS, for a typical user equipment with finite or infinite SIC capability. The results show how the performance gain of SIC depends on many system parameters including path loss exponent, coding rate, fading distributions and BS accessibilities and densities. We show for contemporary OFDM-based HCNs, infinite SIC capability is often unnecessary. In fact, under typical system parameters, most of the gain of SIC comes from the ability of canceling only a single non-accessible BS.