For model architecture, The ResNet18 [1]} and MobileNetV1 [2]} are built as the backbones following ReActNet [3]}. In the ReActNet (Shown as in Figure REF ), the 3\(\times \) 3 depthwise and 1\(\times \) 1 pointwise convolution are replaced by standard 3\(\times \) 3 and 1\(\times \) 1 convolution. The duplication and concatenation operations are designed for addressing the channel number difference. In our case, we simply replace the RSign and RPReLU functions in ReActNet with our \(DySign\)  and \(DyPReLU\) . The ReActNet can be regarded as the baseline in our experiment.
