Inspired by BitFit [1]} that fine-tunes with only bias terms, we untie all the bias terms for each layer and expect to use the free bias terms to specialize the layers with tied weights. As BitFit, bias-based layer adaptation introduces very few additional parameters without inference overhead.
