During the collaborative learning between peer networks, if the outputs of peer networks are very diverse, the mutual knowledge transfer could become poor. Since the self-learning via self-distillation can improve the power of knowledge transfer [1]}, CTSL-MKT further introduces the self-learning of each peer network into collaborative learning via the response-based knowledge self-distillation. To conduct self-learning for each peer network \(N_{k}\) ,
we use the outputs \(\bar{\emph {\textbf {p}}}_k^{t}\)  of the pre-trained network \(N_{k}\)  to supervise itself
with the following
self-distillation loss:
\(\small L_{SD}^k(\emph {\textbf {p}}_k^{t},\bar{\emph {\textbf {p}}}_k^{t})=\sum _{x\in X}\sum _{i=1}^m\sigma _i^t(\bar{\emph {\textbf {z}}}_k(x),t)log\frac{\sigma _i^t(\bar{\emph {\textbf {z}}}_k(x),t)}{\sigma _i^{t}(\emph {\textbf {z}}_k(x),t)}~.\) 
