Longitudinal spin-relaxation in nitrogen-vacancy centers in electron irradiated diamond

Abstract: We present systematic measurements of longitudinal relaxation rates ($1/T_1$) of spin polarization in the ground state of the nitrogen-vacancy (NV$^-$) color center in synthetic diamond as a function of NV$^-$ concentration and magnetic field $B$. NV$^-$ centers were created by irradiating a Type 1b single-crystal diamond along the [100] axis with 200 keV electrons from a transmission electron microscope with varying doses to achieve spots of different NV$^-$ center concentrations. Values of ($1/T_1$) were measured for each spot as a function of $B$.

\section{\label{sec:Intro}Introduction}
Nitrogen-vacancy (NV$^-$) centers in diamond~\cite{Jelezko:2006} are
useful for quantum information~\cite{GurudevDutt:2007},
magnetometry (see the review by Rondin \textit{et al.}~\cite{Rondin:2014})
and nanoscale sensing applications (see the review by
Schirhagl \textit{et al.}~\cite{Schirhagl:2014}). NV$^-$ centers have been used to
detect single electron spins~\cite{Grotz:2011,Mamin:2012,Shi:2015} and
small ensembles~\cite{Mamin:2013,Staudacher:2013,Rugar:2014,Haberle:2015,DeVience:2015}
and single nuclear spins~\cite{Sushkov:2015},
study magnetic resonance on a molecular scale, measure electric fields, strain and temperature,
detect low concentrations of paramagnetic molecules and ions, and image
magnetic field distributions of physical or biological systems.
These applications are made possible by the unique properties of the NV$^-$ center
level structure, shown in Fig.~\ref{fig:levels}, which allows manipulation of the
ground-state spin state by optical fields and microwaves and measurement of the
interactions of the ground-state spin with the local environment by monitoring the fluorescence
intensity. Understanding spin relaxation processes is important in optimizing these techniques.
Previous measurements of the dependence of longitudinal relaxation rate ($1/T_1$) of magnetic field
have shown enhanced rates near $B=0$ G, $B=595$ G (Refs.~\cite{Armstrong:2010,Jarmola:2012,Anishchik:2015,Mrozek:2015}),
and $B=514$ G (Refs.(~\cite{Jarmola:2012,Hall:2015}). The enhancements at zero field and
$B=595$ G have been linked to interactions with NV centers whose orientation
makes their energies degenerate at these fields,
while the enhancement at $B=514$~G is related to interactions with substitutional nitrogen
(P1 centers).
Previous work has also shown that the $1/T_1$ rate
depends on NV concentration~\cite{Mrozek:2015}.
In this paper we describe systematic measurements of the longitudinal relaxation
rates of ensembles of NV$^-$ centers created with different, controlled
radiation doses on a single diamond crystal, achieved through irradiation with a transmission electron microscope (TEM).
This method of preparing NV$^-$ centers is more convenient for many laboratories than irradiation in
accelerators and also could facilitate the creation of microscopic structures on a diamond chip for special
applications.

\begin{figure}
  \centering
    \resizebox{\columnwidth}{!}{\includegraphics{levels3}}
      \caption{\label{fig:levels}Energy level diagram of NV$^-$ center. Solid vertical lines
      denote radiative transitions, whereas dashed lines denote transitions thought to be nonradiative.}
\end{figure}

The diamond sample was a [100] cut Type 1b single-crystal plate (Element 6),
grown by the high-pressure, high-temperature technique, with an initial nitrogen concentration
of 200 ppm and dimension of \SI{3}{\milli\metre} $\times$ \SI{3}{\milli\metre} $\times$ \SI{0.3}{\milli\metre}.
It was irradiated using a
transmission electron microscope~\cite{Kim:2012} (Tecnai G20 FEI with a Schottky field
emitter electron source). The accelerating voltage was \SI{200}{\kilo\volt}.
At these energies the electrons can be expected to penetrate about \SI{140}{\micro\metre} into the
diamond\cite{estar}. However, it is unlikely that they will have sufficient energy to create vacancies below
a depth of about ~\SI{20}{\micro\metre}\cite{Koike:1992}.
Seventeen circular spots with a diameter of \SI{10}{\micro\metre} were irradiated.
After irradiation, the sample was maintained at \SI{800}{\degreeCelsius} for three hours
in the presence of nitrogen gas at less than atmospheric pressure.
Then the sample was placed in a fluorescence microscope and illuminated with a mercury lamp through a filter cube
(Olympus U-MWG2), which reflected excitation light from~\SI{510}{\nano\metre} to \SI{550}{nm} to the sample, and passed the emission through
a ~\SI{590}{\nano\metre} long-pass filter. The red fluorescence was photographed with a Peltier-cooled CCD camera (GT Vision GXCAM-5C).
The pixel intensities of the fluorescence are shown in Fig.~\ref{fig:luminosity}; the
irradiation parameters of the samples are given in Table~\ref{tab:irrad}.
Spots~5--13 were irradiated with an electron intensity of \SI{3530}{\per\nano\metre\squared\per\second}
to reach the dose listed in the table. Spot~14 was obtained by irradiating
with an electron intensity of \SI{2530}{\per\nano\metre\squared\per\second} to compare results for irradiation at
different rates but same total intensity.
\begin{figure}
  \centering
    \resizebox{\columnwidth}{!}{\includegraphics{points}}
      \caption{\label{fig:luminosity}Fluorescence microscope image of the irradiated and annealed sample with a cross-section of fluorescence intensity. The colorbar indicates the intensity of the fluorescence.}
\end{figure}

\begin{table}
\centering
\caption{Parameters of the irradiated spots numbered as per the labels in Fig.~\ref{fig:luminosity}:
Integrated (red) fluorescence intensity upon irradiation with green light, electron dose and estimated
NV$^-$ concentration.}
\label{tab:irrad}
\begin{tabular}{cccc}
\hline
\hline
    & \multicolumn{1}{c}{Integrated}		   	& \multicolumn{1}{c}{Electron}                      & Estimated\\
    & \multicolumn{1}{c}{Fluorescence}	   	& \multicolumn{1}{c}{Dose}                          & NV$^-$\\
Spot & \multicolumn{1}{c}{Intensity}		    &                                                   & Concentration\\
Nr.  & \multicolumn{1}{c}{(arb. units)}     & \multicolumn{1}{c}{(\si{\per\centi\metre\squared})} & (ppm) \\
\hline
5   & 1400						                      & $1.1 \times 10^{19}$                                & 0.2\\
6   & 2600 					   	                    & $2.1 \times 10^{19}$                                & 0.3\\
7   & 5700 					   	                    & $4.2 \times 10^{19}$                                & 0.7\\
8   & 10000					   	                    & $8.5 \times 10^{19}$                                & 1.2\\
9   & 6300 					   	                    & $1.7 \times 10^{20}$                                & 0.7\\
10  & 2900					   	                    & $3.4 \times 10^{20}$                                & 3.3\\
11  & 50000					   	                    & $6.8 \times 10^{20}$                                & 5.5\\
12  & 39000					   	                    & $1.3 \times 10^{21}$                                & 4.3\\
13  & 65000					   	                    & $2.5 \times 10^{21}$                                & 7.1\\
14  & 8300 					   	                    & $6.1 \times 10^{19}$                                & 3.9\\
\hline
\end{tabular}
\end{table}

The experimental setup used to measure longitudinal relaxation times as a
function of magnetic field is shown in Fig.~\ref{fig:exp}.
One should note that our procedure assumes that the relaxation
rates for the $m_S=0$ and $m_S=\pm1$ sublevels are identical.
NV centers were excited with \SI{512}{\nano\metre} light from an
external cavity diode laser (Toptica DL100 pro).
The laser power before the microscope lens was around \SI{7}{\milli\watt}.
The microscope lens had a focal length of ~\SI{4.5}{\milli\metre} and numerical aperture of 0.55.
The sample holder was mounted on a three-axis positioning stage (Thorlabs Max341)
  and placed inside a three-axis Helmholtz coil system, allowing
control over the magnetic field direction and magnitude.
Microwaves (MW) from a frequency generator (Stanford Research Systems
SG386) passed through a switch (Minicircuits ZASWA-2-50DR+) and were amplified with a
\SI{16}{\watt} MW amplifier (Minicircuits ZHL-16W-43-S+). The MW were delivered to the sample by
a \SI{0.071}{\milli\metre} diameter copper wire placed close to the diamond surface, and the
wire was terminated into \SI{50}{ohm} after the sample.

The fluorescence from the sample was collected using the same microscope lens that
focused the light onto the sample, and then passed through a dichroic mirror (Thorlabs DMLP567),
which passed wavelengths longer than \SI{567}{\nano\metre}.
After passing through an additional filter that further suppressed the green excitation light,
the  fluorescence could be deflected
to either a CMOS camera for visual adjustments of the sample, wire or position on sample,
or to the avalanche photodiode (Thorlabs APD110A/M) for overall fluorescence measurements.

The current in the Helmholtz coils was adjusted so that the magnetic field
pointed in the [111] crystallographic direction. An optically detected
magnetic resonance (ODMR) signal was obtained by scanning the microwave
frequency under continuous laser irradiation and measuring the fluorescence (see Fig.~\ref{fig:odmr}).
In this configuration two ODMR peaks appear on either side of the
microwave frequency of \SI{2.87}{\giga\hertz}, which is the frequency of the NV resonance in the
absence of magnetic field. The two inner peaks are more intense and
correspond to the three possible alignments of the NV axis that make
the same angle with the magnetic field. The two outer peaks correspond to the
alignment of the NV axis that is parallel to the magnetic field.
Then, the microwave generator was set to the frequency corresponding
to one of the outer ODMR peaks (both were measured).

The laser light hitting the sample could be turned on and off with an
accousto-optic modulator (AA OptoElectronic MT200-A0.5-VIS).
An pulse generator (PulseBlaster ESR-PRO-500)
was used to control the accousto-optic modulator and MW switch. To measure the longitudinal relaxation
time, a decay curve was generated as follows. First, the NV$^-$ centers were pumped into
$m_S=0$ level of the ground state with a green laser pulse that lasted~\SI{50}{\micro\second}. Then
the laser light was blocked for a variable time $\tau$, after which the sample was again illuminated.
The fluorescence as a function of time was recorded on an oscilloscope (Yokogawa DL6154), averaged
1024 times and saved to disk.
The procedure was repeated, but a
microwave $\pi$ pulse was applied after the laser pump pulse. This sequence was repeated five times.
The fluorescence
obtained after time $\tau$ with the $\pi$ pulse was subtracted from the fluorescence obtained
after time $\tau$ without the $\pi$ pulse to eliminate contributions to the signal from other
NV$^-$ alignments and other sources of common-mode noise~\cite{Jarmola:2012}. The fluorescence immediately after
time $\tau$ was normalized to the fluorescence after the spins have been pumped into the $m_S=0$ ground-state
sublevel, and this quantity $I$ was plotted as a function of $\tau$.
The plot was fit with a stretched exponential of the form $\exp{-(\frac{\tau}{T_{1}})^{\beta}}$,
where $T_{1}$ is the longitudinal relaxation time, and $\beta$ is a parameter between zero and one that
describes the distribution of relaxation times in a large ensemble of NV$^-$ centers with similar but not identical
relaxation times. A value of $\beta=1$ indicates a $\delta$-function distribution of relaxation times~\cite{Johnston:2006}.

\begin{figure}
\centering
\resizebox{0.9\columnwidth}{!}{\includegraphics{t1_experiment_v2}}
      \caption{\label{fig:exp}Schematic diagram of the experiment.}
\end{figure}

\begin{figure}
\centering
\resizebox{0.7\columnwidth}{!}{\includegraphics{ODMR_v4}}
      \caption{\label{fig:odmr}Fluorescence vs. MW frequency at $B=30$ G.}
\end{figure}

In order to compare the NV concentrations of the different spots, we measured
their fluorescence with a fluorescence microscope (see Fig.~\ref{fig:luminosity}).
The relationship between the fluorescence intensity and NV$^-$ concentration was determined using a diamond
sample with a known, uniform NV$^-$ concentration of 10 ppm in the same setup. Assuming that
the fluorescence intensity is proportional to the NV$^-$ concentration, the relative
concentration of our spots should be relatively well known (see Fig.~\ref{fig:concentration}).
However, the overall normalization should be
considered to be only an order-of-magnitude estimate. Our estimated NV$^-$ concentrations are systematically
lower than concentrations quoted for similar electron doses~\cite{Acosta:2010,Jarmola:2012,Mrozek:2015}; however,
these experiments used higher electron energies.

\begin{figure}
  \centering
      \resizebox{0.9\columnwidth}{!}{\includegraphics{Concentration_dose_v3}}
      \caption{\label{fig:concentration} Estimated NV$^-$ concentration vs electron dose. Error bars are not
      given as the main error is systematic and could be a factor of five.}
\end{figure}

The Hamiltonian $H_S$ of the $^3A_2$ state (see Fig.~\ref{fig:levels}) of the electronic spin of
the NV$^-$ center is given by
$H_S= DS_z^2 + E(S_x^2-S_y^2)+g_S \mu_B \vec{B}\cdot\vec{S}$,
where $DS_z^2$ corresponds to the zero-field splitting,  $E(S_x^2-S_y^2)$ corresponds to electric fields, which
can arise in the lattice due to strain, and $g_S\mu_B\vec{B}\cdot\vec{S}$ corresponds to the Zeeman splitting.
The strain parameter $2E$ can be observed as a small splitting in the zero-field ODMR signal. We have estimated this
parameter for each spot by fitting parabolas to the peaks,
and it is plotted in Fig.~\ref{fig:strain} as a function of estimated concentration.
The radiation damage causes distortions in the crystal structure, as shown by the fact that the strain
splitting increases with concentration, though more slowly above $~3$~ppm. The distortion could also change
the distance between the N and V defects, which would alter the zero-field splitting $D$.
Whereas an earlier study reported an increase of the zero-field
splitting $D$ on the order of 20 MHz at high radiation doses~\cite{Kim:2012}, our measurements show no change in $D$.

\begin{figure}
    \centering
      \resizebox{0.9\columnwidth}{!}{\includegraphics{EDplotnew}}
      \caption{\label{fig:strain} Strain splitting ($2E$) and zero-field splitting ($D$) as a function of
      NV$^-$ concentration.}
\end{figure}

Figure~\ref{fig:rates} shows the measured longitudinal relaxation rates, obtained by fitting the decay curves from our
experiments, as a function of magnetic field for various spots.
The rates are comparable to previously measured rates in bulk samples~\cite{Jarmola:2012,Mrozek:2015}.
The increase in $1/T_1$ at zero magnetic field is caused by the fact that all ODMR components are degenerate there\cite{Mrozek:2015}.
The effect is analogous to level-crossing resonances.
It is also possible to see a hint of the relaxation rates dropping and increasing again as the magnetic field value increases
from zero, which is related to partial overlapping of the ODMR components at lower magnetic field values\cite{Mrozek:2015}.
More measurements near zero field would be desirable, but we could not resolve the $\pm1$ ODMR peaks at lower fields.

Figure~\ref{fig:beta_t1} shows that in our fits the value of $\beta$ approaches unity as the NV$^-$ concentration decreases.
This result is consistent with the measurements in Fig. 2(a) of~\cite{Jarmola:2012}, which showed that at low temperatures,
the NV$^-$ density, i.e., interaction with nearby spins, dominated the $1/T_1$ rate,
wherease at higher temperatures, relaxation induced by phonons dominated.
Similarly, the nearly linear relationship between longitudinal relaxation rate and NV concentration
is consistent with dipole-dipole interactions driving relaxation at these densities,
since the dipole field drops off as $r^{-3}$ whereas the average distance to the nearest NV center changes as $\rho^{-1/3}$.

\begin{figure}
  \centering
    \resizebox{\columnwidth}{!}{\includegraphics{Allplotsv270}}
      \caption{\label{fig:rates}Longitudinal relaxation rates as a function
      of magnetic field for different spots.}
\end{figure}

\begin{figure}
  \centering
    \resizebox{0.9\columnwidth}{!}{\includegraphics{t1_beta_inone_56_46-cropped}}
      \caption{\label{fig:beta_t1} $1/T_1$ (filled circles) and the
      parameter $\beta$ (open circles) as a function NV$^-$ concentration at $B=30$~G.}
\end{figure}

In conclusion, we have made a systematic measurement of longitudinal relaxation times as a function of magnetic field and
and NV$^-$ concentration for a type Ib diamond.
The results suggest that at high concentrations the NV$^-$ centers in the spots do not have a narrowly defined relaxation rate,
but rather some distribution of relaxation rates.
However, as the concentration decreases, the relaxation rate becomes
better defined, as indicated by the stretched-exponential fits of the polarization-decay curves
approaching unity.
Another interesting feature is the zero-field resonance in relaxation rate when plotted against magnetic
field (Fig.~\ref{fig:rates})~\cite{Jarmola:2012,Mrozek:2015}.
Finally, the longitudinal relaxation rate was measured for values of NV$^-$ concentration spanning an order of magnitude,
and the value of the relaxation rate increased by almost a factor of five over this range.
These measurements could help to guide the preparation of microscale NV$^-$ sensors on diamond using a TEM.

This work was supported by ESF Project Nr. 2013/0028/1DP/1.1.1.2.0/13/APIA/VIAA/054.
We thank Raimonds Poplausks and Kaspars Vai\v{c}ekonis for help with the experiments.
D.B. acknowledges support by DFG through the DIP program (FO 703/2-1) and by the
AFOSR/DARPA QuASAR program.

Quantum Physics (quant-ph); Mesoscale and Nanoscale Physics (cond-mat.mes-hall); Materials Science (cond-mat.mtrl-sci)


SSD-KD: A Self-supervised Diverse Knowledge Distillation Method for Lightweight Skin Lesion Classification Using Dermoscopic Images

Abstract: Skin cancer is one of the most common types of malignancy, affecting a large population and causing a heavy economic burden worldwide. Over the last few years, computer-aided diagnosis has been rapidly developed and make great progress in healthcare and medical practices due to the advances in artificial intelligence. However, most studies in skin cancer detection keep pursuing high prediction accuracies without considering the limitation of computing resources on portable devices. In this case, knowledge distillation (KD) has been proven as an efficient tool to help improve the adaptability of lightweight models under limited resources, meanwhile keeping a high-level representation capability. To bridge the gap, this study specifically proposes a novel method, termed SSD-KD, that unifies diverse knowledge into a generic KD framework for skin diseases classification. Our method models an intra-instance relational feature representation and integrates it with existing KD research. A dual relational knowledge distillation architecture is self-supervisedly trained while the weighted softened outputs are also exploited to enable the student model to capture richer knowledge from the teacher model. To demonstrate the effectiveness of our method, we conduct experiments on ISIC 2019, a large-scale open-accessed benchmark of skin diseases dermoscopic images. Experiments show that our distilled lightweight model can achieve an accuracy as high as 85% for the classification tasks of 8 different skin diseases with minimal parameters and computing requirements. Ablation studies confirm the effectiveness of our intra- and inter-instance relational knowledge integration strategy. Compared with state-of-the-art knowledge distillation techniques, the proposed method demonstrates improved performances for multi-diseases classification on the large-scale dermoscopy database.

\section{Introduction}
This bundle provides two classfiles, namely \verb+cas-sc.cls+ and
\verb+cas-dc.cls+  and corresponding template files for typesetting
journal articles supposed to go through Elsevier's updated workflow.
\verb+cas-sc.cls+ is meant for one-column, the other
\verb+cas-dc.cls+ for two-column layout. These are now accepted for
submitting articles both in Elsevier's electronic submission system and
elsewhere.

\subsection{Usage}
\begin{enumerate}
\item \verb+cas-sc.cls+ for single column journals.

\begin{vquote}
 \documentclass[<options>]{cas-sc}
\end{vquote}
\item \verb+cas-dc.cls+ for single column journals.

\begin{vquote}
 \documentclass[<options>]{cas-dc}
\end{vquote}
\end{enumerate}
and have an option \verb+longmktitle+ to handle long front matter.

\section{Front matter}

\begin{vquote}
\title [mode = title]{This is a specimen $a_b$ title}
\tnotemark[1,2]

\tnotetext[1]{This document is the results of the research
   project funded by the National Science Foundation.}

\tnotetext[2]{The second title footnote which is a longer text
   matter to fill through the whole text width and overflow
   into another line in the footnotes area of the first page.}

\end{vquote}

\begin{vquote}

\author[1,3]{J.K. Krishnan}[type=editor,
                        auid=000,bioid=1,
                        prefix=Sir,
                        role=Researcher,
                        orcid=0000-0001-0000-0000]
\cormark[1]
\fnmark[1]
\ead{jkk@example.in}
\ead[url]{www.jkkrishnan.in}

\credit{Conceptualization of this study,
          Methodology, Software}

\affiliation[1]{organization={Department of Physics,
                  J.K. Institute of Science},
                addressline={Jawahar Nagar},
                city={Trivandrum},
                postcode={695013},
                state={Kerala},
                country={India}}

\author[2,4]{Han Thane}[style=chinese]

\author[2,3]{William {J. Hansen}}[%
   role=Co-ordinator,
   suffix=Jr,
   ]
\fnmark[2]
\ead{wjh@example.org}
\ead[URL]{https://www.university.org}

\credit{Data curation, Writing - Original draft preparation}

\end{vquote}

\begin{vquote}

\affiliation[2]{organization={World Scientific University},
                addressline={Street 29},
                postcode={1011 NX},
                postcodesep={},
                city={Amsterdam},
                country={The Netherlands}}

\author[1,3]{T. Rafeeq}
\cormark[2]
\fnmark[1,3]
\ead{t.rafeeq@example.in}
\ead[URL]{www.campus.in}

\affiliation[3]{organization={University of Intelligent
                    Studies},
                addressline={Street 15},
                city={Jabaldesh},
                postcode={825001},
                state={Orissa},
                country={India}}

\cortext[cor1]{Corresponding author}
\cortext[cor2]{Principal corresponding author}
\fntext[fn1]{This is the first author footnote, but is common
   to third author as well.}
\fntext[fn2]{Another author footnote, this is a very long
  footnote and it should be a really long footnote. But this
  footnote is not yet sufficiently long enough to make two
  lines of footnote text.}

\nonumnote{This note has no numbers. In this work we
  demonstrate $a_b$ the formation Y\_1 of a new type of
  polariton on the interface between a cuprous oxide slab
  and a polystyrene micro-sphere placed on the slab.
  }
\end{vquote}

\begin{vquote}

\begin{abstract}[S U M M A R Y]
This template helps you to create a properly formatted
 \LaTeX\ manuscript.

\begin{abstract} ... \end{abstract} and \begin{keyword}
... \end{keyword} which contain the abstract and keywords
respectively.  Each keyword shall be separated by
a \sep command.
\end{abstract}

\begin{keywords}
quadrupole exciton \sep polariton \sep WGM \sep BEC
\end{keywords}

\maketitle
\end{vquote}

\subsection{Title}

\verb+\title+ command have the below options:
\begin{enumerate}
\item \verb+title:+ Document title
\item \verb+alt:+ Alternate title
\item \verb+sub:+ Sub title
\item \verb+trans:+ Translated title
\item \verb+transsub:+ Translated sub title
\end{enumerate}

\begin{vquote}
 \title[mode=title]{This is a title}
 \title[mode=alt]{This is a alternate title}
 \title[mode=sub]{This is a sub title}
 \title[mode=trans]{This is a translated title}
 \title[mode=transsub]{This is a translated sub title}
\end{vquote}

\subsection{Author}
\verb+\author+ command have the below options:

\begin{enumerate}
\item \verb+auid:+ Author id
\item \verb+bioid:+ Biography id
\item \verb+alt:+ Alternate author
\item \verb+style:+ Style of author name, eg.\ chinese
\item \verb+prefix:+ Prefix, eg.\ Sir
\item \verb+suffix:+ Suffix
\item \verb+degree:+ Degree
\item \verb+role:+ Role
\item \verb+orcid:+ ORCID
\item \verb+collab:+ Collaboration
\item \verb+anon:+ Anonymous author
\item \verb+deceased:+ Deceased author
\item \verb+twitter:+ Twitter account
\item \verb+facebook:+ Facebook account
\item \verb+linkedin:+ LinkedIn account
\item \verb+plus:+ Google plus account
\item \verb+gplus:+ Google plus account
\end{enumerate}

\begin{vquote}
\author[1,3]{Author Name}[type=editor,
    auid=000,bioid=1,
    prefix=Sir,
    role=Researcher,
    orcid=0000-0001-0000-0000,
    facebook=<facebook id>,
    twitter=<twitter id>,
    linkedin=<linkedin id>,
    gplus=<gplus id>]
\end{vquote}

\begin{figure}
\includegraphics[width=\textwidth]{sc-sample.pdf}
\caption{Single column output (classfile: cas-sc.cls).}
\end{figure}

\begin{figure}
\includegraphics[width=\textwidth]{dc-sample.pdf}
\caption{Double column output (classfile: cas-dc.cls).}
\end{figure}

\subsection{Various Marks in the Front Matter}

The front matter becomes complicated due to various kinds
of notes and marks to the title and author names. Marks in
the title will be denoted by a star ($\star$) mark;
footnotes are denoted by super scripted Arabic numerals,
corresponding author by an Conformal asterisk (*) mark.

\subsubsection{Title marks}

Title mark can be entered by the command, \verb+\tnotemark[<num>]+
and the corresponding text can be entered with the command
\verb+\tnotetext[<num>]+ \verb+{<text>}+. An example will be:

\begin{vquote}
\title[mode=title]{Leveraging social media news to predict
                      stock index movement using RNN-boost}

\tnotemark[1,2]

\tnotetext[1]{This document is the results of the research
   project funded by the National Science Foundation.}

\tnotetext[2]{The second title footnote which is a longer
   text matter to fill through the whole text width and
   overflow into another line in the footnotes area of
   the first page.}
\end{vquote}

\verb+\tnotemark+ and \verb+\tnotetext+ can be anywhere in
the front matter, but should be before \verb+\maketitle+ command.

\subsubsection{Author marks}

Author names can have many kinds of marks and notes:

\begin{vquote}
    footnote mark : \fnmark[<num>]
    footnote text : \fntext[<num>]{<text>}
    affiliation mark : \author[<num>]
    email : \ead{<emailid>}
    url : \ead[url]{<url>}
    corresponding author mark : \cormark[<num>]
    corresponding author text : \cortext[<num>]{<text>}
\end{vquote}

\subsubsection{Other marks}

At times, authors want footnotes which leave no marks in
the author names. The note text shall be listed as part of
the front matter notes. Class files provides
\verb+\nonumnote+ for this purpose. The usage

\begin{vquote}
\nonumnote{<text>}
\end{vquote}

\noindent and should be entered anywhere before the \verb+\maketitle+
command for this to take effect.

\subsection{Abstract and Keywords}

Abstract shall be entered in an environment that starts
with\break \verb+\begin{abstract}+ and ends with
\verb+\end{abstract}+. Longer abstracts spanning more than
one page is also possible in slass file even in double
column mode. We need to invoke \verb+longmktitle+ option in the
class loading line for this to happen smoothly.

The key words are enclosed in a \verb+{keyword}+
environment.

\begin{vquote}
\begin{abstract}
 This is an abstract. \lipsum[3]
\end{abstract}

\begin{keywords}
 First keyword \sep Second keyword \sep Third
    keyword \sep Fourth keyword
\end{keywords}
\end{vquote}

\section{Main Matter}
Main matter contains sections, paragraphs, equations and floats like
tables, figures, textboxes etc.
\subsection{Tables}
\subsubsection{Normal tables}

\begin{vquote}
\begin{table}
  \caption{This is a test caption.}
  \begin{tabular*}{\tblwidth}{@{} LLLL@{} }
   \toprule
    Col 1 & Col 2\\
   \midrule
    12345 & 12345\\
    12345 & 12345\\
    12345 & 12345\\
    12345 & 12345\\
    12345 & 12345\\
    12345 & 12345\\
   \bottomrule
  \end{tabular*}
\end{table}
\end{vquote}

\subsubsection{Span tables}

\begin{vquote}
\begin{table*}[width=.9\textwidth,cols=4,pos=h]
  \caption{This is a test caption.}
  \begin{tabular*}{\tblwidth}{@{} LLLLLL@{} }
   \toprule
    Col 1 & Col 2 & Col 3 & Col4 & Col5 & Col6 & Col7\\
   \midrule
    12345 & 12345 & 123 & 12345 & 123 & 12345 & 123 \\
    12345 & 12345 & 123 & 12345 & 123 & 12345 & 123 \\
    12345 & 12345 & 123 & 12345 & 123 & 12345 & 123 \\
    12345 & 12345 & 123 & 12345 & 123 & 12345 & 123 \\
    12345 & 12345 & 123 & 12345 & 123 & 12345 & 123 \\
    12345 & 12345 & 123 & 12345 & 123 & 12345 & 123 \\
    12345 & 12345 & 123 & 12345 & 123 & 12345 & 123 \\
   \bottomrule
  \end{tabular*}
\end{table*}
\end{vquote}

\subsection{Figures}
\subsubsection{Normal figures}
\begin{vquote}
\begin{figure}
	\centering
		\includegraphics[scale=.75]{Fig1.pdf}
	\caption{The evanescent light - $1S$ quadrupole coupling.
   See also Fig. \protect\ref{FIG:2}).}
	\label{FIG:1}
\end{figure}
\end{vquote}

\subsubsection{Span figures}

\begin{vquote}
\begin{figure*}
	\centering
	  \includegraphics[width=\textwidth,height=2in]{Fig2.pdf}
	\caption{Schematic of formation of the evanescent polariton on
	linear chain of \PMS.}
  \label{FIG:2}
\end{figure*}\end{vquote}

\subsection{Theorem and theorem like environments}

CAS class file provides a few hooks to format theorems and
theorem like environments with ease. All commands the
options that are used with \verb+\newtheorem+ command will work
exactly in the same manner. Class file provides three
commands to format theorem or theorem like environments:

\begin{enumerate}
\item \verb+\newtheorem+ command formats a theorem in
\LaTeX's default style with italicized font for theorem
statement, bold weight for theorem heading and theorem
number typeset at the right of theorem heading. It also
optionally accepts an argument which will be printed as an
extra heading in parentheses. Here is an example coding and
output:

\begin{vquote}
\newtheorem{theorem}{Theorem}
\begin{theorem}\label{thm}
 The \WGM evanescent field penetration depth into the
 cuprous oxide adjacent crystal is much larger than the
 \QE radius:
 \begin{equation*}
  \lambda_{1S}/2 \pi \left({\epsilon_{Cu2O}-1}
    \right)^{1/2} = 414 \mbox{ \AA} \gg a_B = 4.6
    \mbox{ \AA}
 \end{equation*}
\end{theorem}
\end{vquote}

\item \verb+\newdefinition+ command does exactly the same
thing as with except that the body font is up-shape instead
of italic. See the example below:

\begin{vquote}
\newdefinition{definition}{Definition}
\begin{definition}
 The bulk and evanescent polaritons in cuprous oxide
 are formed through the quadrupole part of the light-matter
 interaction:
 \begin{equation*}
  H_{int} = \frac{i e }{m \omega_{1S}} {\bf E}_{i,s}
    \cdot {\bf p}
 \end{equation*}
\end{definition}
\end{vquote}

\item \verb+\newproof+ command helps to define proof and
custom proof environments without counters as provided in
the example code. Given below is an example of proof of
theorem kind.

\begin{vquote}
\newproof{pot}{Proof of Theorem \ref{thm}}
\begin{pot}
 The photon part of the polariton trapped inside the \PMS
 moves as it would move in a micro-cavity of the effective
 modal volume $V \ll 4 \pi r_{0}^{3} /3$. Consequently, it
 can escape through the evanescent field. This evanescent
 field essentially has a quantum origin and is due to
 tunneling through the potential caused by dielectric
 mismatch on the \PMS surface. Therefore, we define the
 \emph{evanescent} polariton (\EP) as an evanescent light -
 \QE coherent superposition.
\end{pot}
\end{vquote}

\end{enumerate}

\subsection{Enumerated and Itemized Lists}

CAS class files provides an extended list processing macros
which makes the usage a bit more user friendly than the
default LaTeX list macros. With an optional argument to the
\verb+\begin{enumerate}+ command, you can change the list
counter type and its attributes. You can see the coding and
typeset copy.

\begin{vquote}
\begin{enumerate}[1.]
  \item The enumerate environment starts with an optional
        argument `1.' so that the item counter will be suffixed
        by a period as in the optional argument.
  \item If you provide a closing parenthesis to the number in the
        optional argument, the output will have closing
        parenthesis for all the item counters.
  \item You can use `(a)' for alphabetical counter and `(i)' for
        roman counter.
  \begin{enumerate}[a)]
    \item Another level of list with alphabetical counter.
    \item One more item before we start another.
    \begin{enumerate}[(i)]
      \item This item has roman numeral counter.
\end{vquote}

\begin{vquote}
      \item Another one before we close the third level.
    \end{enumerate}
    \item Third item in second level.
  \end{enumerate}
  \item All list items conclude with this step.
\end{enumerate}

\section{Biography}

\verb+\bio+ command have the below options:
\begin{enumerate}
 \item \verb+width:+ Width of the author photo (default is 1in).
 \item \verb+pos:+ Position of author photo.
\end{enumerate}

\begin{vquote}
\bio[width=10mm,pos=l]{tuglogo.jpg}
 \textbf{Another Biography:}
  Recent experimental \cite{HARA:2005} and theoretical
  \cite{DEYCH:2006} studies have shown that the \WGM can travel
  along the chain as "heavy photons". Therefore the \WGM
  acquires the spatial dispersion, and the evanescent
  quadrupole polariton has the form (See Fig.\ref{FIG:3}):
\endbio
\end{vquote}

\section[CRediT...]{CRediT authorship contribution statement}

Give the authorship contribution after each author as

\begin{vquote}
 \credit{Conceptualization of this study, Methodology,
         Software}
\end{vquote}

To print the details use \verb+\printcredits+

\begin{vquote}
\author[1,3]{J.K. Krishnan}[type=editor,
                        auid=000,bioid=1,
                        prefix=Sir,
                        role=Researcher,
                        orcid=0000-0001-0000-0000]
\end{vquote}

\begin{vquote}
\cormark[1]
\fnmark[1]
\ead{jkk@example.in}
\ead[url]{www.jkkrishnan.in}

\credit{Conceptualization of this study, Methodology, Software}

\affiliation[1]{organization={Department of Physics,
                      J.K. Institute of Science},
                addressline={Jawahar Nagar},
                city={Trivandrum},
                postcode={695013},
                state={Kerala},
                country={India}}

\author[2,4]{Han Thane}[style=chinese]

\author[2,3]{William {J. Hansen}}[%
   role=Co-ordinator,
   suffix=Jr,
   ]
\fnmark[2]
\ead{wjh@example.org}
\ead[URL]{https://www.university.org}

\credit{Data curation, Writing - Original draft preparation}

  . . .
  . . .
  . . .
  \printcredits
\end{vquote}

\section{Bibliography}

For CAS categories, two reference models are recommended.
They are \file{model1-num-names.bst} and \file{cas-model2-names.bst}.
Former will format the reference list and their citations according to
numbered scheme whereas the latter will format according name-date or
author-year style. Authors are requested to choose any one of these
according to the journal style. You may download these from

The above bsts are available in the following location for you to
download:

\url{https://support.stmdocs.in/wiki/index.php?title=Model-wise_bibliographic_style_files}
\hfill $\Box$

\end{document}

\section{Introduction}\label{intro}

Skin cancer is among the most common human malignancies globally, especially among the fair-skinned population. Over the past decades, skin cancer has pervaded different cultures and caused a huge economic burden in the healthcare maintenance. Meanwhile, both malignant melanoma (MM), which causes most deaths in skin cancers, and keratinocyte carcinomas (squamous cell carcinoma and basal cell carcinoma) have a promising prognosis if they are detected and well treated at the early stage \citep{apalla2017epidemiological}. Therefore, early diagnosis plays an essential role in skin cancer’s efficient management and successful treatment.

Benefiting from the unprecedented advances in optical imaging techniques, huge amounts of high-quality skin images were collected in multiple modalities. Dermoscopy, a noninvasive digital imaging technique, has been widely used for skin cancer diagnosis since it allows the in vivo evaluation of colors and structures of the epidermis that are hard to be visualized by the naked eye \citep{celebi2019dermoscopy,barata2018survey}. In traditional skin cancer diagnosis, expert dermatologists can inspect the skin lesions following valid algorithms such as ABCD rules \citep{nachbar1994abcd}, 7-point Checklist \citep{argenziano1998epiluminescence} using dermoscopy images. However, such time consuming and expensive diagnostic methods cause a great burden on clinical diagnosis. Therefore, automatic skin cancer detection is in urgent need to relieve dermatologists’ heavy work pressure in clinical diagnosis.

Convolutional neural networks (CNN) have been widely used in computer-aided diagnosis (CAD) for skin cancer detection and achieved remarkable performance, because of its excellent and robust capability in feature extraction and categorical classification \citep{esteva2017dermatologist,litjens2017survey,brinker2018skin}. In recent years, more and more research efforts have been made to develop advanced deep CNN architectures to improve their diagnostic performance for skin cancer detection. Esteva et al. \citep{esteva2017dermatologist} involved the pre-trained Inception V3 models in skin lesions’ classification and presented dermatologist-leveled prediction results for the first time. Abbas et al. \citep{abbas2019dermodeep} proposed the DermoDeep framework for the classification of melanoma and nevus that consisted visual-features fusion and deep learning without pre- and post-processing steps.  Kawahara et al. \citep{kawahara2016deep} found that the features extracted from the ImageNet pre-trained model outperformed general handcrafted features in skin lesion classification tasks. With the further development of deep learning technology, more novel structures have also been proposed and being used in this task. The integration strategies of various deep learning models and the interpretive analysis of the predictions they produce have also been extensively studied. For example, Gessert et al. \citep{gessert2020skin} proposed an ensemble framework for skin lesion classification using multi-resolution Efficient Net architectures with a search strategy. Barata et al. \citep{barata2021explainable} came up with an explainable framework that can provide insightful information about deep learning models’ decisions by incorporating the taxonomies information and special attention blocks. Meanwhile, the exploration of the combined effect of multiple modalities and the introduction of clinical diagnostic knowledge have further promoted the development of deep learning in the diagnosis of skin diseases. Kawahara et al. \citep{kawahara2018seven} proposed a multi-task deep convolutional neural network containing clinical, dermoscopic images and patient metadata. Their approach incorporated clinical rule-based knowledge into deep learning and demonstrated benchmark results for 7-point criteria and diagnostics. Based on this study, Bi et al. \citep{bi2020multi} further considered the feature information that multi-modal input can share in the training phase of deep learning architecture and achieved further improvement in melanoma detection. Pacheco et al. \citep{pacheco2021attention} proposed a MetaBlock module to improve classifiers' performance by utilizing the clinical metadata. Wang et al. \citep{wang2021incorporating} proposed a novel method that introduced the inter-dependencies between different criteria using a constrained classifier chain, which made the resulting predictions more acceptable to physicians. Tang et al. \citep{tang2022fusionm4net} came up with a new algorithm that fused both feature-level and decision-level information from multiple modalities and achieved state-of-the-art performance in skin cancer diagnosis.

These remarkable achievements using varieties of CNN architectures have a great impact on the progress of automated skin cancer diagnosis. However, these research works related to CNNs also indicate that sophisticated models with more neural network layers and blocks tend to have better predictive performance with sufficient training data and proper training procedures. These works further suggest that, CNNs necessitate a huge number of parameters and massive floating point operations to obtain a satisfactory performance. For example, ResNet50 \citep{he2016deep}, as one of the most well-known and effective deep learning  models, has about 25.6 millions of parameters, which requires 98 MB and 4.11 GFlops to process the algorithm. Although these complex and powerful deep learning models can allow us to obtain better prediction results, they also require significant computing resources (e.g., memories and hardware resources). These strict requirements may impede their practicability in restrained environments and the embedded devices that have limited computation power and memories. Therefore, exploring portable and efficient networks with comparable performance for skin cancer auto-detection that can work on embedded equipment or limited resources has both practical and theoretical significance.

Knowledge Distillation (KD), proposed as an extended model compression method, can transfer the learned knowledge from a complicated model (the teacher) to a simple model (the student) by sharing the soft labels during the training phase, thus improving the representation power of the lightweight CNNs preferably \citep{hinton2015distilling, tung2019similarity}. In recent years, many novel and practical deep learning models based on KD ideas have been proposed. Romero et al. \citep{romero2015fitnets} proposed the FitNet, an approach that added an additional requirement of predicting the outputs of the intermediate layers to the traditional KD algorithm and made it suitable for thin and deep networks. Yim et al. \citep{yim2017gift} measured a flow of feature correlations among adjacent layers in residual blocks and transferred them to a student network, then Liu et al. \citep{liu2019knowledge} extended it to a dense cross-layer transfer scenario. Park et al. \citep{park2019relational} built a dubbed relational KD deep learning framework that considered both distance-wise and angle-wise distillation losses and improved the prediction ability of student models significantly. Peng et al. \citep{peng2019correlation} proposed a correlation congruence distillation scheme to model the relation between samples in the kernel form. Xu et al. \citep{xu2020knowledge} further found that self-supervision signals can effectively transfer the hidden information from the teacher to the student via their proposed SSKD architecture, which substantially benefited the scenarios with few-shot and noisy-labels. Ding et al. \citep{ding2021distilling} proposed an data augmentation-based  knowledge distillation framework for classification and regression tasks via synthetic samples \citep{ding2020ccgan,ding2021efficient}.

Meanwhile, KD has been proven effective in the compression of deep learning network structures on the basis of ensuring model prediction and learning capabilities among varieties of areas \citep{gou2021knowledge,wang2021knowledge}. In particular, KD shows great potential for applications in computer-aided diagnostics. For example, Qin et al. \citep{qin2021efficient} proposed an efficient architecture by distilling knowledge from well-trained medical image segmentation networks to train a lightweight network and obtained significant improvements for the segmentation capability while retaining the runtime efficiency. Abbasi et al. \citep{abbasi2021classification} implemented knowledge distillation on unlabeled medical data via an unsupervised learning manner and showed a significant improvement in the prediction of diabetic retinopathy. Chen et al. \citep{chen2022lightweight} compared multiple deep learning models’ performance regarding cervical cells classification using knowledge distillation and showed the importance of model selection.  At the same time, knowledge distillation methods have also shown great potential in the field of skin disease diagnosis although there still exists quite limited attention in this area. Back et al. \citep{back2021robust} used an ensemble learning strategy to help a student network learn from multiple teacher networks progressively in the portable diagnosis of herpes zoster skin disease using clinical images, and achieved robust performances. To our best knowledge, however, there are still no studies on the classification of multiple skin diseases based on large-scale dermocopy image datasets.

\begin{figure*}
\centering
\includegraphics[width=1\textwidth]{./figures/flowchart.png}
\caption{\label{fig:flowchart}An illustration of the proposed knowledge distillation framework that distills and transfers diverse knowledge to the student model for skin cancer diagnosis. The upper branch (in green) denotes the pretrained teacher classification model, and the lower branch (in blue) denotes the untrained student classification model. Both models consist of a set of convolutional blocks to extract feature maps of a batch of skin images, followed by average pooling operations to produce embeddings, and a fully connected (FC) layer to yield logits for skin disease classification. The three types of knowledge and corresponding loss functions are quantified based on channel feature maps, embeddings and logits, respectively. In particular, our novel channel relational distillation loss is defined in the feature map stream. Finally, the diverse knowledge-integrated distillation loss is computed and backpropagated (along dashed reverse arrow lines) to train the student network. The self-supervised learning strategy can be conveniently incorporated to further boost the training of the student model. }
\end{figure*}

Despite that these innovative studies have advanced the application of KD in the field of computer-aided diagnosis, they still mainly rely on the sharing and transferring the subject-level category information between the teacher network and the student network. In detail, existing skin diagnosis KDs have largely ignored the inter and intra relations within input samples, which may preserve important knowledge to diagnose skin diseases from the teacher network. This work explores the intra-instance relational knowledge, as well as its integrated analysis with traditional inter-instance relational knowledge and dark knowledge, to enhance the effectiveness of information transfer from the teacher network to the student network. Consequently, the proposed diverse knowledge-aware KD can effectively improve the predictive capabilities of the student model.

In this work, we develop a novel knowledge distillation framework, termed SSD-KD, that integrates diverse conventional knowledge with a novel type of intra-instance relational knowledge to improve the auto-diagnosis of multiple types of skin diseases based on lightweight deep learning models. Specifically, we formulate a dual relational knowledge distillation architecture introducing different relational representations and softened network outputs to distill and transfer diverse knowledge from the teacher model to student model. Moreover, we also incorporate the self-supervised knowledge distillation strategy into the framework to enable the student model capture richer structured knowledge from the self-supervision predictions of the teacher model. Additionally, to make the proposed SSD-KD framework more suitable for the skin lesions auto-detection task, which always suffers from data imbalance problems, we replace the general cross-entropy loss function in conventional KDs with an improved weighted version to benefit the skin disease categories with less subjects. We have evaluated our proposed method in the publicly available skin lesion datasets ISIC 2019, which contains over 20000 dermoscopy images and 8 types of skin diseases. The comparison with state-of-the-art methods and ablation studies are also performed to demonstrate the effectiveness of our proposed framework.

The main contributions of this study can be summarized as follows: \par

\begin{itemize}
     \item We propose a novel self-supervised diverse knowledge distillation method for lightweight multi-class skin diseases classification using dermoscopy images, named SSD-KD. We replace the traditional single relational modeling block with dual relational blocks in our method, which assists the student model capture both intra- and inter-instance relational information from the teacher model. Hence, our diverse knowledge-integrated KD can further improve the transferable ability of KD and the performance for skin disease classification.
     \item We employ the self-supervision based auxiliary learning strategy in our knowledge distillation framework, which enables richer structured knowledge to be transferred, and stabilizes the training dynamics of the intra- and inter-instance relational distillation scheme.
     \item We utilize the weighted cross entropy loss function and combine it with our SSD-KD structure to assist the lightweight student model better mimic the behavior of the teacher model in capturing additional information from the diseases with limited subjects.
     \item The results demonstrate that our carefully designed method can help the student model achieve comparable or even better performance than the teacher model in the multi-class skin lesion classification task. From a practical clinical point of view, our proposed method can further advance the application of remarkable deep learning models in scenarios with limited computing resources.
\end{itemize}\par
The rest of the paper is organized as follows: the detailed introduction of the proposed method is presented in the second section, followed by the experiment setup and results analysis in the third section. The discussion and the future works are shown in the fourth section while the conclusion of the study is put in the last section.

\section{Methodology}

In this section, we describe the proposed method that aims to develop an effective knowledge distillation framework for skin disease detection. The overall pipeline of the proposed method is shown in Figure~\ref{fig:flowchart}. In the proposed framework, we distill and transfer three types of knowledge from a large and pretrained teacher model to a portable student model that will be trained from scratch. The first type of knowledge adopts a customized version of the conventional knowledge distillation scheme \citep{hinton2015distilling} that defines the ``dark knowledg'' via logits (i.e., the output vector from the last layer of a CNN model). The second type of knowledge utilizes the inter-instance relational knowledge \citep{park2019relational} to model the relationship of a batch of training samples at the penultimate layer.  In addition, we define the intra-instance relational knowledge between channel feature maps of each sample at the last convolutional layer.  This type of knowledge further exploits the inherent structures within model responses to facilitate the training of the student model.

For convenience, we introduce some notations to be used in the following section. Let us denote the overall training set of skin images as $\mathcal{X}$ with groundtruth labels as $\mathcal{Y}$. Following the general routine in deep learning, we split the training set $(\mathcal{X}, \mathcal{Y})$ into non-overlapping subsets randomly. The number of skin images (i.e., batch size) in each subset is denoted as $B$, i.e., $\{ (\boldsymbol{x}_1, y_1), \cdots,  (\boldsymbol{x}_B, y_B) \} \subset \mathcal{X}$. For an input sample $\boldsymbol{x}_i \ (i=1,\cdots, B)$, we denote its logits vector by $\boldsymbol{l}(\boldsymbol{x}_i)$, the embedding vector by $\boldsymbol{e}_i$ (i.e., $\boldsymbol{e} (\boldsymbol{x}_i)$ for brevity), and channel feature maps by $\boldsymbol{f}(\boldsymbol{x}_i)$, respectively. Here $\boldsymbol{l}(\boldsymbol{x}_i) = [l_1(\boldsymbol{x}_i), \cdots, l_C (\boldsymbol{x}_i)  ]$, where $C$ represents the number of unique classes of skin images; $\boldsymbol{f}(\boldsymbol{x}_i) = [ f_1(\boldsymbol{x}_i), \cdots,  f_K(\boldsymbol{x}_i)] $ with $f_k(\boldsymbol{x}_i) \in \mathbb{R}^{H_f\times W_f} \ (k=1,\cdots, K)$, where $H_f, W_f$ denote the height and width of a feature map, respectively.

\subsection{Logit-based dark knowledge}

\cite{hinton2015distilling} proposed a logit-based knowledge distillation method (a.k.a BLKD). Essentially, BLKD utilizes the (softened) logits of a teacher model as informative knowledge, which is then transferred to assist the training of a student model.

BLKD includes two loss terms: 1) a term that matches the softened logits of a teacher model with those of a student one, and 2) a regular classification term that matches the predictions of a student model with the groundtruth label. For an image $\boldsymbol{x}_i$, denote its softened logits from a model as $\boldsymbol{p}(\boldsymbol{x}_i, T)$ at a temperature $T$, where $\boldsymbol{p}(\boldsymbol{x}_i, T) = [ p_1(\boldsymbol{x}_i, T), \cdots,  p_C(\boldsymbol{x}_i, T) ]$. Here $p_c(\boldsymbol{x}_i, T)$ denotes the soft probability that $\boldsymbol{x}_i$ belongs to a class $c$. With the softmax function, $p_c(\boldsymbol{x}_i, T)$ can be computed via the logit $l_c (\boldsymbol{x}_i)$,
\begin{equation}
    p_c(\boldsymbol{x}_i, T) = \frac{\textrm{exp}(l_c(\boldsymbol{x}_i) / T)}{\sum_{j=1}^C \textrm{exp} (l_j (\boldsymbol{x}_i) / T )}
    \label{eq:softmax}
\end{equation}
where a higher temperature $T$ provides softer probabilities for class predictions, and tends to indicate richer informative structures between classes. In BLKD, the loss of the first term measures the distribution differences of soft probabilities from a teacher (i.e., $\boldsymbol{p}^t(\boldsymbol{x}_i, T)$) and a student (i.e., $\boldsymbol{p}^s(\boldsymbol{x}_i, T)$) using the Kullback–Leibler (KL) divergence,

\begin{equation}
\begin{split}
        L_{KD} =  & \sum_{i=1}^B D_{KL} ( \boldsymbol{p}^t(\boldsymbol{x}_i , T) \ || \  \boldsymbol{p}^s(\boldsymbol{x}_i, T) ) \\
        = & \sum_{i=1}^B \sum_{c=1}^C p_c^t(\boldsymbol{x}_i, T) \ \textrm{log} \ \frac{p_c^t(\boldsymbol{x}_i, T)}{p_c^s(\boldsymbol{x}_i, T)}
\end{split}
\label{eq:kd_blkd}
\end{equation}
where $L_{KD}$ denotes the loss of the first term in BLKD that are computed on a batch of images.

The predictions of a model can be obtained by simply setting $T=1$ in Eq.~(\ref{eq:softmax}). Then, as the second loss term in BLKD, a cross entropy loss can be computed between model predictions and groundtruth labels. Unlike general computer vision tasks, however, the second loss term is not  well suited for our skin cancer diagnosis problems. This is mainly due to the data imbalance issue in skin disease application, namely, we may only have few skin images for some diseases. Therefore, in this case, we turn to the weighted cross entropy (WCE). Denote by $w_c$ the weight for a class $c$, where $w_c$ can be computed as the inverse of the frequency for a disease class $c$ over the training dataset. Denote also the one-hot encoded label of $\boldsymbol{x}_i$ as $\boldsymbol{y}(\boldsymbol{x}_i) = [y_c(\boldsymbol{x}_i), \cdots, y_C(\boldsymbol{x}_i)]$.  Then, the WCE loss is expressed as,
\begin{equation}
    L_{WCE} = \sum_{i=1}^B \sum_{c=1}^C - w_c \cdot y_c(\boldsymbol{x}_i) \ \textrm{log} \ p_c^s(\boldsymbol{x}_i, T=1)
\label{eq:wce}
\end{equation}

Then, the customized overall loss term of BLKD equals,
\begin{equation}
    L_{BLKD} = (1-\lambda_{kd}) \cdot L_{WCE} + \lambda_{kd} \cdot L_{KD}
\label{eq:blkd}
\end{equation}
where $\lambda_{kd}$ is a hyperparameter to balance the two loss terms in BLKD.

\subsection{Inter-instance relational knowledge}

An inter-instance relational knowledge was proposed to model the structural relations among a batch of training data \citep{park2019relational}, termed data relational knowledge distillation (DRKD). Though there exist different variants of inter-instance knowledge (e.g., a kernelized form in \citep{peng2019correlation}), this study selects to use the popular DRLK due to its effectiveness and computational efficiency. To deal with the data scarcity challenge in skin cancer diagnosis, we also utilize such knowledge to additionally supervise the training of the student model.

DRKD maps the embeddings of different instances (within a batch) into relational representations, then it minimizes the structural differences of such representations between the teacher and student models, such that the student can better mimic the teacher. Given instances $\{ \boldsymbol{x}_1, \cdots, \boldsymbol{x}_B \}$ and their embeddings denoted by $\{ \boldsymbol{e}_1, \cdots, \boldsymbol{e}_B \}$, \cite{park2019relational} defined two types of inter-instance relational knowledge: the distance-wise relation $\psi_d (\boldsymbol{e}_i, \boldsymbol{e}_j)$ between any two instances, and the angle-wise relation $\psi_a (\boldsymbol{e}_i, \boldsymbol{e}_j, \boldsymbol{e}_k)$ between any three instances. Here the potential functions $\psi_d (\boldsymbol{e}_i, \boldsymbol{e}_j)$ and $\psi_a (\boldsymbol{e}_i, \boldsymbol{e}_j, \boldsymbol{e}_k)$ are computed based on network embeddings $\boldsymbol{e}_i, \boldsymbol{e}_j, \boldsymbol{e}_k$ via the normalized Euclidean distance and angular distance, respectively. The loss term of DRKD is defined as,
\begin{equation}
\begin{split}
    L_{DRKD} &= \lambda_d \cdot \sum_{(\boldsymbol{x}_i, \boldsymbol{x}_j) \in \mathcal{X}_B^2} l(\psi_d^t(\boldsymbol{e}_i, \boldsymbol{e}_j), \psi_d^s(\boldsymbol{e}_i, \boldsymbol{e}_j)) \\
    &+ \lambda_a \cdot \sum_{(\boldsymbol{x}_i, \boldsymbol{x}_j, \boldsymbol{x}_k) \in \mathcal{X}_B^3} l(\psi_a^t(\boldsymbol{e}_i, \boldsymbol{e}_j, \boldsymbol{e}_k), \psi_a^s(\boldsymbol{e}_i, \boldsymbol{e}_j, \boldsymbol{e}_k))
\end{split}
\label{eq:drkd}
\end{equation}
where $\mathcal{X}_B$ represents a training subset comprised of a batch of instances, $l$ denotes the Huber function \citep{huber1992robust}; $\lambda_d, \lambda_a$ denote the weight for the distance-wise loss and the angle-wise loss \citep{park2019relational}, respectively.

\subsection{Intra-instance relational knowledge}

To further improve the performance of a student model, we propose to exploit the intra-instance relational knowledge using channel feature maps for each instance. We term this novel type of knowledge distillation method as channel relational knowledge distillation (CRKD). This work only extracts channel relations of feature maps at the last convolutional layer, since these feature maps contain more fine-grained and critical features for making final predictions. It is worth mentioning that the instance-level congruence term in \citep{peng2019correlation} actually refers to the KL divergence of logits between a teacher and a student (i.e., as in Eq.~\ref{eq:kd_blkd}), which is clearly different from our formulation.

As illustrated in Figure~\ref{fig:flowchart}, given an input image $\boldsymbol{x}_i$, a (teacher or student) model extracts high-level features $f_k(\boldsymbol{x}_i)$ for $k=1,\cdots, K$. Then, CRKD can be performed following two sequential steps: the channel adaptation step, and the intra-instance relation generation step. Since the number of channels of a student model may not be equal to that of a teacher model, it is infeasible to align the intra-instance relations from two models directly. Therefore, for the student model, we adopt an adaptation module to match its channel number with that of the teacher. Specifically, we utilize a $1 \times 1$ convolution \citep{lin2014network} that linearly projects the channel feature maps of the student model to a space that shares a same channel number with the teacher model \citep{romero2015fitnets}.

In the second step, we will model the relation between an arbitrary (vectorized) feature map pair of an instance $\boldsymbol{x}_i$ as $\big(\textrm{Vec}(f_k(\boldsymbol{x}_i)), \textrm{Vec}(f_{k'}(\boldsymbol{x}_i)) \big)$ utilizing the un-normalized cosine similarity metric,
\begin{equation}
    r\big(f_k(\boldsymbol{x}_i), f_{k'}(\boldsymbol{x}_i)\big) = \big<\textrm{Vec}(f_k(\boldsymbol{x}_i), \textrm{Vec}(f_{k'}(\boldsymbol{x}_i) \big>
\end{equation}
Denote the overall intra-instance relational matrix as $\boldsymbol{R} \in \mathbb{R}^{K\times K}$, then $\boldsymbol{R}$ can be generated with entries $\boldsymbol{R}_{k, k'}$ as,
\begin{equation}
   \boldsymbol{R}_{k, k'} (\boldsymbol{x}_i) = r\big(f_k(\boldsymbol{x}_i), f_{k'}(\boldsymbol{x}_i)\big), \ k, k' \in \{ 1, \cdots, K\}
\end{equation}
A more computationally efficient way is adopt the Grammian matrix \citep{johnson2016perceptual,ding2021towards} to generate this relational matrix. To encourage the student learn the intra-instance relations of the teacher, we define the CRKD loss term over a batch of samples,
\begin{equation}
    L_{CRKD} = \sum_{i=1}^B \frac{1}{K H_f W_f} \big|\big| \boldsymbol{R}^t (\boldsymbol{x}_i) - \boldsymbol{R}^s (\boldsymbol{x}_i) \big|\big|_F
\end{equation}
where $F$ denotes the Frobenius norm of matrices; $H_f, W_f$ refer to the height and width of a feature map of the teacher model, respectively.

The intra-instance relation formulation above appears to resemble the angle-wise relation in DRKD, but here we are considering a two-tuple relation scenario. Another major distinction is that we are modeling the relation between any two channel feature maps of one instance rather than the embeddings from different instances. Our CRKD also differs from \citep{yim2017gift} and \citep{liu2019knowledge} that aim to measure the correlations across different network layers which tend to involve considerably more computational costs and instability issues. Besides, without the channel adaptation module as in our method, they require a similar network architecture (e.g., from a same architecture family) for the teacher and student models. This strict requirement, however, restrains their applicability in our considered scenario to establish more portable skin lesion diagnosis models.

A physical interpretation of our intra-instance relational modeling is to extract the texture information of an input image $\boldsymbol{x}_i$ in the feature space. The extracted textures have been demonstrated crucial in computer vision tasks, such as image recognition \citep{geirhos2018imagenet} and visual tracking attacks \citep{ding2021towards}. Since textural features also contain important discriminative knowledge in the skin lesion diagnosis task, this type of relation and knowledge can provide auxiliary favorable cues to facilitate the training of our student model.

\subsection{Diverse knowledge integration}

To further boost the student training process, we integrate diverse levels of knowledge and transfer them to the student model. As depicted in Figure~\ref{fig:flowchart}, our diverse knowledge distillation (D-KD) loss can be expressed as,
\begin{equation}
    L_{D-KD} = \lambda_{blkd} \cdot L_{BLKD} + \lambda_{drkd} \cdot L_{DRKD} + \lambda_{crkd} \cdot L_{CRKD}
\label{eq:overall}
\end{equation}
where $\lambda_{blkd}, \lambda_{drkd}, \lambda_{crkd}$ denote the hyperparameter to balance the loss terms for BLKD, DRKD, and CRKD, respectively.

In addition, we can incorporate the self-supervised training strategy \citep{xu2020knowledge} into the diverse knowledge-integrated distillation loss $L_{D-KD}$ to additionally improve the training dynamics of the student model. Self-supervised learning (SSL) is an effective strategy to learn generic visual representations from unlabeled data for downstream tasks (e.g., image classification, semantic segmentation) in the computer vision field \citep{chen2020simple,he2020momentum,jing2020self}. Moreover, a very recent work \citep{liu2022self} shows that this learning strategy can well alleviate the data imbalance issue, which further validated our motivation to incorporate the self-supervision knowledge.

A typical SSL scheme is to employ a contrastive loss \citep{van2018representation} that maximizes the agreement in feature space of positive sample pairs generated from different augmentation transformations on a same image \citep{he2020momentum,jing2020self}. For the first time, Xu et al. \citep{xu2020knowledge} introduced SSL into knowledge distillation (SSKD), and conducted a thorough study on the effectiveness of this learning strategy in the KD setting. To perform the self-supervision, SSKD pre-trains a teacher model with the contrastive loss, then transfers the self-supervised predictions from the teacher to the student. Following \citep{xu2020knowledge}, we integrate the self-supervised training strategy into our D-KD framework. Then, the loss function for the self-supervised diverse knowledge distillation (SSD-KD) is,
\begin{equation}
    L_{SSD-KD} = L_{D-KD} + \lambda_{sskd} \cdot L_{SSKD}
\label{eq:ssd-kd}
\end{equation}
where $\lambda_{sskd}$ denotes the hyperparameter for the self-supervision regularizer.

In Algorithm ~\ref{alg}, we describe the details of the proposed method.

\begin{algorithm}[tb]
	\footnotesize
	\SetAlgoLined
	\KwData{A training dataset $(\mathcal{X}, \mathcal{Y})$, a pretrained teacher model, an untrained student model, batch size $B$, the maximum number of training epochs $N$, hyperparameters $\lambda_{blkd}, \lambda_{drkd}, \lambda_{crkd}, \lambda_{sskd}$, a minibatch stochastic gradient descent optimizer, an early stop epoch $K_{stop}$.}
	Initialize the student model using random initialization, set the early stopping flag as False\;
	Split $(\mathcal{X}, \mathcal{Y})$ into $\ceil{N/B}$ non-overlapping batches randomly\;
	\For{$n=1$ \KwTo $N$}{
	Reset the knowledge distillation loss $L$ as 0\;
	\For{$iter=1$ \KwTo $\ceil{N/B}$}{
	Draw a batch of skin images $\{ \boldsymbol{x}_1, \cdots, \boldsymbol{x}_B \}$\;
	Input the image batch into the teacher model, computes intra, inter-instance relational knowledge, and logit-based knowledge\;
	Input the image batch into the student model, computes intra, inter-instance relational knowledge, and logit-based knowledge\;
	Compute the knowledge distillation loss using Eq. (\ref{eq:ssd-kd}) for SSD-KD\;
	Update the student model using the optimizer\;
	}
	If loss not decrease within $K_{stop}$ epochs: stop the program\;
	Else: continue the program\;
	}
	\textbf{Return}: An optimized portable student model.
	\caption{The algorithm of our method.}
	\label{alg}
\end{algorithm}

\section{Experiments and results}
In this section, we conduct experiments to empirically demonstrate the effectiveness of the proposed method. We will describe in detail our experimental setup, evaluation metrics, and then present the comparison results and our analysis.

\subsection{Dataset}

The dataset we used in this study is ISIC 2019 \citep{tschandl2018ham10000,codella2018skin,combalia2019bcn20000}, the large-scale publicly available dermoscopy imaging database, that contains 25331 dermoscopic images acquired from different resources along with their diagnosis labels as their ground truth. Part of the images also have corresponding meta information, such as gender, age, anatomy site, etc. The ISIC 2019 contains 8 different skin diseases including malignant melanoma (MM), melanocytic nevus (MN), basal cell carcinoma (BCC), actinic keratosis (AK), benign keratosis (BKL), dermatofibroma (DF), vascular lesion (VASC) and squamous cell carcinoma (SCC). An overview of details for each diseases along with the exampled images is presented in  Figure~\ref{fig:data}.

\begin{figure*}
\centering
\includegraphics[width=1\textwidth]{./figures/data_examples.png}
\caption{\label{fig:data}Example images along with their detailed information from the ISIC 2019 dataset.}
\end{figure*}

\subsection{Implementation details}

In the experiment part, we selected ResNet50 as the teacher model while MobileNetV2 was chosen as the student model. ResNet50 is one of the most powerful convolutional neural network (CNN) architectures and has demonstrated outstanding representation and generalization abilities in medical imaging analytic area \citep{raghu2019transfusion, zhang2019medical, xie2021survey}. However, ResNet50 increases the complexity of architecture and requires huge computational power during training. Therefore, ResNet50 is a suitable choice for teacher model in our project. On the other hand, MobileNetV2, as an advanced version of MobileNet, benefits from its reduced network size and number of parameters, and it has been proven as an effective feature extractor for object detection for mobile applications \citep{sandler2018mobilenetv2, tougaccar2021intelligent, srinivasu2021classification}. In this project, the carefully selected teacher model and student model not only have huge differences in model size, but also process of dramatic distinctions in the backbone structure. Unlike other knowledge distillation studies that only focus on the same structure with different number of layers, our task is more challenging and this setup can better validate the practicality and applicability of the KD methods in the field of dermoscopy imaging.

We implemented our proposed SSD-KD method using PyTorch \citep{paszke2019pytorch} on one NVIDIA Tesla V100 GPU card (32 GB memory) card at the UBC ARC Sockeye, which is a high-performance computing platform available to UBC researchers across all disciplines. For a total of 25331 images from the ISIC2019, following general practices, we randomly selected 10\% of the images to compose a test set, while the rest images were also randomly divided into training and validation with a ratio of 8: 2. Before feeding images into the deep learning models, we uniformly resized all the images to $224 \times 224$ and applied data augmentation using common image processing operations, including horizontal and vertical flips, adjustments in brightness, contrast, saturation, image scaling, and random noise. The teacher and student models were pre-trained on ImageNet \citep{deng2009imagenet} with the weights gathered from torchivision API and then fine-tuned for 150 epochs using the mini-batch SGD optimizer with an initial learning rate of 0.001, momentum of 0.9, and the weight decay of 0.001. For hyperparameters in D-KD and SSD-KD, we used default values for parameters within modules BLKD \citep{hinton2015distilling}, DRKD \citep{park2019relational}, and SSKD \citep{xu2020knowledge}. Then for D-KD, we set $\lambda_{blkd}, \lambda_{drkd}, \lambda_{crkd}$ as 1, 1, and 1000, respectively. The parameters were tuned on the validation dataset with grid search. In SSD-KD, since the SSKD module already included BLKD in it \citep{xu2020knowledge}, we therefore treated SSKD as a whole module and used their default parameters. We then set $\lambda_{drkd}, \lambda_{crkd}, \lambda_{sskd}$ as 1, 1000 and 1, respectively.

In the training phase, we used a conditional reduction strategy for the adjustment of the learning rate. If the model did not improve for 10 consecutive epochs, the learning rate value was reduced by a rate of 0.1. Early stopping was also used for every 15 consecutive epochs if the model did not improve. In the initial experiments, we found that increasing the batch size within a certain range will significantly improve the learning ability of the model. Therefore, while considering the computational cost, on the premise of obtaining a better teacher model, we set the batch size as 128. Additionally, because of the severe data imbalance problems between different skin diseases, we used the weighted cross-entropy as the loss function. The weights are determined according to the frequency of diagnosis labels and thus assigning more attention to the classes that less appeared in the training dataset.

\subsection{Evaluation metrics}

We employed four widely used metrics to evaluate the prediction performance for multi-classes skin diseases of our proposed method. The metrics include Accuracy (ACC), Balanced Accuracy (BACC), mean Average Precision (mAP), and Area Under the Curve (AUC).

\begin{table*}[width=.9\textwidth,cols=4,pos=h]

 \caption{The comparison of the prediction accuracy and computation costs on ISIC 2019 between the teacher model(ResNet50) and the student model (MobielNetv2).}\label{tbl1}
 \begin{tabular*}{\tblwidth}{@{}LLLLLLL@{} }
\toprule
 Model & Input Size & ACC & Model Size & Number of parameters & Time per inference step & GFlops\\
\midrule
ResNet50 & $224\times224$ & 0.820 & 98MB & 25.6M& 58.2ms & 4.11\\
MobileNetV2 & $224\times224$ & 0.754 & 16MB &3.5M & 22.9ms & 0.31\\
\bottomrule
\end{tabular*}
\end{table*}
\begin{table*}[width=.9\textwidth,cols=4,pos=h]

 \caption{Comparison of single teacher/models and other state-of-the-art knowledge distillation methods. Best results are highlighted in bold for each column.}
 \label{tbl2}
 \begin{tabular*}{\tblwidth}{@{}LLLLLLL@{} }
\toprule
Teacher & Student & KD Method & ACC & BACC & AUC& mAP\\
\midrule
ResNet50 & N/A & N/A & 0.820 &0.822 & 0.975 & 0.740 \\
MobileNetV2 & N/A & N/A & 0.754 & 0.767 & 0.959 & 0.625 \\
ResNet50 & MobileNetV2 & BLKD \citep{hinton2015distilling} & 0.838 &0.824 & 0.976 & 0.751 \\
ResNet50 & MobileNetV2 & FitNet \citep{romero2015fitnets} & 0.807 &0.806 & 0.971 & 0.700 \\
ResNet50 & MobileNetV2 & DRKD \citep{park2019relational} & 0.816 &0.804 & 0.972 & 0.744 \\
ResNet50 & MobileNetV2 & SSKD \citep{xu2020knowledge} & 0.840 &0.840 & \textbf{0.978} & 0.759 \\
ResNet50 & MobileNetV2 & SSKD+DRKD+CRKD (ours) & \textbf{0.846} &\textbf{0.843} & 0.977& \textbf{0.796 }\\
\bottomrule
\end{tabular*}
\end{table*}

\begin{table*}[width=.9\textwidth,cols=4,pos=h]
\caption{Comparison of different modules equipped in SSD-KD for ablation studies. Best results are highlighted in bold for each column.}
\label{tbl3}
\begin{tabular*}{\tblwidth}{@{}LLLLLLL@{} }
\toprule
Teacher & Student & KD Method & ACC & BACC & AUC& mAP\\
\midrule
ResNet50 & MobileNetV2 & BLKD+DRKD & 0.839 &0.827 & 0.977 & 0.755 \\
ResNet50 & MobileNetV2 & BLKD+CRKD & 0.839 &0.824 & 0.977 & 0.773 \\
ResNet50 & MobileNetV2 & SSKD+DRKD & 0.841 &0.840 & 0.978 & 0.769 \\
ResNet50 & MobileNetV2 & SSKD+CRKD & 0.842 &0.839 & 0.977 & 0.773 \\
ResNet50 & MobileNetV2 & BLKD+DRKD+CRKD (D-KD) & 0.844 &0.839 & \textbf{0.978} & 0.788 \\
ResNet50 & MobileNetV2 & SSKD+DRKD+CRKD (SSD-KD) & \textbf{0.846} &\textbf{0.843} & 0.977 & \textbf{0.796} \\
\bottomrule
\end{tabular*}
\end{table*}

\subsection{Comparison of the teacher and student models}

To clearly indicate the differences between the teacher model and the student model in various indicators, we compared the classification accuracy along with the computational costs in detail. In Table~\ref{tbl1}, we measured the computational costs by four commonly used complexity standards, including model size, number of parameters, time for every inference step, and the flops. Both teacher model and student model were trained without any knowledge of distillation. As shown in Table~\ref{tbl1}, the teacher model achieved much better performance in skin lesion classification with an accuracy of 82.0\% , while the lightweight student model only got 75.4\% in the same task. However, the model size of the teacher model reached 98M and the number of parameters was 25.6M, which was over 6 times that of the student model. In addition, the time for each inference step of the teacher model was 58.2ms, while the student model only used half of this time. The flops, which were used to measure the number of operations of the network, indicated that the teacher model owned over 10 times of complexity in terms of the operation numbers. The above data comparison showed that the teacher model can indeed achieve an essential improvement in the model's predictive ability, but it is also accompanied by a greater computational requirement. This is well aligned with our assumptions for this project.

\subsection{Comparison with other KD methods}

To demonstrate the effectiveness of our proposed method, several popular representative knowledge distillation methods were listed for comparison. All these listed methods followed exactly the same training settings as introduced above to make sure the comparison was fair and the results were listed in Table~\ref{tbl2}, Figure~\ref{fig:conf_mat}, and Figure~\ref{fig:Roc}. As we can see, all knowledge distillation methods greatly improved the prediction performance of the student model, where the accuracy improvement ranged from 5\% to 9\% and the balanced accuracy improvement ranged from 4\% to 8\%. The increase in mean average precision even reached a maximum of 18\%. These results were comparable to the teacher model, and some parts were even higher than the teacher model. It proved that the KD method can improve the representative ability of the lightweight student model with the help of the teacher model in the classification task for multiple skin diseases via the dermoscopy imaging technique. Among all the methods listed in Table~\ref{tbl2}, we also found that our proposed method achieved the best performance in ACC, BACC, and mAP, and the second-best in AUC. Especially in the balanced accuracy, the obvious improvement indicated that our method is more suitable for the classification of multiple skin diseases with unbalanced sample numbers. In Figure~\ref{fig:conf_mat} and Figure~\ref{fig:Roc}, we presented the detailed confusion matrix of prediction and the ROC curves along with the AUC values for each diseases, which also supported our above analysis.

\begin{figure*}[h]
  \centering

  \subfigure[MobileNetV2]{
  \includegraphics[width = 5cm]{./figures/conf_mat/conf_mat_mobilenet.png}
  }
  \quad
  \subfigure[ResNet50]{
  \includegraphics[width = 5cm]{./figures/conf_mat/conf_mat_resnet.png}
  }
  \quad
  \subfigure[BLKD]{
  \includegraphics[width = 5cm]{./figures/conf_mat/conf_mat_blkd.png}
  }
  \quad
  \subfigure[FitNet]{
  \includegraphics[width = 5cm]{./figures/conf_mat/conf_mat_fitnet.png}
  }
  \quad
  \subfigure[DRKD]{
  \includegraphics[width = 5cm]{./figures/conf_mat/conf_mat_rkd.png}
  }
  \quad
  \subfigure[SSKD]{
  \includegraphics[width = 5cm]{./figures/conf_mat/conf_mat_sskd.png}
  }
  \quad
  \subfigure[BLKD+DRKD]{
  \includegraphics[width = 5cm]{./figures/conf_mat/conf_mat_blkd_rkd.png}
  }
  \quad
  \subfigure[BLKD+CRKD]{
  \includegraphics[width = 5cm]{./figures/conf_mat/conf_mat_blkd_crd.png}
  }
  \quad
  \subfigure[SSKD+DRKD]{
  \includegraphics[width = 5cm]{./figures/conf_mat/conf_mat_sskd_rkd.png}
  }
  \quad
  \subfigure[SSKD+CRKD]{
  \includegraphics[width = 5cm]{./figures/conf_mat/conf_mat_sskd_crd.png}
  }
  \quad
  \subfigure[BLKD+DRKD+CRKD]{
  \includegraphics[width = 5cm]{./figures/conf_mat/conf_mat_blkd_rkd_crd.png}
  }
  \quad
  \subfigure[SSKD+DRKD+CRKD]{
  \includegraphics[width = 5cm]{./figures/conf_mat/conf_mat_sskd_rkd_crd.png}
  }
  \caption{Visualizations of confusion matrices from skin lesion classification models that are trained: without KD (as shown in subfigures (a) -- (b)), and with different KDs (as shown in subfigures (c) -- (l)).}
  \label{fig:conf_mat}
\end{figure*}

\begin{figure*}[htbp]
  \centering

  \subfigure[MobileNetV2]{
  \includegraphics[width = 5cm]{./figures/roc_curve/roc_curve_mobilenet.png}
  }
  \quad
  \subfigure[ResNet50]{
  \includegraphics[width = 5cm]{./figures/roc_curve/roc_curve_resnet.png}
  }
  \quad
  \subfigure[BLKD]{
  \includegraphics[width = 5cm]{./figures/roc_curve/roc_curve_blkd.png}
  }
  \quad
  \subfigure[FitNet]{
  \includegraphics[width = 5cm]{./figures/roc_curve/roc_curve_fitnet.png}
  }
  \quad
  \subfigure[DRKD]{
  \includegraphics[width = 5cm]{./figures/roc_curve/roc_curve_rkd.png}
  }
  \quad
  \subfigure[SSKD]{
  \includegraphics[width = 5cm]{./figures/roc_curve/roc_curve_sskd.png}
  }
  \quad
  \subfigure[BLKD+DRKD]{
  \includegraphics[width = 5cm]{./figures/roc_curve/roc_curve_blkd_rkd.png}
  }
  \quad
  \subfigure[BLKD+CRKD]{
  \includegraphics[width = 5cm]{./figures/roc_curve/roc_curve_blkd_crd.png}
  }
  \quad
  \subfigure[SSKD+DRKD]{
  \includegraphics[width = 5cm]{./figures/roc_curve/roc_curve_sskd_rkd.png}
  }
  \quad
  \subfigure[SSKD+CKD]{
  \includegraphics[width = 5cm]{./figures/roc_curve/roc_curve_sskd_crd.png}
  }
  \quad
  \subfigure[BLKD+DRKD+CRKD]{
  \includegraphics[width = 5cm]{./figures/roc_curve/roc_curve_blkd_rkd_crd.png}
  }
  \quad
  \subfigure[SSKD+DRKD+CRKD]{
  \includegraphics[width = 5cm]{./figures/roc_curve/roc_curve_sskd_rkd_crd.png}
  }
  \caption{Visualization of ROC curves from skin lesion classification models that are trained: without KD (as shown in in subfigures (a) -- (b)), and with different KDs (as shown in subfigures (c) -- (l)).}
  \label{fig:Roc}
\end{figure*}

\subsection{Ablation studies}

As introduced in the methodology section, our proposed method was based on BLKD, while adding the distilled knowledge between instances and within the channels, along with a self-supervised learning strategy. Therefore, it was essential to explore the roles and effects of different components of the overall framework. In Table~\ref{tbl3}, we conducted ablation studies to compare the performance of different methods with different modules. In the training phase, we also fixed all the training optimization parameters and settings to make sure the comparison was fair.

Firstly, we removed self-supervision mechanism and trained BLKD along with DRKD and CRKD separately as shown in the first two rows of the Table~\ref{tbl3}. Compared with BLKD, we found that both DRKD and CRKD methods consistently improved the accuracy, balanced accuracy, AUC, mAP values. Based on this result, we drew preliminary conclusions that the incorporation of relational-based KD can benefit the predictive ability of the student model by introducing more knowledge. Next, we involved the self-supervision in BLKD, which is noted as SSKD in the Table~\ref{tbl2} and Table~\ref{tbl3}. DRKD and CRKD methods were also considered individually in this scenario. According to the third and forth rows of the comparison, we noticed that the self-supervised method further improved the results while DRKD and CRKD methods were included, especially the ACC and BACC. This results demonstrated that while introducing DRKD and CRKD method to knowledge distillation, the adoption of the SSKD module can achieve further improvement of representative ability of the student model. Finally, we carried out another two sets of experiments that introduced both DRKD and CRKD modules and trained them with and without the self-supervision strategy to further validate our hypothesis. As shown in the last two rows of the Table~\ref{tbl3}, our jointly optimized knowledge distillation methods achieved the best performance with the highest prediction accuracy and balanced accuracy, which agreed with our preliminary conclusions mentioned above. Each component of our architecture contributed to the improved performance of the lightweight student model, thus making the proposed method more effective.

\subsection{Visualizations of class activation maps}

In addition to the analysis and comparisons, we further observed the prediction performance of the test set samples before and after using our knowledge distillation method. We noticed that the skin lesion classes of many samples in the test set were correctly predicted by the teacher model and the student model with knowledge distillation, but are diagnosed as the wrong classes by the student model without the knowledge distillation. For example, 17 melanomas were predicted as benign keratoses by the student model without any knowledge distillation but they were classified correctly after the implementation of our KD method. To show how the KD methods helped the learning of the student model, we visualized the class activation mapping (CAM) from the teacher model, the student model before the implementation of KD, and the student model after using KD applying Grad-CAM \citep{selvaraju2017grad}, which was commonly used to locate discriminative regions for object detection and classification tasks. As presented in Figure~\ref{fig:CAM}, the located discriminative areas of student model after the KD implementation was more align with the teacher model's activation maps; Hence, it enhanced the performance of the student model.

\begin{figure*}[htbp]
  \centering

  \subfigure[Original]{
  \includegraphics[width = 3cm,height = 2.2cm]{./figures/cam/original_1.jpg}
  }
  \quad
  \subfigure[Teacher]{
  \includegraphics[width = 3cm,height = 2.2cm]{./figures/cam/teacher1}
  }
  \quad
  \subfigure[Student before SSD-KD]{
  \includegraphics[width = 3cm,height = 2.2cm]{./figures/cam/student_no_1.png}
  }
  \quad
  \subfigure[Student after SSD-KD]{
  \includegraphics[width = 3cm,height = 2.2cm]{./figures/cam/student_yes_1.png}
  }

  \caption{Visualization of class activation mapping (CAM). The example images are correctly predicted by the teacher model and the student model only after implementing our proposed method. (a) is the original dermoscopy image while the other three subfigures (b)-(d) denote their corresponding CAM ouput from different models.}
  \label{fig:CAM}
\end{figure*}

\section{Discussion and future works}
The multi-class classification task has made great progress in recent years due to the in-depth research of medical imaging and deep learning technology. However, excellent model capabilities are often accompanied by complex structures and high training and storage costs, which limit their practical applications in the scenarios with restricted computational resources, such as portable devices. Correspondingly, although the small lightweight model can greatly reduce the computational cost, it cannot extract more representative features, so that the predictive ability of the model is degraded. For example, in this paper, both ResNet50 and MobileNetV2 are trained on the large-sacle dataset of ISIC. However, the prediction accuracy of ResNet50 can reach 82\%, while the lightweight MobileNet can only reach 75\%. It can be seen that using the KD methods to improve the learning performance of lightweight models while maintaining the complexity advantage is an important direction in both theoretical and application levels.

Considering the morphological characteristics of skin lesions, we added a number of modules based on the traditional KD framework, including the widely used DRKD and SSKD, as well as CRKD and weighted soften knowledge that were specifically designed for skin lesions classification via dermoscopy imaging. While the number of skin cancer lesions is far less than that of benign skin lesions, in our proposed method, we integrated weighted softened output and generic feature representations in a self-supervised manner, and also improved the ability of a lightweight model to obtain more useful information from classes with fewer subjects. Therefore, our proposed method can better help the lightweight student model to extract features, thereby achieving improved results. In addition, when we performed ablation studies, we also noticed that although the KD methods improved the student model substantially, the improvement brought by adding different modules to BLKD was not very obvious. To verify that the presented results are sufficiently convincing, we tried ten experiments under the same conditions and calculated the standard deviation of all calculated metrics. The standard deviation values were all less than 0.01, demonstrating the robustness of the trends shown by our results. In fact, further tuning of the hyper-parameters should make the model perform better when adding a combination of different modules, but in this paper we fixed the hyperparameter values of all modules for a completely fair comparison.

Additionally, after a series of pre-experiments, we selected the two most representative models, based on their distinct characteristics, as the teacher model and the student model. As the shown in table~\ref{tbl1}, our criteria for selecting models was to ensure that the teacher model has excellent performance, while the complexity of the student model is suitable for practical portable applications. Although we achieved our expectation of improving the predictive power of the student model, we strongly believed that this was only a starting point for this line of research. In our future works, we plan to further introduce other architectures, such as multi-teacher KD and KD with the help of the teaching assistant mechanism, so as to improve the generalizability of the obtained results. Moreover, our proposed method can also be applied to other medical problems with similar scenarios after fine-tuning, helping to improve lightweight models in other areas.

\section{Conclusion}

In this paper, we proposed an efficient knowledge distillation framework, called SSD-KD, which aimed to address the challenging task of multi-class classification of dermoscopic skin lesion imaging by lightweight deep learning models. Different from existing KD methods, SSD-KD considered both intra- and inter-instance based knowledge to guide the lightweight model capture richer structural knowledge that are important for skin lesion diagnosis. Additionally, the self-supervised auxiliary training strategy and weighted cross entropy loss were also involved to further boost its performance.

We conducted experiments to evaluate our proposed method on the large-scale open-accessed dermoscopic skin lesion imaging dataset ISIC 2019, and we presented detailed comparisons with the existing KD methods. We also applied a CAM method to visualize the changes of activation maps generated by the models. Experiments demonstrated the effectiveness of our proposed SSD-KD with a promising result in multi-diseases classification. With low model complexity and storage requirements while improving predictive performance, our method could greatly assist in the promotion of deep learning techniques to clinical practices.

\printcredits

\bibliographystyle{cas-model2-names}

Computer Vision and Pattern Recognition (cs.CV)