Detailed analysis of the cell-inactivation mechanism by accelerated protons and light ions

Abstract: Published survival data for V79 cells irradiated by monoenergetic protons, helium-3, carbon, and oxygen ions and for CHO cells irradiated by carbon ions have been analyzed using the probabilistic two-stage model of cell inactivation. Three different classes of DNA damages formed by traversing particles have been distinguished, namely severe single-track damages which might lead to cell inactivation directly, less severe damages where cell inactivation is caused by their combinations, and damages of negligible severity that can be repaired easily. Probabilities of single ions to form these damages have been assessed in dependence on their linear energy transfer (LET) values.Damage induction probabilities increase with atomic number and LET. While combined damages play crucial role at lower LET values, single-track damages dominate in high-LET regions. The yields of single-track lethal damages for protons have been compared with the Monte Carlo estimates of complex DNA lesions, indicating that lethal events correlate well with complex DNA double-strand breaks. The decrease in the single-track damage probability for protons of LET above approx. 30 keV/$\mu$m, suggested by limited experimental evidence, is discussed, together with the consequent differences in the mechanisms of biological effects between protons and heavier ions. Applications of the results in hadrontherapy treatment planning are outlined.

\section{Introduction}
\label{sec:Introduction}
Hadron radiotherapy, based on irradiating tumours by beams of accelerated protons and other ions, is expected to significantly increase the cure rate of cancer in near future; compare e.g.\ \citep{MedAustron, CNAO, Brahme - Design, Heidelberg}. The well-known rationale for using proton beams in radiotherapy lies in the characteristic pattern of their energy deposition to matter, the Bragg peak, leading to the possibility of achieving highly conformal dose distributions \citep{Wilson, Lomax1, Lomax2}. Beams of light ions possess additional advantages: They are characterized by higher linear energy transfer (LET) values, leading to enhanced relative biological effectiveness (RBE) and diminishing oxygen enhancement ratio (OER), e.g.\ \citep{ion-rationale}. They are therefore expected to be especially suitable for curing tumours resistant to conventional radiotherapy; compare e.g.\ \citep{ion-rationale, chordomas}. Another advantage consists in the possibility of online dose monitoring by positron emission tomography, PET \citep{PET}. Furthermore, recent clinical studies indicate that unconventional fractionation schemes might be used for selected tumours, reducing the overall treatment time to a few sessions only \citep{Tsujii - PTCOG Paris 2004}.

Several dedicated hadrontherapy facilities have been launched recently, compare \citep{Particles}. In most of them only proton beams are available; however, in several centres, existing or being built, protons as well as light ions up to carbon or oxygen may be used. For the choice of an optimal ion in given clinical situations, analyses of energy loss, scattering and fragmentation phenomena for different ions and comparisons of their dose distributions are necessary \citep{Brahme - Design, TRiP1, fragmentation}. These physical analyses have to be complemented then by biology-oriented studies, analyzing the mechanisms of biological effects for different ions using available data and adequate models.

However, existing hadrontherapy treatment planning approaches have been based on detailed description of the physical processes only, and have not addressed the biological phase of the underlying radiobiological mechanism in detail. E.g.\ the treatment planning procedure used at HIMAC (Chiba, Japan) is based on interpolating the $\alpha, \beta$ coefficients of the linear-quadratic (LQ) fits to experimental survival data, and on the similarity in biological effects between carbon and neutron beams of similar LET values \citep{Kanai 1997, Kanai 1999}; compare also \citep{Kagawa 2002}. In the approach used in clinical hadrontherapy applications at GSI (Darmstadt, Germany), the local effect model (LEM), the effect of ion tracks on a microscopic scale is assumed to be equal to that of correspondingly high photon doses (compare \citep{TRiP2}); again, the underlying biological processes of damage induction and repair have not been taken into account explicitly. On the contrary, the present paper summarizes the results of a detailed biology-oriented analysis of cell inactivation data for protons, helium-3, carbon and oxygen ions. Experimental data gathered by several groups \citep{Belli, Perris, Goodhead, Folkard96, Wilma, Kiefer} have been analysed with the help of a detailed probabilistic radiobiological model \citep{PhysMedBiol2005}.

\section{Probabilistic two-stage model of radiobiological effects}
\label{sec:ProbabilisticRadiobiologicalModel}

The processes running in a cell after irradiation (under conditions usual in radiotherapy) proceed in two distinct, subsequent phases. The first stage includes processes running immediately after the impact of individual ionizing particles, i.e.\ energy transfer events, radical formation and diffusion, chemical reactions and formation of DNA damages. The other stage is formed then by the subsequent cellular response which follows as a reaction to the effect of all ionizing particles having hit a cell nucleus or chromosomal system at a given dose; this stage includes the processes of damage repair or misrepair and further biological processes leading to cell survival or inactivation. Distinguishing these two phases represents the basis of the probabilistic two-stage model of biological effects of ionizing particles.

In the full scheme of the model, the following characteristics are taken into account: (i) the stochastic distribution of particle tracks over the irradiated cell population, (ii) the distribution of transferred energy, (iii) the probability of single traversing particles to induce DNA damages of different severity, (iv) saturation or synergetic combinations of individual damages, and (v) the effects of cellular repair systems. The model scheme, presented in \citep{PhysMedBiol2005}, allows to describe different classes of survival curves, including also low-dose hypersensitivity phenomena, on the basis of detailed characteristics of the mentioned processes, especially the effects of damage induction and repair processes. However, in the following comparison of the effects of different light ions that are expected to be used in hadrontherapy, a simplified model scheme will be used, taking into account only DNA damages that are practically unrepairable.

The cell inactivation probability at applied dose $D$ is given by
\begin{equation} \label{sumaPkqk}
	s(D) = \sum_k P_k(D) \: q_k \ .
\end{equation}
Here, the number $k$ ($k\geq 0$) of primary particles (particle tracks) traversing individual cell nuclei is given by Poisson distribution
\begin{equation} \label{Poisson}
	P_k (D) = \frac{(hD)^k}{k!} \exp(-hD) \ ,
\end{equation}
the average number per 1 Gy, $h$, being proportional to the geometric cross-section of the nuclei, $\sigma$. The survival probability after $k$ particles have traversed the nucleus has been denoted by $q_k$. It can be calculated by
\begin{equation} \label{qk}
	q_k = 1 - 	\sum_{i=0}^k (^k_{\, i}) a^i (1-a)^{k-i}
				\sum_{j=0}^{k-i} (^{k-i}_{\, \ j}) b^j (1-b)^{k-i-j}
				[1-r_{ij}^{ab}] \ ,
\end{equation}
where combinatorial numbers $(^k_{\, i}) = \frac{k!}{i!(k-i)!}$. Here $a$ and $b$ stand for the induction probability of different damages and $r_{ij}^{ab}$ for repair probabilities of their different combinations; $r_{00}^{ab} \equiv 1 \equiv r_{01}^{ab}$ (compare below). Three different classes of damage formed by individual particles have been distinguished:
\begin{enumerate}
	\item Severe damage formed by a single track that is capable of causing cell inactivation even if this is the only damage to the cell ("single-track" or "single-particle induced" damage). The probability of inducing such a damage, per track, has been denoted by $a$.
	\item Damages of lower severity, at least two of which must combine to be lethal ("combined" or "two-track" damages). The probability of their formation (per track) has been denoted by $b$.
	\item Lesions that do not represent any significant challenge to the cell and its repair systems. Such negligible lesions have not been included in Eq.\ (\ref{qk}).
\end{enumerate}
The above classification of DNA damage is illustrated in Figure~\ref{fig:classification}.

\begin{figure}[!htb]
	\centering
		\includegraphics[angle=0, width=0.60\textwidth]{classification.eps}
	\caption{Schematic illustration of the damage classification according to damage complexity.}
	\label{fig:classification}
\end{figure}

Eq.\ (\ref{qk}) represents the general formula for cell inactivation after a passage of $k$ particles. The effects of cell repair systems have been included in repair probabilities $r^{ab}_{ij}$. In the present analysis, however, only unrepairable damages have been considered, for which $r^{ab}_{ij} \equiv 0$. In other words, the probabilities $a$ and $b$ derived here represent the lowest estimates of damage induction at corresponding energy transfers (LET values), since unrepairable damages form, essentially, subsets of all (both repairable and unrepairable) damages in the above classification (Eq.\ (\ref{qk}), Figure~\ref{fig:classification}).

Considering unrepairable damages only, cell inactivation follows, then, if at least one single-track or at least two combined-type damages have been formed by the $k$ particles traversing the nucleus. Indeed, for $r^{ab}_{ij} \equiv 0$, Eq.\ (\ref{qk}) may be simplified to
\begin{equation} \label{qk2}
	q_k = (1-a)^k - 	(1-a)^k  \sum_{j=2}^k (^k_{\, j})  b^j (1-b)^{k-j}
	= (1-a)^k [(1-b)^k + kb(1-b)^{k-1}]
\end{equation}
(with $q_0 = 1$ and $q_1 = 1-a$),
i.e., the cell survives if no single-track and not more than a single $b$-type damage have been formed by the $k$ particles.
\footnote{The corresponding Eqs.\ (9-15) in \citep{PhysMedBiol2005} represented simplified versions of formulas derived from Eq.\ (\ref{qk}), as combined damages $b$ were taken into account in a simplified manner only: The synergetic effects were represented by terms $(1-b^2)^{k(k-1)/2}$ only, yielding e.g.\ $q_k = (1-a)^k (1-b^2)^{k(k-1)/2}$ after neglecting the repair probabilities. By comparing this formula with Eq.\ (\ref{qk2}) it is clear that the role of combined damages was slightly underestimated, especially in the region of higher particle numbers. I.e.\ the analyses based on precise formulas derived from Eq.\ (\ref{qk}) yield slightly higher combined-damage induction probabilities as compared to the simplified formulas used in \citep{PhysMedBiol2005}. This fact, however, does not affect the conclusions drawn in that paper, especially those concerning the role of single-particle and combined damage induction and repair processes with respect to the shapes of survival curves.}

\section{Analysis of experimental data}
\label{sec:AnalysisOfExperimentalData}

Survival data for V79 cells irradiated by protons at 0.57~--~7.4~MeV \citep{Belli, Folkard96, Perris, Goodhead}, helium-3 at 3.4~--~6.9~MeV \citep{Folkard96}, carbon at 2.4~--~266~MeV/u \citep{Wilma} and oxygen ions at 1.9~--~396~MeV/u \citep{Kiefer} have been analysed using the model reviewed in the preceding section. The list of particle energies, LET values and ranges in water is given in Table~\ref{tab:energies}.

\begin{table}
\caption{\label{tab:energies}Physical parameters of particles in the analyzed data sets. Energy, LET values and ranges as given by the authors, or calculated using PSTAR tables \citep{PSTAR} or SRIM-2003 \citep{SRIM}.}
\begin{indented}
\item[]\begin{tabular}{@{}lllll}
\br
&Energy&LET&Range in water&Reference\\
&[MeV/u]&[$\mathrm{keV/\mu m}$]&[mm]&\\
\mr
p&7.4&5.8&0.72&\citet{Perris}\\
&5.01&7.7&0.37&\citet{Belli}\\
&3.66&10.1&0.21&\citet{Folkard96}\\
&3.20&11.0&0.17&\citet{Belli}\\
&3.0&11.7&0.15&\citet{Perris}\\
&1.83&17.8&0.065&\citet{Folkard96}\\
&1.41&20.0&0.043&\citet{Belli}\\
&1.4&20.3&0.042&\citet{Goodhead}\\
&1.16&23.0&0.040&\citet{Goodhead}\\
&1.07&27.6&0.027&\citet{Folkard96}\\
&0.76&30.5&0.016&\citet{Belli}\\
&0.64&34.6&0.012&\citet{Belli}\\
&0.57&37.8&0.010&\citet{Belli}\\
\mr
$^3\mathrm{He}$&2.30&58.9&0.074&\citet{Folkard96}\\
&1.39&88.3&0.033&\\
&1.13&105.8&0.024&\\
\mr
C&266.4&13.7&140&\citet{Wilma}\\
&190.7&16.8&80&\\
&76.9&32.4&15.9&\\
&18.0&103&1.2&\\
&11.0&153.5&0.5&\\
&5.4&275.1&0.15&\\
&4.2&339.1&0.1&\\
&2.4&482.7&0.045&\\
\mr
O&396&18&202&\citet{Kiefer}\\
&88&46&15.2&\\
&10.7&238&0.36&\\
&1.9&754&0.030&\\
\br
\end{tabular}
\end{indented}
\end{table}

To facilitate a systematic analysis of the given data sets, the damage induction probabilities for a given ion have been considered in dependence on LET value $\lambda$ instead of deriving their values for individual measured survival curves independently. These LET dependences have been represented by flexible test functions involving a low number of auxiliary parameters, $a_{0-2}$ and $b_{0-2}$:
\begin{equation} \label{ab}
	a(\lambda) = a_0 (1 - \exp(-(a_1 \lambda)^{a_2})) \ , \qquad
	b(\lambda) = b_0 (1 - \exp(-(b_1 \lambda)^{b_2})) \ .
\end{equation}
Parameter values have been determined using standard optimization methods; mainly the SIMPLEX and MIGRAD methods implemented in the MINUIT multivariate minimization tool~\citep{MINUIT} have been applied to dedicated computer codes written in FORTRAN. Weighted least-square method has been used to construct the objective function, $\chi^2$, as described previously \citep{PhysMedBiol2005}.

Model representations of survival curves for V79 cells irradiated by different ions are shown in Figures~\ref{fig:p}-\ref{fig:O}. In Figure~\ref{fig:ab}, the probabilities of single-track and combined damage induction are plotted as functions of LET for different ions. Values of auxiliary model parameters are listed in Table~\ref{tab:ModelParameters}.

\begin{figure}[!htb]
	\centering
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{p1.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{p2.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{p3.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{p4.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{p5.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{p6.eps}
	\caption{Survival curves for V79 cells irradiated by protons; data taken from \citep{Belli, Folkard96, Perris, Goodhead}.}
	\label{fig:p}
\end{figure}

\begin{figure}[!htb]
	\centering
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{3He1.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{3He2.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{3He3.eps}
	\caption{V79 cells irradiated by low-energy $\mathrm{^3He}$ ions; data from \citep{Folkard96}.}
	\label{fig:3He}
\end{figure}

\begin{figure}[!htb]
	\centering
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{C1.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{C2.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{C3.eps}
	\caption{Representation of survival curves for V79 cells irradiated by carbon ions; data from \citep{Wilma}.}
	\label{fig:C}
\end{figure}

\begin{figure}[!htb]
	\centering
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{O1.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{O2.eps}
	\caption{Survival of V79 cells irradiated by oxygen ions; data measured by \citet{Kiefer}.}
	\label{fig:O}
\end{figure}

\begin{figure}[!htb]
	\centering
		\includegraphics[angle=-90, width=0.34\textwidth]{p-ab.eps}
		\includegraphics[angle=-90, width=0.34\textwidth]{3He-ab.eps}
		\includegraphics[angle=-90, width=0.34\textwidth]{C-ab.eps}
		\includegraphics[angle=-90, width=0.34\textwidth]{O-ab.eps}
	\caption{Probabilities of single-track and combined-damage induction for protons (top left), helium-3 (top right), carbon (bottom left) and oxygen ions (bottom right) in dependence on LET. For helium-3, the combined damages might have been neglected at all, $b=0$.}
	\label{fig:ab}
\end{figure}

\begin{table}
\caption{\label{tab:ModelParameters}Values of model parameters, describing the dependence of damage induction probabilities on ion LET values for V79 and CHO-K1 cells, Eq.\ (\ref{ab}).}
\begin{indented}
\item[]\begin{tabular}{@{}l|llll|l}
\br
&&&V79&&CHO-K1\\
\mr
&p&$^3$He&C&O&C\\
\mr
$a_0$&0.045&0.25&0.41&0.80&0.50\\
$a_1$ [$\mu$m/keV]&0.036&0.015&0.0063&0.0042&0.0064\\
$a_2$&3.57&2.79&3.21&4.17&2.0\\
$b_0$&0.017&0.0&1.0&0.39&0.26\\
$b_1$ [$\mu$m/keV]&0.13&&0.0012&0.0093&0.0011\\
$b_2$&2.81&&1.0&1.44&0.69\\
\br
\end{tabular}
\end{indented}
\end{table}

A similar analysis has been performed for survival data of CHO-K1 cells irradiated by carbon ions, measured experimentally by \citet{Wilma}. Model representation of survival curves and the derived damage induction probabilities are shown in Figure~\ref{fig:C-CHO}; the parameter values are given in Table~\ref{tab:ModelParameters}.

\begin{figure}[!htb]
	\centering
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.34\textwidth]{C-CHO1.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.34\textwidth]{C-CHO2.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.34\textwidth]{C-CHO3.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.34\textwidth]{C-CHO4.eps}
		\includegraphics[angle=-90, width=0.34\textwidth]{C-CHO-ab.eps}
	\caption{Survival curves (data from \citep{Wilma}) and damage induction probabilities for CHO-K1 cells irradiated by carbon ions.}
	\label{fig:C-CHO}
\end{figure}

The present results correspond to the geometrical cross section for V79 cells chosen uniquely for all data sets to be $\sigma$ = 87.8 $\mathrm{\mu m^2}$, based on the value reported in \citep{Wilma}, in order to enable comparisons between different ions. Values in the range of 50~--~135 $\mathrm{\mu m^2}$, also reported in the literature (compare e.g. \citep{Kraft-Scholz-50, Belli}), have yielded similar results (figures not shown here). For CHO cells, the value of geometrical cross section $\sigma$ = 108 $\mathrm{\mu m^2}$ was taken, as reported in the given experiment \citep{Wilma}.

\section{Discussion}
\label{sec:Discussion}

\subsection{Single-track damage induced by protons: Possible indications for differences in radiobiological mechanisms between protons and ions}
\label{sec:DiscussionProtons}

Measured data for protons of LET values above approximately 30 keV$/\mu$m \citep{Belli} suggest that the single-track damage formation probability, $a$, might decrease in this region; compare Figure~\ref{fig:p-nonmonotonous} where more detailed fits to the data are shown, based on the parameterization
\begin{eqnarray} \label{ab-nonmonotonous}
	a(\lambda) &=& a_0 (1 - \exp(-(a_1 \lambda)^{a_2})) \exp(-(a_3 \lambda)^{a_4}) \ , \nonumber \\
	b(\lambda) &=& b_0 (1 - \exp(-(b_1 \lambda)^{b_2})) \ .
\end{eqnarray}
Whereas the value of the goodness-of-fits criterion is $\chi^2=350$ for the data of \citet{Belli} in the joint fits to proton-induced survival shown in Figure~\ref{fig:p}, the corresponding value for the detailed fits shown in Figure~\ref{fig:p-nonmonotonous} is $\chi^2_{\mathrm{detailed}}=55$. The goodness-of-fits to this data set can be, in fact, further improved if, in addition to a decreasing single-track damage probability, the combined-damages are allowed to dominate in the whole protons' LET range and their repair is taken into account, compare \citep{PhysMedBiol2005}.

To verify the hypothesis of decreasing single-track damage probability for protons of high LET, further experimental studies are necessary, as the existing evidence is limited to two survival curves measured under a single experimental setup only. Even though this decrease corresponds to very limited proton ranges in tissues only (approximately 15 $\mu$m or smaller) and will probably not manifest in clinical applications, this finding would be important for understanding the mechanisms underlying the radiobiological effects: For heavier ions, no such complex behaviour has been observed in the studied LET ranges, the damage induction probabilities exhibiting saturation characteristics only (Figure~\ref{fig:ab}). It means that for heavier ions the RBE effects termed often as "overkill" occur as a result of too high energy deposits to individual cells. For protons, on the other hand, this evidence suggests that they are already the effects of single particles on the level of DNA damage formation that get saturated and are responsible for the decrease of biological effectiveness observed experimentally \citep{Belli}.

\begin{figure}[!htb]
	\centering
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{p-detailed-1.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{p-detailed-2.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{p-detailed-3.eps}
		\includegraphics[angle=-90, width=0.32\textwidth]{p-detailed-ab-0,2.eps}
	\caption{Detailed fits to proton-induced V79 inactivation, indicating a decrease in the single-track damage formation for protons at LET values above approximately 30 keV$/\mu$m; data measured by \citet{Belli}, calculations based on Eq.\ (\ref{ab-nonmonotonous}).}
	\label{fig:p-nonmonotonous}
\end{figure}

\subsection{The influence of repair processes}
\label{sec:DiscussionRepair}

As mentioned in the preceding paragraph, the effects of cellular repair processes might be important in representing the outcome of proton irradiation. Also the fits to $^3$He data in the high-dose region might be improved significantly by incorporating the repair probabilities (Figure~\ref{fig:He-repair}). In this region, however, the experimental conditions are very challenging and, consequently, the data in this region have a limited reliability \citep{Folkard96}; therefore, the authors have interpreted the upward-bending of survival curves as artificial effects only. However, the results shown in Figure~\ref{fig:He-repair} demonstrate that similar upward bending might follow as a result of repair processes; compare also the general discussion in \citep{PhysMedBiol2005}. The damage induction probabilities are almost identical with those presented in Figure~\ref{fig:ab} (in both cases, the combined-damage probabilities have been neglected, $b=0$); the survival values at high doses being mainly influenced. Again, additional experimental data would be helpful in solving this issue.

\begin{figure}[!htb]
	\centering
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{3He-repair-1.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{3He-repair-2.eps}
		\includegraphics[trim=0cm 0.4cm 0cm 1.2cm, clip, angle=-90, width=0.32\textwidth]{3He-repair-3.eps}
		\includegraphics[angle=-90, width=0.32\textwidth]{3He-repair-ab.eps}
		\includegraphics[trim=0.7cm 1.2cm 1.3cm 1.6cm, clip, angle=-90, width=0.32\textwidth]{3He-repair-r.eps}
	\caption{Upper panel: Representation of survival curves for V79 cells irradiated by helium-3 ions (data from \citep{Folkard96}) when repair $r^a$ of single-track damage taken into account. Lower panel: Damage induction and repair probabilities.}
	\label{fig:He-repair}
\end{figure}

The goodness-of-fits for heavier ions does not increase significantly when the repair processes are taken into account. For the sake of simplicity, only unrepairable damages have, therefore, been considered in the present work (Section~\ref{sec:AnalysisOfExperimentalData}). The induction probabilities for both single-track and combined damages have been estimated on the basis of survival data only. This slightly limits the reliability of estimating the roles of combined-track events, $b$, in the higher-LET regions, whereas the unrepairable single-track damages, $a$, have been assessed reliably in the whole studied LET regions. Somewhat higher values of damage induction probabilities would be obtained if the influence of repair processes were included. A corresponding concept, including a wider class of damages and their repair by the cells, has been used in analyzing radiobiological data for cell lines of different radiosensitivity; the results will be presented and discussed in detail elsewhere \citep{CHO-xrs}.

\subsection{On the interpretation of the damage classification}
\label{sec:DiscussionInterpretation}

Clustered DNA lesions of high complexity (locally multiply damaged sites, complex lesions) are thought to represent significant challenges to cellular repair systems, and to play crucial roles in cell inactivation by ionizing particles \citep{Ward, Goodhead-LMDS, Ottolenghi}. The existing estimates of their yields have been based mostly on Monte Carlo calculations involving track structure simulations and simplified models of DNA and/or chromatin structure. Due to their complexity and computational costs, these approaches have been performed for photons and low-energy particles only. On the other hand, the present approach, based on analyzing survival data, enables to derive the yields of lethal events also for high-$Z$ high-energy ions (Figure~\ref{fig:ab}). By comparing the present results with the mentioned Monte Carlo calculations, mechanistic interpretation of the damage classification used in the present work might be sought, and the fraction of lethal events among different classes of complex lesions can be estimated.

As the first step in this direction, the present results for lethal damage induced by protons in V79 cells have been compared to estimates of complex lesions calculated on the basis of a fast Monte Carlo damage simulation model proposed by \citet{Semenenko+Stewart}. In Figure~\ref{fig:damageYields}, the LET-dependent yields of single-track lethal events, $a$, are compared to the total number of double-strand breaks (DSBs) and DSBs composed of at least 4, 6, 8, and 10 elementary lesions (single-strand breaks (SSBs), base damages or abasic sites). This figure indicates that the present results are consistent with the trends predicted by Monte Carlo calculations. In Figure~\ref{fig:lethalRatios}, the ratio between the number of lethal events and DSBs of different complexity is shown, again as a function of LET. Whereas only 0.05~--~0.5\% of all DSBs formed (denoted by DSB total in Figures \ref{fig:damageYields} and~\ref{fig:lethalRatios}) are lethal, this portion increases to approximately 3~--~4\% for DSBs composed of at least 8 elementary lesions (DSB 8+); this number being uniform (LET-independent), i.e.\ being consistent with the hypothesis of a uniform repair probability for these specific damages.

\begin{figure}[!htb]
	\centering
		\includegraphics[trim=0.3cm 0.5cm 0.3cm 1.5cm, clip, angle=0, width=0.80\textwidth]{damage_yields.eps}
	\caption{Comparisons of damage yields for protons of different LET values: single-track lethal damage as derived in the present work, and Monte Carlo estimates of total DSB numbers and yields of DSBs of higher complexity, consisting of at least 4, 6, 8, or 10 elementary DNA lesions (DBS 4+, 6+, 8+, 10+, respectively).}
	\label{fig:damageYields}
\end{figure}

\begin{figure}[!htb]
	\centering
		\includegraphics[trim=0.3cm 0.5cm 0.3cm 0.5cm, clip, angle=0, width=0.80\textwidth]{lethal_ratios.eps}
	\caption{The ratio of lethal events among all DSBs (DSB total) and complex DSBs formed by at least 4, 6, 8, or 10 elementary lesions.}
	\label{fig:lethalRatios}
\end{figure}

These results indicate that the single-track lethal damages, $a$, might be related to clustered lesions of very high complexity, and correlate best with DSB 8+, containing at least 6 elementary DNA lesions in addition to 2 SSBs forming the DSB. Similarly, the combined damages, $b$, might correspond to pairwise combinations of DSBs formed by different tracks, leading to large-scale chromosomal aberrations; compare e.g.\ \citep{Sachs}. Further damage induction and repair studies are, however, necessary to identify the biophysical nature of lethal events reliably. Furthermore, this comparison has been performed for protons only, given the limitations of existing Monte Carlo approaches. Efforts will therefore be made to extend these calculations to heavier ions and higher energies, using correspondingly simplified Monte Carlo-based models.

\subsection{Applications in hadrontherapy treatment planning}
\label{sec:DiscussionTreatmentPlanning}

The analyzed proton and helium-3 data \citep{Belli, Folkard96, Perris, Goodhead} concern irradiation by low-energy particles only, with ranges in tissue of approx. 10~$\mu$m -- 0.7~mm for protons and 24 -- 74~$\mu$m for $^3$He ions, respectively (Table~\ref{tab:energies}), corresponding to track ends of these particles. On the other hand, data for carbon \citep{Wilma} and oxygen ions \citep{Kiefer} have been gathered over wide energy (and LET) ranges of these particles, corresponding to penetration depths of approx. 45 $\mu$m to 140 mm for carbon ions and 30 $\mu$m to 200 mm for oxygen ions, covering the majority of clinically relevant ranges. Figures~\ref{fig:C} and~\ref{fig:O} illustrate that the simplified probabilistic model enables to represent cell survival over such wide ranges in a systematic manner by using a low number of parameters only (the present analysis involves 6 parameters, $a_0, a_1, a_2$ and $b_0, b_1, b_2$, plus the cross section of cell nuclei, $\sigma$). This is a prerequisite if the model is to be used in treatment planning applications. In the present analysis, cell inactivation effects have been studied for monoenergetic particles as function of their LET value (corresponding to their energy). For treatment planning applications, an adequate physical model, describing the spectra of particle energy and LET values as function of penetration depths and including also energy-loss straggling, lateral scattering and fragmentation processes, has to be used together with the model of biological effects presented here. Such a physical model might be based on detailed Monte Carlo calculations (e.g.\ \citep{SRIM}) or semi-phenomenological parameterization of measured experimental data \citep{TRiP1, Chu}. Calculations of survival as function of penetration depth for several cell lines, using the radiobiological module presented here and a simplified description of physical processes based on SRIM-2003 calculations, have been already performed, showing good agreement with measured data; the results will be presented separately \citep{depth-survival}. To enhance the predictive power of the model, relation between the damage induction probabilities $a, b$ and track structure characteristics of different ions will be sought, with the aim to help in identifying the optimal ion in different clinical situations. Additional systematic experimental data, especially for inactivation induced by light ions (helium to nitrogen), would be helpful in solving this issue, too.

\section{Conclusion}
\label{sec:Conclusion}
Systematic analysis of published survival data for V79 cells irradiated by light ions has been performed using a simplified scheme of the probabilistic two-stage model. The probabilities of single ions to induce severe damage to DNA have been derived for different ions in dependence on their LET values. In the lower-LET regions, combined damages dominate, while survival at higher LET values is governed by single-particle damages (Figure~\ref{fig:ab}). The present results give quantitative estimates of the increase in damage complexity with increasing atomic number $Z$ and LET value. The analysis of survival data for CHO-K1 cell line shows then the differences in DNA damage induction for different cell lines.

The derived probabilities of inducing single-track lethal damages have been compared with the Monte Carlo estimates of the yields of complex DNA lesions. Good agreement between these two methodologically different approaches was demonstrated, indicating the biophysical interpretation of lethal events and the importance of damage complexity with respect to its biological consequences.

The results of this work will be used in proposing detailed biology-oriented approaches for hadrontherapy treatment planning. The results may also contribute to understanding the differences between the mechanisms of biological effects of different ions.

\ack
The author is grateful to Prof R Stewart (Purdue University, USA) for providing him with the results of Monte Carlo simulations of different damage types for low-energy protons and to Prof M Lokaj\'{\i}\v{c}ek for valuable comments on the manuscript. The present work was supported by the grant "Modelling of radiobiological mechanism of protons and light ions in cells and tissues" (Czech Science Foundation, GACR 202/05/2728). Cooperation within the ENLIGHT project ("European Network for LIGht Ion Hadron Therapy", European Commission, QLG1-CT-2002-01574) is acknowledged, too.

\References

	\bibitem[Belli \etal(1998)]{Belli}
	Belli M, Cera F, Cherubini R, Dalla Vecchia M, Haque AMI, Ianzini F, Moschini G, Sapora O, Simone G, Tabocchini MA, Tiveron P 1998
	RBE-LET relationships for cell inactivation and mutation induced by low energy protons in V79 cells: further results at the LNL facility.
	\emph{Int. J. Radiat. Biol.} \textbf{74} 501--509

	\bibitem[Berger \etal(2000)]{PSTAR}
	Berger MJ, Coursey JS, and Zucker MA 2000
	ESTAR, PSTAR, and ASTAR: Computer Programs for Calculating Stopping-Power and Range Tables for Electrons, Protons, and Helium Ions (version 1.2.2).
	Available: http://physics.nist.gov/Star. National Institute of Standards and Technology, Gaithersburg, MD.
	Originally published as: Berger MJ, NISTIR 4999, National Institute of Standards and Technology, Gaithersburg, MD (1993).

	\bibitem[Brahme \etal(2001)]{Brahme - Design}
	Brahme A, Lewensohn R, Ringborg U, Amaldi U, Gerardi F, Rossi S 2001
	Design of a centre for biologically optimised light ion therapy in Stockholm.
	\emph{Nucl. Instrum. Meth.} \textbf{B 184} 569-588.

	\bibitem[Cella \etal(2001)]{Lomax2}
	Cella L, Lomax A, Miralbell R 2001
	Potential role of intensity modulated proton beams in prostate cancer radiotherapy.
	\emph{Int. J. Radiat. Oncol. Biol. Phys.} \textbf{49} 217-223.

	\bibitem[Chu \etal(1993)]{Chu}
	Chu WT, Ludewigt BA, Renner TR 1993
	Instrumentation for treatment of cancer using proton and light-ion beams
	\emph{Rev. Sci. Instrum.} \textbf{64} 2055-2122

	\bibitem[Debus \etal(1998)]{Heidelberg}
	Debus J, Gross KD, Pavlovic M (Eds) 1998
	Proposal for a Dedicated Ion Beam Facility for Cancer Therapy. GSI, Darmstadt.

	\bibitem[Enghardt \etal(2004)]{PET}
	Enghardt W, Parodi K, Crespo P, Fiedler F, Pawelke J, Ponisch F 2004
	Dose quantification from in-beam positron emission tomography
	\emph{Radiother. Oncol.} \textbf{73} Suppl.\ 2, S96-S98.

	\bibitem[Folkard \etal(1996)]{Folkard96}
	Folkard M, Prise K M, Vojnovic B, Newman H C, Roper M J, Michael B D 1996
	Inactivation of V79 cells by low-energy protons, deuterons and helium-3 ions
	\emph{Int. J. Radiat. Biol.} \textbf{69} 729--738

	\bibitem[Goodhead \etal(1992)]{Goodhead}
	Goodhead DT, Belli M, Mill AJ, Bance DA, Allen LA, Hall SC, Ianzani F, Simone G, Stevens DL, Stretch A, Tabocchini MA, Wilkinson RE	1992
	Direct comparison between protons and alpha-particles of the same LET. I. Irradiation methods and inactivation of asynchronous V79, HeLa and C3H 10T1/2 cells
	\emph{Int. J. Radiat. Biol.} \textbf{61} 611--624

	\bibitem[Goodhead(1994)]{Goodhead-LMDS}
	Goodhead DT 1994
	Initial events in the cellular effects of ionizing-radiations - clustered damage in DNA
	\emph{Int. J. Radiat. Biol.} \textbf{65} 7-17

	\bibitem[Hrom\v{c}\'{\i}kov\'{a} \etal(2005)]{CHO-xrs}
	Hrom\v{c}\'{\i}kov\'{a} H, Kundr\'{a}t P, Lokaj\'{\i}\v{c}ek M 2005
	Analysis of damage induction and repair processes after carbon irradiation in different cell lines
	(in preparation)

	\bibitem[James(1994)]{MINUIT}
	James F 1994
	MINUIT Minimization package - Reference Manual
	\emph{CERN Program Library Long Writeup} D506, CERN Geneva

	\bibitem[Kagawa \etal (2002)]{Kagawa 2002}
	Kagawa K, Murakami M, Hishikawa Y, Abe M, Akagi T, Yanou T, Kagiya G, Furusawa Y, Ando K, Nojima K, Aoki M, Kanai T 2002
	Preclinical biological assessment of proton and carbon ion beams at Hyogo Ion Beam Medical Center.
	\emph{Int. J. Radiat. Oncol. Biol. Phys.} \textbf{54} 928--938

	\bibitem[Kanai \etal (1997)]{Kanai 1997}
	Kanai T, Furusawa Y, Fukutsu K, Itsukaichi H, Eguchi-Kasai K, Ohara H 1997
	Irradiation of Mixed Beam and Design of Spread-out Bragg Peak for Heavy-Ion Radiotherapy.
	\emph{Radiat. Res.} \textbf{147} 78--85

	\bibitem[Kanai \etal (1999)]{Kanai 1999}
	Kanai T, Endo M, Minohara S, Miyahara N, Koyama-Ito H, Tomura H, Matsufuji N, Futami Y, Fukumura A, Hiraoka T, Furusawa Y, Ando K, Suzuki M, Soga F, Kawachi K 1999
	Biophysical characteristics of HIMAC clinical irradiation system for heavy-ion radiation therapy.
	\emph{Int. J. Radiat. Oncol. Biol. Phys.} \textbf{44} 201--210

	\bibitem[Kr\"{a}mer \etal(2000)]{TRiP1}
	Kr\"{a}mer M, J\"{a}kel O, Haberer T, Kraft G, Schardt D, Weber U 2000
	Treatment planning for heavy-ion radiotherapy: physical beam model and dose optimization.
	\emph{Phys. Med. Biol.} \textbf{45} 3299-3317.

	\bibitem[Kr\"{a}mer and Scholz(2000)]{TRiP2}
	Kr\"{a}mer M, Scholz M 2000
	Treatment planning for heavy-ion radiotherapy: calculation and optimization of biologically effective dose
	\emph{Phys. Med. Biol.} \textbf{45} 3319-3330

	\bibitem[Krengli and Orecchia(2004)]{CNAO}
	Krengli M, Orecchia R 2004
	Medical aspects of the National Centre for Oncological Hadrontherapy (CNAO - Centro Nazionale Adroterapia Oncologica) in Italy.
	\emph{Radiother. Oncol.} \textbf{73} Suppl.\ 2, S21-S23.

	\bibitem[Kundr\'{a}t \etal(2005)]{PhysMedBiol2005}
	Kundr\'{a}t P, Lokaj\'{\i}\v{c}ek M, Hrom\v{c}\'{\i}kov\'{a} H 2005
	Probabilistic two-stage model of cell inactivation by ionizing particles
	\emph{Phys. Med. Biol.} \textbf{50} 1433-1447

	\bibitem[Kundr\'{a}t(2005)]{depth-survival}
	Kundr\'{a}t P 2005
	Towards biology-oriented hadrontherapy treatment planning
	(in preparation)

	\bibitem[Mayer \etal(2004)]{MedAustron}
	Mayer R, Mock U, Jager R, Potter R, Vutuc C, Eiter H, Krugmann K, Hammer J, Hirn B, Hawliczek R, Knocke-Abulesz TH, Lukas P, Nechville E, Pakisch B, Papauschek M, Wolfgang R, Rhomberg W, Sabitzer H, Schratter-Sehn A, Felix S, Wedrich I, Auberger T 2004
	Epidemiological aspects of hadron therapy: A prospective nationwide study of the Austrian project MedAustron and the Austrian Society of Radiooncology (OEGRO).
	\emph{Radiother. Oncol.} \textbf{73} Suppl.\ 2, S24-S28.

	\bibitem[Ottolenghi \etal(1997)]{Ottolenghi}
	Ottolenghi A, Merzagora M, Paretzke HG 1997
	DNA complex lesions induced by protons and alpha-particles: Track structure characteristics determining linear energy transfer and particle type dependence
	\emph{Radiat. Environ. Biophys.} \textbf{36} 97-103

	\bibitem[Perris \etal(1986)]{Perris}
	Perris A, Pialoglou P, Katsanos AA, Sideris EG 1986
	Biological effectiveness of low-energy protons. 1. Survival of Chinese-hamster cells
	\emph{Int. J. Radiat. Biol.} \textbf{50} 1093--1101

	\bibitem[Sachs and Brenner(1993)]{Sachs}
	Sachs RK, Brenner DJ 1993
	Effect of LET on chromosomal aberration yields. 1. Do long-lived, exchange-prone double-strand breaks play a role
	\emph{Int. J. Radiat. Biol.} \textbf{64} 677-688

	\bibitem[Schall \etal(1996)]{fragmentation}
	Schall I, Schardt D, Geissel H, Irnich H, Kankeleit E, Kraft G, Magel A, Mohar MF, Munzenberg G, Nickel F, Scheidenberger C, Schwab W 1996
	Charge-changing nuclear reactions of relativistic light-ion beams ($5 \leq Z \leq 10$) passing through thick absorbers
	\emph{Nucl. Instrum. Meth.} \textbf{B 117} 221-234

	\bibitem[Scholz and Kraft(1995)]{Kraft-Scholz-50}
	Scholz M, Kraft G 1995
	Track structure and the calculation of biological effects of heavy charged-particles
	\emph{Adv. Space Res.} \textbf{18} 5-14

	\bibitem[Schulz-Ertner \etal(2002)]{chordomas}
	Schulz-Ertner D, Haberer T, Jakel O, Thilmann C, Kramer M, Enghardt W, Kraft G, Wannenmacher M, Debus JR 2002
	Radiotherapy for chordomas and low-grade chondrosarcomas of the skull base with carbon ions
	\emph{Int. J. Radiat. Oncol. Biol. Phys.} \textbf{53} 36-42.

	\bibitem[Semenenko and Stewart (2004)]{Semenenko+Stewart}
	Semenenko VA, Stewart RD 2004
	A fast Monte Carlo algorithm to simulate the spectrum of DNA damages formed by ionizing radiation
	\emph{Radiat. Res.} \textbf{161} 451-457

	\bibitem[Sisterson(2005)]{Particles}
	Sisterson J (Ed) (2005)
	\emph{Particles Newsletter} 36.
	http://ptcog.web.psi.ch/archive.html

	\bibitem[Stoll \etal(1995)]{Kiefer}
	Stoll U, Schmidt A, Schneider E, Kiefer J 1995
	Killing and mutation of Chinese-hamster V79 cells exposed to accelerated oxygen and neon ions.
	\emph{Radiat. Res.} \textbf{142} 288-294

	\bibitem[Tsujii \etal(2004)]{Tsujii - PTCOG Paris 2004}
	Tsujii H, Mizoe J, Kamada T, Baba M, Kato S, Kato H, Tsuji H, Yamada S, Yasuda S, Ohno T, Yanagi T, Hasegawa A, Sugawara T, Ezawa H, Kandatsu S, Yoshikawa K, Kishimoto R, Miyamoto T 2004
	Overview of clinical experiences on carbon ion radiotherapy at NIRS.
	\emph{Radiother. Oncol.} \textbf{73} Suppl.\ 2, S41-S49

	\bibitem[Wambersie \etal(2004)]{ion-rationale}
	Wambersie A, Hendry J, Gueulette J, Gahbauer R, Potter D, Gregoire V 2004
	Radiobiological rationale and patient selection for high-LET radiation in cancer therapy
	\emph{Radiother. Oncol.} \textbf{73} Suppl.\ 2, S1-S14.

	\bibitem[Ward (1985)]{Ward}
	Ward JF 1985
	Biochemistry of DNA lesions
	\emph{Radiat. Res.} \textbf{104} S103-S111

	\bibitem[Weyrather \etal(1999)]{Wilma}
	Weyrather WK, Ritter S, Scholz M, Kraft G 1999
	RBE for carbon track-segment irradiation in cell lines of differing repair capacity
	\emph{Int. J. Radiat. Biol.} \textbf{75} 1357--1364

	\bibitem[Wilson (1946)]{Wilson}
	Wilson RR 1946
	Radiological use of fast protons.
	\emph{Radiology} \textbf{47} 487-491

	\bibitem[Ziegler(2004)]{SRIM}
	Ziegler JF 2004
	SRIM-2003
	\emph{Nucl. Instrum. Meth.} \textbf{B 219} 1027-1036

	\bibitem[Zurlo \etal(2000)]{Lomax1}
	Zurlo A, Lomax A, Hoess A, Bortfeld T, Russo M, Goitein G, Valentini V, Marucci L, Capparella R, Loasses A 2000
	The role of proton therapy in the treatment of large irradiation volumes: A comparative planning study of pancreatic and biliary tumors.
	\emph{Int. J. Radiat. Oncol. Biol. Phys.} \textbf{48} 277-288.

\endrefs

\end{document}

Medical Physics (physics.med-ph); Biological Physics (physics.bio-ph)


Automatic Image Stylization Using Deep Fully Convolutional Networks

Abstract: Color and tone stylization strives to enhance unique themes with artistic color and tone adjustments. It has a broad range of applications from professional image postprocessing to photo sharing over social networks. Mainstream photo enhancement softwares provide users with predefined styles, which are often hand-crafted through a trial-and-error process. Such photo adjustment tools lack a semantic understanding of image contents and the resulting global color transform limits the range of artistic styles it can represent. On the other hand, stylistic enhancement needs to apply distinct adjustments to various semantic regions. Such an ability enables a broader range of visual styles. In this paper, we propose a novel deep learning architecture for automatic image stylization, which learns local enhancement styles from image pairs. Our deep learning architecture is an end-to-end deep fully convolutional network performing semantics-aware feature extraction as well as automatic image adjustment prediction. Image stylization can be efficiently accomplished with a single forward pass through our deep network. Experiments on existing datasets for image stylization demonstrate the effectiveness of our deep learning architecture.

\section{Introduction}
\label{sec:intro}
Stylistic enhancement adjusts an image for enhancing artistic styles that convey unique themes. Unlike conventional image enhancement focusing on fixing photographic artifacts (under/over exposure, insufficient contrast, etc.), stylistic enhancement involves dramatic color and tone adjustments to achieve distinctive visual effects. For example, the \textit{X-PRO II} filter from mobile photo App Instagram expresses a wistful mood by simulating the cross processing procedure of photographic films. Professional image editing software (such as Adobe Lightroom) and social mobile Apps (such as Instagram) provide users with predefined styles, which are often hand-crafted through a trial-and-error process.

Conventional automatic photo adjustment has difficulty in representing complex color transforms between images before and after adjustment. Most of them merely model global color transforms without considering local semantic contexts. Although more sophisticated adjustments introduce spatially varying effects according to local image statistics, they still lack a semantic understanding of image contents. On the contrary, professional photographers often manually enhance images in a semantics-aware manner.
For instance, when enhancing photos to create a nostalgic theme, photographers might apply more exaggerated adjustments to a photo of \textit{Broadway} taken in year 1950 than a photo of \textit{Burj Khalifa}, which is the tallest skyscraper built in year 2010, as the former is more fitting with the theme.
At a local scale, they employ selection tools to isolate semantic regions (faces, buildings, etc.), which are enhanced with distinct sets of adjustments. For instance, there may exist a demand to apply exaggerated adjustments to foreground objects to help them stand out. The ability of applying distinct adjustments to semantic regions enables a broader range of visual styles.

As stylistic adjustments interact with image semantics and contexts in a complicated manner, it is challenging to manually define the relationships between them. To automatically learn stylistic enhancement from a small set of image exemplars, in this paper, we propose a novel deep learning architecture.
Unlike existing work that integrates hand-crafted features with a small-scale multilayer neural network~\cite{YanZW+16}, our solution is a large-scale deep network. It consists of a fully convolutional network (FCN) for automatic semantic feature extraction and $1$x$1$ convolutional layers for nonlinear transformation and adjustment prediction. Recently, fully convolutional networks~\cite{long2015fully,chen2016semantic,LiYu16} have proven to be efficient and powerful deep learning architectures for image processing and visual understanding tasks, such as semantic image segmentation, contour detection and salient object detection, that need to generate high-resolution outputs. In our deep network, feature maps with sufficiently large receptive fields are computed to model contexts. We further employ 1x1 convolutional layers, which predict color transforms according to contexts and pixel features. We seamlessly integrate the FCN with 1x1 convolutional layers, and an input image can be enhanced with a single forward pass in our deep network.

\begin{figure*}[t]
  \includegraphics[width=1.0\linewidth]{figs/graph_overview_icip.pdf}
  \caption{The architecture of our deep neural network for stylistic image enhancement.}
  \label{Fig:architecture}
\end{figure*}

\section{Related Work}
Researchers have made an enormous amount of effort to develop automatic or semi-automatic methods for tone adjustment~\cite{Twoscale_Tone:06}, color management~\cite{wang2010data}, and image smoothing~\cite{jointbilateral,BiHY15}.
In this section, we focus on the most relevant work in data-driven image enhancement.
Data-driven approaches are capable of learning new effects from examples, and thus offer a flexible set of adjustments.
Bychkovsky {\em et al.}~\cite{Bychkovsky} predict global tonal adjustments using a Gaussian process regression model built from a large dataset of images. Their regression model only extracts image global features and does not accommodate semantics-aware local adjustments. Kang {\em et al.}~\cite{PersonalizationSB:2010} introduce user preference in image global enhancement, and retouch a novel image by finding most similar examples in a database and transferring their tone and color adjustments.
Joshi {\em et al.}~\cite{joshi2010personal} retouch imperfect personal photos by leveraging existing high-quality photos of the same person.
Lee {\em et al.}~\cite{LS+16} develop an unsupervised technique for learning content-specific style rankings and transfers highly ranked styles from exemplars to an input photo. However, their styles are still limited to global color and tone transforms.
Wang {\em et al.}~\cite{wang2011example} approximate complex spatially varying tone and color adjustments with piecewise polynomial functions, which rely on low-level image statistics only and are not aware of image semantics.
Yan {\em et al.}~\cite{YanZW+16} present a method for local tone and color adjustments using a combination of image global statistics, contextual semantic features and pixelwise color and spatial features.
Shih {\em et al.} \cite{shih2013data} synthesize images associated with different times of day by learning locally affine models after locating a matching video within a time-lapse video database.
Gatys {\em et al.}~\cite{gatys2016image} perform image style transfer using convolutional neural networks. A new image is synthesized by matching the coarse structures of a content image and the texture features of a style image.

\section{Problem Definition}
Given a set of exemplar image pairs, each representing a photo before and after pixel-level color and tone adjustments following a particular style, we wish to learn a computational model that can automatically adjust a novel input photo in the same style. We cast this learning task as a regression problem as in \cite{YanZW+16}. For completeness, let us first review the original problem definition before presenting our new deep learning based architecture.

We seek a color transformation function $\phi$ such that, for every pixel $p_i$ in the exemplar images, the color transform returned by $\phi$ is $\phi(\theta,x_i)$, which maps the pixel color at $p_i$ before adjustment, $c_i={[L_i ~ a_i ~b_i]}^T$ (CIELab color space), to its corresponding pixel color $y_i$ after adjustment. Here $\theta$ denotes the model parameters and $x_i$ the feature vector at pixel $p_i$. The quadratic color basis $V(c_i)=[L_i^2 ~ a_i^2 ~ b_i^2 ~ L_ia_i ~L_ib_i~ a_ib_i ~L_i~ a_i ~ b_i~1]^T$ is used to absorb high-frequency pixelwise color variations. The product of the color transform $\phi(\theta,x_i)$ and the color basis is a prediction of the enhanced color. Since our color space has 3 channels and the quadratic color basis is a 10-dimensional vector, $\phi(\theta,x_i)$ is in fact a 3x10 matrix. The model parameters of $\theta$ are learnt by minimizing the following objective function, which measures the squared differences between the predicted and groundtruth enhanced colors.
\begin{equation}
\label{eq: cost function}
    \begin{split}
    \arg \min_{\theta} \sum_{i} {||\phi(\theta,x_i)V(c_i)-y_i||}^2
    \end{split}
\end{equation}

\section{Network Architecture}
The architecture of our proposed deep network is shown in Figure~\ref{Fig:architecture}. It relies on automatically learned features at the pixel and contextual levels. The {\em pixel feature} simply consists of all color channels at the considered pixel.
Due to the content-aware nature of our proposed tasks, our network has a fully convolutional sub-network dedicated for extracting contextual semantic features for every pixel in the input image. The final feature map from this sub-network is high-dimensional with 4096 channels, but its resolution is much lower, only one eighth of the resolution of the original input image. The {\em contextual} feature of a pixel consists of two components, which are deep CNN features extracted over two differently sized receptive fields centered at the pixel.

We upsample the input image by a factor of 2, and feed the original and upsampled images into two separate copies of the sub-network for semantic feature computation to extract two feature maps. These feature maps represent semantic contexts at two different scales.
When we compute the feature map from the upsampled image, the size of the receptive field is reduced from $224 \times 224$ to $112 \times 112$. This two-scale scheme is better suited for context modeling. To ensure meaningful features are extracted at image boundaries, we apply reflection padding to the original and upsampled images. Each of these two feature maps then goes through its own {\em interpolation} layer, which makes the feature map reach the full resolution of the original input.

The pixel feature and two-scale contextual feature at every pixel are concatenated and passed through two subsequent 1x1 convolutional layers with nonlinear ReLU activation before being used in the output layer (another 1x1 convolutional layer with 30 channels) to predict per-pixel color transforms. The predicted color transform at pixel $p_i$, $\phi(\theta,x_i)$, is multiplied with its corresponding quadratic color basis vector, $V(c_i)$, in an \textit{enhancement} layer to produce the final adjusted color at $p_i$. Here 1x1 convolutional layers with nonlinear activation are used to perform nonlinear feature transformation. Both pixel features and per-pixel color basis vectors are defined in the CIELab color space. The entire network presented here can be trained or fine-tuned.

Let us now elaborate on the fully convolutional sub-network for semantic feature computation.
Fully convolutional networks are introduced in \cite{long2015fully}, and can be set up by transforming a convolutional neural network (CNN).
The CNN can be pretrained on a large-scale dataset, such as ImageNet~\cite{deng2009imagenet}, to learn discriminative feature representations generically useful for a variety of tasks.
We transform the VGG-16 network~\cite{simonyan2014very} pretrained on ImageNet into a fully convolutional network.
The original VGG-16 network consists of 5 groups of convolutional layers (\textit{conv1}-\textit{conv5}), 5 pooling layers (\textit{pool1}-\textit{pool5}) and 2 fully connected layers (\textit{fc6} and \textit{fc7}). We replace layers \textit{fc6} and \textit{fc7} with 1x1 convolutional layers \textit{conv6} and \textit{conv7} while kernel parameters in \textit{conv6} and \textit{conv7} are copied from \textit{fc6} and \textit{fc7}, respectively. After this change, the spatial dimensions of the feature map at the deepest layer \textit{conv7} is reduced by a factor of 32 due to the 5 pooling layers. Then the stride of pooling kernels in layers \textit{pool4} and \textit{pool5} is reduced from $2$ to $1$. Thus the spatial resolution of feature maps is not reduced across these two layers. As a result, the overall spatial resolution is merely reduced by a factor of 8. Feature maps with high spatial resolutions have been shown to be vital in attaining superior performance in image processing~\cite{long2015fully}.
Changing the kernel stride at a pooling layer alone makes the following convolutional layer have a different receptive field and extract meaningless features.
We use \textit{dilated convolution}~\cite{yu2015multi} to increase the input stride of feature maps for layers \textit{conv5} and \textit{conv6} by factors of 2 and 4 respectively to compensate the change of pooling kernels in \textit{pool4} and \textit{pool5}. To obtain pixelwise semantic features, we fine-tune the above transformed network on the ADE20K dataset~\cite{ADE20Kdataset} with 150 object categories. The ADE20K dataset is specifically designed for scene parsing.
Finally, if an input image of size $H \times W \times 3$ is sent into this fine-tuned FCN, a feature map of size $\frac{H}{8} \times \frac{W}{8} \times 4096$ is extracted from layer \textit{conv7}.

There exist significant qualitative differences between our new deep learning based framework and the method in~\cite{YanZW+16}. First, our proposed framework here extracts semantic features automatically learned from raw data using a fully convolutional neural network while the method in \cite{YanZW+16} only makes use of traditional neural networks and hand-crafted features (e.g. intensity distribution, label histograms, etc). Automatically learned features are much more powerful than hand-crafted features.
Second, our entire deep network is end-to-end trainable and final results are predicted at the pixel level while the method in \cite{YanZW+16} can only train fully connected layers and final results are predicted at the superpixel level. Thus the method in \cite{YanZW+16} does not fully exploit the power of deep learning.
Third, to form a contextual feature descriptor for reflecting the local semantics and appearance around a pixel or superpixel, Yan  {\em et al.} \cite{YanZW+16} rely on a traditional scene parser~\cite{scene_parsing} to label background image regions, and a set of cascaded object detectors \cite{wang2013regionlets} to detect and classify foreground objects into a small set of predefined categories. By merging scene parsing results and object detections into a semantic label map, they compute label histograms using a multiscale spatial pooling scheme. However, both the scene parser and the object detectors they use have been substantially outperformed by recent deep learning models~\cite{long2015fully,ren2015faster,liu2015ssd,chen2016semantic}. Furthermore, in contrast to the continuous deep CNN features used in our contextual features, their category-oriented discrete label map is more prone to quantization errors, which can lead to severe artifacts in the final results.
Fourth, the fully convolutional network in our proposed new framework has an efficient GPU implementation. CNN feature computation in our new framework can be completed on the order of one second. In contrast, the scene parser and object detectors used in \cite{YanZW+16} are two orders of magnitude slower.

\begin{figure*}[!t]
  \centerline{
    \includegraphics[width=0.18\textwidth]{figs/82975753_7361722ac5_z-orig.png}\hfill
    \includegraphics[width=0.18\textwidth]{figs/82975753_7361722ac5_z-truth.png}\hfill
    \includegraphics[width=0.18\textwidth]{figs/82975753_7361722ac5_z-ours.png}\hfill
    \includegraphics[width=0.18\textwidth]{figs/82975753_7361722ac5_z-yan.png}\hfill
    \includegraphics[width=0.216\textwidth]{figs/scene_label_1.PNG}
  }
  \centerline{
    \includegraphics[width=0.18\textwidth]{figs/2773613631_7bcaa9260c_z-orig.png}\hfill
    \includegraphics[width=0.18\textwidth]{figs/2773613631_7bcaa9260c_z-truth.png}\hfill
    \includegraphics[width=0.18\textwidth]{figs/2773613631_7bcaa9260c_z-ours.png}\hfill
    \includegraphics[width=0.18\textwidth]{figs/2773613631_7bcaa9260c_z-yan.png}\hfill
    \includegraphics[width=0.216\textwidth]{figs/scene_label_2.PNG}
  }
  \centerline{\small \hfill Input Image \hfill\hfill Ground Truth \hfill\hfill Our Result \hfill\hfill Result from \protect\cite{YanZW+16} \hfill Scene Labels from \protect\cite{YanZW+16}
  }
  \caption{Comparisons with the method in \protect{\cite{YanZW+16}} on examples from the Foreground Popout style. In the top row, their method mislabels `sea' as `mountain' and the saturation of this mislabeled region is incorrectly increased. In the bottom row, although their method labels the `building' region correctly, it still adjusts its color incorrectly, which reveals the limitation of their feature description. In contrast, our proposed method produces much more robust visual results by using contextual semantic features extracted using deep fully convolutional networks.}
\label{fig:comp-yan}
\end{figure*}

\section{Experimental Results}
\subsection{Results on the Uniform Dataset}
Here we report image stylization results from our deep network on the Uniform dataset introduced in \cite{YanZW+16}. This dataset includes 115 images and three local enhancement styles. In the following, we briefly review these three styles.

The first style, ``\textbf{Foreground Pop-out}", increases the contrast and color saturation of foreground objects while decreasing the color saturation of background. In general, this style makes the foreground objects more salient and colorful while deemphasizing the background. The second style, ``\textbf{Local Xpro}", is more complex compared to the previous effect. It generalizes the popular ``cross processing" effect in a local manner. When creating this style, a photographer first defined multiple profiles in Photoshop, each of which is specifically tailored for a subset of semantic categories. All the profiles share a common series of operations, such as hue/saturation adjustment and brightness/contrast manipulation. Nonetheless, each profile defines a distinct set of adjustment parameters tailored for its corresponding semantic categories. Although the profiles roughly follow the ``cross processing" style, the choice of local profiles and additional minor image editing operations were heavily influenced by the photographer's personal taste which can be naturally learned through exemplars. The third style, ``\textbf{Watercolor}", gives viewers artistic impressions. It creates ``brush" effects over the input image. All pixels inside the region covered by a single brush stroke share the same color. This mimics the ``watercolor" painting style. This style also gives rise to complex spatially varying color adjustments.

We have trained our entire deep network from end to end to learn all three styles from this dataset. As in \cite{YanZW+16}, for each style, 70 images are used for training and the remaining 45 images are used for testing.
We have calculated the mean per-pixel $L^2$ distance in the CIELab color space between our enhanced images and the groundtruth results as well as between the input images and the groundtruth results. They are shown in Table~\ref{table:local compare}.

\begin{table}[!bt]
\centering
\caption{Mean per-pixel $L^2$ distances between input images and groundtruth adjusted images and mean per-pixel $L^2$ testing errors between automatically adjusted results and groundtruth results.}
\label{table:local compare}
\resizebox{0.5\textwidth}{!}
{
\renewcommand{\arraystretch}{1.5}
\begin{tabular}{|l|l|l|l|l|}
\hline
Style              & \begin{tabular}[l]{@{}l@{}}Groundtruth\\ $L^2$ distance\end{tabular} & Yan et al. \cite{YanZW+16} & Our Method \\ \hline
Foreground Pop-out & 13.86                                                                 & 7.08                            & 7.10       \\ \hline
Local Xpro         & 19.71                                                                 & 7.43                            & 6.98       \\ \hline
Watercolor         & 15.30                                                                 & 7.20                            & 6.88       \\ \hline
\end{tabular}
}
\end{table}

We primarily compare our proposed method here against the one in \cite{YanZW+16} because the method in \cite{YanZW+16} is the most recent work capable of performing spatially varying photo adjustment. It has been demonstrated that their method outperforms all earlier relevant techniques. Comparison of numerical errors on testing images are shown in the third and fourth columns of Table~\ref{table:local compare}.
Our method achieves much lower numerical errors on the Local Xpro and Watercolor styles, and comparable errors on the Foreground Pop-out style.

Two visual comparisons are shown in Fig~\ref{fig:comp-yan} on the {\em Foreground Pop-out} style.
First, contextual features in \cite{YanZW+16} are built upon hand-crafted features and discrete label maps, whose accuracy and reliability cannot match the performance of continuous deep features computed in the proposed method. Therefore, their stylized results are more likely to have artifacts due to incorrect labeling. One such example from the Foreground Popout style is shown in the top row of Fig~\ref{fig:comp-yan} , where `sea' is mislabeled as `mountain' with their method and the saturation of this mislabeled region is incorrectly increased.
Second, their method requires the definition of a set of semantic categories. When this set is limited (20 categories used in \cite{YanZW+16}), their contextual feature would not work well on testing images containing object categories beyond this predefined set. Another example from the Foreground Popout style is shown in the bottom row of Fig~\ref{fig:comp-yan}, where the saturation of `building' is not increased with their method even though it is labeled correctly. In contrast, contextual features extracted using our new FCN based architecture tend to be more robust.

\begin{table}[bt]
\centering
\caption{Statistics of Cold and Spirng styles. Middle: mean per-pixel $L^2$ distance between original images and the ground truth. Right: mean per-pixel $L^2$ distance between our enhanced results and the ground truth.}
\label{table: spring cold}
{
\renewcommand{\arraystretch}{1.5}
\begin{tabular}{|l|l|l|}
\hline
Style & \begin{tabular}[c]{@{}l@{}}Ground Truth ($L^2$ distance) \end{tabular} & Our Method \\ \hline
Cold   & 16.95                                                                 & 1.98       \\ \hline
Spring & 18.56                                                                 & 3.56       \\ \hline
\end{tabular}
}
\end{table}

\begin{figure}[!bt]
\captionsetup[subfigure]
  { labelformat=empty}
\centering
\subfloat[Input image]{\includegraphics[width=0.15\textwidth]{figs/4677364341_58a614bf9d_z-orig.png}}~
\subfloat[Spring effect]{\includegraphics[width=0.15\textwidth]{figs/4677364341_58a614bf9d_z-truth-spring.png}}~
\subfloat[Our result]{\includegraphics[width=0.15\textwidth]{figs/4677364341_58a614bf9d_z-ours-spring.png}}\\
\subfloat[Input image]{\includegraphics[width=0.15\textwidth]{figs/4677364341_58a614bf9d_z-orig.png}}~
\subfloat[Cold effect]{\includegraphics[width=0.15\textwidth]{figs/4677364341_58a614bf9d_z-truth-cold.png}}~
\subfloat[Our result]{\includegraphics[width=0.15\textwidth]{figs/4677364341_58a614bf9d_z-ours-cold.png}}
\caption{Examples of global `Spring' and `Cold' styles. \textbf{Left}: Input image. \textbf{Middle}: Ground truth. \textbf{Right}: Our result.}
\label{fig: spring cold}
\end{figure}

\subsection{Global Styles}
Our deep network can readily learn global image adjustment styles as well. We asked a photographer to create additional stylistic effects which are saved as ``action" files in Photoshop, each of which contains a series of operations such as saturation adjustment, tone adjustment, and curve tuning. An ``action" is globally applied to the original images in the Uniform dataset~\cite{YanZW+16} to form a set of image exemplars for a global enhancement style. It is also convenient for us to obtain from the Internet various stylistic ``action" files shared by photo retouching enthusiasts. Here we demonstrate model training and testing for two global effects as examples.

The first global effect (called ``Spring") gives viewers the feeling of vitality, and the second global effect (called ``Cold") tends to make a photo look vintage. Fig~\ref{fig: spring cold} shows enhancement examples in these two styles.
The mean per-pixel $L^2$ distance between the original images and their groundtruth enhanced images reaches 16.95 and 18.56 for the ``Cold" and ``Spring" styles, respectively. This indicates that these two effects make significant color changes to the original images. The small testing errors of our trained models shown in Table~\ref{table: spring cold} numerically demonstrate the strength of our method in learning such global effects. Comparing the middle column and the right column in Fig~\ref{fig: spring cold}, we can see that the enhanced results from our model look nearly the same as the ground truth.

\section{Conclusions}
In this paper, we have presented a novel deep learning architecture for exemplar-based image stylization, which learns local enhancement styles from image pairs. Our deep learning architecture is an end-to-end deep fully convolutional network performing semantics-aware feature extraction as well as automatic image adjustment prediction. Image stylization can be efficiently accomplished with a single forward pass through our deep network. Experiments on existing datasets for image stylization demonstrate the effectiveness of our deep network.

\bibliographystyle{IEEEbib}

\section{Foreground Popout effect}
\label{sec:popout}

\begin{figure}[H]
\captionsetup[subfigure]{labelformat=empty}
\centering
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0087.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/popout_images/0087.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/popout_images/0087.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0092.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/popout_images/0092.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/popout_images/0092.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0095.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/popout_images/0095.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/popout_images/0095.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0071.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/popout_images/0071.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/popout_images/0071.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0113.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/popout_images/0113.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/popout_images/0113.png}}\\
\caption{Examples of Foreground Popout effect.
\textbf{First column:} Input image. \textbf{Second column:} Ground truth.
\textbf{Third column:} Our result.}
\label{fig:popout2}
\end{figure}

\section{Local Xpro effect}
\label{sec:xpro}

\begin{figure}[H]
\captionsetup[subfigure]{labelformat=empty}
\centering
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0089.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/xpro_images/0089.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/xpro_images/0089.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0093.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/xpro_images/0093.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/xpro_images/0093.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0097.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/xpro_images/0097.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/xpro_images/0097.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0100.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/xpro_images/0100.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/xpro_images/0100.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0104.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/xpro_images/0104.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/xpro_images/0104.png}}\\
\caption{Examples of Local Xpro effect.
\textbf{First column:} Input image. \textbf{Second column:} Ground truth.
\textbf{Third column:} Our result.}
\label{fig:xpro2}
\end{figure}

\section{Watercolor effect}
\label{sec:watercolor}

\begin{figure}[H]
\captionsetup[subfigure]{labelformat=empty}
\centering
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0074.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/watercolor_images/0074.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/watercolor_images/0074.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0090.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/watercolor_images/0090.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/watercolor_images/0090.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0082.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/watercolor_images/0082.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/watercolor_images/0082.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0101.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/watercolor_images/0101.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/watercolor_images/0101.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0115.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/watercolor_images/0115.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/watercolor_images/0115.png}}\\
\caption{Examples of Watercolor effect.
\textbf{First column:} Input image. \textbf{Second column:} Ground truth.
\textbf{Third column:} Our result.}
\label{fig:watercolor2}
\end{figure}

\section{Comparison with [Yan {\em et al.} 2016] on Foreground Popout effect}
\label{sec:compare1}

\begin{figure}[H]
\captionsetup[subfigure]{labelformat=empty}
\centering
\subfloat{\includegraphics[width=0.24\textwidth]{orig_images/0095.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{groundtruth/popout_images/0095.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{enhanced/popout_images/0095.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{enhanced_by_zhicheng/popout_images/0095.png}}\\
\subfloat{\includegraphics[width=0.24\textwidth]{orig_images/0071.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{groundtruth/popout_images/0071.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{enhanced/popout_images/0071.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{enhanced_by_zhicheng/popout_images/0071.png}}\\
\subfloat{\includegraphics[width=0.24\textwidth]{orig_images/0113.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{groundtruth/popout_images/0113.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{enhanced/popout_images/0113.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{enhanced_by_zhicheng/popout_images/0113.png}}\\
\caption{Comparison with [Yan {\em et al.} 2016] on Foreground Popout effect.
\textbf{First column:} Input image. \textbf{Second column:} Ground truth.
\textbf{Third column:} Our result. \textbf{Fourth column:} result of [Yan {\em et al.} 2016] }
\label{fig:watercolor1}
\end{figure}

\clearpage
\section{Comparison with [Yan {\em et al.} 2016] on Local Xpro effect}
\label{sec:compare2}

\begin{figure}[H]
\captionsetup[subfigure]{labelformat=empty}
\centering
\subfloat{\includegraphics[width=0.24\textwidth]{orig_images/0088.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{groundtruth/xpro_images/0088.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{enhanced/xpro_images/0088.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{enhanced_by_zhicheng/xpro_images/0088.png}}\\
\subfloat{\includegraphics[width=0.24\textwidth]{orig_images/0104.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{groundtruth/xpro_images/0104.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{enhanced/xpro_images/0104.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{enhanced_by_zhicheng/xpro_images/0104.png}}\\
\caption{Comparison with [Yan {\em et al.} 2016] on Local Xpro effect.
\textbf{First column:} Input image. \textbf{Second column:} Ground truth.
\textbf{Third column:} Our result. \textbf{Fourth column:} result of [Yan {\em et al.} 2016] }
\label{fig:watercolor1}
\end{figure}

\section{Comparison with [Yan {\em et al.} 2016] on Watercolor effect}
\label{sec:compare3}

\begin{figure}[H]
\captionsetup[subfigure]{labelformat=empty}
\centering
\subfloat{\includegraphics[width=0.24\textwidth]{orig_images/0101.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{groundtruth/watercolor_images/0101.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{enhanced/watercolor_images/0101.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{enhanced_by_zhicheng/watercolor_images/0101.png}}\\
\subfloat{\includegraphics[width=0.24\textwidth]{orig_images/0077.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{groundtruth/watercolor_images/0077.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{enhanced/watercolor_images/0077.png}}~
\subfloat{\includegraphics[width=0.24\textwidth]{enhanced_by_zhicheng/watercolor_images/0077.png}}\\
\caption{Comparison with [Yan {\em et al.} 2016] on Watercolor effect.
\textbf{First column:} Input image. \textbf{Second column:} Ground truth.
\textbf{Third column:} Our result. \textbf{Fourth column:} result of [Yan {\em et al.} 2016] }
\label{fig:watercolor1}
\end{figure}

\clearpage
\section{Cold effect}
\label{sec:cold}

\begin{figure}[H]
\captionsetup[subfigure]{labelformat=empty}
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0071.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/cold_images/0071.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/cold_images/0071.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0114.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/cold_images/0114.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/cold_images/0114.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0113.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/cold_images/0113.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/cold_images/0113.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0112.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/cold_images/0112.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/cold_images/0112.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0111.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/cold_images/0111.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/cold_images/0111.png}}\\
\caption{Examples of Cold effect.
\textbf{First column:} Input image. \textbf{Second column:} Ground truth.
\textbf{Third column:} Our result.}
\label{fig:popout1}
\end{figure}

\clearpage

\section{Spring effect}
\label{sec:spring}

\begin{figure}[H]
\captionsetup[subfigure]{labelformat=empty}
\centering
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0095.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/spring_images/0095.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/spring_images/0095.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0096.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/spring_images/0096.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/spring_images/0096.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0097.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/spring_images/0097.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/spring_images/0097.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0098.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/spring_images/0098.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/spring_images/0098.png}}\\
\subfloat{\includegraphics[width=0.3\textwidth]{orig_images/0099.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{groundtruth/spring_images/0099.png}}~
\subfloat{\includegraphics[width=0.3\textwidth]{enhanced/spring_images/0099.png}}\\
\caption{Examples of Spring effect.
\textbf{First column:} Input image. \textbf{Second column:} Ground truth.
\textbf{Third column:} Our result.}
\label{fig:popout1}
\end{figure}

\clearpage

\end{document}
\section{Introduction}
\label{sec:intro}

These guidelines include complete descriptions of the fonts, spacing, and
related information for producing your proceedings manuscripts. Please follow
them and if you have any questions, direct them to Conference Management
Services, Inc.: Phone +1-979-846-6800 or email
to \\\texttt{ip17@securecms.com}.

\section{Formatting your paper}
\label{sec:format}

All printed material, including text, illustrations, and charts, must be kept
within a print area of 7 inches (178 mm) wide by 9 inches (229 mm) high. Do
not write or print anything outside the print area. The top margin must be 1
inch (25 mm), except for the title page, and the left margin must be 0.75 inch
(19 mm).  All {\it text} must be in a two-column format. Columns are to be 3.39
inches (86 mm) wide, with a 0.24 inch (6 mm) space between them. Text must be
fully justified.

\section{PAGE TITLE SECTION}
\label{sec:pagestyle}

The paper title (on the first page) should begin 1.38 inches (35 mm) from the
top edge of the page, centered, completely capitalized, and in Times 14-point,
boldface type.  The authors' name(s) and affiliation(s) appear below the title
in capital and lower case letters.  Papers with multiple authors and
affiliations may require two or more lines for this information. Please note
that papers should not be submitted blind; include the authors' names on the
PDF.

\section{TYPE-STYLE AND FONTS}
\label{sec:typestyle}

To achieve the best rendering both in printed proceedings and electronic proceedings, we
strongly encourage you to use Times-Roman font.  In addition, this will give
the proceedings a more uniform look.  Use a font that is no smaller than nine
point type throughout the paper, including figure captions.

In nine point type font, capital letters are 2 mm high.  {\bf If you use the
smallest point size, there should be no more than 3.2 lines/cm (8 lines/inch)
vertically.}  This is a minimum spacing; 2.75 lines/cm (7 lines/inch) will make
the paper much more readable.  Larger type sizes require correspondingly larger
vertical spacing.  Please do not double-space your paper.  TrueType or
Postscript Type 1 fonts are preferred.

The first paragraph in each section should not be indented, but all the
following paragraphs within the section should be indented as these paragraphs
demonstrate.

\section{MAJOR HEADINGS}
\label{sec:majhead}

Major headings, for example, "1. Introduction", should appear in all capital
letters, bold face if possible, centered in the column, with one blank line
before, and one blank line after. Use a period (".") after the heading number,
not a colon.

\subsection{Subheadings}
\label{ssec:subhead}

Subheadings should appear in lower case (initial word capitalized) in
boldface.  They should start at the left margin on a separate line.

\subsubsection{Sub-subheadings}
\label{sssec:subsubhead}

Sub-subheadings, as in this paragraph, are discouraged. However, if you
must use them, they should appear in lower case (initial word
capitalized) and start at the left margin on a separate line, with paragraph
text beginning on the following line.  They should be in italics.

\section{PRINTING YOUR PAPER}
\label{sec:print}

Print your properly formatted text on high-quality, 8.5 x 11-inch white printer
paper. A4 paper is also acceptable, but please leave the extra 0.5 inch (12 mm)
empty at the BOTTOM of the page and follow the top and left margins as
specified.  If the last page of your paper is only partially filled, arrange
the columns so that they are evenly balanced if possible, rather than having
one long column.

In LaTeX, to start a new column (but not a new page) and help balance the
last-page column lengths, you can use the command ``$\backslash$pagebreak'' as
demonstrated on this page (see the LaTeX source below).

\section{PAGE NUMBERING}
\label{sec:page}

Please do {\bf not} paginate your paper.  Page numbers, session numbers, and
conference identification will be inserted when the paper is included in the
proceedings.

\section{ILLUSTRATIONS, GRAPHS, AND PHOTOGRAPHS}
\label{sec:illust}

Illustrations must appear within the designated margins.  They may span the two
columns.  If possible, position illustrations at the top of columns, rather
than in the middle or at the bottom.  Caption and number every illustration.
All halftone illustrations must be clear black and white prints.  Colors may be
used, but they should be selected so as to be readable when printed on a
black-only printer.

Since there are many ways, often incompatible, of including images (e.g., with
experimental results) in a LaTeX document, below is an example of how to do
this \cite{Lamp86}.

\section{FOOTNOTES}
\label{sec:foot}

Use footnotes sparingly (or not at all!) and place them at the bottom of the
column on the page on which they are referenced. Use Times 9-point type,
single-spaced. To help your readers, avoid using footnotes altogether and
include necessary peripheral observations in the text (within parentheses, if
you prefer, as in this sentence).

\begin{figure}[htb]

\begin{minipage}[b]{1.0\linewidth}
  \centering
  \centerline{\includegraphics[width=8.5cm]{image1}}
  \centerline{(a) Result 1}\medskip
\end{minipage}
\begin{minipage}[b]{.48\linewidth}
  \centering
  \centerline{\includegraphics[width=4.0cm]{image3}}
  \centerline{(b) Results 3}\medskip
\end{minipage}
\hfill
\begin{minipage}[b]{0.48\linewidth}
  \centering
  \centerline{\includegraphics[width=4.0cm]{image4}}
  \centerline{(c) Result 4}\medskip
\end{minipage}
\caption{Example of placing a figure with experimental results.}
\label{fig:res}
\end{figure}

\section{COPYRIGHT FORMS}
\label{sec:copyright}

You must include your fully completed, signed IEEE copyright release form when
form when you submit your paper. We {\bf must} have this form before your paper
can be published in the proceedings.

\section{REFERENCES}
\label{sec:ref}

List and number all bibliographical references at the end of the
paper. The references can be numbered in alphabetic order or in
order of appearance in the document. When referring to them in
the text, type the corresponding reference number in square
brackets as shown at the end of this sentence \cite{C2}. An
additional final page (the fifth page, in most cases) is
allowed, but must contain only references to the prior
literature.

\bibliographystyle{IEEEbib}

Computer Vision and Pattern Recognition (cs.CV)