Radio-to-UV monitoring of AO 0235+164 by the WEBT and Swift during the 2006--2007 outburst

Abstract: The blazar AO 0235+164 was claimed to show a quasi-periodic behaviour in the radio and optical bands. Moreover, an extra emission component contributing to the UV and soft X-ray flux was detected, whose nature is not yet clear. A predicted optical outburst was observed in late 2006/early 2007. We here present the radio-to-optical WEBT light curves during the outburst, together with UV data acquired by Swift in the same period. We found the optical outburst to be as strong as the big outbursts of the past: starting from late September 2006, a brightness increase of 5 mag led to the outburst peak in February 19-21, 2007. We also observed an outburst at mm and then at cm wavelengths, with an increasing time delay going toward lower frequencies during the rising phase. Cross-correlation analysis indicates that the 1 mm and 37 GHz flux variations lagged behind the R-band ones by about 3 weeks and 2 months, respectively. These short time delays suggest that the corresponding jet emitting regions are only slightly separated and/or misaligned. In contrast, during the outburst decreasing phase the flux faded contemporaneously at all cm wavelengths. This abrupt change in the emission behaviour may suggest the presence of some shutdown mechanism of intrinsic or geometric nature. The behaviour of the UV flux closely follows the optical and near-IR one. By separating the synchrotron and extra component contributions to the UV flux, we found that they correlate, which suggests that the two emissions have a common origin.

\section{Introduction}

\object{AO 0235+164} at redshift $z=0.94$ is one of the best-studied BL Lac objects.
The analysis of its radio and optical light curves extending over 25 years led \citet{rai01}
to suggest a quasi-periodic occurrence of the main outbursts every $5.7 \pm 0.5$ years.
The next outburst was predicted around February--March 2004, and a multiwavelength campaign was organised by the Whole Earth Blazar Telescope (WEBT)\footnote{{\tt http://www.to.astro.it/blazars/webt/} \\ \citep[see e.g.][]{vil06,vil07,boe07,rai07b}} to follow the expected event.
The radio-to-optical observations by the WEBT were complemented by the optical--UV and X-ray data acquired by the XMM-Newton satellite during 3 pointings in January and August 2004, and January 2005, and by optical spectroscopic observations with the 3.6 m Telescopio Nazionale Galileo (TNG).
The results of this intense observing effort were published by (\citealt{rai05,rai06b,rai06a,rai07a}; see also \citealt{hag07b} for an analysis of colour variability).
The predicted outburst was not observed, and time-series analysis on the light curves extended to 2005 revealed a possible characteristic variability time scale of $\sim 8$ years.
Moreover, the XMM-Newton observations suggested the presence of an extra emission component in the source spectral energy distribution (SED), in addition to the synchrotron and inverse-Compton ones.
The origin of this component, peaking in the UV/soft X-ray frequency range, was ascribed either to thermal emission from an accretion disc, or to synchrotron emission from an inner jet region.

An increased radio activity was detected in the 2005--2006 observing season \citep{bac07}, and a
dramatic optical brightening was finally observed in late 2006 -- early 2007.
This triggered a new WEBT multiwavelength campaign as well as ToO observations by the Swift satellite.
In this paper, we present the radio-to-optical observations performed from spring 2005 (i.e.\ the end of the period considered
in \citealt{rai06b}) to October 2007,
and the data acquired by the UltraViolet and Optical Telescope (UVOT) instrument onboard Swift.
X-ray data from the Swift X-ray Telescope (XRT) and Burst Alert Telescope (BAT) instruments will be presented in another paper (Kadler et al., in preparation).

This paper is organised as follows:
the procedures we adopted to treat the WEBT and UVOT data are outlined in Sect.~2, with particular attention paid to the subtraction of the southern AGN contribution from the optical and UV photometry.
The multifrequency light curves are presented and discussed in Sect.~3 while in Sect.~4 time lags among variations at different frequencies are derived by means of statistical analysis.
Spectral energy distributions with simultaneous data from near-IR to UV are constructed in Sect.~5, with the aim of separating the synchrotron and the extra emission components and of understanding their relationship. Finally, the results of our work are discussed in Sect.~6.

\section{Data reduction and analysis}

\subsection{Radio-to-optical observations}

   \begin{figure*}
   \sidecaption
   \includegraphics[width=12cm]{ottico-nir.eps}
      \caption{Optical and near-IR light curves of AO 0235+164 in the 2006--2007 observing season.
The optical magnitudes are corrected for the ELISA contribution.
Maximum and minimum brightness levels are highlighted with horizontal dotted lines.}
         \label{ottico-nir}
   \end{figure*}

The new WEBT campaign on AO 0235+164 saw a wide international participation.
Data were collected starting from the end of the period considered in \citet{rai06b}, i.e.\ from spring 2005 to October 2007.
Table \ref{obs} shows the list of participating observatories, the size of the telescopes,
and the bands/frequencies at which we acquired the data.

We collected optical and near-IR data as instrumental magnitudes of the source and comparison stars
in the same field.
Magnitude calibration was performed using preferably Stars 1, 2, and 3 by \citet{smi85} in the optical bands\footnote{In some cases, we also used Stars 9, 10, and 11 by \citet{gon01}.}, and the \citet{gon01} photometric sequence in the near-IR.
Optical light curves were carefully assembled and cleaned \citep[see e.g.][]{vil02}.

We collected radio data as calibrated flux densities; no systematic offset among datasets provided by different observers was evident;
data binning was needed in a few cases to reduce the noise.
We also included data from the VLA/VLBA Polarization Calibration Database\footnote{\tt http://www.vla.nrao.edu/astro/calib/polar/}
(PCD) in our light curves.

\begin{table}
\caption{Ground-based observatories participating in this work.}
\label{obs}
\centering
\begin{tabular}{l r c  }
\hline\hline
Observatory    & Tel.\ size     & Bands\\%               & $N_{\rm obs}$\\
\hline
\multicolumn{3}{c}{\it Radio}\\
Crimean (RT-22), Ukraine & 22 m          & 22, 37 GHz          \\%lat=+44:23:52.6, long=2h 15m 55.1s
Mauna Kea (SMA), USA     &$8 \times 6$ m$^a$ & 0.85, 1 mm      \\
Medicina, Italy          & 32 m          & 5, 8, 22 GHz        \\
Mets\"ahovi, Finland     & 14 m          & 37 GHz              \\
Noto, Italy              & 32 m          & 43 GHz              \\
UMRAO, USA               & 26 m          & 5, 8, 14.5 GHz      \\
\hline
\multicolumn{3}{c}{\it Near-infrared}\\
Campo Imperatore, Italy  & 110 cm        & $J, H, K$          \\
Roque (NOT), Spain       & 256 cm        & $J, H, K$          \\
\hline
\multicolumn{3}{c}{\it Optical}\\
Abastumani, Georgia      &  70 cm        & $R$                \\%Long=+34.0125 deg, Lat=+44.7266 deg, Alt=650 m
Crimean, Ukraine         &  70 cm        & $B, V, R, I$       \\
Greve, Italy             &  32 cm        & $R$                \\
Hankasalmi, Finland      &  40 cm        & $V, R, I$          \\
Jakokoski, Finland       &  50 cm        & $R$                \\%Long=+29.9969 deg, Lat=+62.7275 deg, Alt=155 m
Kitt Peak (MDM), USA     & 130 cm        & $B, V, R, I$       \\
L'Ampolla, Spain         &  36 cm        & $R$                \\%Long=+0.670136 deg, Lat=+40.807 deg, Alt=75 m
Michael Adrian, Germany  & 120 cm        & $R$                \\%Long=+8.4114 deg, Lat=+49.9254, Alt=103 m
New Millennium, Italy    &  36 cm        & $R$                \\
Roque (KVA), Spain       &  35 cm        & $R$                \\
Roque (NOT), Spain       & 256 cm        & $U, B, V, R, I$    \\
Rozhen, Bulgaria         & 50/70 cm      & $B, V, R, I$       \\%Long=+24.7439 deg, Lat=+41.6931, Alt=1759 m
Sabadell, Spain          &  50 cm        & $R$                \\
Sobaeksan, South Korea   &  61 cm        & $V, R, I$          \\%Long=+128 27 27.36, Lat=+36 56 03.89, Alt=1378 m
St.\ Petersburg, Russia  &  40 cm        & $V, R, I$          \\
Torino, Italy            & 105 cm        & $R, I$             \\
Toyonaka, Japan          &  35 cm        & $B, V, R, I$       \\
Tuorla, Finland          & 103 cm        & $R$                \\%Long=+22.17 deg, Lat=+60.27.
Valle d'Aosta, Italy     &  81 cm        & $V, R, I$          \\%Long=+7.47833 deg, Lat=+45.7895 deg, Alt=1600 m
Yunnan, China            & 102 cm        & $R$                \\
\hline
\multicolumn{3}{l}{$^a$ Radio interferometer including 8 dishes of 6 m size.}
\end{tabular}
\end{table}

\subsection{UV observations by Swift}

The Swift satellite observed the source from January 28 to February 26, 2007.
The UVOT instrument \citep{rom05} acquired data in the $U$, UV$W1$, UV$M2$, and UV$W2$ filters, which
were processed with the {\tt uvotmaghist} task of the HEASOFT 6.3 package.
Following the recommendations contained in the release notes, we used an aperture radius of 5 arcsec
in agreement with the standard photometric aperture defined in the calibration files (CALDB updated as July 2007).
Background was extracted in an annulus centred on the source, with inner and outer radii of 8 and 18 arcsec, respectively.

\subsection{Correction for ELISA contribution}

A major point in the analysis of the optical and UV radiation coming from AO 0235+164 is to eliminate the contamination from the southern AGN, named ELISA by \citet{rai05}.
Since it lies only 2 arcsec south of the source and it is very faint \citep{nil96}, this object is not resolved from AO 0235+164 in most optical frames as well as in the UVOT ones.
Consequently, it gives a contribution to the AO 0235+164 photometry, which affects more the blue than the red part of the optical--UV spectrum and is stronger when the blazar is fainter.
To correct for this effect, \citet{rai05} estimated the $UBVRI$ magnitudes of ELISA from images taken at medium-size telescopes participating in the 2003--2004 WEBT campaign.
More recently, \citet{rai07a} analysed an ELISA optical spectrum taken at the 8 m Very Large Telescope (VLT). The $UBVRI$ magnitudes derived from this spectrum confirm the
previous photometry results and lower the uncertainties to 0.05 mag; only in the case of the $V$ filter a slightly fainter value appears more appropriate.
Hence, to subtract the ELISA contribution from the optical fluxes,
we adopted the following ELISA magnitudes:
$U=20.80$, $B=21.40$, $V=20.95$, $R=20.50$, and $I=19.90$.
Zero-magnitude fluxes were taken from \citet{bes98}.

We then estimated the ELISA contribution in the UV$W1$, UV$M2$, and UV$W2$ filters from the HST/FOS composite spectrum published by \citet{bur96}\footnote{We note that the flux in the UV$W1$ band is affected by the contribution of the \ion{C}{iii}] emission line, and that the HST/FOS UV spectrum nicely reconnects in flux with the VLT optical spectrum.}.
We subtracted these ELISA flux densities from the total flux densities obtained with
{\tt uvotmaghist} and converted the results into magnitudes with zero-magnitude fluxes of
$3.963, \, 4.544, \, 5.400 \, \times 10^{-9} \rm \, erg \, cm^{-2} \, s^{-1} \, \AA^{-1}$.
UV magnitudes of ELISA are: 20.45, 20.50, and 20.65 in the UV$W1$, UV$M2$, and UV$W2$ bands,
respectively, again with a 0.05 mag uncertainty.

As in the previous works, the ELISA contamination in the near-IR is assumed to be negligible,
and null in the mm and cm bands.

   \begin{figure}
   \resizebox{\hsize}{!}{\includegraphics{uvot_clean.eps}}
      \caption{UVOT light curves of AO 0235+164 compared to the ground-based $U$ data from the
NOT (red diamonds) and to the $R$-band data (bottom panel).
All data have been corrected for the ELISA contribution.
The cubic spline interpolation through the 1-day binned $R$-band light curve is shown in each panel,
properly rescaled (see text).}
         \label{uvot_clean}
   \end{figure}

\section{Light curves behaviour}

The final light curves are shown in Figs.\ \ref{ottico-nir}--\ref{radop}.
The optical and UV magnitudes have been corrected for the ELISA contribution,
as explained in the previous section.

Figure \ref{ottico-nir} displays the optical and near-IR light curves of the 2006--2007 observing season.
Notwithstanding the differences in time sampling, the general trend is similar at all wavelengths, as expected.
At the beginning of the observing season the source was in a faint state, but
by in late September the source brightness started to increase.
We observed a fast flare to peak at JD = 2454077.4 (December 7, 2006).
It was characterized by a 2.0 mag brightening in 4 days in the $R$ band, and
by noticeable intranight variability \citep[see also][]{hag07a}. After the drop of 1.5 mag in 3 days,
the source brightness continued to rise smoothly, with minor flares superposed.
The outburst culminated on JD = 2454151--3 (February 19--21, 2007). This peak is well sampled in the
optical bands, but was missed by the near-IR observations, which registered the brightest level some days later,
when the optical flux was already in a decreasing phase.
We notice that in this observing season the variability amplitude in the best-sampled $R$ band was $\sim 5$ mag.

Observations by the Swift satellite were triggered in late January during the culminating phase of the outburst.
The $U$ and UV light curves from UVOT are shown in Fig.\ \ref{uvot_clean} as blue circles, and compared to the $R$-band light curve. In the $U$-band panel, ground-based data from the Nordic Optical Telescope (NOT) were plotted as red diamonds. UVOT data have been binned daily, and a lower limit to the error of 0.1 mag has been adopted to take into account statistical and systematic errors.
The grey line in each panel represents a cubic spline interpolation through the 1-day binned $R$-band light curve
(rescaled according to average colour indices: $U-R = 2.1$; $W1-R = 2.1$; $M2-R = 2.3$; and $W2-R = 2.7$).
It helps to recognize that the trend of the $R$-band light curve is well followed by all the UVOT light curves.
Also, the variability amplitude appears nearly the same in the optical and UVOT bands.

The increase of the optical and near-IR fluxes was followed soon after by an increase of the millimetric and then of the centimetric fluxes, with a progressive time delay going toward longer wavelengths.
Figure \ref{radop} shows the radio light curves starting from the end of the observing period presented in \citet{rai06b}, compared to the $R$-band one (top panel).
Optical observations in the 2005--2006 season were rather sparse;
they show moderate source activity at a level similar to that observed in the previous season \citep{rai06b}.
The figure also shows that, after the solar conjunction in 2007, the source was again in a very faint optical state.

The second panel from the top displays 1 mm and 850 $\mu$m data from the SubMillimeter Array (SMA).
A cubic spline interpolation through the 10-day binned 1 mm light curve is plotted to better follow the behaviour of the source flux at this wavelength.
The 1 mm flux density increased by a factor $\sim 8$ from the beginning of the 2006--2007 season to the time of the observed maximum, on March 6, 2007. After this date, the only data point
acquired before the solar conjunction epoch suggests that the outburst peak was already over.
Moreover, data after solar conjunction show that the mm flux density had reduced by more than a factor of 2.
We notice that a first hump in the 1 mm light curve, around JD = 2454100, may be correlated to the fast optical flare observed at JD $\sim$ 2454080.

In the following panels, radio light curves at lower and lower frequencies are displayed.
We have plotted the cubic spline interpolation through the 20-day binned light curve at 37 GHz, the best-sampled one, to allow an easier comparison of the flux density behaviour at different wavelengths.
In general, the radio spectrum appears ``inverted": the flux density is higher at higher frequencies. This is not true in the period (JD $\sim$ 2453900--2454050) leading to the minimum flux levels, where the flux density appears higher at longer wavelengths, and during the outburst decreasing phase, where the radio spectrum seems to be flat.

The radio emission had already undergone a rather bright phase in 2005--2006, a kind of ``minor outburst" like those often observed in the past, which sometimes are accompanied by noticeable optical events and sometimes not.
A flux minimum was reached around JD = 2454000--2454100, and then the flux started to increase again, all these variations occurring first at the higher and then at the lower frequencies.
However, the 22 GHz flux density reached a maximum value similar to that observed at 37 GHz almost at the same time, and the flux decrease was also similar and contemporaneous. This is highlighted in the fourth panel of Fig.\ \ref{radop} by the overlapping of the spline through the 37 GHz data with that through the 22 GHz ones.
The lower-frequency light curves are more sparsely sampled, but they too suggest a progressively delayed rising phase, and a contemporaneous decreasing phase.

While the general multifrequency behaviour during the outburst seems in agreement with the predictions of shock-in-jet models \citep[see e.g.][]{hug89,val92}, the decay stage might suggest a more complex picture.
Indeed, it seems as if a kind of ``shutdown" occurred when the perturbation that was responsible for the flux increase reached the jet region emitting at cm wavelengths. The shape of the 37 GHz outburst, sharp and without a plateau, may indicate that the outburst observed at this frequency has already been affected by the shutdown.
The signature of the shutdown would be mainly visible in the 22 GHz light curve, which is characterized by an abrupt flux decay.

   \begin{figure*}
   \sidecaption
   \includegraphics[width=12cm]{radop.eps}
      \caption{Comparison among the $R$-band magnitudes ({\it top panel}),
corrected for the ELISA contribution,
and the radio flux densities (Jy) at different frequencies.
Cubic spline interpolations through the 10-day binned light
curves in the $R$ and 1 mm bands have been drawn in the first and second panel, respectively.
Similarly, the cubic spline interpolation through the 20-day binned light curve at 37 GHz has been drawn in the other panels, to highlight the time delays between the flux variations at 37 GHz and those at lower frequencies. In the fourth panel the cubic spline interpolation through the 15-day binned light curve at 22 GHz is also shown (cyan line).
Minimum (and maximum) flux density levels are indicated with horizontal dotted lines.}
         \label{radop}
   \end{figure*}

It is interesting to compare this recent behaviour of the source with its past history.
Figure \ref{long_term} displays the light curves in the optical $R$ band and at two radio frequencies in the last $\sim 12.5$ years, including the previous radio-optical outburst observed in 1997--1998. We notice that the last outburst had a shorter duration with respect to the outburst in 1997--1998. Moreover, if we compare the 2005--2006 and 2006--2007 events, we see that the latter was much ``harder" (i.e.\ stronger at higher frequencies) than the former. Similarly, the 1999--2000 post-outburst event is harder than the preceding major one. Other very hard (optical only) events are visible along all the period shown in Fig.\ \ref{long_term}, starting from the 1995--1996 event \citep{tak98}.
As in the case of 3C 454.3 discussed by \citet{vil07}, these occurrences might be due to (temporary) different orientations of the optical and radio emitting regions, harder events being observed when the optical emitting region is more aligned with the line of sight than the radio one, and hence the corresponding radiation is more beamed.
Alternatively, harder and softer events might be intrinsically different. In this case harder events would correspond to situations where the perturbation vanishes earlier in the jet, sometimes being unable to reach the radio emitting regions.
Regardless of their origin, hard flares in average should be shorter, since they cannot fully develop.

In this framework we could also explain the sharpness of the last radio outburst, and the fast contemporaneous flux decrease at all wavelengths. They might be due either to a geometric effect, i.e.\ a jet bending in the radio emitting region (as proposed for 3C 454.3; see \citealt{vil07}), or to an intrinsic perturbation power off in the same jet region.
We speculate that, had this shutdown occurred in an even inner part of the jet, we would have not observed the outburst in the radio band, i.e.\ the event would have been even harder.

   \begin{figure*}
   \sidecaption
   \includegraphics[width=12cm]{long_term.eps}
      \caption{Comparison among the $R$-band magnitudes ({\it top panel}),
corrected for the ELISA contribution,
and the radio flux densities (Jy) at 37 and 8 GHz over the last $\sim 12.5$ years.}
         \label{long_term}
   \end{figure*}

\section{Cross-correlation analysis}

In the previous section we saw that the brightness increase and consequent outburst
observed in the optical and near-IR bands were soon followed by similar events at mm and cm wavelengths.
We can of course estimate time delays by calculating the time separation between peaks in the different light curves.
But this procedure neglects the light curves behaviour as a whole, and strongly depends on the time sampling.
For instance, when comparing the $R$-band to the 1 mm light curve,
we find a lag of about 1 month if we consider the December 2006 optical flare,
while the lag is reduced to a couple of weeks if we consider the outburst peak.
Hence, in order to obtain a better estimate of the average time delays between the flux changes observed at different frequencies, we apply the discrete correlation function (DCF; \citealt{ede88,huf92}) analysis.
Since we are analysing a limited time period with long data gaps,
we use the cubic spline interpolations through the
binned light curves that had already been shown in Fig.\ \ref{radop}.
In this way we certainly oversmooth the variations hidden by the solar conjunction period,
but better define the major signals we are looking for.

Cross-correlation between the $R$-band and 1 mm, 37 GHz, and 22 GHz splines yields the DCF curves displayed in Fig.\ \ref{ritardi}.
In all cases the distribution of points shows a significant maximum ($\sim 1$)
at positive time lags $\tau$, which indicates a strong correlation with variations
at the lower frequency lagging behind the higher-frequency ones.
The delays corresponding to the DCF peaks are 20, 60, and 70 days for the 1 mm, 37 GHz, and 22 GHz curves, respectively.
We obtain a better estimate of the time lag by calculating the centroid of the DCF \citep{pet98},
which is defined as $\tau_{\rm c}=\sum \tau_i \, {\rm DCF}_i  / \sum {\rm DCF}_i$
for all the points $i$ with ${\rm DCF}_i > k \, {\rm DCF_{peak}}$.
By considering $0.7 \le k \le 0.8$, we obtain lags between 22 and 23 days for the $R$-mm, 56--58 days for the $R$-37 GHz, and 70--72 days for the $R$-22 GHz cross-correlations.

However, we have already discussed in the previous section that the 22 GHz outburst seems to be affected by a sort of shutdown causing the sudden flux decrease. In the absence of this effect, the outburst peak and following decreasing phase would likely have been delayed. Hence, the 70--72 day delay indicated by the DCF appears as a ``compromise" between the more delayed rising phase and the less delayed decaying phase.
Indeed, by looking at the fourth panel of Fig.\ \ref{radop}, we see that the flux density increase at 22 GHz followed that at 37 GHz after about 25 days, the peak was reached 1 week later, and no lag is visible during the flux fading.

In the 14.5, 8, and 5 GHz light curves, the time of the outburst peak cannot be defined because of the inferior sampling, but we nonetheless infer that the ascending part of the outburst was delayed by $\sim 50$, $\sim 65$, and $\ga 90$ days with respect to the rising part of the 37 GHz outburst, while, again, no significant delay is visible in the dimming phase.

The application of the DCF method to auto-correlate the composite $R$-band historical light curve\footnote{This composite light curve has been obtained by considering the $R$-band data back to JD = 2447000, and $B$-band data converted into $R$-band ones (through the mean colour index $B-R=1.7$) before
that date \citep{rai06b}.} results in a reinforcement of the $\sim 8.5$ year signal already found by \citet{rai06b}.

   \begin{figure}
   \resizebox{\hsize}{!}{\includegraphics{ritardi.eps}}
      \caption{Discrete correlation functions between the cubic spline interpolations through
the binned $R$-band and 1 mm light curves (blue dots), the $R$-band and 37 GHz ones (red diamonds), and the $R$-band and 22 GHz ones (green triangles).
Vertical strips highlight the range of time lags $\tau$ included between the location of the peak
and the centroid of the distributions. }
         \label{ritardi}
   \end{figure}

\section{Spectral energy distributions}

One of the major issues raised by the works of \citet{jun04} and \citet{rai05,rai06b,rai06a}
is the existence of an extra emission component in the SED of AO 0235+164,
in addition to the synchrotron and inverse-Compton components.
This component is represented by a bump in the UV and soft X-ray energy range, and its nature is not yet clear.
In the case of quasar-type blazars, a similar feature is clearly visible only during faint states, and it is generally ascribed to thermal emission from an accretion disc.
However, when these sources undergo
a flaring state, the bump tends to disappear, since it is overwhelmed by the synchrotron emission
\citep[see e.g.][for the case of 3C 454.3]{rai07b}.
In contrast, by comparing SEDs of AO 0235+164 obtained at various observing epochs in which X-ray observations were performed,
\citet{rai06b} pointed out that the presence of the extra component can also be inferred in bright states, and that the component appears to be variable.
But they could not discriminate between the two most obvious interpretations, i.e.\
thermal emission from an accretion disc and an additional synchrotron component from an inner jet region.

With the aim of shedding some light on this, we selected epochs in which the source was in different brightness states and in which contemporaneous near-IR-to-UV data were available.
Magnitudes were corrected for both Galactic and foreground-galaxy absorption by following the
prescriptions of \citet{jun04} (see Table 5 by \citealt{rai05}).
We then converted de-reddened magnitudes into fluxes using the calibration by \citet{bes98}
in the optical, and the zero-mag fluxes for the UVOT ultraviolet bands mentioned in Sect.\ 2.3.

We show three of these SEDs in Fig.\ \ref{sed}, compared with those corresponding
to the XMM-Newton pointings of January and August 2004, published in \citet{rai06b}.
We built the brightest SED with data taken on February 19, 2007, at the culmination of the outburst; since there were no simultaneous $JHK$ data, we inferred near-IR magnitudes
by considering the $R - JHK$ colour indices corresponding to the brightest near-IR level in
the light curves (see Fig.\ \ref{ottico-nir}).
In the other two cases (SEDs of January 31 and February 5, 2007) all data are contemporaneous (taken within few hours).
They show intermediate flux levels, with a noticeable spectral-slope variation:
the near-IR fluxes are nearly the same, but toward the UV the two SEDs diverge.

We found that the least-order polynomial that yields a good fit to the near-IR-to-optical part of the SEDs
is a cubic; we assume that these fits, which are displayed in Fig.\ \ref{sed},
can reasonably represent the contribution of the synchrotron component.
The UV excess is clearly recognizable in all SEDs, and
represents the extra emission component.
We estimated the contribution of this component in the UV bands as the difference between the total and the extrapolated synchrotron flux densities.

    \begin{figure}
   \resizebox{\hsize}{!}{\includegraphics{sed_0235.eps}}
      \caption{Near-IR-to-UV spectral energy distributions at various epochs:
the SED corresponding to the outburst maximum (February 19, 2007) is compared to
the faint-state SEDs obtained during the January and August 2004 XMM-Newton pointings \citep{rai06b}
and to two intermediate-state SEDs preceding the outburst peak.}
         \label{sed}
   \end{figure}

Figure \ref{extra} shows the relationship between the extra-component and the synchrotron-component flux
densities for the three UV bands: a fair linear correlation can be seen (correlation coefficient $r=0.996, 0.995, 0.994$ for $W1$, $M2$, $W2$, respectively).
Notwithstanding the uncertainties affecting our fitting procedure, we think that this result indicates
that this extra component can hardly be explained in terms of thermal emission from the disc, which is
not expected to depend on the synchrotron radiation, but more likely it is produced by the same mechanism
that is responsible for the synchrotron emission.

   \begin{figure}
   \resizebox{\hsize}{!}{\includegraphics{extra.eps}}
      \caption{The contribution to the flux density from the extra component
versus that from the synchrotron component in the three UVOT bands
(see text for details). Lines represent linear fits to the data.}
         \label{extra}
   \end{figure}

\section{Discussion and conclusions}

The claim by \citet{rai01} of a possible quasi-periodic occurrence of the major radio and optical outbursts of AO 0235+164 every $5.7 \pm 0.5$ years led to a mobilization of the international blazar community to observe the next event, predicted to peak in early 2004. A WEBT campaign was organised, with a huge international participation \citep{rai05,rai06b,rai06a,rai07a}, but the source remained in a faint state and time series analysis on the updated light curves suggested a possible longer period of about 8--8.5 years, delaying the occurrence of the next outburst toward the 2006--2007 observing season.
An increased radio activity was registered in 2005--2006 \citep{bac07}, and a major optical outburst was finally observed in late 2006 -- early 2007, about 8.5 years after the previous major outburst peak, thus confirming the \citet{rai06b} prediction.

We observed the same event simultaneously in the near-IR and UV frequency ranges, and then at millimetric and centimetric wavelengths, with a progressive time delay toward lower frequencies.
We estimated that the 1 mm and 37 GHz outbursts lagged behind the $R$-band one by about 3 weeks and 2 months, respectively.
This latter lag appears a factor $\sim 2$
longer than previously estimated \citep{rai05,rai06b} when considering the historical light curves until spring 2005.
This may reflect some real change either in the jet structure at still unresolved scales or in its energetics with respect to the past.
Indeed, there are sources that exhibit a characteristic behavior for many years and then suddenly change \citep[see e.g.][]{smi96,all96}.
Another possibility is that lack of data for long time intervals, chiefly in the optical bands because of solar conjunctions, affects the results of the DCF analysis.
Even in the case of the 1997--1998 outburst, which was intensively observed thanks to the
efforts of the just-born WEBT \citep{rai01}, the central part of the event was missed in the optical, and the dimming phase was poorly sampled at 37 GHz. As for the 2006--2007 outburst, notwithstanding the exceptional sampling, we cannot
rule out that solar conjunction hid a further optical peak. This would shorten the radio lags.

In any case, the delays estimated in the present paper appear rather short if compared,
for example, with the approximately twice-longer lags estimated by \citet{vil07} for the quasar-type blazar 3C 454.3 in correspondence of its 2004--2006 exceptional outburst, and by \citet{vil04b} and \citet{bac06} for BL Lacertae.
The short time delays in AO 0235+164 had already been noticed in the past \citep{web00,rai01}, and this was one of the reasons why microlensing was proposed as a possible explanation of the major variability events in this source \citep[see e.g.][and references therein]{rai07a}.
In contrast, when interpreting the multifrequency variability of AO 0235+164 in the framework of the helical jet model by \citet{vil99}, the short time delays imply that the corresponding jet-emitting regions are only slightly misaligned  \citep{ost04}. If, besides the geometrical effect, the outburst is also produced by energetic processes inside the jet, short time lags imply that the emitting regions are also closeby.

At lower ($< 37$ GHz) radio frequencies, the outburst behaviour changes: while the rising phase is progressively delayed, as expected, the decaying phase is observed simultaneously at all wavelengths. In particular, the 22 GHz flux density reaches a maximum value similar to the 37 GHz flux density almost at the same time and then suddenly decreases.
Although the observed general multifrequency properties of the outburst can be explained by models dealing with shocks propagating along an inhomogeneous jet \citep[see e.g.][]{hug89,val92}, the emission behaviour during the dimming phase might suggest that the scenario is more complicated than what is depicted in these models.
Indeed, it seems as if a kind of shutdown occurred in the jet between the mm emission region and the 37--22 GHz region.
We can speculate that either a jet bending or an intrinsic power off of the perturbation may produce this effect, which
could also account for the different flavours of events observed in the historical light curves of the source: the harder outbursts, which are stronger at the higher frequencies,
suggest that a shutdown (of any origin) has occurred between the higher- and the lower-frequency emitting regions.
Understanding whether the geometric or intrinsic scenario would be more plausible is not easy.
One can envisage that if the flux fading is due to a jet bending, the perturbation could continue to travel along the jet, following a misaligned path, until it could eventually be observed at radio wavelengths as soon as the jet (helical) path turns again toward the line of sight. For those sources, for which the Very Long Baseline Interferometry (VLBI) resolution can separate these regions, we should be able to observe a brightening of a VLBI knot.

We notice that our analysis of the multifrequency light curves in terms of harder/softer events presents some similarities with the generalized shock model by \citet{val92}, who distinguish between high- and low-peaking flares.
We finally mention that \citet{hag07a} interpreted the photometric and polarimetric variations of AO 0235+164 during the December 2006 flare as due to the propagation of a transverse shock, accompanied by a small change in the viewing angle of the jet.

Another major issue we investigated is the existence of an extra emission component mostly contributing in the UV and soft X-ray energy range of the source SED.
Swift-UVOT observations during the culminating phase of the optical outburst show that the behaviour of the UV emission strictly follows that of the optical one. Moreover, when constructing SEDs with simultaneous near-IR-to-UV data, and separating the synchrotron and extra-component
contributions, we found that they are correlated.
Although this result is affected by uncertainties due to the decomposition procedure, it nevertheless
suggests that an interpretation of the extra component in terms of radiation from the accretion disc is rather unlikely \citep[see][for the various possible interpretations]{rai06b,rai06a}. Also the hypothesis of two independent synchrotron components  appears now inadequate.
A still viable explanation could be that of anomalous absorption of the optical to near-UV emission in a spectrum that would otherwise be power-law-like from the near-IR to UV.
Indeed, examples of noticeable intrinsic absorption are found when analysing the spectra of broad absorption line (BAL) quasars, where several lines between $\sim$ 1000 and 1500 \AA\ (rest frame) can heavily reduce the flux in the corresponding spectral regions \citep[see e.g.][]{tur88}.
A further possibility is that the extra component is the result of inverse-Compton scattering of radio photons off the relativistic electrons producing the IR--optical emission. Indeed, as stated above, the close correlation and short time delays between the optical and radio variations suggest that a lot of radio photons are available in the optical emitting region.

\begin{acknowledgements}
We are grateful to Philip Hughes for useful discussions.
We acknowledge the use of public data from the Swift data archive.
This research has made use of data obtained through the High Energy Astrophysics Science Archive Research Center Online Service,
provided by the NASA/Goddard Space Flight Center.
This work is partly based on observations made with the Nordic Optical Telescope, operated
on the island of La Palma jointly by Denmark, Finland, Iceland,
Norway, and Sweden, in the Spanish Observatorio del Roque de los
Muchachos of the Instituto de Astrofisica de Canarias.
This research has made use of data from the University of Michigan Radio Astronomy Observatory,
which is supported by the National Science Foundation and by funds from the University of Michigan.
The Mets\"ahovi team acknowledges the support from the Academy of Finland.
AZT-24 observations are made within an agreement between  Pulkovo,
Rome and Teramo observatories.
The St.-Petersburg team was supported by the Russian Foundation for Basic Research, grant
08-02-00245.
The Torino team acknowledges financial support by the Italian Space Agency through contract
ASI-INAF I/088/06/0 for the Study of High-Energy Astrophysics.
University of Joensuu acknowledges co-operation with local Amateur Astronomer Association Seulaset.
ACG work is supported by NNSF of China grant No.10533050.
JHF's work is supported by the National Natural Science Foundation of China (10573005,10633010) and the 973 project (No. 2007CB815405).
\end{acknowledgements}


Learning to Fuse Music Genres with Generative Adversarial Dual Learning

Abstract: FusionGAN is a novel genre fusion framework for music generation that integrates the strengths of generative adversarial networks and dual learning. In particular, the proposed method offers a dual learning extension that can effectively integrate the styles of the given domains. To efficiently quantify the difference among diverse domains and avoid the vanishing gradient issue, FusionGAN provides a Wasserstein based metric to approximate the distance between the target domain and the existing domains. Adopting the Wasserstein distance, a new domain is created by combining the patterns of the existing domains using adversarial learning. Experimental results on public music datasets demonstrated that our approach could effectively merge two genres.

\section{Introduction}

Computational creativity is a lively research area that focuses on understanding and facilitating human creativity through the implementation of software programs \cite{colton2009computational}. With the rapid advances in data-driven algorithms such as deep learning \cite{hinton2006fast}, the exploration into computational creativity via machine learning approaches has become increasingly popular.  Examples showing the potential of deep learning for creative approaches are the artistic style transfer on images \cite{Gatys2015} and videos \cite{ruder2016artistic}. Music, with its complex hierarchical and sequential structure and its inherent emotional and aesthetic subjectivity, is an intriguing research subject at the core of human creativity. While there has been some work on generative models for music, research that investigates the capabilities of deep learning for creative applications such as style transfer is limited. This paper aims to fill this space by exploring the idea of style fusion in music with generative adversarial dual learning.

In the field of unsupervised generative learning, generative adversarial networks (GAN) \cite{goodfellow2014generative} have recently gained considerable attention.
It is important to note, however, that GANs are designed to learn from a single domain and cannot discover cross-domain knowledge.
Inspired by dual learning in machine translation, several recent publications propose methods to pair unlabeled data points in different domains by optimizing bi-directional reconstruction errors \cite{yi2017dualgan,kim2017learning,zhu2017unpaired}. Although these methods are designed to address the challenge of multimodal learning, they do not address the problem of domain fusion.

The goal of this paper is to provide a solution for fusing two or more groups of sequential patterns using unsupervised methods. Specifically, we focus on the problem of generating music with a fused music genre.
To combine genres using generative adversarial learning, we propose a framework for integrating multiple GANs. Unlike previous work, our study targets the creation of an unknown domain by mixing two given domains; a multi-way GAN-based model is proposed to absorb existing patterns and yield a new mixture.
With the utilization of the Wasserstein measure \cite{arjovsky2017wasserstein}, we pose a distance constraint on the new domain that automatically balances its relation to the given domains.
As a result, our model is capable of generating a mixed pattern after convergence. The main contributions of this paper are:
\begin{itemize}
    \item \textbf{A novel framework for unsupervised music fusion}: The dual learning scheme is extended to involve multiple domains by leveraging mutual regularization. In this way, the proposed framework enables the new domain to keep equal similarities with the given domains and produce a mixture of existing sequential patterns.
    \item \textbf{A sequence fusion method using GANs}: To apply unsupervised fusion learning, we extend the generative adversarial model so that multiple generators and discriminators can be updated by one another.
    \item \textbf{Formulation of an objective function indicating fusion progress}: An objective function based on Wasserstein distance is designed to integrate the information from multiple domains and represent learning progress.

\end{itemize}

\section{Related Work}\label{rw}
\textbf{Music Genre Fusion:} Fusion is mostly known as the sub-genre of jazz that emerged in the late '60s, and that combines several musical styles such as funk, rock, and blues with the jazz harmony and improvisation \cite{GeraldDavid2005}. The term can be generalized, however, to any combinations of music genres. The creative combination of two or more genres obviously requires not only intimate knowledge of the stylistic characteristics of the involved genres but also extensive compositional experience and skill to combine genres in a meaningful and satisfying way.
Recently, Engel et al.~\cite{engel2017neural} proposed a new approach of generating new musical sounds through interpolating the latent space learned by WaveNet \cite{oord2016wavenet} autoencoders.
Additionally, Gatys et al. demonstrated the Deep Neural Networks' (DNNs) capabilities of learning the artistic styles by blending the style of the training materials to the testing images using ConvNet \cite{Gatys2015}. Both examples show the great potential of using DNNs for merging two abstract concepts without the need for domain knowledge and explicitly defined rules.

\textbf{Cross-domain GAN:} GAN \cite{goodfellow2014generative}, which has quickly risen to one of the most popular generative approaches, learns patterns without requiring adversarial examples or labels. In CGAN \cite{mirza2014conditional}, the generator $G$ has a conditional parameter $c$ and will learn the conditional distribution of the data. However, those conditions need to be provided manually, somewhat similar to supervised learning. CoGAN \cite{liu2016coupled} is proposed to learn a joint distribution with only samples drawn from the marginal distributions. DiscoGAN \cite{kim2017learning} solves the cross-domain pairing by minimizing bi-directional reconstruction loss, while DualGAN \cite{yi2017dualgan} takes advantage of the dual learning paradigm in machine translation. CycleGAN \cite{zhu2017unpaired} presents a method whose mapping functions are cycle-consistent and proposes a cycle consistency loss function to further reduce the space of possible mapping functions.

Different from the previous studies, our work raises a new problem, i.e., merging two sequential patterns into one. We propose a GAN-based framework that merges two domains without manually tuning the distance between the given domains and the target domain.

\section{Learning to Blend Music with FusionGAN}\label{method}

\subsection{Problem Definition}
This paper aims to characterize music fusion across different genres in an unsupervised manner. Based on the data $\XA$ and $\XB$ from existing music domains $\D_{A}, \D_{B}$ respectively, our goal is to create a new domain $\D_{F}$ which resembles both $\D_{A}$ and $\D_{B}$. In the first phase, the new domain $\D_{F}$ is expected to learn from $\D_{A}, \D_{B}$ and keep an equal distance from both.  Specifically, the generator $\GF$ of $\D_{F}$ obtains feedback from the discriminators $\DA,\DB$ of existing domains $\D_{A}, \D_{B}$, while the discriminator $\DF$ collects data from $\GF$ and  $\D_{A}, \D_{B}$ for iterative updating. The $\GF$ from the new domain minimizes the distance from the domains $\D_{A}, \D_{B}$, respectively, and keeps the same distance from $\D_{A}$ and $\D_{B}$.

\subsection{FusionGAN}
First, the framework initializes individual GAN models for existing domains. Maximum likelihood estimation using Long Short Term Memory (LSTM) is applied to initialize the sequence generators $\GA(\GB)$. Then $\DA$($\DB$) are trained given the domain data and sequence sampled from $\GA(\GB)$. Then GAN is employed to iteratively enhance the $\GA(\GB)$ and $\DA$($\DB$).
After pre-training, we build a GAN model for the new domain using the feedback from the models constructed in the pre-training procedure. First, a new domain $\D_{F}$ is initialized randomly and trained by both $\D_{A}$ and $\D_{B}$. Following the dual learning strategy, $\D_{A}$ is enhanced by $\D_{B}$ and $\D_{F}$ in the second phase. Similarly, $\D_{B}$ is improved by $\D_{A}$ and $\D_{F}$. Such learning proceeds until convergence. The framework overview is shown in Figure \ref{fig:fusion_framework}.

\begin{figure}[!hpbt]
    \centering
    \includegraphics[width=2.4in]{img/fusion_framework.png}
    \caption{FusionGAN framework}
    \label{fig:fusion_framework}
\end{figure}

In FusionGAN, we employ the Wasserstein-1 distance for all discriminators because it is sensible when the learning distributions are supported by low dimensional manifolds \cite{arjovsky2017wasserstein}.
To avoid the intractable infimum in Wasserstein-1 distance, we resort to its Kantorovich-Rubinstein duality \cite{villani2008optimal}: $ W\left( \Pp_{r},\Pp_{\theta }\right) =\sup_{\left\| f\right\| _{L}\leq 1}\E_{x\sim \Pp_{r}}\left[ f\left( x\right) \right]-\E_{x\sim \Pp_{\theta }}\left[ f\left( x\right) \right] $.
For the vanilla GAN, the goal is to find the optimal configuration of the parameters $\phi$ of discriminator ($f=\Dd$). When the discriminator is optimized, the maximized Wasserstein distance can be used as reward in the policy gradient process of the generator. Similarly, we define a three-way Wasserstein distance as the objective function: $W\left( \Pp_{\XA},\Pp_{\XB},\Pp_{\theta }\right)=\Ll $ to measure the integrated distance between $\Pp_{\theta}$ and $\Pp_{\XA},\Pp_{\XB}$. This measure should consider the relationship between every pair of domains. The framework consists of three pairs of generators and discriminators, i.e., $\GA-\DA, \GB-\DB, \GF-\DF$, and two input data, i.e., $\XA, \XB$. For each discriminator, there are five possible inputs, i.e., $\XA,\XB, \GA, \GB, \GF$. Collecting all possible inputs w.r.t. the three discriminators, the objective function to maximize is defined as:
{
\footnotesize
\begin{equation}\label{loss}
\begin{aligned}
\Ll &=&&W\left( \Pp_{\XA},\Pp_{\XB},\Pp_{\theta }\right) \\
&&&\E_{x\sim \Pp_{\XA}}\left[\DA\left( x \right) \right]-\E_{x\sim \Pp_{\XB}}\left[\DA\left( x \right) \right] - \\
&&& \E_{z\sim p(z)}\left[\DA\left( \GA\left( z\right) \right)\right] - \E_{z\sim p(z)}\left[\DA\left( \GB\left( z\right) \right) \right]-\\
&&&\E_{z\sim p(z)}\left[\DA\left( \GF\left( z\right) \right)\right] - \\
&&&\E_{x\sim \Pp_{\XA}}\left[\DB\left( x \right) \right]+ \E_{x\sim \Pp_{\XB}}\left[\DB\left( x \right) \right]-\\
&&&\E_{z\sim p(z)}\left[\DB\left( \GA\left( z\right) \right)\right] - \E_{z\sim p(z)}\left[\DB\left( \GB\left( z\right) \right) \right]-\\
&&&\E_{z\sim p(z)}\left[\DB\left( \GF\left( z\right) \right)\right] +\\
&&&\E_{x\sim \Pp_{\XA}}\left[\DF\left( x \right) \right]+ \E_{x\sim \Pp_{\XB}}\left[\DF\left( x \right) \right] +\\
&&&\E_{z\sim p(z)}\left[\DF\left( \GA\left( z\right) \right)\right] + \E_{z\sim p(z)}\left[\DF\left( \GB\left( z\right) \right) \right]-\\
&&&\E_{z\sim p(z)}\left[\DF\left( \GF\left( z\right) \right)\right],\\
\end{aligned}
\end{equation}
}where the first five terms are for $\DA$, the second five terms are for $\DB$, and the last five terms are for $\DF$. This loss is used to update all the generators and discriminators. The following two subsections, \ref{sec:f_fusiongan} and \ref{sec:ab_fusiongan},  will present the update rules using Eq. \ref{loss}.

\subsection{$\D_{F}$ Update of FusionGAN}\label{sec:f_fusiongan}
Removing the terms that are unrelated to $\D_{F}$ in Eq. \ref{loss} (zero derivative w.r.t. $\DF$ and $\GF$), we have the objective function for updating $\DF$ and $\GF$:
{
\footnotesize
\begin{equation}
\begin{aligned}
\Ll_{F} &=&&\E_{x\sim \Pp_{\XA}}\left[\DF\left( x \right) \right]+ \E_{x\sim \Pp_{\XB}}\left[\DF\left( x \right) \right] +\\
&&&\E_{z\sim p(z)}\left[\DF\left( \GA\left( z\right) \right)\right] + \E_{z\sim p(z)}\left[\DF\left( \GB\left( z\right) \right) \right]-\\
&&&\E_{z\sim p(z)}\left[\DF\left( \GF\left( z\right) \right)\right]-\\
&&&\E_{z\sim p(z)}\left[\DA\left( \GF\left( z\right) \right)\right] - \\
&&&\E_{z\sim p(z)}\left[\DB\left( \GF\left( z\right) \right)\right]. \\
\end{aligned}
\end{equation}
}First, $\Ll_{F}$ is calculated w.r.t. $\GF$ so as to update $\GF$. Considering differentiating $\nabla _{\theta}\Ll_{F}$, the optimal $\GF$ is approximated as its convergence. The proof is provided below to show that this derivative is principled under the optimality assumption.

\begin{theorem}[FusionGAN Optimality Theorem]\label{theorem}
Let $\Pp_{\XA}$ and $\Pp_{\XB}$ be any distribution, Let $\Pp_{\theta}$ be the distribution of $\GF_{\theta}(Z)$ with $Z$, a random variable with density $p$ and $\GF_{\theta}$ a function satisfying $\E_{z\sim p}[L(\theta,z)] < +\infty$, where $L$ is Lipschitz constants. All functions are well-defined. Then there is a solution $\DF:\X\rightarrow\Rd$  to the problem:
{
\footnotesize
\begin{equation}
W\left( \Pp_{\XA},\Pp_{\XB},\Pp_{\theta }\right)=\max _{\left\| \DF\right\| _{L}\leq 1} \Ll_{F}.
\end{equation}} and we have:
{
\footnotesize
\begin{equation}\label{opttheorem}
\begin{aligned}
\nabla _{\theta }\Ll_{\GF} =&\nabla _{\theta }W\left( \Pp_{\XA},\Pp_{\XB},\Pp_{\theta}\right)&\\
=&-\E_{z\sim p(z)} \nabla _{\theta } \sum_{i\in \{A,B,F\}}\Dd_{i}\left( \GF\left( z\right) \right)&\\
\end{aligned}
\end{equation}
}

\end{theorem}

\begin{proof}[Proof of Theorem \ref{theorem}]
Let us define $ \Ll_{F}= V(\tilde{\DF},\theta)$ where $\tilde{\DF}$ lies in $\F=\{\tilde{\DF}:\X \rightarrow \Rd, \tilde{\DF} \in C_{b}(\X), \left\|\DF\right\| _{L}\leq 1 \}$ and $\theta \in \Rd_{d}$. By the Kantorovich-Rubinstein duality,  there is an $f \in \F$ that satisfies:
{
\footnotesize
\begin{equation*}
    W\left( \Pp_{\XA},\Pp_{\XB},\Pp_{\theta }\right) =\sup_{\tilde{\DF}\in\F}V(\tilde{\DF},\theta)=V(\DF,\theta).
\end{equation*}
}
Let us define $X^{*}(\theta)=\{\DF\in\F: V(\DF,\theta)=W\left( \Pp_{\XA},\Pp_{\XB},\Pp_{\theta }\right)\}$, which shows that $X^{*}(\theta)$ is non-empty. According to Theorem 1 in \cite{arjovsky2017wasserstein}: (1) if $\GF$ is continuous in $\theta$, so is $W\left( \Pp_{\XA},\Pp_{\XB},\Pp_{\theta }\right)$, (2) If $\GF$  is locally Lipschitz and satisfies $\E_{z\sim p}[L(\theta,z)] < +\infty$, then $W\left( \Pp_{\XA},\Pp_{\XB},\Pp_{\theta }\right)$ is continuous and differentiable for any $\DF \in X^{*}(\theta)$ when both terms are well-defined. Let $\DF \in X^{*}(\theta)$, which we knows exists since $X^{*}(\theta)$is non-empty for all $\theta$. Then:
{\footnotesize
\begin{equation}\label{wdis}
\begin{aligned}
&\nabla _{\theta }W\left( \Pp_{\XA},\Pp_{\XB},\Pp_{\theta }\right)&\\
=&\nabla _{\theta }V(\DF,\theta)=\nabla _{\theta }\Ll_{F}&\\
=&-\nabla _{\theta }\E_{z\sim p(z)}\lbrack \DF\left( \GF\left( z\right) \right)+\DA\left( \GF\left( z\right) \right) +\DB\left( \GF\left( z\right) \right) \rbrack, &
\end{aligned}
\end{equation}}since $\DF$ is 1-Lipschitz. Furthermore, $\GF_{\theta}(z)$ is locally Lipschitz as a function of ($\theta, z$). Therefore, $\GF_{\theta}(z)$ is locally Lipschitz on ($\theta, z$) with constants $L(\theta, z)$. By Radamacher’s Theorem, $\DF(\GF_{\theta}(z))$ has to be differentiable almost everywhere for ($\theta, z$) jointly. Rewriting this, the set $A = \{(\theta, z) : \DF  \circ \GF $ is not differentiable$\}$ has measure 0. By Fubini’s Theorem, this implies that for almost every $\theta$ the section $A_{\theta} = \{z : (\theta, z) \in A\}$ has measure 0. Let's fix a $\theta_{0}$ such that the measure of $A_{\theta_{0}}$ is null. For this $\theta_{0}$ we have $\nabla _{\theta}\DF(\GF_{\theta}(z))|\theta_{0}$ is well-defined for almost any $z$, and since $p(z)$ has a density, it is defined $p(z)$ almost everywhere. Given the condition $\E_{z\sim p}[L(\theta,z)] < +\infty$, we have
{\footnotesize
\begin{equation}
    \E_{z\sim p(z)}[\left\|\nabla _{\theta}\DF(\GF_{\theta}(z))\right\|] \leq \E_{z\sim p(z)}[L(\theta_{0},z)] < +\infty,
\end{equation}
}so $\E_{z\sim p(z)}[\nabla _{\theta}f(g_{\theta}(z))] $ is well-defined for almost every $\theta_{0}$. To keep the notation simple, we leave it implicit in the following equation that the $\E$ subjects to $z\sim p(z)$.  :
{\footnotesize
\begin{equation}\label{wdisapp}
\begin{aligned}
&\frac{1}{\left\| \theta-\theta_{0} \right\|}\sum_{i} \bigg[\E[\Dd_{i}(\GF_{\theta}(z))] -\E[\Dd_{i}(\GF_{\theta_{0}}(z))] -\\
&\langle(\theta-\theta_{0}),\E\lbrack\nabla _{\theta}[\Dd_{i}(\GF_{\theta}(z))]\rbrack|_{\theta_{0}}\rangle \bigg]\\
=&\frac{1}{\left\| \theta-\theta_{0} \right\|}\sum_{i} \E \bigg[ \Dd_{i}(\GF_{\theta}(z)) -\Dd_{i}(\GF_{\theta_{0}}(z)) -\\
&\langle(\theta-\theta_{0}),\nabla _{\theta}[\Dd_{i}(\GF_{\theta}(z))]|_{\theta_{0}}\rangle \bigg]\\
\leq & \frac{1}{\left\| \theta-\theta_{0} \right\|}\sum_{i} \big\| \left\| \theta-\theta_{0} \right\|L(\theta_{0},z) + \left\| \theta-\theta_{0} \right\|\cdot\|\nabla _{\theta}[\Dd_{i}(\GF_{\theta}(z))]\big\|\\
\leq & \sum_{i}2L(\theta_{0},z) = 6L(\theta_{0},z),
\end{aligned}
\end{equation}}where $i \in \{A,B,F\}$. By differentiability, the term inside the integral converges $p(z)$ to 0 as $\theta \rightarrow \theta_{0}$. Furthermore, and since $\E_{ z \sim p(z)} [6L(\theta_{0},z) ] < +\infty$, we get by dominated convergence that Eq. \ref{wdis} converges to 0 as $\theta \rightarrow \theta_{0}$. So the result of Eq. \ref{wdis} equals to:
{\footnotesize
\begin{equation*}
-\E_{z\sim p(z)}\lbrack \nabla _{\theta }\DF\left( \GF\left( z\right) \right)+\nabla _{\theta }\DA\left( \GF\left( z\right) \right) +\nabla _{\theta }\DB\left( \GF\left( z\right) \right) \rbrack.
\end{equation*}
}
\end{proof}

Therefore, the update rules for the parameters $\theta$ of $\GF$ is Eq. \ref{opttheorem}.
\begin{figure}[!hpbt]
    \centering
    \includegraphics[width=2.2in]{img/fusion_fg.png}
    \caption{FusionGAN: update $\GF$}\label{fig:fg_update}
\end{figure}

The physical meaning is that all three discriminators can offer feedback to generator $\GF$ as shown in Fig. \ref{fig:fg_update}. The left cyan rectangle indicates $\D_{A}$, the middle pink one is $\D_{F}$, and the right yellow one $\D_{B}$. Similarly, the gradient w.r.t. the parameters $\phi$ of $\DF$ is:
{
\footnotesize
\begin{equation}\label{fd_update}
\begin{aligned}
\nabla_{\phi}\Ll_{F} &=&&\nabla_{\phi}\big[\E_{x\sim \Pp_{\XA}}\left[\DF\left( x \right) \right]+ \E_{x\sim \Pp_{\XB}}\left[\DF\left( x \right) \right] +\\
&&&\E_{z\sim p(z)}\left[\DF\left( \GA\left( z\right) \right)\right] + \E_{z\sim p(z)}\left[\DF\left( \GB\left( z\right) \right) \right]-\\
&&&\E_{z\sim p(z)}\left[\DF\left( \GF\left( z\right) \right)\right]\big].\\
\end{aligned}
\end{equation}
}

\begin{figure}[!hpbt]
    \centering
    \includegraphics[width=2.2in]{img/fusion_fd.png}
    \caption{FusionGAN: update $\DF$}
    \label{fig:df_update}
\end{figure}

As shown in Fig. \ref{fig:df_update}, the update for $\DF$ actually implicitly balances itself between $\D_{A}$ and $\D_{B}$. However, this update does not cover the degree of blending which may lead to an unbalanced mixture.  Specifically, a good learning process for the discriminator $\DF$ should consider two factors: (1) minimizing the distance with $\D_{A}$, $\D_{B}$ and $\D_{F}$ simultaneously and (2) maintaining a equal distance from $\D_{A}$ and $\D_{B}$ so as to satisfy both domains. To satisfy (2) and push the discriminator $\DF$ to have a well-proportioned blending, an additional constraint is employed to further balance the ratio of existing domains, i.e., $\D_{A}, \D_{B}$:
{
\footnotesize
\begin{equation}\label{fd_update_add}
\begin{aligned}
\Ll_{F-bal} &=&& \left\| \E_{z\sim p(z)}\left[\DF\left( \GA\left( z\right) \right)\right] - \E_{z\sim p(z)}\left[\DF\left( \GB\left( z\right) \right) \right]\right\| + \\
&&& \left\| \E_{x\sim \Pp_{\XA}}\left[\DF\left( x \right) \right] - \E_{x\sim \Pp_{\XB}}\left[\DF\left( x \right) \right]\right\|.
\end{aligned}
\end{equation}
}

\subsection{$\D_{A}$ and $\D_{B}$ Update of FusionGAN}\label{sec:ab_fusiongan}
After updating  $\D_{F}$, FusionGAN will improve $\D_{A}$ and $\D_{B}$. Since $\D_{A}$ and $\D_{B}$ are symmetric in the framework, we only discuss $\D_{A}$ as shown in Fig. \ref{fd_update_add}. Keeping related terms in Eq. \ref{loss}, we get the loss function for $\D_{A}$:
{
\footnotesize
\begin{equation}\label{aloss}
\begin{aligned}
\Ll_{A} &=&&\E_{x\sim \Pp_{\XA}}\left[\DA\left( x \right) \right]-\E_{x\sim \Pp_{\XB}}\left[\DA\left( x \right) \right] - \\
&&& \E_{z\sim p(z)}\left[\DA\left( \GA\left( z\right) \right)\right] - \E_{z\sim p(z)}\left[\DA\left( \GB\left( z\right) \right) \right]-\\
&&&\E_{z\sim p(z)}\left[\DA\left( \GF\left( z\right) \right)\right] -\\
&&&\E_{z\sim p(z)}\left[\DB\left( \GA\left( z\right) \right)\right] +\\
&&&\E_{z\sim p(z)}\left[\DF\left( \GA\left( z\right) \right)\right].
\end{aligned}
\end{equation}
}
\begin{figure}[!hpbt]
    \centering
    \includegraphics[width=2.1in]{img/fusion_ad.png}
    \caption{FusionGAN: update $\DA$}\label{fig:ad_update}
\end{figure}

Taking the derivative of $\Ll_{A}$ w.r.t. $\DA$, we have
{
\footnotesize
\begin{equation}\label{ad_update}
\begin{aligned}
\nabla_{\DA}\Ll_{A} &=&&\nabla_{\DA}\big[\E_{x\sim \Pp_{\XA}}\left[\DA\left( x \right) \right]-\E_{x\sim \Pp_{\XB}}\left[\DA\left( x \right) \right] - \\
&&& \E_{z\sim p(z)}\left[\DA\left( \GA\left( z\right) \right)\right] - \E_{z\sim p(z)}\left[\DA\left( \GB\left( z\right) \right) \right]-\\
&&&\E_{z\sim p(z)}\left[\DA\left( \GF\left( z\right) \right)\right] \big]. \\
\end{aligned}
\end{equation}
}

There are four negative terms and one positive term in Eq. \ref{ad_update} w.r.t. $\DA$. The training implicitly increases the positive terms and decreases the value of the negative terms. However, the inequalities in the two groups are not controlled. To further control the distance among the three domains, we propose an inequality constraint for $\DA$:
{
\footnotesize
\begin{equation}\label{ad_update_add}
\begin{aligned}
&\DA\left( \XA\right) \geq\DA\left( \GF\left( z\right)  \right)  \geq\DA\left( \XB\right)
\end{aligned}
\end{equation}
}

Then combining Eq. \ref{ad_update_add} into loss function Eq. \ref{ad_update}, it can be rewritten as:
{
\footnotesize
\begin{equation}\label{ad_update_add_rew}
\begin{aligned}
&\Ll_{A-bal} = \|\DA\left(\XA\right) -\DA\left(\GF\left(z\right) \right)\| +\| \DA\left(\GF\left(z\right)\right)-\DA\left(\XB\right)\|.
\end{aligned}
\end{equation}
}
\begin{figure}[!hpbt]
    \centering
    \includegraphics[width=2.1in]{img/fusion_ag.png}
    \caption{FusionGAN: update $\GA$}\label{fig:ag_update}
\end{figure}

Some inequalities that have been implicitly included in GAN training are not covered in Eq. \ref{ad_update_add}, such as $\DA\left( \XA\right) \geq \DA\left( \GA\right)$. Similar to the $\GF$ update in Fig. \ref{fig:ag_update}, the derivative of $\Ll_{A}$ w.r.t. $\GA$ is:
{
\footnotesize
\begin{equation}\label{ag_update}
\begin{aligned}
\nabla_{\GA}\Ll_{A} &=&& \E_{z\sim p(z)}\nabla_{\GA}\big[-\DA\left( \GA\left( z\right) \right) -\\
&&&\DB\left(\GA\left(z\right)\right) +\DF\left(\GA\left(z\right)\right)\big].
\end{aligned}
\end{equation}
}

\subsection{Algorithm Description}

{\footnotesize
\begin{algorithm}[!h]
    \caption{FusionGAN}
    \label{algo:fgan}
    \SetAlgoLined
    \KwIn{Input data $\XA,\XB$ from $\D_{A},\D_{B}$}
    \KwOut{a generator and a discriminator: $\GF and \DF$}
    Randomly initialize $\GA, \DA, \GB, \DB, \GF, \DF$, generators/discriminators employ LSTM/TextCNN\;\label{algo:init}
    \tcp{Pre-training}
    Train $\GA$ using maximum likelihood estimation on real data $\XA$\;\label{sgan_start}
    Generate negative samples $\NA$ using $\GA$\; \label{gen_neg}
    Train binary classifier $\DA$ with $\NA$ and $\XA$\; \label{min_entro}
    \Repeat{$\DA$ converges}{ \label{pregan_start}
        \For{$\GA$ training}
        {
            Sample a sequence $\SL_{1:T}=(s_{1},...,s_{T}) \sim \GA$\;
            Derive $Q$ value  as rewards $\R$ from $\DA$\;
            Update parameters $\theta$ of $\GA$ by policy gradient: $\theta \leftarrow \theta+\alpha\nabla_{\theta}\R$  \;
        }
        \For{$\DA$ training}
        {
            Generate negative examples $\sim \GA$\;
            Train $\DA$ with negative samples and $\XA$\;
        }
    }\label{sgan_end}
    Repeat line \ref{sgan_start} to \ref{sgan_end} with $\D_{B}$, i.e., $\DB, \GB, \XB$\;
    Repeat line \ref{sgan_start} to \ref{min_entro} with $\D_{F}$, i.e., $\DF, \GF, \mathbf{X}_{F}=\XA+\XB$\;
    \tcp{FusionGAN training}
    \Repeat{$\DA, \DB, \DF$ converges}{
        \For{$\D_{F}$ training}{\label{fusion_f_start}
            Update parameters of $\DF$ using the sum of Eq. \ref{fd_update} and Eq. \ref{fd_update_add} \label{fusion_df}: $\theta_{\DF} \leftarrow \theta_{\DF}+\alpha\nabla_{\DF}(\Ll_{F}+\Ll_{F-bal}) $ \;
            Update parameters of $\GF$ using Eq. \ref{opttheorem} $\theta_{\GF} \leftarrow \theta_{\GF}+\alpha\nabla_{\GF}\Ll_{F} $ \;\label{fusion_gf}
        }\label{fusion_f_end}
        \For{$\D_{A}$ training}{\label{fusion_ab_start}
          Update parameters of $\DA$ using the sum of Eq. \ref{ad_update} and Eq. \ref{ad_update_add_rew} :  $\theta_{\DA} \leftarrow \theta_{\DA}+\alpha\nabla_{\DA}(\Ll_{A}+\Ll_{A-bal}) $\;
        Update parameters of $\GA$ using Eq. \ref{ag_update}: $\theta_{\GA} \leftarrow \theta_{\GA}+\alpha\nabla_{\GA}\Ll_{A} $\;
        }\label{fusion_ab_end}
        Repeat line \ref{fusion_ab_start} to \ref{fusion_ab_end} with $\D_{B}$, i.e., $\DB, \GB, \XB$\;
    }
\end{algorithm}
}
The Algorithm \ref{algo:fgan}\footnote{https://github.com/aquastar/fusion\_gan} initializes the parameters of the discriminators and generators (line \ref{algo:init}). TextCNN \cite{zhang2015text} and LSTM are employed to build the discriminators and generators respectively. Given  $\XA$, the generator is updated by maximum likelihood estimation (line \ref{sgan_start}). From lines \ref{gen_neg} to \ref{min_entro}, $\DA$ receives the sequences generated by $\GA$ as negative data, and real data as positive samples. Between lines \ref{pregan_start} and \ref{sgan_end}, a GAN procedure is applied to optimize $\DA$ and $\GA$. $\DA$ returns regression scores that are treated as rewards for the policy gradient of $\GA$. Similarly, $\DA$ improves itself by the real world data $\XA$ and the data generated by $\GA$. Our method proceeds to update $\DB$ and $\GB$ by using the same process from lines \ref{sgan_start} to \ref{sgan_end}. $\D_{F}$ repeats the same pre-training procedure from \ref{sgan_start} to \ref{sgan_end} with $\DF, \GF, \mathbf{X}_{F}=\XA+\XB$. From line \ref{fusion_f_start} to \ref{fusion_ab_end}, the framework iteratively updates the generators and discriminators from $\D_{F}, \D_{A}, \D_{B}$. Lines \ref{fusion_f_start} to \ref{fusion_f_end} are for $\D_{F}$, while lines \ref{fusion_ab_start} to \ref{fusion_ab_end} are for $\D_{A}$ and $\D_{B}$. The update rules for the parameters of $\DF$ and $\GF$ have been covered in subsection (\ref{sec:f_fusiongan}), while those for the parameters of $\GA$ and $\DA$ are described in subsection (\ref{sec:ab_fusiongan}). Again, $\D_{B}$ follows the same process as that of $\D_{A}$. The algorithm stops when the discriminators converge.

\IEEEpeerreviewmaketitle

\section{Evaluation}\label{exp}

\begin{figure*}[!hbpt]
        \centering
        {
            \subfigure{
                \begin{tikzpicture}
                    \begin{axis}[
                        xtick={0,5,10,15},
                        yticklabels={,,},
                        scaled y ticks={base 10:0},
                        legend pos=north east,
                        ymajorgrids=true,
                        grid style=dashed,
                        legend style={font=\tiny},
                    ]
                    \addplot[
                        color=orange,
                        mark=*,
                    ]
                    coordinates {
(0,124689) (1,12475) (2,5047) (3,2573) (4,1493) (5,1024) (6,694) (7,451) (8,355) (9,264) (10,208) (11,167) (12,105) (13,59) (14,50) (15,48) (16,32) (17,25) (18,25) (19,31)                };
                    \legend{jazz}

                    \end{axis}
                \end{tikzpicture}
                \label{}
            }\hspace{0pt}%
            \subfigure{
                \begin{tikzpicture}
                    \begin{axis}[
                        xtick={0,5,10,15},
                        yticklabels={,,},
                        scaled y ticks={base 10:0},
                        legend pos=north east,
                        ymajorgrids=true,
                        grid style=dashed,
                        legend style={font=\tiny},
                    ]
                    \addplot[
                        color=magenta,
                        mark=*,
                    ]
                    coordinates {
(0,18739) (1,8697) (2,1390) (3,997) (4,109) (5,96) (6,20) (7,21) (8,4) (9,1) (10,4) (11,0) (12,0) (13,3) (14,0) (15,0) (16,0) (17,0) (18,0) (19,0)                  };
                    \legend{folk}

                    \end{axis}
                \end{tikzpicture}
                \label{}
            }\hspace{0pt}%
            \subfigure{
                \begin{tikzpicture}
                    \begin{axis}[
                        xtick={0,5,10,15},
                        yticklabels={,,},
                        legend pos=north east,
                        ymajorgrids=true,
                        grid style=dashed,
                        legend style={font=\tiny},
                    ]
                    \addplot[
                        color=cyan,
                        mark=triangle*,
                    ]
                    coordinates {
(0,8047) (1,594) (2,248) (3,157) (4,99) (5,61) (6,50) (7,51) (8,35) (9,21) (10,19) (11,16) (12,11) (13,10) (14,14) (15,4) (16,0) (17,0) (18,1) (19,0)                     };
                    \legend{GAN}

                    \end{axis}
                \end{tikzpicture}
            }\hspace{0pt}%
            \subfigure{
                \begin{tikzpicture}
                    \begin{axis}[
                        xtick={0,5,10,15},
                        yticklabels={,,},
                        legend pos=north east,
                        ymajorgrids=true,
                        grid style=dashed,
                        legend style={font=\tiny},
                    ]
                    \addplot[
                        color=cyan,
                        mark=triangle*,
                    ]
                    coordinates {
(0,7970) (1,1080) (2,389) (3,232) (4,112) (5,73) (6,49) (7,29) (8,27) (9,25) (10,13) (11,10) (12,10) (13,7) (14,3) (15,0) (16,0) (17,0) (18,0) (19,0)                     };
                    \legend{MLE}

                    \end{axis}
                \end{tikzpicture}
                \label{}
            }\hspace{0pt}%
            \subfigure{
                \begin{tikzpicture}
                    \begin{axis}[
                        xtick={0,5,10,15},
                        yticklabels={,,},
                        legend pos=north east,
                        ymajorgrids=true,
                        grid style=dashed,
                        legend style={font=\tiny},
                    ]
                    \addplot[
                        color=cyan,
                        mark=triangle*,
                    ]
                    coordinates {
(0,555) (1,2078) (2,386) (3,291) (4,145) (5,111) (6,74) (7,40) (8,39) (9,17) (10,17) (11,9) (12,8) (13,8) (14,5) (15,3) (16,1) (17,1) (18,1) (19,2)                     };
                    \legend{RL}

                    \end{axis}
                \end{tikzpicture}
                \label{}
            }\hspace{0pt}%
             \subfigure{
                \begin{tikzpicture}
                    \begin{axis}[
                        xtick={0,5,10,15},
                        yticklabels={,,},
                        scaled y ticks={base 10:0},
                        legend pos=north east,
                        ymajorgrids=true,
                        grid style=dashed,
                        legend style={font=\tiny},
                    ]
                    \addplot[
                        color=cyan,
                        mark=triangle*,
                    ]
                    coordinates {
(0,13636) (1,2061) (2,596) (3,374) (4,163) (5,106) (6,59) (7,54) (8,47) (9,20) (10,19) (11,14) (12,10) (13,6) (14,7) (15,8) (16,0) (17,4) (18,3) (19,2)                     };
                    \legend{MC}
                    \end{axis}
                \end{tikzpicture}
                \label{}
            }\hspace{0pt}%
            \subfigure{
                \begin{tikzpicture}
                    \begin{axis}[
                        xtick={0,5,10,15},
                        yticklabels={,,},
                        legend pos=north east,
                        ymajorgrids=true,
                        grid style=dashed,
                        legend style={font=\tiny},
                    ]
                    \addplot[
                        color=cyan,
                        mark=triangle*,
                    ]
                    coordinates {
(0,133) (1,119) (2,121) (3,139) (4,101) (5,124) (6,117) (7,121) (8,123) (9,127) (10,130) (11,105) (12,135) (13,125) (14,131) (15,132) (16,136) (17,128) (18,135) (19,123)                     };
                    \legend{RM}
                    \end{axis}
                \end{tikzpicture}
            }\hspace{0pt}%
            \subfigure{
                \begin{tikzpicture}
                    \begin{axis}[
                        xtick={0,5,10,15},
                        yticklabels={,,},
                        legend pos=north east,
                        ymajorgrids=true,
                        scaled y ticks={base 10:0},
                        grid style=dashed,
                        legend style={font=\tiny},
                    ]
                    \addplot[
                        color=violet,
                        mark=*,
                    ]
                    coordinates {
(0,11181) (1,508) (2,48) (3,11) (4,5) (5,3) (6,3) (7,2) (8,0) (9,0) (10,0) (11,0) (12,1) (13,0) (14,0) (15,0) (16,0) (17,0) (18,0) (19,0)                };
                    \legend{Fusion}
                    \end{axis}
                \end{tikzpicture}
                \label{}
            }\hspace{0pt}%
            \subfigure{
                \begin{tikzpicture}[trim axis right]
                    \begin{axis}[
                        scaled y ticks={base 10:0},
                        xtick=data,
                        xticklabels={C,, D,, E,,F\#,,G\#,,A\#,},
                        yticklabels={,,},
                        legend pos=south west,
                         ymajorgrids=true,
                        grid style=dashed,
                    ]

                    \addplot[
                        color=orange,
                        mark=*,
                    ]
                    coordinates {
(0,13948) (1,8166) (2,11687) (3,11631) (4,8240) (5,15458) (6,6584) (7,14181) (8,10243) (9,10258) (10,14071) (11,7061)                     };

                    \end{axis}
                \end{tikzpicture}
            }\hspace{0pt}%
            \subfigure{
                \begin{tikzpicture}
                    \begin{axis}[
                        xtick=data,
                        xticklabels={C,, D,, E,,F\#,,G\#,,A\#,},
                        yticklabels={,,}
                        legend pos=south west,
                        ymajorgrids=true,
                        grid style=dashed,
                    ]

                    \addplot[
                        color=magenta,
                        mark=*,
                    ]
                    coordinates {
(0,5169) (1,427) (2,5303) (3,372) (4,4155) (5,2145) (6,1286) (7,5024) (8,492) (9,4677) (10,979) (11,3921)                     };
                    \end{axis}
                \end{tikzpicture}
                \label{}
            }\hspace{0pt}%
            \subfigure{
                \begin{tikzpicture}
                    \begin{axis}[
                        xtick=data,
                        xticklabels={C,, D,, E,,F\#,,G\#,,A\#,},
                        yticklabels={,,}
                        legend pos=south west,
                        ymajorgrids=true,
                        grid style=dashed,
                    ]
                    \addplot[
                        color=cyan,
                        mark= triangle*,
                    ]
                    coordinates {
(0,869) (1,629) (2,448) (3,1650) (4,368) (5,863) (6,985) (7,503) (8,1020) (9,369) (10,959) (11,699)                     };

                    \end{axis}
                \end{tikzpicture}
                \label{}
            }\hspace{0pt}%
            \subfigure{
                \begin{tikzpicture}
                    \begin{axis}[
                        xtick=data,
                        xticklabels={C,, D,, E,,F\#,,G\#,,A\#,},
                        yticklabels={,,},
                        legend pos=south west,
                        ymajorgrids=true,
                        grid style=dashed,
                    ]
                     \addplot[
                        color=cyan,
                        mark= triangle*,
                    ]
                    coordinates {
(0,1170) (1,538) (2,814) (3,629) (4,578) (5,1010) (6,481) (7,1099) (8,572) (9,749) (10,868) (11,557)                     };

                    \end{axis}
                \end{tikzpicture}
                \label{}
            }\hspace{0pt}%
            \subfigure{
                \begin{tikzpicture}
                    \begin{axis}[
                        xtick=data,
                        xticklabels={C,, D,, E,,F\#,,G\#,,A\#,},
                        yticklabels={,,},
                        legend pos=south west,
                        ymajorgrids=true,
                        grid style=dashed,
                    ]
                    \addplot[
                        color=cyan,
                        mark= triangle*,
                    ]
                    coordinates {
(0,5270) (1,1) (2,134) (3,92) (4,629) (5,151) (6,501) (7,279) (8,3) (9,117) (10,22) (11,356)                     };

                    \end{axis}
                \end{tikzpicture}
                \label{}
            }\hspace{0pt}%
            \subfigure{
                \begin{tikzpicture}
                    \begin{axis}[
                        xtick=data,
                        xticklabels={C,, D,, E,,F\#,,G\#,,A\#,},
                        yticklabels={,,},
                        legend pos=south west,
                        ymajorgrids=true,
                        grid style=dashed,
                    ]

                    \addplot[
                        color=cyan,
                        mark= triangle*,
                    ]
                    coordinates {
(0,2104) (1,926) (2,1915) (3,1287) (4,1364) (5,1811) (6,888) (7,2099) (8,1162) (9,1608) (10,1640) (11,1185)        };

                    \end{axis}
                \end{tikzpicture}
                \label{}
            }\hspace{0pt}%
            \subfigure{
                \begin{tikzpicture}
                    \begin{axis}[
                        xtick=data,
                        legend pos=south west,
                        xticklabels={C,, D,, E,,F\#,,G\#,,A\#,},
                        yticklabels={,,},
                        ymajorgrids=true,
                        grid style=dashed,
                    ]

                   \addplot[
                        color=cyan,
                        mark= triangle*,
                    ]
                    coordinates {
(0,258) (1,227) (2,198) (3,216) (4,221) (5,176) (6,244) (7,218) (8,169) (9,223) (10,209) (11,205)                    };
                    \end{axis}
                \end{tikzpicture}
                \label{}
            }\hspace{0pt}%
            \subfigure{
                \begin{tikzpicture}
                    \begin{axis}[
                        xtick=data,
                        xticklabels={C,, D,, E,,F\#,,G\#,,A\#,},
                        yticklabels={,,},
                        legend pos=south west,
                        ymajorgrids=true,
                        grid style=dashed,
                    ]
                    \addplot[
                        color=violet,
                        mark=*,
                    ]
                    coordinates {
(0,3849) (1,135) (2,3948) (3,219) (4,1314) (5,1200) (6,172) (7,2941) (8,115) (9,487) (10,147) (11,328)                     };
                    \end{axis}
                \end{tikzpicture}
            }\hspace{0pt}%
        }
        \caption{upper line: \textbf{DD} (\textbf{x-axis}: duration length;\textbf{y-axis}: percentage); lower line: \textbf{NPD} (\textbf{x-axis}: pitch class; \textbf{y-axis}: percentage)
        }
        \label{fig:duradis}
\end{figure*}

Training data includes two datasets called the Weimar jazz dataset\footnote{http://jazzomat.hfm-weimar.de} and the Essen Associative Code (EsAC) folk dataset\footnote{http://www.esac-data.org}. Both of these datasets contain symbolic data of single-voiced tracks, which support our objective of generating solo melodies. The baseline methods include:

\begin{enumerate}
    \item \textbf{Random Mixing (RM)}: \textbf{RM} randomly selects notes and duration from the pitch and duration distribution on the combination of $\XA$ and $\XB$.
    \item \textbf{Monte Carlo Sampling (MC)}: After calculating the note pitch and duration distributions on the combination of $\XA$ and $\XB$, \textbf{MC} samples notes and duration subject to these distributions.
    \item \textbf{Maximum Likelihood Estimation (MLE)}: Employing LSTM, \textbf{MLE} directly learns a pattern from the mixture of $\XA$ and $\XB$ by predicting the next element.
    \item \textbf{GAN}: The model directly learns a pattern from the mixture of $\XA$ and $\XB$ using GAN.
    \item \textbf{Reinforcement Learning (RL)}:  After applying GAN on $\DA-\GA, \DB-\GB$, we exchange the discriminators $\DA$ and $\DB$ which are treated as rewards function to $\GB$ and $\GA$ respectively.
    \end{enumerate}

\subsection{Quantitative Study}
Due to the aesthetic nature of the task, a quantitative evaluation can only give a first impression of the power of the generative system. Here, we investigate the distance between pitch and note duration distributions, knowingly discarding all sequential information of the melodies.

The ideal output should keep equally close to the given domains and minimize this distance. Two distributions of properties are selected for the study.
\begin{enumerate}
    \item \textbf{Duration Distribution (DD)}: Notes are categorized by their durations.
    \item \textbf{Normalized Pitch Distribution (NPD)}: All tokens are categorized into C/C\#/D/D\#/E/F/F\#/G/G\#/A/A\#/B.
\end{enumerate}

To quantify the similarity between distributions, two symmetric metrics, the Euclidean distance (\textbf{EUD}) and the Wasserstein-1 metric (Earth Mover distance (\textbf{EM})), are employed to calculate the distance between the distributions of generated sequences and those of domains $\XA$ and $\XB$. Intuitively, the distance between $\XA$ and $\XB$, between $\XA, \D_{F}$,  and between $\XB, \D_{F}$ should satisfy the triangle inequality, and the optimal configuration should minimize the inequality, i.e.,
{\footnotesize
\begin{equation*}
\label{sdpfim}
\begin{aligned}
& \underset{}{\text{min}}
& & Diff = \|\XA- \GF(z)\|+\|\XB-\GF(z)\| - \|\XA - \XB\|, \\
& \text{}
&& Ratio =\frac{\|\XA- \GF(z)\| + \|\XB-\GF(z)\|}{\|\XA - \XB\| }.  \\
\end{aligned}
\end{equation*}
}

\begin{table}[!hpbt]
\centering
\caption{Distance between Distributions (the lower the better)}
\label{tab:distcomp}
\scalebox{1}{
\begin{tabularx}{\linewidth}{c|XX|XX
\toprule
   & \multicolumn{2}{l|}{\textbf{EUD}} & \multicolumn{2}{l}{\textbf{EM}} \\ \cline{2-5}
         & Diff          & Ratio           & Diff          & Ratio          \\ \hline
\multicolumn{5}{c}{\textbf{DD}}                            \\ \hline
\textbf{RM} & 39742.2 & 1.375 & 2757.6 & 1.461 \\
\textbf{MLE} & 24546.4 & 1.231 & 2005.2 & 1.335 \\
\textbf{GAN} & 24765.2 & 1.233 & 2064.3 & 1.345 \\
\textbf{RL} & 37971.2 & 1.358 & 2629.0 & 1.439 \\
\textbf{MC} & 13988.7 & 1.132 & 1289.2 & 1.215 \\
\hline
\textbf{Fusion} & 19452.6 & 1.183 & 1831.9 & 1.306 \\
\hline
\multicolumn{5}{c}{\textbf{NPD}}                            \\ \hline
\textbf{RM} &  19586.6 & 1.647 & 5231.0 & 1.643 \\
\textbf{MLE} & 15921.4 & 1.526 & 4147.5 & 1.510 \\
\textbf{GAN} & 16807.1 & 1.555 & 4098.0 & 1.504 \\
\textbf{RL}  & 16927.1 & 1.559 & 4399.2 & 1.541 \\
\textbf{MC} & 11175.0 & 1.369 & 2660.2 & 1.327 \\
\hline
\textbf{Fusion} & 11564.6 & 1.382 & 3182.5 & 1.391 \\
\hline
\bottomrule
\end{tabularx}
}
\end{table}

Figure~\ref{fig:duradis} (top) shows the note duration distributions (DD) and Table \ref{tab:distcomp} (top) gives the corresponding distance measures. The note duration distribution of jazz and folk have shapes that resemble the power law distribution. The shape of \textbf{RM} is very different from jazz or folk. Visually, \textbf{RL}'s duration distribution does not match that of jazz and folk because the portion of notes at the length of 0 and 1 (the peak) were relatively higher than that of jazz or folk. The other baselines (\textbf{GAN}, \textbf{MLE}, \textbf{MC}) appear similar to the given domains because they conform to the power law distributions. \textbf{Fusion} is roughly at the same level of \textbf{MC}. Note that, given our evaluation metrics, \textbf{MC} has the best performance in this experiment since it directly samples notes from the true distribution. This disregard of sequential information, however, results in unnatural rhythmic qualities of the music clips generated by \textbf{MC}.

Figure~\ref{fig:duradis} (bottom) shows the pitch distributions (NPD) and Table \ref{tab:distcomp} (bottom) gives the corresponding distance measures.
\textbf{RL}'s pitch distribution is ---due to overfitting--- different from that of jazz and folk, which shows its low capacity in modeling pitch distribution. The single peak at D\# as generated by \textbf{GAN} does not match either the jazz or folk distributions. The distributions of the other baselines are difficult to categorize as a good or bad match. Since \textbf{MC} is directly sampled from real distributions, its configurations should be exactly the same as that of the optimal fusion. \textbf{MC} shows two valleys at C\# and F\#, and two peaks at C and G, which should be expected. \textbf{RM} satisfies one of them (F\# is a valley), while \textbf{RM} meets only one standard (C is the peak). \textbf{GAN} fails in all the characteristic pitch classes, and both \textbf{MLE} and \textbf{Fusion} meet all of them. Quantitatively, the ratio of \textbf{RM} in \textbf{NPD} is the highest which means that \textbf{RM} is the worst in modeling pitch. \textbf{GAN} and \textbf{RL} improve \textbf{RM} by around 10\%. As one of the two distributions that look most similar to the original distributions, \textbf{MLE} outperforms \textbf{RM}, \textbf{RL}, and \textbf{GAN}. Our method \textbf{Fusion} has the second lowest ratio after \textbf{MC}. However, as mentioned above, the \textbf{MC} does not consider the sequential consistence, resulting in bad evaluations in the listening test described in Sect.~\ref{case}.

\subsection{User Study via Listening Test}\label{case}
To evaluate the subjective quality of the generated sequences, a user study was conducted via Amazon Mechanical Turk. The listening test was divided into three steps:

\begin{enumerate}
    \item \textbf{Qualification test}: \textit{``Listen to the music, and choose one genre below that most matches the music''}. This is to test the evaluators' qualification level. The provided choices are \textit{(A) jazz, (B) folk, (C) neither}.
    \item \textbf{Fusion recognition}: \textit{``Listen to the attached music, and then choose the choice that most matches the music''} Possible answers are \textit{(A) pure jazz, (B) pure folk, (C) mixture of jazz and folk, (D) neither}. Each of the pieces of music generated by our proposed method and the baselines are assigned to one such question.
    \item \textbf{Musicality}: \textit{``Listen to the attached music in 2nd step, who do you think its composer''}, the allowed answers are \textit{(A) expert, (B) newbie, (C) robot}. Similarly, each generated sample is associated with one such question.
\end{enumerate}
We randomly drew from the training data and prepared 800 sets for the first question. In the first step, 66.5\% of candidates chose the correct answer, which is an acceptable rate given the inherent difficulties of genre identification for average listeners. The statistics resulting from the valid answers are shown in Table \ref{amt}.
Each of the methods generated 500 samples, which were randomly selected for the second and third questions.

As shown in Table \ref{amt}, the best system based on the percentage of \textit{mixture} is \textbf{MLE}. A closer look at \textbf{MLE}, however, reveals the unbalanced distribution between \textit{jazz} and \textit{folk}, which implies its bias towards the jazz genre. On the other hand, \textbf{RM} is the best system based on the balance between \textit{jazz} and \textit{folk}, but the high percentage of \textit{neither} suggests the confusion of the listeners. It is clear that a single criterion is insufficient for determining the best system.
Therefore, we design a metric to summarize the results and represent the fusion level ($\mathbf{FL}$) of the evaluated systems with the following equation:
{\small
\begin{equation*}
\label{fusionlvl}
\begin{aligned}
& \underset{}{\text{maximize}}
& &
\mathbf{FL}=1-\frac{\|C_{jazz}-C_{folk}\|+C_{neither}}{C_{jazz}+C_{folk}+C_{mixture}+C_{neither}},\\
\end{aligned}
\end{equation*}
}where $C_{i}$ indicates the count of $i$. $\|C_{jazz}-C_{folk}\|$ means the unbalanced error, and $C_{neither}$ can be treated as another type of error. A higher $\mathbf{FL}$ value implies better fusion.\\
In the $Musicality$ test, \textbf{MLE} (45.5\%), \textbf{GAN} (42.0\%), and \textbf{Fusion} (43.8\%) were voted by the majority as \textit{expert}. Overall, \textbf{Fusion} shows a balanced performance in both $Fusion recognition$ and $Musicality$ tests, which demonstrates the effectiveness of our proposed method.

\begin{table}
\caption{Listening test results
}
\label{amt}
\begin{center}
\vspace{-0.5em}
 \begin{tabularx}
 {\linewidth}{c|>{\hsize=.7\hsize}X >{\hsize=.7\hsize}X >{\hsize=.7\hsize}XX|X}
 \toprule

\multicolumn{6}{c}{\textbf{Fusion recognition}}                            \\ \hline
    & \textit{jazz} & \textit{folk} & \textit{mixture} & \textit{neither} &\textbf{FL}\\ \hline
\textbf{RM}   & 25.0\% & 22.5\% & 12.5\%  & 40\%& 57.5\%\\ \hline
\textbf{MLE}   & 43.6\% & 9.1\% & 30.9\%  & 16.4\% & 49.1\%\\ \hline
\textbf{GAN}   & 34.0\% & 17.0\% & 26.0\% &14\%& 69\%\\ \hline
\textbf{RL}   & 20.1\% & 28.3\% & 20.8\%  & 30.8\%& 61\% \\ \hline
\textbf{MC}   & 32.0\% & 2.0\% & 14.0\% & 52\%& 16\% \\ \hline
\textbf{Fusion}   & 35.9\% & 25.0\% & 20.0\% & 19.1\% & 70\% \\ \hline
\end{tabularx}
\smallskip

\begin{tabularx}{\linewidth}{c|X X X}
\multicolumn{4}{c}{\textbf{Musicality}}                            \\ \hline
     & \textit{expert} & \textit{newbie} & \textit{robot} \\ \hline

\textbf{RM}   & 22.5\% & 50.0\% & 27.5\%  \\ \hline
\textbf{MLE}   & 45.5\% & 21.8\% & 32.7\%  \\ \hline
\textbf{GAN}   & 42.0\% & 32.0\% & 26.0\%  \\ \hline
\textbf{RL}   & 32.1\% & 37.7\% & 30.2\%   \\ \hline
\textbf{MC}   & 30.0\% & 36.0\% & 34.0\% \\ \hline
\textbf{Fusion}   & 43.8\% & 28.1\% & 28.1\% \\
\bottomrule
\end{tabularx}
\vspace{-0.5em}
\end{center}
\end{table}
\vspace{-0.5em}

\section{Conclusion}\label{conclusion}
In this paper, we proposed a three-way GAN-based learning framework to integrate multiple domains. A Wasserstein distance based metric is introduced to indicate the blending progress. The evaluation investigated objective metrics looking at the pitch and note duration distributions of the generated data. A user study explicitly illustrates the validity of the proposed method as compared to the baselines, as the melodies generated by our model were preferred by the majority of users. While the listening test results are encouraging, future work will benefit from removing key dependence of the training data, a careful look into perceptually meaningful evaluation metrics that take into account the sequential nature of music, and a careful listening test design, including human-composed melodies for comparison.

\bibliographystyle{IEEEtranS}

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multimedia (cs.MM); Sound (cs.SD); Audio and Speech Processing (eess.AS)