Conversion-Based Dynamic-Creative-Optimization in Native Advertising

Abstract: Yahoo Gemini native advertising marketplace serves billions of impressions daily, to hundreds millions of unique users, and reaches a yearly revenue of many hundreds of millions USDs. Powering Gemini native models for predicting advertise (ad) event probabilities, such as conversions and clicks, is OFFSET - a feature enhanced collaborative-filtering (CF) based event prediction algorithm. The predicted probabilities are then used in Gemini native auctions to determine which ads to present for every serving event (impression). Dynamic creative optimization (DCO) is a recent Gemini native product that was launched two years ago and is increasingly gaining more attention from advertisers. The DCO product enables advertisers to issue several assets per each native ad attribute, creating multiple combinations for each DCO ad. Since different combinations may appeal to different crowds, it may be beneficial to present certain combinations more frequently than others to maximize revenue while keeping advertisers and users satisfied. The initial DCO offer was to optimize click-through rates (CTR), however as the marketplace shifts more towards conversion based campaigns, advertisers also ask for a {conversion based solution. To accommodate this request, we present a post-auction solution, where DCO ads combinations are favored according to their predicted conversion rate (CVR). The predictions are provided by an auxiliary OFFSET based combination CVR prediction model, and used to generate the combination distributions for DCO ad rendering during serving time. An online evaluation of this explore-exploit solution, via online bucket A/B testing, serving Gemini native DCO traffic, showed a 53.5% CVR lift, when compared to a control bucket serving all combinations uniformly at random.

\section{Introduction}\label{sec: introduction}
Yahoo Gemini native marketplace\footnote{See https://gemini.yahoo.com/advertiser/home} serves users with ads that are rendered to resemble the surrounding content (see Figure~\ref{fig: native ad} for a typical Gemini native ad). Operating with a yearly run-rate of several hundred million USDs\footnote{Due to commercial confidentiality matters, we report rough estimations or relative numbers regarding traffic volumes and revenue.}, Gemini native is one of Yahoo fastest growing and main businesses. With more than two billion impressions daily, and an inventory of a several hundred thousand active ads at any given time, this marketplace performs real-time \textit{first price}\alex{That's not up to date. We don't do GSP anymore}\oren{done} auctions that take into account budget considerations and targeting.

\begin{figure}[t]
\centering
\includegraphics[width=1.0\columnwidth]{nativeToyotaHighlander}
\caption{A typical Yahoo Gemini native ad captured from Yahoo home-page stream. The ad consists of a title, an image, a description, and a transparent sponsorship notification.}
\label{fig: native ad}
\end{figure}

In order to rank the native ads for incoming users and their specific context according to the \textit{cost per click} (CPC) price type where advertisers pay for clicks, a score (or expected revenue) is calculated by multiplying the \textit{predicted click-through rate} (pCTR) by the bid for each active ad. The pCTR is provided by a model that is periodically updated by \textsc{Offset}\xspace\ - a feature-enhanced \textit{collaborative-filtering} (CF) based event-prediction algorithm
\cite{aharon2013off}\cite{aharon2017adaptive}\cite{kaplan2021dynamic}.
\textsc{Offset}\xspace is a one-pass\footnote{Goes through the data only once.} algorithm that updates its latent factor model for every new batch of logged data using a \textit{online gradient descent} (OGD) \alex{Online gradient descent (OGD)?}\oren{done} based learning approach.
\IEEEpubidadjcol

\textit{Dynamic creative optimization} (DCO) is a recent product of the native portfolio Yahoo offers to its advertisers \cite{koren2020dynamic}. With DCO ads, advertisers may provide several assets for each native ad attribute, which are used to generate a plethora of combinations (e.g., having 3 titles, 2 images, and 3 descriptions, yields $3\times 2\times 3 = 18$ combinations or virtual native ads). Since different combinations may attract different crowds it may be beneficial to present certain combinations more frequently than others to maximize certain metrics such as \textit{click through-rate} (CTR) and revenue. The DCO product provides a simple and quick way for advertisers to identify the best ads that may be assembled from groups of assets without conducting a time consuming and expensive in-house A/B testings.

As advertisers spend is shifting toward \textit{optimize CPC} (oCPC) bidding strategy, where advertisers are interested in conversions\footnote{A conversion may follow (and be affiliated to) an impression (post-view) or a click (post-click), and may include events such as purchasing, registration, or app install.} but still pay for clicks, and since high CTR combinations do not necessarily provide high \textit{conversion rates} (CVR)\footnote{CVR is defined as the number of conversions divided by the number of impressions.}, we were required to provide a conversion based DCO solution that favors high CVR combinations. However, the event counting based approach of \cite{koren2020dynamic} that works well for clicks is problematic when conversions are considered, since CVRs are generally two orders of magnitude lower than CTRs.

In this work we present the approach used to serve the DCO best combinations for increasing CVR in Gemini native marketplace. To maintain similar system resources (i.e., model sizes and serving latency), we adhere to a two-stage solution (see \cite{aharon2019carousel}\cite{koren2020dynamic}). During the first stage, the Serving system (or Serving) conducts a regular auction where all combinations of a DCO ads participate as a single native ad \alex{where all combinations of a DCO ad participate as one item}\oren{done}. The second stage is invoked only in case a DCO ad wins the auction, and a DCO combination is drawn according to some combination distribution (a \textit{post-auction approach}). To calculate the combination distributions, we use an \textit{auxiliary combination CVR prediction model} which is not consumed by the Serving system and is used to provide the combination \textit{predicted CVRs} (pCVR) per certain traffic segments. The predictions are then turned into combination distributions where \alex{combinations having a higher pCVR}\oren{done} combinations having a higher pCVR are assigned with higher probabilities. As more conversions are accumulated, and certain DCO ad combinations are predicted to have higher CVRs, the distributions will drift from a uniform distribution, and the system will impress those combinations more frequently than others. The proposed approach may be seen as a simple and practical solution to the well know \textit{online decision problem} (see \cite[Chapter 2]{thompson2018now}), that is designed to satisfy both system requirements (such as model sizes and serving delay), and the inherent marketplace behaviour (such as time varying CVRs, and temporal trends).

After a short alpha testing phase, optimizing internal DCO ads, the system was launched for a beta testing phase, serving traffic of selected advertisers. The conversion based DCO solution was compared to a control bucket serving all combinations uniformly at random, and showed a staggering $53.5\%$ CVR lift. After the successful beta testing, the proposed conversion based DCO product is now fully deployed and available for all advertisers. The main contributions of this work are:
\begin{itemize}
\item We introduce a practical web scale optimization problem of maximizing native DCO ads CVR, under real-life constraints.
\item We present a post-auction DCO combination CVR prediction based approach. The proposed solution is simple, practical,
and easy to implement in any ad serving systems.
\alex{I think that the "tailored to" part is not a feature, but an anti-feature. I think we should emphasize that it's light weight, and easy to implement in ad serving systems}\oren{done}
\item Our solution was tested in scale, serving Gemini native users, and exhibits impressive CVR lifts in comparison to a control bucket serving all combinations uniformly at random.
\item The conversion based DCO system, is currently fully deployed, serving all Gemini native conversion ad traffic.
\end{itemize}

The rest of the paper is organized as follows. In Section~\ref{sec: background}, we bring relevant background, and cover related work in Section~\ref{sec:Related work}. We state the problem in Section~\ref{sec:problem statement}, and present our solution in Section \ref{sec:our approach}. Performance evaluation is considered in Section~\ref{sec:evaluation}. We conclude and discuss future work in Section~\ref{sec:Concluding remarks}.

\section{Relevant Background}\label{sec: background}
\subsection{{OFFSET} -- Event-Prediction Algorithm}\label{sec: offset}

Powering the Gemini native models is \textsc{Offset}\xspace (One-pass Factorization of Feature Sets): a feature enhanced collaborative-filtering (CF)-based ad event-prediction algorithm \cite{aharon2013off}\cite{aharon2017adaptive}\cite{aharon2019soft}\cite{arian2019feature}\cite{kaplan2021dynamic}. According to \textsc{Offset}\xspace, the \textit{predicted event probability} (pET) of a user $u$ and ad $a$ is roughly given by
\begin{equation}\label{eq:event probability prediction}
    \mathrm{pET}_{u,a} = \sigma(s_{u,a})\in [0,1]\ ,
\end{equation}
where $\sigma(x)=\left(1+e^{-x}\right)^{-1}$ is the \textit{Logistic sigmoid} function, and
\begin{equation}\label{eq: score}
	s_{u,a}=b+\nu_{u}^T \nu_{a}\ ,
\end{equation}
$\nu_{u},\ \nu_{a}\in \rm I\!R^D$ denote the user and ad \textit{latent factor} (LF) vectors respectively, and $b\in \rm I\!R$ denotes the model bias. The product $\nu_{u}^T \nu_{a}$ indicates the tendency of user $u$ towards ad $a$, where a higher score translates into a higher pET.
\Comment{It is noted that $\Theta=\{\nu_{u},\nu_{a},b\}$ are the model parameters which are learned from the logged data.} \alex{Well, they aren't. The feature latent vectors are the model's parameters}\oren{you are correct. I hope the reader is capable of this leap}.
It is noted that $\{\nu_{u},\nu_{a}\}$ are constructed from the model parameters $\Theta$ which are learned from the logged data and described in the sequel.

As mentioned above, \textsc{Offset}\xspace is used to drive many of Gemini native models, including (a) \textit{Click model} for predicting a click event (pCTR); (b) \textit{Conversion model} for predicting a conversion given a click event (pCONV); and (c) \textit{Ad close model} for predicting an ad close event (pCLOSE) \cite{silberstein2020ad}.

The ad and user LF vectors are constructed using their features to overcome data sparsity issues (ad events such as click, conversion, or close, are quite rare). For ads, a simple summation of their features LF vectors (e.g., ad id, campaign id, advertiser id, ad categories, etc.), all in dimension $D$, is used. The interaction between the different $d$-dimension user feature LF vectors to produce the user $D$-dimension LF vector is more complicated to support non-linear dependencies \alex{non-linear pairwise interactions?}\oren{I think it's good enough} between feature pairs.

\begin{figure}[t]
\centering
\includegraphics[width=0.95\columnwidth]{offsetModelFit.pdf}
\caption{Example of a user latent factor vector construction for $o=4,\ s=2$ and $K=3$ features (i.e., age, gender, and geo). The three relevant feature value vectors of dimension $d=10$ are retrieved from the model, appended with 1s, and multiplied entry wise to produce the final user vector of dimension $D=18$ on the right.}
\label{fig: user vector construction}
\vspace{0.1cm}
\end{figure}

The user vectors are constructed using their $K$-feature learned vectors $v_k\in \rm I\!R^d$, $k\in\{1,...,K\}$ (e.g., gender values, age values, , device types, geo values, etc.). In particular, $o$ entries are allocated to each pair of user features, and $s$ entries are devoted to each feature vector alone. The dimension of a single feature value vector is therefore $d=(K-1)\cdot o + s$,
whereas the dimension of the combined user vector is $D=\binom{K}{2} \cdot o + K\cdot s$. An illustration of this construction is depicted in Fig.~\ref{fig: user vector construction}. The advantage over the conventional CF approach is that the model includes only $O(K)$ LF vectors, one for each feature value (e.g., three for gender - female, male, and unknown) instead of hundreds of millions of unique user LF vectors. Note that \textsc{Offset}\xspace may be seen as an enhancement of \alex{a special case of $\to$ an enhancement of}\oren{done} a \textit{factorization machine} (FM) \cite{rendle2010factorization} or as a variant of a \textit{field-aware factorization machine} (FFM) \cite{juan2017field}\cite{juan2016field}.

To learn the model parameters $\Theta$, \textsc{Offset}\xspace minimizes the \textit{logistic} loss (or LogLoss) of the training data set $\mathcal{T}$ (i.e., past negative and positive events) using a one-pass \textit{online gradient descent} (OGD)
\begin{equation*}
\operatornamewithlimits{argmin}_{\Theta}\!\!\!\! \sum_{(u,a,y)\in \mathcal{T}} \mathcal{L}(u,a,y)\ ,
\end{equation*}
where $\mathcal{L}(u,a,y)$ equals
\Comment{\begin{multline}
\mathcal{L}(u,a,y)=\\-(1-y)\log\left(1-\mathrm{pEVENT}_{u,a}\right)-y \log \left(\mathrm{pEVENT}_{u,a}\right)+\frac{\lambda}{2}\sum_{\theta\in\Theta}\theta^2\ ,
\end{multline}}
\[
-(1-y)\log\left(1-\mathrm{pET}_{u,a}\right)-y \log \left(\mathrm{pET}_{u,a}\right)+\frac{\lambda}{2} \|\Theta\|_2^2\ ,
\]
\alex{In the formula above $\displaystyle \frac{\lambda}{2}\sum_{\theta\in\Theta}\theta^2$ $\to$ $\frac{\lambda}{2} \|\Theta\|_2^2$? More compact, and standard notation.}\oren{done}
$y \in \{0,1\}$ is the indicator (or label) for the event involving user $u$ and ad $a$, and $\lambda$ denotes the $L2$ regularization parameter. The OGD step sizes are determined by a variant of the \textit{adaptive gradient} (AdaGrad) algorithm \cite{duchi2011adaptive} (see \cite{aharon2017adaptive}).

The \textsc{Offset}\xspace algorithm applies an incremental training approach, where it continuously updates its model parameters with each batch of new training events (e.g., every 15 minutes for the click model, or 4 hours for the conversion model).
The \textsc{Offset}\xspace algorithm also includes an \textit{adaptive online hyper-parameter tuning mechanism} \cite{aharon2017adaptive}. This mechanism utilizes the parallel \textit{map-reduce} architecture of the Gemini native backend platform, and attempts to tune \textsc{Offset}\xspace hyper-parameters (e.g., OGD initial step size and AdaGrad parameters) to match the varying marketplace conditions (such as trend and temporal effects). We note that other components of \textsc{Offset}\xspace, such as similarity weights used for applying ``soft'' recency and frequency rules\footnote{How recent and how frequent a user may be impressed with the same ad or campaign.} are not described here for brevity (see \cite{aharon2019soft}).

\subsubsection{Weighted multi-value feature type}\label{sec:offset wmv feature}
The most generic feature type used by the \textsc{Offset}\xspace algorithm is the \textit{weighted multi-value feature} type. According to this feature type, the model includes a $d$-dimension LF vector for each of the $m$ feature values seen so far. In this case, the $d$-dimension vector of this weighted multi-value feature for user $u$ is\footnote{The definition is also valid for ad entities, however, the feature values LF vectors and resulting vector in this case are of dimension $D$.}
\[
v=\frac{1}{\sqrt{n}}\sum_{i=1}^n w_{\ell_i}\ v_{\ell_i}\ ,
\]
\alex{Since we're not actually using the weights anywhere in this paper, it might be more beneficial to just describe multi-value features}\oren{I agree but I think it's ok}
where $\{\ell_i\}$ and $\{w_{\ell_i}\}$ are the $n$ feature values and accompanied weights associated with user $u$, and $\{v_{\ell_i}\}$ are the model LF vectors associated with values $\{\ell_i\}$. It is emphasized that the weights are given as part of the users' data and are not model parameters that are needed to be learned. Each time \textsc{Offset}\xspace encounters a new value for the feature, it assigns a random Gaussian vector to it with zero mean and covariance matrix $\eta\cdot I_d$, where $0\le \eta\ll 1$ and $I_d$ is the identity matrix\footnote{Other more sophisticated ``cold-start'' strategies are beyond the scope of this work.}. It is noted that simpler feature types such as \textit{categorical features} (e.g., age and gender), or non-weighted multi-value features (e.g., user category feature where all weights equal 1), are merely special cases of the weighted multi-value feature type.

\subsection{Serving ads}\label{sec:serving}
 When a user arrives at a Yahoo O\&O\footnote{\textit{Owned and Operated.}
 site such as \textit{Yahoo news}, \textit{Yahoo mail}, and \textit{Yahoo finance}.} or Syndication\footnote{When Yahoo presents ads at third party site and shares revenue with the site owner.}\alex{Is the distinction between O\&O or Syndication important? I think it just adds redundant information which confuses the readers}\oren{It shows that we are a major exchange with our own sites as well as other sites} site, and a native slot should be populated by an ad, an auction takes place. At first, the Serving system generates a list of scores for all eligible active ads. In general, an ad's eligibility to a certain user in a certain context is determined by \textit{targeting}, which is beyond the scope of this work, and relates to user characterization (such as age, gender, and geo) specified by the advertiser to approach certain crowds.

\paragraph*{Auction}
The score is a measure that attempts to rank ads according to their expected revenue with respect to the arriving user and her context (i.e., her features, e.g., age, gender, geo, day, hour, site, device type, etc.). Roughly, an ad's score is defined as
\begin{equation}
\label{eq:rankingScore}
\mathrm{Score}_{u,a} = \mathrm{bid}_a \cdot \mathrm{pCTR}_{u,a}\ ,
\end{equation}
where $\mathrm{pCTR}_{u,a}$ (predicted click probability) is calculated by the \textsc{Offset}\xspace main click model (see Eq. \eqref{eq:event probability prediction} for a click event), and $\mathrm{bid}_a$
(in USD) is the amount of money the advertiser is willing to spend for a click on a \textit{manual cost per click} (mCPC) biding strategy ad $a$
\alex{We'd like to stress that a DCO ad, along with all its assets, participates as one unit in the auction, and receives a single pCTR score.}\oren{I already added it before and it is mentioned explicitly in the DCO definition section}
For an \textit{optimized cost per click} (oCPC) biding strategy ad $\mathrm{bid}_a=\mathrm{pCONV}_{u,a}\cdot \mathrm{tCPA}$, where $\mathrm{pCONV}_{u,a}$ (predicted conversion probability given a click event) is calculated by the \textsc{Offset}\xspace main conversion model (see Eq. \eqref{eq:event probability prediction} for a conversion given click event), and the \textit{target cost per action} (tCPA) is the average cost the advertiser is expecting to spend on a conversion while still paying for clicks. Other, more sophisticated biding strategies such as \textit{enhanced optimized cost per click } (eCPC), where part of the traffic is used to estimate the conversion price, are outside of our scope.

According to the \textit{first price} auction, the winning ad is an eligible ad with the highest score $\mathrm{Score}_{u,a}$ (see \eqref{eq:rankingScore}).

\Comment{To encourage advertiser truthfulness, the cost incurred by the winner of the auction is according to the \textit{generalized second price} (GSP) \cite{edelman2007internet}, which is defined as
\begin{equation}
\label{eq:gsp}
\mathrm{gsp} = \frac{\mathrm{Score}_{u,b}}{\mathrm{Score}_{u,a}} \cdot \mathrm{bid}_a = \frac{\mathrm{pCTR}_{u,b}}{\mathrm{pCTR}_{u,a}} \cdot \mathrm{bid}_b\ ,
\end{equation}
where $a$ and $b$ correspond to the winner of the auction and the runner-up, respectively. Note that by definition $\mathrm{gsp}\le\mathrm{bid}_a$, which means the winner will pay no more than its bid. Moreover, the winner will pay the minimal price required for winning the auction. In particular, if both ads have the same pCTR with respect to the user, the winner will pay the bid of the runner-up (i.e., $\mathrm{bid}_b$).}

\subsection{Native DCO ads}\label{sec:dco ads}
Native ads are designed to resemble the surrounding content of the site (or Web page) and are considered less intrusive than other ad types (see \cite{wojdynski2016native}). In general, Gemini native ads format usually includes several attributes such as title, image, and a short description (see Figure~\ref{fig: native ad} for a typical Gemini native ad). Other ad formats with richer structures such as Video and Carousel ads \cite{aharon2019carousel}, are outside the scope of this work.

\begin{figure}[t]
\centering
\includegraphics[width=0.95\columnwidth]{nativeToyotaDcoCombinations.png}
\caption{A typical native DCO ad and its assets (2 titles, 3 images, and 3 descriptions) along with the resulting $18 = 2\times 3 \times 3$ combinations of virtual native ads.}
\label{fig: DCO ad}
\end{figure}

Two years ago, Yahoo started offering DCO native ads to its customers. With DCO ads, advertisers are allowed to upload up to 3 assets per each native ad attribute (e.g., 3 titles, 2 images, and 3 descriptions). This creates a plethora of combinations of ``conventional'' native ads that may be presented to users. In Figure~\ref{fig: DCO ad}, a real DCO ad and its assets are shown (i.e., 2 titles, 3 images, and 3 descriptions) along with the resulting $2\times 3\times 3 = 18$ combinations (or virtual native ads since combinations have no actual ad ID in the Gemini native system) that are generated by selecting one asset of each attribute. In case a DCO ad wins the auction and is about to be impressed, one of the combinations should be selected and rendered for presentation.

\section{Related work}\label{sec:Related work}
The problem of \textit{dynamic creative optimization} (DCO)\footnote{See https://en.wikipedia.org/wiki/Dynamic\_Creative\_Optimization .}, or finding the top combinations in terms of some metric such as CTR, CVR, or revenue, can be formulated as a reinforcement learning problem, where agents take actions to maximize some notion of cumulative reward \cite{sutton2018reinforcement}. In particular, our problem is closely related to the rich literature of the \textit{multi-armed bandit} (MAB) problem, which comes in several flavors, such as \textit{best arm identification} \cite{audibert2010best,karnin2013almost} and regret minimization \cite{bubeck2012regret}. The MAB problem is also studied in many settings, including mortal bandits \cite{chakrabarti2009mortal}, restless bandits \cite{slivkins2008adapting}, distributed bandits with feedback delay \cite{hillel2013distributed}, bandits with non-stationary rewards \cite{besbes2014stochastic}, and contextual bandits \cite{li2010contextual}. Our problem is also closely related to the \textit{online decision problem} \cite{thompson2018now}. Moreover, the use of an auxiliary prediction model to provoke actions is inspired by \textit{Thompson sampling}, which is widely used to address such problems (see \cite{chapelle2011empirical} for online advertising application). Other works that also adopt a holistic approach (as opposed to our post-auction approach) are \cite{hofmann2013balancing} and \cite{slivkins2013ranked}. Explore-exploit techniques are widely used to optimize Web content such as new items and ads. For example, \cite{agarwal2009online} describes an explore-exploit based system for selecting news items for Yahoo Homepage. There are also many commercial tools for conducting various ad related optimizations on social platforms (see \cite{lempel2012hierarchical} and references therein). It is also worth mentioning that similar DCO products are offered also by other online advertising providers such as Facebook, Google, and LinkedIn.

In the sequel we shall demonstrate that due to system requirements and constraints, our solution is based on a post-auction approach, which separates ad ranking and DCO ad rendering (as oppose to works that adopt a more holistic approach, e.g., \cite{agrawal2012analysis}\cite{hofmann2013balancing}\cite{li2010contextual}\cite{slivkins2013ranked}). The post-auction approach was used in \cite{koren2020dynamic} for a DCO product, however, with CTR as optimization goal. A \textit{successive elimination} algorithm based on robust CTR measurements was used to identify the ``best'' combinations (see also \cite{aharon2019carousel} for a similar setting which considers \textit{asset optimization} for Carousel ads). It is noted that although we consider a similar DCO product in the same native advertising marketplace as \cite{koren2020dynamic}, we are interested in maximizing CVR which is much lower than CTR in general. To overcome the extreme data sparsity condition, we apply a totally \textbf{different} \alex{radically different?}\oren{totally is not enough :-)} approach than that of \cite{koren2020dynamic}, which is based on \textit{event prediction} instead of \textit{event counting}. In addition, the proposed solution does not ``eliminate" combinations (as opposed to that of \cite{koren2020dynamic}) and inherently can follow combination CVR trends \alex{It's a contradiction. It cannot be stateless and follow trends. In fact, there is a state - the model which is used to generate the combinations}\oren{done}. Therefore, we believe the proposed CVR based DCO solution deserves its own separate reporting.

Since CVR prediction is a major part of our solution, we mention a few works of the available vast literature of ad CVR prediction. An \textsc{Offset}\xspace based conversion-given-click prediction is described in \cite{aharon2015serving}. For more recent works on conversion rate prediction via post-click event modeling, see \cite{su2020attention}\cite{wen2020entire} and references therein.

\section{Problem statement}\label{sec:problem statement}
Assume we have a DCO ad with $M$ attributes and $m_i$ assets for the $i$th attribute $i=1\ldots M$. Therefore, there are $N=\Pi_{i=1}^M m_i$ combinations (or virtual native ad). The goal is to use the additional \textit{degree-of-freedom} of having $N$ combinations, to maximize the DCO ad CVR and increase revenue.

It is noted that by maximizing the CVR (as opposed to CTR maximization \cite{koren2020dynamic}) we cannot guarantee a direct increase in revenue since according to the oCPC bidding strategy, advertisers still pay for clicks. However, since our main click and conversion-given-click prediction models are accurate, increasing the DCO ad CVR will cause the system to increase its bid (see oCPC explanation in Section \ref{sec:serving}) pushing DCO ads to win more auctions and spend more (if budget allows) \alex{It can also decrease bids, right? I'm not sure the statement is correct}\oren{if budget permits optimized bid will increase. The bottom line here is that immediate revenue lift is not guaranteed but long term lift is expected}. Moreover, for the mCPC bidding strategy DCO ads (see mCPC explanation in Section \ref{sec:serving}), the advertisers are expected to increase their budgets since they pay less for their conversions if CVR increases. Therefore, in the long-run, CVR improvement is expected to increase revenue.

\subsection{Requirements, constraints and conditions}\label{sec:requirements}
Since the conversion DCO system should be incorporated into the existing Gemini native architecture, there are several system requirements and inherent constraints that considerably affect the nature of the selected solution.

Starting with the system requirements: (a) \textit{Model sizes} --
\textsc{Offset}\xspace click and conversion model sizes that are consumed by the Serving system cannot be considerably increased. Since most of the models consists of ad vectors, considering different combinations of DCO ads as new ads is not scalable; (b) \textit{Query-per-second (QPS) and service-level-agreement (SLA)} -- QPS cannot be considerably decreased (or alternatively SLA cannot be considerably increased). This means that the Serving system cannot make a query per DCO combination (e.g., using the combination number as an ad feature); (c) \textit{Serving complexity} -- computation complexity of the Serving system should be restricted to simple processing per ad (e.g., sigmoid function of the ads' score according to \eqref{eq:event probability prediction} and \eqref{eq: score}); and (d) for legacy reasons\footnote{Dictated by Gemini \textit{carousel asset optimization} \cite{aharon2019carousel}, and CTR based DCO \cite{koren2020dynamic} products.} the interface to the Serving system is limited to combination distributions per DCO ad and certain traffic segments (see Section \ref{sec: defining traffic segments}). The Serving system then uses the combination distributions in a predefined manner to draw the DCO combination for rendering the DCO ad.

It is worth emphasizing that we are \textbf{extremely} sensitive to Serving system resources surge, as traffic is spread globally among many servers for maintaining harsh SLA requirements. However, we are almost \textbf{agnostic} to backend resources used to train our models, which is carried out over a couple of production grids.

Turning to the inherent constraints: (a) \textit{Delay} --
the time it takes for the system to update the model, index it, serve users, and log the users' actions\footnote{Applies to actions such as swipes and clicks but not to conversions.}. In the current production environment the average delay is more than an hour;
(b) \textit{Incremental mode} -- data processing is currently done in batches of a few minutes to a few hours worth of data (e.g., 15 minutes and 4 hours for click and conversion models, respectively). This means that approaches that require policy changes after each action cannot be considered.

Finally, we describe the setting conditions that challenge the algorithm selection and system design: (a) \textit{Data sparsity} -- CVR is usually two orders of magnitude lower than CTR, hence, event counting based approach as the one used for optimizing click DCO ads \cite{koren2020dynamic} is problematic; (b) \textit{Reporting delays} -- conversion events may be reported up to 30 days after they occur. This is a real challenge that is hard to deal with since we do not want to wait so long before updating our models; (c) \textit{Varying impression rate} -- impression rates of DCO ads (as any other ad) are dictated mainly by their popularity, bid, and budget availability. These factors determine the ad delivery rate (the rate it wins auctions and gets impressions) and may change significantly over time; (d) \textit{Varying combination CVRs} -- CVR in general (as CTR) is known to vary over time (day over night, week days over weekends, etc.). In addition, popularity trends and ad fatigue may also cause ad (and combinations within DCO ads) CVR variations. The last two setting conditions make generating comparable combination CVRs measurements, a challenging task

\section{Our approach}\label{sec:our approach}
\subsection{Overview}
Due to the constraints dictated by the Gemini native architecture (see Section \ref{sec:requirements}) we chose to separate the processes of ad auction and DCO optimization, and adhere to a \textit{post-auction} approach. Hence, for each incoming impression we let the Serving system use \textsc{Offset}\xspace models to select the winning ad (see Section \ref{sec:serving}) and only then try to ``optimize'' the best combination for the current impression in case a DCO ad wins the auction. Since CVR is much lower than CTR, we use predictions instead of event counting to generate combination distributions. In particular, we train an \textit{auxiliary combination CVR prediction model} (see Section \ref{sec:auxiliary model}) and turn its predictions into combination distributions per DCO ad and certain \textit{traffic segments}, defined by a crossing contextual user features (see Section \ref{sec: defining traffic segments})\alex{, defined by a combination of user and contextual features, }\oren{done} at the end of every training period. Then, the distributions (packed in a DCO model file) are periodically sent to the Serving system, allowing it to draw the selected combination according to the relevant distribution before rendering the winning DCO ad. It is noted that we maximize CVR (probability of a conversion given an impression event) and not the probability of a conversion given a click event. We do that since after the auction, when the combination distributions are used by Serving, the impression has not happened yet and therefore CVR is the correct term to maximize. \alex{We'd like to emphasize that the novelty of our approach stems not from a novel conversion prediction model, but from our utilization of an existing well-proven conversion prediction framework to facilitate our light-weight post-auction approach to overcome extreme data sparsity issues (this is to address a reviewer's comment)}\oren{done}

We would like to emphasize that the novelty of our approach stems not from a novel conversion prediction model, but from our utilization of an existing well-proven conversion prediction framework to facilitate our light-weight post-auction approach to overcome challenges such as extreme data sparsity condition (see Section \ref{sec:requirements}).

\subsection{Traffic segments definition}\label{sec: defining traffic segments}
We define the traffic segments by configurable segment keys that relate to categorical unweighted user features
(e.g., age, gender, and device). Assuming we use gender (i.e., male, female, and unknown) and device (i.e., mobile, desktop, tablet, and unknown) as segment keys, then for each DCO ad we have $3\times 4=12$ segments (e.g., female $\times$ desktop).
For each DCO logged impression and conversion events, the segment keys are extracted and used as user features for training the auxiliary prediction model (see Section \ref{sec:auxiliary model}). During serving time, in case a DCO ad wins the auction, the segment keys are extracted from the incoming user's features, and used to locate the corresponding traffic segment distribution for drawing the combination and rendering the DCO ad (see Section \ref{sec:rendering DCO ads in serving time}).

\subsection{Rendering DCO ads during serving time}\label{sec:rendering DCO ads in serving time}
Assume the auction winning ad is some eligible DCO ad $A$ with $N$ combinations $\mathcal{C}_A=\{C_n\}_{n=1}^N$. The Serving system extracts the segment keys of the incoming user's features to determine the traffic segment $S$ and locates the corresponding combination distribution $Q_{A,S}=\{Q_{C_n}\}_{n=1}^N$ in the DCO model file. Then, it draws one combination according to $Q_{A,S}$ before rendering $A$.

\subsection{\hspace{-0.1cm}Auxiliary combination CVR prediction model}\label{sec:auxiliary model}
To predict DCO ads combinations' CVRs, we train an \textsc{Offset}\xspace based event prediction model (see Section \ref{sec: offset}). \alex{Consider, instead of the prev. sentence: "To generate the combination distributions, we train an \textsc{Offset}\xspace based model model for predicting pCVR, and transform its predictions to distributions."}\oren{something in the middle}.
\begin{itemize}
\item Impressions are used as negative events while conversions (both post-click and post-view) are used as positive events.
\item Since conversions are reported with long delays (up to 30 days after the actual view or click), we do not join the conversion and the impression that entailed it before training the model. Hence, each positive event is also trained as a negative event. Therefore, CVRs are slightly under-predicted and their predictions should be bias-corrected before turned into actual distributions (see Section \ref{sec: prediction correction}).
\item To reduce the backend resources required to train the auxiliary prediction model we downsample the negative events (i.e., impressions) before training. Hence, predictions should be bias-corrected \alex{consider ``corrected'' $\to$ bias-corrected}\oren{done} also to account for this downsampling operation \alex{consider: for this downsampling operation $\to$ to account for this downsampling}\oren{done} before turned into actual distributions (see Section \ref{sec: prediction correction}).
\end{itemize}
It is noted that unlike the main click and conversion-given-click models that are consumed (or indexed) by the Serving system for ranking the ads (see Section \ref{sec:serving}), the auxiliary model is used only for generating DCO combination distributions. In addition, we train the auxiliary model over \textit{all} conversion ad traffic and not only DCO ad traffic for triggering collaborative filtering patterns that help in ``filling'' the gaps due to conversion data sparsity.

Next, we consider the ad and user features used to train the auxiliary model, and tie the latter to the traffic segments defined in Section \ref{sec: defining traffic segments}.

\subsubsection{User features}\label{sec:user features}
Since the auxiliary model is used to predict the DCO ads combinations' CVRs for each traffic segment of interest (see Section \ref{sec: defining traffic segments}), we use the corresponding segment keys as user features. For example, if segment traffics are defined by \textit{gender} and \textit{device}, then we use those as user features.

It is noted that adding more user features may provide more accurate CVR predictions. However, there is little benefit in doing that in our setting, since we are interested in the CVR predictions of certain traffic segments that are defined by certain segment keys. Therefore, even if we add more features, we would have to average the predictions over the extra features' values to get the predictions over the traffic segments of interest, and actually ``lose'' the improved accuracy. It is concluded that in our setting it is sufficient to set the traffic segment keys as user features.

\subsubsection{Ad features}\label{sec:ad features}
We may use all ``standard'' ad features available for Gemini native models, such as \textit{ad ID\ \footnote{ID is a unique string that is generated by the system and assigned to every ad related entity, such as the ad itself, its assets, its campaign, and its advertiser.}}, \textit{campaign ID},  \textit{advertiser ID}, and \textit{ad category} (a multi-value feature with few dozens of values such as \textit{sports} and \textit{electronics} - see Section \ref{sec:offset wmv feature}). To predict the CVR for each combination of each DCO ad, we require a special ad feature that ties together the assets that form each combination.

\paragraph{Combination assets ad feature}\label{sec:assets feature}
Each event (impression or conversion) which involves a DCO ad, includes information regarding the actual assets used to render the ad. For example, assuming a certain DCO ad was impressed using a certain combination with description ID - De123, image ID - Im456, and title ID - Ti789, then a weighed multi-value feature \{(De123,1),(Im456,1),(Ti789,1)\} with unit weights (see Section \ref{sec:offset wmv feature}) is generated and used to train the auxiliary model. It is noted that assets multi-value representation is used instead of representing the combination number as an ad feature, to create dependencies among the DCO ad's combinations that share assets (e.g., share the same title and description but include different images). This in turn triggers collaborative patterns that help ``filling'' the gaps caused by data sparsity. Finally, for all events that involve non-DCO ads, the feature is assigned with ``NONDCO'' value and a unit weight, i.e.,  \{(NONDCO,1)\}.

\subsection{Turning predictions into distributions}\label{sec: predictions to distributions}
At the end of every training period of the auxiliary prediction model (e.g., 4 hours), we scan it for DCO ads. Then, for each DCO ad we use the auxiliary model to calculate the DCO ad combinations' CVR predictions for each traffic segment. To do that we gather the particular DCO ad features values (see Section \ref{sec:ad features}), query the model, and construct all the DCO ad's combinations LF vectors (see Section \ref{sec: offset}). Since the system is designed such that traffic key segments coincides with the auxiliary prediction model user features, to get the user LF vector we query the model using the segment keys values that define the segment (e.g., female and desktop for gender and device segment keys), and construct the user vector (see Section \ref{sec: offset}). After we have the user and combinations LF vectors, we extract the model bias and use \eqref{eq:event probability prediction} to calculate the combinations' CVR predictions. Next, the predictions are bias-corrected to compensate for the \textit{non-join} and impression \textit{downsampling} operations (see Section \ref{sec: prediction correction}). Finally, we use \textit{SoftMax}\footnote{See https://en.wikipedia.org/wiki/Softmax\_function .} to translate the true predictions into distributions and add a uniform component to produce the final distribution. All distributions are gathered into a DCO distributions file (or table) and sent to the Serving system to be used during serving time (see Section \ref{sec:rendering DCO ads in serving time}).

See Algorithm \ref{algo:P2D} for a formal description of the \textit{predictions to distributions} (P2D) algorithm.
\input{p2dAlgo}

\Comment{The following actions are omitted from Algorithm~\ref{algo:P2D} for clarity and are mentioned here instead.
\begin{itemize}
    \item \textit{Handling ads with fewer conversions} - DCO ads with less than a predefined number of conversions $N_c$ (a system parameter, e.g., $N_c=1$) are assigned with uniform distributions for all traffic segments.
    \item \textit{Handling traffic segments with low maximal prediction $P_M$} - traffic segments with $P_M<P_{min}$ (a system parameter, e.g., $P_{min} = 1e-9$) are assigned with uniform distribution.
    \item \textit{Omitting inactive DCO ads} - to reduce the auxiliary prediction model size and training time, we omit ads (DCO and non-DCO) that were inactive (i.e., had neither impressions nor conversions) for a predefined time period (e.g., a week) and treat them as new ads if they reappear in traffic.
\end{itemize}}

\subsubsection{Why \textit{SoftMax}?}\label{sec:why softMax}
Our choice of using \textit{SoftMax} facilitates a controlled mechanism to provide an explore-exploit trade-off and lets the system follow trends while presenting ``better'' combinations more frequently. However, there are other solutions that hold similar characteristics, e.g., we could use the predictions and select the closest distribution on the unit simplex, or just normalize each prediction by the sum of predictions. In the sequel we provide a theoretical justification to our choice.

Assuming a conversion DCO ad $A$ has won the auction for an impression that belongs to a traffic segment $u$, we would like to maximize the chance for getting a conversion by exploiting the additional degree-of-freedom and select the combination that has the best chance to entail a conversion. Accordingly, we would like to maximize
\vspace{-0.18cm}
\begin{multline}\label{eq:optimization}
Pr(\mathrm{conversion}|u,A)=\\ \sum_{C\in \mathcal{C}_A}\ Pr(\mathrm{conversion}|C\  \mathrm{selected},u,A)Pr(C\  \mathrm{selected}|u,A)\ ,
\end{multline}
where the probability $Pr(\mathrm{conversion}|C\ \mathrm{selected},u,A)$ can be approximated by the auxiliary model true CVR prediction $P_C$ (see step 13 of Algorithm \ref{algo:P2D}), and the probability $\Pr(C\ \mathrm{selected}|u,A)$ is the traffic segment combination probability $Q_C$ we need to set for DCO ad $A$. A trivial solution to maximize \eqref{eq:optimization} is to assign all the probability mass to the combination that has the highest CVR prediction. However, presenting the same combination repeatedly to all users falling into the traffic segment (every time DCO ad $A$ wins the auctions) is undesired. This is since it might enhance the ad fatigue phenomenon and also prevent the system from exploring and following trends causing other combinations to be attractive in terms of CVR. Therefore, we would like to ``pull'' the distribution towards a uniform distribution in a controlled manner and create a trade-off between exploration and exploitation. A natural way to do the latter is to add an \textit{Entropy} regularization term into the optimization problem of \eqref{eq:optimization} and rewrite it as
\vspace{-0.2cm}
\begin{multline}\label{eq:optimization with regularization}
Q^* = \operatornamewithlimits{argmax}_Q \left\{ \sum_{C\in\mathcal{C}_A}P_C\ Q_C - \alpha \sum_{C\in\mathcal{C}_A} Q_C \ln(Q_C)\ :\ \right.\\ \left. \sum_{C\in\mathcal{C}_A} Q_C = 1,\ Q_C\geq 0\ \forall{C\in\mathcal{C}_A}, \right\}\ ,
\end{multline}
where the two constraints are added to ensure $Q=\{Q_C\}$ is a distribution function (in a vector representation), and $\alpha>0$ is the regularization parameter. Next, we apply a well known result in convex analysis \cite[Section 4.4.10]{doi:10.1137/1.9781611974997} that the \textit{convex conjugate} of $\phi(z) = \sum_{i=1}^d z_i \ln(z_i)$ defined over the \textit{unit simplex} is\footnote{Note that the result of \eqref{eq:convex conjugate} is a scalar while the result of \eqref{eq:optimization with regularization} is a vector.}
\begin{equation}\label{eq:convex conjugate}
\phi^*(x) = \max_z \left\{ x^T z - \phi(z) : \sum_{i=1}^d z_i = 1,\ z \geq 0 \right\} = \ln \left( \sum_{i=1}^d e^{x_i} \right)\ ,
\end{equation}
and according to the \textit{conjugate-subgradient} theorem \cite[Theorem 4.20]{doi:10.1137/1.9781611974997}, we have
\[
Q^* = \left.\left(\nabla \phi^*(x)\right)\right |_{x=P/\alpha} = \frac{\{e^{P_C/\alpha}\}}{\sum_{C\in\mathcal{C}_A}e^{P_C/\alpha}} = SoftMax(P/\alpha)\ .
\]
where $P=\{P_C\}$ are the combinations' predictions (in a vector presentation). Setting $\alpha=P_M/\beta$ \alex{this is something I don't understand - I know it works in practice, but why actually the entropy regularization parameter $\alpha$ should depend on the maximum $P_C$?}\oren{it's mathematically correct and meant to support an explainable relative presentation of the assigned probabilities, i.e., the numerator is 1 for $P_M$ and $<1$ for all the rest} where $P_M=\max \{P_C\}$, and recalling that \textit{SoftMax} is invariant under shift, so
\[
SoftMax\left(\beta P/P_M\right)=SoftMax\left(-\beta\left(1-P/P_M\right)\right)\ ,
\]
we get the \textit{SoftMax} combination distribution component of  Algorithm \ref{algo:P2D} (step 16). The final presentation of the \textit{SoftMax} argument is preferred to allow relative interpretation, e.g., setting $\beta=6.93$ means that a $10\%$ difference between the ``best'' combination and its runner-up entails approximately twice the probability. It is noted that the uniform component $\lambda$ is added to ensure a minimal amount of exploration. This constraint may be also added to the optimization problem, however, for clarity matters it is added in a straightforward sub-optimal manner.

\subsubsection{Prediction correction}\label{sec: prediction correction}
As mentioned earlier we downsample the impressions (e.g., factor $r_{ds}=100$) to reduce the auxiliary model training resources. In addition, we also do not ``join'' conversions with their impressions to avoid long training delays (conversions may be reported up to 30 days after they occur). Therefore, to get the ``correct'' CVR prediction we have to compensate for these two actions. The following is a rough approximation that is accurate only on average. Assuming we have $V$ conversions and $S$ skips (i.e., impressions without conversions) for a certain ad. Then, the average ``raw'' CVR with \textit{downsampling} of $r_{ds}$ and \textit{non-join} operation may be written as
\[
\mathrm{CVR'}=\frac{V}{V+\frac{V+S}{r_{ds}}}\ .
\]
Since the correct average $CVR=V/(V+S)$,
it is easily verified that
\begin{equation}
    \mathrm{CVR}=\min\left\{1,\frac{\mathrm{CVR'}}{r_{ds}(1-\mathrm{CVR'})}\right\}\ ,
\end{equation}
where the minimum operation is required to keep $\mathrm{CVR}<1$ for very high average ``raw'' conversion rates $CVR'>r_{ds}/(1+r_{ds})$ which is $\sim 0.99$ for $r_{ds}=100$.

\subsection{Benefits of our approach}\label{sec: benefits of our approach}
\begin{itemize}
    \item The post-auction approach is selected for its reduced system complexity; As it does not increase the main click and conversion model sizes (consumed periodically by the Serving system and used in serving time), nor does it reduce the serving QPS (see Section~\ref{sec:requirements}). This is since the combination selection is done after the auction and no further index queries are required to rank the combinations. Instead, only a simple draw based selection, done according to the combination distribution, is applied.
    \item Unlike the CTR based DCO solution (reported in \cite{koren2020dynamic}), which uses event (i.e., clicks and skips) counting, the current solution uses combination CVR prediction provided by an auxiliary CF model capable of revealing collaborative conversion patterns that help in ``filling'' the gaps caused by data sparsity.
    \item The solution is able to follow and discover trends that affect the attractiveness of different combinations throughout the ad's lifespan. In addition, it can combat the ad fatigue effect by reducing the probabilities of combinations the users are weary of, and increasing the probabilities of other less exploited combinations.
    \item Unlike the CTR based DCO successive elimination solution (reported in \cite{koren2020dynamic}), the current solution does not eliminates combinations and therefore may be easily trained on traffic of other candidate models. This makes the ramp-up process of such a model rather easy in comparison to that of the CTR based model of \cite{koren2020dynamic}.

\end{itemize}
We do not claim that our solution is optimal, however, as we shell demonstrate in the sequel, we do manage to utilize the additional degree-of-freedom and considerably increase the DCO ads' CVRs.

\Comment{
\subsection{Auxiliary prediction model offline evaluation}
\subsubsection{Settings}
\subsubsection{Metric}
\subsubsection{Baselines}
\subsubsection{Results}
}

\section{Performance Evaluation}\label{sec:evaluation}
In this section we report the online performance of the conversion based DCO system. We describe the setting, define the performance metrics and baselines, present the results and discuss them.

It is noted that since proprietary data is used for evaluating our solution, it is evident that others can not reproduce our results. This caveat is common in papers describing commercial systems and should not undermine the overall contribution of the work. Moreover, due to commercial confidentially matters we report performance lifts only (e.g., CVR lift). However, this should not affect the trends and observations we present.

\paragraph{\textbf{Settings}}
After a phase of offline evaluation used to select the auxiliary prediction model features and parameters, the conversion based DCO system was launched for an alpha online test phase, optimizing internal ads. The online tests demonstrated the potential of the proposed solution and were also used for tuning various system parameters. As a result, we trained the auxiliary model with all available ad features (see Section \ref{sec:ad features}). As user features we use \textit{gender} and \textit{device} that are also used as traffic segment keys (a total of $3\times 4=12$ segments for each DCO ad). In addition, we set the user features independent and overlapping LF vector sizes to $s=o=12$, so the final ad and user LF vector sizes are $D=36$ (see Section \ref{sec: offset}). It is noted that other auxiliary model training parameters such as OGD step-size, AdaGrad parameters, and regularization parameter, are tuned automatically by \textsc{Offset}\xspace built-in online hyper parameter tuning mechanism (see Section \ref{sec: offset} and \cite{aharon2017adaptive}). For the P2D algorithm (see Algorithm \ref{algo:P2D}), we set the \textit{SoftMax} factor $\beta=13.86$ (so a $10$\% predicted CVR difference between the ``best'' and runner-up combinations results in a probability ratio of $4$), and the uniform component $\lambda$ times the number of combinations to $0.1$ (i.e., 10\% of the probability mass). The aforementioned parameter set demonstrated the best CVR lifts we experienced in the limited grid search we performed during our online alpha test phase.

After demonstrating good online performance, the system was pushed to production (serving $90\%$ of all traffic) for a beta test phase, done with a few selected advertisers, and included several dozens of conversion DCO ads with a total daily average of over million impressions. Although the beta experiments were coordinated, the advertisers controlled all aspects of their conversion DCO ads such as ad content, daily budget, and target CPA (tCPA).

\Comment{
\paragraph{\textbf{Baseline}}
At any stage we compared the conversion based DCO bucket metrics to those of a \textbf{control bucket} operating with no conversion based DCO system, and rendering DCO ads using combinations that are chosen uniformly at random. The uniform DCO bucket was deployed on the remaining $10\%$ of the traffic. We note that since we deal with a real Web scale system and not an offline experimental system, implementing and deploying other more sophisticated baselines is problematic. Lastly, it is worth mentioning that using our CTR based DCO system of \cite{koren2020dynamic} as baseline is not informative since the two systems optimize different metrics.}

\paragraph{\textbf{Baselines}}
At any stage we compared the conversion based DCO bucket metrics to those of two \textbf{control buckets} each serving $5\%$ of the traffic. The \textbf{Uniform bucket} was operating with no conversion based DCO system, and rendering DCO ads using combinations that are chosen uniformly at random. The \textbf{CTR bucket} was operating the CTR based DCO system of \cite{koren2020dynamic} which is aimed at maximizing the DCO combinations' CTR.
We note that since we deal with a real Web scale system and not an offline experimental system, implementing and deploying other more sophisticated baselines is problematic.

\Comment{
\begin{table
\begin{center}
    \begin{tabular}{|c|c|c|c|c|c|}
    \hline
    Test period & CVR lift & CTR lift & Delivery lift & CPM lift & CPA lift\\
    \hline\hline
    28 days & 53.5\% & 1.69\% & 0.01\% & 0.44\%  & -34.6\% \\
    \hline
    \end{tabular}
    \end{center}
    \caption{Conversion based DCO test phase results summary.}\label{table: bucket results}
\end{table}}

\begin{table
\begin{center}
    \begin{tabular}{|c|c|c|c|c|c|}
    \hline
    baseline & CVR lift & CTR lift & Delivery lift & CPM lift & CPA lift\\
    \hline\hline
    Uniform & 53.5\% & 1.69\% & 0.01\% & 0.44\%  & -34.6\% \\
    \hline
    CTR &  38.0\% & -0.92\% & 0.00\% & -3.57\%  & -30.1\% \\
    \hline
    \end{tabular}
    \end{center}
    \caption{Conversion based DCO test phase results summary.}\label{table: bucket results}
    \vspace{-0.5cm}
\end{table}

\paragraph{\textbf{Metrics}}
To evaluate our system we use the following.
\begin{itemize}
    \item CVR - conversion rate, i.e., the number of conversions divided by the number of impressions, where higher is better. Increasing the CVR metric is the goal of the system. We expect positive lifts\footnote{Lifts are calculated by $(M_{DCO}/M_{baseline}-1)\cdot 100$, for any positive metric $M>0$.} if our system works properly.
    \item CTR - click through rate, i.e., the number of clicks divided by the number of impressions, where higher is better. We do not expect a major CTR lift over the Uniform baseline, since our system is not designed to maximize CTR. Moreover, we do expect a CTR drop in comparison to the CTR baseline which is designed to maximize CTR.
    \item Delivery - the number of impressions of the targeted traffic (i.e., DCO traffic). The A/B testing mechanism ensures that different buckets will get their fair share of incoming impressions opportunities. However, the amount of DCO traffic falling into each bucket depends on the results of the auctions which happens only after the incoming impressions were distributed to the different buckets.
    \item CPM - cost per thousand impressions, i.e., the total cost divided by the number of impressions and multiplied by 1000, where higher is better.
    \item CPA  - cost per action, i.e., the total cost divide by the number of conversions, where lower is better. Lower CPA means advertisers spend less for each conversion\alex{on each conversion}\oren{done}.
\end{itemize}
While the CVR, CTR, and Delivery metrics, are easily measured and interpreted, the CPM and CPA metrics, as we shall explain in the sequel, should be handled more carefully.

\paragraph{\textbf{Results and discussion}}\label{sec:results}
The results of the conversion based DCO system beta test are summarised in Table \ref{table: bucket results}. We emphasize that the reported results were measured over the traffic of all DCO ads (oCPC and mCPC) that had at-least one conversion during the evaluation period, while other DCO ads traffic is ignored. Over a period of 28 days (four weeks), we measured an impressive $53.5\%$ and $38.0\%$ average CVR lifts of the conversion based DCO bucket in comparison to the Uniform and CTR buckets, respectively. Since the performance were measured over millions of impressions and the lift is large, the superiority of our approach over the baselines is established with high confidence. In particular, the positive CVR lifts are statistically significant with $p-value < 10^{-14}$. \alex{I think that since the statistical model doesn't take delay into account, i.e. you aren't modeling the hazard function, and just using the #conv / #impr, I think this number isn't very informative. If I were a reviewer, it would negatively impress me. But I don't have that much experience, so it's up to you.}\oren{you are probably correct but the reviewers probably won't notice and p value report is expected}

In addition to the CVR hefty lift we also measured a $1.69\%$ average CTR lift ($p-value=0.016$) over the Uniform baseline and only a $-0.92\%$ CTR drop (negative lift) when compared to the CTR baseline, which means that there is some positive correlation between DCO combinations that entail more conversions and those that entail more clicks. However, it is quite evident that the system is indeed tuned for optimizing CVR. Moreover, the measured CTR drop in comparison to the CTR baseline is expected since the latter is designed to maximize CTR.

Examining the minuscule Delivery lifts of $0.01\%$ and $0.00\%$, it is concluded that the targeted traffic of DCO ads with at least one conversion is almost equal (after proper normalization) between the conversion based DCO and baseline buckets. This is easily explain by recalling that both buckets rank the ads using the same main click and conversion-given-click prediction models which are trained over the entire traffic. Moreover, the DCO combination selection is done after the auction and has no direct influence on the delivery. It is also noted that since normalized delivery is almost equal between the all buckets, CPM lift which is considered next, is actually revenue lift.

The measured CPM lift of $0.44\%$ over the Uniform bucket seems quite disappointing and does not reflects the staggering CVR lift. However, the fact that all buckets use the same main models for ad ranking and therefore have similar deliveries, combined with the modest CTR lift and the fact that all DCO ads pay for clicks, explains the CPM result. Moreover, the baselines actually benefit from the increased CVR of DCO ads caused by the conversion based DCO system. This is since the main ranking models are trained over the entire traffic which is dominated by the conversion based DCO system traffic ($90\%$ of all traffic). The measured $-3.57\%$ CPM significant drop when compared to the CTR baseline may be explained by similar reasons and by the fact that we actually measure also a CTR drop. It is concluded that in the current A/B test setting where the main ranking models are dominated by the conversion based DCO system traffic, we can not measure actual revenue lifts. As a matter of fact, the conversion based DCO system is expected to increase revenue proportionally to the CVR lift.
This is since the expected revenue (ad score) for oCPC ads, which consist of the lion share of the targeted DCO traffic, is $\mathrm{pCTR}\cdot \mathrm{pCONV} \cdot \mathrm{tCPA} \approx \mathrm{pCVR}\cdot \mathrm{tCPA} \approx \mathrm{CVR} \cdot \mathrm{tCPA}$ (see Section \ref{sec:serving}) \alex{where does the approx equality $\mathrm{pCTR}\cdot \mathrm{pCONV} \cdot \mathrm{tCPA} \approx \mathrm{pCVR}\cdot \mathrm{tCPA}$ come from?}\oren{assuming post-click conversions we have (clicks/impressions) x (conversions/clicks) = conversions/impressions = CVR} assuming our main models are accurate.

We measure a huge CPA drop of $-34.6\%$ and $-30.1\%$, which means that advertisers allegedly pay much less for conversions. However, for oCPC ads this is actually an \textit{artifact} stemming from the settings of our A/B testing where all buckets share the same main models used for ad-ranking. When measuring CPA, the baselines actually ``suffer'' from the situation where delivery is determined by the main ranking models, which reflects the surge in the DCO ads' CVRs, but combinations are actually drawn uniformly at random (at the Uniform bucket) or according to their CTR (in the CTR bucket). Hence, in the baselines buckets advertisers wrongfully spend more for less conversions. As a matter of fact, assuming our main models are accurate, no CPA drop is expected for oCPC DCO ads.

We end this section by noting that the evaluation was dominated by a few active DCO ads with relatively high CVRs. Milder CVR lifts are expected for an evaluation done with larger and more typical DCO demand.

\section{Concluding remarks}\label{sec:Concluding remarks}
In this work we presented a simple yet practical post-auction two-stage solution to the conversion optimization DCO problem. During the first stage of serving, a regular Gemini native auction is performed and if a DCO ad wins the auction, the Serving system uses the combination distributions generated by our combination CVR prediction based DCO solution, to draw the rendered combination. Since the solution is based on predictions provided by an auxiliary prediction model, and not on event counting used by the CTR based DCO system of \cite{koren2020dynamic}, it is capable of following trends and mitigate data sparsity issues. After showing good online performance, significantly increasing the DCO ads' CVRs, the conversion DCO product was deployed and has been serving all Gemini native traffic since.
We note that different approaches to the conversion based DCO were considered as well. The simplest approach would have been to consider each combination as a new native ad and add it to the ad inventory. This however, could increase in theory the ad inventory and main click and conversion model sizes by a factor of the average number of combinations (e.g., $27$ assuming $3$ attributes with $3$ assets each). Another approach considered is treating the different combinations as ad features and incorporating those into \textsc{Offset}\xspace models. This approach would reduce the QPR of the Serving system since we need to calculate pCTRs and pCONVs for each combination per DCO ad before conducting the auction, thereby increasing the SLA by a factor of the average number of combinations. The proposed solution, requires minor modification of the Serving system and is not claimed to be optimal, does not increase \textsc{Offset}\xspace click and conversion model sizes and only slightly impacts the Serving system SLA. Moreover, it is demonstrated to use the additional degree-of-freedom provided by having multiple assets per attribute to increase CVR considerably over the baselines.

Future work may include improving the auxiliary prediction model accuracy, experimenting with additional user features and refining traffic segmentation. Another direction, is experimenting with a CTR prediction based DCO model. Once proven, we plan to combine both solutions and provide a unified CTR/CVR prediction based DCO solution to advertisers.

\balance
\bibliographystyle{plain}

Information Retrieval (cs.IR); Machine Learning (cs.LG)


A large C+N+O abundance spread in giant stars of the globular cluster NGC 1851

Abstract: Abundances of C, N, and O are determined in four bright red giants that span the known abundance range for light (Na and Al) and s-process (Zr and La) elements in the globular cluster NGC 1851. The abundance sum C+N+O exhibits a range of 0.6 dex, a factor of 4, in contrast to other clusters in which no significant C+N+O spread is found. Such an abundance range offers support for the Cassisi et al. (2008) scenario in which the double subgiant branch populations are coeval but with different mixtures of C+N+O abundances. Further, the Na, Al, Zr, and La abundances are correlated with C+N+O, and therefore, NGC 1851 is the first cluster to provide strong support for the scenario in which AGB stars are responsible for the globular cluster light element abundance variations.

\section{Introduction}
\label{sec:intro}

Globular clusters have long been regarded as simple stellar populations
which may be described by a single age, helium abundance ($Y$),
metallicity ($Z$), and initial mass function \citep{rb86}. For many years
these simple stellar populations have played a prominent role in astronomy
by providing a lower limit to the age of the Universe and for testing the
predictions of stellar evolution and stellar nucleosynthesis (e.g., see review
by \citealt{gratton04}).

However,
a revolution in the field of globular cluster research is underway,
prompted by the recent
discoveries of complex structure on the main
sequence, subgiant branch (SGB), red giant branch (RGB),
and/or horizontal branch
(HB) within some Galactic and extra-Galactic globular clusters
indicating that these simple stellar populations in fact contain
discrete, multiple populations
(e.g., \citealt{bedin04,dantona05,piotto05,sollima05,mackey08}).
At present, these multiple populations can be best explained in terms
of distinct ages and/or compositions, although the sequence of events
leading to their formation remains largely unexplained \citep{renzini08}.
In some cases, extremely
large helium abundances, up to $Y$ = 0.40, are required to explain
the various populations (e.g., \citealt{dantona04,norris04}). While
theoretical models have struggled to account for such high
abundances (e.g., \citealt{karakas06,bekki07}), some successful
efforts are beginning to appear (e.g., \citealt{dercole08,pumo08}).
The intrigue surrounding the multiple populations in globular clusters has
been enhanced by the speculation that clusters with extended HBs,
generally the most massive clusters,
are the remnants of dwarf galaxies from which the halo of the Galaxy was
built \citep{lee07}.

The globular cluster NGC 1851 has a bimodal HB and
displays
multiple SGBs \citep[hereafter M08]{milone08},
but no evidence for
multiple main sequences.
Based on the width of the main sequence and
RGB, M08 suggested that the maximum possible
abundance variation is $\Delta$[Fe/H] = 0.1 dex or $\Delta Y$ = 0.026.
From an analysis of eight bright RGB stars using high resolution
spectroscopy, \citet[hereafter YG08]{yg08}
showed that the dispersion in [Fe/H] was less than 0.1 dex,
and that NGC 1851 displays the
usual star-to-star abundance variations for the light elements O, Na,
and Al; the exact mechanism for these
abundance patterns, seen in all globular clusters,
remains unknown (e.g., \citealt{gratton04}).
However, YG08 also showed that NGC 1851 harbors a star-to-star abundance
variation for the $s$-process elements Zr and La and that the abundances
of these elements were correlated with the light element abundances.
Furthermore, within the small sample
there was a hint that the abundances of the $s$-process elements was bimodal,
which suggests that the RGB may consist of two populations with
distinct chemical compositions.
Therefore, as well as showing unusual photometric properties, NGC 1851
also displays peculiar chemical abundance patterns not seen in other clusters.

Several explanations for the
double SGB in NGC 1851 have been offered.
If age is the sole parameter,
M08 suggested a 1 Gyr difference
with the fainter subgiant branch (fSGB) population
being older.
M08 also considered a combination of increasing [Fe/H] by 0.2 dex and
helium from $Y$ = 0.247 to 0.30,
though this
possibility was excluded by the magnitudes of the blue HB stars.
An alternative explanation was proposed by \citet{cassisi08}
in which the two SGB populations
are coeval, but with the fSGB population having a
total C+N+O abundance increased by a factor of 2.
In this scenario, the two
populations may have comparable He abundance
(at most $\Delta Y$ = 0.032), in contrast to
the large He variations inferred in
other clusters which display multiple populations. We have obtained
spectra with the goal of measuring the abundance sum C+N+O to test the
\citet{cassisi08} scenario.

\section{Observations and abundance analysis}
\label{sec:data}

We selected four bright RGB
stars from YG08, two Na-, Al-, Zr-, and La-normal targets
as well as two Na-, Al-, Zr-, and La-rich objects.
In Figure \ref{fig:cmd} we show the locations of our targets
in a color-magnitude diagram.
If the double SGB populations are due to distinct
chemical compositions, and if these chemical compositions are also
present on the RGB,
then our small, but carefully selected, sample should allow us to
identify any C+N+O abundance variation, if present.

\begin{figure}
\epsscale{1.1}
\plotone{f1.ps}
\caption{The color-magnitude diagram for NGC 1851 using the
\citet{grundahl99} photometry. The large plus symbols represent
the bright giants in this study. The filled squares represent the
additional four stars in YG08. \label{fig:cmd}}
\end{figure}

The stars were
observed with the high resolution spectrograph MIKE \citep{mike} on the
Magellan Clay Telescope on 2007 December 23. We used the 0.7\arcsec\ slit
which provided a spectral resolution of $R$ $\simeq$ 37,000 in the blue CCD
(3350-5000\AA) and $R$ $\simeq$ 30,000 in the red CCD (4900-9500\AA) as
measured from the ThAr lines. Our total
exposure times per star ranged from 40 to 80 minutes resulting in
signal-to-noise ratios (S/N) per four pixel resolution element of
70-100 at 4320\AA\ and 210-260 at 6600\AA. The data were reduced using the
same procedure described by \citet{yong06}.

Although the new data are of lower spectral resolution
($R$ $\simeq$ 30,000 vs.\ $R$ $\simeq$ 55,000),
the S/N and wavelength coverage are superior to the
spectra analyzed by YG08. Using these new spectra, we rederived the
stellar parameters using the same tools and techniques described in YG08
and found that
the revised stellar parameters are in good agreement with the
stellar parameters previously obtained. The new dispersion
in Fe abundances for these four stars is $\sigma$[Fe/H] = 0.03 dex
compared with 0.11 dex for the YG08 analysis. This decrease is
due to the increased wavelength coverage and S/N which allows us to more
accurately measure the equivalent widths of a larger number of Fe lines.

The abundances of C and N were derived by comparing
observed spectra with synthetic spectra generated
using the LTE line analysis and spectrum synthesis program MOOG \citep{moog}.
For C, we analyzed lines from
the $(0,0)$ and $(1,1)$ bands of the $A-X$ electronic
transition of the CH molecule near 4310\AA\ using the line list from
\citet{ch}. For N, we analyzed lines from
the $(2,0)$ band of the $A-X$ electronic
transition of the CN molecule near 8000\AA\ using the line list from
\citet{reddy02} which is based on the \citet{cn} list.
In both cases, we adjusted the abundances of C or N until the
synthetic spectra matched the observed spectra
(see Figures \ref{fig:cnospec1} and \ref{fig:cnospec2}).
We rederived the O abundances again from the 6300\AA\ [OI] line
using the new spectra, the revised C
and N abundances, and the revised stellar parameters.
Since the abundances of C, N, and O are coupled, we
iterated until self-consistent abundances were obtained.
We also updated the abundances for Na, Al, Zr and La
using the new spectra and revised stellar parameters
(a complete abundance analysis of our data will be
presented elsewhere). We present the stellar parameters and abundances
in Table \ref{tab:abund}.

\begin{figure}[h!]
\epsscale{1.1}
\plotone{f2.ps}
\caption{Observed spectra (circles) and synthetic spectra (lines) for
C (upper), N (middle), and O (lower) for Star 395.
The synthetic spectra show the best fit (thick black line) and unsatisfactory
fits (thin red and green lines) $A$(C) $\pm$ 0.30 dex, $A$(N) $\pm$ 0.20 dex,
and $A$(O) $\pm$ 0.20 dex. \label{fig:cnospec1}}
\end{figure}

\begin{figure}[h!]
\epsscale{1.1}
\plotone{f3.ps}
\caption{Same as Figure \ref{fig:cnospec1} but for Star 95.
\label{fig:cnospec2}}
\end{figure}

\begin{deluxetable*}{cccccccccccccc}
\tablecolumns{14}
\tablewidth{0pc}
\tablecaption{Stellar parameters and abundances\label{tab:abund}}
\tablehead{
\colhead{Star\tablenotemark{a}} &
\colhead{$V$} &
\colhead{\teff~(K)} &
\colhead{$\log g$ (cgs)} &
\colhead{$v_t$ (km s$^{-1}$)} &
\colhead{$A$(Fe)} &
\colhead{$A$(C)} &
\colhead{$A$(N)} &
\colhead{$A$(O)} &
\colhead{$A$(CNO)} &
\colhead{$A$(Na)} &
\colhead{$A$(Al)} &
\colhead{$A$(Zr)} &
\colhead{$A$(La)}
}
\startdata
003 & 13.68 & 4125 & 0.7 & 1.86 & 6.28 & 6.46 & 7.60 & 7.75 & 8.00 & 5.25 & 5.63 & 1.51 & 0.16 \\
095 & 13.58 & 4125 & 0.7 & 1.92 & 6.32 & 6.51 & 7.60 & 7.60 & 7.92 & 5.25 & 5.69 & 1.64 & 0.22 \\
329 & 13.46 & 3925 & 0.0 & 1.92 & 6.25 & 6.81 & 8.00 & 7.88 & 8.26 & 5.50 & 5.79 & 1.87 & 0.42 \\
395 & 13.48 & 4025 & 0.5 & 2.01 & 6.29 & 6.41 & 8.45 & 7.38 & 8.49 & 5.74 & 6.04 & 1.85 & 0.53
\enddata
\tablenotetext{a}{Star names taken from \citet{stetson81}.}
\end{deluxetable*}

\section{Results }
\label{sec:results}

As mentioned, our sample was selected to span the full range of the
known star-to-star abundance variation of Na, Al, Zr, and La.
Our new analysis confirms the large abundance spread for these
elements as well as the correlations between the
light (Na and Al) and heavy (Zr and La) elements established by
YG08. The range in Na, Al, Zr, and La abundances
far exceeds the measurement uncertainties of 0.06, 0.05, 0.12, and 0.08
dex respectively, though we note that the abundance amplitudes for
these elements are slightly smaller than those reported by YG08.

Adopting a solar abundance
$\log~\epsilon$(Fe) = 7.50 \citep{grevesse98},
we find a mean cluster abundance
of [Fe/H] = $-$1.22 ($\sigma$ = 0.03). The revised cluster Fe
abundance is in good agreement with the YG08 value and previous
estimates in the literature, but the
abundance dispersion is considerably smaller for the
reasons noted above. The abundance range for Fe may be entirely
attributable to observational uncertainties (0.07 dex).

For C and N, we find a large range in abundances. The 0.40 dex
range in $\log~\epsilon$(C) and the 0.85 dex range in $\log~\epsilon$(N)
greatly exceed the measurement uncertainties
(0.04 dex and 0.11 dex respectively).
The abundance amplitude for N, $\Delta\log~\epsilon$(N) = 0.85, is smaller
than that seen in other clusters, e.g., RGB bump stars in NGC 6752 and main
sequence stars in M13 show 2.0 dex amplitudes \citep{briley02,6752nh}.
This may be due to the small sample
and/or advanced evolutionary status where extensive CN-cycling may
reduce the amplitude of the N abundance dispersion.
Curiously, we do not detect any obvious anticorrelation between C and N
as seen in every well studied Galactic globular cluster. We
identify correlations between the abundances of N and Na as well as N
and Al, but the corresponding anticorrelations between C and Na and
C and Al are not obvious. Nevertheless, we find correlations between
the abundances of N and Zr as well as N and La as expected
given the known behavior of N and Al and the
correlations between Al and Zr and Al and La found YG08.

Our first significant result
is that the abundance sum
C+N+O shows a large variation. Within our small but biased sample,
the amplitude of the abundance variation is $\Delta$$A$(C+N+O) = 0.57 dex.
The 0.57 dex spread far exceeds the estimated uncertainty of 0.14 dex.
Bright giants in M4 and NGC 6712,
globular clusters of comparable metallicity to NGC 1851,
show C+N+O abundance amplitudes of 0.22 dex and 0.35 dex respectively
\citep{smith05,6712}. Unevolved stars in NGC 6397,
NGC 6752 and 47 Tuc show C+N+O abundances that
may be regarded as constant within the $\sim$0.3 dex
uncertainties \citep{carretta05}.

Our second main result
is that the abundances of
Na, Al, Zr, and La show a positive correlation with the abundance sum C+N+O
(see Figure \ref{fig:cno}).
In this figure, we overplot results for M4 and NGC 6712 and find that for
the available data, there are no positive correlations between $A$(X) and C+N+O
in these other clusters. Finally, our small sample does not exclude
the possibility that the C+N+O abundance may be bimodal within this
cluster.

\begin{figure}
\epsscale{1.1}
\plotone{f4.ps}
\caption{The abundances of Na, Al, Zr, and La vs.\ the abundance sum C+N+O.
A representative error bar and a linear least squares fit
to the data (including formal slope and error) are shown. Data for
M4 \citep{smith05} and NGC 6712 \citep{6712} are included for comparison.
\label{fig:cno}}
\end{figure}

\section{Discussion}
\label{sec:discussion}

We seek to test the \citet{cassisi08} prediction in which
the double SGB in NGC 1851
is due to two coeval populations whose chemical compositions
for C+N+O differ by a factor of two.
Note that we have not directly observed stars on the SGB.
Instead, we have observed a sample of bright RGB stars in which we have
found a large spread in the C+N+O abundances. If the SGBs
comprise populations with different C+N+O abundances, then we
assume that these populations with distinct compositions are also
present on the main sequence and RGB. Str{\" om}gren photometry
obtained by \citet{grundahl99}
reveals that the bimodality seen on the SGB can also be
found on the RGB. Therefore,
the large spread in C+N+O abundances in RGB stars offers
support, albeit not definitive, for the \citet{cassisi08} scenario.
We tentatively confirm that the double SGB
in NGC 1851 is due to two coeval populations with C+N+O abundances
differing by a factor of two.

Based on the relative numbers of stars, YG08 speculated that the
brighter subgiant branch (bSGB) stars were related to the RGB stars with
``normal'' Zr and La abundances and that these stars populate the
red HB. YG08 also suggested that the fSGB stars
are related to the RGB stars with Zr and La enhancements and that these
stars populate the blue HB.
If the fSGB is older than the bSGB as M08 suggest, then the YG08
speculation would require the Zr-rich,
La-rich stars to be older than the Zr-normal, La-normal stars.
In this scenario, the sequence of events leading to cluster formation
would be complicated and require the AGB (or other) stars to
preferentially pollute the older Zr-rich La-rich population
but without polluting the gas from which the
younger Zr-normal La-normal population formed.
However, if the C+N+O abundance of the $s$-process rich
stars is larger than that of the $s$-process normal ones,
the fSGB may be coeval
to the bSGB, or even a bit younger than the bSGB, as shown in Cassisi
et al. (2008). Due to the observational errors, and to the fact that
the age difference we are looking for does not exceed some 100Myr, it
may be very difficult or impossible to constrain the precise mass
range of the progenitors of the $s$-process rich stars, but a coherent picture
of the chemical evolution history is emerging.

The \citet{cassisi08} scenario
has important implications for other clusters which
display multiple SGBs. \citet{villanova07} studied the
SGB region of $\omega$ Centauri, the most massive globular cluster
which is suspected of harboring up to five populations with discrete
ages and/or compositions. From the large range in the magnitudes
of the SGB populations, \citet{villanova07} inferred a large
range in ages. They found that there was an extremely old, but metal-rich
population which greatly complicates any formation scenario.
Based on our NGC 1851 results, the inferred ages of SGB
stars in $\omega$ Cen requires detailed knowledge of the relative
abundances of C+N+O. We speculate that the old metal-rich population
(Villanova et al.'s ``SGB group D'') has a large excess of
[(C+N+O)/Fe] relative to the more metal-poor populations and therefore a
younger relative age than currently inferred. Such compositions
would be consistent with our understanding of the composition of the ejecta
from AGB stars \citep{karakas07}
and the prominent role of AGB stars in the chemical evolution
of $\omega$ Cen \citep{norris95}.

AGB stars have long been suspected as being responsible, in part or in
whole, for the star-to-star abundance variation for the light elements
(C, N, O, F, Na, Mg, and Al) found in every well studied Galactic
globular cluster, including NGC 1851. However, various theoretical yields
and chemical evolution models involving AGB stars have yet to provide
a satisfactory solution (e.g., \citealt{fenner04}). AGB stars may be
expected to produce a substantial increase in the C+N+O abundance as they
enhance Na and Al and deplete O and Mg. AGB stars should also produce
$s$-process elements, albeit via different processes than for
the O, Na, Mg, and Al nucleosynthesis. If AGB stars are responsible
for the light element abundance variations in globular clusters,
we may expect
a large range in C+N+O abundances along
with correlations between (1) C+N+O and Na,
(2) C+N+O and Al, and (3) C+N+O and $s$-process elements.
On the other hand, these proposed abundance trends are dependent on the
number of third dredge up episodes
and therefore
on the adopted convection efficiency
(e.g., \citealt{ventura05a,renzini08})
and mass loss
(e.g., \citealt{ventura05b}).
Further
uncertainty in the yields comes from the uncertainties in the nuclear
reaction rates
(e.g., \citealt{lugaro04,ventura06,karakas06mg,izzard07}).
Although the brightest AGB stars in the LMC and SMC can become
C-rich, it is
possible that nucleosynthesis in the most massive AGBs is scarcely
affected by the third dredge up, with only a minor increase in the total
C+N+O abundance of the second generation stars
\citep{dantona08}.
Nevertheless, the absence of
the signatures listed above has forced astronomers to
consider
alternative sources such as massive stars
(e.g., \citealt{decressin07}).

The correlations between C+N+O and Na, Al, Zr, and La presented here
offer strong support for the scenario in which AGB stars are responsible
for the light element abundance variations. Given the peculiar photometric
and chemical properties of NGC 1851, it is surprising that this cluster
offers the strongest support for the globular cluster AGB pollution scenario.
It would be ironic if NGC 1851 is the only globular cluster for which
the AGB pollution scenario is applicable. An alternative interpretation
is that the AGB pollution scenario is applicable to all globular clusters.
For NGC 1851, the masses of the polluting AGB stars was lower such that
the products of third dredge-up, increased C+N+O and $s$-process elements,
are clearly evident.

We note that abundance spreads in $A$(CNO) are likely present
in $\omega$ Centauri \citep{norris95}, although they were
concerned with possible problems with the N measurements.
Indeed, NGC 1851 bears a
striking resemblance to $\omega$ Centauri. Both clusters possess
multiple SGBs, large abundance spreads for
$s$-process elements and likely spreads in $A$(CNO),
and a scatter in the Str{\" om}gren $m_1$ index,
traditionally used as a metallicity indicator \citep{grundahl99}.
We reiterate the comments by YG08 that NGC 1851 is a ``bridge'' between
$\omega$ Centauri (variation of all elements) and NGC 6752-like clusters
(constant Fe and C+N+O but large variations of light elements C-Al).

Finally, recent work by \citet{salaris08} also suggests a
relationship between the SGB and HB
populations in NGC 1851. Specifically, the bSGB stars
evolve onto the red HB while the fSGB stars populate
the red and blue HB. If this is correct, we would therefore expect
stars on the red HB to be primarily Zr-normal and La-normal. NGC 1851
may provide critical information on HB morphology
and the second parameter effect and
additional studies of this cluster are of great interest.

\acknowledgments

DY thanks Bertrand Plez for providing the CH line list and
Bacham Reddy for providing the CN line list.
We thank the anonymous referee for helpful comments.
FG acknowledges financial support from the Carlsberg Foundation, the
Danish AsteroSeismology Centre at Aarhus University and support from the
Danish National Research Council to the project: "Stars: Cental engines of
the evolution of the  Universe."
FD acknowledges partial support from  the italian PRIN MIUR 2007
'Multiple stellar populations in globular clusters: census,
characterization and
origin'.
We acknowledge support from the Australian Research Council's
Discovery Projects funding scheme
under grants DP0663562 (JEN) and DP0664105 (AIK).

Solar and Stellar Astrophysics (astro-ph.SR); Astrophysics of Galaxies (astro-ph.GA)