Fatigue-Aware Ad Creative Selection

Abstract: In online display advertising, selecting the most effective ad creative (ad image) for each impression is a crucial task for DSPs (Demand-Side Platforms) to fulfill their goals (click-through rate, number of conversions, revenue, and brand improvement). As widely recognized in the marketing literature, the effect of ad creative changes with the number of repetitive ad exposures. In this study, we propose an efficient and easy-to-implement ad creative selection algorithm that explicitly considers user's psychological status when selecting ad creatives. The proposed system was deployed in a real-world production environment and tested against the baseline algorithms. The results show superiority of the proposed algorithm.

\section{Introduction}
\label{sec:Introduction}
In online display advertising, it is a common practice to prepare multiple ad creatives (a combination of an image and text or a video, see Figure \ref{fig:creative} for example) for each advertising campaign. The DSP (Demand-Side Platform) selects the most effective ad creative from the candidates.

\begin{figure}[htb]
  \centering
  \includegraphics[width=\linewidth]{sample_creatives.png}
  \caption{Variation of ad creatives for a mobile game title. The messages are all different:``Don't miss the special event!'' (left), ``Have you already tried these coupons?'' (center), and ``Waiting for your return'' (right). Users jump to login screen of the game when click on these ads.}
  \label{fig:creative}
  \Description{}
\end{figure}

Since new ad creatives are continuously added every day, every hour in the typical online advertising scene there is little time to test the effectiveness of each ad creative beforehand. The {\it bandit algorithm}\cite{lattimore_bandit_2019} is best suited to such situation. The bandit algorithms give us a way to choose one from multiple choices under uncertainty of the outcome by balancing exploration and exploitation.

The effectiveness of ad creative depends not only on the quality of ad creative but also on the audience. For example, one might prefer an informative ad creative while another like an emotional ad creative. The DSP, therefore, often considers {\it context} as well as the ad creative. The bandit algorithm that learn contexts is called {\it contextual bandit} and have been extensively studied at a practical level \cite{li_contextual-bandit_2010,chapelle_empirical_2011,tang_automatic_2013,bietti_contextual_2018}.

One serious side-effect of the contextual bandit is that the ad selection becomes deterministic once the algorithm has {\it learned} enough volume of data. The basic bandit algorithm, regardless of its implementation (e.g. UCB, Thompson sampling, or $\epsilon$-greedy), assumes the reward of arms are unchanged. The degree of exploration is decreasing with the number of plays and eventually goes to zero. As a result, targeted users keep watching same ad creative that is considered as the best by the algorithm given users' information.

The marketing science literature has been extensively studying the consequences of repetitive ad exposure. A seminal work by Zajonc\cite{zajonc_attitudinal_1968} shows mere repeated exposure can incur positive attitude among audience from the experiment results. On the other hand, many researchers point out there is a threshold above which additional ad exposure has negative effect for users' attitude due to boredom or advertising fatigue. As surveyed in \cite{pechmanAdvertisingRepetitionCritical1988,schmidt_advertising_2015}, {\it Two-factor model} theory is widely supported by the researchers in this domain. This theory claims that repetitive advertising has two phases: in the early stage the attitude toward the product improves with the number of repetition but at some point advertising fatigue dominates the positive effect and the net effect (positive effect minus negative effect) plummets (Figure \ref{fig:two factor}).

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.5\linewidth]{two_factor.png}
  \caption{A graphical image of "Two-factor model" for advertising repetition. Previous researches have found that the repetitive advertising has both positive and negative effects on consumers' attitude toward advertised products and the net effect is described as an inverted U-shaped curve\cite{schmidt_advertising_2015}.}
  \label{fig:two factor}
  \Description{}
\end{figure}

The over-exposure of same ad is especially prevalent in display advertising. The real-time bidding (RTB) enables advertisers to reach out to targeted users whenever user browses web or use apps with advertising slots. The distribution of the counts of each user-ad creative pairs from real RTB data (Figure \ref{fig:adexposures}) shows that 47.5\% of creative-user pairs appear multiple times within 24 hours, which leads that 64.6\% of the users experience multiple exposure to same ad creative.

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.5\linewidth]{adexpousres.png}
  \caption{Number of exposure to same ad creative for each user within 24 hours. $N = 27,750$}
  \label{fig:adexposures}
  \Description{}
\end{figure}

Recently the concept of advertising fatigue has received growing attention from machine learning and data mining researchers\cite{maUserFatigueOnline2016a,guptaFactoringExposureDisplay2012,agarwalSpatiotemporalModelsEstimating2009a,abramsPersonalizedAdDelivery2007a}. In particular, Agarwal et al. (2009)\cite{agarwalSpatiotemporalModelsEstimating2009a} and Ma et al. (2016)\cite{maUserFatigueOnline2016a} proposed a fatigue-aware news recommendation system which evidently improved upon existing algorithm. Similar idea is applied to LinkedIn recommendation system\cite{leeModelingImpressionDiscounting2014}. The DSPs also need to consider the psychological status of the audience when choosing an ad creative. To capture users' perception toward each ad creative, we introduce a quantitative measure of advertising fatigue. When we have only one ad creative in hand, fatigue will be well proxied by the number of ad exposures in the past. However, it is not the case in display advertising. When we have multiple creatives, the past exposure to other creatives should be considered because the creatives more or less resemble to each other and expected to increase fatigue in a similar way. We conjecture the gain in fatigue is correlated with the degree of similarity of the ad creative on user's screen and what he has seen in the past.
To get a more concrete idea, let's assume user's attitude toward ad creative evolves according to Figure \ref{fig:two factor}. Then the magnitude of the improvement of attitude by the additional ad exposure ($=$marginal effect of ad exposure) decreases as user's advertising fatigue increases (Figure \ref{fig:marginaleffect}). Suppose that a user has seen same ad creative repeatedly then the effect of the ad creative is negative due to the high level of fatigue (point A). In this case, we can restore the effect of advertising by using a new ad creative (point B). The new ad creative can be a slight modification of the original or totally different ones. A typical set of creatives is already shown in Figure \ref{fig:creative}. That is, the DSP needs to switch ad creative when user gets high level of fatigue.

\begin{figure}[htb]
  \centering
  \includegraphics[width=0.5\linewidth]{marginaleffect.png}
  \caption{The level of fatigue and effect of additional ad exposure (a graphical image). According to the two-factor model, user who has seen same ad creative many times gets advertising fatigue and the effect of ad exposure is negative (A). But new ad creatives can reduce fatigue level and restore effectiveness (B).}
  \label{fig:marginaleffect}
  \Description{}
\end{figure}

We propose a new algorithm for ad creative selection for DSP that explicitly considers advertising fatigue. The algorithm not only selects the best ad creative based on the context but also estimate the level of fatigue for each user and take it into account.

Since the display advertising environment has strict latency requirements, the DSP server can spare only a few milliseconds for computation since it should deliver thousands of ad creatives to numerous users within unnoticeable duration. To save as much as computation time, we first develop a very simple calculation formula for advertising fatigue which capture the similarity of current ad creative and past ad exposure. Then we propose the implementable algorithm that add fatigue features to the existing reward predictor. To check the feasibility and the performance, we conducted an online experiment on real-world display advertising. The result shows that adding fatigue feature improves CTR (click-through rate, the percentage of being clicked out of impressions) and CVR (conversion rate, the percentage of users that complete desired goal such as log-in) over existing algorithm without any systemic problem. We also conducted post-experiment analysis and find the algorithm indeed reduced the fatigue. Therefore, our experiments show an apparent relationship between fatigue and CTR/CVR.

The rest of the paper is as follows. In the next section, we review existing work in Section \ref{sec:Related Works}. Then we formalize the problem and explain the existing system in Section \ref{sec:Ad Creative Selection}.
Section \ref{sec:Fatigue-Aware Ad Creative Selection} proposes the new algorithm that takes the advertising fatigue into account.
Section \ref{sec:Evaluation} describes our online experiment and the results. Section \ref{sec:Discussion} conducts post-experiment analysis. Finally, Section \ref{sec:Conclusion} concludes.

\section{Related Works}
\label{sec:Related Works}
We review related works in both marketing science and machine learning.

\subsection{User fatigue in Machine Learning Literature}
Agarwal et al. (2009)\cite{agarwalSpatiotemporalModelsEstimating2009a} perform explanatory analysis on user's repeat-exposure to recommended articles using data from {\it Yahoo!} front page. They incorporate decaying factors to their CTR predictor based on the analysis. Ma et al. (2016)\cite{maUserFatigueOnline2016a} also conduct an extensive analysis for user's past exposure on data from {\it Bing} news recommendation engine. They propose to use users' past exposure and reaction to same news item and items from same category as features for the machine learning model. User fatigue is also considered explicitly in \cite{guptaFactoringExposureDisplay2012,abramsPersonalizedAdDelivery2007a}. The present paper extends their researches in a number of ways. First we extends the literature from recommendation to display advertising. Second, we define and calculate item similarity to measure fatigue. Third, we deployed our proposed algorithm in a production environment and proved its efficiency and feasibility. Not explicitly mentioning user fatigue, Lee et al. (2014)\cite{leeModelingImpressionDiscounting2014} improves recommendation algorithms in {\it LinkedIn} using user's past exposure to the recommended items (other members) and the actions to them.

\subsection{Repetitive Ad in Marketing Science}
The relationship between the number of repetitions and the effectiveness of ad have been actively studied in marketing science literature for a long time. Extensive reviews and meta-analysis point out that the optimal number of repeated ad exposures depends many conditions and thus is hard to determine\cite{pechmanAdvertisingRepetitionCritical1988,schmidt_advertising_2015}. In the context of online advertising, an analysis of large scale \textit{natural experiments} finds that  wear-out occurs in a heterogeneous manner\cite{lewis_worn-out_2015}. A lab experiment reveals that ad creatives with high quality is immune to wear-out effects\cite{chen_effects_2016}.

\subsection{Effect of Varied Ad}
The effectiveness of varied ad creative has been examined in marketing science literature \cite{chatterjeeModelingClickstreamImplications2003,unnavaEffectsRepeatingVaried1991a,schumannPredictingEffectivenessDifferent1990}. In particular, Chatterjee et al. (2003)\cite{chatterjeeModelingClickstreamImplications2003} examined effectiveness of (i)repeating same ad creative and (ii)varying ad creative in lab experiments. The point estimate of CTR (click-through rate) of varied ad creative exceeded the same ad creative while their test statistics is not significant due to its small sample size.

\subsection{Bandit Algorithms in Online Ad/Recommendation System}
The bandit algorithms have been already applied to variety of advertising/recommendation system \cite{li_contextual-bandit_2010,chapelle_empirical_2011,tangEnsembleContextualBandits2014}. In particular, an application to ad format selection is closest to ours\cite{tang_automatic_2013}. Our research is also related to the {\it non-stationary bandit} problem, in which the reward is changing with time and the optimal arm is not always the same. Among various algorithms, directly estimating the decaying factor of rewards are proved to be effective\cite{liu_time-decaying_2014,levine_rotting_2017}.
We also consider continuous creation and deletion of arms, which is considered in \cite{chakrabarti_mortal_nodate}. While existing literature focus on relatively simple mechanism of reward changing (e.g. decrease with time or pulls), we introduce a numerical measure of ad fatigue of users and tackle the problem. Several studies explicitly consider user abandoning due to the users' psychological status in contextual bandit\cite{lei_actor-critic_2017,cao_dynamic_2019}. In these studies, profitable interventions are considered to have a negative effect on users psychological status and possibly cause user churn. In this scenario, a good contextual bandit algorithm needs to save the frequency of uncomfortable treatments to keep the users engaged. The proposed algorithms solve a constrained optimization problem to find the best policy \cite{lei_actor-critic_2017} or calculate the optimal sequence of interventions \cite{cao_dynamic_2019} at each time of intervention. However, both studies are limited to the simulation and not applied to real production environment.

\section{Problem Setting and Existing Ad Creative Selection System}
\label{sec:Ad Creative Selection}

In this section we introduce real-world problem setting and explain current deployed system.

\subsection{Ad Creative Selection for Retargeting to Game Users}
In this paper, we focus on a DSP that targets the churned game users who have stopped playing for a while.
But the same idea can be applied to a variety of settings in advertising and recommendation.
The DSP intensively buys ad views of these users through Real-Time Bidding (RTB) and urge them to return to the game. The example of ad creatives has been already shown in Figure \ref{fig:creative}, which promote new features of the game, issuing coupons, or simply persuading them to return. Some of the users click on the ad, jump to the login screen, and login the game (conversion).

To convert users, the DSP needs to select the most effective ad creative for each user. The existing system first predicts CTR for each ad creative based on contextual information and then chooses one with the highest predicted CTR.

Formally, each time $t$, the CTR $r_t$ is predicted based on the context vector $\mathbf{x}_t$ for ad creative $a_t$,
\begin{equation}
\hat{r_t}(a_t) = \sigma\left(\bm{\theta_0} \cdot \mathbf{x}_t + \bm{\theta}(a_t)\cdot \mathbf{x_t}\right),
\label{eq:basic}
\end{equation}
where $\bm{\theta_0}$ is a $d$-dimensional weight vector that is invariant to ad creative, and $\bm{\theta}(a_t)$ is a $d$-dimensional {\it action-specific} weight vector. Both vectors include bias term. $\sigma(\cdot)$ represents the sigmoid (logistic) function. The context vector $\mathbf{x}_t$ includes information from bid request (e.g. device, site/app and SSP), and hour of the day. The algorithm selects the ad creative with the highest expected reward.

One serious problem with the algorithm is that it does not generate good training data. The algorithm chooses same ad creative for same context. However, we need the results for the other ad creatives to train the model. The bandit algorithm is an established remedy to the problem that add randomization to selection algorithms. Following \cite{chapelle_empirical_2011}, the existing system adopts the Thompson sampling. That is, for each time $t$, action-specific weight vector $\bm{\theta}(a_t)$ is sampled from the distribution $\mathcal{N}(\bm{\mu}(a_t), \alpha \bm{\Sigma}(a_t))$ for each candidate ad creative, where scalar $\alpha \in (0,1]$ controls the degree of exploitation. Action-invariant vector $\bm{\theta}_0$ is set to $\bm{\mu}_0$ since it does not matter for selection.

Then the system chooses the best ad creative from candidates set $\mathcal{A}_t$, namely, $a_t^\ast = \mathop{\rm arg~max}\limits\limits_{a_t \in \mathcal{A}_t}\hat{r}_t(a_t)$. The action space $\mathcal{A}_t$ is different across $t$. $\mathcal{A}_t$ is determined based on the characteristics of user and ad-slot by the system. Moreover, the set of available ad creative changes over time. As explained in \cite{chakrabarti_mortal_nodate}, ads are continuously added and deleted from circulation. The algorithm chooses newly-added ad creative with a probability of $1/|A_t|$ for exploration. The whole procedure is described in Algorithm \ref{alg:contextual}.

\begin{algorithm}[htb]
\caption{Baseline Ad Creative Selection Algorithm (Baseline)}
\label{alg:contextual}
\begin{algorithmic}
\REQUIRE $\mathcal{N}(\bm{\mu}(a), \alpha\bm{\Sigma}(a))$
\FOR{$t = 1, \cdots, T$}
\STATE Take feature vector $\mathbf{x}_{t}$,
\FOR{$a_t = 1, \cdots, |\mathcal{A}_t|$}
\IF{$\bm{\mu}(a_t)$, $\bm{\Sigma}(a_t)$ is available}
\STATE Sample $\bm{\theta}(a_t) \sim \mathcal{N}(\bm{\mu}(a_t), \alpha\bm{\Sigma}(a_t))$
\STATE Calculate $\hat{r}_{t}(a_t)$ according to eq.(\ref{eq:basic}).
\ELSIF{$\bm{\mu}(a_t)$, $\bm{\Sigma}(a_t)$ is NOT available due to lack of data}
\STATE Assign $\hat{r}_t(a_t)$ very large value at the probability of $\frac{1}{|\mathcal{A}_t|}$ and very small values at the probability of $1-\frac{1}{|\mathcal{A}_t|}$
\ENDIF

\ENDFOR
\STATE Select $a^\ast = \mathop{\rm arg~max}\limits\limits_{a \in \mathcal{A}_t} \hat{r}_{t}(a)$
\STATE Observe $r_t \in \{0, 1\}$
\IF{$t$ is the last impression of the day}
\STATE Update $(\bm{\mu}(a), \bm{\Sigma}(a))$ using data from the previous day
\ENDIF
\ENDFOR
\end{algorithmic}
\end{algorithm}

\subsection{Parameter Estimation}
\label{sec:Parameter Estimation}
We estimate $\left(\bm{\mu}(a_t), \bm{\Sigma}(a_t)\right)$ as posterior Gaussian distribution of the weight vector $\bm{\theta(a_t)}$ of eq.\ref{eq:basic}. That is, we first estimate $\bm{\theta} = (\bm{\theta_0}, \bm{\theta}(a_t))$ as point estimates from L2-regularized logistic regression and derive $\bm{\Sigma_0}, \bm{\Sigma}(a_t) \in \bm{\Sigma}$ since the inverse of Hessian of the objective function (negative log-likelihood plus L2-penalty) becomes variance-covariance matrix of the weight vector.
Since the derivation is same as Chapell and Li (2011)\cite{chapelle_empirical_2011} we do not detail here. It is basically using the result that the L2-regularized logistic regression is equivalent to MAP estimation and utilize Laplace transformation (see Bishop (2006)\cite{bishopPatternRecognitionMachine2006}). Due to the intractable size of our Hessian matrix, we only use diagonal elements of Hessian.

To estimate both common weight parameters $\bm{\mu}_0$ and action-specific $\bm{\mu}(a)$, $a \in \mathcal{A}$ at once, we first expand feature vector $\mathbf{x}_t$ by interacting a binary action vector $\mathbf{a} \in \{0, 1\}^{|\mathcal{A}|}$ whose $k$th element takes one when corresponding action is taken while the others take zeros. That is, for $\mathbf{a}_t = (0,1,0)^T$, $\mathbf{a}_t \otimes \mathbf{x}_t = (\mathbf{0}_d^T, \mathbf{x}_t^T, \mathbf{0}_d^T)^T$ where $0_d$ denotes a zero vector of length $d$. Finally, by combining action-invariant component $\mathbf{x}_t$, we have $\mathbf{z_t} = (\mathbf{x}_t^T, \mathbf{a}_t \otimes \mathbf{x}_t)^T$. We use $\mathbf{z}_t$ for both training and predicting procedures. The actual implementation is detailed in Section \ref{sec:Evaluation}.

\section{Fatigue-Aware Ad Creative Selection}
\label{sec:Fatigue-Aware Ad Creative Selection}
As explained in Section \ref{sec:Introduction}, we need to consider user's psychological status when predicting reward. Formally, our {\it fatigue aware} reward predictor becomes
\begin{eqnarray}
\label{eq:predictor with fatigue} \hat{r}_{i, t}(a_t) = \sigma(\mathbf{x}_t \cdot \bm{\theta}_0 + \mathbf{x}_t \cdot \bm{\theta}(a_t) + b_1\kappa_{i,t}(a_t)+ b_2\kappa_{i,t}(a_t)^2).
\end{eqnarray}
The equation \ref{eq:predictor with fatigue} departs from eq. (\ref{eq:basic}) in two ways. First, the new predictor uses the metric for the level of advertising fatigue $\kappa_{i,t}(a_t)$. $\kappa_{i,t}(a_t)$ measures the estimated ad fatigue when a user $i$ is exposed to ad creative $a_t$ at time $t$. The specific calculation of $\kappa_{i,t}(a_t)$ is detailed below. Second, the new predictor take into account user $i$ while eq. (\ref{eq:basic}) does not contain subscript $i$. In the sense, the fatigue-aware predictor can personalize the ad creative selection.

$\kappa$ is included in the function as a quadratic form. However, both $b_1$ and $b_2$ can take any values.
While the marketing literature find inverse-U shape relationship described in Figure \ref{fig:two factor}, we \textbf{do not} presume the relationship between the two. The actual values of weights $(b_1, b_2)$ for fatigue metric are learned from the data. With the new predictor the DSP server chooses ad creative $a_t$ from a set of candidates $\mathcal{A}_t$. The whole procedure is Algorithm \ref{alg:fatigue}.

\begin{algorithm}[htb]
\caption{Fatigue-Aware Ad Creative Selection (FA)}
\label{alg:fatigue}
\begin{algorithmic}
\REQUIRE: $\mathcal{N}\left(\bm{\mu}(a), \alpha\bm{\Sigma}(a)\right)$, $\mathbf{h}_{i,t}$, $\bm{s}$
\FOR{$t = 1, \cdots, T$}
\STATE Take feature vector $\mathbf{x}_{t}$ and history for $i$ at $t$ $\mathbf{h}_{i, t}$
\FOR{$a_t = 1, \cdots, |\mathcal{A}_t|$}
\IF{Action specific distribution is available}
\STATE get a vector of similarity scores for $a_t$, i.e. $\mathbf{s}(a_t)$
\STATE Calculate $\kappa_{i, t}(a_t) = \mathbf{h}_{i,t} \cdot \mathbf{s}(a_t)$
\STATE Sample $\bm{\theta}(a_t) \sim \mathcal{N}(\bm{\mu}(a_t), \alpha\bm{\Sigma}(a_t))$
\STATE Calculate $\hat{r}_{t}(a_t)$ according to eq.(\ref{eq:predictor with fatigue}).
\ELSIF{Action-specific distribution is NOT available}
\STATE Same as Algorithm \ref{alg:contextual}
\ENDIF
\ENDFOR
\STATE Select $a^\ast = \mathop{\rm arg~max}\limits\limits_{a \in \mathcal{A}_t} \hat{r}_{t}(a)$
\STATE Observe $r_t \in \{0, 1\}$
\IF{$t$ is the last impression of the day}
\STATE Update $\mathcal{N}(\bm{\mu}(a), \alpha\bm{\Sigma}(a)), b_1, b_2$ on data from previous day
\ENDIF
\ENDFOR
\end{algorithmic}
\end{algorithm}

\subsection{The Calculation of Fatigue}
\label{subsec:calc fatigue}
Here, we detail the calculation of fatigue metric $\kappa$. The basic idea is that the level of fatigue is determined by the number of past exposure to the same or similar ad creatives. Suppose the similarities among three ad creatives are as shown in Figure \ref{fig:similarity_matrix}.
\begin{figure}[htb]
  \centering
  \includegraphics[width=0.3\linewidth]{similarity_matrix.png}
  \caption{An example of similarity matrix.}
  \label{fig:similarity_matrix}
  \Description{}
\end{figure}
Then the level of fatigue for each ad creative candidates can be sum of similarity between each candidate and creatives to which the user are exposed in the past. Figure \ref{fig:fatigue_image} shows an example. This user has seen the blue ad creative three times and the yellow creative one time. Then the ad fatigue becomes approximately 3.4 (3 for the same creative and .39 for yellow creative) when he sees the blue.
\begin{figure}[htb]
  \centering
  \includegraphics[width=0.9\linewidth]{fatigue_image.png}
  \caption{Calculation of fatigue. The values of fatigue depends on the history of ad exposure and chosen ad creative. In this case, blue ad creative has been seen three times and incur the highest fatigue.}
  \label{fig:fatigue_image}
  \Description{}
\end{figure}

Formally, let $\mathbf{h}_{i, t}$ be a vector representation of history of ad exposures to user $i$ at time $t$. Each element in $\mathbf{h}_{i, t}$ represents the number of exposures to each creative. In the above example, $\mathbf{h_{i, t}} = (3, 1, 0)^T$. We record the histories for each user-advertiser pair. Hence, each user has multiple history vectors. The expected level of fatigue to ad creative $a_t$ is calculated as a weighted sum of history: $ \kappa_{i, t}(a_t) = \mathbf{h}_{i, t} \cdot \mathbf{s}(a_t)$, where the weight vector $\mathbf{s}(a_t)$ measure how similar $a_t$ and the other creatives are. That is,
\begin{eqnarray}
\mathbf{s}(a_t) = (s(a_t, 1), s(a_t, 2), \cdots, s(a_t, |\mathcal{A}_t|))^T, s(j,k) \in [0,1].
\end{eqnarray}

\subsection{Calculation of Similarity}
\label{sec;calculation of similarity}
Each similarity score $s(a, a')$ is calculated as a weighted average of {\it text similarity} and {\it image similarity}. The text similarity is calculated by the cosine similarity of the bag of words (BoW) representations of the description texts.
We extracted the words from the texts by using Mecab\cite{Mecab}, which is the most commonly used morphological analyzer for the Japanese language, with NEologd dictionary\cite{neologd}, which is a frequently updated Japanese dictionary containing up-to-date words.
We used the gensim library\cite{gensim} for the BoW calculation.

The image similarity is also calculated by the cosine similarity of the vector representation of the images.
The vector representation is extracted from each image by using pre-trained MobileNetV2\cite{MobileNetV2} implemented in Keras\cite{Keras}.
Here, we use the output from the last pooling layer as the representation.
We put three times higher weight to the text similarity because the text similarity (i.e., BoW representation) is much easier to interpret than the image similarity (i.e., weight of the neural network).

In a real world online display advertising environment, complicated models such as DNN are hard to be implemented because DSPs are required extremely short response time not to deteriorate user experience. By calculating similarity in advance, we can compress information from image and text into scalar variables it drastically reduces computing cost.

\subsection{Implementation}
The online advertising is an computationally complex environment. To deliver thousands of ad creatives to numerous users within unnoticeable duration, the system can spare only a few milliseconds to calculate fatigues for all the candidates of each impression. To save time, we first calculate the similarity scores for each pair of creatives off-line and store in the server.
We update the histories for each user-advertiser pair in \textbf{real-time}.
Owing to the extremely high frequency of impressions, recording all the ad creative exposures for all the users could lead to an explosion of the data volume, which needs to be capped at some point.
Therefore, the system records the first impression of a user to each ad creative every minute. That is, if a user watches the same ad creative more than once within a minute, it is counted as one impression.\footnote{This data reduction affect less than 40\% of the users.} To further reduce the data volume and consider forgetting, the histories more than 24 hours before are deleted from the database. The whole system is described in Figure \ref{fig:architecture}.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{architecture.png}
  \caption{Summary of fatigue-aware ad creative selection system.}
  \label{fig:architecture}
  \Description{}
\end{figure}

\section{Online Experiments}
\label{sec:Evaluation}
Since the reward is changed by the past intervention by the system, offline evaluation is not very meaningful in this case. Notice that replay method needs the exact same intervention history to each user in this case. Hence we run an online experiment in a production environment to examine the effectiveness of the proposed algorithm. In this section, we first describe the setting of the experiment and then show the results.

\subsection{Setting}
We deploy our Fatigue-Aware (FA) selection algorithm in a DSP provided by CyberAgent, inc., a Japan-based major online advertising company. Three advertising campaigns for three different mobile game titles are selected for the experiment according to various conditions \footnote{We used four campaigns but it turns out that one of them has only one creative.}. A part of impressions for these campaigns are used for the experiment. The sample size of the training data reaches a maximum of one million a day with the hundreds of thousand of features. To deal with the size and the sparsity of the training data, we employ negative down-sampling and the hashing trick\cite{weinberger_feature_2009}. We set sampling rate for negative data to 5\% and limit the length of features to $2^{24}$ for the system without fatigue parameters (Baseline) and $2^{24} + 2$ for FA by hashing-trick. The scalar parameter $\alpha$ that governs the degree of exploration of the algorithm are set to $0.01$ for both FA and Baseline. With the small value of $\alpha$, the Thompson sampling becomes {\it optimistic}\cite{chapelle_empirical_2011}. We apply stochastic gradient decent (SGDCliassifier in skleran package of Python) to obtain $\bm{\mu}$ with regularization parameter $\lambda = 0.0011$. The parameter vector is updated daily basis using a batch of data from the previous day.
These hyper-parameters are tuned using the replay method \cite{li_contextual-bandit_2010} for Baseline. FA shares all the above setting except for the inclusion of the fatigue parameter. To deal with cold start problem, FA is trained on data from the existing algorithm in the pre-experiment period. During the experiment, it learns only data that it produces.

Baseline (Alg.\ref{alg:contextual}) and FA (Alg.\ref{alg:fatigue}), are tested against the result of randomly-chosen ad creative (Random) in a standard A/B testing. That is, they share the impressions equally based on the users' id and update their parameters based on the logs they generate. The experiment ran for one week.

\subsection{Main Results}
Table \ref{tab:Main} shows the overall results of the experiments. Each metric is normalized with respect to the result of random algorithm (e.g., CTRs for FA and Baseline are divided by the CTR for Random). FA out-performed both baseline and random algorithm in CTR (click-through rate), CVR (conversion rate, the share of post-click conversions in all impressions), and Post-impression CVR including non-click conversions. The results clearly show that the proposed algorithm successfully increased both clicks and conversions. Baseline algorithm outperforms FA for post-click CVR. That is, while baseline collects less clicks than FA, baseline gets conversions more likely than FA once it gets click. But the difference is not statistically significant.

\begin{table}[htb]
\centering
\begin{tabular}{lrrrrr}
  \hline
 Alg. & Impressions & CTR & CVR & \shortstack{Post-Click \\CVR} & \shortstack{Post-Imp\\ CVR} \\
  \hline
  FA & 1,097,261 & \textbf{1.08}$^{\ast}$ & \textbf{1.12} & 1.04 & \textbf{1.09}$^{\ast\ast\ast}$ \\
  Baseline & 1,081,393 & 1.04 & 1.10 & \textbf{1.05} & 1.04 \\
  Rand & 1,087,931 & 1.00 & 1.00 & 1.00 & 1.00 \\
   \hline
\end{tabular}
\caption{The results of the experiments. The numbers are normalized except for the number of impressions. Normalization is performed with respect to the random algorithm. The results are for the aggregate of all three campaigns. Superscripts implies FA is statistically better than Baseline. $P<0.1:^\ast,  P <0.05:^{\ast\ast},  P<0.01 :^{\ast\ast\ast}$}
\label{tab:Main}
\end{table}

\subsection{Heterogeneity in campaigns}
To further examine the results, Table \ref{tab:campaigns} shows the CTRs for each campaign. The detailed results reveal that the baseline is not always superior to random. As for conversion rates, the performance of the baseline is poor in campaign A. On the other hand, our fatigue-aware algorithm stably outperforms the other two. The instability of baseline possibly comes from the data volume as campaign A has least size of data. With combination of small size of training data and sparsity, the estimates could be very unstable. In real-world production environment, the data volume cannot be controlled by experimenters. Still, fatigue bandit algorithm successfully learned the data.

\begin{table}[ht]
\centering
\begin{tabular}{lrrrr}
  \hline
& \multicolumn{3}{c}{Campaigns}\\
Alg & A & B & C \\
  \hline
FA & \textbf{1.21} & \textbf{1.04} & \textbf{1.32} \\
Baseline & 0.95 & 1.03 & 1.11 \\
Rand & 1.00 & 1.00 & 1.00 \\
   \hline
\end{tabular}
\caption{CTRs for three campaigns. Numbers are again normalized with respect to random algorithm.}
\label{tab:campaigns}
\end{table}

\section{Discussion}
\label{sec:Discussion}
In this section, we further investigate the results focusing on the fatigue. Figure \ref{fig:fatigue_dist} shows the distribution of user fatigue at time that the algorithm chooses the ad creative. While the distributions of fatigue are similar among three campaigns, Campaign A has the mean highest similarity score (Table \ref{tab:simiarity}) and higher fatigue level. The distribution is dense around 0 to 10, which is equivalent to 0 to 10 times exposure to same ad creative within past 24 hours.

\begin{figure}[htb]
 \includegraphics[width=\linewidth]{fatigue_dist.png}
  \caption{The distribution of fatigue level at each impression for three campaigns used in the experiment. The fatigue level is calculated according to Section \ref{subsec:calc fatigue}. Bin width is 5.}
  \label{fig:fatigue_dist}
  \Description{}
\end{figure}

\begin{table}[ht]
\centering
\begin{tabular}{cccc}
  \hline
Campaign & \# creatives & mean & sd \\
  \hline
A & 21 & 0.48 & 0.19 \\
  B & 12 & 0.39 & 0.09 \\
  C & 5 & 0.34 & 0.05 \\
   \hline
\end{tabular}
\caption{Summary of similarity scores for each campaign}
\label{tab:simiarity}
\end{table}

Figure \ref{fig:freq_fatigue} shows the relation between the number of ad exposures and the the level of fatigue. Apparently, users feel least fatigue with random algorithm (dashed line). Random algorithm just distribute ad creatives evenly regardless of their performance while two other algorithms choose ``better'' ad creative to maximize clicks. We see the clear difference between fatigue-aware (dotted line) and baseline (solid line). Fatigue-aware algorithm tries to save the accumulation of fatigue by changing ad creatives when the level of fatigue is high. The fatigue-aware algorithm indeed works as we expected.

\begin{figure}[htb]
  \centering
  \includegraphics[width=4.5cm]{freq_fatigue.png}
  \caption{The relationship between the number of ad exposures in past 24 hours for each user (frequency) and the level of fatigue. Fitted values by local linear regressions of fatigue on frequency.}
  \label{fig:freq_fatigue}
  \Description{}
\end{figure}

Next, we check the association between the level of fatigue and the metrices (CTR and CVR). Figure \ref{fig:cv_fatigue} shows the local linear regressions CTR/CVR on fatigue. To eliminate bias generated by algorithms we use logs from random algorithm only. The two panels shows the difference between CTR and CVR. While CTR constantly decreases with the fatigue, CVR shows more complex relationship. In other words, we see only negative effects of fatigue on CTR but find both positive and negative effects for CVR. Possible explanation is that while audience tend to click on new and fresh ad creative for simple curiosity, they do not login game until they understand the message brought by ad creative after repetitive exposures. Hence, it is possible that the DSP should change the algorithm for different goals. This could explain why Baseline is better than FA in terms of post-click CVR. Baseline tends to show the same ad creatives consistently and gets more chance to get conversions once it gets clicks. But overall performance (CTR and CVR) is in favor of FA.

\begin{figure}[htb]
  \centering
  \begin{minipage}{.4\linewidth}
  \includegraphics[width=\linewidth]{click_fatigue.png}
    \end{minipage}
\begin{minipage}{.4\linewidth}
 \includegraphics[width=\linewidth]{imp_cv_fatigue.png}
    \end{minipage}
  \caption{The relationship between the level of fatigue and CTR (left) and post-impression CVR (right). Local linear regression of CTR/CVR on fatigue. Data is from random algorithm. Each line represents each campaign.}
  \label{fig:cv_fatigue}
  \Description{}
\end{figure}

\section{Conclusion}
\label{sec:Conclusion}
In this paper, we proposed a fatigue-aware ad creative selection algorithm that explicitly considered the changes in users' attitude by the repetitive ad exposures. We introduced a numerical measure of the level of fatigue and set up easy-to-implement and efficient algorithm that changes ad creative according to the level of the fatigue. The proposed algorithm was applied to a running DSP in a production environment for one week. The results show the superiority of the proposed algorithm over the baseline algorithm.

\begin{acks}
The authors thank the Dynalyst team of CyberAgent, especially Hidetoshi Kawase, Yuta Kurosaki, and Shuhei Kimura for providing immense support to perform the experiments, and Kazuki Taniguchi, Masahiro Nomura, Kota Yamaguchi, Mayu Otani (AILab), Takanori Maehara (RIKEN), and Junpei Komiyama (NYU) for helpful advice.
\end{acks}

\newpage
\bibliographystyle{ACM-Reference-Format}

Computers and Society (cs.CY); Machine Learning (cs.LG); Machine Learning (stat.ML)

Fatigue-Aware Ad Creative Selection


Radioactivity and Electron Acceleration in Supernova Remnants

Abstract: We argue that the decays of radioactive nuclei related to $^{44}$Ti and $^{56}$Ni ejected during supernova explosions can provide a vast pool of mildly relativistic positrons and electrons which are further accelerated to ultrarelativistic energies by reverse and forward shocks. This interesting link between two independent processes - the radioactivity and the particle acceleration - can be a clue for solution of the well known theoretical problem of electron injection in supernova remnants. In the case of the brightest radio source Cas A, we demonstrate that the radioactivity can supply adequate number of energetic electrons and positrons for interpretation of observational data provided that they are stochastically pre-accelerated in the upstream regions of the forward and reverse shocks.

\section{Introduction}

Supernova remnants (SNRs) are generally believed to be prime
candidates for production of both hadronic and electronic
components of galactic cosmic rays (CRs) via the diffusive shock
acceleration (DSA)  mechanism (see e.g. \cite{malkov01} for a
review). While the main aspects of the theory are well understood,
the key issue related to electrons is the so-called injection
problem which despite certain theoretical attempts (see e.g.
\cite{levinson94, bykov99}), remains an open question.  The
injection of  electrons is a serious challenge because the
electron gyroradius  is  small compared to   the shock
thickness which  is of the order of  the proton  gyroradius. In fact
this is a more general problem, related not only to DSA but also
to other electron acceleration mechanisms, e.g. through different
scenarios of stochastic acceleration \cite{fermiII}. In this paper
we  explore whether the  pool of suprathermal electrons and
positrons  related to the  decay products of radioactive nuclei
$^{56}$Ni and $^{44}$Ti can serve as an effective injector for
further acceleration of electrons in SNRs by the forward and  reverse shocks.

It is well established  that  the supernova ejecta contain
huge amount of radioactive nuclei. The decays of
these unstable nuclei have  been  proposed  as a source  of
low energy  positrons (see e.g. ref. \cite{chan93, martin10})
responsible for the  0.511 MeV annihilation
line observed from the direction of the Galactic Center.
In the case of the core-collapse supernova Cas~A,
approximately  0.1$M_{\odot}$ of $^{56}$Ni  has been  ejected
just after the explosion\cite{krause08}. The nuclei $^{56}$Ni decay with a
half lifetime  $t_{1/2}=6.1$~days into $^{56}$Co. Over the
first years after the explosion,  the decay products  of $^{56}$Co
($t_{1/2}=77$ days)  support the supernova optical light emission.
At later epochs, less abundant
radioactive nuclei with longer lifetimes  contribute to the
production of low energy suprathermal electrons, positrons and
gamma-rays. In particular, the detection of  characteristic
gamma-ray   \cite{iyudin94} and hard X-ray lines \cite{renaud06}
from $^{44}$Ti  gives a robust estimate of the total mass of
radioactive  $^{44}$Ti  ($t_{1/2}=63$ years) produced
in Cas~A:  $2\cdot 10^{-4}M_{\odot}$ . Recently a comparable amount
of $^{44}$Ti has been found also in the   youngest galactic supernova remnant -
SNR~G1.9+0.3 \cite{borkowski10}.

\begin{figure}[t]
\includegraphics[width=6.0cm,angle=270]{fig1.eps}
\caption{Schematic view of a young supernova remnant in the
context of the ``radioactive origin'' of relativistic electrons.
The forward shock propagates in the circumstellar medium outward,
while the  reverse shock  propagates into the ejecta (gray
color) outward in the laboratory frame and inward in the frame of
expanding ejecta. Pre-existing energetic electrons are produced in
the circumstellar medium via the Compton scattering of gamma-rays
from the decay of $^{56}$Co. The  radioactive decays of  $^{44}$Ti
provide energetic electrons and positrons in the ejecta.}
\end{figure}

Cas A, an approximately 300 year old  remnant,
shows bright  broad-band emission extending from radio to  gamma-rays.
It consists of both thermal and nonthermal components, indicating the presence
of hot  thermal plasma, strong magnetic field,  relativistic electrons, and likely also protons,
accelerated up to multi-TeV  energies.
All these components constitute a significant fraction
of the bulk motion kinetic energy of the shell expanding with a
speed of 4000 to 6000 km s$^{-1}$ \cite{patnaude09}. Most likely,
acceleration of electrons takes place  both in forward and reverse shocks.

Thin non-thermal X-ray filaments detected at the periphery of the remnant \cite{gotthelf01}
reveal the presence of a strong $\sim $ 1~mG magnetic field \cite{voelk05} and
multi-TeV electrons accelerated at the forward shock of Cas A.   Synchrotron X-rays are
produced both  in the reverse and  forward shocks \cite{helder08}.  The  time
variations of synchrotron X-radiation  found for a number of
filaments and  knots associated with the reverse shock, indicate that
magnetic field in these compact  structures also is  very large,
close  to 1~mG\cite{Uchi08}.   Because of  large  magnetic fields,
gamma-rays produced via inverse Compton scattering is strongly suppressed,
except  for some regions in the reverse shock with relatively small
magnetic field. Even so, the total energy in protons, assuming that the detected
GeV \cite{abdo10} and  TeV gamma-rays \cite{aharonian01,albert07,acciari10}
are of purely hadronic origin,  does not significantly exceed $10^{49}$erg \cite{abdo10}.
On the other hand,  the  bright  synchrotron
radio emission of Cas~A  indicates to the  existence of  huge amount of relativistic electrons
accelerated by forward and reverse shocks
with total  energy as large as $10^{48} \ \rm erg$ \cite{atoyan00}.
That constitutes approximately  $10^{-3}$ fraction of the  explosion (mechanical) energy.
A significant fraction  of this energy is contained in compact
radio-knots of Cas~A \cite{tuffs86, anderson95}
where  the pressure  of  relativistic  electrons is  comparable to
the thermal pressure  of the shell.

\section{Production of energetic electrons and positrons}

Below  we assume that the radioactive
elements are distributed uniformly throughout the ejecta. Although
these elements are synthesized predominantly
in the core of  the ejecta,  during the  explosion
they can be well  mixed  in  the ejecta.

The ratio of number density of energetic MeV positrons $n_+$ from
$\beta $-decay of $^{44}$Ti to the baryonic density of the ejecta
$n_{ej}$ is given by
\begin{equation}
\frac {n_+}{n_{ej}}=0.94\frac {M_{Ti}}{44M_{ej}}\left[ 1-\exp
\left( -\frac {t\ln 2}{t_{1/2}}\right) \right] .
\end{equation}
Here $t$ is the time since supernova explosion, $M_{ej}$ is
the mass of ejecta, and  $M_{Ti}$  is the  total mass of the ejected nuclei $^{44}$Ti.
Eq. (1)  takes into  account that positrons appear in $94\% $ of the $^{44}$Sc decay.

The rate of Coulomb energy losses of electrons and positrons is described as
\[
\frac{\dot{E}}{E}=\frac {4\pi r_e^2m_ec^4n_{ej}}{vE}\Lambda  \left< \frac ZA\right> =
\frac {r_e^2m_ec^4}{vE}\Lambda \frac {3(k-3)M_{ej}}{2km_pV_{ej}^3t^3}
\sim
\]
\begin{equation}
\frac {0.025}{t}\frac {m_ec^3}{vE}\frac {\Lambda }{40}
\left( \frac {M_{ej}}{M_{\odot }}\right) ^{5/2}
\left( \frac {E_{SN}}{10^{51}\ \mathrm{erg}}\right) ^{-3/2}\left( \frac {t}{63\ \mathrm{yr}}\right) ^{-2}.
\end{equation}
Here $r_e$ is the classical electron radius, $m_p$ and $m_e$ are the proton and electron masses respectively,
$\Lambda \simeq 40$
is the Coulomb logarithm in fully ionized plasma,
$E_{SN}$ is the total energy of explosion, and
$V_{ej} = \left( 10(k-5)E_{SN} /3(k-3)M_{ej}\right) ^{1/2}$ is the characteristic velocity of ejecta
with a power-law  density distribution characterized by the  index $k \sim 10$ \cite{chevalier82b}.
In Eq. (2)  the mean ratio of the atomic number to the mass number $\left< \frac ZA\right> $ is
taken 0.5.
Note  that,  in addition to positrons with energy $E_+\sim $1 MeV,
one electron of  energy $E_-\sim $0.1 MeV is produced  per a $^{44}$Ti decay.
However,  because of difference in energies
the positrons have more chances to be accelerated before they are  thermalized.  Therefore
the  fraction of the  accelerated  positrons  $n_+/(n_++ n_-) \geq 1/2$.

For supernova explosions with small ejecta masses, $M_{ej}<5M_{\odot}$,
the energy losses of positrons from decays of $^{44}$Ti are  not  significant
(see also \cite{martin10}). For  larger ejecta  masses, the energetic positrons
are thermalized  before they
are injected into the reverse shock.  In any case,  these particles  cannot
travel and approach  the forward shock. In this regard, $^{44}$Ti
cannot provide electrons and positrons for acceleration by the
forward shock. Nevertheless,  the forward shock  can be
supplied by  suprathermal  electrons, but  through a different (indirect)  way
related to the Compton scattering of  MeV  gamma-rays - the products of
$^{56}$Co decays.

The number density of energetic  electrons of Compton origin produced by
MeV gamma-rays from $^{56}$Co decays in the
circumstellar medium with number density $n$ is estimated as
\begin{equation}
\frac {n_-}{n}=\xi _{\gamma }\frac {M_{Ni}}{56m_p}\frac {\sigma _{\rm T}}{4\pi r^2}
\sim 1.2\cdot 10^{-7}\xi _{\gamma }\frac {M_{Ni}}{M_{\odot }}r^{-2}_{\rm pc}.
\end{equation}
Here $\sigma _{\rm T}$ is the Thompson cross-section, $r$ is the
distance from the center of the supernova explosion and
$\xi _{\gamma }$ is the fraction of gamma-rays which escape
the expanding ejecta. For photons of  energy of $E \sim 0.5$ MeV
the cross-section of the Compton scattering is
$\sigma _{\rm C}=0.4\sigma _{\rm T}$. It is  taken into account in Eq.(3) that
in a single act of decay of  $^{56}$Co
on average  2.5  gamma-ray photons are produced.
We should note that a similar idea  for the production  of
energetic electrons in SNRs via the Compton scattering of
gamma-rays from the annihilation of $^{56}$Co decay positrons has been
earlier suggested by Bychkov \cite{bychkov77}. This gives
additional  0.5 gamma-photons per a decay of $^{56}$Co.

In the interstellar medium, the timescale of the Coulomb and ionization losses of
energetic electrons  is  of the order of
$10^5$ years. During $300$ years they cannot  diffuse away beyond
3 pc, given that  the diffusion coefficient that characterizes
their propagation does not exceed the standard value of the diffusion coefficient in
the interstellar medium,  $D \sim 10^{28}$ cm s$^{-1}$. Therefore
they will be picked  up by the  arriving  SNR shock.

The fraction of gamma-rays  that escape  the supernova ejecta
is determined by the optical depth $\tau $:
\[
\tau =\left< \frac ZA\right> \sigma _{\rm C}\int n_{ej}dr
=\frac {3(k-3)M_{ej}\sigma _{\rm C}}{4\pi (k-1)m_pV^2_{ej}t^2}\left< \frac ZA\right> \sim
\]
\begin{equation}
0.6\left( \frac {M_{ej}}{M_{\odot }}\right) ^2
\left( \frac {E_{SN}}{10^{51}\ \mathrm{erg}}\right) ^{-1}
\left( \frac {t}{77\mathrm{days}}\right) ^{-2}.
\end{equation}

In order to escape the ejecta without significant lose of energy,
the Compton optical depth for gamma-rays  $\tau$  should not significantly exceed 1.
This determines the time $t$ and the corresponding amount of
non-decaying $^{56}$Co. As it follows from Eq.(4) gamma-rays
from decays of $^{56}$Co can escape the ejecta only if the mass of latter
does not exceed several solar masses. For larger  ejecta masses, the contribution of
gamma-rays from longer-lived
isotopes, e.g.  $^{57}$Co ($t_{1/2}=272$ days, mass $\sim 0.003\ M_{\odot }$ \cite{meyer95}),
becomes more important.

Note that for any reasonable parameters, the Compton optical depth
in the interstellar medium is much smaller than one (even in the
galactic scales), therefore only a small fraction of energy
released at $^{56}$Co decays is transferred to energetic electrons
in the circumstellar medium. The main fraction of energy goes to
the heating of the ejecta.

\section*{Acceleration of electrons}

At the plane non-modified shock with compression ratio $\sigma$,
the far-upstream and downstream momentum distributions of particles,
$F_0(p)$ and $F(p)$, respectively,  are related as

\begin{equation}
F(p)=\gamma \int ^p_0\frac {dp'}{p'}\left( \frac {p'}{p}\right) ^{\gamma }F_0(p').
\end{equation}
Here $\gamma =3\sigma/(\sigma -1)$ is the  Krymsky's index.

Let us assume now that the suprathermal  electrons with a mean energy
$E_{\mathrm{inj}}$ are injected into the plane shock.
For a non-modified strong shock with
compression ratio $\sigma =4$  we have the following expression for
the pressure of accelerated electrons:
\begin{equation}
P_-=\frac 43n_-E_{\mathrm{inj}}\ln \frac {E_{\max }}{E_{\mathrm{inj}}}.
\end{equation}
Here $E_{\max }$ is the maximum energy of electrons accelerated at
the shock. In young SNRs  $E_{\max }$ is of the order of $10-100$ TeV.
Using the number density given by Eq. (1), we can estimate the ratio of the pressure
of positrons $P_+$ to the ram pressure of the reverse shock, $\rho u_r^2$,
propagating  at $t>>t_{1/2}$  into the
ejecta  with a  speed $u_r$:

\[
\frac {P_+}{\rho u_r^2}=\frac {4}{3}\frac {0.94M_{Ti}}{44M_{ej}}
\frac{E_{\mathrm{inj}}}{m_pu_r^2}\ln \frac {E_{\max }}{E_{\mathrm{inj}}}
\sim
\]
\begin{equation}
2.7\frac {M_{Ti}}{M_{ej}}E_{\mathrm{inj}}^{\mathrm{MeV}}
\left( \frac {u_r}{10^3\mathrm{km}\ \mathrm{s}^{-1}}\right) ^{-2}
\ln \frac {E_{\max }}{E_{\mathrm{inj}}}.
\end{equation}
A similar estimate  for the ratio of the electron pressure to the ram pressure $\rho u_f^2$ of the
forward shock propagating in the circumstellar medium with a  speed $u_f$,  gives
\[
\frac {P_-}{\rho u_f^2}=\frac {4}{3}\xi _{\gamma }\frac {M_{Ni}}{56m_{p}}\frac {\sigma _{\rm T}}{4\pi r^2}
\frac{E_{\mathrm{inj}}}{m_pu_f^2}\ln \frac {E_{\max }}{E_{\mathrm{inj}}}\sim
\]
\begin{equation}
1.5\cdot 10^{-5}\xi _{\gamma }\frac {M_{Ni}}{M_{\odot }}E_{\mathrm{inj}}^{\mathrm{MeV}} r^{-2}_{\mathrm{pc}}
\left( \frac {u_f}{10^3\mathrm{km}\ \mathrm{s}^{-1}}\right) ^{-2}
\ln \frac {E_{\max }}{E_{\mathrm{inj}}}.
\end{equation}

From these equations  follows that  the ratio of the electron pressure
to the ram pressure can vary, depending on the several principal
model parameters, within a broad range,  from   $10^{-7}$ to
$10^{-3}$. We  assume   that electrons  are
injected with their original energy $\sim$1 MeV. However their
energy can  be significantly larger  if  particles are
pre-accelerated in the upstream regions of the shocks.

\section{Pre-acceleration of electrons}

High-energy particles
accelerated at strong  shocks excite plasma waves and produce small-scale shocks  and
turbulence in the upstream region. The  turbulence may
amplify magnetic fields at the shocks of  young SNRs \cite{bell04}.
Also, the dissipation of  the turbulence results
in substantial gas heating upstream of the shock. The latter
 limits the total compression ratio of the shock  modified by cosmic rays.  This
is an important feature of  modern nonlinear shock acceleration models (see for a review
ref.\cite{malkov01}).
At  these conditions,  some
pre-acceleration of energetic electrons via the stochastic (second
order Fermi) mechanism  which also energizes thermal electrons and ions
in this region seems  rather  plausible. Note that in principle the stochastic
acceleration can be realized also via  ensemble of random shocks.
Also we should emphasize that there is an essential  difference between
the pre-existing energetic (supra-thermal) electrons and those, which in principle could be
injected at the shock front from the thermal pool. While the pre-existing
energetic electrons pass through the whole extended turbulent
region upstream of the shock,  the particles injected in the
shock front occupy a narrow region  at the shock. That is why
pre-acceleration of these electrons  is  not significant.
The reacceleration of sub-keV electrons from the thermal pool
of upstream plasma is problematic  also
because of strong Coulomb losses (see Eq. (2)).

The energy $E_{\mathrm{inj}}$ is determined by the efficiency of
stochastic acceleration upstream of the shock. The rate of
stochastic (second order) acceleration is $\tau _{st}^{-1}\sim
u_t^2/D$ while the rate of DSA is $\tau _{D}^{-1}\sim u^2/D$,
where $u_t$ is the  the velocity of turbulence (plasma waves) and
$D$ is the diffusion coefficient. The maximum energy of protons is
of the order of 100 TeV in young SNRs. Then for $u_t/u\sim 0.1$,
the maximum energy accelerated through the stochastic mechanism is
expected $E_{\mathrm{inj}}\sim 1$ TeV.  However, this  should be
considered as an optimistic upper limit, given that the diffusion
coefficient  for the low-energy particles in the turbulent region
upstream of the shock can be significantly larger than the Bohm
diffusion.

A more realistic estimate is given below. We shall consider the reacceleration of particles by multiple
 small-scale shocks in the upstream region of the SNR shock. A particle  is picked-up by the
 small-scale shocks, accelerated and advected downstream where it loses energy adiabatically.
 Then  the particle is picked-up by the next small-scale shock,  {\it etc}.

The energy density
 of relativistic electrons just
 downstream of the small-scale shock can be found after integration of Eq. (5).
 Because of the  adiabatic expansion in the
 downstream region,  this value drops by a factor of $\sigma _s^{4/3}$,
 where $\sigma _s$ is the compression
 ratio of the small-scale shock. So  the energy density  $\epsilon _-$ after
 one acceleration cycle is

\begin{equation}
\frac {\epsilon _-}{\epsilon _0}=\frac {\gamma _s}{\gamma _s-4}\sigma _s^{-4/3}=
\frac {3\sigma _s}{4-\sigma _s}\sigma _s^{-4/3}.
\end{equation}
Here $\epsilon _0$ is the electron energy density at the beginning of the cycle.

It is interesting to compare the relative change of the electron energy density to  the relative
change of the gas pressure $P$. Using
the Rankine-Hugoniot conditions we find

\begin{equation}
\frac P{P_0}=\frac {4\sigma _s-1}{4-\sigma _s}\sigma _s^{-5/3}.
\end{equation}
Here $P _0$ is the gas pressure in the beginning of the cycle.

One can see that the relative changes of the electron energy density and of the gas
pressure are similar.  For example, for $\sigma _s=3$ we have
${\epsilon _- }/{\epsilon _0}=2.08$ and ${P}/{P _0}=1.76$. For weaker
shocks, the change of the electron energy density is higher than the change
of the gas pressure. This means that after many cycles,  the relative change of the gas
pressure is comparable or smaller than the change of the electron
energy density. In other words,  the gas heating in the upstream
region of the SNR shock is accompanied by a similar or stronger
electron reacceleration.

Although the gas heating can not directly estimated from observations of SNRs,
one can constrain it (a lower bound)
assuming non-negligible amplification of the magnetic field.
Numerical studies  of the Bell's instability show that the energy
 density of the heated gas is comparable or
 higher than the energy of the amplified magnetic field\cite{bell04,zirakashvili08,riquelme09}.
 Namely,  within the synchrotron-loss interpretation of thin
X-ray filaments in young SNRs (see e.g. ref.\cite{voelk05}), the field  in the upstream region
can  be amplified by a factor of 5 to 10.  Therefore the
gas pressure should be increases by a factor as large as   100.
The similar level of the gas heating is needed
 to limit the strong shock modification
 and to avoid the appearance of the  concave CR spectra (see e.g. ref.\cite{malkov01}).
 It is sufficient  to have 8 cycles to provide
 a 100-fold increase of the gas pressure at the shocks with $\sigma _s=3$. The corresponding increase of
 $E_{inj}=\epsilon _-/n_-$ equals several hundreds.

The modeling of the Bell's instability
 with DSA \cite{zirakashvili08} shows  that the upstream region of a young SNR of width $L\sim 10^{18}$ cm
 is filled with a supersonic MHD turbulence with Mach number 3-4, while  the distance between small-scale
 shocks is $l\sim 10^{16}$ cm. For these parameters and for turbulent motions
$u_t/u\sim 0.1$ the expected number of cycles is $Lu_t/lu\sim 10$.

One should note that the
 pre-accelerated electrons  may have an impact  on the upstream
turbulence and thus regulate their own acceleration efficiency. In
particular,  the higher number density of pre-existing electrons would make
lower the energy $E_{\mathrm{inj}}$. Under these conditions,  the
energy density of pre-accelerated electrons may be of the order of
the energy density of the upstream turbulence. The latter is believed to be several
percent of the ram pressure $\rho u^2$ at cosmic ray modified  shocks. So the upper limit for the
number density of pre-accelerated  electrons is $n_-E_{\mathrm{inj}}\sim 10^{-2}\rho u^2$
in Eq. (6).

Even for a modest
energy  $E_{\mathrm{inj}}=100$ MeV,  one can obtain, according to Eq.(7),
quite high ratio $P_+/\rho u_r^2\sim 0.1$.
The shock may be slightly  modified by the pressure of accelerated
electrons and positrons!

\section{Applications to SNRs}

The above discussed picture of pre-acceleration of electrons and
positrons from  the products of decays of radioactive short-lived
elements can be relevant to the  reverse shock of Cas A. This can
explain why the pressure of energetic positrons (electrons) in the
shocked ejecta is  comparable to the gas pressure in the supernova
shell. The same could be true also for the radio-knots if they are
fast moving clumps of the shocked ejecta. At the present epoch,
the  pressure of energetic electrons at the forward shock of Cas~A
is not very high,   as it follows from Eq.(8). However, most
likely  it was much higher  in the past when the radius of the
remnant was smaller than 0.1 pc. Since the forward shock of Cas~A
propagates in a dense stellar wind of the supernova progenitor
with a density profile $\sim r^{-2}$, the accelerated electrons
have been produced mainly in the past when the synchrotron cooling
in the amplified field was significant. Now these electrons are
located inside the forward shock. This  can explain the  rather
steep radio-spectrum  of Cas A.

In Cas~A,   the energy of pre-accelerated electrons $E_{\rm inj}$
can not exceed  100-200 MeV, otherwise this would  be in conflict
with the observed  synchrotron radio spectrum.
The  spectral flatening seen at 20 MHz \cite{baars77}
can  be attributed,  for the magnetic field at the reverse
shock of the order of 100-200 $\mu $G, to the lower energy cut-off in the
electron spectrum at 100 MeV.  The magnetic
field at the forward shock of  Cas A is  larger.
However since radio-emitting electrons have been  accelerated
in this region in the past,  because of adiabatic losses
their low-energy cut-off is now located below 100 MeV

We conclude that the high radio brightness of Cas~A
is caused  by  the dense stellar wind where the
forward shock propagates, and by  a relatively high amount of radioactive
$^{44}$Ti  the decay of which
provides supra-thermal electrons and positrons for the  further
acceleration  by  the reverse shock. This is in contrast to  other historical
young SNRs like Tycho, Kepler and SN1006. They are results of Ia supernova explosions
in  uniform medium.  Therefore, in these objects the electrons
accelerated  by forward shocks, are  produced predominantly at later epochs.
In addition, the ejecta of Cas~A, because of the dense stellar wind  has been shocked
very early,  likely just after the explosion. The radiative instabilities
operated in the shocked ejecta, could  result in the  formation of ejecta
clumps \cite{hwang03,hwang09}, which presently are observed as radio-knots.

Since the reverse shock of Cas~A contains about 1$\%$ of  the
explosion energy, the energy fraction of electrons and
positrons is close to $10^{-3}$. The electrons accelerated at the forward shock have a similar energetics. So
 we expect that in Cas~A  approximately
 $10^{-3}$ fraction of supernova energy is transferred to the accelerated
electrons and positrons.  This conclusion is in agreement with estimates
based on radio observations \cite{atoyan00}.

The fraction of energy $10^{-3}$ found for positrons in the reverse shock of
Cas~A is expected to be
the same for all young core-collapse supernova. However GeV
positrons  leave the remnant only at late stages when its
radius becomes  a factor of 10 larger than the radius at the transition
to the Sedov phase when the positrons have been accelerated. Since the energy of
particles adiabatically drops (inverse proportional to the
remnant's radius), the energy fraction of positrons will  be reduced down
to $10^{-4}$. The
luminosity in galactic CR positrons at multi-GeV energies
based on the recent measurements of the Pamela collaboration \cite{adriani09} is close to
$10^{38}$ erg s$^{-1}$. Given the overall  mechanical power of the galactic core
collapse supernova  $10^{42}$ erg s$^{-1}$, our model can explain the flux
of the primary CR positrons by reverse shocks of young SNRs
without invoking other source populations (for
a review on different potential sources of galactic CR positrons see \cite{Positrons}).

It is important to note that our model applied to Cas~A  predicts
the positron-to-electron ratio close to 1. The reason is that (i)
the estimated energetics of leptons in forward and reverse shocks
in  Cas A based on radio observations are comparable, and (ii) our
model implies that while electrons are accelerated in the forward
shock, in the reverse shock the content of positrons is equal or
larger than the content of electrons.  If so, Cas A, as well as other young SNRs
alone cannot provide the total flux of galactic CR
electrons. In fact this is a model-independent statement based on
the estimates of numbers of electrons in young SNRs.  For old SNRs
the situation is  different. While the reverse shocks disappear in
these objects, the forward shock continue to accelerate electrons
(although to modest energies, $E \leq 1$~TeV). In our model, the
electrons produced via the Compton scattering of gamma-rays from
$^{56}$Co are accelerated by  forward shocks in  old Ia SNRs
expanding in the uniform medium.  According to the observed light
curves, the ejecta of Ia supernova contains $\sim $ 0.6$M_{\odot }$
of $^{56}$Ni just after the supernova explosion.  The energetics
of galactic Ia supernova is of the order of $3\cdot 10^{41}$ erg
s$^{-1}$, implying approximately  one supernova per century. On
the other hand, the  production rate of galactic CR
electrons is close to $10^{39}$ erg s$^{-1}$\cite{berezinsky90}.
So a  fraction of $0.3\% $  of energy of Ia supernova must  be
transferred to CR electrons. The similar ratio of cosmic
ray electron pressure to the ram pressure  is estimated for an old
remnant with the radius $30$ pc and the shock speed  $300$ km
s$^{-1}$ if $E_{\mathrm{inj}}=3$GeV (see Eq. (9)). The required  higher
value of $E_{\mathrm{inj}}$ can be explained by  a lower number density of
the circumstellar medium where  the Ia supernova explosions occur.

We should note that another source of the suprathermal electrons
at supernova shocks has been recently  suggested  by Morlino
\cite{morlino11}. Partially ionized
 multi-GeV ions accelerated at the shock can produce multi-MeV electrons via
 photo-ionization by optical Galactic emission. The fraction $\eta =n_-/n$
 of the corresponding electrons is
 estimated  as $\eta \sim 0.1x_{He}\gamma ^{-1}u^2/c^2$ at cosmic-ray modified shocks.
 Here $x_{He}\sim 0.1$ is the fraction of Helium in the interstellar medium,
 $\gamma \sim I_{He}/\epsilon _{ph}$ is the gamma-factor of He$^{+}$ ion ionized by Galactic
 optical photons with energy $\epsilon _{ph}$, and $I_{He}=54$ eV is the ionization
potential of Helium. This results in $\eta \sim 10^{-4}u^2/c^2$ in young SNRs where $\gamma \sim 100$
 and ions are photo-ionized by eV optical photons, and $\eta \sim 10^{-3}u^2/c^2$ in the
 old remnants where $\gamma \sim 10$ and ions are photo-ionized by ultraviolet photons. These numbers
 are comparable or higher than numbers given by Eq. (3).
 Even without any preacceleration by MHD turbulence this mechanism
 results in the electron to proton ratio $K_{ep}\sim x_{He}m/m_p\sim 10^{-4}$. Although the preacceleration of
 these electrons is more problematic because they are produced closer to the shock by 10-100 GeV ions, it is not
 excluded. Then the corresponding injection energy necessary for explanation of galactic cosmic ray
 electrons can  be below 1 GeV closer to $E_{inj}=100$ MeV  as argued above
 for reverse shock of Cas A.

Finally, in the context of the proposed model,
one can expect harder CR positron spectrum. The positrons
of higher energies leave the remnant earlier and are subject to
lower adiabatic losses in comparison with the positrons of lower energies.
This effect does not have an impact  on the spectra of
electrons  accelerated  predominantly
by forward shocks in  old SNRs.
The harder source spectrum
of positrons is in agreement with the recent Pamela measurements \cite{adriani09}.

According to the scenario proposed in this paper, only forward
shocks of young SNRs produced by supernova explosions with a small
ejecta masses $M_{ej}<2M_{\odot }$, can contain large amount of
accelerated electrons. The relevant  SNRs belong to  the
Ia/b/c and, probably,  IIb (like Cas~A)  type supernovae. Note
that the  brightest in TeV gamma-rays young SNR RX J1713.7-3946
most likely belongs to Ib/c type SNR with a  small ejecta mass
\cite{zirakashvili10a}. In the case of {\bf IIP}
supernova with large ejecta masses   gamma-rays from $^{56}$Co
decay  cannot  effectively escape the ejecta and  "feed" the
forward shock by suprathermal electrons for further acceleration.
If so, we should expect forward shocks of IIP SNRs to be  dim in radio and
non-thermal X-rays. On the other hand,  large amount  of electrons
and positrons from decays of $^{44}$Ti  can be accelerated at reverse
shocks  of young SNRs of all types including the most frequent IIP
supernovae. In this regard, the youngest
galactic SNR  G1.9+0.3 is  of a special interest. It shows both large content of
$^{44}$Ti and  ongoing acceleration of electrons by reverse shock
\cite{borkowski10} - two key components  required in  our model.

\section{Summary}

 The "radioactive"  origin  of electron injection, related to both the  forward and
reverse shocks,  seems to be  a natural scenario in SNRs with the following key  components:

1) the energetic positrons (and possibly also electrons) from $^{44}$Ti decay are
accelerated at  reverse shocks of young SNRs;

2) the energetic electrons from the Compton scattering of
$^{56}$Co-decay gamma-rays are accelerated at forward shocks of both old and young SNRs
of type Ia/b/c and IIb;

3) a modest  pre-acceleration (presumably of stochastic origin) to
energies $E_{\mathrm{inj}}\sim 0.1$ GeV in the upstream regions of  the
forward and reverse shocks is a necessary condition in Cas A  for explanation
of  the energetics in  relativistic electrons;

4) the  proposed scenario can explain not only the overall flux of galactic CR electrons
by SNRs, but also the  recently reported tendency of gradual increase of the
positron-to-electron ratio
with energy.