TWIN: TWo-stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou

Abstract: Life-long user behavior modeling, i.e., extracting a user's hidden interests from rich historical behaviors in months or even years, plays a central role in modern CTR prediction systems. Conventional algorithms mostly follow two cascading stages: a simple General Search Unit (GSU) for fast and coarse search over tens of thousands of long-term behaviors and an Exact Search Unit (ESU) for effective Target Attention (TA) over the small number of finalists from GSU. Although efficient, existing algorithms mostly suffer from a crucial limitation: the \textit{inconsistent} target-behavior relevance metrics between GSU and ESU. As a result, their GSU usually misses highly relevant behaviors but retrieves ones considered irrelevant by ESU. In such case, the TA in ESU, no matter how attention is allocated, mostly deviates from the real user interests and thus degrades the overall CTR prediction accuracy. To address such inconsistency, we propose \textbf{TWo-stage Interest Network (TWIN)}, where our Consistency-Preserved GSU (CP-GSU) adopts the identical target-behavior relevance metric as the TA in ESU, making the two stages twins. Specifically, to break TA's computational bottleneck and extend it from ESU to GSU, or namely from behavior length $10^2$ to length $10^4-10^5$, we build a novel attention mechanism by behavior feature splitting. For the video inherent features of a behavior, we calculate their linear projection by efficient pre-computing \& caching strategies. And for the user-item cross features, we compress each into a one-dimentional bias term in the attention score calculation to save the computational cost. The consistency between two stages, together with the effective TA-based relevance metric in CP-GSU, contributes to significant performance gain in CTR prediction.

\section{Introduction} \label{sec:intro}
As one of the most popular short video sharing apps in China, \textit{Kuaishou} strongly relies on its powerful recommendation system (RS).
Every day, RS helps hundreds of millions of active users to filter out millions of uninterested videos and reach their interested ones,
leaving tens of billions of click through logs.
These tremendous data not only feed the training of RS, but also boost technique revolutions that keep lifting both the user experience and business effectiveness on this platform.

In modern RSs, a fundamental task is Click Through Rate (CTR) prediction which aims to
predict the probability that a user would click an item / video \cite{chen2016deep,guo2017deepfm,zhang2019field}. Accurate CTR prediction directs RS to serve each user one's favourite contents and deliver each video to its interested audiences. To achieve this, CTR models should be highly personalized and make full use of
scarce user information.
Consequently, \textit{life-long user behavior modeling}, i.e., extracting a user's hidden interests from rich long-term historical behaviors, usually acts as a key component in CTR models
\cite{zhou2018deep,zhou2018atrank,liu2007framework,zhou2019deep,chung2014empirical}.

Industrial life-long behavior modeling algorithms mostly follow the two cascading stages \cite{pi2020search}:
(1) a General Search Unit (GSU) that conducts a fast coarse search
over tens of thousands of long-term behaviors and outputs a small number of most target-relevant ones, and
(2) an Exact Search Unit (ESU) that
performs effective
\textit{Target Attention} (TA) over the small number of finalists from GSU.
The reason behind this two-stage design is twofold. On one hand, to precisely capture the user interest, TA is a proper choice
for emphasizing target-relevant behaviors and suppressing target-irrelevant ones. On the other hand, the
expensive computational cost of TA limits its applicable sequence length to at most a few hundreds.
To this end, a simple and fast GSU as a pre-filter is essential for cutting-off industrial scale behavior sequences which could easily reach $10^4$ in just a few months.

\begin{figure}[t]
  \centering
  \hspace{-0.3cm}
  \includegraphics[width=0.4\textwidth]{picture/croped_fig1.pdf}
  \caption{Inconsistency between GSU \& ESU in a Conventional Two-Stage Algorithm.
  Assume that an
  ``Oracle'' (blue) could afford to use the identical relevance metric as that in ESU on $10^4$ behaviors, namely find ``the real top-100''.

  While GSU (orange) uses an ineffective and inconsistent coarse search.
  Among the top-100 returned by GSU
  (x axis), only 40 hit the real top-100 (y axis). This inconsistency (gray) indicates the potential improvement space left for TWIN.
  }
  \vspace{-0.4cm}
  \label{fig:inconsist}
\end{figure}

Recent years have witnessed a great many emerging studies on two-stage life-long behavior modeling, while their key difference lies in the GSU strategies that coarsely select target-relevant behaviors.
For example, SIM Hard simply selects behaviors from the same category as the target item,
while SIM Soft calculates the target-behavior relevance score from pre-trained item embeddings by the inner product and selects behaviors with the highest relevance \cite{pi2020search}.
ETA approximates the relevance score calculation using locality-sensitive hashing (LSH) and Hamming distance \cite{chen2021end}.
SDIM samples behaviors with the same hash signature as the target-behavior through multi-round hash collision \cite{cao2022sampling}, among others.
Despite being extensively studied, existing two-stage life-long behavior modeling algorithms still suffer from a crucial limitation:
the \textit{inconsistency} between GSU and ESU (shown in Fig \ref{fig:inconsist}\footnote{The Oracle is identically copied from the ESU of SIM-hard (trained from the top-100 behaviors retrieved by GSU),
except that it ranks $10^4$ rather than $10^2$ behaviors.
Experiments are conducted on a small demo dataset from Kuaishou, not feasible for normal datasets due to the extremely high cost of Oracle (details in Section \ref{sec:rq2}).
}). Specifically, the target-behavior relevance metric used in GSU is both coarse and inconsistent with the TA used in ESU.
As a result, GSU may probably miss relevant behaviors,
but retrieve ones considered irrelevant by ESU, wasting ESU's precious computational resources.
In such case, the TA in ESU, no matter how attention is allocated, mostly deviates from the real user interests and thus degrades the overall CTR prediction accuracy.

To address such \textit{inconsistency}, we propose TWIN: TWo-stage Interest Network for lifelong user behavior modeling, where
a Consistency-Preserved GSU (CP-GSU) adopts the \textit{identical} target-behavior relevance metric as the TA in ESU, making the two stages \textit{twins}.
To extend the expensive TA to CP-GSU,
TWIN breaks TA's key computational bottleneck, namely the linear projection of $10^4$ behaviors, by effective behavior feature split, simplified TA architecture and highly optimized online infrastructure.
1). Specifically, for the video inherent features of a behavior (e.g. video id, author, duration, topic) which are shared across users / behavior sequences,
we accelerate their projection by
efficient pre-computing \& caching strategies.
2). And for the the user-video cross features of a behavior (e.g. user's click timestamp, play time, rating), where caching is not applicable, we
simplify the TA architecture by compressing their projection into bias terms.
With optimized online infrastructure, we successfully extend the applicable sequence length of TA from $10^2$ in ESU to $10^4-10^5$ in CP-GSU.
The consistency between two stages, together with the
effective
TA-based relevance metric in CP-GSU, contributes to significant performance gain in CTR prediction.

Overall, we make the following contributions:
\begin{itemize}[leftmargin=*]
    \item In our proposed TWIN, CP-GSU precisely and consistently retrieves behaviors that are not only target-relevant, but also considered important by ESU, maximizing the retrieval effectiveness of behavior modeling.
    To the best of our knowledge, we are the first to successfully address the inconsistency in the two-stage life-long behavior modeling problem.

    \item We validate the effectiveness of TWIN through extensive offline experiments on Kuaishou's 46 billion scale industrial dataset and online A/B tests.
     We verify our validity through ablation studies and show that TWIN brings significant online benefits.

    \item We build efficient industrial infrastructure to apply TWIN on the real online RS. Effective pre-computing \& caching strategies are proposed to reduce the computational bottleneck of TWIN, i.e., the linear projection of behaviors in CP-GSU, by 99.3\% and to meet the low latency requirements of the online serving system. TWIN has now been deployed on the RS of Kuaishou, serving the main traffic of 346 million active users every day.
\end{itemize}

\section{Related Work} \label{sec:related_work}

\subsection{Click-Through-Rate (CTR) Prediction}
CTR prediction which aims to predict a user's personalized interests, is crucial for nowadays RSs. In early days, CTR models are shallow and mainly focus on exploiting feature interactions, such as factorization machines (FM) \cite{rendle2010factorization} and field-aware factorization machines (FFM) \cite{juan2016field}. With the success of deep learning, deep CTR models are extensively studied and become mainstream choices. For example, \citet{chen2016deep} and \citet{zhang2016deep} first apply deep models for CTR tasks. Wide\&Deep \cite{cheng2016wide} combines a wide linear model and a deep model, which takes the advantages of both memorization of feature interaction and generalization of deep architecture. DeepFM \cite{guo2017deepfm} and DCN \cite{wang2017deep, wang2021dcn} improve the wide part of Wide\&Deep to increase the feature interaction ability. xDeepFM \cite{lian2018xdeepfm} and AFM \cite{xiao2017attentional} further exploit convolution-like layers and attention mechanism to improve the deep part and boost model performance.

User behavior modeling, i.e.,
capturing a user's hidden interest from the summarization of one's historical behaviors, plays an important role in CTR prediction. Limited by computational resources, early algorithms are mostly in target-independent manners and thus can be efficiently pre-calculated offline \cite{covington2016deep,yu2016dynamic,song2016multi}. To better extract a user's interest in specific items, various TA mechanisms are adopted.
The most representative one is DIN \cite{zhou2018deep}, which represents the user interest by a weighted pooling over historical behaviors and the pooling weights are calculated from TA to emphasis target-relevant behaviors.
DIEN \cite{zhou2019deep} further introduces the temporal relationship of behaviors using ARGRU, an attention-based variant of classic GRU \cite{cho2014learning}. After that, several works try to improve DIN and DIEN from different aspects. DSIN \cite{feng2019deep} splits behaviors into multiple sessions and conducts self-attention inside each one to
emphasise intra-session relationships. MIND \cite{li2019multi} and DMIN \cite{xiao2020deep} argue that representing user interest in one single vector is not enough. MIND uses the Capsule network \cite{sabour2017dynamic} inside to represent user interests as multiple vectors while DMIN treats representations from different self-attention heads of Transformers \cite{vaswani2017attention} as different user interests. Akin to DMIN, BST \cite{chen2019behavior}, SASRec \cite{kang2018self} and BERT4Rec \cite{sun2019bert4rec} also use transformers to model user sequential behaviors, improving the model's performance and parallelism.

\begin{table}[t]
\small
\caption{Comparison of SOTA user interest models. The bottom part lists the two-stage models. Length refers to the max sequence length of user behaviors in original papers.}
\begin{tabular}{lllcc}
\toprule
\multirowcell{2}{Method} & \multirowcell{2}{Length} & \multirowcell{2}{GSU Stragegy} & \multirowcell{2}{End2End} &\multirowcell{2}{Consist}  \\
\\
\midrule
DIN \cite{zhou2018deep} & $\sim 10^3$ & N/A & N/A & N/A\\
DIEN \cite{zhou2019deep} & $\sim 10^2$ & N/A & N/A & N/A\\
MIMN \cite{pi2019practice} & $\sim 10^3$ & N/A & N/A & N/A\\
\midrule
UBR4CTR \cite{qin2020user} & $\sim 10^2$ & BM25 & \XSolidBrush& \XSolidBrush \\
SIM Hard \cite{pi2020search} & $\sim 10^3$ & Category Filter & \XSolidBrush& \XSolidBrush \\
SIM Soft \cite{pi2020search} & $\sim 10^3$ & Inner Product & \XSolidBrush& \XSolidBrush \\
{ETA \cite{chen2021end}} &{$\sim 10^3$} & LSH \& Hamming  & {\Checkmark}& {\XSolidBrush}\\
{SDIM \cite{cao2022sampling}} &{$\sim 10^3$} &Hash Collision & {\Checkmark}&{\XSolidBrush} \\
TWIN (ours) & $\sim 10^4$ & Target Attention & \Checkmark & \Checkmark\\
\bottomrule
\label{tab:comparison}
\end{tabular}
\end{table}

\subsection{Long-Term User Behavior Modeling}
As the effectiveness of TA and interest modeling have been confirmed in modern industrial RSs, researchers start to model increasing longer behaviors and try to dig out fine-grained interests from different user behavior stages. For example, \citet{liu2007framework} proposes to combine long-term and short-term interests in CTR prediction. To process long-term user behaviors up to length of $10^3$, MIMN \cite{pi2019practice} stores the user behaviors as a memory matrix at the user interest center (UIC) and update the memory when new user behavior comes. However, MIMN is hard to extend to longer behavior sequence and it generates the same memory matrix for different candidate items, carrying useless noise and impairing the TA. Recently, SIM \cite{pi2020search} and UBR4CTR \cite{qin2020user} introduce the two-stage cascaded framework to solve those challenges and achieved SOTA results on the CTR task at that time. SIM consists of a simple and fast GSU, which retrieves the top ``similar'' items for the target item from thousands of user behaviors, and an attentive ESU to perform TA. In the original SIM, there are two instances with different GSU designs. The GSU of SIM Hard retrieves related items by filtering items from the same category, while the GSU of SIM Soft uses the inner product with pre-trained item embeddings. UBR4CTR utilizes BM25 by regarding items and features as documents and terms.

Though the two-stage design takes a big step forward, the original GSU still faces high computational burden and has different retrieval metric with ESU, which brings the inconsistency problem. More recently, ETA \cite{chen2021end} uses locality-sensitive hash (LSH) to encode item embeddings from ESU and retrieve similar items from long-term behaviors via Hamming distance (HD) based on those hash signatures. SDIM \cite{cao2022sampling} samples behavior items with the same hash signature as the target item through multiround hash collision, whose ESU then linearly aggregates those sampled behavior items to obtain user interests.
It is positive that ETA and SDIM adopt End2End training, so that the embeddings of the two stages are the same.
Nevertheless, inconsistency still exists in the retrieval strategies of ETA and SDIM, specifically the network structure and parameters.
Because they just approximate the similarity or relevance computation of ESU in the GSU, which probably misses important behaviors and retrieves unrelated ones, misleading the CTR model.
In this paper, we propose to unify the way of similarity computation in GSU\&ESU by equally using TA in GSU, and maintain the end-to-end train method, i.e., CP-GSU, whose parameters are updated from ESU. Theoretically, GSU and ESU share the same architecture and parameters. So compared to ETA and SDIM, our model is consistent in two stages, has less chance to miss ``similar'' items in stage 1, and improves the performance significantly. We detail the differences of our model with others in Table \ref{tab:comparison}.

\begin{figure*}[t]
  \centering
  \hspace{-0.3cm}
  \includegraphics[width=0.93\textwidth]{picture/sim3_framework_compressed.pdf}
  \caption{TWIN in Kuaishou's CTR prediction system.
   Different from conventional two-stage behavior modeling algorithms,
   TWIN adopts identical target-behavior relevance metric in CP-GSU and ESU, including not only identical network architecture (shown in the left) but also identical parameter values (shown in the middle bottom). This is challenging since MHTA was designed with high computational cost and thus only applicable to ESU (with 100 behaviors), not to CP-GSU (with $10^4$ behaviors).
    We address this challenge by proposing: 1). efficient feature split and projection strategies that process item inherent features and user-item cross features in different manners (shown in the right bottom); 2). simplified target attention architecture that accelerates the efficiency of target attention through compressing cross features into bias terms
    (shown in the left).
}
  \vspace{-0.4cm}
  \label{fig:framework}
\end{figure*}

\begin{table}
\caption{Important Notations Used in Section 3}\label{notations}
\begin{center}
\begin{tabular}{ll|ll|ll}
   \toprule
		  $f$ & predictor		&	$\sigma$ &sigmoid 			&$\hat y$ & predicted CTR	\\
$\mathcal D$&dateset			&	$\mathbb R$ & real number set  	&$d$& feature dimension\\
$\ell$&loss&$\mathbf x$&feature vector		&	$y$& ground truth label\\
\midrule
\end{tabular}
\begin{tabular}{ll|ll}
 $E$ & embedding dictionary&$\mathbf x_\text{emb}$ &embedded feature \\
$\mathbf x_\text{hot}$& one/multi hot coding &$v$& vocabulary size	\\
$K$ & behavior features & $L$ & behavior length \\
$J$ & cross feature number &$a$ & head index  \\
$H$ & inherent feature dimension & $K_h$ & inherent features \\
$C$ & cross feature dimension &  $K_c$ & cross features\\
$\boldsymbol \beta$ & cross feature weight &$\boldsymbol \alpha$ & attention weight \\
\midrule
\end{tabular}\end{center}
\begin{tabular}{ll}
$d_k$ & dimension of projected inherent features  \\
$d_{out}$ & dimension of projection in original MHTA \\
$W^h, W^c, W^v, W^o$ & linear projection parameters \\
$\mathbf w_j^c$ & a diagonal block of projection parameter $W^c$ \\
\bottomrule
\end{tabular}
\end{table}

\section{TWIN in KuaiShou CTR Prediction} \label{sec:method}
At the beginning, we first review the general preliminaries of the CTR prediction problem
in Section \ref{sec:preliminaries}. Then we describe the model architecture of our CTR prediction system in Kuaishou in Section \ref{sec: architecture}. We further dig into details of our proposed
 \textit{consistency-preserved}
lifelong user behavior modeling module, named
TWo-stage Interest Network (TWIN),
in Section \ref{sec: TIM}. Finally, we introduce essential accelerating strategies which guarantee the successful online deployment of TWIN on the main traffic of Kuaishou in Section \ref{sec:deploy}. The notations used are summarized in Table \ref{notations}.

\subsection{Preliminaries}\label{sec:preliminaries}
The aim of CTR prediction is to predict the probability that a \textit{user} would $click$ an \textit{item} given specific \textit{contexts}.
Accurate CTR prediction not only lifts the user experience by serving preferred contents, but also benefits the business effectiveness of content producers and platforms by reaching interested audiences.
Consequently, CTR prediction has become the core component in various industrial RSs, especially short video recommendation platforms like Kuaishou.

CTR prediction is usually formulated as a binary classification problem, where the goal is to learn a predictor function $f:\mathbb R^d \rightarrow \mathbb R$ given a training dataset $\mathcal D =\{(\mathbf x_1,y_1),...,(\mathbf x_{|\mathcal D|}, y_{|\mathcal D|})\}$.
Specifically, $\mathbf{\mathbf x_i\in\mathbb R^d}$ is the feature vector of the $i$-th training sample (namely, the concatenation of the user, item and contexts features), and $y_i \in \{0,1\}$ is the ground truth label denoting whether the user clicks (1) the item or not (0).
The predicted CTR is calculated as:
\begin{equation}
\hat y_i = \sigma(f(\mathbf x_i)).
\end{equation}
$\sigma(\cdot)$ is the sigmoid function that scales the prediction of $f$ to $(0,1)$.

The model is trained by minimizing the negative log-likelihood:
\begin{equation}
    \ell(\mathcal D) = -\frac{1}{|\mathcal D|} \sum_{i=1}^{|\mathcal D|} y_i \log(\hat y_i) + (1-y_i)\log(1-\hat y_i).
\end{equation}
For conciseness, we omit the training sample index $i$ in the following sections when no confusion is caused.

\subsection{The Architecture of the CTR Prediction System in Kuaishou}\label{sec: architecture}
We now illustrate the architecture of our CTR prediction system at Kuaishou. Details are shown in Fig \ref{fig:framework}.
\subsubsection{The Embedding Layer}
At the bottom, out model starts from a feature embedding layer that transforms raw features of a training sample to embedding vectors.

Without loss of generality, we assume that all features are in the categorical form after essential pre-processing. For a feature $A$ with vocabulary size $v_A$, we first encode the categorical information into a one-hot / multi-hot code $\mathbf x_{\text{A},\text{hot}} \in \{0,1\}^{v_A}$. For example,

\textsf{WeekDay=Mon} ~~ ~~~~~$\Longrightarrow$ $\mathbf x_\text{WeekDay, hot} = [1,0,0,0,0,0,0]^\top$,

 \textsf{Topic=\{Funny, Pet\}}~~~~~~~$\Longrightarrow$ $\mathbf x_\text{Topic, hot} = [...,0,1,0,...,0,1,0...]^\top$.

\noindent Note that in most industrial  systems, the vocabulary size (especially that of user / author / video ids) can easily scale to hundreds of millions. Thus, a common strategy is to transform the extreme high dimensional hot codes to low dimensional embeddings,
\begin{equation}
\mathbf x_\text{A,emb} = E_A \mathbf x_\text{A,hot},
\end{equation}
where $E_A\in \mathbb R^{d_A \times v_A}$ is the embedding dictionary of $A$, and $d_A$ is the embedding dimension. In our system, we set the embedding dimension to $64$ for id features with large vocabulary, and to $8$ for others, such as video topic, video played timestamp.

In all the upper layers, we take the embedding vectors as input and thus omit the subscript ``$\text{emb}$'' for conciseness.
\subsubsection{The Deep Networks}
The overall architecture of our CTR prediction is shown in Fig \ref{fig:framework}.
The upper module, consisting of many stacked neural networks and ReLUs, acts as a mixer that learns the interaction between the outputs of
three intermediate modules:
\begin{itemize}[leftmargin=*]
\item \textbf{TWIN}, the proposed consistency-preserved life-long user behavior modeling module. TWIN extracts user interest through two cascading stages of behavior modeling sub-modules:
1). Consistency-Preserved General Search Unit (\textbf{CP-GSU}) which performs a coarse search for 100 most relevant behaviors from tens of thousands of long term historical behaviors; 2). Exact Search Unit (\textbf{ESU}) which adopts an attention mechanism over the 100 finalists of CP-GSU to capture the exact user interest.

Different from conventional algorithms that usually consist of a ``light'' GSU and a ``heavy'' ESU,
our proposed CP-GSU follows the identical relevance evaluation metric as that of ESU,
making the two cascading stages \textit{TWINS}. Consequently, CP-GSU consistently retrieves items that are considered important by ESU, maxmizing the behavior modeling effectiveness.
\item \textbf{Short-term behavior modeling} which extracts user interests from the 50 most recent behaviors. This module focuses on the user's short term interest in the latest few days, and acts as a strong complement to TIM.
\item \textbf{Others Task Modelings}. Besides behavior modeling, we also concatenate the outputs of various other task modelings, which model the user's gender, age, occupation, location, the video's duration, topic, popularity, quality, and contexts features such as the played date, timestamp, page position, etc.
\end{itemize}

\subsection{TWIN: TWo-stage Interest Network}\label{sec: TIM}
We name the proposed algorithm TWIN to highlight that CP-GSU follows the identical relevance evaluation metric as that of ESU. Note that this consistency is nontrivial because:
\begin{itemize}[leftmargin=*]
\item Effective behavior modeling algorithms are usually based on Multi-Head Target Attention (MHTA) \cite{vaswani2017attention}, which precisely captures user interest by emphasising target relevant behaviors.
Unfortunately, due to the high computational complexity,
the applicable behavior sequence length  of MHTA is mostly limited to a few hundreds.

\item To exhaustively capture user's long-term interest, CP-GSU should cover user behaviors in the last several months, which could easily reach tens of thousands. This sequence length is far beyond the capacity of conventional MHTA given the strict low latency requirements of online systems.
\end{itemize}

\noindent This section aims to answer this key question: how to improve the efficiency of MHTA so that we can
extend it from ESU to CP-GSU, or namely from a sequence length of hundreds to a sequence length of at least tens of thousands?

\subsubsection{Behavior Feature Splits and Linear Projection}
Following the standard notations of the MHTA \cite{vaswani2017attention}, we define the features of a length $L$ behavior sequence $[s_1,s_2,...,s_L]$ as matrix $K$, where each row denotes the features of one behavior.
In practice, the linear projection of $K$ in the attention score computation of MHTA is the key computational bottleneck that hinders the application of MHTA on extremely-long user behavior sequences. We thus propose the followings to reduce its complexity.

We first split the behavior features matrix $K$ into two parts,
\begin{equation}\label{eq:be_splits}
K \triangleq\left[
\begin{array}{cc}
K_{h} & K_c
\end{array}\right] \in \mathbb R^{L \times (H+C)},
\end{equation}
We define $K_h \in \mathbb R^{L \times H}$ as the \textit{inherent} features of behavior items (e.g. video id,  author, topic, duration) which are
independent of the specific user / behavior sequence,
and $K_c \in \mathbb R^{L \times C}$ as the user-item cross features (e.g. user click timestamp, user play time, clicked page position, user-video interactions).
This split allows high efficient computation of the following linear projection $K_hW^h$ and $K_cW^c$.

For the inherent features $K_h$, although the dimension $H$ is large (64 for each id feature), the linear projection is actually not costly. The inherent features of a specific item are shared across users / behavior sequences. With essential caching strategies, $K_h W^{h}$ could be efficiently ``calculated'' by a look up and gathering procedure. Details of online deployment will be introduced in Section \ref{sec:deploy}.

For the user-item cross features $K_c$, caching strategies are not applicable because:
1). Cross features describe the interaction details between a user and a video, thus not shared across users behavior sequences.
2). Each user watches a video for at most once. Namely, there is no duplicated computation in projecting cross features.
We thus reduce the computational cost by simplifying the linear projection weight.

  Given $J$ cross features, each with embedding dimension $8$ (since not id features with huge vocabulary size). We have $C = 8J$.
 We simplify the linear projection as follows,
\begin{equation}
K_c W^c \triangleq \left[
\begin{array}{ccc}
K_{c,1} \mathbf w_{1}^c,&...&,K_{c,J}\mathbf w_{J}^c
\end{array}\right]
,
\end{equation}
where $K_{c,j} \in \mathbb R^{L \times 8}$ is a column-wise slice of $K_c$ for the $j$-th cross feature, and $\mathbf w_{j}^c \in \mathbb R^{8}$ is its linear projection weight. Using this simplified projection, we compress each cross feature into one dimension, i.e., $K_c W^c \in \mathbb R^{L\times J}$. Note that
this simplified projection is
equivalent to restricting $W^c$ to a diagonal block matrix.

\subsubsection{Complexity Analysis} \label{sec:complexity}
In the conventional MHTA, the time complexity of linear projection of $K$, namely from dimension $L\times(H+C)$ to $L\times d_{out}$, is $O(L \times (H+C) \times d_{out})$.

While in our MHTA for TWIN, the item inherent features $K_hW^h$ is pre-computed and efficiently gathered in $O(L)$, which is independent of the dimension $H$. And the user-item cross features $K_cW^c$ is
reduced to low dimensional computation of $O(L\times C)$.

Since $C \ll H$, and $C \ll d_{out}$,
it is this theoretical acceleration that allows the consistent implementation of MHTA in both CP-GSU and ESU.

\subsubsection{Target Attention in TWIN}
Based on the linear projection of behaviors $K_hW^h$ and $K_cW^c$, we now define the target-behavior relevance metric that is used uniformly in both CP-GSU and ESU.

Without loss of generality, we assume that there has been no interaction between the user and the target item and denote the target item's inherent features as $\mathbf q \in \mathbb R ^{H}$.
With proper linear projection $W^q$, the relevance score $\boldsymbol \alpha \in \mathbb R^L$ between the target item and the historical behaviors is calculated as:
\begin{equation}\label{equ:alpha}
\boldsymbol{\alpha} = \frac{(K_h W^{h})(\mathbf q^\top W^{q})^\top}{\sqrt d_k}+ (K_c W^c)\boldsymbol{\beta},
\end{equation}
where $d_k$ is the dimension of projected query and key.
This relevance score is calculated by the inner product between the \textit{query}, i.e., the inherent features of the target, and the \textit{keys}, i.e., the inherent features of behaviors. Additionally, the cross features, since compressed to 1 dimension, serve as bias terms. We use
 $\boldsymbol \beta \in \mathbb R^J$ as the learnable parameter for the relative importance of cross features.

In CP-GSU, this relevance score $\boldsymbol \alpha$ is used to cut-off the $L= 10^4$ long-term historical behaviors to 100 most relevant ones.
And in ESU, we perform a weighted average pooling over the 100 finalists:
\begin{equation}
\text{Attention}(\mathbf q^\top W^q,K_hW^h,K_cW^c,KW^v)= \text{Softmax}(\boldsymbol \alpha)^\top K W^v,
\end{equation}
where
$W^v$ is a projection matrix. We slightly abuse the notation by setting $L =100$.
This projection $KW^v$ is only performed over 100 behaviors and thus can be conducted efficiently online. We do not need to split $K$ as we did when computing $\boldsymbol \alpha$ for $10^4$ behaviors.

To jointly attend to information from different representation subspaces, we adopt 4 heads in our MHTA. Thus the final output of TWIN is defined as
\begin{equation}
\begin{aligned}
&\text{TWIN} = \text{Concat}(\text{head}_1,...,\text{head}_4)W^o, \\
\text{head}_a &= \text{Attention}(\mathbf q^\top W_a^q,K_hW_a^h,K_cW_a^c,KW_a^v), a\in\{1,...,4\},
\end{aligned}
\end{equation}
$W^o$ is a projection that learns relative importance between heads.
\subsection{System Deployment}\label{sec:deploy}
\begin{figure}[t]
  \centering
  \hspace{-0.3cm}
  \includegraphics[width=0.46\textwidth]{picture/sim3_system.pdf}
  \caption{The Deployment of TWIN in Online CTR Prediction System.
  We propose essential precomputing \& caching strategies to reduce the key computational bottleneck, the linear projection of $10^4$ behaviors' inherent features.
  With proper frequency control, we cut off tailed videos and limit the size of candidate video pool to 8 billion.
As a result, the \textit{inherent feature projector} can cyclically refresh the linear projection of all candidate videos every 15 minutes, minimizing the accuracy loss from caching. And the \textit{embedding server} which stores the projection of 8 billion candidate videos, can cover 97\% of the online requests and achieve satisfactory effectiveness given limited computational resources.
  }
  \vspace{-0.4cm}
  \label{fig:cbss}
\end{figure}

We deploy TWIN on the ranking system of Kuaishou, serving the main traffic of 346 million daily active users.
In this section, we introduce our hands-on experience in the deployment. Details of our system architecture is shown in Fig \ref{fig:cbss}.
\subsubsection{Training System}
Our TWIN module is trained jointly with the whole CTR prediction model on Kuaishou's large-scale distributed nearline learning system.

Every day, hundreds of millions of users visit Kuaishou, watch and interact with short videos, and leave 46 billion watch and interaction logs per day. Each log is collected, preprocessed in realtime and used for model training in less than 8 minutes. This nearline training system incrementally updates the model parameters using the latest knowledge from user-video interactions that take place in less than 8 minutes ago.

In addition,
our message queue system continuously synchronizes the latest parameter values from the training system to the
offline inferring and online serving systems, once every 5 minutes. This synchronization ensures that the online CTR prediction service is always based on the update-to-date model.

 \subsubsection{Offline Inferring}\label{sec:delay in cach}
The offline inferring system aims to accelerate the online serving by providing a lookup service.
When receiving lookup keys, a batch of video ids,
this service returns the lookup values, the corresponding projected inherent features in concatenation, i.e. $K_hW^h_a$ for all heads $a \in \{1,...,4\}$.

Specifically, the offline inferring system consists of two parts. 1).
\textit{An inherent feature projector}, which cyclically pre-computes the linear projection of inherent features using the latest embeddings and TWIN parameters $W^h_a$ synchronized from the training system. With proper frequency control, this projector can refresh the projected inherent features of an
8 billion scale candidate video pool every 15 minutes, minimizing the accuracy loss from caching
2). \textit{An embedding server}, which stores results of the inherent feature projector into a key-value structure and provides the aforementioned key lookup service. By cutting off tailed videos, the 8 billion keys can cover 97\% of online request, balancing efficiency and effectiveness.

\subsubsection{Online Serving}
Once a request is received, the online serving system queries the offline inferring system for the projected inherent features $K_hW^h_a$, and calculates the other parts of Eq \ref{equ:alpha} in realtime for the user-item relevance score $\boldsymbol \alpha$.
We then sort the behaviors by the descending order of the scale in $\boldsymbol \alpha$, and select the top-100 behaviors for ESU. This design reduces the computational bottleneck of TWIN, i.e., linear projection of $K_h$ with $10^4$ rows, by 99.3\% in practice.

Note that ESU with only 100 behaviors is light enough for all calculations to be conducted in realtime using up-to-date parameters synchronized from the training system. As the result, the $K_hW^h$ calculated by ESU is slightly more up-to-date than that in CP-GSU, which further lifts the performance of our TA mechanism.

With accelerated design, TWIN is successfully deployed on the ranking system of Kuaishou, serving the main traffic of 346 million  active users, with a peak request of 30 million videos per second.

\section{EXPERIMENT}
In this section, we detail the offline and online experiments on real industrial data to evaluate our proposed method with the purpose of answering the following four research questions (RQs).
\begin{itemize}[leftmargin=*,partopsep=0pt,topsep=0pt]
    \setlength{\itemsep}{0pt}
    \setlength{\parsep}{0pt}
    \setlength{\parskip}{0pt}
\item \textbf{RQ1:} How does TWIN perform in the offline evaluations compared to other SOTAs in lifelong user behavior modeling?
\item \textbf{RQ2:} How \textit{consistent} can TWIN achieve, compared to other SOTAs? Or namely, why TWIN works?
\item \textbf{RQ3:} How does the effectiveness of TWIN change as the length of the user behavior sequence grows?
\item \textbf{RQ4:}
What are the effects of key components and different implementations in the proposed method?
\item \textbf{RQ5:} How does TWIN perform in real online RS?
\end{itemize}

\subsection{\textbf{Dataset}}
\begin{table}[t]
\small
\caption{Statistics of the industrial dataset  constructed on the daily collected user logs from Kuaishou.}
\begin{tabular}{llc}
\toprule
Data & Field & Size \\
\midrule
\multirow{4}{*}{Daily Log Info} & Users & 345.5 million \\
& Videos & 45.1 million \\
& Samples & 46.2 billion \\
& Average User Actions & 133.7 / day \\
\midrule
\multirow{2}{*}{Historical Behaviors} & Average User Behaviors & 14.5 thousand \\
& Max User Behaviors & 100 thousand \\
\bottomrule
\label{tab:dataset}
\end{tabular}
\end{table}
To evaluate TWIN in real-world situation for lifelong user behavior modeling, we need a large scale CTR prediction dataset with rich user historical behaviors
that can ideally scale up to tens of thousands behaviors per user.
Unfortunately, existing public datasets are either relatively small or lack of sufficient user historical behaviors.
For example, in the widely used Amazon dataset \cite{mcauley2015image, he2017translation, kang2018self} each user has less than ten historical behaviors
 on average. In the Taobao dataset \cite{pi2019practice, zhu2018learning, zhou2018deep, zhou2019deep, pi2020search} the average sequence length of user behaviors is at most 500. We thus collect an industrial dataset from Kuaishou, one of the top short-video sharing platforms in China.

We construct samples from daily user logs, with user's clickthroughs as labels.
As shown in Table \ref{tab:dataset}, the size of daily active users on Kuaishou is around 346 million.
Every day 45 million short videos are posted and these videos are played 46 billion times in total. On average, each user watches 133.7 short videos per day.
To utilize rich behavior information, we collect full user historical behaviors from older logs back to months ago. On average, each user watched 14,500 videos in the past six months, which provides models a testbed with fertile user historical behaviors to learn from. We cutoff the maximum user behavior sequence length to 100,000, which is about the annual total number of views for heavy users.

\subsection{\textbf{Baselines}}
To demonstrate effectiveness, we compare TWIN with
the following
SOTA lifelong user behaviors modeling algorithms.
\begin{itemize}[leftmargin=*,partopsep=0pt,topsep=0pt]
\setlength{\itemsep}{0pt}
\setlength{\parsep}{0pt}
\setlength{\parskip}{0pt}
\item \textbf{Avg-Pooling}, the average pooling of user lifelong behaviors.
\item \textbf{DIN}~\cite{zhou2018deep}, the most widely adopted approach for short-term behavior modeling, which leverages TA for target-specific interests.
\item \textbf{SIM Hard}~\cite{pi2020search}. GSU selects behaviors from the same category with the target item and ESU follows the TA in DIN. In our scenarios, the total number of the video categories is 37.
\item \textbf{ETA } \cite{chen2021end}.
Locality-sensitive hash (LSH) is used to generate a hash signature for the target video and behaviors.
Then GSU uses Hamming distance as the target-behavior relevance metric.

\item \textbf{SDIM} \cite{cao2022sampling}. GSU selects behaviors with the same hash signature as the target video through multi-round hash collision. In the original paper, ESU linearly aggregates sampled behaviors from multiple rounds to obtain user interests. In our experiments, a more powerful TA is adopted in ESU for fair comparison.

\item \textbf{SIM Cluster}. Since ``category'' requires expensive human annotations and is usually unreliable in short video scenarios, we implement SIM Cluster as an improved variant of SIM Hard.
We group videos into 1,000 clusters based on pre-trained embeddings. GSU retrieves behaviors from the same cluster as the target item.

\item \textbf{SIM Cluster+} is a refinement of SIM Cluster, where the number of clusters is expanded from 1,000 to 10,000.
\item \textbf{SIM Soft}~\cite{pi2020search}. GSU uses inner product score of videos' pre-trained embeddings to retrieve relevant behaviors. Inner product is a more refined retrieval method than Hamming distance and hash collision, but with higher computational cost.

\end{itemize}
 \noindent
In summary, ETA and SDIM adopt end-to-end training methods, but use rough retrieval methods to avoid high-complexity calculations. SIM Cluster (+), and SIM Soft use refined retrieval methods, but the compromise is that they have to use pre-trained embeddings and generate offline inverted index in advance.
Note that SIM Hard and SIM Soft are simultaneously proposed \cite{pi2020search}, but SIM Soft has not yet been defeated by the follow-up work ETA and SDIM ~\cite{chen2021end, cao2022sampling}.
We do not compare with UBR4CTR \cite{qin2020user} because its iterative training is not suitable for our streaming scenario. Furthermore, UBR4CTR is confirmed performing worse than SIM Hard and ETA \cite{chen2021end}.

\subsection{\textbf{Experimental Setting}}
We use samples in 23 consecutive hours of a day as training data and ones in the following hour for test. We evaluate all algorithms in 5 consecutive days and report the averaged performance over days.
For offline evaluation, we use two widely adopted metrics: AUC and GAUC. AUC signifies the probability that a positive sample's score is higher than that of a negative one, reflecting a model's ranking ability. GAUC performs a weighted average over all user's AUC, and the weights are set to the number of samples of this user. GAUC eliminates the bias among users and evaluates model performance at a finer and fair granularity.

For two-stage models, we use the recent 10,000 behaviors as input of GSU and retrieve 100 behaviors for TA in ESU.
For DIN, we use the most recent 100 behaviors due to its bottleneck in processing long sequences.
Though CP-GSU of TWIN uses four heads in the attention score computation, we recursively traverses the top ranked items by the four heads until it collects 100 unique behaviors.
For all models, the embedding layer uses the AdaGrad optimizer and the learning rate is 0.05. DNN parameters are updated by Adam with learning rate 5.0e-06. The batch size is set as 8192.

\begin{table}[]
\normalsize
\caption{Offline comparison with SOTAs (RQ1).
We report the \textit{mean} and standard deviation (\textit{std}) over 5 consecutive days. The best and second-best results are highlighted in bold and underlined respectively . Note that the improvement of TWIN over the best compared model in AUC is $0.29\%$ and that in GAUC is $0.51\%$. These improvements are much larger than $0.05\%$, a value enough to bring online benefits.}
\begin{tabular}{lcc}
\toprule
Method & AUC (\textit{mean} \footnotesize{$\pm$ \textit{std}} ) $\uparrow$  & GAUC (\textit{mean} \footnotesize{$\pm$ \textit{std}} ) $\uparrow$  \\
\midrule
Avg-Pooling  & 0.7855 \footnotesize{$\pm$ 0.00023} & 0.7168 \footnotesize{$\pm$ 0.00019} \\
DIN  & 0.7873 \footnotesize{$\pm$ 0.00014} & 0.7191 \footnotesize{$\pm$ 0.00012} \\
\hline
SIM Hard  & 0.7901 \footnotesize{$\pm$ 0.00016} & 0.7224 \footnotesize{$\pm$ 0.00021} \\
ETA   & 0.7910 \footnotesize{$\pm$ 0.00004} & 0.7243 \footnotesize{$\pm$ 0.00011} \\
SIM Cluster  & 0.7915 \footnotesize{$\pm$ 0.00017} & 0.7253 \footnotesize{$\pm$ 0.00018} \\
SDIM  & 0.7919 \footnotesize{$\pm$ 0.00009} & 0.7267 \footnotesize{$\pm$ 0.00006} \\
SIM Cluster+  & 0.7927 \footnotesize{$\pm$ 0.00009} & 0.7275 \footnotesize{$\pm$ 0.00011} \\
SIM Soft  & \underline{0.7939 \footnotesize{$\pm$ 0.00014}} & \underline{0.7299 \footnotesize{$\pm$ 0.00013}} \\
\midrule
TWIN  & \textbf{0.7962 \footnotesize{$\pm$ 0.00008}} & \textbf{0.7336 \footnotesize{$\pm$ 0.00011}} \\
Improvement & +0.29\% & +0.51\% \\
\bottomrule
\label{tab:overall}
\end{tabular}
 \vspace{-15px}
\end{table}
\subsection{Overall Performance (RQ1)}
Table~\ref{tab:overall} shows the performance of all models. Note that due to the large amount of users and samples in our dataset, an improvement of $0.05\%$ in AUC and GAUC in the offline evaluation is significant enough to bring online gains for the business.

\textbf{First, TWIN significantly outperforms all baselines, especially two-stage SOTAs with inconsistent GSU.} This validates the key advantage of TWIN in life-long behavior modeling, i.e., the \textit{powerful} and \textit{consistent} TA in CP-GSU.
Specifically, CP-GSU precisely retrieves behaviors that are considered highly relevant by ESU, saving ESU's precious computational resources for the most important user information.
While in others, the ineffective and inconsistent GSU may miss important behaviors but introduce noisy ones, degrading TA performance.
In addition, the gain from Avg-pooling to DIN shows the ability of TA in retrieving effective information. And the gain from DIN to other two-stage SOTAs verifies the necessity of modeling long behaviors. These two together support our motivation: extending TA to long sequences.

\textbf{Second, only end-to-end training is not enough.}
We observe that TWIN apparently outperforms ETA and SDIM,
two strong baselines whose embeddings in GSU are also trained in end-to-end manner.
Specially, ETA uses LSH \& Hamming distance and SDIM uses multiround hash collision. Both GSU strategies are less precise compared to TA and inconsistent with
the target-behavior relevance metric used in their ESUs.
While CP-GSU in TWIN is not only trained end-to-end, but also consistent with TA in ESU.
This shows that a precise relevance metric is crucial to GSU, validating our advantage over the existing end-to-end algorithms.

 \textbf{Third, modeling lifelong behaviors at a finer granularity is effective.}
We compare the variants of SIM: SIM Hard with 37 categories, SIM Cluster(+) with 1,000
 / 10,000 clusters and SIM Soft where the target-behavior relevance score is calculated individually for each behavior. We observe consistent performance improvement as every finer granularity retrieval method is used in GSU.
This is because GSU retrieves behaviors more accurately when it can capture the relevance score between videos more finely.
From this perspective, we further
attribute our advantage over SIM Soft to the fact that TWIN adopts more accurate relevance metric.
Note that the poor performance of SIM Hard is partly because the video category used for retrieval is sometimes not reliable in short video RS, which is a little surprising for engineers from e-shop industry.

\begin{figure}[t]
  \centering
\hspace{-0.3cm}
    \includegraphics[width=0.39\textwidth]{picture/croped_fig7.pdf}

  \caption{Consistency between GSU \& ESU (RQ2).
Oracle uses ESU's relevance metric on $10^4$ behaviors to get ``the real top-100''. Among the 100 return by GSU of SIM Hard, only 40 hits. TWIN lifts the hit number to 94. Theoretically, TWIN could achieve 100\% hit rate since we adopt consistent GSU. However, our practical performance is limited by the deployment constrain, the 15 minutes refresh delay in caching.

  }

\label{fig:consis}
\end{figure}

\subsection{Consistency Analysis (RQ2)}\label{sec:rq2}
As claimed, our superior behavior modeling ability comes from the consistent relevance metrics in CP-GSU and ESU.
But how consistent we really achieve (Figure~\ref{fig:consis})?

For each well-trained two-stage model, we reuse the parameters of its ESU as its Oracle
to retrieve ``the real top-100'' from 10,000 behaviors.
In other words, these real top-100 are the ground-truth that ESU considers to be really important.
We then traverse the GSU output size of all compared algorithms
from 10 to $10^4$ to examine how many outputs hit the real top-100.
Note that each compared algorithm has its own Oracle and top-100. But we only plot one Oracle curve since all Oracles perfectly hit ground truth.

SIM Soft achieves an improvement in retrieval consistency benefiting from a more precise retrieval strategy in GSU.
Further, TWIN achieves 94 hits when returning 100 behaviors, which validates our advantage in preserving the consistency between two stages.
Note that this is the most noteworthy value, since 100 is the upper bound for ESU input, given the constraints of inference time and computational complexity of TA.
We did not reach our theoretical 100\% consistency in practice due to refresh delay in caching, as described in Section \ref{sec:delay in cach}.
In addition, we show some case studies of retrieval consistency in the Appendix.

Based on the above results, we speculate that CP-GSU has a stronger ability to match user interests with the target video than conventional GSUs. This is attributed to the consistency. Via sharing the same structure and parameters with ESU, CP-GSU is able to accurately judge and retrieve videos with similar content.
In addition, as the CP-GSU parameters are updated in real time, the model can capture the user's dynamically evolving interests.

\subsection{Effects of Behavior Length (RQ3)}
\begin{figure}[t]
  \centering
\hspace{-0.3cm}
  \subfigure{\label{fig:length-auc}
    \includegraphics[width=0.23\textwidth]{picture/hyperpara_wuauc_raw.pdf}}
    \subfigure{\label{fig:length-wuauc}
    \includegraphics[width=0.23\textwidth]{picture/hyperpara_wuauc_gain.pdf}}

  \caption{

  Performance when GSU Takes Different Behavior Sequence Lengths (RQ3).
  Left: GAUC values.
  Right: Relative GAUC improvements over length = 10,000.

  As the sequence length increases, all model's performance improves and the performance gap between TWIN and baselines increases.}
  \vspace{-0.1cm}
\label{fig:length}
\end{figure}
We aim to test the effectiveness of TWIN under different behavior sequence lengths and to further tap the potential of TWIN.
Note that only input sequence lengths of GSUs
are changed and the output length is kept 100. Results are shown in Figure~\ref{fig:length}.

We observe that 1). TWIN performs the best consistently, and 2) the performance gap between TWIN and other methods becomes larger as the sequence length increases.
This indicates that TWIN has a better effectiveness in modeling extremely long sequences.

\subsection{Ablation Study (RQ4)}\label{sec:ablation}

\begin{figure}[t]
  \centering
  \hspace{-0.3cm}
  \subfigure{\label{fig:length-auc}
    \includegraphics[width=0.223\textwidth]{picture/ablation_parameter_wuauc.pdf}}
   \hspace{-0.3cm}
    \subfigure{\label{fig:length-wuauc}
    \includegraphics[width=0.26\textwidth]{picture/ablation_cache_wuauc.pdf}}
  \vspace{-0.4cm}
  \caption{Effects of key Components in TWIN (RQ4).
  Left: \textit{TWIN w/o Para-Cons} (consistent network structure + inconsistent parameters) outperforms SIM Soft (both inconsistent), but performs worse than TWIN (both consistent).
  Right: directly removing the user-item cross features (\textit{TWIN w/o Bias}) saves little computation but leads to a significant drop in GAUC compared to TWIN.
  Compared to TWIN with the original MHTA (\textit{TWIN w/ Raw MHTA}), compressing the item inherent features to biases for efficiency (TWIN) hardly harms the performance but greatly speeds up inference time.}
  \vspace{-0.1cm}
\label{fig:ablation}
\end{figure}
We conduct ablation studies by applying different operations to TWIN to evaluate the contribution of our key model designs: 1) the consistency between two stages, and 2) efficient MHTA.

TWIN preserves the consistency between two stages in two folds: the network structure and parameters. To study the benefits brought by each, we implement a variant named \textit{TWIN w/o Para-Con}, that doesn't preserve the parameter consistency.
Specially, we first train an auxiliary model \textit{TWIN-aux} which uses the identical network structure and training data with TWIN but is trained separately. We then synchronize the GSU parameters from \textit{TWIN-aux} to \textit{TWIN w/o Para-Con}. This is to ensure that \textit{TWIN w/o Para-Con} is still updated in real time and that the gap between TWIN and \textit{TWIN w/o Para-Con} are all caused by parameter inconsistency.

As shown in Figure~\ref{fig:ablation} (left), \textit{TWIN w/o Para-Con} performs significantly better than SIM Soft (inconsistent in both structure and parameters) but slightly worse than TWIN.
It indicates that both structure consistency and parameter consistency are beneficial, but network structure consistency contributes more.

To compute MHTA efficiently for the industrial deployment, we split user behavior features and compress each user-item cross feature into a one-dimensional bias term. To study the impact of such modification and the benefits of preserving user-item cross features in the attention computation, we implement two variants and compare their performances and inference time:
TWIN with the original MHTA where a straightforward linearly projection $KW$ is used and no feature split is conducted, and TWIN without using the user-item cross features in MHTA, which are respectively abbreviated as \textit{TWIN w/ Raw MHTA} and \textit{TWIN w/o Bias}.

As shown in Figure~\ref{fig:ablation} (right), TWIN beats \textit{TWIN w/o Bias} significantly and performs almost the same as \textit{TWIN w/ Raw MHTA}, validating that our proposed modification to MHTA hardly compromises the performance.
Regarding to the computational cost, as caching is inapplicable for \textit{TWIN w/ Raw MHTA} when the user-item cross features are used in the linear projection of $K$ (details in Section~\ref{sec:complexity}), the inference time of \textit{TWIN w/ Raw MHTA} significantly increases. In contrast, removing the user-item cross features (TWIN w/o Bias) does not save much computation, yet harms performance.

\begin{table}[t]
\small
\centering
\caption{The relative Watch Time improvement of TWIN in online A/B test compared with SIM Hard and SIM Soft (RQ5).
In Kuaishou's
scenario, 0.1\% increase is a significant improvement that brings great business effectiveness.
}
\label{tab::abtest}
\vspace{-0.3cm}
\setlength{\tabcolsep}{0.9mm}{\begin{tabular}{lcccc}
\toprule
\textbf{Scenarios}
& \textbf{Featured-Video Tab}
& \textbf{Discovery Tab}
& \textbf{Slide Tab} \\
\midrule
\textbf{v.s. SIM Hard}
& +4.893\% & +3.712\% & +6.249\% \\
\textbf{v.s. SIM Soft}
& +2.778\% & +1.374\% & +2.705\% \\
\bottomrule
\end{tabular}
}
\vspace{-0.3cm}
\end{table}

\subsection{Online Result (RQ5)}

To evaluate the online performance of TWIN, we conduct strict online A/B tests on Kuaishou's short-video recommendation platform. Table~\ref{tab::abtest} compares the performance of TWIN with SIM Hard and SIM Soft under three representative business scenarios in Kuaishou (\textbf{Featured-Video} Tab, \textbf{Discovery} Tab, and \textbf{Slide} Tab).
Different from
e-commerce, where the commonly used online evaluation metrics are
CTR and GMV, the short-video recommendation scenarios
usually use Watch Time, which measures the total amount of time users spend on viewing the videos. As shown, TWIN remarkably outperforms SIM Hard and SIM Soft in all scenarios. As 0.1\% increase in Watch Time is considered as an effective improvement in Kuaishou, TWIN achieves significant business gain.

\vspace{-0.1cm}
\section{Conclusion}
To address the inconsistency issue of conventional life-long behavior modeling algorithms,
we proposed a consistency-preserved TWo-stage Interest Model, which successfully extended the effective but computational expensive MHTA from ESU to CP-GSU, or namely from a sequence length of $100$ to length $10^4 - 10^5$.
Specifically, we designed novel MHTA mechanism as well as highly efficient infrastructure, including behavior features splits \& compression, pre-computing \& caching, online training \& parameter synchronization. We accelerated the computational bottleneck by 99.3\%, which contributed to the successful deployment of TWIN on Kuaishou, serving the main traffic of hundreds of millions of active users.
The consistency between two stages, together with the effective TA-based relevance metric in CP-GSU, maximized the retrieval effectiveness of behavior modeling and significantly lifted the performance of CTR prediction.
To the best of our knowledge, TWIN is the first to achieve consistency in the two-stage life-long behavior modeling problem.

\clearpage

\bibliographystyle{ACM-Reference-Format}


Spatially resolved stellar population parameters in the BCGs of two fossil groups

Abstract: We report the results of Gemini/GMOS long-slit spectroscopic observations along the major and minor axes of the central galaxies in two fossil groups, SDSS J073422.21+265133.9 and SDSS J075828.11+374711.8 (the NGC 2484 group). Spatially resolved kinematics and stellar population parameters (ages, metallicities and $\alpha$-element abundance ratios) derived using ~20 Lick indices are presented. Despite remarkable similarities in their morphologies, photometric properties (luminosity and colour) and kinematics, the two galaxies exhibit significantly different stellar population parameters.SDSS J073422.21+265133.9 exhibits a strong metallicity gradient (Delta[Z/H]/Delta R ~ -0.4) all the way into the centre of the galaxy. It also exhibits an age profile that suggest a relatively recent, centrally concentrated burst of star formation superimposed on an older, more spatially extended population. NGC 2484, a well known X-ray AGN, exhibits a flat core-like structure in its metallicity gradient, but no detectable age gradient. The alpha-element abundance ratio ([E/Fe]) profiles of the two galaxies are also significantly different. SDSS J073422.21+265133.9 exhibits a slightly positive gradient (Delta [E/H]/Delta R ~ 0.1), perhaps again suggesting a more recent central burst of star formation, while NGC 2484 shows a negative gradient (Delta [E/H]/Delta R ~-0.1), indicating that star formation may have happened "inside out".Our analysis of these two galaxies of similar mass, morphology and kinematics therefore suggests two different mechanisms to have been in action during their formation. Consequently, we conclude that the central galaxies of fossil groups can not be considered a homogeneous group with regard to their formation processes or star formation histories.

\section{Introduction}
Fossil galaxy groups are defined as X-ray luminous
($L_{X,\,\textrm{\scriptsize bol}} > 0.5 \times 10^{42}$ $h_{50}^{-2}$
erg~s$^{-1}$) galaxy aggregates with a greater than 2 magnitude gap
between the brightest and second brightest galaxies within half the
virial radius \citep{jones03}. The commonly stated paradigm for the
formation of these systems is that the large magnitude gap is
generated by dynamical friction acting on the massive $L^{*}$ galaxies
near the centre of the group, which causes them to spiral inwards and
merge with the central galaxy. For the magnitude gap to be sustained,
this scenario requires the group undergoes no significant recent
mergers or infall events. According to this paradigm, fossil groups
therefore represent highly evolved, but otherwise undisturbed,
examples of galaxy groups that formed early in the history of the
universe. Hence the study of fossil groups has become of great
interest, as this scenario suggests that they represent the
undisturbed end-product of galaxy group evolution. Contrary to this
hierarchical merging scenario it was also suggested that fossil groups
could simply be \emph {failed groups} that formed with a top-heavy
luminosity function absent of $L^{*}$ galaxies (Mulchaey \& Zabludoff
1999). As such they represent an important benchmark against which
other systems can be compared.

Although fossil groups as whole systems are believed to have collapsed
early and have assembled most of their virial masses at higher
redshifts, in comparison with non-fossil groups
\citep{donghia05,dariush07}, first-ranked galaxies in fossil groups
may have merged {\it later} than non-fossil bright central galaxies
\citep{diazgimenez08}. The question to which this work is addressed is
then: does the formation process of fossil groups create significant
differences in the stellar population gradients of their first-ranked
galaxies when compared to non-fossil bright cluster galaxies
(hereafter BCGs)?

There are relatively few studies on the stellar population gradients
of BCGs, and for fossil groups, only one study has been carried-out so
far, concluding that first-ranked galaxies in fossil groups show
comparatively flat metallicity gradients indicative of a major merger
origin \citep{eigenthaler13}. \citet{loubser} presented a systematic
study of the relations between the stellar population gradients of 24
BCGs in non-fossil clusters and properties of the cluster where they
reside (richness, mass, etc). This study may serve as a benchmark
against which results on the properties of first-ranked galaxies of
fossil groups can be compared. Other smaller samples which focused on
the detailed study of stellar populations of BCGs are
\citet{brough07,spolaor,gorgas90,fisher95,sanchezblazquez06,mehlert,carter,davidgegrinder95}.
Studies of BCGs in non-fossil groups have shown that these galaxies
possess central stellar populations that are generally old, of super
solar metallicity and moderately $\alpha$-element enhanced, i.e.  they
exhibit similar ages and metallicities as non-BCGs of the same mass.
However, some studies hint towards higher [E/Fe] values in BCGs
\citep{linden07,loubser09}.  BCGs exhibit fairly strong negative
metallicity gradients, and shallow age gradients, both positive and
negative. Where available $\alpha$-element abundance ratios are shown
to be relatively shallow \citep{loubser}.  However, it is not simple
to infer a consistent formation history for BCGs that can at the same
time explain the gradients in the three parameters, age, metallicity,
and abundances. In any case, what is evident is that the dispersions
in the stellar population parameters are high, indicating a number of
possible formation scenarios for BCGs that depend on the details of
the merging history for each galaxy.

Given our small sample, a full statistical comparison of fossil to
non-fossil BCGs is not possible in this paper, but must wait until a
larger sample of first-ranked fossil group galaxies have been studied
at the same level of detail. In this work we therefore simply continue
the accumulation of stellar population parameters age, [Fe/H], [E/Fe]
(a proxy for [$\alpha$/Fe] \citep{proctorsansom02}), and [Z/H] for
BCGs in fossil groups.

The paper is organised as follows. In Section \ref{observations} the
sample selection, observations and data reduction are described.
Section \ref{kinematicsstellarpopulations} presents galaxy kinematics
and stellar population parameters.  Section \ref{discussion} discusses
the obtained results while Sect.\ \ref{conclusions} shows our
conclusions.

\section{Sample  selection,  observations  and  data  reduction}
\label{observations}

We report on Gemini GMOS spectroscopic data for the BCGs of the two
fossil groups SDSS J073422.21+265144.9 and SDSS J075828.10+374711.8
(hereafter SDSS0734 and SDSS0758).  The sample was selected from the
list of fossil groups in \citet{diazgimenez08}, originally identified
in the Sloan Digital Sky Survey (SDSS) DR6 \citep{adelman08}.
\citet{diazgimenez08} used a Friends-of-Friends analysis to find
groups within the SDSS with an over-density contrast of 200, more than
10 spectroscopically confirmed members, masses greater than
5$\times$10$^{13}$ $h^{−1}$ M$_{\odot}$ and redshifts lower than 0.1.
Using these criteria, \citet{diazgimenez08} identified five groups
with $\Delta$m$_{12}$ values larger than 2.0. We had originally
intended to observe all five bona-fide fossil groups from
\citet{diazgimenez08}, however, given the time allocation constraints,
we obtained data for only two of them (galaxies II and IV in
\citealt{diazgimenez08}).  The two galaxies possess nearly identical
photometric properties and reside in groups with very similar
properties. Their $g$ band magnitudes, $g-r$ colours, sizes,
minor/major axis ratios ($b/a$), and redshifts are given in Table
\ref{photometricdata}. The $\Delta$m$_{12}$ values, velocity
dispersions, number of spectroscopically confirmed members within the
virial radius and richness of the groups in which these BCGs reside
are also given in Table \ref{photometricdata}. Magnitudes, colours and
redshifts were taken from the SDSS DR9.  The absolute $g$ band
magnitude was taken from the SDSS DR9 {\small photoz} table. For
effective radii, we considered the isophotal radius at $\mu_{K}$=20
mag arcsec$^{-2}$ from the 2MASS extended objects catalogue
\citep{2MASS} to be a good estimate (see also
\citealt{proctor08}). Based on the photometric similarities of both
galaxies and the groups in which they resdie, the selected targets are
ideal for studying the uniformity of the formation scenario for fossil
galaxy groups.  Hence we carried out spectroscopic observations along
the major and minor axes of these two fossil BCGs to study their
spatially resolved stellar population parameters.

\begin{table}
\caption{Overall properties of the observed fossil BCGs and the groups in which they reside.}
\begin{center}
\begin{tabular}{ l  . . }
\hline
BCGs                       &            \multicolumn{1}{c}{SDSS0734}            &             \multicolumn{1}{c}{SDSS0758}             \\
\hline
$\alpha_{\rm J2000}$        &  \multicolumn{1}{c}{$07^{\rm h}34^{\rm m}22\fs2$}  &   \multicolumn{1}{c}{$07^{\rm h}58^{\rm m}28\fs1$}   \\
$\delta_{\rm J2000}$        &  \multicolumn{1}{c}{$+26\degr51\arcmin44\farcs9$}  &   \multicolumn{1}{c}{$+37\degr47\arcmin11\farcs8$}   \\
$g$ [mag]                   &                        15.35                       &                         13.87                        \\
$M_g$ [mag]                 &                       -22.67                       &                        -22.58                        \\
$g-r$ [mag]                 &                         1.00                       &                          0.89                        \\
$r_{\rm eff}$ [arcsec]      &                        15.2                        &                         30.2                         \\
$r_{\rm eff}$ [kpc]         &                        24.1                        &                         25.7                         \\
$b/a$                       &                         0.80                       &                          0.78                        \\
$z$                         &                         0.0796                     &                          0.0408                      \\
\hline
Groups                        &            \multicolumn{1}{c}{SDSS0734}            &             \multicolumn{1}{c}{SDSS0758}             \\
\hline
$\Delta$m$_{12}$ [mag]                   &   2.01                     &    2.64        \\
$\sigma$ [km s$^{-1}$]                     &   551                     &     563       \\
Members                     &   16                     &      38      \\
Richness                    &    5                    &        4    \\
\hline

\end{tabular}
\end{center}
{\bf Notes:} Magnitudes, colours and redshifts were taken from the
SDSS DR9, while coordinates, minor/major axis ratios, and effective
radii were taken from NED. Members is the number of galaxies within the
virial radius with SDSS spectroscopy. Group richness is defined as
the number of spectroscopically confirmed members brighter than
0.4L$^*$.
\label{photometricdata}
\end{table}

The observations were  performed using GMOS  on Gemini North  on  December  24 in 2011 (GN-2011B-Q-107) in long-slit  mode.  Observations  were
carried out  using the   B600+G5307 grating  and a   long-slit of  1 arcsec   slit-width, yielding  a dispersion  of 0.92  \AA~pixel$^{-1}$. We measured  a
 spectral resolution  of  4.7~\AA\ over  a  wavelength  range  of  3500 to  6300~\AA \space from  the  FWHM   of  multiple  arc calibration lines.  Three
exposures of 1800~s were performed along each  of the major and  minor axes  of both  galaxies. Exposures were dithered in the spectral direction in order
to provide coverage across the GMOS chip gaps$^{1}$.   Arc  and flat-field observations were interleaved between  the target  exposures. The   bias  frames
provided by   the observatory were   taken  from a  preceding run  on December 12 in 2011. The seeing was $\geq$1.2 arcsec.

\begin{figure*}
\centering
\includegraphics[width=17cm]{figure1.eps}
\caption{\label{spectra} De-redshifted spectra along the major axis of
  the observed BCGs of the two fossil groups. The central bin and one in the
  outskirts around $\log(r/r_{\rm eff})\sim-0.3$ are shown.  Sky-line
  residuals have been removed for better visualisation. The spectra
  have been normalised to the flux at $\lambda$5000\AA. SDSS0758 shows
  prominent [OIII] $\lambda\lambda4959,5007$\AA \space emission in the
  center and is a well-known X-ray AGN \citep{sun09}.}
\end{figure*}

The data reduction was carried out using the {\small IRAF} Gemini
{\small GMOS} package and {\small STARLINK} software. Early in the
data reduction process, it became apparent that the data suffered from
significant contamination by scattered light, which had serious
repercussions for flat-fielding and sky-subtraction. In order to
minimise scattered light effects, the reduction procedure carried out
differed slightly from the usual sequence.  First, both target and
flat-field frames were bias subtracted, cleaned of cosmic rays and bad
pixels, and subsequently wavelength calibrated using Cu-Ar
comparison-lamp exposures.  Next we used the only regions in our
observations in which the scattered light could be measured directly,
i.e.\ the {\it bridges} in the long-slit\footnote{see the definition
of GMOS chip gaps and bridges on the Gemini homepage {\tt
http://www.gemini.edu/node/10663}}, to estimate scattered light
levels, again in both target and flat-field frames. We found that some
20\% of the light reaching the detector was in the form of scattered
light.  Estimates of the scattered light contamination were made by
interpolating between the two bridges, along the spatial
direction. These levels were then subtracted from each
exposure. Flat-fielding of the decontaminated target frames was then
carried out using the corrected flat-field frames.  The resultant
light profiles { along the spatial direction were found to be much
  flatter at large radii, i.e.\ more consistent with the constant sky
  level one would expect at such radii.  Sky subtraction was then
  performed in the usual manner, by fitting a linear function to
  background windows along the spatial direction on each side of the
  galaxy spectrum outside the effective radii. The data were then
  binned along each axis so that the outer bins still yielded a
  reasonable S/N. In the center, typically 2 pixels were binned while
  in the outskirts, around $r_{\rm eff}$, about 50 pixel rows were
  added.  We measure a typical S/N of $\sim 70$ in the center and
  $\sim 15$ in the outskirts. Figure \ref{spectra} shows spectra along
  the major axes for both galaxies. The central bin and one in the
  outskirts around $\log(r/r_{\rm eff})\sim-0.3$ are presented.

\section{Kinematics and Stellar population parameters}
\label{kinematicsstellarpopulations}

The recession velocities and velocity dispersions of each bin along
the major and minor axes were measured using the {\small IRAF fxcor}
routine.  \citet{bruzualcharlot03} SSP models that had been broadened
to the Gemini resolution of 4.7\AA\ were used as templates for this
procedure. The SSP model showing the highest cross-correlation peak
was selected for the final measures. However, variation in the values
derived using different templates was found to be less than the errors
quoted in our results. The resulting values are shown in
Fig.\ref{kins}. Radial velocities are shown with respect to the
systemic velocities. We find no evidence for rotation along either
axis of either galaxy.  The velocity dispersion profiles are both
shallow.  Indeed, there is no evidence for any reduction in velocity
dispersion with radius in SDSS0758. It is therefore clear that both
galaxies reside in substantial dark matter halos. The systemic
velocities of both galaxies and the measured central values of the
velocity dispersion are listed in Table \ref{slopes}.  Radial
velocities and velocity dispersions at each radial bin are given in
Appendix \ref{kinematicappendix}.

\begin{figure*}
\centering
\includegraphics[width=12cm]{newfigure2.eps}
\caption{\label{kins}Kinematics of the observed BCGs of the two fossil
  groups. Radial velocity and velocity dispersion profiles are
  shown. Red points are measured along the major axis while open ones
  present the minor axis.  The horizontal lines show the systemic
  velocities and central velocity dispersions. There is no evidence
  for rotation along either axis of either galaxy.  The velocity
  dispersion profiles are both shallow.}
\end{figure*}

\subsection{Measurement of stellar population parameters}
\label{sspmeasurement}
To derive stellar population parameters, Lick/IDS line-strength
indices as defined in \citet{worthey94} and \citet{worthey97} have
been measured. The data were corrected for internal velocity
dispersion as described in \citet{proctorsansom02}.  In the absence of
Lick standard star observations, full calibration to the Lick system
could not be performed. However, as noted in many previous studies
(e.g. \citealt{proctorsansom02}), the only indices that significantly
depend on such a calibration are Mg$_1$ and Mg$_2$. These indices were
therefore omitted from all further analysis. NaD was also excluded
because of its contamination by interstellar absorption. The observed
wavelength range did not cover the redshifted TiO indices. As a result
a total of 22 Lick indices from H$\delta$ to Fe5782 were measured with
20 of these used in our analysis. The measurement of stellar
population parameters age, [Z/H] and [E/Fe] was then carried out using
the multi-index $\chi^{2}$-fitting technique detailed in
\citet{proctorsansom02}. [Fe/H] was then derived from [Z/H] and [E/Fe]
using [Fe/H]=[Z/H]--0.94[E/Fe].  The 20 indices were fit, first
subject to three-sigma clipping. This removed indices contaminated by
effects such as sky-line residuals and chip gap residuals as well as
highlighting positions in the galaxies affected by emission lines (in
this case the central two pixels of SDSS0758 only). The stability of
the fits with respect to the exclusion of the remaining indices was
then tested to ensure that no individual index was creating tension in
the fit.  In most cases the indices causing such tensions were found
to be remaining sky-line residuals, or other clear distortions in the
spectrum. Those indices were also excluded from the final fits. The
resultant fits generally included between 17 and 20 indices.  The
derived stellar population parameters are shown in Fig. \ref{agez}.

\begin{figure*}
\includegraphics[width=12cm]{figure3.eps}
\caption{Spatially resolved stellar population parameters for the
  observed BCGs of fossil groups. The data are measured with the
  multi-index $\chi^{2}$-fitting technique detailed in
  \citet{proctorsansom02}. Filled, red points are measured along the
  major axis while open, black points present the minor axis.  The
  horizontal dashed lines show the measured central values. The
  vertical lines indicate the typical seeing during our
  observations. Solid lines show linear least squares fits to the
  data.}
\label{agez}
\end{figure*}

\subsection{Comparison with full-spectrum fitting}
\label{fullspectrumfitting}
To verify the stability of our measurements, we also compared our
results from the multi-index $\chi^{2}$-fitting technique with a
full-spectrum fitting technique.  Therefore, the open-source package
ULySS \citep{koleva09} was used to fit SSP models directly to the
observed galaxy spectra.  We utilized Pegase$\,$HR models resolved in
[$\alpha$/Fe] \citep{alphamodels} based on the Elodie$\,$3.2 library
of stellar spectra \citep{elodie32}\footnote{\tt
  http://ulyss.univ-lyon1.fr/optional.html}.  The library involves an
interpolator to provide a stellar spectrum at any point in the
parameter space ($T_{\rm eff}$, log~$g$, [Fe/H], and [$\alpha$/Fe]).
All Pegase$\,$HR models are computed assuming a Salpeter IMF and
Padova 1994 evolutionary tracks providing synthetic SSPs with ages
between 1$ -$20000$\,$Myr, metallicities between $-2.3$ and
$0.69\,$dex, and [$\alpha$/Fe] enhancement between 0 and 0.4.  A
multiplicative polynomial is used to adjust the overall spectral shape
to the SSP model. To achieve reasonable values an order of 40 was
chosen.  The ULySS parameter {\tt \textbackslash CLEAN} was considered
in the fitting procedure to exclude possible outliers in the galaxy
spectrum, resulting from any remaining night sky emission.  200
Monte-Carlo simulations were computed for every spectral bin,
repeating a fit successively with random Gaussian noise. The dimension
of the added noise was based on a user-defined signal-to-noise ratio
provided for each spectral bin.  The S/N ratios were measured with
{\small IRAF splot} at a rest-frame wavelength of $\sim5140\,$\AA. The
resulting point distributions were then used to calculate average SSP
ages, [Fe/H] (see also \citealt{eigenthaler13}) and
[$\alpha$/Fe] values. [Z/H] was then also calculated from these data
using [Z/H]=[Fe/H]+0.94[E/Fe]. Outliers clearly detached from the main point
distributions have been excluded for these measurements. Error bars
were estimated as the standard deviation of the point
distributions. For the comparison, we analysed the major axis spectra
of both galaxies with ULySS, and averaged the stellar population
parameters from both galaxy sides.

Besides stellar population parameters, ULySS also allows us to measure
the kinematic properties of our galaxies since the package measures
the shift in wavelength and broadening of each SSP model to match the
observed galaxy spectrum. ULySS determines the broadening relative to
the SSP model dispersion $\sigma_{\rm{model}}$, amounting to 13 km
s$^{-1}$ for the Pegase HR models. Taking into account instrumental
dispersion, velocity dispersions are computed via the relation: $\sigma
_{\rm{phys}}^2 = \sigma _{\rm{ulyss}}^2 + \sigma _{\rm{model}}^2 -
\sigma _{\rm{instr}}^2$.

We also investigated the use of composite stellar populations within ULySS
by attempting to fit the {\it relatively} high S/N central bins of each
galaxy with a combination of two independent SSPs. However, even in
these central bins the S/N was not sufficient to permit the
convergence of the fits to a stable solution given the large number of
free parameters in such a fit. Therefore we report here only fits
utilising single SSP models.

Figure \ref{comparison} shows the comparison of our results from the
multi-index $\chi^{2}$-fitting technique with the corresponding values
derived from ULySS full-spectrum fitting.  We confirm the kinematic
properties of both galaxies. However, we note that our ULySS results
show systematically larger values for velocity dispersion with an
average offset of $\sim 30$ km s$^{- 1}$ compared to the values
derived from the {\small IRAF fxcor} routine. The cause of this slight
discrepancy remains unidentified.  However, the offset (which is only
slightly larger than the velocity dispersion errors) was found to have
no significant effect on the stellar population parameters derived.

Qualitatively, both the kinematic and stellar population trends show
fair agreement. However, there are some noticeably quantitative
differences in the results of the stellar population analyses. Most
noticeably, the [E/Fe], although
showing similar gradients, are systematically lower in the ULySS full
spectral fitting results. There are also some slight systematic
differences in the derived [Fe/H] values. For SDSS0734 we confirm the
negative [Fe/H] gradient but find a shallower slope, while for
SDSS0758, we confirm the flat inner metallicity gradient, but find a
flat gradient also at larger galactic radii. Interestingly, however,
the derived [Z/H] values (which are a combination of [Fe/H] and
[E/Fe]) show good agreement at all points except for the two high
[Z/H] inner points of SDSS0734. It is therefore evident that these
differences arise as the result of the differing model set used
(Thomas, Maraston \& Bender in the multi-index fitting analysis and
Pegase$\,$HR models in the ULySS full spectral fitting
analysis). With regards age, for SDSS0734 agreement is very
good. However, the age gradient for SDSS0758 shows the largest
discrepancy in our comparison. While we find a flat age gradient from
multi-index fitting, we obtain systematically lower ages from
full-spectrum fitting which even show a negative trend around
$\log(r/r_{\rm eff})\sim-0.7$. We note that in this galaxy the
deviations in age and [Fe/H] follow the age-metallicity
degeneracy, i.e. higher [Fe/H] is compensated for by younger age.
It is our intention to continue to make these comparisons in future
data analyses in the hope of identifying consistent systematic
difference between the model sets.

\begin{figure}
\centering
\includegraphics[width=\columnwidth]{newfigure4.eps}
\caption{\label{comparison}Comparison between the results from the
  multi-index $\chi^{2}$-fitting technique detailed in
  \citet{proctorsansom02} and the full-spectrum fitting technique
  ULySS. Red symbols show the major axis data from Fig.\ref{agez},
  while open triangles show the results from full-spectrum fitting. In
  both cases the data have been folded along the major axis.}
\end{figure}

\section{Discussion}
\label{discussion}
Figs. \ref{agez} and \ref{comparison} show the spatially resolved
stellar population parameters resulting from our analysis.  It can be
seen from Figure \ref{agez} that there is good agreement between
values on either side of each axis and between the major and minor
axes of each galaxy. This evidences the robustness of the multi-index
fitting technique that we employ.  However, we note that there is a
hint of ellipticity in the [Z/H] and perhaps [Fe/H] profiles of
SDSS0758 such that they decline faster along the minor axis. SDSS0758
has a minor/major axis ratio of $\sim$0.8, which would cause a
displacement in log radius of points along the minor axis of -0.1 with
respect to the major axis values, consistent with the small
displacement evident in Fig.\ \ref{agez}. We detect no such
displacement in SDSS0734, despite it exhibiting a similar ellipticity
to SDSS0758.

The aim of this work is to begin the process of comparing the star formation
histories and/or merger histories of the BCGs of fossil systems to those of
normal systems. As well as the comparison of the central ages and metallicities of
such systems, another, and potentially more powerful, approach is the comparison
of the gradients in these parameters. As a result there has been much recent
interest in the use of metallicity gradients as a probe of the merger histories
of galaxies. However, such comparisons are severely hampered by the number of free
parameters that describe the star formation and merger histories of large
early-type galaxies, such as the BCGs in our sample. Key amongst these
parameters are whether the mergers that the central galaxies experiences are
¨wet¨ or ¨dry¨ (whether or not there is significant gas present during the
merger).

In the case of dry mergers the metallicity gradient of the merger remnant
depends on both the mass ratio of the merger event and on the pre-merger
gradients of the galaxies involved (e.g. \citealt{dimatteo09}). Conceivably,
age gradients could also be produced in the case of dry mergers if the two
merging galaxies have differing pre-merger ages.

For wet mergers, the effects on age gradients are generally more clearly
defined, since the gas present in the merging galaxies is known to be funneled
towards the galactic centers during the merger (e.g. \citealt{rupke10}), where
bursts of star formation then take place (e.g. \citealt{ellison13}). These
central starbursts then induce positive age gradients (younger towards the
centre). However, the prediction of metallicity gradients in merger remnants is
\emph{much} more complicated, since the gas driven to the centre of the merger
remnant can drive the gradient either up or down depending on the specifics of
the quantity, source and metallicity of the gas \citep{torrey12}.

Clearly, the interpretation of the gradients in individual galaxies
is, at best, a complex process.  This underlies the stated aim of this
work - to begin the process of measuring gradients in fossil group
BCGs in order to ultimately perform a \emph{statistical} comparison to
the BCGs of normal systems.  In the following we therefore refrain
from attempting detailed interpretation of the individual galaxies,
confining ourselves to general comments and comparisons between the
individual galaxies.\\

Gradients of the stellar population parameters were obtained by
fitting the radial profiles with linear relations of the form;

\begin{displaymath}
X[r/{r_{{\rm{eff}}}}] = {X_{{\rm{eff}}}} + {\nabla _{X}}\log\left(  {r/{r_{{\rm{eff}}}}} \right),
\end{displaymath}

\noindent where $X$ denotes the corresponding parameter.  The fits
have been weighted by the errors associated with each value.
Gradients were computed excluding the seeing affected innermost
point. We note that our derived gradients in general fit the data well
out to the effective radius.  The computed gradient slopes $\nabla
_{X}\equiv\Delta X/\Delta\log(r/r_{\rm eff})$ are given in Table
\ref{slopes}. The significance of these gradients were tested using
their correlation coefficients. All were found to be of greater than
3$\sigma$ except for the [E/Fe] in SDSS0758 (only 2$\sigma$) and log(age)
in SDSS0758 which in any case we find to exhibit no discernible
gradient.  The gradients in both galaxies were also found not to be
significantly affected by the exclusion of the low signal-to-noise
outer points.

We can readily see that despite the photometric
similarities between the two galaxies, there are significant
differences in the radial gradients of their stellar population
parameters.

Fig. \ref{agez} shows that, in SDSS0734 there is clear evidence that
the central region of the galaxy is younger than its outer
regions. There is also a suggestion that the central region has lower
[E/Fe] than the outer regions. Both [Fe/H] and [Z/H] show negative
gradients over the whole radial range probed.  This galaxy therefore
appears to have formed either in a single burst, but star formation
with the central regions continuing for 1-2 Gyr longer than in the out
regions i.e. long enough for Type Ia supernovae to begin driving the
central [E/Fe] down, or the galaxy may have formed an old population
upon which a central burst was then superimposed a few Gyr later. In
either case we might characterise the central regions of this galaxy
as having formed over a relatively extended period of time.

In SDSS0758, on the other hand, the galaxy exhibits no detectable age
gradient but a negative [E/Fe] gradient.  Although the galaxy also
exhibits negative [Fe/H] and [Z/H] gradients, there is a strong
suggestion of a flattening in this gradient in the central
regions. Indeed the [Fe/H] gradient appears to be flat all the way out
to $\log(r/r_{\rm eff})=-1$, and from our full-spectrum fitting
results, we measure a flat gradient even out to $\log(r/r_{\rm
  eff})=-0.5$.  The [E/Fe] gradient in this galaxy suggests that this
galaxy also formed over a relatively extended period of time, except
in this case the later star formation took place in the outer regions,
rather than in the central regions as was the case in SDSS0734. This
galaxy therefore resembles a bulge and disc galaxy in which star
formation was truncated very early, but slightly later (perhaps
$\sim$1 Gyr) in the outer regions. This is supported by the flat
(perhaps even negative) [Fe/H] gradient in the central region which
can arise if star formation in the central was truncated prior to the
star formation further out.

It is the aim of this project to compare the BCGs of fossil galaxy
groups with BCGs in non-fossil aggregates.  However, with a sample of
only two galaxies we can draw no firm conclusions at this point, other
than to note that these galaxies possess stellar population parameters
and gradients consistent with the range of values exhibited by normal
BCGs \citep{loubser}.

\begin{table*}
\begin{centering}
\caption{Central values and radial  gradients of the measured  stellar population parameters. System  velocities and central velocity  dispersions are
also given. Errors are listed below the corresponding values.}
\begin{tabular}{c..c.ccc...}
\hline
            &   \multicolumn{1}{c}{RV}    &  \multicolumn{1}{c}{$\sigma_{0}$}  &  \multicolumn{1}{c}{log(age)$_{0}$}  &  \multicolumn{1}{c}{[Fe/H]$_{0}$}   &  \multicolumn{1}{c}{[E/Fe]$_{0}$}  &   \multicolumn{1}{c}{[Z/H]$_{0}$}   &            $\nabla$log(age)             &   \multicolumn{1}{c}{$\nabla$[Fe/H]}  &  \multicolumn{1}{c}{$\nabla$[E/Fe]}  &  \multicolumn{1}{c}{$\nabla$[Z/H]}   \\
\hline
  SDSS0734  &            23868            &                254                 &                 0.96                 &                 0.14                &                0.28                &                0.40                 &                  0.08                   &                 -0.45                 &                 0.13                 &                -0.33                 \\
            &                6            &                 17                 &                 0.01                 &                 0.01                &                0.04                &                0.03                 &                  0.05                   &                  0.07                 &                 0.07                 &                 0.10                 \\

  SDSS0758  &            12209            &                224                 &                 1.10                 &                -0.16                &                0.37                &                0.19                 &                  0.01                   &                 -0.22                 &                -0.13                 &                -0.34                 \\
            &                9            &                  9                 &                 0.03                 &                 0.02                &                0.03                &                0.01                 &                  0.06                   &                  0.09                 &                 0.05                 &                 0.09                 \\
\hline
\end{tabular}
\label{slopes}
\end{centering}
\end{table*}

\section{Conclusions}
\label{conclusions}

We have presented spatially resolved stellar population parameters
log(age), [Fe/H], [Z/H] and [E/Fe] for two first-ranked fossil group
galaxies, SDSS0734 and SDSS0758.  These two galaxies have quite
different stellar population parameters, despite their similarities in
morphology, absolute magnitude, colour and central velocity
dispersions, as well as the groups in which they resdie. We suggest
that one of them, SDSS0734, may have had a recent central burst of
star formation superimposed onto an old stellar population, which
explains a positive age gradient, a steep negative metallicity
gradient and positive [E/Fe] gradient. For the other galaxy, SDSS0758,
however, a well-known X-ray AGN \citep{sun09}, we argue that the star
formation was complete $\sim$ 10 Gyr ago. However, the negative
gradient in [E/Fe] and flat central metallicity profile suggest that
star formation may have continued slightly longer in the outer regions
than in the central regions of the galaxy (i.e.\ inside out
formation). Our results therefore suggest that there may be
considerable dispersion in the range of measured stellar population
parameters of first-ranked galaxies in fossil groups. We plan to study
a larger sample of these galaxies in detail in order to have a better
notion of how the central galaxies in fossil groups formed and to
investigate if there are any differences in their properties with
respect to the properties of non-fossil BCGs.

\section*{Acknowledgments}

This work is based on observations obtained at the Gemini Observatory,
which is operated by the Association of Universities for Research in
Astronomy, Inc., under a cooperative agreement with the NSF on behalf
of the Gemini partnership: the National Science Foundation (United
States), the Science and Technology Facilities Council (United
Kingdom), the National Research Council (Canada), CONICYT (Chile), the
Australian Research Council (Australia), Minist\'{e}rio da Ci\^{e}ncia
e Tecnologia (Brazil) and Ministerio de Ciencia, Tecnolog\'{i}a e
Innovaci\'{o}n Productiva (Argentina). This research made use of
NASA's Astrophysics Data System, as well as IRAF and STARLINK
software.  IRAF is distributed by the National Optical Astronomy
Observatories, which is operated by the Association of Universities
for Research in Astronomy, Inc.  (AURA) under cooperative agreement
with the National Science Foundation.  RNP acknowledges financial
support from the Brazilian agency FAPESP (program number
2008/57331-0). CMdO also acknowledges FAPESP (program number
2006/56213-9) and CNPq. PE acknowledges support from FONDECYT through
grant 3130485. We thank Mina Koleva for useful discussions on the
full-spectrum fitting package ULySS.