adSformers: Personalization from Short-Term Sequences and Diversity of Representations in Etsy Ads

Abstract: In this article, we present a general approach to personalizing ads through encoding and learning from variable-length sequences of recent user actions and diverse representations. To this end we introduce a three-component module called the adSformer diversifiable personalization module (ADPM) that learns a dynamic user representation. We illustrate the module's effectiveness and flexibility by personalizing the Click-Through Rate (CTR) and Post-Click Conversion Rate (PCCVR) models used in sponsored search. The first component of the ADPM, the adSformer encoder, includes a novel adSformer block which learns the most salient sequence signals. ADPM's second component enriches the learned signal through visual, multimodal, and other pretrained representations. Lastly, the third ADPM "learned on the fly" component further diversifies the signal encoded in the dynamic user representation. The ADPM-personalized CTR and PCCVR models, henceforth referred to as adSformer CTR and adSformer PCCVR, outperform the CTR and PCCVR production baselines by $+2.66\%$ and $+2.42\%$, respectively, in offline Area Under the Receiver Operating Characteristic Curve (ROC-AUC). Following the robust online gains in A/B tests, Etsy Ads deployed the ADPM-personalized sponsored search system to $100\%$ of traffic as of February 2023.

\section{Introduction}
\label{introduction}

Etsy~\footnote{\url{https://www.etsy.com}} is an online marketplace that brings together millions of buyers and sellers of unique handmade and vintage items. Etsy Ads’ vision is to give sellers more agency in their success by providing them with the tools to market their shops. In this vein, Etsy Ads supplements Etsy's organic search results with relevant listings (items) sponsored by Etsy sellers (ads). Best-in-class ad relevance is rooted in personalization. Personalization is where seller imagination meets buyer inspiration. While Etsy Ads are not limited to the search product (we also have recommendation ads), this article focuses on sponsored search personalization.

\subsection{Introduction to Etsy's Ad Ranking}
\label{sec:ad_ranking}

We first provide an overview of Etsy Ads ranking and bidding systems for sponsored search, before introducing our approach to personalizing these systems. Etsy Ads allows sellers on the platform to sponsor their listings through a second-price cost-per-click auction campaign. To ensure the efficiency of its ranking and bidding systems, Etsy Ads relies on accurate real-time predictions of the probability that a user will click or purchase a given sponsored listing based on the ad's context.

As with many industrial search and recommendations systems, Etsy Ads uses Learning to Rank (LTR) \cite{liu2011learning, cao2007learning} as the primary information retrieval methodology to build the ranking system shown in Figure \ref{fig:ltr}. The system is divided according to the classic two-stage dichotomy: \textit{candidate retrieval} and then \textit{ranking}. We give more details on the candidate retrieval stage in the Appendix \ref{appendix:candidates} and focus our exposition on the ranking phase of sponsored search. First, the candidate retrieval step winnows the full inventory from over 100 million listings to under 1000 matched candidates. Then, two separate pointwise-ranking models, the CTR and the PCCVR, form the foundation of the re-ranking pass. As direct drivers of user engagement, the CTR and PCCVR prediction models are critical to the stakeholder balance of buyer engagement, seller return, and marketplace revenue.

\begin{figure}[tp]
  \centering
  \includegraphics[scale=0.4]{images/learning_to_rank.png}
  \caption{In our Learning to Rank system, ads are ranked by the value score in Equation \ref{eq:score_fn}, which combines the CTR and PCCVR predictions.}
  \label{fig:ltr}
\end{figure}

\subsubsection{Model design CTR and PCCVR} Both the CTR and PCCVR tasks are formulated as binary classification problems, with a binary cross entropy loss function
\begin{equation}
L = -\frac{1}{N} \sum_{\left( \mathbf{x}, y \right ) \in D} \left( ylogp \left( \mathbf{x} \right) + \left( 1-y \right) log \left( 1-p \left(\mathbf{x}\right)\right)\right),
\end{equation}
where $D$ represent all samples in the training dataset, $y \in \{ 0, 1 \}$ is the label, $p(x)$ is the predicted probability, and $x$ is the input vector containing (listing, query, and context) attributes. In the CTR case, $p(\mathbf{x})$ denotes the predicted probability $p(y_{CTR}=1)$ that the candidate listing will be clicked, while for PCCVR $p(\mathbf{x})$ denotes the predicted conditional probability $p(y_{PCCVR}=1|y_{CTR}=1)$ that the listing will be purchased, having been clicked. We combine the CTR and PCCVR predictions into a value score, which we then use to sort sponsored search results. We often experiment with the value function, which currently is

\begin{equation}
\label{eq:score_fn}
value = f \left( p \left( y_{CTR}=1 \right), bid \right),
\end{equation}
where the bid is a function of the PCCVR prediction and other business metrics, $bid = f(p(y_{PCCVR}=1|y_{CTR}=1))$. Etsy Code as Craft post \cite{EtsyCodeAsCraftCsb} describes the initial journey to context-specific bidding. CTR and PCCVR scores must reflect true probabilities in order to assign monetary value and forecast budget depletion, therefore we introduce a calibration layer. We achieve calibration through the parametric approach of Platt Scaling \cite{platt1999probabilistic, guo2017calibration}.

\subsubsection{Ads personalization} In this article, we introduce an approach to personalizing the CTR and PCCVR models by encoding session user data. We define a "session" as a one-hour shopping window. To gain insight into a user's journey during a session, we analyze user implicit feedback, such as search queries, item favorites, views, add-to-carts, and purchases. Figure \ref{fig:session} displays a histogram of the number of listing views from a sample of sessions. The histogram shows a power law distribution suggesting that a small group of users account for a disproportionate number of views. The vast majority of users interact with a small number of listings before leaving the session. This insight about user dynamism drives the decision to personalize ad ranking by encoding short-term sequences of user actions that took place during the most recent one-hour session. The rankers benefit from streaming features being made available within seconds of the user interacting with a query, listing, shop, or category, which optimizes the experience for in-the-moment shopping.

\begin{figure}
  \centering
  \includegraphics[scale=0.3]{images/session_views_hist.png}
  \caption{Distribution of User Listing Views in Short-Term Sessions.}
  \label{fig:session}
\end{figure}

In this article we make the following contributions:

\begin{itemize}
\item We introduce the adSformer diversifiable personalization module (ADPM henceforth), our highly configurable module used to encode sequences of in-session logged-in and logged-out user actions and derive short-term dynamic user representations. The module has three components: 1. the adSformer encoder component; 2. the pretrained representations component; 3. the learned representations component. We explain the ADPM's design and implementation for maximum impact and scalability.
\item We introduce and describe the adSformer encoder component of the ADPM, built with a modified transformer block backbone.
\item We illustrate how we use the ADPM to personalize sponsored search (ad) ranking. To validate ADPM's effectiveness, we include offline and online experimental results and deployment considerations for the resulting adSformer CTR and adSformer PCCVR models.
\item We illustrate our diversity mission through the diversity of representations learned in Etsy Ads and included in the ADPM. Furthermore, we present our pretrained representation learning workflows, with a special focus on visual and multimodal representations.
\item We share reproducibility details including optimal hyperparameters, model design choices, and training approaches, so that our readers can readily extend our results to their short-term personalization use cases.
\end{itemize}

\subsection{Related Work}

 \subsubsection{Personalization from user sequences} A useful survey of deep learning methods employed in personalized online services and CTR prediction is \cite{zhang2021deep}. A few other works include sequences of user actions to improve CTR in \cite{zhou2019deep, pi2020search, pi2019practice, chen2019behavior, xu2022rethinking}. Personalization can be applied at both candidate retrieval and re-ranking stages of the sponsored search journey. In a starkly different approach from ours, Etsy's organic search documented their personalization journey in \cite{supersonalization}. Recent developments in the personalized ranking and recommendations domain emerged to separate personalization from sequences into long-term and short-term sequences. For example, \cite{pancha2022pinnerformer} employs a transformer architecture to learn user representations from long term sequences which are later employed in downstream ranking and recommendation models for the sake of learning and serving efficiency. Furthermore, Pinterest describes an end-to-end ranking application in \cite{xu2022rethinking}, with a brief reference to recent user actions.

 \subsubsection{Transformers in personalization} Transformer encoders have been used in personalized recommender systems and product search, both at retrieval and re-ranking phases. The transformer architecture captures the context of users' intents by encoding the sequential nature of user behavior sequences as seen in previous works from Alibaba, Pinterest, and others \cite{chen2019behavior, kang2018self, sun2019bert4rec, pancha2022pinnerformer}. Focusing on the retrieval phase, TEM, a transformer based model, can tune the amount of personalization applied based on the query characteristics \cite{bi2020transformer}. In addition to employing attention mechanisms and transformer architectures authors in \cite{zhang2020personalized} employ graph neural networks to build a session-aware recommender. In a similar vein, sequence and graph modelling came together as a heterogeneous user graph transformer in \cite{shui2022sequence} to improve personalized recommendations.

\subsubsection{The magic of diversity} Diversity is a cornerstone concept in machine learning. Engineers augment training datasets to improve example diversity thereby learning more robustly \cite{wang2021augmax}. Multicollinearity \cite{hastie2009elements} is a potential side effect of correlated features or lack of signal diversity. A diverse set of architectural interaction modules improve model performance in CTR prediction \cite{zhang2022dhen}. Amazon measures video recommenders diversity through offline coverage metrics, and high-precision non-diversified recommendations look like a hall of mirrors showing only products in a single topic \cite{Rybakov2018}. Ultimately, user interests are marked by diversity, and in the recommendations domain "minority matters" \cite{bao2022minority}. In fact, without diversity we would have no need for personalization. When users' interests are homogeneous a simple popularity recommender caters to all. Therefore, our personalization approach employs a diversity of representations. As a side note, for the sake of consistency and generality, in this article we make an effort to use the terminology of "representation" even when referring to "word embeddings" and any other types of "embeddings". Generally, neural networks are effective at learning representations due to their ability to capture complex patterns in data \cite{bengio2013representation}.

The rest of the article is organized as follows. We introduce our approach to personalization from short-term sequences and the ADPM in Section \ref{sec:perso}. We describe how we use ADPM to personalize the CTR and PCCVR ranking models in Section \ref{sec:ranking}. Since pretrained representations are a central component of the ADPM and highly diverse, we explain our pretrained representation learning workflows in Section \ref{pretrained}. Afterwards we present our offline and online experimental results in Section \ref{sec:experiments} and then we finally conclude.

\section{Personalization from Short-Term Sequences of User Actions}
\label{sec:perso}

  To personalize Etsy Ads sponsored search, we analyze sequences of recent user actions anywhere on Etsy. For a more comprehensive understanding of user behavior, we use data collected from both logged-in users and logged-out browsers. We model logged-in and logged-out impressions jointly. Note that instead of considering the last M user actions as in \cite{pancha2022pinnerformer, chen2019behavior, xu2022rethinking} and others, we instead consider a sliding one hour window of user actions, in reverse chronological order of timestamps, since recency is key in our approach, especially for logged-out users. More formally, let $I = \left( a, e, t \right)$, be a one-hour sequence of user actions, where $a$ is the action type, $a \in \{\text{view}, \text{favorite}, \text{add-to-cart}, \text{purchase}, \text{search}\}$, and $e$ represents the entity (listingId, shopId, categoryId, or text-query) associated with action $a$ performed at timestamp $t$. Due to semantics and infrastructure constraints we further cap the maximum sequence length at $M=50$ actions. Each sequence $I$ is truncated to within one hour of the most recent action, so $t_0-t_{last} \leq 1 hr$. Some users could have up to 50 actions while others could have one or none. Intuitively, by truncating interaction sequences to a one hour window, we apply more or less personalization, depending on the per-user available relevant short-term personalization signal. Next we introduce the ADPM, our approach to configurable personalization from short-term sequences.

\subsection{adSformer Diversifiable Personalization Module (ADPM)}

\begin{figure}[tbp]
  \centering
  \includegraphics[scale=0.3]{images/adSFormer_module_and_block.png}
  \caption{We use the adSformer diversifiable personalization module (ADPM) on the left side to personalize downstream models, such as CTR and PCCVR in Ad Ranking. The adSformer block on the right side is used in the adSformer encoder component of the ADPM.}%
  \label{fig:adSFormer_module_and_block}
\end{figure}

We build ADPM seen in left-hand side of Figure \ref{fig:adSFormer_module_and_block} to encode one-hour user sequences $I$ and derive a short-term dynamic user representation which we then concatenate to the input layer of downstream CTR and PCCVR tasks to build personalized ranking models, the adSformer CTR and PCCVR. The ADPM is highly configurable. Note that the one-hour window length is a hyperparameter we can vary in ADPM's configuration. The ADPM includes three components which produce three different types of representations, which we then concatenate to produce the final ADPM output. The three components are:
\begin{itemize}
\item \textbf{adSformer encoders component.} The adSformer encoder component uses one or more custom adSformer blocks with several attention heads, no dropout and a global max pooling layer as seen in Figure \ref{fig:adSFormer_module_and_block}, right panel. This component learns a deep, expressive representation from the highest impact sequences. We find that an update to the standard transformer block in \cite{vaswani2017attention} works better when considered as part of the pool of diverse representations in ADPM, overcoming the challenge of correlated signals. The adSformer block starts with a representation layer learned for each encoded one-hour sequence of viewed listing IDs. We pad and mask the variable length sequence to a common length of $M=50$. Furthermore, we concatenate the target listing representation at position zero as in \cite{chen2019behavior}. We then add a fully learnable position representation to the concatenated (target, sequence) representation as in \cite{devlin2018bert}. We currently only encode the sequence order index to represent position, and not the actual timestamp. We further detail the adSformer block architecture in the Appendix \ref{sec:block}.
\item \textbf{Pretrained representations component.} Additionally, we use multiple types of pretrained representations together with average pooling to encode the one-hour sequences of actions. We encode concatenated sequences of recent search queries using skip-gram text representations described in Section \ref{sec:fastText}. Depending on downstream performance, we choose from multimodal (AIR) representations detailed in Section \ref{sec:air}, visual representations detailed in Section \ref{sec:visual}, or interaction-based representations detailed in \ref{sec:skip-gram} for encoding sequences of listing IDs. Section \ref{pretrained} details Etsy Ads' workflows for learning these pretrained representations.
\item \textbf{Learned representations component.} The third component of ADPM introduces representations learned for each sequence from scratch in its own vector space as part of the downstream models. Average pooling summarizes each sequence into a dense vector representation.
\end{itemize}

 We concatenate the three components' outputs to produce the final ADPM dynamic user representation. Tracking the trajectory of this dynamic vector through time similarly to dynamic word representations \cite{bamler2017dynamic} can describe users' changing short-term intents, which we leave for future work.

\begin{figure*}[!tbp]
    \centering
    \subfloat[\centering adSformer CTR model]{{\includegraphics[scale=0.35]{images/adsformer_CTR_v3.png}\label{fig:adsformer_ctr} }}%
    \subfloat[\centering adSformer PCCVR model]{{\includegraphics[scale=0.35]{images/adsformer_PCCVR_v3.png}\label{fig:adsformer_pccvr}}}%
    \caption{We personalize the CTR and PCCVR models by including the short-term user representation from ADPM in the input representation layer. Additional architectural differences exist between the adSformer CTR and PCCVR models.}%
    \label{fig:model_archs}%
\end{figure*}

\subsection{ADPM Design and Implementation}
\label{sec:ADPM}
We implement both the adFormer encoder component and the ADPM as custom TensorFlow \cite{abadi2016tensorflow} Keras layers which we can reuse across personalization use-cases with a simple import statement. The ADPM layer is configurable, allowing arguments for customizing the ADPM with respect to: pretrained representations selection, action types selection, learned representation dimensions selection, adSformer encoder architectures, vocabularies selection, session duration, etc. Since sequences of streamed user actions are made available by the Feature System team to all downstream Etsy applications, we can easily scale to multiple downstream personalization tasks while at the same time introducing a per-model customization to avoid sacrificing performance.

\subsubsection{Vocabularies} ADPM's sequence representations are implemented as lookup tables. However, when vocabularies are large, the corresponding weight matrices can take up too much memory and disk space, a significant challenge in deployments. To overcome this challenge and reduce memory consumption, we limit the vocabulary size for each sequence to the top $K$ most frequent entities, tuning $K$ as a hyperparameter in downstream CTR and PCCVR models. For example, the CTR's vocabulary corresponding to sequences of listing IDs was capped to top $K=750,000$ listings, down from 100 million in total. The top $K$ listings carry the largest impact, after which increasing vocabulary size led to only marginal improvements in model performance. Hence vocabulary size impact on model performance follows a power distribution \cite{roberts2017expectation}, as further illustrated in ablation studies in Appendix \ref{appendix:training}. We built the vocabularies dynamically, from the corpus of one-hour sequences present in the downstream models' training datasets, further maximizing impact.

\subsubsection{Software implementation of the pretrained representations component} Pretrained representations are useful for feature encoding, scalability and modularity and we describe learning pretrained representations for every active listing in Section \ref{pretrained}. Running this process offline reduces online compute and model complexity. The pretrained representation vectors are joined to the input representation layer of adSformers through the use of a lookup table and a mapping between vocabulary and index positions. To make pretrained representations easily accessible to downstream models, we wrap the lookup table in a TensorFlow SavedModel which includes average pooling layers and handling of default sentinel padding values. Because both the pretrained representations and downstream adSformer models are re-trained daily, they must be versioned together. By saving the lookup table with the downstream model, we assure that the representations used at training time are the ones used at serving time.

\section{Personalizing Ad Ranking through ADPM}
\label{sec:ranking}

 Next we describe how we personalized both the CTR and the PCCVR models employed in Etsy Ads ranking using the ADPM. The ADPM's configurations differ between the CTR and the PCCVR models, reflecting the differences between these two downstream tasks. Fortunately, the ADPM is configurable and scalable, as described in Section \ref{sec:ADPM}. At a high-level, both architectures stack an input representation layer, an interaction module, and finally a stack of fully-connected layers. Figure \ref{fig:model_archs} shows how layers differ in sizes and input constituency between adSformer CTR and PCCVR, further reflecting the different characteristics of the click-through rate and post-click purchase rate prediction problems.

\subsection{Input Representation Layer in adSformers}

As the input to each of the adSformer CTR and PCCVR models we form a wide feature vector of dimension $d$, $\boldsymbol{x}_{q,l,u} \in \mathbb{R}^{d}$ which constitutes the overall representation of a clicked or purchased ad. We concatenate the dynamic user representation from ADPM with additional (query, listing, and context) input representations. Additionally we encode the query and other textual inputs such as candidate listing's title, using pretrained skip-gram text representations described in Section \ref{sec:fastText}, as well as using text representations trained within the taxonomy space \cite{kacmajor2020semantic} using PCCVR training data. We further fine-tune the skip-gram text representations during CTR and PCCVR model training. Other numerical and categorical features are encoded into a dense feature layer and concatenated in. Features differ to some extent between the adSformer CTR and PCCVR models. Next we detail how the ADPM configuration and other architectural choices differ between the two tasks.

\begin{table}
  \caption{The different CTR and PCCVR ADPM configurations}
  \label{tab:configs}
  \footnotesize
  \begin{tabular}{lrrrr}
    \toprule
    Config & CTR & PCCVR\\
    \midrule
    Pretrained Representation Type & AIR & skip\&visual \\
    fastText Representation Dimension & 256 & 128 \\
    Num adSformer Blocks & 1 & 1 \\
    adSformer Block Dropout & 0 & 0 \\
    Multihead Attention Num Heads & 3 & 2 \\
    Learned Listing Representation Dimension & 32 & 32\\
    Learned Shop Representation Dimension & 16 & 16  \\
    Learned Taxonomy Representation Dimension & 8 & 8 \\
    (e, a) (Learned Component) & (all e, all a) & (all e, all a) \\
    (e, a)((Pretrained Component) & (listing, all a) & (listing, all a) \\
    (e, a) (adSformer Encoder) & (listing, viewed) & (listing, viewed) \\
    (listing, viewed) Vocabulary Size & 750K & 650K\\
     Num OOV Tokens & 1 & 1 \\
  \bottomrule
\end{tabular}
\end{table}

\subsection{adSformer CTR}

In this section we describe the ADPM composition and the overall architecture for the personalized CTR model.

\subsubsection{ADPM configuration in CTR} Table \ref{tab:configs} gives the ADPM configurations for the adSformer CTR model. Specifically, the adSformer encoder includes one adSformer block with three attention heads. Currently, the adSformer encoder encodes only user and browser sequences of recently viewed listings fused with the target listing, since these $(e, a)$ have the highest session frequency. Within the pretrained representations component, the multimodal pretrained representation (AIR) described in Section \ref{sec:air} works best for the CTR task to encode all sequences of listing IDs. Additionally, ADPM's learned representations component encodes all one-hour sequences of recent user actions.

\subsubsection{CTR Model Architecture and Training}

For reproducibility, Table \ref{tab:configs1} in the Appendix shares hyperparameters and other algorithmic choices employed in the adSformer CTR training, as well as training and evaluation datasets details. Figure \ref{fig:model_archs} (a) depicts the model architecture. We compare the adSformer CTR's offline and online model performance against the production model, a feed forward neural network with four layers. Therefore, we preserve hidden layer sizes in the adSformer CTR architecture, to better understand sources of model improvements.

Beyond including the ADPM in the input representation layer, we add a cross network (DCN) to the personalized adSformer CTR architecture \cite{wang2017deep}. As recognized in recent work \cite{zhang2022dhen, lang2021architecture, song2019autoint, li2020interpretable}, learning higher order feature interactions from the input layer helps performance in large scale CTR prediction. For the adSformer CTR, the interaction module is necessary to fully leverage the wide input vector which includes the ADPM. The cross layer exclusion leads to a 1.17\% drop in the area under receiver operating characteristic curve (ROC AUC). Beyond DCN, we plan to explore other interaction module designs through low-rank structures and compression mechanisms \cite{wang2021dcn, roberts2020qr}, attention \cite{li2020interpretable}, or a mixture of interaction modules \cite{zhang2022dhen}.

The adSformer CTR architecture has four times the number of parameters of the baseline NN and includes the adSformer encoder. The large capacity aids learning and generalization by providing a smoother loss landscape and more stable and faster learning dynamics. The adSformer CTR only needs one epoch of training, for a total of 11 hours (one A100 GPU) using Adam \cite{kingma2014adam} optimizer. Throughout model training we decay the learning rate using cosine annealing \cite{loshchilov2016sgdr}. We select the largest batch size that can fit in memory and tune the learning rate to an optimum $lr=0.002$.

\subsection{adSformer PCCVR}
Next we describe the ADPM configuration and architecture for the personalized PCCVR model.

\subsubsection{ADPM configuration in PCCVR} Table \ref{tab:configs} gives the ADPM configuration for the adSformer PCCVR model. The adSformer encoder component includes one adSformer block with two attention heads and encodes sequences of recently viewed listing IDs as well as recently purchased taxonomy IDs. We parameterize the ADPM's pretrained representations component with pretrained skip-gram and visual listing representations. The third ADPM's component, learned representations, parallels adSformer CTR's ADPM configuration.

\subsubsection{PCCVR Model Architecture and Training}
Figure \ref{fig:model_archs} (b) shows the adsformer PCCVR model architecture. Similarly to the adSformer CTR, in order to take advantage of the large and sparse feature space, we use a Deep \& Cross Network (DCN) \cite{wang2017deep, wang2021dcn} consisting of two cross layers stacked before two fully connected dense layers.

While we trained the PCCVR baseline production model on only two weeks of historical click logs (135 million examples), the adSformer PCCVR has more parameters after ADPM addition and thrives with three weeks of data (200mln examples). Table \ref{tab:pccvr_train_window} in the Appendix reveals the offline comparison results for varying dataset sizes. Similarly to the adSformer CTR, we train with cosine annealing \cite{loshchilov2016sgdr}. The LazyAdam optimizer from TensorFlow Addons package helps lazier handling of gradient updates for sparse variables. In contrast to the adSformer CTR which thrived with the vanilla Adam, utilizing LazyAdam reduced training time by three hours without diminished accuracy.

\begin{figure}[h]
\includegraphics[width=3.25in]{images/bias.png}
\centering
\caption{The average prediction scores of the CTR model for impressions at each position.}
\label{fig:debiased-results}
\end{figure}

\subsection{Position Bias Removal in adSformers}

Sampling biases, such as position bias \cite{huang2021deep}, can reduce ad ranking improvements from ADPM. We observed a correlation between the positional rank of a shown ad and the probability of being clicked, which can lead to a feedback loop where the model inadvertently prioritizes certain candidates due to their previous high ranking and not due to improved personalization. To address this challenge, we include an auxiliary debiasing model \cite{zhao2019recommending}. We feed features such as positional rank, platform, and device, into a two-layer fully connected network with 100 units in each layer. We then add the output logits of this auxiliary model to the logits of the main model during training. We discard the bias tower and serve only the main tower at inference time. Figure \ref{fig:debiased-results} shows the reduction in correlation between position and CTR from debiased model as compared to a baseline model that was trained without the bias tower. We include a discussion of additional sampling biases in Appendix \ref{sec:bias-appendix}.

\subsection{To Multitask or Not To Multitask}
\label{multi}
Numerous real world machine learning applications across natural language processing \cite{radford2019language}, computer vision \cite{beal2022billion}, trajectory prediction \cite{roberts2019neural}, personalized ads, search and recommendations \cite{zhao2019recommending, baltescu2022itemsage}, and computational advertising \cite{ma2018entire, ma2018modeling} employ multitask learning \cite{caruana1997multitask}. A technique many decades old, multitask learning commonly improves accuracy of tasks as well as efficiency of machine learning pipelines. However, the case for multitask learning in mature machine learning systems that have been deployed as separate models for many years, is complex in practice. For example, models such as the adSformer CTR and the adSformer PCCVR have been tuned separately with different choices in terms of training datasets, neural network capacities, and ADPM configurations, and we observed different learning dynamics.

\section{Pretrained Representations for ADPM}
\label{pretrained}

 Component two of the ADPM employs pretrained representations to encode one-hour sequences of user actions. Beyond the ADPM, in Etsy Ads we use pretrained representations as part of our candidate retrieval system, as nearest neighbors for recommendation ads, and as features in ranking models. For this reason we give an overview of Etsy Ads pretrained representation learning workflows next. We train word representations for encoding text inputs and several flavors of representations for encoding listing IDs: visual, multimodal (AIR), and skip-gram. We found that different pretrained listing ID representations perform differently in downstream tasks. For example, Figure \ref{fig:emb_knn_comparison} shows the top k nearest neighbors for a listing using both visual and AIR representations in a retrieval task. Because the visual and AIR representations encode different signals for the same listing, the retrieved neighbors are quite different. We find that the AIR representations better encode price, category, and text while the visual representations better encode color, shape, and background image.

\begin{figure}[h]
    \begin{minipage}[b]{0.3\linewidth}
        \subfloat[\centering Target listing]{{
            \includegraphics[width=20mm]{images/emb_target.png}
            \label{fig:emb_target}}}
    \end{minipage}%
    \begin{minipage}[b]{0.7\linewidth}
        \subfloat[\centering AIR representation]{{
            \includegraphics[width=0.85\linewidth]{images/emb_air_knn.png}
            \label{fig:emb_air_knn}}}

        \subfloat[\centering Visual representation]{{
            \includegraphics[width=0.85\linewidth]{images/emb_visual_knn.png}
            \label{fig:emb_visual}}}
    \end{minipage}

    \caption{Given a target listing of an emerald ring shown in (a), we visualize the top $6$ nearest neighbors based on the listing's pretrained AIR representation (b) and listing's visual representation (c).}
    \label{fig:emb_knn_comparison}
\end{figure}

\subsection{Visual Representations}
\label{sec:visual}
Pinterest authors lament the scarcity of visual representations across large-scale industry applications \cite{beal2022billion}. Conversely, in Etsy Ads, we employ rich image signals across a variety of downstream tasks, such as visually similar candidate generation, search by image, learning other pretrained representations, and the ADPM-based adSformer CTR and PCCVR ranking models.

\subsubsection{Model design} We train image representations using a multitask classification architecture \cite{pinterest19unified}, an improvement over the "classic" classification as a proxy to metric learning \cite{pinterest18classification}. By using multiple classification heads, such as taxonomy, color, and material, our representations are able to capture more diverse information about the image. We used an EfficientNetB0 architecture \cite{efficientnet} and weights pretrained on ImageNet \cite{deng2009imagenet} as the backbone, and replaced the final layer with a 256-dimensional convolutional block, our desired output representation size. We apply image random rotation, translation, zoom, and a color contrast transformation to augment the dataset during training.

\subsubsection{Multitask learning of visual representations} Rather than using a single dataset of listing images with multiple label columns, we concatenate heterogeneous dataset sources containing product images with different selected attributes as the labels. Listing attributes such as color and material input by sellers are optional, so they can be sparse. To mitigate this concern we implemented a dataset sampler, pictured in Figure \ref{fig:viz-multitask-arch-3ds}, which takes in the separate datasets, each with its own unique label type. The sampler evenly distributes examples from each dataset to construct balanced training batches and ascertain that only loss coming from the respective task’s classification head is considered during backpropagation. One additional benefit of this approach is that we can mix in datasets of images from different visual domains to support more use cases, such as user-uploaded review photos. The multitask visual representations are integrated into the visually similar ad recommendations as well as the brand new and built by Etsy Ads \textit{Search by Image} shopping experience. We give more details in the Appendix \ref{sec:search_by_image:appendix}.

\begin{figure}[h]
\includegraphics[width=3.25in]{images/multitask_architecture_3ds.png}
\centering
\caption{Multitask and multi-dataset visual representations training architecture. The data sampler combines examples from an arbitrary number of datasets corresponding to respective classification heads.}
\label{fig:viz-multitask-arch-3ds}
\end{figure}

\subsection{Text Representations}
\label{sec:fastText}

We pretrain word representations and leverage across a multitude of Etsy Ads applications. We employ the fastText library \footnote{https://fasttext.cc/}, which implements \cite{bojanowski2017enriching} and \cite{joulin2016bag}. The fastText paradigm learns subword-level information useful when building vectors for unknown words. We train on the primary text fields (title, description, tags) of all active Etsy listings.

\subsection{Multimodal Representations: Learning AIR}
\label{sec:air}

\subsubsection{Model design} We design Ads Information Retrieval (AIR) pretrained listing representations with the explicit goal of driving ad clicks. Therefore the ADPM's pretrained component in the adSformer CTR is parameterized by AIR. The AIR model is a neural network with a \textit{pseudo-two-tower} architecture, where the source and candidate towers share all trainable weights. The architecture with the shared encoder is pictured in Figure \ref{fig:air-encoder}.

\begin{figure}[h]
\includegraphics[width=3.25in]{images/AIR_model_architecture.png}
\centering
\caption{Learning AIR: pseudo-two-tower architecture with shared weights.}
\label{fig:air-encoder}
\end{figure}

\subsubsection{Learning} We compile a dataset of listing pairs that have been clicked together on various pages and we consider one listing as the "source" and the other listing as the "candidate" within the same example. The training dataset includes source-candidate pairs, where for each source we consider its corresponding candidate pair as the positive label, while a sample of the other examples in the same batch are the negative labels. For each of the source and candidate listings, we preprocess and concatenate into an input layer a set of multimodal features. For example, we concatenate the multitask visual representations described in Section \ref{sec:visual}, text representations of listing’s title, tags and taxonomy path average pooled from fastText pretrained representations described in Section \ref{sec:fastText}, and a series of other normalized features. We train the AIR model with a softmax loss \cite{pinterest19unified} and a classification objective. During training, we inference source and candidate 256-dimensional representations for each pair in the batch. We then compute a matrix of cosine similarity scores between each example’s source and candidate representations \cite{nigam2019semantic}. Finally, we compute the classification loss using these scores.

\subsection{Skip-Gram Listing Representations}
 \label{sec:skip-gram}

\subsubsection{Model design} Visual and AIR pretrained listing representations do not capture signals from the sequential browsing behavior of users within a web session. Co-viewed listings within a session are inherently related by virtue of having appealed to the particular style and tastes of a user with a specific intention. Therefore we view a sequence of listings in a browsing session as if they were words and learn a listing representation by employing the skip-gram \cite{skipgram} model, similarly to Airbnb \cite{grbovic2018real}, and to prior work at Etsy \cite{etsy2018iteminteraction}.

\subsubsection{Learning} Our training data consists of a set of listing views sequences. We learn a vector representation $\mathbf{v}_{l_i} \in \mathbb{R}^d$ of dimension $d=64$ for each unique listing in the training set. We use a hierarchical softmax loss function (\cite{morin2005hierarchical} and \cite{mnih2008scalable}), which showed much better results than classic negative sampling. We found that fastText library worked well for training, as long as we disabled functionality to consider subwords. For an evaluation set of search sessions with a purchase, we determine the representation quality by the average position of the purchased listing from a set of candidate listings sorted by representation cosine similarity to the clicked listing's representation within the session \cite{grbovic2018real}, illustrated in Figure \ref{fig:skip-gram-eval}.

\begin{figure}[h]
\includegraphics[scale=0.25]{images/skip-gram-embeddings-eval.png}
\centering
\caption{Search ranking evaluation metric on skip-gram listing representations. Lower ranking of purchased listings in each click group represent a higher degree of quality.}
\label{fig:skip-gram-eval}
\end{figure}

\section{adSformers Experimental Results}
\label{sec:experiments}

\subsection{Offline Experiments}

We evaluate the offline performance of adSformers using PR-AUC and ROC-AUC metrics. We run extensive offline experiments and ablation studies to study the effect of the ADPM in both the adSformer CTR and the adSformer PCCVR models. In Table \ref{tab:emb_eval_comparison} we can see how different configurations of the pretrained representation component in ADPM lead to different test ROC AUC and PR AUC. We notice that combining all three flavors of pretrained representations in CTR's ADPM configuration leads to the highest test ROC AUC. However we decided to include only AIR in the final adSformer CTR, since both ROC AUC and PR AUC are top ranking and memory requirements are lower. In PCCVR's ADPM configuration we found that skip-gram and visual representations performed best together in terms of PR AUC, however the ADPM configuration with only the adSformer encoder component and the learned component performed well. One hypothesis is that the adSformer encoder learns a signal similar to the signal from the pretrained skip-gram.

\begin{table}[h]
    \centering
    \footnotesize
     \caption{Ablation studies showing offline lift in metrics against the baseline, non-personalized, models for various configurations of the ADPM's pretrained representation component.}
    \begin{tabular}{lrrrr}
        \hline
        & \multicolumn{2}{c}{CTR} & \multicolumn{2}{c}{PCCVR} \\
        Pretrained Selection &
              ROC-AUC &
              PR-AUC &
              ROC-AUC &
              PR-AUC \\
        \midrule
        Skipgram &
              +1.99\% &
              +5.88\% &
              +1.35\% &
              +9.93\% \\
        Visual &
              +1.90\% &
              +5.37\% &
              +1.33\% &
              +10.21\% \\
        AIR &
              +2.14\% &
              \textbf{+6.44\%} &
              +1.23\% &
              +8.19\% \\
        Skipgram/Visual &
              +2.10\% &
              +5.71\% &
              +1.32\% &
              \textbf{+10.38\%}\\
        AIR/Skipgram &
              +2.20\% &
              +6.44\% &
              +1.25\% &
              +7.51\% \\
        AIR/Visual &
              +2.16\% &
              +6.22\% &
              +1.17\% &
              +8.54\% \\
        AIR/Skipgram/Visual &
              \textbf{+2.26\%} &
              +6.17\% &
              +1.14\% &
              +7.56\% \\
        No Pretrained Representations &
              +1.60\% &
              +4.89\% &
              \textbf{+1.37\%} &
              +10.24\% \\
        \bottomrule
    \end{tabular}
    \label{tab:emb_eval_comparison}
\end{table}

 The adSformer CTR and PCCVR differ in ADPM's pretrained component optimal configuration. We are not surprised, given that a user may have different intents during their browsing journey. For example, a buyer shown an ad impression is likely to be earlier in the purchase funnel and the click intent (CTR prediction) may be around shopping for inspiration or price comparisons. Conversely, post-click the purchase intent (PCCVR prediction) may be more narrowly focused on stylistic variations and shipping profiles of a candidate listing. We present additional offline comparison results in Appendix \ref{appendix:training}.

In Etsy Ads, we train models daily and evaluate on next-day test datasets. In Figure \ref{fig:cvr-offline-eval} we can see the PR AUC and ROC AUC lift for adSformer PCCVR and metrics variability across a period of ten days, when comparing to the baseline.

\begin{figure}[h]
\includegraphics[width=3.25in]{images/offline_eval_cvr.png}
\centering
\caption{Daily offline AUC evaluation of adSformer personalized PCCVR model against the baseline non-personalized PCCVR model from 2023-01-11 to 2023-01-21.}
\label{fig:cvr-offline-eval}
\end{figure}

\subsection{Online A/B Experiments}

To measure the effectiveness of the adSformer system, we deployed the best trained models to our production environment and evaluated them in online A/B tests. In Table \ref{tab:ads-exp} we have preliminary online test results. Compared to the current production model, the adSformer PCCVR demonstrates an increase in Return on Ad Spend (ROAS), PCCVR, and Gross Merchandised Sale (GMS). For Etsy Ads sellers this will translate to an increase in orders with larger sale values at a lower cost. The CTR model shows improved engagement metrics and the adSformer CTR and PCCVR models deployed in tandem for A/B test demonstrate even larger increases in ROAS and GMS when comparing to individual model contributions.

\begin{table}
    \centering
    \caption{Statistically significant online A/B experiment results (p-value < 0.001).}
    \label{tab:ads-exp}
    \begin{tabular}{lrrrr}
        \toprule
        Metric & adSformer CTR & adSformer PCCVR  & Tandem \\
        \cmidrule(r){1-1}\cmidrule(l){2-4}
        Ads CTR & +2.55\% & -0.02\% & +2.42\% \\
        Ads PCCVR & +0.29\% & +1.26\%  & +0.90\%\\
        Ads ROAS & +1.02\% & +3.51\% & +5.34\%\\
        Ads GMS & +3.02\% & +2.24\% & +4.17\% \\
        \bottomrule
    \end{tabular}
\end{table}

\subsection{Infrastructure Considerations in Deployed adSformers System}

Introducing short term sequences into a production training and serving path includes infrastructure challenges, and a heavy investment in streaming architecture. Thankfully, much of this infrastructure was already standardized across Etsy. Etsy's Feature System team uses a low latency streaming system powered by Kafka, PubSub for writing interaction event logs to a low latency feature store. The logged-in and logged-out user features are made available through a scalable micro service. The fetched in-session features are captured and preserved in training logs as close as possible to model serving, which minimizes skew between training and serving feature set. Our model serving system uses memcache to cache model scores thereby reducing latency and volume to downstream feature fetching and model inference services. The key used to write and read requests must be updated to include user and browser ID (for logged-out users). The real-time nature of the interaction sequences demands that either the cache is removed or that the expiration TTL is lowered dramatically. At scale, in our production systems, we pay close attention to latency and cloud cost surges due to increased volume to downstream services.

\section{Conclusion}

 Personalizing search ad ranking through the adSformer diversifiable personalization module (ADPM) is a leap toward meeting our goal of a highly personalized, best in class, customer experience, as demonstrated through strong A/B test results. In next steps we plan to extend ADPM to personalizing recommendation ads, as well as to the ad candidate retrieval step. Future work includes experimenting with the ADPM architecture, by further diversify components through graph representations, as well as other adSformer encoder architectures. Moreover, we plan to vary the duration of the short-term sequences (currently set to one hour), as well as experiment with the overall adSformer CTR-PCCVR architectures. Furthermore, we continue improving our pretrained representations through strengthening our ads computer vision capabilities. Finally, we plan to extend our artificial intelligence driven personalization efforts to expanding the seller toolbox via diffusion-based image and video generation.

\section{Acknowledgments}
We thank our colleagues in Etsy Ads ML for their related work leading to developing and deploying the adSformers. In alphabetical order: Julie Chien, Eric Herrmann, Julia Hoffman, Rezwana Karim, Mason Kirchner, Melissa Louie, Alec Malstrom, Seoyoon Park, Maharshi Thakker, Justin Tse, Gerardo Veltri. We also thank the Rivulet streaming features team, Information Retrieval Platform, and the ML Platform team for their support.

\bibliographystyle{ACM-Reference-Format}


United States Politicians' Tone Became More Negative with 2016 Primary Campaigns

Abstract: There is a widespread belief that the tone of US political language has become more negative recently, in particular when Donald Trump entered politics. At the same time, there is disagreement as to whether Trump changed or merely continued previous trends. To date, data-driven evidence regarding these questions is scarce, partly due to the difficulty of obtaining a comprehensive, longitudinal record of politicians' utterances. Here we apply psycholinguistic tools to a novel, comprehensive corpus of 24 million quotes from online news attributed to 18,627 US politicians in order to analyze how the tone of US politicians' language evolved between 2008 and 2020. We show that, whereas the frequency of negative emotion words had decreased continuously during Obama's tenure, it suddenly and lastingly increased with the 2016 primary campaigns, by 1.6 pre-campaign standard deviations, or 8% of the pre-campaign mean, in a pattern that emerges across parties. The effect size drops by 40% when omitting Trump's quotes, and by 50% when averaging over speakers rather than quotes, implying that prominent speakers, and Trump in particular, have disproportionately, though not exclusively, contributed to the rise in negative language. This work provides the first large-scale data-driven evidence of a drastic shift toward a more negative political tone following Trump's campaign start as a catalyst, with important implications for the debate about the state of US politics.

\section*{Results}

To quantify negative language in quotes, we used Linguistic Inquiry and Word Count (LIWC) \autocite{pennebaker2007liwc}, which provides a dictionary of words belonging to various
linguistic,
psychological,
and topical
categories,
and whose validity has been established by numerous studies in different contexts and domains \autocite{alpers2005cancer, kahn2007emotions, settanni2015facebook}.
We computed a negative-emotion score for each of the 24 million quotes via the percentage of the quote's constituent words that belong to LIWC's \textit{negative emotion} category.
Analogously, we computed scores for LIWC's three subcategories of the \textit{negative emotion} category---\textit{anger,} \textit{anxiety,} and \textit{sadness}---as well as for the \textit{swear words} category.
We collectively refer to these five categories as ``negative language''.%
\footnote{%
As a robustness check, we replicated the analysis using the dictionary provided by Empath \autocite{fast2016empath}
instead of LIWC.
In contrast to LIWC, the categories in Empath's dictionary were not hand-crafted but generated by a deep-learning model based on a small set of seed words.
The results, which are consistent with those from the LIWC-based analysis, are provided in Supplementary Fig.\xspace{} S1, S2, S3, and S4, and in Supplementary Table{} S1.}

In order to obtain a time series for each word category, we averaged individual quote scores by month.
By giving each quote the same weight when averaging, we obtain \textit{quote-level aggregates;}
by giving each speaker the same weight, \textit{speaker-level aggregates.}
(Formal definitions in \textit{Materials and Methods}.)
Quote-level aggregates give more weight to more frequently quoted politicians, and thus better capture the overall tone of political language as reflected in the news.
Whenever, on the contrary, we reason about politicians, rather than about the overall media climate created by all politicians' joint output, we use speaker-level aggregates as the more appropriate aggregation.

\Tabref{tab:scores} summarizes the prevalence of the above five word categories via the means and standard deviations of the respective quote-level aggregates during the pre-campaign period (September 2008 through May 2015).
We observe that
one in 54 words expresses a negative emotion;
one in 155, anger;
one in 339, anxiety;
and one in 285, sadness.
Swear words, at a rate of one in 2,329, are least common.
Given this wide range of frequencies, and in order to make effect sizes comparable across word categories, we standardize all monthly scores category-wise by subtracting the respective pre-campaign mean and dividing by the respective pre-campaign standard deviation.%
\footnote{%
In order to facilitate the comparability and interpretability of results, standardization always uses the means and standard deviations computed on quote-level aggregates involving all speakers, even in analyses of speaker-level aggregates or of quote-level aggregates when omitting individual speakers.}
The resulting effect sizes, in units of pre-campaign standard deviations, become more palpable when expressed as multiples of the corresponding pre-campaign means.
For this purpose, the rightmost column of \Tabref{tab:scores} lists coefficients of variation, \abbrevStyle{i.e.}\xspace, ratios of standard deviations and means.

\begin{table}
\centering
\caption{
\textbf{Overall statistics of LIWC negative-language word categories.}
Means $\mu$ and standard deviations $\sigma$ were calculated over monthly quote-level aggregates from the pre-campaign period (September 2008 through May 2015).
The value $n=1/\mu$ in the fourth column implies that, in an average quote, on average every $n$-th word belongs to the respective category.
The coefficients of variation, $\sigma/\mu$, shown in the fifth column, allow to easily translate pre-campaign standard deviations (as shown on the $y$-axes of time series plots) into fractions of pre-campaign means.
The most frequent words per category are listed in Supplementary Tables{} S2 and S3.
}
\label{tab:scores}
\begin{tabular}{lrrrr}
    \toprule
    Word category     &    Mean $\mu$ & Std.\ dev.\ $\sigma$ & $1/\mu$ & $\sigma/\mu$ \\
    \midrule
    Negative emotion &  0.01853 &  0.00090 &      54 &          4.8\% \\
    Anger            &  0.00647 &  0.00055 &     155 &          8.4\% \\
    Anxiety          &  0.00295 &  0.00022 &     339 &          7.3\% \\
    Sadness          &  0.00351 &  0.00014 &     285 &          4.1\% \\
    Swear words      &  0.00043 &  0.00015 &    2,329 &         33.9\% \\

    \bottomrule
\end{tabular}
\end{table}

\begin{figure*}[htb]
	\centering
	\includegraphics[width=\linewidth]{img/Negativity.pdf}
	\caption{
	\textbf{Temporal evolution of negative language.}
	Columns correspond to negative-language word categories from LIWC;
	rows correspond to aggregation methods for computing monthly averages.
	Points show monthly averages, expressed as pre\hyp campaign $z$-scores (\abbrevStyle{i.e.}\xspace, subtracting pre\hyp campaign mean from raw frequency values, and dividing by pre\hyp campaign standard deviation).
	Lines (with 95\% confidence intervals) were obtained via ordinary least squares regression, with coefficients shown in legends
	(\abbrevStyle{cf.}\xspace\ \Eqnref{eqn:regression} and \Figref{fig:fig0}(b) for interpretation of coefficients; tabular summary in Supplementary Tables{} S4, S6, S8, S9).
	\textbf{\mbox{(a--e)}}~\textit{Quote-level aggregation} micro\hyp averages over all quotes per month, \abbrevStyle{i.e.}\xspace, speakers have weight proportional to their number of quotes in the respective month. Panel (a) shows the same data as \Figref{fig:fig0}.
	\textbf{(f--j)}~\textit{Speaker-level aggregation} macro\hyp averages by speaker, \abbrevStyle{i.e.}\xspace, all speakers with at least one quote in a given month have equal weight in that month.
	\textbf{(k--o)}~\textit{Quote-level aggregation by party} performs the analysis of (a--e), but separately for quotes from Democrats \abbrevStyle{vs.}\xspace\ Republicans (coefficients omitted for clarity; \abbrevStyle{cf.}\xspace\ Supplementary Tables{} S8 and S9).
	Significance of regression coefficients: ***~$p<0.001$, **~$p<0.01$, *~$p<0.05$.
	We observe drastic shifts toward a more negative tone at the modeled June 2015 discontinuity (Trump's campaign start).
	}
	\label{fig:emotions}
\end{figure*}

\xhdr{Temporal evolution of negative language}
\Figref{fig:emotions} visualizes the evolution of negative language between September 2008 and April 2020, with
one row per aggregation method,
one column per word category,
and one point per monthly aggregate score.
In order to quantify the shape of the curves, we fitted ordinary least squares linear regression models with a discontinuity in June 2015, the starting month of Donald Trump's primary campaign.
For a given word category, we model the aggregate score $y_t$ for month $t$ as
\begin{align}
	y_t &= \alpha_0
	+ \beta_0 \,t
	+ \alpha \,i_{t}
	+ \beta \,i_{t} \,t
	+ \varepsilon_{t},
\label{eqn:regression}
\end{align}
where $t \in \{-81, \dots, 58\}$, with $t=0$ corresponding to June 2015;
$i_t$ indicates whether $t$ is located before \abbrevStyle{vs.}\xspace\ after the campaign start (\abbrevStyle{i.e.}\xspace, $i_t=0$ for $t < 0$, and $i_t=1$ for $t \geq 0$);
and $\varepsilon_t$ is the residual error.
As illustrated in \Figref{fig:fig0}(b), the coefficient $\alpha$ captures the immediate jump coinciding with the campaign start,
and $\beta$, the change in slope,
such that $\beta_0$ and $\beta_0 + \beta$ describe the slopes of the regression lines before and after the campaign start, respectively.
\Figref{fig:emotions} plots the fitted regression lines, alongside 95\% confidence intervals.

We first focus on the time series of quote-level aggregates (\Figref{fig:emotions}(a--e)).
The regression coefficients indicate a significant, sudden increase in the relative frequency of negative emotion words (\Figref{fig:emotions}(a)) in June 2015, by $\alpha=1.6$ ($p=4.1\times 10^{-6}$) pre-campaign standard deviations (SD),
translating to a relative increase of 8\% over the pre-campaign mean (\abbrevStyle{cf.}\xspace\ \Tabref{tab:scores}).
All three subcategories of negative emotion words, as well as swear words, also saw significant jumps in June 2015:
anger, by 1.3 SD ($+11\%$ of the pre-campaign mean, $p=0.0015$, \Figref{fig:emotions}(b));
anxiety, by 1.5 SD ($+11\%$, $p=5.6\times 10^{-4}$, \Figref{fig:emotions}(c));
sadness, by 0.86 SD ($+4\%$, $p=0.021$, \Figref{fig:emotions}(d));
and swear words, by 0.92 SD ($+31\%$, $p=2.9\times 10^{-6}$, \Figref{fig:emotions}(e)).%
\footnote{Note that the pre-campaign regression line for swear words (\Figref{fig:emotions}(e)) underestimates the values just before the discontinuity, mostly due to outliers at the left boundary (2008/09). For swear words, the actual June 2015 jump should thus be considered to be smaller than the estimate $\alpha$.
}
The June 2015 discontinuity was not only associated with a sudden increase in negative language, but also with a change in slope:
whereas the frequency of negative emotion words (\Figref{fig:emotions}(a)) had steadily and significantly decreased over the first 6.5 years of Obama's tenure by
$\beta_0=-0.023$ SD per month ($p=9.5 \times 10^{-7}$),
\abbrevStyle{i.e.}\xspace, by about a quarter SD per year,
this trend came to a halt in June 2015, with a (non-significantly) positive slope of $\beta_0+\beta=0.0076$ (compound $p=0.21$) from June 2015 onward.%
\footnote{%
These results hold when removing outlier months whose quote-level aggregate score lies more than three standard deviations from the mean (Supplementary Fig.\xspace{} S1, and Supplementary Tables{} S5 and S7).}

We emphasize that the June 2015 discontinuity was chosen \textit{ex ante} based on incoming hypotheses grounded in the general public's subjective impression that Donald Trump's entering the political scene had changed the tone of US politics~\autocite{pew2019discourse}.
A data-driven analysis, conducted \textit{ex post,} revealed that June 2015 is in fact the optimal discontinuity for modeling the data:
out of 140 regression models analogous to \Eqnref{eqn:regression}, but each using another one of the 140 months of our analysis period as the discontinuity, the model with the June 2015 discontinuity yielded the best fit for negative emotion words and anger (Supplementary Fig.\xspace{} S5).
(For anxiety and sadness, slightly better fits were obtained when using different discontinuities; for swear words, a 2009 discontinuity led to a better fit due to outliers around that time; see Supplementary Fig.\xspace{} S5.)

\xhdr{Role of speaker prominence}
When repeating the analysis using speaker-level, rather than quote-level, aggregates, \abbrevStyle{i.e.}\xspace, weighing all speakers equally when averaging,
all of the above effects persisted qualitatively, but were reduced quantitatively (\Figref{fig:emotions}(f--j)):
in each of the five word categories, the immediate increase $\alpha$ at the June 2015 discontinuity dropped by between one-third and one-half,
remaining significant for negative emotions, anger, and swear words ($p=9.1\times 10^{-5},$ 0.0025, and 0.030, respectively),
but becoming non-significant for anxiety and sadness ($p=0.11$ and $0.43$, respectively).
The change of slope observed in the quote-level analysis also persisted in the speaker-level analysis.

The fact that speaker-level effects are weaker than quote-level effects indicates that prominent, highly quoted speakers contribute disproportionally to the increase in negative language.
To confirm this conclusion more directly, we divided the speakers into four quartiles with respect to the number of quotes attributed to them, and repeated the speaker-level analysis for each quartile individually.
\Figref{fig:verbosity} shows
(1)~that the abrupt increase in negative emotion words emerges in all strata of speaker prominence except the least prominent stratum, and
(2)~that quotes by more prominent speakers generally contain more negative emotion words.
(Results for other word categories in Supplementary Fig.\xspace{} S3.)

\begin{figure*}[htb]
	\centering
	\includegraphics[width=\linewidth]{img/verbosity_negemo.pdf}
	\caption{
		\textbf{Role of speaker prominence.}
		The set of 18,627 US politicians was split into four evenly-sized quartiles with respect to their total number of quotes (\abbrevStyle{i.e.}\xspace, prominence);
		each panel shows the time series for negative emotion words obtained by performing monthly speaker-level aggregation on the respective quartile separately.
		That is, the figure shows the data of \Figref{fig:emotions}(f) after stratifying speakers by prominence.
		Lines (with 95\% confidence intervals) were obtained via ordinary least squares regression, with coefficients in legends (\abbrevStyle{cf.}\xspace\ \Eqnref{eqn:regression} and \Figref{fig:fig0}(b) for interpretation of coefficients).
		(Tabular summary of regression coefficients in Supplementary Tables{} S11, S12, S13, and S14.)
		Significance of regression coefficients: ***~$p<0.001$, **~$p<0.01$, *~$p<0.05$.
		We observe that the abrupt increase in negative emotion words emerges in all strata of speaker prominence except the least prominent stratum, and that quotes by more prominent speakers overall contain more negative emotion words.
		The figure focuses on one category of negative\hyp language words (negative emotion); for the other four categories, see Supplementary Fig.\xspace{} S3.
    }
	\label{fig:verbosity}
\end{figure*}

\xhdr{Biographic correlates of negative language}
The patterns identified above---a sudden increase in negative language in June 2015 followed by a change in slope---hold across party lines,%
\footnote{All $\alpha$ coefficients of \Figref{fig:emotions}(k--o) are positive. For Republicans, all are significant ($p<0.05$); for Democrats, the coefficients for anger ($\alpha=0.57$) and sadness ($\alpha=0.43$) are non-significant ($p>0.05$).}
but are more pronounced for Republicans,
as seen in \Figref{fig:emotions}(k--o), which tracks the (quote-level) evolution of negative language over time, analogously to \Figref{fig:emotions}(a--e), but separately for quotes by Democrats \abbrevStyle{vs.}\xspace\ Republicans.
For instance, the party-wise estimates of the June 2015 increase in negative emotion words is
$\alpha=0.89$ pre-campaign SD for Democrats ($p=0.014$)
and $\alpha=2.3$ SD for Republicans ($p=5.6 \times 10^{-9}$).

We further considered the possibility that the distribution of speaker characteristics may have changed over time; \abbrevStyle{e.g.}\xspace, members of one party or gender may have become more frequently quoted over time.
To account for potential confounding due to such factors, we repeated the
regression analysis with added control terms for four biographic attributes:
party affiliation (Republican, Democrat),
the party's federal role (Opposition, Government),
Congress membership (Non-Congress, Congress),
and gender (Male, Female).%
\footnote{Due to small sample size, we discarded speakers of a non-binary gender according to Wikidata.}
For a given month, party affiliation fully determines the party's federal role, so for each month, the set of speakers can be partitioned into $2^3=8$ speaker groups, one per valid attribute combination.
We computed monthly aggregate scores $y_{gt}$ separately for each speaker group $g$, obtaining eight aggregate data points per month, and modeled them jointly using the following extended version of \Eqnref{eqn:regression}:
\begin{align}
	y_{gt} &= \alpha_0
	+ \beta_0 \,t
	+ \alpha \,i_{t}
	+ \beta \,i_{t} \,t
	+ \gamma \,\text{Democrat}_g
	+ \delta \,\text{Government}_g
	+ \zeta \,\text{Congress}_g
	+ \eta \,\text{Female}_g
	+ \varepsilon_{gt},
\label{eqn:extended_regression}
\end{align}
where $\text{Democrat}_g=1$ (or 0) if group $g$ contains Democrat (or Republican) speakers, and analogously for $\text{Government}_g$, $\text{Congress}_g$, and $\text{Female}_g$.

Inspecting the fitted coefficients (shown in \Figref{fig:coefficients} for quote-level aggregation; for speaker-level aggregation, see Supplementary Fig.\xspace{} S6), we make two observations:
First, the sudden June 2015 jump $\alpha$ in negative language (\Figref{fig:coefficients}(b)) appears even after adjusting for the four biographic attributes.
Second, we observe systematic correlations of negative language with biographic attributes, whereby quotes by members of the opposition party, Congress members, and Democrats contain, \textit{ceteris paribus,} significantly more negative language.
Moreover, quotes by females contained more anxiety and sadness, and quotes by males, more negative emotion, anger, and swear words.
The fact that quotes by members of the opposition party contain more negative language may reflect their role as corrective agents in the democratic process.
The fact that quotes by Congress members contain more negative language echoes previous work that has highlighted high-ranked politicians' role not only as deputies of their own party, but also as antagonists of the opposing party, epitomized in the finding that ``ideological moderates won't run'' \autocite{thomsen2014moderates} for Congress.

\begin{figure*}[htb]
	\centering
	\includegraphics[width=\linewidth]{img/coefficients.pdf}
	\caption{
	\textbf{Biographic correlates of negative language.}
	Coefficients (with 95\% confidence intervals) of ordinary least squares regression (\Eqnref{eqn:extended_regression}) for modeling time series of word categories (quote-level aggregation; speaker-level aggregation in Supplementary Fig.\xspace{} S6) while adjusting for
	party affiliation ($\gamma$),
    the party's federal role ($\delta$),
    Congress membership ($\zeta$),
    and gender ($\eta$)
    (tabular summary in Supplementary Tables{} S15 and S16).
    Positive coefficients mark word categories that are, \textit{ceteris paribus,} used more frequently
    by Democrats than by Republicans,
    by members of the governing than by members of the opposition party,
    by Congress members than by others,
    or by females than by males
    (and \textit{vice versa} for negative coefficients).
    We observe that quotes by members of the opposition party, Congress members, and Democrats contain significantly more negative language.
    Importantly, the sudden June 2015 jump in negative language ($\alpha$) remains significant after adjusting for biographic attributes.
	}
	\label{fig:coefficients}
\end{figure*}

\xhdr{Role of individual politicians}
In order to determine to what extent the above population-level effects mirror individual-level effects, we fitted regression models (\abbrevStyle{cf.}\xspace\ \Eqnref{eqn:regression}) to individual speakers' time series and analyzed the corresponding $\alpha$ coefficients (\abbrevStyle{i.e.}\xspace, the size of the June 2015 jump; for completeness, $\beta$ in Supplementary Fig.\xspace{} S7).
In order to avoid data sparsity issues, this analysis focuses on the 200 most-quoted speakers, with $\alpha$ plotted in \Figref{fig:individuals_alpha}(a--e).
Additionally, \Figref{fig:individuals_alpha}(f--j) plots the fraction of speakers with
positive $\alpha$ among the speakers with at least $n$ quotes, as a function of $n$.
We observe that, although many individual $\alpha$ coefficients are non-significant ($p>0.05$, gray confidence intervals in \Figref{fig:individuals_alpha}(a--e)), the majority of coefficients are positive, particularly among the top most-quoted speakers (as manifested in the increasing curves of \Figref{fig:individuals_alpha}(f--j)).
For instance, for negative emotion words (\Figref{fig:individuals_alpha}(f)),
all of the top four most-quoted politicians have positive $\alpha$.
Among the top 50, 74\% have positive $\alpha$;
among the top 100, 63\%;
and among the top 200, 59\%.
In other words, the June 2015 discontinuity emerges not only by aggregating at the population level, but mirrors a disruption that can also be perceived in the most-quoted politicians' individual language.
We further illustrate (Supplementary Fig.\xspace{} S8) using as examples the two presidents (Barack Obama, Donald Trump), vice presidents (Joe Biden, Mike Pence), and runners-up (Mitt Romney, Hillary Clinton) from the study period.
Four of these six speakers were associated with a significant ($p<0.05$) positive $\alpha$ for negative emotion words, and none with a significant negative $\alpha$.%
\footnote{Interestingly, although both Donald Trump ($\alpha=3.6$, $p=0.026$) and Barack Obama ($\alpha=3.7$, $p=0.0094$) followed the population-wide pattern by increasing their frequency of negative emotion words in June 2015, their pre- and post-discontinuity slopes were opposite to the population-wide pattern: their language first became more negative, then less negative.}

\begin{figure*}[htb]
	\centering
 	\includegraphics[width=\linewidth]{img/individuals_combined.pdf}
	\caption{
	\textbf{Role of individual politicians: single-speaker study.}
	Results of ordinary least squares regressions (\Eqnref{eqn:regression}) fitted separately to the time series of each of the 200 most quoted speakers.
	\textbf{(a--e)}~Each speaker's $\alpha$ coefficient (capturing the size of the June 2015 jump, with 95\% confidence intervals) as a function of the speaker's number of quotes.
	Significant coefficients ($p<0.05$) in color, others in gray.
	\textbf{(f--j)}~Fraction of speakers with positive $\alpha$ among the speakers with at least $n$ quotes, as a function of $n$.
	We observe that, although many individual $\alpha$ coefficients are non-significant (a--e), the majority of coefficients are positive, particularly among the most-quoted speakers (as manifested in the increasing curves of (f--j)). That is, the June 2015 jump in negative language emerges even at the
	individual level for a majority of the most-quoted politicians.
	}
	\label{fig:individuals_alpha}
\end{figure*}

Next, we sought evidence whether individual politicians contributed particularly strongly to the overall increase in negative language.
We proceeded in an ablation study: for each of the 50 most quoted speakers, we repeated the quote-level regression (\Figref{fig:emotions}(a--e)) on a dataset consisting of all quotes except those from the respective speaker.
If $\alpha$ is particularly low when removing a given speaker, that speaker contributed particularly strongly to the overall June 2015 increase in negative language.
\Figref{fig:ablation}(a) shows that no single speaker's removal leads to an important change in $\alpha$ for negative emotion words, with one exception: Donald Trump.
By removing Donald Trump's quotes, the June 2015 increase in negative emotion words drops by 40\%, from $\alpha=1.6$ to $\alpha=0.98$ pre-campaign SD.%
\footnote{
More precisely, $(1.622-0.974)/1.622 = 0.40$.
When considering quotes by Republicans only (\abbrevStyle{cf.}\xspace\ \Figref{fig:emotions}(k--o)), $\alpha$ drops by 43\% when removing Trump's quotes, from 2.3 to 1.3 pre-campaign SD (Supplementary Tables{} S9 and S10).}
Put differently, by adding Donald Trump's quotes, the June 2015 increase in negative emotion words is boosted by 63\%.
Note that this is not merely an artifact of Trump's being quoted especially frequently: Obama was quoted about twice as frequently as Trump over the course of the 12-year period, yet removing his quotes does not notably affect the June 2015 increase in negative emotion words.
Qualitatively similar results hold for the other word categories, in particular for swear words (\Figref{fig:ablation}(e); $\beta$ in Supplementary Fig.\xspace{} S9 for completeness).
Although the size of the June 2015 jump in negative emotion words decreased drastically when removing Trump's quotes, note that it remained highly significant ($p=0.0032$).
That is, Trump was the main, but not the sole, driver of the effect.

\begin{figure*}[htb]
	\centering
	\includegraphics[width=\linewidth]{img/ablation_alpha.pdf}
	\caption{
	\textbf{Role of individual politicians: ablation study.}
	Results from ordinary least squares regression (\Eqnref{eqn:regression}; quote-level aggregation) on data sets obtained by removing all quotes by one target speaker and retaining all quotes from all 18,626 other speakers, using each of the 50 most quoted speakers as a target speaker.
	Each point shows the $\alpha$ coefficient (capturing the size of the June 2015 jump, with 95\% confidence intervals) obtained after removing the target speaker's quotes from the analysis, as a function of the target speaker's number of quotes.
	Dashed horizontal lines mark the coefficients obtained on the full data set without removing any speaker (\abbrevStyle{cf.}\xspace\ \Figref{fig:emotions}(a--e)).
	We observe that no single speaker's removal leads to a notable change in $\alpha$, except Donald Trump: \abbrevStyle{e.g.}\xspace, by removing Trump's quotes, $\alpha$ drops by 40\% for negative emotion words (a).
	}
	\label{fig:ablation}
\end{figure*}

\xhdr{Positive emotion words}
Complementary to the five word categories related to negative language analyzed above, we also conducted an exploratory analysis of positive language, as captured by LIWC's \textit{positive emotion} category.
As seen in \Figref{fig:posemo}(a), the time series of positive emotion words does not simply mirror that of negative emotion words. In particular, no particular changes are observed in June 2015. Rather, positive emotion words appear stable well into Trump's term, and then decline.
(The best regression fit is achieved when using July 2018 as the discontinuity, with a sharp drop in quality of fit after February 2019; Supplementary Fig.\xspace{} S10.)
As in the case of negative emotion words, the same pattern emerges within each party (\Figref{fig:posemo}(c)), with a more pronounced drop in positive emotion words for Democrats than for Republicans.

\begin{figure*}[htb]
	\centering
	\includegraphics[width=\linewidth]{img/Positive.pdf}
	\caption{
	\textbf{Temporal evolution of positive emotion words.}
	Points show monthly averages, expressed as pre\hyp campaign $z$-scores.
	One panel per aggregation method for computing monthly averages.
    We observe that positive emotion words appear stable well into Trump's term, and then decline.

    }
	\label{fig:posemo}
\end{figure*}

\section*{Discussion}

The goal of this work has been to determine the accuracy of the average American's subjective impression (as of mid 2019)
that ``the tone and nature of political debate in the United States has become more negative in recent years'', and
that Donald Trump ``has changed the tone and nature of political debate [\dots{}] for the worse''~\autocite{pew2019discourse}.
Based on an analysis of 24 million quotes uttered by 18,627 US politicians between 2008 and 2020, we conclude that both of the above impressions are largely correct.

US politicians' tone indeed became suddenly and significantly more negative with the start of the 2016 primary campaigns in June 2015, and the frequency of negative language has remained elevated ever since.
Intriguingly, the shift at this incision point coincides with a similar abrupt shift in political polarization on online platforms \autocite{anderson2021social, frimer2022incivility}.
The sudden increase in negative language reported here was not only significant, but also strong; \abbrevStyle{e.g.}\xspace, the frequency of negative emotion words jumped up by 1.6 pre-campaign standard deviations, or by 8\% of the pre-campaign mean.
The disruption becomes particularly stark when contrasted with the first 6.5 years of Obama's tenure, during which negative language had decreased steadily---at odds with a commonly held belief that Trump merely continued an older trend \autocite{fritze2021trump,james2021effects}.
The potential of negativity, incivility, and fear as tools to support political campaigns
has been long known \autocite{brooks2007incivility, gerstle2019negativity} and might explain the increase of negative language during the campaigns.
It cannot, however, explain the fact that the boost in negative language has continued for years after the campaigns had ended.

Rather, our results show that political debate during Donald Trump's entire term was characterized by a negative tone, and they specifically point to Trump as a key driver of this development:
when removing Trump's quotes from the corpus, the magnitude of the June 2015 jump in the frequency of negative emotion words drops by 40\%.
Interestingly, Trump's own negative tone (Supplementary Fig.\xspace{} S8(z)) followed long-term trends opposite to the population-wide trends, with an initial increase and a subsequent decrease (and with a sudden June 2015 increase akin to that of the population).
But as his language was overall far more negative than the average
(Trump's mean monthly average was
8.0 pre-campaign standard deviations larger than the overall mean monthly average), he strongly skewed the overall tone toward the negative end when he moved to the center of the media's attention \autocite{patterson2017news}.

Despite Trump's disproportionate impact, the increase in negative language was, however, not due to Trump alone.
It remained significant in various complementary analyses:
when removing Trump's quotes from the analysis,
when giving equal weight to all speakers,
when analyzing each party separately,
and when analyzing the most-quoted speakers individually.
The negative tone of others might be partly provoked by Trump's statements and actions, but as we do not have access to a counterfactual world without Trump, our analysis cannot speak to this possibility.

Our analysis also cannot disambiguate to what extent the observed shift in negative language is caused by a real shift in what politicians say \abbrevStyle{vs.}\xspace\ a shift in what the media choose to report.
It is well known that media outlets are biased in what they report \autocite{gentzkow2010drives, puglisi2011newspaper, schiffer2006assessing}, typically towards negative news that tend to increase engagement on readers' behalf \autocite{grabe2006negativenews}, and this bias may have drifted during the 12 years analyzed here.
Future work may investigate this possibility by comparing quotes reported in the media to complete records of certain politicians' utterances in certain contexts, \abbrevStyle{e.g.}\xspace, via Congressional records or public speeches \autocite{niculae2015quotus}.
(But whether the shift was caused by politicians or by the media, the effect would be identical: a more negative tone as perceived by news-reading citizens.)

Moreover, although word-counting is a powerful tool for detecting broad trends in large textual data \autocite{pennebaker2001personality, kahn2007emotions}, and although our results are robust to the specific choice of dictionary (Empath \autocite{fast2016empath} yields qualitatively identical results to LIWC \autocite{pennebaker2007liwc}; Supplementary Fig.\xspace{} S1, S2, S3, S4, and S11, and Supplementary Table{} S1), word-counting is crude and insensitive to nuances in context \autocite{beasley2015emotions}.
Our findings should thus be considered a starting point and hypothesis generator for more detailed analyses based on a closer inspection of the text and context of quotes, \abbrevStyle{e.g.}\xspace, using a combination of advanced natural language processing tools and human annotation in order to shed light on the relation between negative language and polarization and partisanship: Has politicians' tone become more negative specifically when they talk about opponents?

The present study is also limited with respect to its time frame.
Although Quotebank \autocite{vaucher2021quotebank} is the largest existing corpus of speaker-attributed quotes in terms of size and temporal extent, the 12 years it spans are but a short sliver of the United States' long political history.
It is thus unclear when the decrease in negative language at the beginning of the corpus (during the initial 6.5 years of Obama's tenure) started.
Gentzkow \abbrevStyle{et al.}\xspace\ \autocite{gentzkow2019partisanship} observed a concurrent decrease in partisan language in Congressional speeches during Obama's tenure (2009--2016), which might be yet another manifestation of the processes that underlie the initial decrease in negative language observed here.
Gentzkow \abbrevStyle{et al.}\xspace's corpus of Congressional speeches, however, spans a much longer time (144 years, 1873--2016), during which overall partisanship increased much more---especially from the 1980s onward---than it eventually decreased during Obama's tenure.
We must therefore consider the possibility that, analogously, the 2009--2015 decrease in negative language may have been merely a short anomaly in a longer increasing trend in negative language.
(Unfortunately, Gentzkow \abbrevStyle{et al.}\xspace's corpus ends with Obama's tenure, so we cannot compare trends in negative language to trends in partisan language during Trump's tenure.)

We saw that the June 2015 rise in negative language was not accompanied by a simultaneous drop in positive language. Rather, maybe in line with a general positivity bias in human language \autocite{Dodds2015human}, positive language remained stable until it eventually dropped during Trump's term.
What happened at this point is an open question that lies beyond the scope of this work.
The phenomenon highlights, however, that positive and negative emotion words are not necessarily complementary.
From the start of the 2016 primary campaigns through the first half of Donald Trump's term, political tone was both highly positive and highly negative---akin to Trump's own style, characterized by typical features of affective polarization such as positive self-representation and negative other-presentation \autocite{ahmadian2017trump, kreis2017tweets, lewandowsky2020using}:
``Sorry losers and haters, but my I.Q.\ is one of the highest -and you all know it! Please don't feel so stupid or insecure,it's not your fault''.%
\footnote{\url{https://web.archive.org/web/20151013150633/https://twitter.com/realDonaldTrump/status/332308211321425920}}

This said, a key contribution of this work is the conclusion that, despite Donald Trump's key role in setting the tone of political debate, the shift towards a more negative tone permeates all of US politics.
The consequences are tangible, as shown by research that highlights the detrimental effects of affective polarization on altruism \autocite{whitt2020tribalism}, trust \autocite{mutz2005videomalaise}, and opinion formation \autocite{druckman2021affect, forgette2006television},
and by polls showing that politics has become a stressful experience for Americans \autocite{pew2019discourse, apa2017coping}, exacting an ever increasing toll on their physical, emotional, and social well-being \autocite{Smith2019cost, apa2018genZ}.
Finding ways to break out of this cycle of negativity is one of the big challenges faced by the United States today.

\section*{Materials and Methods}

\xhdr{US politicians}
We considered as politicians all people for whom the Wikidata knowledge base \autocite{wikidata} (version of 27 October 2021) lists ``politician'' (Wikidata item Q82955) or a subclass thereof as an occupation (P106).
Given our focus on the United States, we included only those politicians whose party affiliation (P102) was listed as Democrat (Q29552) or Republican (Q29468).
We considered as members of Congress those for whom Wikidata listed a US Congress Bio ID%
\footnote{For instance, \url{https://bioguide.congress.gov/search/bio/S000148}}
(P1157), making no distinction between former and active members of Congress.
Due to the small number of speakers of a non-binary gender, we included only speakers whose gender (P21) was listed in Wikidata as male (Q6581097) or female (Q6581072).

\xhdr{Quotebank}
The analyzed quotes were obtained from Quotebank \autocite{vaucher2021quotebank}, a publicly available \autocite{vaucher2021zenodo} corpus of 235 million unique speaker-attributed quotes extracted from 127 million English news articles published between September 2008 and April 2020, provided by the large-scale online media aggregation service Spinn3r.com.
While Spinn3r.com collects and supplies content from a comprehensive set of news domains \autocite{west2021postmortem}, it also includes much content beyond news alone, including ``social media, weblogs, news, video, and live web content'' \autocite{spinn3r_doc}.
Therefore, Quotebank was extracted from a filtered data set consisting only of content from a set of about 17,000 online news domains, defined as the set of domains appearing at least once in the large \textit{News on the Web} corpus \autocite{nowCorpus}, which has been collecting large numbers of news articles from Google News and Bing News since 2010 and may thus be considered to provide a comprehensive list of English-language media outlets.
We emphasize that the \textit{News on the Web} corpus was only used for defining the set of news domains. It was not used for obtaining the news articles themselves, which originated from Spinn3r.com only.

We use the quote-centric (as opposed to the article-centric) version of Quotebank, which contains one entry per unique quote and aggregates information from all news articles in which the quote occurs.
In constructing Quotebank, a machine learning algorithm (based on the large pre-trained BERT language model \autocite{DevlinCLT19bert}) was used to infer, for each quote, a probability distribution over all speaker names mentioned in the text surrounding the quote (and an additional ``no speaker'' option), specifying each speaker's estimated probability of having uttered the quote.
For a given quote, we maintained only the name with the highest probability and consider it to indicate the speaker of the quote (a heuristic that was shown to have an accuracy of around 87\% \autocite{vaucher2021quotebank}).
A speaker name may be ambiguous. In such cases, Quotebank does not attempt to disambiguate the name,
but rather provides a list of all speakers to whom the name may refer, where speakers are identified by their unique ID from the Wikidata knowledge base \autocite{wikidata}.
In our analysis, we attributed quotes that were linked to ambiguous speaker names (less than 6\% of all quotes, see below) to each speaker to whom the respective name may refer.

To further clean the data set, we discarded quotes that were clearly non-verbal (\abbrevStyle{e.g.}\xspace, consisting of URLs, HTML tags, or dates only).
Moreover, on some days, Spinn3r.com, which provided the news articles for Quotebank, failed to deliver content due to technical problems \autocite{vaucher2021quotebank}.
We therefore identified missing days as those having less than 10\% of the median number of unique quotes and dropped eight (out of 140) months with 20 or more missing days:
May 2010,
June 2010,
January 2016,
March 2016,
June 2016,
October 2016,
November 2016,
January 2017.

\xhdr{Quotes by US politicians}
Keeping only quotes attributed to US politicians (see above), we obtained 24 million quotes attributed to 18,627 unique US politicians.
Out of these,
4,487 were female, 14,140 male;
9,390 were Democrats, 9,237 Republicans;%
\footnote{
Out of an original set of 18,954 US politicians appearing in Quotebank, Wikidata listed 327 as (former) members of both parties, usually because they switched membership during their careers. Out of these 327, we manually checked the 21 politicians with over 10,000 quotes, of whom 16 could be unambiguously assigned to one party for the study period. The remaining 311 politicians were dropped from the analysis.}
and 1,790 were labeled as members of Congress.
The most prolific speakers were
Barack Obama (1.5m quotes),%
\footnote{It seems unreasonable that any single person could utter over 300 different quote-worthy statements a day. Instead, the large number can be explained by news outlets attending to different parts of a politician's spoken output. For instance, a long sentence can be quoted in various ways.}
Donald Trump (763k quotes),
Mitt Romney (281k quotes),
Hillary Clinton (230k quotes),
George W.\ Bush (200k quotes),
John McCain (161k quotes),
and Joe Biden (127k quotes).
For a list of the 30 most frequently quoted politicians, see Supplementary Table{} S17.
Although, as mentioned, ambiguous names led to some quotes being attributed to multiple speakers, this happened rarely:
the vast majority (94.3\%) of quotes were attributed to a single politician,
4.3\% to two politicians,
1.2\% to three politicians,
and 0.13\% to four politicians.
Supplementary Fig.\xspace{} S12 shows the number of quotes and the number of unique speakers per month.

\xhdr{Aggregation methods}
Consider a fixed LIWC word category $c$ and a fixed month $t$.
Let $S$ be the set of speakers with at least one quote during month $t$.
Let $Q_s$ be the set of quotes attributed to speaker $s \in S$ during month $t$,
and let $Q = \bigcup_{s \in S} Q_s$ be the set of all quotes from month $t$.%
\footnote{The (few) quotes attributed to multiple speakers are included in $Q$ once per speaker.}
Let $\score{q}$ be quote $q$'s score for word category $c$.

Then, the \textit{quote-level aggregate score} for word category $c$ in month $t$ is defined as
$$ y^\text{quote}_t
= \frac{1}{|Q|} \sum_{q \in Q} \score{q}
= \sum_{s \in S} \underbrace{\frac{|Q_s|}{|Q|}}_\text{depends on $s$}  \underbrace{\left[ \frac{1}{|Q_s|} \sum_{q \in Q_s} \score{q} \right]}_\text{average of speaker $s$},$$
and the \textit{speaker-level aggregate score,} as
$$ y^\text{speaker}_t
= \frac{1}{|S|} \sum_{s \in S} \left( \frac{1}{|Q_s|} \sum_{q \in Q_s} \score{q} \right)
= \sum_{s \in S} \underbrace{\frac{1}{|S|}}_{\text{constant}} \underbrace{\left[ \frac{1}{|Q_s|} \sum_{q \in Q_s} \score{q} \right]}_\text{average of speaker $s$}. $$
That is, in quote-level aggregation, every speaker contributes with weight proportional to their number of quotes,
whereas in speaker-level aggregation, all speakers contribute with equal weight.

\xhdr{Data availability}
The Quotebank corpus is publicly available on Zenodo at \url{https://doi.org/10.5281/zenodo.4277311}.
Aggregated data derived from Quotebank are available on GitHub at \url{https://github.com/epfl-dlab/Negativity_in_2016_campaign}.

\xhdr{Code availability}
All analysis code is available on GitHub at \url{https://github.com/epfl-dlab/Negativity_in_2016_campaign}.

\printbibliography

\clearpage


Radiative $b$ Decays and the Detection of Supersymmetric Dark Matter

Abstract: The upper bond on the branching ratio for $b\to s\gamma$ decays implies a stringent lower bound on the mass of the pseudoscalar Higgs boson of the MSSM if sparticles are heavy. This leads to an upper bound on the expected event rate in experiments searching for heavy supersymmetric dark matter. Scenarios with lighter sparticle spectrum and light pseudoscalar Higgs boson are still possible, but only if $\mu < 0$, which again implies a small LSP counting rate.

\subsection*{Acknowledgements}
I wish to thank my collaborators Francesca Borzumati and Mihoko Nojiri;
without them, ref.\cite{8} would not have been written and I would have missed
the opportunity to have a truly cold lunch. This work was supported in part by
the U.S. Department of Energy under Grant No.~DE-FG02-95ER40896, by the
Wisconsin Research Committee with funds granted by the Wisconsin Alumni
Research Foundation, as well as by a grant from the Deutsche
Forschungsgemeinschaft under the Heisenberg program.

High Energy Physics - Phenomenology (hep-ph)