As the largest e-commerce platform, Taobao helps advertisers reach billions of search requests each day with its sponsored search service, which has also contributed considerable revenue to the platform. How to design a suit of marketing optimization tool to cater various advertiser demands while balancing platform revenue and consumer experience is significant to a healthy and sustainable marketing ecosystem, among which bidding strategy plays a critical role. Traditional keyword-level bidding optimization only provides a coarse-grained match between advertisement and impression. Meanwhile impression-level expected value bidder is not applicable to various demand optimization of massive advertisers, not to mention its lack of mechanism to balance benefits of three parties. In this paper we propose , a bidding optimization framework which designs an impression-level bidding strategy to reflect advertiser's conversion willingness and budget control. In this way, with a simplified control ability for advertisers, CIA is capable of fulfilling various e-commerce advertiser demands in different levels, such as GMV optimization, style comparison etc. Additionally, a replay based simulation system is designed to predict the performance of different take-rate. CIA unifies the benefits of three parties in the marketing ecosystem without changing the classic expected Cost Per Mille mechanism. Our extensive offline simulations and large-scale online experiments on platform verify the high effectiveness of the CIA framework. Moreover, CIA has been deployed online as a major bidding tool for advertisers in TSA.
Other novel work also exists in display advertising. Considering the sequential decision processes in impression bidding, utilized reinforcement learning to bid @cite_10 . With the assumption of i.i.d impression features, they derived an optimal real-time bidder under budget and auction volume constraints. A similar idea was also used in JD DSP business, with deep Q-learner in its bidder @cite_14 . Reinforcement learning is a straightforward solution to the scenario of delayed rewards in online advertising. improved single-impression bidder by attributing post-impression actions into previous bids to decouple the sequential bidding process @cite_3 . Another related line of bidding optimization research lies in smart pacing, which helps budget-limited advertisers decide whether to bid or an impression or not @cite_11 @cite_7 .
@cite_14 @cite_7 @cite_3 @cite_10 @cite_11
2745512453 2092560364 2739259168 2562337727 1996312099
We present LADDER, the first deep reinforcement learning agent that can successfully learn control policies for large-scale real-world problems directly from raw inputs composed of high-level semantic information. The agent is based on an asynchronous stochastic variant of DQN (Deep Q Network) named DASQN. The inputs of the agent are plain-text descriptions of states of a game of incomplete information, i.e. real-time large scale online auctions, and the rewards are auction profits of very large scale. We apply the agent to an essential portion of JD's online RTB (real-time bidding) advertising business and find that it easily beats the former state-of-the-art bidding policy that had been carefully engineered and calibrated by human experts: during JD.com's June 18th anniversary sale, the agent increased the company's ads revenue from the portion by more than 50 , while the advertisers' ROI (return on investment) also improved significantly. Today, billions of display ad impressions are purchased on a daily basis through a public auction hosted by real time bidding (RTB) exchanges. A decision has to be made for advertisers to submit a bid for each selected RTB ad request in milliseconds. Restricted by the budget, the goal is to buy a set of ad impressions to reach as many targeted users as possible. A desired action (conversion), advertiser specific, includes purchasing a product, filling out a form, signing up for emails, etc. In addition, advertisers typically prefer to spend their budget smoothly over the time in order to reach a wider range of audience accessible throughout a day and have a sustainable impact. However, since the conversions occur rarely and the occurrence feedback is normally delayed, it is very challenging to achieve both budget and performance goals at the same time. In this paper, we present an online approach to the smooth budget delivery while optimizing for the conversion performance. Our algorithm tries to select high quality impressions and adjust the bid price based on the prior performance distribution in an adaptive manner by distributing the budget optimally across time. Our experimental results from real advertising campaigns demonstrate the effectiveness of our proposed approach. Predicting click and conversion probabilities when bidding on ad exchanges is at the core of the programmatic advertising industry. Two separated lines of previous works respectively address i) the prediction of user conversion probability and ii) the attribution of these conversions to advertising events (such as clicks) after the fact. We argue that attribution modeling improves the efficiency of the bidding policy in the context of performance advertising. Firstly we explain the inefficiency of the standard bidding policy with respect to attribution. Secondly we learn and utilize an attribution model in the bidder itself and show how it modifies the average bid after a click. Finally we produce evidence of the effectiveness of the proposed method on both offline and online experiments with data spanning several weeks of real traffic from Criteo, a leader in performance advertising. The majority of online display ads are served through real-time bidding (RTB) --- each ad display impression is auctioned off in real-time when it is just being generated from a user visit. To place an ad automatically and optimally, it is critical for advertisers to devise a learning algorithm to cleverly bid an ad impression in real-time. Most previous works consider the bid decision as a static optimization problem of either treating the value of each impression independently or setting a bid price to each segment of ad volume. However, the bidding for a given ad campaign would repeatedly happen during its life span before the budget runs out. As such, each bid is strategically correlated by the constrained budget and the overall effectiveness of the campaign (e.g., the rewards from generated clicks), which is only observed after the campaign has completed. Thus, it is of great interest to devise an optimal bidding strategy sequentially so that the campaign budget can be dynamically allocated across all the available impressions on the basis of both the immediate and future rewards. In this paper, we formulate the bid decision process as a reinforcement learning problem, where the state space is represented by the auction information and the campaign's real-time parameters, while an action is the bid price to set. By modeling the state transition via auction competition, we build a Markov Decision Process framework for learning the optimal bidding policy to optimize the advertising performance in the dynamic real-time bidding environment. Furthermore, the scalability problem from the large real-world auction volume and campaign budget is well handled by state value approximation using neural networks. The empirical study on two large-scale real-world datasets and the live A B testing on a commercial platform have demonstrated the superior performance and high efficiency compared to state-of-the-art methods. Search engine ad auctions typically have a significant fraction of advertisers who are budget constrained, i.e., if allowed to participate in every auction that they bid on, they would spend more than their budget. This yields an important problem: selecting the ad auctions which these advertisers participate, in order to optimize different system objectives such as the return on investment for advertisers, and the quality of ads shown to users. We present a system and algorithms for optimizing budget constrained spend. The system is designed be deployed in a large search engine, with hundreds of thousands of advertisers, millions of searches per hour, and with the query stream being only partially predictable. We have validated the system design by implementing it in the Google ads serving system and running experiments on live traffic. We have also compared our algorithm to previous work that casts this problem as a large linear programming problem limited to popular queries, and show that our algorithms yield substantially better results.