Ad exchanges are widely used in platforms for online display advertising. Autonomous agents operating in these exchanges must learn policies for interacting profitably with a diverse, continually changing, but unknown market. We consider this problem from the perspective of a publisher, strategically interacting with an advertiser through a posted price mechanism. The learning problem for this agent is made difficult by the fact that information is censored, i.e., the publisher knows if an impression is sold but no other quantitative information. We address this problem using the Harsanyi-Bellman Ad Hoc Coordination (HBA) algorithm [1, 3], which conceptualises this interaction in terms of a Stochastic Bayesian Game and arrives at optimal actions by best responding with respect to probabilistic beliefs maintained over a candidate set of opponent behaviour profiles. We adapt and apply HBA to the censored information setting of ad exchanges. Also, addressing the case of stochastic opponents, we devise a strategy based on a Kaplan-Meier estimator for opponent modelling. We evaluate the proposed method using simulations wherein we show that HBA-KM achieves substantially better competitive ratio and lower variance of return than baselines, including a Q-learning agent and a UCB-based online learning agent, and comparable to the offline optimal algorithm.
Much has been written about ad display and sponsored search auctions, for each participating agent of the auction, either as publisher or advertiser. A key paper in the area of ad exchanges is that of Muthukrishnan @cite_10 , who laid out several research directions in this domain, informed by exposure to the practice in this domain.
@cite_10
1554763265
An emerging way to sell and buy display ads on the Internet is via ad exchanges. RightMedia [1], AdECN [2] and DoubleClick Ad Exchange [3] are examples of such real-time two-sided markets. We describe an abstraction of this market. Based on that abstraction, we present several research directions and discuss some insights.