We propose a cost-effective framework for preference elicitation and aggregation under the Plackett-Luce model with features. Given a budget, our framework iteratively computes the most cost-effective elicitation questions in order to help the agents make a better group decision. We illustrate the viability of the framework with experiments on Amazon Mechanical Turk, which we use to estimate the cost of answering different types of elicitation questions. We compare the prediction accuracy of our framework when adopting various information criteria that evaluate the expected information gain from a question. Our experiments show carefully designed information criteria are much more efficient, i.e., they arrive at the correct answer using fewer queries, than randomly asking questions given the budget constraint.
Pairwise elicitation questions may be the most widely explored in the literature due to their simplicity . In contrast, @cite_9 focused on elicitation of full rankings, though their proposed framework also allows for partial orders. @cite_0 @cite_1 studied a larger set of queries, which includes asking a person to rank her top @math choices over all alternatives. In this paper, we consider an even broader set of queries, asking an agent to rank her top @math choices over a subset of @math alternatives ( @math ). This enables us to elicit preferences in a more cost-effective manner.
@cite_0 @cite_9 @cite_1
1607093780 2560254791 2159545104 2024021823
Voting is a general method for aggregating the preferences of multiple agents. Each agent ranks all the possible alternatives, and based on this, an aggregate ranking of the alternatives (or at least a winning alternative) is produced. However, when there are many alternatives, it is impractical to simply ask agents to report their complete preferences. Rather, the agents' preferences, or at least the relevant parts thereof, need to be elicited. This is done by asking the agents a (hopefully small) number of simple queries about their preferences, such as comparison queries, which ask an agent to compare two of the alternatives. Prior work on preference elicitation in voting has focused on the case of unrestricted preferences. It has been shown that in this setting, it is sometimes necessary to ask each agent (almost) as many queries as would be required to determine an arbitrary ranking of the alternatives. In contrast, in this paper, we focus on single-peaked preferences. We show that such preferences can be elicited using only a linear number of comparison queries, if either the order with respect to which preferences are single-peaked is known, or at least one other agent's complete preferences are known. We show that using a sublinear number of queries does not suffice. We also consider the case of cardinally single-peaked preferences. For this case, we show that if the alternatives' cardinal positions are known, then an agent's preferences can be elicited using only a logarithmic number of queries; however, we also show that if the cardinal positions are not known, then a sublinear number of queries does not suffice. We present experimental results for all elicitation algorithms. We also consider the problem of only eliciting enough information to determine the aggregate ranking, and show that even for this more modest objective, a sublinear number of queries per agent does not suffice for known ordinal or unknown cardinal positions. Finally, we discuss whether and how these techniques can be applied when preferences are almost single-peaked. Stated preference elicitation techniques, such as discrete choice experiments and best-worst scaling, are now widely used in health research to explore the public’s choices and preferences. In this paper, we propose an alternative stated preference elicitation technique, which we refer to as ‘trio-wise’. We explain this new technique, its relative advantages, modeling framework, and how it compares to the best-worst scaling method. To better illustrate the differences and similarities, we utilize best-worst scaling Case 2, where individuals make best and worst (most and least) choices for the attribute levels that describe a single profile. We demonstrate this new preference elicitation technique using an empirical case study that explores preferences among the general public for ways to involve them in decisions concerning the health care system. Our findings show that the best-worst scaling and trio-wise preference elicitation techniques both retrieve similar preferences. However, the capability of our trio-wise method to provide additional information on the strength of rank preferences and its ability to accommodate indifferent preferences lead us to prefer it over the standard best-worst scaling technique. Ranking a set of retrieved documents according to their relevance to a query is a popular problem in information retrieval. Methods that learn ranking functions are difficult to optimize, as ranking performance is typically judged by metrics that are not smooth. In this paper we propose a new listwise approach to learning to rank. Our method creates a conditional probability distribution over rankings assigned to documents for a given query, which permits gradient ascent optimization of the expected value of some performance measure. The rank probabilities take the form of a Boltzmann distribution, based on an energy function that depends on a scoring function composed of individual and pairwise potentials. Including pairwise potentials is a novel contribution, allowing the model to encode regularities in the relative scores of documents; existing models assign scores at test time based only on individual documents, with no pairwise constraints between documents. Experimental results on the LETOR3.0 data set show that our method out-performs existing learning approaches to ranking. We seek to elicit individual design preferences through human-computer interaction. During an iteration of the interactive session, the computer queries the subject by presenting a set of designs from which the subject must make a choice. The computer uses this choice feedback and creates the next set of designs using knowledge accumulated from previous choices. Under the hypothesis that human responses are deterministic, we discuss how query schemes in the elicitation task can be viewed mathematically as learning or optimization algorithms. Two query schemes are defined. Query type 1 considers the subject’s binary choices as definite preferences, i.e., only preferred designs are chosen, while others are skipped; query type 2 treats choices as comparisons among a set, i.e., preferred designs are chosen relative to those in the current set but may be dropped in future iterations. We show that query type 1 can be considered as an active learning problem, while type 2 as a “black-box” optimization problem. This paper concentrates on query type 2. Two algorithms based on support vector machine and efficient global optimization search are presented and discussed. Early user tests for vehicle exterior styling preference elicitation are also presented. [DOI: 10.1115 1.4005104]