In this paper, we present a link between preference-based and multiobjective sequential decision-making. While transforming a multiobjective problem to a preference-based one is quite natural, the other direction is a bit less obvious. We present how this transformation (from preference-based to multiobjective) can be done under the classic condition that preferences over histories can be represented by additively decomposable utilities and that the decision criterion to evaluate policies in a state is based on expectation. This link yields a new source of multiobjective sequential decision-making problems (i.e., when reward values are unknown) and justifies the use of solving methods developed in one setting in the other one.
A final approach to multiobjective problem assumes an interactive setting where a human expert is present and can provide additional preferential information (i.e., how to trade-off between different objectives). This approach loops between the following two steps until a certain criterion is satisfied (e.g., the expert is satisfied with a proposed solution or there is only one solution left): show potential solutions or ask query to the expert receive a feedback answer from the expert The feedback answer from the expert allows to guide the search for a preferred solution among all Pareto-optimal ones @cite_25 , or elicit unknown parameters of user preference model @cite_12 .
@cite_25 @cite_12
1990421110 2563621803 1990671169 2268703685
In many multiobjective optimization problems, the Pareto Fronts and Sets contain a large number of solutions and this makes it difficult for the decision maker to identify the preferred ones. A possible way to alleviate this difficulty is to present to the decision maker a subset of a small number of solutions representatives of the Pareto Front characteristics. In this paper, a two-steps procedure is presented, aimed at identifying a limited number of representative solutions to be presented to the decision maker. Pareto Front solutions are first clustered into "families", which are then synthetically represented by a "head-of-the-family" solution. Level Diagrams are then used to represent, analyse and interpret the Pareto Front reduced to its head-of-the-family solutions. The procedure is applied to a reliability allocation case study of literature, in decision-making contexts both without or with explicit preferences by the decision maker on the objectives to be optimized. While Pareto-based multiobjective optimization algorithms continue to show effectiveness for a wide range of practical problems that involve mostly two or three objectives, their limited application for many-objective problems, due to the increasing proportion of nondominated solutions and the lack of sufficient selection pressure, has also been gradually recognized. In this paper, we revive an early developed and computationally expensive strength Pareto-based evolutionary algorithm by introducing an efficient reference direction-based density estimator, a new fitness assignment scheme, and a new environmental selection strategy, for handling both multiobjective and many-objective problems. The performance of the proposed algorithm is validated and compared with some state-of-the-art algorithms on a number of test problems. Experimental studies demonstrate that the proposed method shows very competitive performance on both multiobjective and many-objective problems considered in this paper. Besides, our extensive investigations and discussions reveal an interesting finding, that is, diversity-first-and-convergence-second selection strategies may have great potential to deal with many-objective optimization. HighlightsWe integrate user appraisals in a POMDP-based dialogue manager procedure.We employ additional socially-inspired rewards in a RL setup to guide the learning.A unified framework for speeding up the policy optimisation and user adaptation.We consider a potential-based reward shaping with a sample efficient RL algorithm.Evaluated using both user simulator (information retrieval) and user trials (HRI). This paper investigates some conditions under which polarized user appraisals gathered throughout the course of a vocal interaction between a machine and a human can be integrated in a reinforcement learning-based dialogue manager. More specifically, we discuss how this information can be cast into socially-inspired rewards for speeding up the policy optimisation for both efficient task completion and user adaptation in an online learning setting. For this purpose a potential-based reward shaping method is combined with a sample efficient reinforcement learning algorithm to offer a principled framework to cope with these potentially noisy interim rewards. The proposed scheme will greatly facilitate the system's development by allowing the designer to teach his system through explicit positive negative feedbacks given as hints about task progress, in the early stage of training. At a later stage, the approach will be used as a way to ease the adaptation of the dialogue policy to specific user profiles. Experiments carried out using a state-of-the-art goal-oriented dialogue management framework, the Hidden Information State (HIS), support our claims in two configurations: firstly, with a user simulator in the tourist information domain (and thus simulated appraisals), and secondly, in the context of man-robot dialogue with real user trials. In this paper we investigate the use of coactive learning in a multitask setting. In coactive learning, an expert presents the learner with a problem and the learner returns a candidate solution. The expert then improves on the solution if necessary and presents the improved solution to the learner. The goal for the learner is to learn to produce solutions which cannot be further improved by the expert while minimizing the average expert effort. In this paper, we consider the setting where there are multiple experts (tasks), and in each iteration one expert presents a problem to the learner. While the experts are expected to have different solution preferences, they are also assumed to share similarities, which should enable generalization across experts. We analyze several algorithms for this setting and derive bounds on the average expert effort during learning. Our main contribution is the balanced Perceptron algorithm, which is the first coactive learning algorithm that is both able to generalize across experts when possible, while also guaranteeing convergence to optimal solutions for individual experts. Our experiments in three domains confirm that this algorithm is effective in the multitask setting, compared to natural baselines.