Motivated by the successful application of the theory of regular languages to formal verification of finite-state systems, there is a renewed interest in developing a theory of analyzable functions from strings to numerical values that can provide a foundation for analyzing quantitative properties of finitestate systems. In this paper, we propose a deterministic model for associating costs with strings that is parameterized by operations of interest (such as addition, scaling, and min), a notion of regularity that provides a yardstick to measure expressiveness, and study decision problems and theoretical properties of resulting classes of cost functions. Our definition of regularity relies on the theory of string-to-tree transducers, and allows associating costs with events that are conditional upon regular properties of future events. Our model of cost register automata allows computation of regular functions using multiple “write-only” registers whose values can be combined using the allowed set of operations. We show that classical shortest-path algorithms as well as algorithms designed for computing discounted costs, can be adopted for solving the min-cost problems for the more general classes of functions specified in our model. Cost register automata with min and increment give a deterministic model that is equivalent to weighted automata, an extensively studied nondeterministic model, and this connection results in new insights and new open problems.
In @cite_28 , the authors introduce weighted logic for infinite words. In order to address convergence of the weighted sum, the authors assume discounting along later edges in a path ( , future discounting). Extending the results of this paper to discounted weighted computations over infinite words remains open.
@cite_28
2119375992
We introduce a weighted logic with discounting and we establish the Buchi-Elgot theorem for weighted automata over finite words and arbitrary commutative semirings. Then we investigate Buchi and Muller automata with discounting over the max-plus and the min-plus semiring. We show their expressive equivalence with weighted MSO-sentences with discounting. In this case our logic has a purely syntactic definition. For the finite case, we obtain a purely syntactically defined weighted logic if the underlying semiring is additively locally finite. A possibly immortal agent tries to maximise its summed discounted rewards over time, where discounting is used to avoid infinite utilities and encourage the agent to value current rewards more than future ones. Some commonly used discount functions lead to time-inconsistent behavior where the agent changes its plan over time. These inconsistencies can lead to very poor behavior. We generalise the usual discounted utility model to one where the discount function changes with the age of the agent. We then give a simple characterisation of time- (in)consistent discount functions and show the existence of a rational policy for an agent that knows its discount function is time-inconsistent. In this paper, we give the first constant-factor approximation algorithm for the rooted Orienteering problem, as well as a new problem that we call the Discounted-Reward traveling salesman problem (TSP), motivated by robot navigation. In both problems, we are given a graph with lengths on edges and rewards on nodes, and a start node @math . In the Orienteering problem, the goal is to find a path starting at @math that maximizes the reward collected, subject to a hard limit on the total length of the path. In the Discounted-Reward TSP, instead of a length limit we are given a discount factor @math , and the goal is to maximize the total discounted reward collected, where the reward for a node reached at time @math is discounted by @math . This problem is motivated by an approximation to a planning problem in the Markov decision process (MDP) framework under the commonly employed infinite horizon discounted reward optimality criterion. The approximation arises from a need to deal with exponentially large state spaces that emerge when trying to model one-time events and nonrepeatable rewards (such as for package deliveries). We also consider tree and multiple-path variants of these problems and provide approximations for those as well. Although the unrooted Orienteering problem, where there is no fixed start node @math , has been known to be approximable using algorithms for related problems such as @math -TSP (in which the amount of reward to be collected is fixed and the total length is approximately minimized), ours is the first to approximate the rooted question, solving an open problem in [E. M. Arkin, J. S. B. Mitchell, and G. Narasimhan, Proceedings of the @math th ACM Symposium on Computational Geometry, 1998, pp. 307-316] and [B. Awerbuch, Y. Azar, A. Blum, and S. Vempala, SIAM J. Comput., 28 (1998), pp. 254-262]. We complement our approximation result for Orienteering by showing that the problem is APX-hard. The authors develop a theory characterizing optimal stopping times for discrete-time ergodic Markov processes with discounted rewards. The theory differs from prior work by its view of per-stage and terminal reward functions as elements of a certain Hilbert space. In addition to a streamlined analysis establishing existence and uniqueness of a solution to Bellman's equation, this approach provides an elegant framework for the study of approximate solutions. In particular, the authors propose a stochastic approximation algorithm that tunes weights of a linear combination of basis functions in order to approximate a value function. They prove that this algorithm converges (almost surely) and that the limit of convergence has some desirable properties. The utility of the approximation method is illustrated via a computational case study involving the pricing of a path dependent financial derivative security that gives rise to an optimal stopping problem with a 100-dimensional state space. We consider two person zero-sum stochastic games. The recursive formula for the valuesvλ (resp.v n) of the discounted (resp. finitely repeated) version can be written in terms of a single basic operator Φ(α,f) where α is the weight on the present payoff andf the future payoff. We give sufficient conditions in terms of Φ(α,f) and its derivative at 0 for limv n and limvλ to exist and to be equal. We address the problem of designing surrogate losses for learning scoring functions in the context of label ranking. We extend to ranking problems a notion of order-preserving losses previously introduced for multiclass classification, and show that these losses lead to consistent formulations with respect to a family of ranking evaluation metrics. An order-preserving loss can be tailored for a given evaluation metric by appropriately setting some weights depending on this metric and the observed supervision. These weights, called the standard form of the supervision, do not always exist, but we show that previous consistency results for ranking were proved in special cases where they do. We then evaluate a new pairwise loss consistent with the (Normalized) Discounted Cumulative Gain on benchmark datasets. A batch service queue is considered where each batch size and its time of service is subject to control. Costs are incurred for serving the customers and for holding them in the system. Viewing the system as a Markov decision process (i.e., dynamic program) with unbounded costs, we show that policies which minimize the expected continuously discounted cost and the expected cost per unit time over an infinite time horizon are of the form: at a review point when x customers are waiting, serve min x, Q customers (Q being the, possibly infinite, service capacity) if and only if x exceeds a certain optimal level M. Methods of computing M for both the discounted and average cost contexts are presented. Temporal logic is two-valued: formulas are interpreted as either true or false. When applied to the analysis of stochastic systems, or systems with imprecise formal models, temporal logic is therefore fragile: even small changes in the model can lead to opposite truth values for a specification. We present a generalization of the branching-time logic CTL which achieves robustness with respect to model perturbations by giving a quantitative interpretation to predicates and logical operators, and by discounting the importance of events according to how late they occur. In every state, the value of a formula is a real number in the interval [0,1], where 1 corresponds to truth and 0 to falsehood. The boolean operators and and or are replaced by min and max, the path quantifiers ∃ and ¬ determine sup and inf over all paths from a given state, and the temporal operators ♦ and □ specify sup and inf over a given path; a new operator averages all values along a path. Furthermore, all path operators are discounted by a parameter that can be chosen to give more weight to states that are closer to the beginning of the path.We interpret the resulting logic DCTL over transition systems, Markov chains, and Markov decision processes. We present two semantics for DCTL: a path semantics, inspired by the standard interpretation of state and path formulas in CTL, and a fixpoint semantics, inspired by the µ-calculus evaluation of CTL formulas. We show that, while these semantics coincide for CTL, they differ for DCTL, and we provide model-checking algorithms for both semantics. Two players are about to play a discounted infinitely repeated bimatrix game. Each player knows his own payoff matrix and chooses a strategy which is a best response to some private beliefs over strategies chosen by his opponent. If both players' beliefs contain a grain of truth (each assigns some positive probability to the strategy chosen by the opponent), then they will eventually (a) accurately predict the future play of the game and (b) play a Nash equilibrium of the repeated game. An immediate corollary is that in playing a Harsanyi-Nash equilibrium of a discounted repeated game of incomplete information about opponents' payoffs, the players will eventually play an equilibrium of the real game as if they had complete information. Abstract Delay discounting (DD) is a measure of the degree to which an individual is driven by immediate gratification vs. the prospect of larger, but delayed, rewards. Because of hypothesized parallels between drug addiction and obesity, and reports of increased delay discounting in drug-dependent individuals, we hypothesized that obese individuals would show higher rates of discounting than controls. Obese and healthy-weight age-matched participants of both sexes completed two versions of a DD of money task, allowing us to calculate how subjective value of @math 50,000 declined as delay until hypothetical delivery increased from 2 weeks to 10 years. On both tasks, obese women (N = 29) showed greater delay discounting than control women did (N = 26; P values We consider the complete graph K_n on n vertices with exponential mean n edge lengths. Writing C_ ij for the weight of the smallest-weight path between vertex i,j [n], Janson [17] showed that max_ i,j [n] C_ ij log n converges in probability to 3. We extend this results by showing that max_ i,j [n] C_ ij - 3 log n converges in distribution to some limiting random variable that can be identified via a maximization procedure on a limiting infinite random structure. Interestingly, this limiting random variable has also appeared as the weak limit of the re-centered graph diameter of the barely supercritical Erdos-Renyi random graph in [21]. We propose the contextual combinatorial cascading bandits, a combinatorial online learning game, where at each time step a learning agent is given a set of contextual information, then selects a list of items, and observes stochastic outcomes of a prefix in the selected items by some stopping criterion. In online recommendation, the stopping criterion might be the first item a user selects; in network routing, the stopping criterion might be the first edge blocked in a path. We consider position discounts in the list order, so that the agent's reward is discounted depending on the position where the stopping criterion is met. We design a UCB-type algorithm, C3-UCB, for this problem, prove an n-step regret bound O(√n) in the general setting, and give finer analysis for two special cases. Our work generalizes existing studies in several directions, including contextual information, position discounts, and a more general cascading bandit model. Experiments on synthetic and real datasets demonstrate the advantage of involving contextual information and position discounts. It has been reported that 40 of consumers will share an email offer with their friend and 28 of consumers will share deals via social media platforms. This motivates us to study the influence maximization discount allocation problem: given a social network and a limited marketing budget, which set of initial users should be selected to receive the discount, and how much should the discounts be worth? Our goal is to maximize the number of customers who finally adopt the target product. We investigate this problem under both non-adaptive and adaptive settings. In the first setting, we have to commit the set of initial users and corresponding discounts all at once in advance. In the latter case, given that an user has been offered a discount, we are able to know immediately her decision on whether or not to accept that discount, therefore, the decision process is performed in a sequential manner based on the feedback from previously selected users. We propose a simple greedy policy with an approximation ratio of (1-1 e) in non-adaptive setting. For the significantly more complex adaptive setting, we propose a series of adaptive policies with bounded approximation ratio in terms of expected utility. Although most applications of discounting occur in risky settings, the best-known axiomatic justifications are deterministic. This paper provides an axiomatic rationale for discounting in a stochastic framework. Consider a representation of time and risk preferences with a binary relation on a real vector space of vector-valued discrete-time stochastic processes on a probability space. Four axioms imply that there are unique discount factors such that preferences among stochastic processes correspond to preferences among present value random vectors. The familiar axioms are weak ordering, continuity and nontriviality. The fourth axiom, decomposition, is non-standard and key. These axioms and the converse of decomposition are assumed in previous axiomatic justifications for discounting with nonlinear intraperiod utility functions in deterministic frameworks. Thus, the results here provide the weakest known sufficient conditions for discounting in deterministic or stochastic settings. In addition to the four axioms, if there exists a von Neumann-Morgenstern utility function corresponding to the binary relation, then that function is risk neutral (i.e., affine). In this sense, discounting axioms imply risk neutrality. Copyright Springer Science+Business Media, LLC 2013 Behavioral economics examines conditions that influence the consumption of commodities and provides several concepts that may be instrumental in understanding drug dependence. One such concept of significance is that of how delayed reinforcers are discounted by drug dependent individuals. Discounting of delayed reinforcers refers to the observation that the value of a delayed reinforcer is discounted (reduced in value or considered to be worth less) compared to the value of an immediate reinforcer. This paper examines how delay discounting may provide an explanation of both impulsivity and loss of control exhibited by the drug dependent. In so doing, the paper reviews economic models of delay discounting, the empirical literature on the discounting of delayed reinforcers by the drug dependent and the scientific literature on personality assessments of impulsivity among drug-dependent individuals. Finally, future directions for the study of discounting are discussed, including the study of loss of control and loss aversion among drug-dependent individuals, the relationship of discounting to both the behavioral economic measure of elasticity as well as to outcomes observed in clinical settings, and the relationship between impulsivity and psychological disorders other than drug dependence. Rationale: Impulsivity is implicated in drug dependence. Recent studies show problems with alcohol and opioid dependence are associated with rapid discounting of the value of delayed outcomes. Furthermore, discounting may be particularly steep for the drug of dependence. Objectives: We determined if these findings could be extended to the behavior of cigarette smokers. In particular, we compared the discounting of hypothetical monetary outcomes by current, never, and ex-smokers of cigarettes. We also examined discounting of delayed hypothetical cigarettes by current smokers. Methods: Current cigarette smokers (n=23), never-smokers (n=22) and ex-smokers (n=21) indicated preference for immediate versus delayed money in a titration procedure that determined indifference points at various delays. The titration procedure was repeated with cigarettes for smokers. The degree to which the delayed outcomes were discounted was estimated with two non-linear decay models: an exponential model and a hyperbolic model. Results: Current smokers discounted the value of delayed money more than did the comparison groups. Never- and ex-smokers did not differ in their discounting of money. For current smokers, delayed cigarettes lost subjective value more rapidly than delayed money. The hyperbolic equation provided better fits to the data than did the exponential equation for 74 out of 89 comparisons. Conclusions: Cigarette smoking, like other forms of drug dependence, is characterized by rapid loss of subjective value for delayed outcomes, particularly for the drug of dependence. Never- and ex-smokers could discount similarly because cigarette smoking is associated with a reversible increase in discounting or due to selection bias. Metric temporal logic (MTL) is a prominent specification formalism for real-time systems. In this paper, we show that the satisfiability problem for MTL over finite timed words is decidable, with non-primitive recursive complexity. We also consider the model-checking problem for MTL: whether all words accepted by a given Alur-Dill timed automaton satisfy a given MTL formula. We show that this problem is decidable over finite words. Over infinite words, we show that model checking the safety fragment of MTL-which includes invariance and time-bounded response properties-is also decidable. These results are quite surprising in that they contradict various claims to the contrary that have appeared in the literature. The question of the decidability of MTL over infinite words remains open. We propose a deterministic model for associating costs with strings that is parameterized by operations of interest (such as addition, scaling, and minimum), a notion of regularity that provides a yardstick to measure expressiveness, and study decision problems and theoretical properties of resulting classes of cost functions. Our definition of regularity relies on the theory of string-to-tree transducers, and allows associating costs with events that are conditioned on regular properties of future events. Our model of cost register automata allows computation of regular functions using multiple "write-only" registers whose values can be combined using the allowed set of operations. We show that the classical shortest-path algorithms as well as the algorithms designed for computing discounted costs can be adapted for solving the min-cost problems for the more general classes of functions specified in our model. Cost register automata with the operations of minimum and increment give a deterministic model that is equivalent to weighted automata, an extensively studied nondeterministic model, and this connection results in new insights and new open problems. In the domain of decentralized Markov decision processes, we develop the first complete and optimal algorithm that is able to extract deterministic policy vectors based on finite state controllers for a cooperative team of agents. Our algorithm applies to the discounted infinite horizon case and extends best-first search methods to the domain of decentralized control theory. We prove the optimality of our approach and give some first experimental results for two small test problems. We believe this to be an important step forward in learning and planning in stochastic multi-agent systems. We introduce new classes of weighted automata on words. Equipped with pebbles and a two-way mechanism, they go beyond the class of recognizable formal power series, but capture a weighted version of first-order logic with bounded transitive closure. In contrast to previous work, this logic allows for unrestricted use of universal quantification. Our main result states that pebble weighted automata, nested weighted automata, and this weighted logic are expressively equivalent. We also give new logical characterizations of the recognizable series.