The interpretability of an AI agent's behavior is of utmost importance for effective human-AI interaction. To this end, there has been increasing interest in characterizing and generating interpretable behavior of the agent. An alternative approach to guarantee that the agent generates interpretable behavior would be to design the agent's environment such that uninterpretable behaviors are either prohibitively expensive or unavailable to the agent. To date, there has been work under the umbrella of goal or plan recognition design exploring this notion of environment redesign in some specific instances of interpretable of behavior. In this position paper, we scope the landscape of interpretable behavior and environment redesign in all its different flavors. Specifically, we focus on three specific types of interpretable behaviors -- explicability, legibility, and predictability -- and present a general framework for the problem of environment design that can be instantiated to achieve each of the three interpretable behaviors. We also discuss how specific instantiations of this framework correspond to prior works on environment design and identify exciting opportunities for future work. The design of human-aware AI agents must ensure that its decisions are interpretable to the human in the loop. Uninterpretable behavior can lead to increased cognitive load on the human -from reduced trust, productivity to increased risk of danger around the agent BID7 . BID5 emphasises in the Roadmap for U.S. Robotics -"humans must be able to read and recognize agent activities in order to interpret the agent's understanding". The agent's behavior may be uninterpretable if the human: (1) has incorrect notion of the agent's beliefs and capabilities BID15 BID1 ) FORMULA3 is unaware of the agent's goals and rewards BID6 BID12 (3) cannot predict the agent's plan or policy BID8 BID12 . Thus, in order to be interpretable, the agent must take into account the human's expectations of its behavior -i.e. the human mental model BID0 ). There are many ways in which considerations of the human mental model can affect agent behavior. * equal contribution We will now highlight limitations of the proposed framework and discuss how they may be extended in the future.Multiple decision making problems. The problem of environment design, as studied in this paper, is suitable for settings where the actor performs a single repetitive task. However, our formulation can be easily extended to handle an array of tasks that the agent performs in its environment by considering a set of decision making problems for the actor , where the worst-case score is decided by taking either minimum (or average) over the wci(·) for the set of problems.Interpretability Score. The three properties of interpretable agent behavior are not mutually exclusive. A plan can be explicable, legible and predictable at the same time. In general, a plan can have any combination of the three properties. In Equation 2, Int(·) uses one of these properties at a time. In order to handle more than one property at a time, one could formulate Int(·) as a linear combination of the three properties. In general, the design objective would be to minimize the worst-case interpretability score such that the scores for each property are maximized in the modified environment, or at least allow the designer pathways to trade off among potentially competing metrics.Cost of the agent. In Section 1.3 we mentioned an advantage of the design process in the context of interpretabilitythe ability to offload the computational load on the actor, in having to reason about the observer model, to the offline design stage. However, there is never any free lunch. The effect of environment design is more permanent than operating on the human mental model. That is to say, interpretable behavior while targeted for a particular human in the loop or for a particular interaction, does not (usually) affect the actor going forward. However, in case of design of environment, the actor has to live with the design decisions for the rest of its life. That means, for example, if the environment has been designed to promote explicable behavior, the actor would be incurring additional cost for its behaviors (than it would have had in the original environment). This also affects not only a particular decision making problem at hand, but also everything that the actor does in the environment, and for all the agents it interacts with. As such there is a "loss of autonomy" is some sense due to environment design, the cost of which can and should be incorporated in the design process.
We present an approach to redesign the environment such that uninterpretable agent behaviors are minimized or eliminated.
{"src": "scitldr-AIC_train", "paper_id": "rkxg4a3m9N"}