Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> This paper describes the use of PLANETE, a real-time knowledge-based system for program supervision in the framework of an intelligent on-board guidance system (or co-pilot) for a motor vehicle in a realistic situation. The vehicle is equipped with several sensors, producing data, that can be processed by several programs. The role of PLANETE is to convert the requirements of the high-level situation analysis module (expressed in terms of symbolic requests), into the correct low-level processing commands, under computational and time constraints. We discuss the underlying concepts of program supervision, we detail the system architecture (knowledge base and inference engine) and its implementation, and eventually we analyze a realistic example of a partial scenario, describing the path from the events received by the co-pilot, to commands ([de]activations of sensors and programs). <s> BIB001 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> In this paper we present a multisensory vision system that is intended to support the vision requirements of an intelligent robot system. Contrary to many other vision systems, our system has two significant new features. First, it contains multiple sensors, object representations, and image analysis and interpretation methods in order to solve a number of different vision tasks. Secondly, it comprises a vision planner. Upon a task-level vision request from the robot system, the vision planner transforms it into an appropriate sequence of concrete vision operations, executes these operations, and if necessary, finds out alternative sequences of operations. Experimental results demonstrate the clear advantage of this combination of multiple resources with the vision planner in solving typical vision problems for robotic tasks.© (1994) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only. <s> BIB002 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> In this paper, we bring techniques from operations research to bear on the problem of choosing optimal actions in partially observable stochastic domains. We begin by introducing the theory of Markov decision processes (mdps) and partially observable MDPs (pomdps). We then outline a novel algorithm for solving pomdps off line and show how, in some cases, a finite-memory controller can be extracted from the solution to a POMDP. We conclude with a discussion of how our approach relates to previous work, the complexity of finding exact solutions to pomdps, and of some possibilities for finding approximate solutions. <s> BIB003 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> We survey the complexity of computational problems about Markov decision processes: evaluating policies, finding good and best policies, approximating best policies, and related decision problems. <s> BIB004 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> This paper presents a new approach to hierarchical reinforcement learning based on the MAXQ decomposition of the value function. The MAXQ decomposition has both a procedural semantics—as a subroutine hierarchy—and a declarative semantics—as a representation of the value function of a hierarchical policy. MAXQ unifies and extends previous work on hierarchical reinforcement learning by Singh, Kaelbling, and Dayan and Hinton. Conditions under which the MAXQ decomposition can represent the optimal value function are derived. The paper defines a hierarchical Q learning algorithm, proves its convergence, and shows experimentally that it can learn much faster than ordinary “flat” Q learning. Finally, the paper discusses some interesting issues that arise in hierarchical reinforcement learning including the hierarchical credit assignment problem and non-hierarchical execution of the MAXQ hierarchy. <s> BIB005 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> The authors are interested in a knowledge-based technique (called program supervision) for managing the reuse of a modular set of programs. The focus of the paper is to analyse which reuse problems program supervision can solve. First, they propose a general definition for program supervision, a knowledge representation model, and a reasoning model. They then analyse a program supervision solution for reuse in terms of the structure of the programs to re-use and in terms of the effort for building a program supervision knowledge base. The paper concludes with what program supervision can do for program reuse from the points of view of the code developers, the experts, and the end-users. <s> BIB006 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> This document describes concepts and tools that we have developed for the program supervision task. The first part is a reference manual, which introduces the concepts involved in this task as well as the proposed knowledge language and inference engine to achieve this task. The second part is a user's manual, which details both the Yakl description language and the Pegase+ engine. It also provides examples and methodological recommendations to develop a program supervision system with Yakl and Pegase+. <s> BIB007 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> Automated image interpretation is an important task in numerous applications ranging from security systems to natural resource inventorization based on remote-sensing. Recently, a second generation of adaptive machine-learned image interpretation system (ADORE) has shown expertlevel performance in several challenging domains. Its extension, MR ADORE, aims at removing the last vestiges of human intervention still present in the original design of ADORE. Both systems treat the image interpretation process as a sequential decision making process guided by a machine-learned heuristic value function. This paper employs a new leveraging algorithm for regression (RESLEV) to improve the learnability of the heuristics in MR ADORE. Experiments show that RESLEV improves the system’s performance if the base learners are weak. Further analysis discovers the difference between regression and decisionmaking problems, and suggests an interesting research direction. <s> BIB008 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> Abstract This paper describes a mobile robotic assistant, developed to assist elderly individuals with mild cognitive and physical impairments, as well as support nurses in their daily activities. We present three software modules relevant to ensure successful human–robot interaction: an automated reminder system; a people tracking and detection system; and finally a high-level robot controller that performs planning under uncertainty by incorporating knowledge from low-level modules, and selecting appropriate courses of actions. During the course of experiments conducted in an assisted living facility, the robot successfully demonstrated that it could autonomously provide reminders and guidance for elderly residents. <s> BIB009 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> In (Petrick and Bacchus 2002), a "knowledge-level" approach to planning under incomplete knowledge and sensing was presented. In comparison with alternate approaches based on representing sets of possible worlds, this higher-level representation is richer, but the inferences it supports are weaker. Nevertheless, because of its richer representation, it is able to solve problems that cannot be solved by alternate approaches. ::: ::: In this paper we examine a collection of new techniques for increasing both the representational and inferential power of the knowledge-level approach. These techniques have been fully implemented in the PKS (Planning with Knowledge and Sensing) planning system. Taken together they allow us to solve a range of new types of planning problems under incomplete knowledge and sensing. <s> BIB010 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning. <s> BIB011 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> This paper proposes a new hierarchical formulation of POMDPs for autonomous robot navigation that can be solved in real-time, and is memory efficient. It will be referred to in this paper as the Robot Navigation-Hierarchical POMDP (RN-HPOMDP). The RN-HPOMDP is utilized as a unified framework for autonomous robot navigation in dynamic environments. As such, it is used for localization, planning and local obstacle avoidance. Hence, the RN-HPOMDP decides at each time step the actions the robot should execute, without the intervention of any other external module for obstacle avoidance or localization. Our approach employs state space and action space hierarchy, and can effectively model large environments at a fine resolution. Finally, the notion of the reference POMDP is introduced. The latter holds all the information regarding motion and sensor uncertainty, which makes the proposed hierarchical structure memory efficient and enables fast learning. The RN-HPOMDP has been experimentally validated in real dynamic environments. <s> BIB012 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> o We present a value iteration algorithm for learn- ing to act in Partially Observable Markov Decision Processes (POMDPs) with continuous state spaces. Mainstream POMDP research focuses on the discrete case and this complicates its application to, e.g., robotic problems that are naturally modeled using continuous state spaces. The main difculty in dening a (belief-based) POMDP in a continuous state space is that expected values over states must be dened using integrals that, in general, cannot be computed in closed from. In this paper, we provide three main contributions to the literature on continuous- state POMDPs. First, we show that the optimal nite-horizon value function over the continuous innite-dimension al POMDP belief space is piecewise linear and convex, and is dened by a nite set of supporting -functions that are analogous to the -vectors (hyperplanes) dening the value function of a discrete- state POMDP. Second, we show that, for a fairly general class of POMDP models in which all functions of interest are modeled by Gaussian mixtures, all belief updates and value iteration backups can be carried out analytically and exact. Contrary to the discrete case, in a continuous-state POMDP the -functions may grow in size (e.g., in the number of Gaussian components) in each value iteration. Third, we show how the recent point-based value iteration algorithms for discrete POMDPs can be extended to the continuous case, allowing for efcient planning in practical problems. In particular, we demonstrate Perseus, our previously proposed randomized point-based value iteration algorithm, in a simple robot planning problem in a continuous domain, where encouraging results are observed. <s> BIB013 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> In order to behave intelligently, artificial agents must be able to deliberatively plan their future actions. Unfortunately, realistic agent environments are usually highly dynamic and only partially observable, which makes planning computationally hard. For most practical purposes this rules out planning techniques that account for all possible contingencies in the planning process. However, many agent environments permit an alternative approach, namely continual planning, i.e. the interleaving of planning with acting and sensing. This paper presents a new principled approach to continual planning that describes why and when an agent should switch between planning and acting. The resulting continual planning algorithm enables agents to deliberately postpone parts of their planning process and instead actively gather missing information that is relevant for the later refinement of the plan. To this end, the algorithm explictly reasons about the knowledge (or lack thereof) of an agent and its sensory capabilities. These concepts are modelled in the planning language (MAPL). Since in many environments the major reason for dynamism is the behaviour of other agents, MAPL can also model multiagent environments, common knowledge among agents, and communicative actions between them. For Continual Planning, MAPL introduces the concept of of assertions, abstract actions that substitute yet unformed subplans. To evaluate our continual planning approach empirically we have developed MAPSIM, a simulation environment that automatically builds multiagent simulations from formal MAPL domains. Thus, agents can not only plan, but also execute their plans, perceive their environment, and interact with each other. Our experiments show that, using continual planning techniques, deliberate action planning can be used efficiently even in complex multiagent environments. <s> BIB014 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> A major challenge in the path of widespread use of mobile robots is the ability to function autonomously, learning useful features from the environment and using them to adapt to environmental changes. We propose an algorithm for mobile robots equipped with color cameras that allows for smooth operation under illumination changes. The robot uses image statistics and the environmental structure to autonomously detect and adapt to both major and minor illumination changes. Furthermore, the robot autonomously plans an action sequence that maximizes color learning opportunities while minimizing localization errors. Our approach is fully implemented and tested on the Sony AIBO robots. <s> BIB015 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> A major challenge to the deployment of mobile robots in a wide range of tasks is the ability to function autonomously, learning appropriate models for environmental features and adapting these models in response to environmental changes. Such autonomous operation is feasible iff the robot is able to plan an appropriate action sequence. In this paper, we focus on the task of color modeling/learning, and present two algorithms that enable a mobile robot to plan action sequences that facilitate color learning. We propose a long-term action-selection approach that maximizes color learning opportunities while minimizing localization errors over an entire action sequence, and compare it with a greedy/heuristic action-selection approach that plans incrementally, to maximize the utility based on the current state information. We show that long-term action-selection provides a more principled solution that requires minimal human supervision. All algorithms are fully implemented and tested on the Sony AIBO robots. <s> BIB016 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> Flexible general purpose robots need to tailor their visual processing to their task, on the fly. We propose a new approach to this within a planning framework, where the goal is to plan a sequence of visual operators to apply to the regions of interest (ROIs) in a scene. We pose the visual processing problem as a Partially Observable Markov Decision Process (POMDP). This requires probabilistic models of operator effects to quantitatively capture the unreliability of the processing actions, and thus reason precisely about trade-offs between plan execution time and plan reliability. Since planning in practical sized POMDPs is intractable we show how to ameliorate this intractability somewhat for our domain by defining a hierarchical POMDP. We compare the hierarchical POMDP approach with a Continual Planning (CP) approach. On a real robot visual domain, we show empirically that all the planning methods outperform naive application of all visual operators. The key result is that the POMDP methods produce more robust plans than either naive visual processing or the CP approach. In summary, we believe that visual processing problems represent a challenging and worthwhile domain for planning techniques, and that our hierarchical POMDP based approach to them opens up a promising new line of research. <s> BIB017 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> We describe and evaluate the algorithmic techniques that are used in the FF planning system. Like the HSP system, FF relies on forward state space search, using a heuristic that estimates goal distances by ignoring delete lists. Unlike HSP's heuristic, our method does not assume facts to be independent. We introduce a novel search strategy that combines hill-climbing with systematic search, and we show how other powerful heuristic information can be extracted and used to prune the search space. FF was the most successful automatic planner at the recent AIPS-2000 planning competition. We review the results of the competition, give data for other benchmark domains, and investigate the reasons for the runtime performance of FF compared to HSP. <s> BIB018 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> Planning under uncertainty is a central problem in the study of automated sequential decision making, and has been addressed by researchers in many different fields, including AI planning, decision analysis, operations research, control theory and economics. While the assumptions and perspectives adopted in these areas often differ in substantial ways, many planning problems of interest to researchers in these fields can be modeled as Markov decision processes (MDPs) and analyzed using the techniques of decision theory. ::: ::: This paper presents an overview and synthesis of MDP-related methods, showing how they provide a unifying framework for modeling many classes of planning problems studied in AI. It also describes structural properties of MDPs that, when exhibited by particular classes of problems, can be exploited in the construction of optimal or approximately optimal policies or plans. Planning problems commonly possess structure in the reward and value functions used to describe performance criteria, in the functions used to describe state transitions and observations, and in the relationships among features used to describe states, actions, rewards, and observations. ::: ::: Specialized representations, and algorithms employing these representations, can achieve computational leverage by exploiting these various forms of structure. Certain AI techniques-- in particular those based on the use of structured, intensional representations--can be viewed in this way. This paper surveys several types of representations for both classical and decision-theoretic planning problems, and planning algorithms that exploit these representations in a number of different ways to ease the computational burden of constructing policies or plans. It focuses primarily on abstraction, aggregation and decomposition techniques based on AI-style representations. <s> BIB019 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> Existing complexity bounds for point-based POMDP value iteration algorithms focus either on the curse of dimensionality or the curse of history. We derive a new bound that relies on both and uses the concept of discounted reachability; our conclusions may help guide future algorithm design. We also discuss recent improvements to our (point-based) heuristic search value iteration algorithm. Our new implementation calculates tighter initial bounds, avoids solving linear programs, and makes more effective use of sparsity. <s> BIB020 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Action Planning <s> Planning can often be simplified by decomposing the task into smaller tasks arranged hierarchically. Charlin et al. [4] recently showed that the hierarchy discovery problem can be framed as a non-convex optimization problem. However, the inherent computational difficulty of solving such an optimization problem makes it hard to scale to real-world problems. In another line of research, Toussaint et al. [18] developed a method to solve planning problems by maximum-likelihood estimation. In this paper, we show how the hierarchy discovery problem in partially observable domains can be tackled using a similar maximum likelihood approach. Our technique first transforms the problem into a dynamic Bayesian network through which a hierarchical structure can naturally be discovered while optimizing the policy. Experimental results demonstrate that this approach scales better than previous techniques based on non-convex optimization. <s> BIB021
Color segmentation is rarely, if ever, the end purpose of a vision algorithm, especially when used on a mobile robot. Many robot vision algorithms actively consider the purpose of object recognition when considering how to spend computational or action resources -sometimes it may be necessary to perform a sequence of actions before a particular object comes within the robot's view and can be used to model the colors. In such situations, it can be useful for a robot to plan its motion to facilitate color learning. We are interested in visual processing management: given a set of information processing and sensing actions, choose an action sequence that achieves the desired goal reliably, while using the available resources optimally. For instance, if the goal is to learn the color blue from a blue box, only the image regions known to contain a box need to be examined. This problem can be addressed by two different approaches: learning and planning. In the learning approach the robot could use different action sequences to perform each task repeatedly, using techniques such as Reinforcement Learning BIB011 to score the action sequences and choose the best sequence for each task. Learning through repeated trials on a robot, though interesting, would however involve extensive manual effort that might be infeasible in many applications. The alternative is to jointly plan the sensing (where to look ) and information-processing (what to look for ). Challenges to be addressed include: minimizing the amount of domain knowledge to be manually encoded, conflict resolution when multiple feasible actions exist, and planning under uncertainty when the action outcomes are non-deterministic. There exists a significant body of work in the computer vision community on using a user-specified goal to plan a pipeline of visual operators, including classical approaches for active vision  . However, many of these algorithms use deterministic models of the action outcomes: the pre-conditions and the effects of the operators are propositions that are required to be true a priori, or are made true by the application of the operator. Unsatisfactory results are detected by execution monitoring using hand-crafted rules, followed by re-planning the operator sequence or modifying the operator parameters  BIB007 BIB006 . There has also been some work on autonomous object detection and avoidance in vehicles BIB001 , and on interpretation of 3D objects' structure BIB002 but extensions to more general vision tasks have proven difficult. Probabilistic methods have also been used for image interpretation. Dar-rell  used memory-based reinforcement learning and Partially Observable Markov Decision Processes (POMDPs) BIB003 to learn what foveation actions to execute, and when to execute the terminal recognition action, in order to foveate salient body parts in an active gesture recognition system. In the learning phase, manual feedback is provided after the execution of the recognition action in order to assign a reward to the action sequence. The learned recognition policies are transformed into a compact augmented Finite State Machine for online recognition. More recently, Li et al. BIB008 posed image interpretation as a MDP. The reward structure is determined offline by applying all possible sequences of operators to human-labeled images. States are abstracted into a set of image features, and dynamic programming is used to compute value functions for regions in this abstracted feature space. Online image interpretation involves feature extraction and the choice of action that maximizes the value of the learned functions. In robot domains the true state of the world is not known, action outcomes are noisy, and action execution may change the state. Furthermore, computing the utilities of actions by repeated trials is often infeasible in robot domains. AI planning, in general, is a very well researched field  . Several modern AI planning methods exploit the classical planning machinery by relaxing the limiting constraints on prior knowledge of the states and action outcomes  BIB010 . Draper et al.  proposed C-BURIDAN, a planning scheme that is similar to a POMDP formulation since it models actions as probabilistic state transitions between state beliefs to choose the best action to perform. A symbolic STRIPS-like representation  is used for reasoning about the action effects and symbolic sub-goals of the desired goal. Action preconditions and effects are manually specified, and a notion of action costs does not exist. The technique cannot accumulate belief by repeated application of the same action without a manual ordering of actions. On the other hand, Petrick and Bacchus's PKS planner BIB010 describes actions in terms of their effect on the agent's knowledge rather than their effect on the world, using a first order language. The model is hence non-deterministic in the sense that the true state of the world may be determined uniquely by the actions performed, but the agent's knowledge of that state is not. For example, dropping a fragile item will break it, but if the agent does not know that the item is fragile, it will not know if it is broken, and must use an observational action to determine its status. PKS captures the initial state uncertainty and constructs conditional plans based on the agent's knowledge. In the context of visual operators, PKS would plan to use the operators to examine each salient image region, branching based on what is discovered. More recently, Brenner and Nebel BIB014 proposed the Continual Planning (CP) approach, which interleaves planning, plan execution and plan monitoring. It uses the PDDL syntax  and is based on the FF planner BIB018 . Unlike classical planning approaches, an agent in CP postpones reasoning about unknowable or uncertain states until more information is available. The agent allows actions to assert that the action preconditions will be met when the agent reaches that point in the execution of the plan, and replanning is triggered if those preconditions are not met during execution or are met earlier. CP is therefore quite similar to PKS in representation, but works by replanning rather than constructing conditional plans. However, there is no representation of the uncertainty in the observations and actions. In vision applications with noisy observations, the optimal behavior may be to take several images of a scene and accumulate the evidence obtained by running the visual operators more than once. This cannot be readily represented in CP, C-BURIDAN or PKS, A classical approach for planning high-level behaviors on robots is to use a behavior-based architecture, for instance algorithms that build on the subsumption architecture  . The underlying principle is that the representation for information depends on the intended use of the information. Complicated behaviors are decomposed into layers of simple behaviors. A layer implements a particular behavior goal and higher-level goals subsume those of the lower layers. For instance the lowest layer could encode "avoid obstacles" or "follow the wall", followed by a layer that encodes "explore the environment", which in turn is subsumed by the overall goal: "create a map of the environment". The distributed behavior promotes modularity, but there is very little cognitive reasoning in the decision-making, and there is no explicit state or action representation -effects of actions are known only by executing the actions. Such a system would be infeasible in complex robot systems. Hence, most planning systems on robots use some form of knowledge representation, and use reactive planning for simple control tasks within a larger system. Instead of manually encoding the action outcomes, Sridharan and Stone BIB015 BIB016 exploit the structure of the environment (known positions, sizes and color labels of objects) to enable a legged robot to autonomously learn probabilistic models for the motion errors and the feasibility of learning color models at different locations. The robot is then able to robustly plan action sequences that simultaneously maximize color learning opportunities while minimizing the possible localization errors BIB016 . The robot is able to respond to unexpected changes by re-planning from its current state, thereby providing support to the hypothesis that a combination of learning and probabilistic planning is currently the most appealing option for autonomous mobile robot operation. Figure 9 shows examples of planning a sequence of actions to learn colors. The probabilistic (i.e. decision-theoretic) scheme is currently a popular alternative for planning a sequence of actions for practical domains BIB019 BIB009 . Here, the planning task is posed as an instance of probabilistic sequential decision making, or more specifically as a POMDP. POMDPs elegantly capture the incomplete knowledge of the state by maintaining a probability distribution over the true state (i.e. belief state). The non-deterministic action outcomes are also modeled probabilistically. The POMDP model is solved using techniques such as value-iteration BIB011 , and the output policy provides an action sequence that simultaneously maximizes the reliability and minimizes the use of resources. However, practical problems formulated as POMDPs soon become intractable because the state space increases exponentially and the solutions are exponential in the state-space dimensions BIB004 BIB011 . In recent times, the focus has therefore shifted to finding approximate solutions to POMDPs  BIB020 (see   for surveys), and imposing a structure or hierarchy on the problem to make it more tractable BIB012 BIB009 BIB013 . Pineau et al. BIB009 use a hierarchical POMDP for high-level behavior control on a nursing assistant robot. Similar to the MAXQ decomposition for MDPs BIB005 they impose an action hierarchy: the top-level action is a collection of simpler actions represented by smaller POMDPs. Complete solutions (policies) for the smaller POMDPs are combined to provide the total policy. Invoking the top-level policy recursively invokes policies in the hierarchy until a primitive action is reached. POMDPs at all levels are defined over the same state-action space and the relevant space for each POMDP is abstracted using a dynamic belief network. Foka and Trahanias BIB012 proposed a similar system for autonomous robot navigation. A significant portion of the data for hierarchy and model creation is however application-specific, and has to be hand-coded. Furthermore, even with stateof-the-art approximate POMDP solvers BIB020 there is an intractability inherent in POMDPs for domains with large, complex state spaces. Recent work by Sridharan et al.  BIB017 presents a novel hierarchical POMDP framework for a cognitive interaction scenario where a human and a robot jointly converse about and interact with objects on a tabletop. Given input images of the scene with several salient regions-of-interest (ROIs), many visual operators that can be applied on the images, and a query to answer, their approach decomposes the overall planning problem into two sub-problems: the problem of choosing which ROI to perform processing on, and the problem of deciding how to process the chosen ROI. One POMDP is defined at the higher level (HL) where actions consist of choosing the ROI to process, and several POMDPs are defined at the lower level (LL) with one POMDP each for planning the visual operators to apply on a particular ROI. Unlike other hierarchical POMDP approaches, autonomously collected statistics are used to learn the query-specific models at the HL and LL, which are then solved to obtain the overall policy. However, as the state space becomes more complex, a range of hierarchies in the state and action space may be required, in addition to faster POMDP solvers. Still, probabilistic (POMDP) planning methods provide the most promising results, and significant attention is being devoted towards learning the hierarchies autonomously BIB021 . Table 4 Action-planning methods -Probabilistic autonomous schemes provide the more promising results on mobile robots. Table 4 compares the action-planning schemes based on the criteria required for vision-based mobile robot operation. Most classical and non-probabilistic planning methods require prior knowledge of states and action outcomes, which may not be feasible to provide in robot applications. Even when extensive manual encoding of information is not required (e.g. the reactive approach  or continual planning BIB014 ), the techniques either require several expensive robot trials to build useful world models or fail to capture the non-deterministic action outcomes. Even a large subset of the probabilistic planning algorithms for robot domains require a lot of prior information to be manually encoded BIB019 BIB009 . Very rarely do robot action-planning methods learn a majority of the required state and action models without human supervision BIB016 BIB017 -even in such cases some domain knowledge needs to be encoded, though it is significantly easier to provide. To summarize, a combination of learning and probabilistic planning is currently the most appealing direction of research for real-world robot applications that require autonomous operation.