Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Introduction <s> This paper reports on a reactive docking behavior which uses a vision algorithm that grows linearly with the number of image pixels. The docking robot imprints (initializes) on a two-colored docking fiducial upon departing from the dock, then uses region statistics to adapt the color segmentation in changing lighting conditions. The docking behavior was implemented on a marsupial team of robots, where a daughter micro-rover had to reenter the mother robot from an approach zone with a 2 m radius and 140/spl deg/ angular width with a tolerance of /spl plusmn/5 and /spl plusmn/2 cm. Testing during outdoor conditions (noon, dusk) and challenging indoor scenarios (flashing lights) showed that using adaptation and imprinting was more robust than using imprinting atone. <s> BIB001 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Introduction <s> Mobile robot localization is the problem of determining a robot’s pose from sensor data. This article presents a family of probabilistic localization algorithms known as Monte Carlo Localization (MCL). MCL algorithms represent a robot’s belief by a set of weighted hypotheses (samples), which approximate the posterior under a common Bayesian formulation of the localization problem. Building on the basic MCL algorithm, this article develops a more robust algorithm called MixtureMCL, which integrates two complimentary ways of generating samples in the estimation. To apply this algorithm to mobile robots equipped with range finders, a kernel density tree is learned that permits fast sampling. Systematic empirical results illustrate the robustness and computational efficiency of the approach.  2001 Published by Elsevier Science B.V. <s> BIB002 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Introduction <s> This paper describes a machine learning approach for visual object detection which is capable of processing images extremely rapidly and achieving high detection rates. This work is distinguished by three key contributions. The first is the introduction of a new image representation called the "integral image" which allows the features used by our detector to be computed very quickly. The second is a learning algorithm, based on AdaBoost, which selects a small number of critical visual features from a larger set and yields extremely efficient classifiers. The third contribution is a method for combining increasingly more complex classifiers in a "cascade" which allows background regions of the image to be quickly discarded while spending more computation on promising object-like regions. The cascade can be viewed as an object specific focus-of-attention mechanism which unlike previous approaches provides statistical guarantees that discarded regions are unlikely to contain the object of interest. In the domain of face detection the system yields detection rates comparable to the best previous systems. Used in real-time applications, the detector runs at 15 frames per second without resorting to image differencing or skin color detection. <s> BIB003 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Introduction <s> From the Publisher: ::: The accessible presentation of this book gives both a general view of the entire computer vision enterprise and also offers sufficient detail to be able to build useful applications. Users learn techniques that have proven to be useful by first-hand experience and a wide range of mathematical methods. A CD-ROM with every copy of the text contains source code for programming practice, color images, and illustrative movies. Comprehensive and up-to-date, this book includes essential topics that either reflect practical significance or are of theoretical importance. Topics are discussed in substantial and increasing depth. Application surveys describe numerous important application areas such as image based rendering and digital libraries. Many important algorithms broken down and illustrated in pseudo code. Appropriate for use by engineers as a comprehensive reference to the computer vision enterprise. <s> BIB004 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Introduction <s> Abstract This paper describes a mobile robotic assistant, developed to assist elderly individuals with mild cognitive and physical impairments, as well as support nurses in their daily activities. We present three software modules relevant to ensure successful human–robot interaction: an automated reminder system; a people tracking and detection system; and finally a high-level robot controller that performs planning under uncertainty by incorporating knowledge from low-level modules, and selecting appropriate courses of actions. During the course of experiments conducted in an assisted living facility, the robot successfully demonstrated that it could autonomously provide reminders and guidance for elderly residents. <s> BIB005 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Introduction <s> We present a method to learn and recognize object class models from unlabeled and unsegmented cluttered scenes in a scale invariant manner. Objects are modeled as flexible constellations of parts. A probabilistic representation is used for all aspects of the object: shape, appearance, occlusion and relative scale. An entropy-based feature detector is used to select regions and their scale within the image. In learning the parameters of the scale-invariant object model are estimated. This is done using expectation-maximization in a maximum-likelihood setting. In recognition, this model is used in a Bayesian manner to classify images. The flexible nature of the model is demonstrated by excellent results over a range of datasets including geometrically constrained classes (e.g. faces, cars) and flexible objects (such as animals). <s> BIB006 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Introduction <s> This article describes the robot Stanley, which won the 2005 DARPA Grand Challenge. Stanley was developed for high-speed desert driving without human intervention. The robot’s software system relied predominately on state-of-the-art AI technologies, such as machine learning and probabilistic reasoning. This article describes the major components of this architecture, and discusses the results of the Grand Challenge race. <s> BIB007 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Introduction <s> Simultaneous localization and mapping (SLAM) is an important research area in robotics. Lately, systems that use a single bearing-only sensors have received significant attention and the use of visual sensors have been strongly advocated. In this paper, we present a framework for 3D bearing only SLAM using a single camera. We concentrate on image feature selection in order to achieve precise localization and thus good reconstruction in 3D. In addition, we demonstrate how these features can be managed to provide real-time performance and fast matching, to detect loop-closing situations. The proposed vision system has been combined with an extended Kalman Filter (EKF) based SLAM method. A number of experiments have been performed in indoor environments which demonstrate the validity and effectiveness of the approach. We also show how the SLAM generated map can be used for robot localization. The use of vision features which are distinguishable allows a straightforward solution to the "kidnapped-robot" scenario. <s> BIB008 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Introduction <s> The Scale-Invariant Feature Transform (or SIFT) algorithm is a highly robust method to extract and consequently match distinctive invariant features from images. These features can then be used to reliably match objects in diering images. The algorithm was rst proposed by Lowe [12] and further developed to increase performance resulting in the classic paper [13] that served as foundation for SIFT which has played an important role in robotic and machine vision in the past decade. <s> BIB009
Mobile robots are dependent on sensory information for their operation, but they still lack the ability to accurately sense the environmental input. Re-cent developments in sensor technology have provided a range of high-fidelity sensors at moderate costs, such as laser range finders, color cameras, and tactile sensors, thereby making it feasible to use mobile robots in real world applications BIB001 BIB005 BIB007 . However, the sensors typically need extensive manual calibration, which has to be repeated when environmental conditions change. This sensitivity to environmental changes is more pronounced in the case of color cameras -they require extensive manual calibration and are sensitive to illumination changes. Mobile robots however frequently have to operate in environments with non-uniform illuminations. Therefore, even though color is a rich source of information, mobile robot applications have, until recently, relied primarily on non-visual sensors BIB002 . Even when a robot is equipped with a camera, the focus has largely been on using local edge-based gradients computed from the intensity-image BIB009 . Current sensory processing methods on robots therefore fail to exploit the information encoded in color images and fail to eliminate the sensitivity to environmental changes. The need for autonomous operation is a major challenge to the widespread use of mobile robots. In order to be suitable for real-world applications, it is not sufficient for the robot to be equipped with good sensors -it is important for the robot to exploit all the available information. The robot should be able to autonomously learn appropriate models for environmental features, to recognize environmental changes, and adapt the learned models in response to such changes. Mobile robots are typically characterized by constrained computational resources, but they need to operate in real-time in order to respond to the dynamic changes in their environment. Autonomous learning and adaptation on mobile robots is hence a challenging problem. In this article the focus is on learning and adaptation in the context of visual input from a color camera. We specifically focus on: (a) color segmentation i.e. the process of clustering image regions with similar color and/or mapping individual image pixels to color labels such as red, yellow and purple, and on (b) color constancy i.e. the ability to assign the same color labels to objects under different illuminations. Mobile robot applications that do use color information, typically have some form of color segmentation as the first step, but this segmentation is sensitive to illumination changes. Figure 1 shows an example flowchart of visual processing using color information. The first stage of visual processing is color segmentation, which takes as input a color image in one of many color spaces (e.g. RGB or YCbCr) depending on the camera being used and the application domain. Color segmentation involves clustering contiguous image regions into meaningful groups based on color, and/or mapping image pixels to one of a predefined set of color labels. The output of the segmentation module is hence either a list of clustered image regions superimposed on the original image, or an image of the same size as the original image with each pixel labeled with the numerical index of the corresponding color label (e.g. "1" for red, "2" for blue). This clustering and mapping is typically computationally expensive, requires extensive human supervision, and is sensitive to illumination changes. The color-segmented image regions are the input to the object recognition module, where the segmented regions are used to find "objects" and other structures with well-defined properties such as size, shape, and color. These known properties of objects are used to determine heuristics and constraints that help choose the segmented regions that best match the constraints. Object recognition using color or other features is an extensively researched field BIB006  BIB003 that is not the focus of this article. Object recognition typically provides a list of detected objects and their locations in the image. The output of the object recognition module serves as the input to the next stage of visual processing, which could involve some form of 3D scene understanding, or could serve as input to a localization module on mobile robots. Objects and their relative arrangements are used to determine additional information about the 3D scene being observed. If a stereo camera is being used, for instance, the depth information can be combined with the information about the objects to reconstruct the 3D scene. On mobile robots the known map of the world (object sizes and locations in global coordinates) can be exploited to use the computed distances and angles to the detected objects to localize the robot i.e. find its pose (position and orientation) in global coordinates. The robot can also perform simultaneous localization and mapping (SLAM) BIB008 or determine the structure of the world based on the motion (SFM) BIB004 . The reconstructed 3D scene and estimated robot pose can be used by a highlevel module to choose the sequence of actions to perform in order to achieve a specific goal. For instance, if the robot is to navigate to a certain location or retrieve a target object from a specific location, the robot can use the available information to plan its actions so as to maximize the likelihood of success and minimize the use of the available resources. In this article we are primarily interested in the problem of color-based mobile robot operation, and we are specifically focused on the task of robust color segmentation -the first module in Figure 1 . Color segmentation is typically computationally expensive and sensitive to illumination. In order to make color-based mobile robot operation feasible, we need to enable real-time autonomous segmentation of input images into meaningful color labels, and enable adaptation to illumination changes in a dynamically changing environment. We therefore survey a set of representative algorithms that have been proposed to tackle color segmentation, color learning, illumination invariance, and the associated high-level action-planning, on mobile robots. We also look at approaches that just address the underlying vision problems. The goal is to determine the suitability of the state-of-the-art vision algorithms for mobile robot domains, and to identify the challenges that still need to be addressed in order to enable autonomous color-based mobile robot operation. Furthermore, we also identify key results that can be used to achieve autonomous operation with other visual features, thereby moving towards autonomous mobile robot operation under natural conditions.