Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Bootstrapping Learning and Adaptation <s> Simultaneous localization and mapping (SLAM) is an important research area in robotics. Lately, systems that use a single bearing-only sensors have received significant attention and the use of visual sensors have been strongly advocated. In this paper, we present a framework for 3D bearing only SLAM using a single camera. We concentrate on image feature selection in order to achieve precise localization and thus good reconstruction in 3D. In addition, we demonstrate how these features can be managed to provide real-time performance and fast matching, to detect loop-closing situations. The proposed vision system has been combined with an extended Kalman Filter (EKF) based SLAM method. A number of experiments have been performed in indoor environments which demonstrate the validity and effectiveness of the approach. We also show how the SLAM generated map can be used for robot localization. The use of vision features which are distinguishable allows a straightforward solution to the "kidnapped-robot" scenario. <s> BIB001 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Bootstrapping Learning and Adaptation <s> Color segmentation is a challenging yet integral subtask of mobile robot systems that use visual sensors, especially since such systems typically have limited computational and memory resources. We present an online approach for a mobile robot to autonomously learn the colors in its environment without any explicitly labeled training data, thereby making it robust to re-colorings in the environment. The robot plans its motion and extracts structure from a color-coded environment to learn colors autonomously and incrementally, with the knowledge acquired at any stage of the learning process being used as a bootstrap mechanism to aid the robot in planning its motion during subsequent stages. With our novel representation, the robot is able to use the same algorithm both within the constrained setting of our lab and in much more uncontrolled settings such as indoor corridors. The segmentation and localization accuracies are comparable to that obtained by a time-consuming offline training process. The algorithm is fully implemented and tested on SONY Aibo robots. <s> BIB002 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Bootstrapping Learning and Adaptation <s> Mobile robots are increasingly finding application in fields as diverse as medicine, surveillance and navigation. In order to operate in the real world, robots are primarily dependent on sensory information but the ability to accurately sense the real world is still missing. Though visual input in the form of color images from a camera is a rich source of information for mobile robots, until recently most people have focussed their attention on other sensors such as laser, sonar and tactile sensors. There are several reasons for this reliance on other relatively low-bandwidth sensors. Most sophisticated vision algorithms require substantial computational (and memory) resources and assume a stationary or slow moving camera, while many mobile robot systems and embedded systems are characterized by rapid camera motion and real-time operation within constrained computational resources. In addition, color cameras require time-consuming manual color calibration, which is sensitive to illumination changes, while mobile robots typically need to be deployed in a short period of time and often go into places with changing illumination. ::: It is commonly asserted that in order to achieve autonomous behavior, an agent must learn to deal with unexpected environmental conditions. However, for true extended autonomy, an agent must be able to recognize when to abandon its current model in favor of learning a new one, how to learn in its current situation, and also what features or representation to learn. This thesis is a fully implemented example of such autonomy in the context of color learning and segmentation, which primarily leverages the fact that many mobile robot applications involve a structured environment consisting of objects of unique shape(s) and color(s) - information that can be exploited to overcome the challenges mentioned above. The main contributions of this thesis are as follows. ::: First, the thesis presents a hybrid color representation that enables color learning both within constrained lab settings and in un-engineered indoor corridors, i.e. it enables the robot to decide what to learn. The second main contribution of the thesis is to enable a mobile robot to exploit the known structure of its environment to significantly reduce human involvement in the color calibration process. The known positions, shapes and color labels of the objects of interest are used by the robot to autonomously plan an action sequence to facilitate learning, i.e. it decides how to learn. The third main contribution is a novel representation for illumination, which enables the robot to detect and adapt smoothly to a range of illumination changes, without any prior knowledge of the different illuminations, i.e. the robot figures out when to learn. Fourth, as a means of testing the proposed algorithms, the thesis provides a real-time mobile robot vision system, which performs color segmentation, object recognition and line detection in the presence of rapid camera motion. In addition, a practical comparison is performed of the color spaces for robot vision – YCbCr, RGB and LAB are considered. The baseline system initially requires manual color calibration and constant illumination, but with the proposed innovations, it provides a self-contained mobile robot vision system that enables a robot to exploit the inherent structure and plan a motion sequence for learning the desired colors, and to detect and adapt to illumination changes, with minimal human supervision. ::: Keywords. Autonomous Color Learning, Illumination Invariance, Realtime Vision, Legged robots. <s> BIB003 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Bootstrapping Learning and Adaptation <s> A major challenge in the path of widespread use of mobile robots is the ability to function autonomously, learning useful features from the environment and using them to adapt to environmental changes. We propose an algorithm for mobile robots equipped with color cameras that allows for smooth operation under illumination changes. The robot uses image statistics and the environmental structure to autonomously detect and adapt to both major and minor illumination changes. Furthermore, the robot autonomously plans an action sequence that maximizes color learning opportunities while minimizing localization errors. Our approach is fully implemented and tested on the Sony AIBO robots. <s> BIB004
Though color learning, illumination invariance and action-planning have been discussed as separate modules, there exist inter-dependencies that can be exploited to create a predominantly autonomous real-time color-based robot vision system. As an example, we describe our work that enables a mobile robot to autonomously plan actions that facilitate color learning and smooth adaptation to illumination change. Our experiments were run on the Sony ERS-7 Aibos (four-legged robots) [82] and other wheeled robot platforms BIB003 . One standard application domain for the Aibos is the RoboCup initiative  , where teams of four robots play a competitive game of soccer on an indoor soccer field (4m × 6m) -the long-term goal is to create a team of humanoid robots that can compete against a human soccer team under natural conditions by the year 2050. The robot's primary sensor is a color camera in the nose. It has three degrees-of-freedom in each leg, and three more in its head. The field has four color-coded markers spread around the boundary, in addition to the goals. The robot detects these markers in the input images captured at 30Hz, and localizes itself in order to coordinate its activities with its teammates and score goals against the opponents. All processing for vision, localization, motion and strategy is performed on-board using a 576MHz processor. The flow of information is similar to that described in Figure 1 (Section 1), though the architecture is "event-driven". Figure 10 shows an image of the Aibo and the soccer field. When the robot is first deployed in the environment, it has no prior knowledge of the color distributions or the possible illuminations. It knows the positions, sizes and color labels of certain objects in its environment (the structure), and it knows its approximate starting pose. Existing approaches can be used to learn a significant portion of this structure (e.g. SLAM BIB001 ). The robot uses the known structure to heuristically plan a motion sequence and learn models for the color distributions of interest BIB002 , thereby obtaining the color map under the current illumination. This learning takes ≈ 6min of robot time instead of hours of human effort. While learning color models, the robot collects random images of the surroundings to model the illumination: color space histograms and second order statistics as described in Section 5. Furthermore, as the robot moves in the environment during and after the color learning, it collects statistics to learn a model for the motion errors and the feasibility of learning colors at various locations (≈ 1 hour of autonomous learning). For all subsequent changes in object configuration or illumination, our algorithm works as follows. Sudden or large illumination changes, for instance turning on (or off) a few lamps in the room, are detected by comparing the current image statistics with the stored image statistics corresponding to the known illuminations. Once a large illumination change is detected, the robot adapts by using the learned motion-error and color-learning feasibility models to plan an action sequence that simultaneously maximizes color learning opportunities while minimizing localization errors: a new color map is learned. Minor illumination changes, for instance dimming a few lamps in the room, are detected by tracking the changes in the current color distribution models using Gaussian or histogram merging. As an example of the experiments performed to evaluate our algorithms, we describe the experiment that tested the autonomous adaptation to illumination changes. We measured the time taken by the robot to find-and-walk-toobject: it started out near the center of the field with the object placed near the penalty box of the opponent's goal. Sudden + Adapt major,minor 9.72 ± 0.51 0 Sudden + Slow + Adapt major,minor 10.32 ± 0.83 0 Table 5 Time taken to find-and-walk-to-object. The inclusion of Adapt major and Adapt minor enables the robot to operate autonomously over a range of illuminations. When the illumination does not change, the robot can find-and-walk-to-object in 6.18 ± 0.24 seconds. When the illumination is slowly changed as the robot performs the task, using just Adapt major does not help -large variance in sec-ond row. With Adapt minor the results are as good as before (6.24 ± 0.31secs). Next, when the illumination is suddenly changed as the robot starts walking towards the seen object, using just Adapt minor does not help. The robot totally fails to perform the task most of the time, as seen by the large number of failures (fourth row, third column). With the combined strategy, i.e. with both Adapt major and Adapt minor , the robot can perform the task efficiently, the additional time being used to ensure that a change in illumination did occur (9.72 ± 0.51secs). We infer that the improvement is primarily due to Adapt major . In all these experiments, when the illumination changes significantly, the robot is put in conditions similar to the ones for which it has already learned color and illumination models. Hence, once a major change is detected, Adapt major consists of transitioning to the suitable models. Finally the robot is made to find-and-walk-to-object while the illumination is changed significantly initially, and after 3sec is changed slowly over the next 5sec. The robot is able to do the task in 10.32±0.83sec iff both Adapt major and Adapt minor are used. We therefore conclude that a combination of the schemes is essential to operate under a range of illumination intensities (≈ 400Lux to ≈ 1600Lux) and color temperatures (2300K -4000K) BIB004 . Our approach has a couple of key features. First, the robot bootstraps based on the available information. During color learning the robot initially has no knowledge of color distributions and hence chooses to execute only actions that require minimal movement. As the robot learns a few colors it is able to localize better and hence plan and execute subsequent actions more robustly. Second, the robot requires minimal human supervision. The initial specification of the structure and an action sequence suitable for learning are a lot easier to provide than hand-labeling images. All subsequent detection and adaptation to change is autonomous. Furthermore, the learning and adaptation algorithms have been tested in un-engineered indoor corridors on multiple robot platforms, while the action selection algorithm can be used to learn features in addition to color. The results described here are hence a step towards autonomous color-based mobile robot operationthey show that the inter-dependencies between color learning, segmentation, illumination invariance and action planning can be exploited to create an autonomous, learning and adapting mobile robot vision system. Additional videos and images can be viewed online: www.cs.utexas.edu/~AustinVilla/ ?p=research/illuminvar_colorlearn, www.cs.utexas.edu/~AustinVilla/ ?p=research/gen_color