Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Color Learning <s> I define a computational model of color perception and color naming which is based in part on neurophysiological data and which can explain results in psychological, anthropological, and linguistic studies on color naming and categorization. In particular, the model explains the graded nature and foci of color categories. This model constitutes a semantic model of basic color terms, grounded in perception. It allows an artificial cognitive agent to name color samples, point out examples of named colors in its environment, and select objects from its environment by color, consistent with human performance on the same tasks. An algorithm and implementation for the computational model is presented, and evaluated theoretically and experimentally. ::: The contributions of this work are in autonomous agent architecture on the one hand, particularly in the area of symbol grounding and embodiment, and in perceptual modeling, particularly color vision, on the other hand. The application presented is a vertically integrated one, ranging from real visual input to symbolic description, and back. <s> BIB001 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Color Learning <s> Abstract The purpose is to arrive at recognition of multicolored objects invariant to a substantial change in viewpoint, object geometry and illumination. Assuming dichromatic reflectance and white illumination, it is shown that normalized color rgb, saturation S and hue H, and the newly proposed color models c 1 c 2 c 3 and l 1 l 2 l 3 are all invariant to a change in viewing direction, object geometry and illumination. Further, it is shown that hue H and l1l2l3 are also invariant to highlights. Finally, a change in spectral power distribution of the illumination is considered to propose a new color constant color model m1m2m3. To evaluate the recognition accuracy differentiated for the various color models, experiments have been carried out on a database consisting of 500 images taken from 3-D multicolored man-made objects. The experimental results show that highest object recognition accuracy is achieved by l1l2l3 and hue H followed by c1c2c3, normalized color rgb and m1m2m3 under the constraint of white illumination. Also, it is demonstrated that recognition accuracy degrades substantially for all color features other than m1m2m3 with a change in illumination color. The recognition scheme and images are available within the PicToSeek and Pic2Seek systems on-line at: http: //www.wins.uva.nl/research/isis/zomax/. <s> BIB002 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Color Learning <s> An autonomous approach for learning the colors of specific objects assumed to have known body spectral reflectances is developed for daylight illumination conditions. The main issue is to be able to find these objects autonomously in a set of training images captured under a wide variety of daylight illumination conditions, and to extract their colors to determine color space regions that are representative of the objects' colors and their variations. The work begins by modeling color formation under daylight using the color formation equations and the semi-empirical model of Judd, MacAdam and Wyszecki (CIE daylight model) for representing the typical spectral distributions of daylight. This results in color space regions that serve as prior information in the initial phase of learning which consists in detecting small reliable clusters of pixels having the appropriate colors. These clusters are then expanded by a region growing technique using broader color space regions than those predicted by the model. This is to detect objects in a way that is able to account for color variations which the model cannot due to its limitations. Validation on the detected objects is performed to filter out those that are not of interest and to eliminate unreliable pixel color values extracted from the remaining ones. Detection results using the color space regions determined from color values obtained by this procedure are discussed. <s> BIB003 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Color Learning <s> This thesis uses a physical, intrinsic reflection model to analyze the effects of shading and highlights on dielectric, non-uniform materials, such as plastics and paints. Since black-and-white images do not provide sufficient information to interpret these effects appropriately, this work uses color images. The colors seen on a dielectric object are an additive mixture of the object color and the illumination color, i.e.: the colors are a linear combination of those two color vectors. They therefore lie in a plane in the color space. Within the plane, they form a cluster that looks like a skewed T, due to the geometric properties of shading and highlight reflection. ::: When combined with a camera model, this description of physically possible color changes on one object can be used to guide an algorithm in interpreting color images. It is the basis of an algorithm that analyzes color variation in real color images. The algorithm looks for characteristic color clusters in the image and relates their shape to hypotheses on the object color and the amount of shading, as well as the position, strength and color of highlights on the object. It verifies the hypotheses by reapplying them back to the image, testing whether and how they apply to the pixels in the local image area from which the respective hypothesis was derived. In this way, it adapts the image interpretation process to local scene characteristics and reacts differently to color and intensity changes at different places in the image. ::: The result is an image segmentation that ignores color changes due to shading and highlights, as well as a set of intrinsic images, one describing the amount of shading at every pixel in the image, and the other one showing the amount highlight reflection at every pixel. Such intrinsic images can be a useful tool for other areas of computer vision that are currently disturbed by highlights in images, such as stereo vision and motion analysis. The approach can also be used to determine the illumination color from the highlights and as a preprocessor for methods to determine object shape from shading or highlights. This line of research may lead to physics-based image understanding methods that are both more reliable and more useful than traditional methods. (Abstract shortened with permission of author.) <s> BIB004 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Color Learning <s> We investigate the properties of a metric between two distributions, the Earth Mover's Distance (EMD), for content-based image retrieval. The EMD is based on the minimal cost that must be paid to transform one distribution into the other, in a precise sense, and was first proposed for certain vision problems by Peleg, Werman, and Rom. For image retrieval, we combine this idea with a representation scheme for distributions that is based on vector quantization. This combination leads to an image comparison framework that often accounts for perceptual similarity better than other previously proposed methods. The EMD is based on a solution to the transportation problem from linear optimization, for which efficient algorithms are available, and also allows naturally for partial matching. It is more robust than histogram matching techniques, in that it can operate on variable-length representations of the distributions that avoid quantization and other binning problems typical of histograms. When used to compare distributions with the same overall mass, the EMD is a true metric. In this paper we focus on applications to color and texture, and we compare the retrieval performance of the EMD with that of other distances. <s> BIB005 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Color Learning <s> This paper presents a vision system for robotic soccer which was tested on Sony's four legged robot Aibo. The input for the vision system are images of the camera and the sensor readings of the robot's head joints, the output are the positions of all recognized objects in relation to the robot. The object recognition is based on the colors of the objects and uses a color look-up table. The vision system creates the color look-up table on its own during a soccer game. Thus no pre-run calibration is needed and the robot can cope with inhomogeneous or changing light on the soccer field. It is shown, how different layers of color representation can be used to refine the results of color classification. However, the self-calibrated color look-up table is not as accurate as a hand-made. Together with the introduced object recognition which is very robust relating to the quality of the color table, the self-calibrating vision works very well. This robustness is achieved using the detection of edges on scan lines. <s> BIB006 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Color Learning <s> The extraction of high-level color descriptors is an increasingly important problem, as these descriptions often provide links to image content. When combined with image segmentation, color naming can be used to select objects by color, describe the appearance of the image, and generate semantic annotations. This paper presents a computational model for color categorization and naming and extraction of color composition. In this paper, we start from the National Bureau of Standards' recommendation for color names, and through subjective experiments, we develop our color vocabulary and syntax. To assign a color name from the vocabulary to an arbitrary input color, we then design a perceptually based color-naming metric. The proposed algorithm follows relevant neurophysiological findings and studies on human color categorization. Finally, we extend the algorithm and develop a scheme for extracting the color composition of a complex image. According to our results, the proposed method identifies known color regions in different color spaces accurately, the color names assigned to randomly selected colors agree with human judgments, and the description of the color composition of complex scenes is consistent with human observations. <s> BIB007 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Color Learning <s> We present a general-purpose framework for updating a robot’s observation model within the context of planning and execution. Traditional plan execution relies on monitoring plan step transitions through accurate state observations obtained from sensory data. In order to gather meaningful state data from sensors, tedious and time-consuming calibration methods are often required. To address this problem we introduce Reverse Monitoring, a process of learning an observation model through the use of plans composed of scripted actions. The automatically acquired observation models allow the robot to adapt to changes in the environment and robustly execute arbitrary plans. We have fully implemented the method in our AIBO robots, and our empirical results demonstrate its effectiveness. <s> BIB008 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Color Learning <s> Color segmentation is a challenging subtask in computer vision. Most popular approaches are computationally expensive, involve an extensive off-line training phase and/or rely on a stationary camera. This paper presents an approach for color learning on-board a legged robot with limited computational and memory resources. A key defining feature of the approach is that it works without any labeled training data. Rather, it trains autonomously from a color-coded model of its environment. The process is fully implemented, completely autonomous, and provides high degree of segmentation accuracy. <s> BIB009 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Color Learning <s> This article describes the robot Stanley, which won the 2005 DARPA Grand Challenge. Stanley was developed for high-speed desert driving without human intervention. The robot’s software system relied predominately on state-of-the-art AI technologies, such as machine learning and probabilistic reasoning. This article describes the major components of this architecture, and discusses the results of the Grand Challenge race. <s> BIB010 </s> Color Learning and Illumination Invariance on Mobile Robots: A Survey <s> Color Learning <s> Mobile robots are increasingly finding application in fields as diverse as medicine, surveillance and navigation. In order to operate in the real world, robots are primarily dependent on sensory information but the ability to accurately sense the real world is still missing. Though visual input in the form of color images from a camera is a rich source of information for mobile robots, until recently most people have focussed their attention on other sensors such as laser, sonar and tactile sensors. There are several reasons for this reliance on other relatively low-bandwidth sensors. Most sophisticated vision algorithms require substantial computational (and memory) resources and assume a stationary or slow moving camera, while many mobile robot systems and embedded systems are characterized by rapid camera motion and real-time operation within constrained computational resources. In addition, color cameras require time-consuming manual color calibration, which is sensitive to illumination changes, while mobile robots typically need to be deployed in a short period of time and often go into places with changing illumination. ::: It is commonly asserted that in order to achieve autonomous behavior, an agent must learn to deal with unexpected environmental conditions. However, for true extended autonomy, an agent must be able to recognize when to abandon its current model in favor of learning a new one, how to learn in its current situation, and also what features or representation to learn. This thesis is a fully implemented example of such autonomy in the context of color learning and segmentation, which primarily leverages the fact that many mobile robot applications involve a structured environment consisting of objects of unique shape(s) and color(s) - information that can be exploited to overcome the challenges mentioned above. The main contributions of this thesis are as follows. ::: First, the thesis presents a hybrid color representation that enables color learning both within constrained lab settings and in un-engineered indoor corridors, i.e. it enables the robot to decide what to learn. The second main contribution of the thesis is to enable a mobile robot to exploit the known structure of its environment to significantly reduce human involvement in the color calibration process. The known positions, shapes and color labels of the objects of interest are used by the robot to autonomously plan an action sequence to facilitate learning, i.e. it decides how to learn. The third main contribution is a novel representation for illumination, which enables the robot to detect and adapt smoothly to a range of illumination changes, without any prior knowledge of the different illuminations, i.e. the robot figures out when to learn. Fourth, as a means of testing the proposed algorithms, the thesis provides a real-time mobile robot vision system, which performs color segmentation, object recognition and line detection in the presence of rapid camera motion. In addition, a practical comparison is performed of the color spaces for robot vision – YCbCr, RGB and LAB are considered. The baseline system initially requires manual color calibration and constant illumination, but with the proposed innovations, it provides a self-contained mobile robot vision system that enables a robot to exploit the inherent structure and plan a motion sequence for learning the desired colors, and to detect and adapt to illumination changes, with minimal human supervision. ::: Keywords. Autonomous Color Learning, Illumination Invariance, Realtime Vision, Legged robots. <s> BIB011
An alternative to extensive manual color calibration is to enable the robot to autonomously segment images using learned models of color distributions. Attempts to learn color models or make them independent to illumination changes have produced reasonable success BIB002 BIB004 BIB003 ] but the methods either require the spectral reflectances of valid objects to be pre-computed, or require additional transformations. A mobile robot deployed in real-world domains is frequently required to operate in new environments where the surroundings, including objects, may change dynamically. It is therefore not feasible to measure the properties of the environment in advance. The additional transformations between color spaces or for image transformation compensation, operate at the level of pixels or small image regions. Such transformations are expensive to perform on mobile robots that require real-time operation. One question that arises in color learning is the choice of color spaces. Gevers and Smeulders BIB002 evaluate several color spaces to determine their suitability for recognizing multicolored objects invariant to significant changes in viewpoint, object geometry and illumination. They present a theoretical and experimental analysis of several color models. In addition to RGB, Intensity I, normalized RGB (rgb), saturation S, and Hue H, they propose three models: Each of the three channels of c 1 c 2 c 3 represents a ratio of color channel values, while each component of l 1 l 2 l 3 represents normalized square color differences. The model m 1 m 2 m 3 has each channel represent a ratio of color values of neighboring pixels (x 1 , x 2 )-see BIB002 for details. Assuming dichromatic reflection and white illumination, the authors show that rgb, saturation S, Hue H, and the newly proposed models c 1 c 2 c 3 , l 1 l 2 l 3 and m 1 m 2 m 3 are all invariant to the viewing direction, object geometry and illumination. Hue H and l 1 l 2 l 3 are also invariant to highlights, while m 1 m 2 m 3 is independent of the illumination color and inter-reflections under the narrow-band filter assumption. This analysis serves as a reference on the choice of color space, based on the requirements of the specific application. In parallel to the computer and robot vision research of color, human perception and representation of color have been extensively studied over the last several decades. In his thesis, Lammens BIB001 presents a nice analysis of the physiology of human color perception and its representation in language. He analyzes how humans perceive colors and represent them in language, and he proposes appropriate color models based on neurophysiological data and psychophysical findings in color perception. More recent work by Mojsilovic BIB007 provides a computational model for color naming based on experiments where human subjects are asked to name and distinguish between colored patches. She proposes a color naming metric that measures distances between colors analogous to the human perception of color. The metric, based on distances and angular measurements in the LAB and HSL color spaces  , is used to determine the color decomposition of images. Such results from the analysis of the human visual system can be exploited by the robot vision researchers. For instance, in their work on mobile robots, Sridharan and Stone  use the LAB color space  , that organizes colors similar to the human visual system, in order to provide robustness to minor illumination changes. In an approach representative of computer vision methods for robust color learning, Lauziere et al. BIB003 use the prior knowledge of the spectral reflectances of the objects under consideration for learning color models and recognizing objects under varying illumination. The color camera sensitivity curves are measured offline and used to recognize objects better under daylight illumination conditions. However, in addition to being computationally expensive, their method requires prior measurement of the camera characteristics and the spectral properties of the environment. Mobile robots are however frequently required to operate in new environments. On mobile robots, attempts to autonomously learn the color map have rarely been successful. Cameron and Barnes  present a technique based on scene geometry, where edges are detected in the image and closed figures are constructed to find image regions that correspond to certain known environmental features. The color information extracted from these regions is used to build the color classifiers, using the Earth Mover's distance (EMD) BIB005 as the cluster similarity metric. Illumination changes are tracked by associating the current classifiers with the previous ones based on the symbolic color labels. The edge detection, formation of closed figures, and clustering is computationally expensive (order of a few seconds per image) even with off-board processing. However, the technique introduces the promising idea of exploiting the structure inherent in robot application domains to learn color models. Chernova et al. BIB008 propose another approach that exploits the known structure of the application environment to learn color models. The robot moves along a pre-specified sequence of poses. At each pose, edges are detected and objects are found using domain knowledge, and image regions within the detected object were used as training samples to generate the color map. The technique works within controlled lab setting and the appropriate motion sequence is manually specified. For true autonomous behavior the robot should be able to autonomously plan action sequences suitable for color learning and adaptation to environmental changes. Jungel BIB006 modeled colors using three layers of color maps -the lowest layer represents a small subset of easily distinguishable colors, while the higher lay-ers represent a greater number of colors with increasing precision levels. Colors in each level are represented as cuboids relative to a reference color. In order to increase the processing speed very little computation is performed to distinguish between colors whose distributions are similar or overlap in the color space. Instead, domain specific constraints based on the known object properties such as colors and sizes are used during object recognition to disambiguate between object colors. The reference color is tracked with minor illumination changes, and the distributions of other colors in the color space (the cuboids) are displaced by the same amount as the reference color. Though it is an innovative solution for real-time robot applications, cuboids are not a good model for color distributions and different color distributions do not change by the same amount with illumination changes. As a result, the generated map is reported to be not as accurate as the hand-labeled one. [21,  BIB010 101, 22] . In order to navigate autonomously outdoors, issues such as color segmentation and illumination invariance had to be addressed. Mathematical models such as mixture-of-Gaussians (MoG) and histograms were used to model color distributions. For instance, the MoG model for the a priori probability density of pixel values (m) for a particular color label (l) is given by: where there are k weighted components in the model (weight = p(i)), each being a 3D Gaussian with a mean (µ) and a covariance matrix (Σ). However, none of these techniques exploit the domain knowledge significantly, and the modeling of colors is limited to distinguishing between "safe" and "unsafe" road regions. The visual input is used only to find obstacles or image edges, while high-level decisions are predominantly based on non-visual sensors such as laser and GPS. However, the colored objects in mobile robot domains encode a lot of useful information. The challenge is to find an appropriate represent for color distributions that may overlap in the color space. Sridharan and Stone BIB011 BIB009 address the challenge of autonomous real-time modeling of overlapping color distributions. The known structure of the robot's environment (positions, sizes and color labels of objects) is exploited to plan an action sequence suitable for learning the desired colors. Color distributions are modeled using a hybrid model of 3D Gaussians and 3D histograms: where N(µ l , Σ l ) is the 3D Gaussian for color l, while (b 1 , b 2 , b 3 ) are the histogram bin indices corresponding to the pixel values ( robot chooses the best representation for each color distribution based on autonomously collected image statistics. The learned color map provides segmentation and localization accuracies comparable to that obtained from a manually labeled map. The learning scheme however takes ≤ 6min of robot time instead of hours of human effort. Overlapping colors (e.g. red, pink, orange) are modeled efficiently but the sensitivity to illumination still remains. Segmentation performance with learned color maps is shown in Figure 6 . Table 2 Color modeling methods -eliminate manual calibration but cannot adapt. evaluates the color learning algorithms based on the criteria essential for mo-bile robot applications. Algorithms developed for color learning on mobile robots eliminate the need for manual training and do not need prior knowledge of color distributions, irrespective of whether color is used as a primary or secondary input source. However, when color is the main sensory input some robot applications exploit domain knowledge to model colors whose distributions may overlap in the color space (the "(Yes)" in the third row). Such domain knowledge is much easier to provide than hand labeling images or measuring properties of each new scene and the objects in them. Furthermore, existing algorithms can be used to learn most of this domain knowledge (e.g. SLAM algorithms  for learning the objects' locations). To summarize, standard color modeling algorithms are computationally expensive, and even when implemented successfully on mobile robots, they do not address the problem of detecting and adapting to environmental (illumination) changes.