A Review on Graph Based Segmentation <s> VII. ISSUES IN PERFORMANCE EVALUATION <s> This paper presents a database containing 'ground truth' segmentations produced by humans for images of a wide variety of natural scenes. We define an error measure which quantifies the consistency between segmentations of differing granularities and find that different human segmentations of the same image are highly consistent. Use of this dataset is demonstrated in two applications: (1) evaluating the performance of segmentation algorithms and (2) measuring probability distributions associated with Gestalt grouping factors as well as statistics of image region properties. <s> BIB001 </s> A Review on Graph Based Segmentation <s> VII. ISSUES IN PERFORMANCE EVALUATION <s> Unsupervised image segmentation is an important component in many image understanding algorithms and practical vision systems. However, evaluation of segmentation algorithms thus far has been largely subjective, leaving a system designer to judge the effectiveness of a technique based only on intuition and results in the form of a few example segmented images. This is largely due to image segmentation being an ill-defined problem-there is no unique ground-truth segmentation of an image against which the output of an algorithm may be compared. This paper demonstrates how a recently proposed measure of similarity, the normalized probabilistic rand (NPR) index, can be used to perform a quantitative comparison between image segmentation algorithms using a hand-labeled set of ground-truth segmentations. We show that the measure allows principled comparisons between segmentations created by different algorithms, as well as segmentations on different images. We outline a procedure for algorithm evaluation through an example evaluation of some familiar algorithms - the mean-shift-based algorithm, an efficient graph-based segmentation algorithm, a hybrid algorithm that combines the strengths of both methods, and expectation maximization. Results are presented on the 300 images in the publicly available Berkeley segmentation data set <s> BIB002 </s> A Review on Graph Based Segmentation <s> VII. ISSUES IN PERFORMANCE EVALUATION <s> Image segmentation is an important processing step in many image, video and computer vision applications. Extensive research has been done in creating many different approaches and algorithms for image segmentation, but it is still difficult to assess whether one algorithm produces more accurate segmentations than another, whether it be for a particular image or set of images, or more generally, for a whole class of images. To date, the most common method for evaluating the effectiveness of a segmentation method is subjective evaluation, in which a human visually compares the image segmentation results for separate segmentation algorithms, which is a tedious process and inherently limits the depth of evaluation to a relatively small number of segmentation comparisons over a predetermined set of images. Another common evaluation alternative is supervised evaluation, in which a segmented image is compared against a manually-segmented or pre-processed reference image. Evaluation methods that require user assistance, such as subjective evaluation and supervised evaluation, are infeasible in many vision applications, so unsupervised methods are necessary. Unsupervised evaluation enables the objective comparison of both different segmentation methods and different parameterizations of a single method, without requiring human visual comparisons or comparison with a manually-segmented or pre-processed reference image. Additionally, unsupervised methods generate results for individual images and images whose characteristics may not be known until evaluation time. Unsupervised methods are crucial to real-time segmentation evaluation, and can furthermore enable self-tuning of algorithm parameters based on evaluation results. In this paper, we examine the unsupervised objective evaluation methods that have been proposed in the literature. An extensive evaluation of these methods are presented. The advantages and shortcomings of the underlying design mechanisms in these methods are discussed and analyzed through analytical evaluation and empirical evaluation. Finally, possible future directions for research in unsupervised evaluation are proposed. <s> BIB003
Each graph based segmentation technique addresses a solution to some specific segmentation problems. Due to this, a technique might work well for a particular set of images but fail to segment satisfactorily for another set of images. Authors tested their algorithms using limited private set of images or by using standard test images like performance with other algorithms using qualitative evaluation or\and one or more quantitative criteria  BIB002 BIB003 . The main concern is whether the chosen dataset represent all necessary test scenarios like different illuminations, variations, conditions (say, day and night) and contrasts. Given this, creating a database of images and its ground truth for benchmarking an algorithm in intensity based segmentation would be an interesting area of research for computer vision and image processing community for two primary reasons: (1) to evaluate an algorithm with different test images which represent all possible test scenarios and (2) to compare different algorithms in a common platform and rank them based on their performance. Without this dataset and ground truth, it is very difficult to evaluate and compare different graph based segmentation methods. Though at this moment a few benchmarking systems are publically available, they are not very much suitable for evaluating an algorithm in intensity based segmentation. For example, most of the test cases of the Berkeley segmentation dataset and benchmarking BIB001 estimate the ability of an algorithm in identifying textures.