This paper describes PinView, a content-based image retrieval system that exploits implicit relevance feedback collected during a search session. PinView contains several novel methods to infer the intent of the user. From relevance feedback, such as eye movements or pointer clicks, and visual features of images, PinView learns a similarity metric between images which depends on the current interests of the user. It then retrieves images with a specialized online learning algorithm that balances the tradeoff between exploring new images and exploiting the already inferred interests of the user. We have integrated PinView to the content-based image retrieval system PicSOM, which enables applying PinView to real-world image databases. With the new algorithms PinView outperforms the original PicSOM, and in online experiments with real users the combination of implicit and explicit feedback gives the best results.
Later, also @cite_36 and @cite_14 @cite_60 and @cite_44 introduced their image retrieval systems using eye movements. The first one @cite_36 was based on a conceptual interface designed to be controlled completely by implicit gaze, providing a mix of a browsing and search tool. A small-scale online experiment was provided, but it cannot be used for drawing strong conclusions on the accuracy of the retrieval results. The second study mostly concentrated on the accuracy of inferring the relevance in @cite_60 and on fixation-weighted region matching between the query and database images in @cite_14 . The last one @cite_44 used gaze data for genuinely implicit relevance feedback by the means of reranking the results of Google Image Search. However, the system was not fully functional yet as the described experimantal evaluation was done in a non-interactive mode.
@cite_36 @cite_44 @cite_14 @cite_60
2049997905 33281282 1985622654 2065570765
In this paper we propose an implicit relevance feedback method with the aim to improve the performance of known Content Based Image Retrieval (CBIR) systems by re-ranking the retrieved images according to users' eye gaze data. This represents a new mechanism for implicit relevance feedback, in fact usually the sources taken into account for image retrieval are based on the natural behavior of the user in his her environment estimated by analyzing mouse and keyboard interactions. In detail, after the retrieval of the images by querying CBIRs with a keyword, our system computes the most salient regions (where users look with a greater interest) of the retrieved images by gathering data from an unobtrusive eye tracker, such as Tobii T60. According to the features, in terms of color, texture, of these relevant regions our system is able to re-rank the images, initially, retrieved by the CBIR. Performance evaluation, carried out on a set of 30 users by using Google Images and "pyramid" like keyword, shows that about the 87 of the users is more satisfied of the output images when the re-raking is applied. In this thesis visual search experiments are devised to explore the feasibility of an eye gaze driven search mechanism. The thesis first explores gaze behaviour on images possessing different levels of saliency. Eye behaviour was predominantly attracted by salient locations, but appears to also require frequent reference to non-salient background regions which indicated that information from scan paths might prove useful for image search. The thesis then specifically investigates the benefits of eye tracking as an image retrieval interface in terms of speed relative to selection by mouse, and in terms of the efficiency of eye tracking mechanisms in the task of retrieving target images. Results are analysed using ANOVA and significant findings are discussed. Results show that eye selection was faster than a computer mouse and experience gained during visual tasks carried out using a mouse would benefit users if they were subsequently transferred to an eye tracking system. Results on the image retrieval experiments show that users are able to navigate to a target image within a database confirming the feasibility of an eye gaze driven search mechanism. Additional histogram analysis of the fixations, saccades and pupil diameters in the human eye movement data revealed a new method of extracting intentions from gaze behaviour for image search, of which the user was not aware and promises even quicker search performances. The research has two implications for Content Based Image Retrieval: (i) improvements in query formulation for visual search and (ii) new methods for visual search using attentional weighting. Futhermore it was demonstrated that users are able to find target images at sufficient speeds indicating that pre-attentive activity is playing a role in visual search. A current review of eye tracking technology, current applications, visual perception research, and models of visual attention is discussed. A review of the potential of the technology for commercial exploitation is also presented. In this paper we present a novel gaze-based image retrieval application. The application is designed to be run on a dual monitor setup with a separate eye tracking device dedicated to each monitor. A source image is displayed on one monitor and the retrieved images are displayed on the second monitor. The system is completely gaze controlled. The user selects one or more objects or regions in the source image by fixating on them. The system then retrieves images containing similar objects from an image database. These are displayed in a grid on the second monitor. The user can then fixate on one of these images to select it as the new source image and the process can be repeated until a satisfactory image is found. A number of studies have recently used eye movements of a user inspecting the content as implicit relevance feedback for proactive retrieval systems. Typically binary feedback for images or text paragraphs is inferred from the gaze pattern. We seek to make such feedback richer for image retrieval, by inferring which parts of the image the user found relevant. For this purpose, we present a novel Bayesian mixture model for inferring possible target regions directly from gaze data alone, and show how the relevance of those regions can then be inferred using a simple classifier that is independent of the content or the task