A review of quality evaluation of digital libraries based on users ' perceptions <s> Satisfaction. This encompasses the following subcriteria: <s> The purpose of this report is to provide an overview of works on digital library evaluation that included data. The approach was first to analyze and then isolate (or even deconstruct) some 80 evaluation studies along the lines of: 1. construct that were evaluated; 2. context in which evaluation were conducted; 3. criteria that were chosen as a basis for evaluation, and 4. methods that were used. For each of these lines of analysis approaches taken in evaluation studies are identified, enumerated, and approximately ranked: a list of constructs evaluated as to entities or processes is given; context or approaches taken in evaluation are enumerated; numerous criteria used as a base of evaluation are classified; and finally the methodologies used are identified, listed and discussed. Evaluation of digital libraries is not widely practiced. The corpus in this analysis represents the majority of efforts in digital library evaluation that contain data. Conclusions, among others, speculate as to the reasons for a relatively low presence of evaluation in digital libray research and practice. <s> BIB001 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Satisfaction. This encompasses the following subcriteria: <s> Usability, functionality, and accessibility testing of digital library information services and products are essential for providing high quality services to users. This paper details a long-term, evolving effort to develop meaningful evaluations for assessing digital libraries. The authors of this paper have been engaged in a multi-year study to determine appropriate evaluation techniques, tools, and methodologies for the Florida Electronic Library (FEL) and other digital libraries. The evaluation protocols and approaches have been designed over time and iteratively through assessment efforts of the research team of other digital library initiatives and with multiple versions of the FEL. As such, this paper examines the process of developing, applying, and refining appropriate evaluation methodologies for the networked environment of libraries, as well as the implications of these methodologies. The approach taken in the research described herein relies on a combination of evaluation strategies applied iteratively ::: to assess libraries from the perspective of patron needs. A number of specific methods, as shown in this paper, can be ::: readily developed to provide such evaluations. The following goals guide the presentation in this paper of the ::: development through time of the methods and instruments created, tested, refined, and operationalized in functionality, usability, and accessibility testing by the researchers. The goals of this paper are to: 1) demonstrate the potential roles of multiple, iterative evaluation strategies in the development and refinement of digital libraries; 2) detail the methodologies that focus on how the services meet the needs of users; and 3) encourage further discussion of the uses of these multiple evaluation approaches in assessing these libraries. <s> BIB002 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Satisfaction. This encompasses the following subcriteria: <s> Digital library research has developed over the past decade, but little has been done on the identification of evaluation criteria, especially from users' perspectives. This article identifies users' criteria and applies them to the evaluation of existing digital libraries. Forty-eight subjects were instructed to develop and justify a set of essential criteria for the evaluation of digital libraries. At the same time, they were requested to evaluate existing digital libraries by applying the criteria that they were developing. A compilation of criteria developed by participants show that usability and collection quality were the most important criteria for evaluating digital libraries. Service quality, system performance efficiency, and user opinion solicitation were also deemed essential criteria. The author further compares digital library evaluation criteria identified by users and researchers and applied in previous research. The article reveals problems in current digital library design and development, discusses suggestions for improving digital library design, and presents areas for further research. <s> BIB003 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Satisfaction. This encompasses the following subcriteria: <s> Advances in the publishing world have emerged new models of digital library development. Open access publishing modes are expanding their presence and realize the digital library idea in various means. While user-centered evaluation of digital libraries has drawn considerable attention during the last years, these systems are currently viewed from the publishing, economic and scientometric perspectives. The present study explores the concepts of usefulness and usability in the evaluation of an e-print archive. The results demonstrate that several attributes of usefulness, such as the level and the relevance of information, and usability, such as easiness of use and learnability, as well as functionalities commonly met in these systems, affect user interaction and satisfaction. <s> BIB004
Ease of use. It evaluates the user's perception of the ease of use of the system. Organization of information. It evaluates whether the system's structure, layout, and organization meet the user's satisfaction. Labelling. It evaluates from the user's perception whether the system provides clear labelling, and whether the terminology used is easy to understand. Visual appearance. It evaluates the site's design to see whether it is visually attractive. Contents. It evaluates the authority and accuracy of the information provided. Error correction. It tests whether users can recover from mistakes easily, and whether they make mistakes easily because of the system's design. 4. Learnability. It evaluates how easily users can learn to use the system. Saracevic BIB001 analyses 80 evaluation studies taken from the object literature. As a result, he proposes a framework to classify the studies. In this framework, usability encompasses the criteria content, process, format and overall assessment, which are composed of the subcriteria summarized in Figure 2 . Snead et al. BIB002 distinguish between usability and accessibility: 1. Usability. This determines the extent to which a DL, in whole or in part, enables users to use its features intuitively. It encompasses the following subcriteria: • Navigation: the ability to traverse a site using available navigation site tools (e.g. back buttons, links, etc.). • Content presentation: the content is presented in a logical manner that is clear and easy to understand. • Labels: toolbars, buttons, icons, drop-down features are sensibly presented and labelled. • Search process: search features enhance location and retrieval of relevant materials. 2. Accessibility. This determines the extent to which a DL, in whole or in part, enables users with disabilities to interact with the DL. It encompasses the following subcriteria: • Alternative forms of content: users with visual or auditory disabilities are given access to all content through the provision of alternative, equivalent formats. • Colour independent: users with colour deficits and other visual disabilities can access all content (i.e., the DL site does not rely on specific colour to convey content). Tsakonas et al.  BIB004 propose a usability evaluation similar to Jeng's model. As summarized in Figure 2 , the main differences are related to the subcriteria organization. Xie BIB003 conducts an experiment in which users are instructed to develop a set of criteria for DL evaluation. The result regarding usability is summarized in Figure 2 .