A review of quality evaluation of digital libraries based on users ' perceptions <s> Usability and usefulness <s> The purpose of this report is to provide an overview of works on digital library evaluation that included data. The approach was first to analyze and then isolate (or even deconstruct) some 80 evaluation studies along the lines of: 1. construct that were evaluated; 2. context in which evaluation were conducted; 3. criteria that were chosen as a basis for evaluation, and 4. methods that were used. For each of these lines of analysis approaches taken in evaluation studies are identified, enumerated, and approximately ranked: a list of constructs evaluated as to entities or processes is given; context or approaches taken in evaluation are enumerated; numerous criteria used as a base of evaluation are classified; and finally the methodologies used are identified, listed and discussed. Evaluation of digital libraries is not widely practiced. The corpus in this analysis represents the majority of efforts in digital library evaluation that contain data. Conclusions, among others, speculate as to the reasons for a relatively low presence of evaluation in digital libray research and practice. <s> BIB001 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Usability and usefulness <s> This paper reviews how usability has been defined in the context of the digital library, what methods have been applied and their applicability, and proposes an evaluation model and a suite of instruments for evaluating usability for academic digital libraries. The model examines effectiveness, efficiency, satisfaction, and learnability. It is found that there exists an interlocking relationship among effectiveness, efficiency, and satisfaction. It also examines how learnability interacts with these three attributes. <s> BIB002 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Usability and usefulness <s> This study is to develop and evaluate methods and instruments for assessing the usability of digital libraries. It discusses the dimensions of usability, what methods have been applied in evaluating usability of digital libraries, their applicability, and criteria. It is found in the study that there exists an interlocking relationship among effective ness, effi ciency, and satisfaction. It provides operational cri teria for effectiveness, effi ciency, satisfaction, and learn ability. It discovers users’ criteria on ”ease of use,” ”or gani za tion of in formation,” ”terminology and labeling,” ”visual attractiveness,” and ”mistake recovery.” Common causes of ”user lostness” were found. ”Click cost” was examined. <s> BIB003 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Usability and usefulness <s> Usability, functionality, and accessibility testing of digital library information services and products are essential for providing high quality services to users. This paper details a long-term, evolving effort to develop meaningful evaluations for assessing digital libraries. The authors of this paper have been engaged in a multi-year study to determine appropriate evaluation techniques, tools, and methodologies for the Florida Electronic Library (FEL) and other digital libraries. The evaluation protocols and approaches have been designed over time and iteratively through assessment efforts of the research team of other digital library initiatives and with multiple versions of the FEL. As such, this paper examines the process of developing, applying, and refining appropriate evaluation methodologies for the networked environment of libraries, as well as the implications of these methodologies. The approach taken in the research described herein relies on a combination of evaluation strategies applied iteratively ::: to assess libraries from the perspective of patron needs. A number of specific methods, as shown in this paper, can be ::: readily developed to provide such evaluations. The following goals guide the presentation in this paper of the ::: development through time of the methods and instruments created, tested, refined, and operationalized in functionality, usability, and accessibility testing by the researchers. The goals of this paper are to: 1) demonstrate the potential roles of multiple, iterative evaluation strategies in the development and refinement of digital libraries; 2) detail the methodologies that focus on how the services meet the needs of users; and 3) encourage further discussion of the uses of these multiple evaluation approaches in assessing these libraries. <s> BIB004 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Usability and usefulness <s> Digital library research has developed over the past decade, but little has been done on the identification of evaluation criteria, especially from users' perspectives. This article identifies users' criteria and applies them to the evaluation of existing digital libraries. Forty-eight subjects were instructed to develop and justify a set of essential criteria for the evaluation of digital libraries. At the same time, they were requested to evaluate existing digital libraries by applying the criteria that they were developing. A compilation of criteria developed by participants show that usability and collection quality were the most important criteria for evaluating digital libraries. Service quality, system performance efficiency, and user opinion solicitation were also deemed essential criteria. The author further compares digital library evaluation criteria identified by users and researchers and applied in previous research. The article reveals problems in current digital library design and development, discusses suggestions for improving digital library design, and presents areas for further research. <s> BIB005 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Usability and usefulness <s> Advances in the publishing world have emerged new models of digital library development. Open access publishing modes are expanding their presence and realize the digital library idea in various means. While user-centered evaluation of digital libraries has drawn considerable attention during the last years, these systems are currently viewed from the publishing, economic and scientometric perspectives. The present study explores the concepts of usefulness and usability in the evaluation of an e-print archive. The results demonstrate that several attributes of usefulness, such as the level and the relevance of information, and usability, such as easiness of use and learnability, as well as functionalities commonly met in these systems, affect user interaction and satisfaction. <s> BIB006
According to Shackel [54] , the definition of informatics usability was probably first attempted in 1971 by Miller  in terms of measures for 'ease of use'. Since then, a wide variety of definitions for informatics usability have been proposed: for example, Jeng BIB002 reviews 15 different definitions. In this paper, we restrict our usability review to the DL context. Most authors consider usability as a complex concept, composed of several criteria. Figure 2 outlines the usability criteria and subcriteria considered by Evans et al.  , Jeng BIB003 BIB002  , Saracevic BIB001 , Snead et al. BIB004 , Tsakonas et al.  BIB006 , and Xie BIB005 : Figure 1 . Digital library evaluation model proposed by Tsakonas et al.  . Heradio et al. 272 1. Visibility of system status. The system should always keep users informed about what is going on, through appropriate feedback within a reasonable time. 2. Match between system and the real world. The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. 3. User control and freedom. Users often choose system functions by mistake, and will need a clearly marked 'emergency exit' to leave the unwanted state without having to go through an extended dialogue. 4. Consistency and standards. Users should not have to wonder whether different words, situations, or actions mean the same thing. 