A review of quality evaluation of digital libraries based on users ' perceptions <s> Heradio et al. 274 <s> The purpose of this report is to provide an overview of works on digital library evaluation that included data. The approach was first to analyze and then isolate (or even deconstruct) some 80 evaluation studies along the lines of: 1. construct that were evaluated; 2. context in which evaluation were conducted; 3. criteria that were chosen as a basis for evaluation, and 4. methods that were used. For each of these lines of analysis approaches taken in evaluation studies are identified, enumerated, and approximately ranked: a list of constructs evaluated as to entities or processes is given; context or approaches taken in evaluation are enumerated; numerous criteria used as a base of evaluation are classified; and finally the methodologies used are identified, listed and discussed. Evaluation of digital libraries is not widely practiced. The corpus in this analysis represents the majority of efforts in digital library evaluation that contain data. Conclusions, among others, speculate as to the reasons for a relatively low presence of evaluation in digital libray research and practice. <s> BIB001 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Heradio et al. 274 <s> Digital library research has developed over the past decade, but little has been done on the identification of evaluation criteria, especially from users' perspectives. This article identifies users' criteria and applies them to the evaluation of existing digital libraries. Forty-eight subjects were instructed to develop and justify a set of essential criteria for the evaluation of digital libraries. At the same time, they were requested to evaluate existing digital libraries by applying the criteria that they were developing. A compilation of criteria developed by participants show that usability and collection quality were the most important criteria for evaluating digital libraries. Service quality, system performance efficiency, and user opinion solicitation were also deemed essential criteria. The author further compares digital library evaluation criteria identified by users and researchers and applied in previous research. The article reveals problems in current digital library design and development, discusses suggestions for improving digital library design, and presents areas for further research. <s> BIB002 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Heradio et al. 274 <s> Advances in the publishing world have emerged new models of digital library development. Open access publishing modes are expanding their presence and realize the digital library idea in various means. While user-centered evaluation of digital libraries has drawn considerable attention during the last years, these systems are currently viewed from the publishing, economic and scientometric perspectives. The present study explores the concepts of usefulness and usability in the evaluation of an e-print archive. The results demonstrate that several attributes of usefulness, such as the level and the relevance of information, and usability, such as easiness of use and learnability, as well as functionalities commonly met in these systems, affect user interaction and satisfaction. <s> BIB003
Similarly to usability, most authors consider usefulness as a complex concept composed of several criteria. Figure 2 also sums up the criteria proposed by Xie BIB002 , Saracevic BIB001 and Tsakonas et al.  BIB003 . As a result of an experimental study, Xie BIB002 identifies the following criteria: 1. Scope. DL scope has to be clearly defined, so that users can immediately judge whether they have accessed the right DL. 2. Authority. Authority control is the practice of creating and maintaining index terms for bibliographic material. It enables cataloguers to disambiguate items with similar or identical headings (e.g. two authors who happen to have published under the same name can be distinguished from each other by adding middle initials, a descriptive epithet to the heading of both authors, etc.). In addition, authority control is used to collocate materials that logically belong together, although they present themselves differently (e.g. authority records are used to establish uniform titles, which can collocate all versions of a given work together even when they are issued under different titles). 3. Accuracy. If information is inaccurate, there is no reason for people to use it. 4. Completeness. A good DL covers its subjects thoroughly, and is able to provide information that meets the demands of users with varying levels of information need. 5. Currency. DL content should be updated frequently. Saracevic's BIB001 framework does not define usefulness criteria explicitly. Although the criteria summarized in Figure  2 are originally included as usability criteria, we have decided to reclassify them as usefulness criteria, to facilitate comparison of Saracevic's framework with other evaluation proposals. Tsakonas et al.  differentiate between goal and resource criteria: 1. Goal criteria are relevance (topical relevance, commitment with the quality of information), utility and complexity. 2. Resource criteria are currency, level of information (users' information searching behaviour has demonstrated that, although retrieval of full text resources is significant, other levels of information, such as abstracts, are also preferred), reliability and format. In addition, in BIB003 , Tsakonas et al. consider coverage of the deposited documents as an important usefulness criterion.