A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria prioritization <s> There are two main kinds of approach to considering usability of any system: empirical and analytical. Empirical techniques involve testing systems with users, whereas analytical techniques involve usability personnel assessing systems using established theories and methods. We report here on a set of studies in which four different techniques were applied to various digital libraries, focusing on the strengths, limitations and scope of each approach. Two of the techniques, Heuristic Evaluation and Cognitive Walkthrough, were applied in text-book fashion, because there was no obvious way to contextualize them to the Digital Libraries (DL) domain. For the third, Claims Analysis, it was possible to develop a set of re-usable scenarios and personas that relate the approach specifically to DL development. The fourth technique, CASSM, relates explicitly to the DL domain by combining empirical data with an analytical approach. We have found that Heuristic Evaluation and Cognitive Walkthrough only address superficial aspects of interface design (but are good for that), whereas Claims Analysis and CASSM can help identify deeper conceptual difficulties (but demand greater skill of the analyst). However, none fit seamlessly with existing digital library development practices, highlighting an important area for further work to support improved usability. <s> BIB001 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria prioritization <s> In this paper the authors present a preliminary discussion on some of the results from a survey aimed to explore, describe and explain some of the usability characteristics in digital libraries evaluation in the Mexican context. The study is framed in the evaluation of a multinational and monolingual digital library: the Miguel de Cervantes Virtual Library, from the University of Alicante in Spain. The evaluators were Mexican “expert” users (i.e. Spanish-speaking professional university librarians specialized in electronic reference services) who were asked to carry on an evaluation instrument based on usability criteria as taken from some models in developed countries. Some questions that might be answered with future research are outlined in this paper. <s> BIB002 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria prioritization <s> Digital library research has developed over the past decade, but little has been done on the identification of evaluation criteria, especially from users' perspectives. This article identifies users' criteria and applies them to the evaluation of existing digital libraries. Forty-eight subjects were instructed to develop and justify a set of essential criteria for the evaluation of digital libraries. At the same time, they were requested to evaluate existing digital libraries by applying the criteria that they were developing. A compilation of criteria developed by participants show that usability and collection quality were the most important criteria for evaluating digital libraries. Service quality, system performance efficiency, and user opinion solicitation were also deemed essential criteria. The author further compares digital library evaluation criteria identified by users and researchers and applied in previous research. The article reveals problems in current digital library design and development, discusses suggestions for improving digital library design, and presents areas for further research. <s> BIB003 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria prioritization <s> ABSTRACT This paper proposes utilizing a combination of the Quality Function Deployment (QFD)–Kano model as a useful tool to evaluate service quality. The digital library of the University of Guadalajara (Mexico) is presented as a case study. Data to feed the QFD–Kano model was gathered by an online questionnaire that was made available to users on the library's website. The study makes clear that by listening to the voice of the customer (VOC), relevant information may be obtained about issues that should be improved in order to increase customer satisfaction. <s> BIB004
Several works try to identify which evaluation criteria are the most important from the user's perspective. As we shall see, there is as yet a lack of consensus on this issue. An experiment conducted by Kani-Zahibi et al.  shows that finding information easily and quickly in DLs and being able to be easily familiarized with DLs are the two most important DL requirements. In addition, the experiment shows that supporting collaborative knowledge working is a minor requirement, contradicting the opinion of Blandford et al. BIB001 . Xie reports in  that interface usability and system performance are the most important criteria. However, in a previous work BIB003 , she reported that the most relevant criteria were interface usability and collection quality. Zhang  reports an experiment in which different groups of stakeholders (end-users, librarians, DL developers, DL administrators and researchers) were asked to prioritize evaluation criteria for DLs. The research identifies a divergence among the stakeholder groups regarding what criteria should be used for DL evaluation. In the experiment, the service, interface and user evaluation criteria received greater consensus among the stakeholder groups regarding the importance ratings. In contrast, technology, context and content evaluation criteria received more divergent rankings among the groups. According to Zhang, the underlying reason for the lowest agreement on technology evaluation is presumably associated with the end-users' and librarians' unfamiliarity with technological issues. Meanwhile, complexity of content (i.e., the mixture of evaluation objects in terms of meta-information, information and collection) and the indirect relationship between DL use and context might be the two factors causing the larger divergence for content and context evaluation criteria. In BIB002 , Quijano-Solis et al. describe an experiment to register changes in users' perceptions of the main characteristics and preferred search options in DLs. In the experiment, users were first asked to answer a questionnaire related to criteria importance. Then they performed some tasks to become familiar with a DL. Finally, they answered a second questionnaire analogous to the first one. The result of the experiment shows great changes in the users' opinions. For instance, in the second questionnaire 65% of the participants said that searching by title was the preferred way to get information from DLs, contrasting with 39% who marked that option in the first questionnaire. According to Quijano-Solis et al. BIB002 , more research should be done to understand the nature of these changes, including their randomness. Garibay et al. BIB004 propose using the Kano model  to reprioritize evaluation criteria to take into account the relation between the criteria satisfaction perceived by the users compared with the satisfaction level that they would desire the DL to have. In order to adjust the importance of each criterion, the following equation is used: where imp adj is the adjusted importance of the criterion; imp 0 is the importance the criterion has according to the DL users; s 0 indicates how much the DL is currently satisfying the criterion according to the users' opinion; s 1 indicates how much the DL should satisfy the criterion according to the users' opinion; and k is the Kano parameter. The Kano model categorizes the attributes of a product or service based on how well it is able to satisfy customer needs. The model uses three categories, each with a different k value set by an expert team. The Kano categories are as follows: • One-dimensional attributes (performance needs) are typically what we get by just asking customers what they want. These requirements satisfy (or dissatisfy) in proportion to their presence in the product or service. High performance of a product leads to high customer satisfaction. • Attractive attributes (excitement needs). Their absence does not cause dissatisfaction, because they are not expected by customers: therefore customers are unaware of what they are missing. However, achievement of these attributes delights the customer, and satisfaction increases with increasing attribute performance. • Must-be attributes (basic needs). Customers take these for granted when they are fulfilled. However, if the product or service does not meet the basic needs sufficiently, the customer will become very dissatisfied. For example, suppose users value from 1 to 5 the criteria importance and how much the DL satisfies them. Initially, users think the importance of a certain criterion c is 4 (i.e., imp 0 = 4) and the DL satisfies the criterion at a level of 3 (i.e., s 0 = 3). However, users think the level of satisfaction should be 4 (i.e., s 1 = 4). Imagine c falls into the category of attractive attributes, which has been evaluated with k = 2 by the expert team. Hence the adjusted importance is 4.618: