A review of quality evaluation of digital libraries based on users ' perceptions <s> Discussions and challenges <s> There are two main kinds of approach to considering usability of any system: empirical and analytical. Empirical techniques involve testing systems with users, whereas analytical techniques involve usability personnel assessing systems using established theories and methods. We report here on a set of studies in which four different techniques were applied to various digital libraries, focusing on the strengths, limitations and scope of each approach. Two of the techniques, Heuristic Evaluation and Cognitive Walkthrough, were applied in text-book fashion, because there was no obvious way to contextualize them to the Digital Libraries (DL) domain. For the third, Claims Analysis, it was possible to develop a set of re-usable scenarios and personas that relate the approach specifically to DL development. The fourth technique, CASSM, relates explicitly to the DL domain by combining empirical data with an analytical approach. We have found that Heuristic Evaluation and Cognitive Walkthrough only address superficial aspects of interface design (but are good for that), whereas Claims Analysis and CASSM can help identify deeper conceptual difficulties (but demand greater skill of the analyst). However, none fit seamlessly with existing digital library development practices, highlighting an important area for further work to support improved usability. <s> BIB001 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Discussions and challenges <s> The purpose of this report is to provide an overview of works on digital library evaluation that included data. The approach was first to analyze and then isolate (or even deconstruct) some 80 evaluation studies along the lines of: 1. construct that were evaluated; 2. context in which evaluation were conducted; 3. criteria that were chosen as a basis for evaluation, and 4. methods that were used. For each of these lines of analysis approaches taken in evaluation studies are identified, enumerated, and approximately ranked: a list of constructs evaluated as to entities or processes is given; context or approaches taken in evaluation are enumerated; numerous criteria used as a base of evaluation are classified; and finally the methodologies used are identified, listed and discussed. Evaluation of digital libraries is not widely practiced. The corpus in this analysis represents the majority of efforts in digital library evaluation that contain data. Conclusions, among others, speculate as to the reasons for a relatively low presence of evaluation in digital libray research and practice. <s> BIB002 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Discussions and challenges <s> This paper reviews how usability has been defined in the context of the digital library, what methods have been applied and their applicability, and proposes an evaluation model and a suite of instruments for evaluating usability for academic digital libraries. The model examines effectiveness, efficiency, satisfaction, and learnability. It is found that there exists an interlocking relationship among effectiveness, efficiency, and satisfaction. It also examines how learnability interacts with these three attributes. <s> BIB003 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Discussions and challenges <s> ABSTRACT This paper proposes utilizing a combination of the Quality Function Deployment (QFD)–Kano model as a useful tool to evaluate service quality. The digital library of the University of Guadalajara (Mexico) is presented as a case study. Data to feed the QFD–Kano model was gathered by an online questionnaire that was made available to users on the library's website. The study makes clear that by listening to the voice of the customer (VOC), relevant information may be obtained about issues that should be improved in order to increase customer satisfaction. <s> BIB004
The importance of evaluating the quality of DLs from the user's perspective is well recognized by the DL community. However, research on the quality evaluation of DLs based on users' perceptions seems to be in an early stage. As outlined in Figure 2 , there are plenty of definitions for usability and usefulness. Because of this lack of a common lexicon, it is hard to contrast the experimental results obtained by different authors. For instance, Section 3 reports that, according to an experiment conducted by Kani-Zahibi et al.  , the criterion supporting collaborative knowledge working has low importance. Nevertheless, the results presented by Blandford et al. BIB001 support a contrary conclusion. These contradictory results may be a consequence of terminological differences among the criteria definitions managed by the authors. In addition, since most of the experiments have been made with a reduced number of subjects (e.g. Jeng BIB003 uses 41 subjects, and Xie  uses BIB002 , the results may be statistically non-significant. Regarding tools for measuring users' perceptions, two different approaches have been summarized in Section 4: Likert scales and fuzzy linguistic modelling. However, the two approaches have not been compared in the literature yet. Although Likert scales fall within the ordinal level of measurement, many authors treat individual responses as interval data  BIB003   BIB004 , without any justification for the assumption of interval status for Likert data. As we have noted, if the wrong statistical technique is used, researchers increase the chance of coming to the wrong conclusion about the significance of their research. To sum up, the user-centred evaluation of DLs requires the following challenges to be tackled: 1. A consensus on standard definitions for usability and usefulness has to be reached. 2. The minimal threshold of subjects to obtain statistically significant results on the importance and correlation of usability and usefulness criteria should be identified. 3. The assumption of interval status for Likert data in the DL context has to be justified. 4. The advantages and drawbacks of Likert scales compared with fuzzy linguistic modelling have to be identified. We propose Table 7 as a starting point, which should be developed in future work. According to this table, fuzzy linguistic modelling seems to produce better results (the details are presented in Sections 4.1 and 4.2). On the other hand, Likert scales support measurement of the user's opinion dispersion.