A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria for quality evaluation of DLs <s> Evaluation of digital libraries (DLs) is essential for further development in this area. Whereas previous approaches were restricted to certain facets of the problem, we argue that evaluation of DLs should be based on a broad view of the subject area. For this purpose, we develop a new description scheme using four major dimensions: data/collection, system/technology, users, and usage. For each of these dimensions, we describe the major attributes. Using this scheme, existing DL test beds can be characterised. For this purpose, we have performed a survey by means of a questionnaire, which is now continued by setting up a DL meta-library. <s> BIB001 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria for quality evaluation of DLs <s> Digital libraries (DLs) are new and innovative information systems, under constant development and change, and therefore evaluation is of critical importance to ensure not only their correct evolution but also their acceptance by the user and application communities. The Evaluation activity of the DELOS Network of Excellence has performed a large-scale survey of current DL evaluation activities. This study has resulted in a description of the state of the art in the field, which is presented in this paper. The paper also proposes a new framework for the evaluation of DLs, as well as for recording, describing and analyzing the related research field. The framework includes a methodology for the classification of current evaluation procedures. The objective is to provide a set of flexible and adaptable guidelines for DL evaluation. <s> BIB002
This section outlines the criteria that have been proposed to evaluate the quality of DLs from the user's perspective. First, to provide a classification of such criteria, a conceptual model for DL evaluation, which is backed up by the Working Group on Evaluation of the DELOS Network of Excellence, is presented. Then the criteria are summarized. Fuhr et al. BIB001 propose a generic conceptual model for DL evaluation, which is composed of three non-orthogonal components: the users; the DL content; and the technological system that supports the DL content. According to Fuhr's model, 'content is king', and consequently the nature, extent and form of the DL content predetermine both the range of potential users and the required technology. Information retrieval provides the standard measures of precision and recall to evaluate the quality of the systemcontent interaction; however, their applicability to evaluation of the DL user experience has been called into question. So, to Fuhr et al. BIB002 , these measures have not been defined to 'model user satisfaction, result pertinence or system effectiveness for a given task content', and instead they advocate the development of DL-specific evaluation schemes. According to BIB001 , it is not the precision and recall measures that are inappropriate, but rather the benchmarking collections used in competitions such as TREC that 'lack the rich structure and inter-document relationships that are typical for DL collections'. Although Fuhr et al.'s model addresses system-centred evaluations BIB002 , it has remarkably influenced several usercentred proposals. In particular, Tsakonas et al.  propose a user-centred model focused on the relations between the components of Fuhr's model. These relations are shown in Figure 1 . • The content-system pair is related to performance criteria (precision, recall, response time, etc.). • The user-system pair is related to the usability 1 criterion, which defines the quality of the interaction between the user and the system. Usability evaluates whether the system is manipulated effectively by the user, in an efficient and enjoyable way that supports exploitation of all the available functionalities. A usable system is easy to learn, is flexible, and adapts to user preferences and skills. • The user-content pair is related to the usefulness criterion, which evaluates the relevance of the DL content to the user's tasks and needs. This paper is focused on the user-system and the user-content interactions (see the shadowed area in Figure 1 ). At present, there is no consensus on the definition of the usability and usefulness criteria, nor on their importance or correlation. The following subsections review available proposals on such issues.