A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria measurement and analysis <s> THIS paper describes a technique which has been developed for the measurement of race prejudice. This technique differs from most prejudice inventories in that it avoids the following assumptions: (a) that the individual can say, to his own or the investigator's satisfaction, "This is how prejudiced I am," and (b) that, to the extent that the individual can accurately assess his degree of antipathy, he will report honestly the findings of such introspection. Most sociologists would perhaps agree that race attitudes rarely reside on a completely articulate level. Even where the individual holds to intellectual or ideological convictions which would seem to leave no room for out-group antipathies, such do persevere. Thus, we may expect the number of Americans who honestly think themselves "unprejudiced" to be considerably larger than effective research would reveal. Moreover, the number who present themselves as unprejudiced probably exceeds considerably the number who honestly, though often inaccurately, see themselves in this light. Most indirect techniques for the measurement of attitudes have their rationale in observations such as these. The instrument to be described here is not, however, indirect in the usual sense of the word; it does not seek responses to items apparently unrelated to the attitudes investigated. We do, however, seek to measure prejudice in a manner less direct than is true of the usual prejudice scale. In our instrument we seek to measure anti-Negro prejudice. Persons are called upon to respond on social distance scales to whites and Negroes who occupy a variety of occupational positions. The measure of prejudice is derived through the summation of the differences in distance responses to Negroes as opposed to whites in the same occupations. Thus, for lack of a better label, <s> BIB001 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria measurement and analysis <s> The aim of this paper is to model the processes of the aggregation of weighted information in a linguistic framework. Three aggregation operators of weighted linguistic information are presented: linguistic weighted disjunction operator, linguistic weighted conjunction operator, and linguistic weighted averaging operator. A study of their axiomatics is presented to demonstrate their rational aggregation. <s> BIB002 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria measurement and analysis <s> The Perseus Digital Library, (PDL) is one of the primary digital resources for the humanities. Under continuous development since 1987, the project has included an ongoing evaluation component that aims to understand the effects of access to digitized source materials in the humanities. A summary of the PDL genesis and current status is given and the multifaceted and longitudinal evaluation effort is described. A brief synthesis of results is provided and reflections on the evaluation along with recommendations for DL evaluation are given. Introduction Digital libraries marry the missions, techniques, and cultures of physical libraries with the capabilities and cultures of computing and telecommunications. Evaluating digital libraries is a bit like judging how successful is a marriage. A lot depends on how successful the partners are as individuals as well as the emergent conditions made possible by the union. All three entities —the two individuals and the gestalt union--are of course influenced by their context as well. The difficulties arise from the complexity of mutually self-adapting systems interacting in a rich environment. Metrics for success for component parts of a complex system may be distinct from the metrics for success of the marriage (e.g., success for an individual partner is typically necessary but not sufficient to insure success for the marriage). Digital libraries (DLs) are extensions and augmentations of physical libraries (Marchionini & Fox, 1999). As extensions, we might evaluate the individual partners using existing techniques and metrics. Assessing the impacts of libraries on patron's lives and the larger social milieu are the ultimate goals of evaluation but the practical difficulties of assessing such complex and varied impacts causes us to measure the effectiveness and efficiencies of library operations and services as surrogates for these impacts. Metrics such as circulation, collection size and growth rate, patron visits, reference questions answered, patron satisfaction, and financial stability may be used to assess physical library performance in this regard. Clearly, these are metrics that may be points of departure for evaluating digital libraries, but they are not sufficient to characterize the new, rapidly emerging entity. Evaluation criteria for digital technologies can also be useful. For example, metrics such as response time, storage capacity, transfer rate, user satisfaction, and cost per operation may be useful in assessing technological components but may not be sufficient to characterize DL performance, let alone impact. As extensions of physical libraries and digital technologies, these metrics are good starting points but we must look further to consider the effects of DLs as augmentations that provide new services, products, and capabilities. <s> BIB003 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria measurement and analysis <s> Abstract.As experimental digital library testbeds gain wider acceptance and develop significant user bases, it becomes important to investigate the ways in which users interact with the systems in practice. Transaction logs are one source of usage information, and the information on user behavior can be culled from them both automatically (through calculation of summary statistics) and manually (by examining query strings for semantic clues on search motivations and searching strategy). We have conducted a transaction log analysis on user activity in the Computer Science Technical Reports Collection of the New Zealand Digital Library, and report insights gained and identify resulting search interface design issues. Specifically, we present the user demographics available with our library, discuss the use of operators and search options in queries, and examine patterns in query construction and refinement. We also describe common mistakes in searching, and examine the distribution of query terms appearing in the logs. <s> BIB004 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria measurement and analysis <s> An integrated visual thesaurus and results browser to support information retrieval was designed using a task model of information searching. The system provided a hierarchical thesaurus with a results cluster display representing similarity between retrieved documents and relevance ranking using a bullseye metaphor. Latent semantic indexing (LSI) was used as the retrieval engine and to calculate the similarity between documents. The design was tested with two information retrieval tasks. User behaviour, performance and attitude were recorded as well as usability problems. The system had few usability problems and users liked the visualizations, but recall performance was poor. The reasons for poor/good performance were investigated by examining user behaviour and search strategies. Better searchers used the visualizations more effectively and spent longer on the task, whereas poorer performances were attributable to poor motivation, difficulty in assessing article relevance and poor use of system visualizations. The bullseye browser display appeared to encourage limited evaluation of article relevance on titles, leading to poor performance. The bullseye display metaphor for article relevance was understood by users; however, they were confused by the concept of similarity searching expressed as visual clusters. The conclusions from the study are that while visual user interfaces for information searching might seem to be usable, they may not actually improve performance. Training and advisor facilities for effective search strategies need to be incorporated to enhance the effectiveness of visual user interfaces for information retrieval. <s> BIB005 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria measurement and analysis <s> Usability evaluation is an increasingly important part of the user interface design process. However, usability evaluation can be expensive in terms of time and human resources, and automation is therefore a promising way to augment existing approaches. This article presents an extensive survey of usability evaluation methods, organized according to a new taxonomy that emphasizes the role of automation. The survey analyzes existing techniques, identifies which aspects of usability evaluation automation are likely to be of use in future research, and suggests new ways to expand existing approaches to better support usability evaluation. <s> BIB006 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria measurement and analysis <s> In the era of digital libraries, Web-based electronic databases have become important resources for education and research, providing functionality and ease of use superior to print products. Analysis of usage of such online systems can provide valuable information on user behavior, and on usage of electronic information in general. Furthermore, the findings can be used to improve effectiveness of these electronic systems and identify areas for improvement, ranging from user interface and functionality to documentation and product training. This article analyzes usage of the Taiwan-based ScienceDirect OnSite E-journal system, one of the largest and most heavily used full-text Science, Technology, and Medicine (STM) databases worldwide. <s> BIB007 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria measurement and analysis <s> We compare the usage of a Digital Library with many different categories of collections, by examining its log files for a period of twenty months, and we conclude that the access points that the users mostly refer to, depend heavily on the type of content of the collection, the detail of the existing meta-data and the target user group. We also found that most users tend to use simple query structures (e.g. only one search term) and very few and primitive operations to accomplish their request. Furthermore, as they get more experienced, they reduce the number of operations in their sessions. <s> BIB008 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria measurement and analysis <s> Digital libraries (DLs) are among the most complex kinds of information systems, due in part to their intrinsic multi disciplinary nature. Nowadays DLs are built within monolithic, tightly integrated, and generally inflexible systems -- or by assembling disparate components together in an ad-hoc way, with resulting problems in interoperability and adaptability. More importantly, conceptual modeling, requirements analysis, and software engineering approaches are rarely supported, making it extremely difficult to tailor DL content and behavior to the interests, needs, and preferences of particular communities. In this paper, we address these problems. In particular, we present 5SL, a declarative language for specifying and generating domain-specific digital libraries. 5SL is based on the 5S formal theory for digital libraries and enables high-level specification of DLs in five complementary dimensions, including: the kinds of multimedia information the DL supports (Stream Model); how that information is structured and organized (Structural Model); different logical and presentational properties and operations of DL components (Spatial Model); the behavior of the DL (Scenario Model); and the different societies of actors and managers of services that act together to carry out the DL behavior (Societal Model). The practical feasibility of the approach is demonstrated by the presentation of a 5SL digital library generator for the MARIAN digital library system. <s> BIB009 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria measurement and analysis <s> Abstract This study applies the fuzzy set theory to evaluate the service quality of airline. Service quality is a composite of various attributes, among them many intangible attributes are difficult to measure. This characteristic introduces the obstacles for respondent in replying to the survey. In order to overcome the issue, we invite fuzzy set theory into the measurement of performance. By applying AHP in obtaining criteria weight and TOPSIS in ranking, we found the most concerned aspects of service quality are tangible and the least is empathy. The most concerned attribute is courtesy, safety and comfort. <s> BIB010 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria measurement and analysis <s> Evaluation of digital libraries from a user perspective poses immense challenges. Any digital library serves different types of user, with different needs and levels of understanding, and has to be acceptable - not just in terms of basic usability, but also in terms of supporting work as a social, contextualized activity. Different approaches to evaluation are needed for answering different evaluation questions; their broader relationship to design practice also needs to be considered. These issues are discussed in outline in this extended abstract. <s> BIB011 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria measurement and analysis <s> Digital libraries (DLs) are complex information systems and therefore demand formal foundations lest development efforts diverge and interoperability suffers. In this article, we propose the fundamental abstractions of Streams, Structures, Spaces, Scenarios, and Societies (5S), which allow us to define digital libraries rigorously and usefully. Streams are sequences of arbitrary items used to describe both static and dynamic (e.g., video) content. Structures can be viewed as labeled directed graphs, which impose organization. Spaces are sets with operations on those sets that obey certain constraints. Scenarios consist of sequences of events or actions that modify states of a computation in order to accomplish a functional requirement. Societies are sets of entities and activities and the relationships among them. Together these abstractions provide a formal foundation to define, relate, and unify concepts---among others, of digital objects, metadata, collections, and services---required to formalize and elucidate "digital libraries". The applicability, versatility, and unifying power of the 5S model are demonstrated through its use in three distinct applications: building and interpretation of a DL taxonomy, informal and formal analysis of case studies of digital libraries (NDLTD and OAI), and utilization as a formal basis for a DL description language. <s> BIB012 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria measurement and analysis <s> This study presents findings with a particular digital library system, the Belgian‐American Research Collection at the University of Wisconsin Digital Collections Center (UWDCC). It focuses on the ways in which a usability study can inform the future design of the collection and other UWDCC projects. The study uses two formal testing instruments: a focus group and a task‐oriented field test. Both of these techniques yielded solid data about user expectations, problems and actions with the collection. While the findings can aid UWDCC designers as they begin to envision another iteration of the Belgian‐American Research Collection, the findings also point to a need for a shift in the development of more complete digital library systems. This study proposes the development of digital library systems that have a strong orientation toward users and usability as well as advanced search interfaces and full metadata. In the end, it is not just about cataloging and organizing information; it is also about presenting information logically and intuitively for your users. <s> BIB013 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria measurement and analysis <s> This paper presents early findings of a unique analysis that related questionnaire data to site usage as recorded in the transaction log reports of ScienceDirect, for the same people. Its focus is the differences in the online behaviour of three types of navigational users: those accessing the site via a gateway (either via a reference hyperlink or subject search facility), those using the on site search facility and those employing menus. Towards this end 16,865 sessions were analysed and grouped by navigational entry and compared over three types of online behaviour: the viewing of articles in press (AIP), the number of different journals viewed in a session and the viewing of old material. A strong association was found between form of navigation and behavioural trait. Those using menus were more likely to view AIPs, while those using the search facility were more likely to view a greater number of different journals and were more likely to view older material. This supports a hypothesis proposed by Nicholas et al. (2006) that use of the online searching facility increases the visibility of material irrespective of journal and age and results in a greater use of older material and a more diverse journal use compared to other online and off-line information retrieval methods. Although research has been undertaken on the different strategies that users employ to navigate and find their way around a collection of content (e.g. a digital library), this we believe is the first time the effect of different navigational strategies on outcomes (for example, what is viewed) has been investigated. <s> BIB014 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Criteria measurement and analysis <s> It remains unclear whether the recent explosion of medical Internet Digital Libraries (DLs), enabled by substantial investments into eHealth by national governments and international agencies, has brought the desired improvements. As ultimately life-critical applications, medical DLs play a crucial role in delivering evidence to professionals and empowering patients. However, little attention has been given to impact evaluation with domain experts in real settings to assess whether they actually make a difference to clinical practice. ::: ::: In this paper we describe a novel evaluation framework --- Impact-ED --- developed at CeRC to fill the gap in impact evaluation research taking into account the community, content, services and technology dimensions of DLs. We present an account of Impact-ED's application in assessing the impact of the National Resource for Infection Control in the UK (NRIC www.nric.org.uk) --- a real-world medical DL used by over 40 000 professionals monthly. <s> BIB015
As noted by Marchionini BIB003 , the literature sometimes bristles with debates over basic approaches to evaluation, especially with respect to qualitative versus quantitative measures. According to Blandford et al. BIB011  , quantitative approaches (typically involving controlled studies) can be useful in understanding the effects of small but meaningful changes on the design of DLs. On the other hand, qualitative methods, whether applied within a laboratory setting (e.g. think-aloud protocols) or in the user's context of work, can be used in a more exploratory way to identify factors for success. There is an increasing trend to blend quantitative and qualitative data within a study to provide a broader and deeper perspective. This approach is called triangulation BIB003 BIB015 . In obtaining the data for quality evaluation of DLs, automated techniques have been used. In this category we can include the analysis of transaction logs BIB014 BIB004 BIB007 BIB008 , a widely used technique that examines the activity of users in a given time session, and the 5SQual tool, a more complex approach proposed by Moreira et al.  . Such a tool is grounded in the formal model 5S for DLs BIB009 BIB012 . However, automated techniques have been criticized for their lack of ability to produce interpretable and qualitative data that help evaluators understand the nature of the usability problem and the impact it has on the user interaction BIB006 . Furthermore, as mentioned in Section 1, since DLs are destined to serve user communities, the user's opinion is one of the main aspects to be considered in the quality evaluation of DLs. For these reasons, techniques that require the user's participation such as interviews and questionnaires are the focus of this review, and the prime methods for collecting qualitative data [8, 18, 25, 31-33, 35, 36, 38-47, 49-53] . In addition, observations in which user actions are recorded are also common methods for obtaining data  BIB003 BIB013 BIB005 . In these methods, to obtain the users' quality assessment, they are invited to fill in a survey built on the set of criteria. To measure quality, conventional measurement tools used by the users are devised on cardinal or ordinal scales (Likert scales BIB001 ). However, the scores do not necessarily represent the user's preference. This is because respondents have to internally convert preference to scores, and the conversion may introduce distortion of the preference BIB010 . For this reason, a recent method proposes using ordinal fuzzy linguistic modelling  BIB002 to represent the user's perceptions, and computing tools with words based on the linguistic aggregation operators LOWA  and LWA BIB002 to compute the quality assessments. The following subsections sum up these two alternative approaches, used in the context of DL evaluation to compute quality measurements based on users' perceptions: Likert scales and fuzzy linguistic modelling.