A review of quality evaluation of digital libraries based on users ' perceptions <s> Likert scales <s> THIS paper describes a technique which has been developed for the measurement of race prejudice. This technique differs from most prejudice inventories in that it avoids the following assumptions: (a) that the individual can say, to his own or the investigator's satisfaction, "This is how prejudiced I am," and (b) that, to the extent that the individual can accurately assess his degree of antipathy, he will report honestly the findings of such introspection. Most sociologists would perhaps agree that race attitudes rarely reside on a completely articulate level. Even where the individual holds to intellectual or ideological convictions which would seem to leave no room for out-group antipathies, such do persevere. Thus, we may expect the number of Americans who honestly think themselves "unprejudiced" to be considerably larger than effective research would reveal. Moreover, the number who present themselves as unprejudiced probably exceeds considerably the number who honestly, though often inaccurately, see themselves in this light. Most indirect techniques for the measurement of attitudes have their rationale in observations such as these. The instrument to be described here is not, however, indirect in the usual sense of the word; it does not seek responses to items apparently unrelated to the attitudes investigated. We do, however, seek to measure prejudice in a manner less direct than is true of the usual prejudice scale. In our instrument we seek to measure anti-Negro prejudice. Persons are called upon to respond on social distance scales to whites and Negroes who occupy a variety of occupational positions. The measure of prejudice is derived through the summation of the differences in distance responses to Negroes as opposed to whites in the same occupations. Thus, for lack of a better label, <s> BIB001 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Likert scales <s> Acknowledgements 1. Why do we need statistics? 2. Measures of central tendency 3. Measures of dispersion 4. The normal distribution 5. Probability 6. What are statistical tests all about? 7. Hypotheses 8. Significance 9. Simple statistical tests 10. What's in a number 11. Two parametric tests 12. Tests of goodness of fit 13. The design of experiments 14. Sampling 15. Correlation 16. In the last analysis ... 17. Operation schedules Answers to exercises Appendix Index. <s> BIB002 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Likert scales <s> The Perseus Digital Library, (PDL) is one of the primary digital resources for the humanities. Under continuous development since 1987, the project has included an ongoing evaluation component that aims to understand the effects of access to digitized source materials in the humanities. A summary of the PDL genesis and current status is given and the multifaceted and longitudinal evaluation effort is described. A brief synthesis of results is provided and reflections on the evaluation along with recommendations for DL evaluation are given. Introduction Digital libraries marry the missions, techniques, and cultures of physical libraries with the capabilities and cultures of computing and telecommunications. Evaluating digital libraries is a bit like judging how successful is a marriage. A lot depends on how successful the partners are as individuals as well as the emergent conditions made possible by the union. All three entities —the two individuals and the gestalt union--are of course influenced by their context as well. The difficulties arise from the complexity of mutually self-adapting systems interacting in a rich environment. Metrics for success for component parts of a complex system may be distinct from the metrics for success of the marriage (e.g., success for an individual partner is typically necessary but not sufficient to insure success for the marriage). Digital libraries (DLs) are extensions and augmentations of physical libraries (Marchionini & Fox, 1999). As extensions, we might evaluate the individual partners using existing techniques and metrics. Assessing the impacts of libraries on patron's lives and the larger social milieu are the ultimate goals of evaluation but the practical difficulties of assessing such complex and varied impacts causes us to measure the effectiveness and efficiencies of library operations and services as surrogates for these impacts. Metrics such as circulation, collection size and growth rate, patron visits, reference questions answered, patron satisfaction, and financial stability may be used to assess physical library performance in this regard. Clearly, these are metrics that may be points of departure for evaluating digital libraries, but they are not sufficient to characterize the new, rapidly emerging entity. Evaluation criteria for digital technologies can also be useful. For example, metrics such as response time, storage capacity, transfer rate, user satisfaction, and cost per operation may be useful in assessing technological components but may not be sufficient to characterize DL performance, let alone impact. As extensions of physical libraries and digital technologies, these metrics are good starting points but we must look further to consider the effects of DLs as augmentations that provide new services, products, and capabilities. <s> BIB003 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Likert scales <s> Introduction About the Book Social Research and Data Analysis Demystifying Basic Concepts Data Analysis in Context Working with Two Data Sets Descriptive Analysis - Univariate Looking for Characteristics Descriptive Analysis - Bivariate Looking for Patterns Explanatory Analysis Looking for Influences Inferential Analysis From Sample to Population Data Reduction Preparing to Answer Research Questions Real Data Analysis Answering Research Questions <s> BIB004 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Likert scales <s> Dipping my toe into the water of educational research, I have recently used Likert-type rating scales to measure student views on various educational interventions. Likert scales are commonly used to measure attitude, providing a range of responses to a given question or statement . Typically, there are 5 categories of response, from (for example) 1 1⁄4 strongly disagree to 5 1⁄4 strongly agree, although there are arguments in favour of scales with 7 or with an even number of response categories. <s> BIB005 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Likert scales <s> This paper reviews how usability has been defined in the context of the digital library, what methods have been applied and their applicability, and proposes an evaluation model and a suite of instruments for evaluating usability for academic digital libraries. The model examines effectiveness, efficiency, satisfaction, and learnability. It is found that there exists an interlocking relationship among effectiveness, efficiency, and satisfaction. It also examines how learnability interacts with these three attributes. <s> BIB006 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Likert scales <s> Digital library research has developed over the past decade, but little has been done on the identification of evaluation criteria, especially from users' perspectives. This article identifies users' criteria and applies them to the evaluation of existing digital libraries. Forty-eight subjects were instructed to develop and justify a set of essential criteria for the evaluation of digital libraries. At the same time, they were requested to evaluate existing digital libraries by applying the criteria that they were developing. A compilation of criteria developed by participants show that usability and collection quality were the most important criteria for evaluating digital libraries. Service quality, system performance efficiency, and user opinion solicitation were also deemed essential criteria. The author further compares digital library evaluation criteria identified by users and researchers and applied in previous research. The article reveals problems in current digital library design and development, discusses suggestions for improving digital library design, and presents areas for further research. <s> BIB007 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Likert scales <s> Advances in the publishing world have emerged new models of digital library development. Open access publishing modes are expanding their presence and realize the digital library idea in various means. While user-centered evaluation of digital libraries has drawn considerable attention during the last years, these systems are currently viewed from the publishing, economic and scientometric perspectives. The present study explores the concepts of usefulness and usability in the evaluation of an e-print archive. The results demonstrate that several attributes of usefulness, such as the level and the relevance of information, and usability, such as easiness of use and learnability, as well as functionalities commonly met in these systems, affect user interaction and satisfaction. <s> BIB008 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Likert scales <s> ABSTRACT This paper proposes utilizing a combination of the Quality Function Deployment (QFD)–Kano model as a useful tool to evaluate service quality. The digital library of the University of Guadalajara (Mexico) is presented as a case study. Data to feed the QFD–Kano model was gathered by an online questionnaire that was made available to users on the library's website. The study makes clear that by listening to the voice of the customer (VOC), relevant information may be obtained about issues that should be improved in order to increase customer satisfaction. <s> BIB009 </s> A review of quality evaluation of digital libraries based on users ' perceptions <s> Likert scales <s> Social tagging, as a recent approach for creating metadata, has caught the attention of library and information science researchers. Many researchers recommend incorporating social tagging into the library environment and combining folksonomies with formal classification. However, some researchers are concerned with the quality issues of social annotation because of its uncontrolled nature. In this study, we compare social tags created by users from the LibraryThing website with the subject terms assigned by experts according to the Library of Congress Subject Headings (LCSH). The purpose of this study is to examine the difference and connections between social tags and expert-assigned subject terms and further explore the feasibility and obstacles of implementing social tagging in library systems. The results of our study show that it is possible to use social tags to improve the accessibility of library collections. However, the existence of non-subject-related tags may impede the application of social tagging in traditional library cataloguing systems. <s> BIB010
A Likert scale BIB001 is a psychometric scale commonly used in questionnaires, and is the most widely used scale in survey research. When responding to a Likert questionnaire item, respondents are asked to evaluate their level of agreement or disagreement according to any given subjective or objective criteria. To do so, Likert scales provide a range of responses to a given question or statement. In the DL evaluation literature, Likert scales usually include five levels of response BIB006 BIB008  BIB009 . However, some authors advocate the use of six  or even seven levels  , for additional granularity. For instance, consider the evaluation of a DL regarding the usability criterion user opinion solicitation proposed by Xie BIB007 , which is composed of three subcriteria. Table 3 summarizes the user's assessment on a scale of five levels: 1 = very low, 2 = low, 3 = medium, 4 = high, and 5 = very high. BIB003 Learnability $ System interface Tsakonas and Papatheodorou BIB008 Ease of use $ Navigation Ease of use $ Learnability Navigation $ Aesthetics Terminology $ Learnability Usefulness Tsakonas and Papatheodorou BIB008 Reliability $ Format Reliability $ Level of information Level of information $ Coverage In order to conclude an evaluation, the user's perceptions are computed numerically (see Figure 3a) . To do so, each level on the scale is assigned to a numeric value, usually starting at 1 and incremented by 1 for each level. Many of the authors who use Likert scales for DL evaluation treat individual responses as interval data  BIB006   BIB009 . Thus they use the mean as the measure of central tendency, BIB010 and the standard deviation to measure how much each data value deviates from the mean. For instance, Table 4 summarizes the mean and standard deviation of the data presented in Table 3 . Nevertheless, as Blaikie BIB004 points out, Likert scales fall within the ordinal level of measurement. That is, the response levels have a rank order, but the intervals between values cannot be presumed equal; one cannot assume that respondents perceive all pairs of adjacent levels as equidistant. For instance, the intensity of feeling between 'very low' and 'low' may not be equivalent to the intensity of feeling between other consecutive categories on the Likert scale. The legitimacy of assuming an interval scale for Likert-type categories is an important issue, because the appropriate descriptive and inferential statistics differ for ordinal and interval variables. If the wrong statistical technique is used, researchers increase the chance of coming to the wrong conclusion about the significance of their research BIB005 . Unfortunately  BIB006   BIB009 , no statement is made about the assumption of interval status for Likert data, and no argument is made to support it. According to standard statistical texts BIB004 BIB002 , for ordinal data (1) the median and the mode are the typical measures of central tendency, and (2) the range and the interquartile range measure the data dispersion (see Table 4 ).