Testing L2 Talk: A Review of Empirical Studies on Second-Language Oral Proficiency Testing <s> Paired Test Interaction and Assessment <s> Preface 1. Introduction The aims of the book The climate for language testing Research and development: needs and problems Research and development: an agenda Overview of the book Notes 2. Measurement Introduction Definition of terms: measurement, test, evaluation Essential measurement qualities Properties of measurement scales Characteristics that limit measurement Steps in measurement Summary Notes Further reading Discussion questions 3. Uses of Language Tests Introduction Uses of language tests in educational programs Research uses of language tests Features for classifying different types of language test Summary Further reading Discussion questions 4. Communicative Language Ability Introduction Language proficiency and communicative competence A theoretical framework of communicative language ability Summary Notes Further reading Discussion questions 5. Test Methods Introduction A framework of test method facets Applications of this framework to language testing Summary Notes Further reading Discussion questions 6. Reliability Introduction Factors that affect language test scores Classical true score measurement theory Generalizability theory Standard error of measurement: Interpreting individual test scores within classical true score and generalizability theory Item response theory Reliability of criterion-referenced test scores Factors that affect reliability estimates Systematic measurement error Summary Notes Further reading Discussion questions 7. Validation Introduction Reliability and validity revisited Validity as a unitary concept The evidential basis of validity Test bias The consequential or ethical basis of validity Post mortem: face validity Summary Notes Further reading Discussion questions 8. Some Persistent Problems and Future Directions Introduction Authentic language tests Some future directions A general model for explaining performance on language tests Apologia et prolegomenon Summary Notes Further reading Discussion questions Bibliography Author index Subject index <s> BIB001 </s> Testing L2 Talk: A Review of Empirical Studies on Second-Language Oral Proficiency Testing <s> Paired Test Interaction and Assessment <s> Preface 1 Testing, Assessing, and Teaching What Is a Test? Assessment and Teaching Informal and Formal Assessment Formative and Summative Assessment Norm-Referenced and Criterion-Referenced Test Approaches to Language Testing: A Brief History Discrete-point and Integrative Testing Communicative Language Testing Performance-Based Assessment Current Issues in Classroom Testing New Views on Intelligence Traditional and "Alternative" Assessment Computer-Based Testing Exercises For Your Further Reading 2 Principles of Language Assessment Practicality Reliability Student-Related Reliability Rater Reliability Test Administration Reliability Test Reliability Validity Content-Related Evidence Criterion-Related Evidence Construct-Related Evidence Consequential Validity Face Validity Authenticity Washback Applying Principles to the Evaluation of Classroom Tests 1. Are the test procedures practical? 2. Is the test reliable? 3. Does the procedure demonstrate content validity? 4. Is the procedure face valid and "biased for best"? 5. Are the test tasks as authentic as possible? 6. Does the test offer beneficial washback to the learner? Exercises For Your Further Reading 3 Designing Classroom Language Tests Test Types Language Aptitude Tests Proficiency Tests Placement Test Diagnostic Tests Achievement Tests Some Practical Steps to Test Construction Assessing Clear, Unambiguous Objectives Drawing Up Test Specifications Devising Test Tasks Designing Multiple-Choice Test Items 1. Design each item to measure a specific objective 2. State Both stem and options as simply and directly as possible 3. Make certain that the intended answer is clearly the only correct one 4. Use item indices to accept, discard, or revise items Scoring, Grading, and Giving Feedback Scoring Grading Giving Feedback Exercises For Your Further Reading 4 Standardized Testing What Is Standardization? Advantages and Disadvantages of Standardized Tests Developing a Standardized Test 1. Determine the purpose and objectives of the test 2. Design test specifications 3. Design, select, and arrange test tasks/items 4. Make appropriate evaluations of different kinds of items 5. Specify scoring procedures and reporting formats 6. Perform ongoing construct validation studies Standardized Language Proficiency Testing Four Standardized Language Proficiency Tests Test of English as a Foreign Language (TOEFL(R)) Michigan English Language Assessment Battery (MELAB) International English Language Testing System (IELTS) Test of English for International Communication (TOEIC(R)) Exercises For Your Further Reading Appendix to Chapter 4: Commercial Proficiency Tests: Sample Items and Tasks Test of English as a Foreign Language (TOEFL(R)) Michigan English Language Assessment Battery (MELAB) International English Language Testing System (IELTS) Test of English for International Communication (TOEIC(R)) 5 Standards-Based Assessment ELD Standards ELD Assessment CASAS and SCANS Teacher Standards The Consequences of Standards-Based and Standardized Testing Test Bias Test-Driven Learning and Teaching Ethical Issues: Critical Language Testing Exercises For Your Further Reading 6 Assessing Listening Observing the Performance of the Four Skills The Importance of Listening Basic Types of Listening Micro- and Macroskills of Listening Designing Assessment Tasks: Intensive Listening Recognizing Phonological and Morphological Elements Paraphrase Recognition Designing Assessment Tasks: Responsive Listening Designing Assessment Tasks: Selective Listening Listening Cloze Information Transfer Sentence Repetition Designing Assessment Tasks: Extensive Listening Dictation Communicative Stimulus-Response Tasks Authentic Listening Tasks Exercises For Your Further Reading 7 Assessing Speaking Basic Types of Speaking Micro- and Macroskills of Speaking Designing Assessment Tasks: Imitative Speaking PhonePass Test Designing Assessment Tasks: Intensive Speaking Directed Response Tasks Read-Aloud Tasks Sentence/Dialogue Completion Tasks and Oral Questionnaires Picture-Cued Tasks Translation (of Limited Stretches of Discourse) Designing Assessment Tasks: Responsive Speaking Question and Answer Giving Instructions and Directions Paraphrasing Test of Spoken English (TSE) Designing Assessment Tasks: Interactive Speaking Interview Role Play Discussions and Conversations Games Oral Proficiency Interview (OPI) Designing Assessment: Extensive Speaking Oral Presentations Picture-Cued Story-Telling Retelling a Story, News Event Translation (of Extended Prose) Exercises For Your Further Reading 8 Assessing Reading Types (Genres) of Reading Microskills, Macroskills, and Strategies for Reading Types of Reading Designing Assessment Tasks: Perceptive Reading Reading Aloud Written Response Multiple-Choice Picture-Cued Items Designing Assessment Tasks: Selective Reading Multiple-Choice (for Form-Focused Criteria) Matching Tasks Editing Tasks Picture-Cued Tasks Gap-Filling Tasks Designing Assessment Tasks: Interactive Reading Cloze Tasks Impromptu Reading Plus Comprehension Questions Short-Answer Tasks Editing (Longer Texts) Scanning Ordering Tasks Information Transfer: Reading Charts, Maps, Graphs, Diagrams Designing Assessment Tasks: Extensive Reading Skimming Tasks Summarizing and Responding Note-Taking and Outlining Exercises For Your Further Reading 9 Assessing Writing Genres of Written Language Types of Writing performance Micro- and Macroskills of Writing Designing Assessment Tasks: Imitative Writing Tasks in [Hand] Writing Letters, Words, and Punctuation Spelling Tasks and Detecting Phoneme-Grapheme Correspondences Designing Assessment Tasks: Intensive (Controlled) Writing Dictation and Dicto-Comp Grammatical Transformation Tasks Picture-Cued Tasks Vocabulary Assessment Tasks Ordering Tasks Short-Answer and Sentence Completion Tasks Issues in Assessing Responsive and Extensive Writing Designing Assessment Tasks: Responsive and Extensive Writing Paraphrasing Guided Question and Answer Paragraph Construction Tasks Strategic Options Test of Written English (TWE(R)) Scoring Methods for Responsive and Extensive Writing Holistic Scoring Primary Trait Scoring Analytic Scoring Beyond Scoring Responding to Extensive Writing Assessing Initial Stages of the Process of Composing Assessing Later Stages of the Process of Composing Exercises For Your Further Reading 10 Beyond Tests: Alternatives in Assessment The Dilemma of Maximizing Both Practicality and Washback Performance-Based Assessment Portfolios Journals Conferences and Interviews Observations Self- and Peer-Assessments Types of Self- and Peer-Assessment Guidelines for Self- and Peer- Assessment A Taxonomy of Self- and Peer-Assessment Tasks Exercises For Your Further Reading 11 Grading and Student Evaluation Philosophy of Grading: What Should Grades Reflect? Guidelines for Selecting Grading Criteria Calculating Grades: Absolute and Relative Grading Teachers' Perceptions of Appropriate Grade Distributions Institutional Expectations and Constraints Cross-Cultural Factors and the Question of Difficulty What Do Letter Grades "Mean"? Alternatives to Letter Grading Some Principles and Guidelines for Grading and Evaluation Exercises For Your Further Reading Bibliography Name Index Subject Index <s> BIB002 </s> Testing L2 Talk: A Review of Empirical Studies on Second-Language Oral Proficiency Testing <s> Paired Test Interaction and Assessment <s> The increasing popularity of paired format in oral testing has engendered legitimate scrutiny of its reliability and validity as compared with the more traditional interviewer–interviewee arrangement. Although characteristics such as the gender, cultural/L1 background, and language proficiency of one's interlocutor likely affects the discourse produced with a partner, the question remains whether this interlocutor effect influences scores on the test. In this article, the construct interlocutor effect is further examined. Initially, transcript data from two large-scale speaking tests were analyzed by working backward from testtaker scores to locate discourse features to support those scores; identity formulations, such as “proficient” and “competent,” as constructed in and through the discourse test takers produce, emerged as a salient feature of the talk. Specifically, we posit that a test taker brings a language proficiency identity to a test task, and this identity is constructed, mediated, and display... <s> BIB003 </s> Testing L2 Talk: A Review of Empirical Studies on Second-Language Oral Proficiency Testing <s> Paired Test Interaction and Assessment <s> Contents: Modelling language knowledge and performance - Investigating variability in oral proficiency testing: the oral interview, the paired and group oral, test tasks - Discourse patterns in the paired mode: style of dyadic interaction - Impact of partner's proficiency level - Impact of candidate's previous learning experience - Test takers' perceptions of the paired oral - Future research in measuring oral performance through paired-task performance. <s> BIB004 </s> Testing L2 Talk: A Review of Empirical Studies on Second-Language Oral Proficiency Testing <s> Paired Test Interaction and Assessment <s> The present study takes as an empirical point of departure the nature of interaction in second language speaking tests. We examine the relationship between ratings of students' performance in an oral proficiency test and the social practice of conducting 'test talk'. Using conversation analysis, our focal point is how students in peer- driven test interactions manage trouble related to the task-at-hand. Given that students were assessed not only on their linguistic skills, but also on their interactional ability and treatment of topics assigned, our emphasis on task management stems from a hypothesis that orientation to the test task is intimately connected to overall test outcome. We demonstrate that different types of task-related trouble (TRT) reveal diverse understandings of the test task and that 'doing-being a successful task manager' is connected to a moderate orientation to the task and test format. Students displaying such task management strategies were also assessed as highly proficient, whereas other task management strategies identified in our study correlated with low scores and grades. However, the relationship between subskill ratings and task management was not always clear-cut. We argue that the diverging understandings of the test task that learners display become part of how they are assessed and that certain task management strategies are rated less favorably than others. Our study holds promise for the fine-tuning of oral proficiency subskill ratings and raises questions as to the framing of test tasks, since this appears to have implications for student performance and evaluation. <s> BIB005 </s> Testing L2 Talk: A Review of Empirical Studies on Second-Language Oral Proficiency Testing <s> Paired Test Interaction and Assessment <s> This study looks at the real and perceived effects of inter-candidate variation in linguistic ability during pair-tested oral examinations in a southern Italian context. Pair testing has not been taken up in schools and universities in southern Italy despite its wide use in international testing systems. Colleagues and prospective candidates often report to teachers their belief that test results can be adversely or positively affected by candidate pairing. To investigate the existence and validity of these beliefs, a series of tests was devised consisting of paired oral tests and pre- and post-test questionnaires. It was found that while the great majority of candidates believed that their test scores could be, and in many cases had been, affected by their candidate pairing, no evidence came to light to support these beliefs. In fact the linguistic ability of a candidate�s partner appeared to have little or no effect on his or her performance. <s> BIB006 </s> Testing L2 Talk: A Review of Empirical Studies on Second-Language Oral Proficiency Testing <s> Paired Test Interaction and Assessment <s> Poststructuralist perspectives on the discursive construction of identity and critical approaches to linguistics, discourse analysis, and language testing have contributed to a growing awareness of the complex social nature of performance in speaking tests. In this article, I explore how the concept of co-construction is useful to understand how candidate and interviewer identities are created in the discourse of speaking tests but argue for an expanded notion of co-construction that draws upon the concept of intertextuality to recognise the myriad of other ‘voices’, representing those who craft the tests, are interwoven into the discourses produced. The study adopts a qualitative approach, drawing upon discourse analysis to examine co-construction and intertextuality in two levels of Cambridge speaking tests. The findings reveal how interviewers and candidates co-construct particular identities through their interaction with each other, the interlocutor frame, and the test materials, and this can account... <s> BIB007 </s> Testing L2 Talk: A Review of Empirical Studies on Second-Language Oral Proficiency Testing <s> Paired Test Interaction and Assessment <s> Paired speaking tasks are now commonly used in both pedagogic and assessment contexts, as they elicit a wide range of interactional skills. The current study aims to offer an investigation of the interaction co-constructed by learners at different proficiency levels who are engaged in a paired speaking test, and to provide insights into the conceptualization of interactional competence and key distinguishing interactional features across levels. The findings suggest that our understanding of interactional competence both in the classroom and as a construct underlying tests and assessment scales needs to broaden to include not just interactional features such as topic development organization, but also listener support strategies and turn-taking management. A more comprehensive understanding of interactional competence has the potential to complement available descriptions of interactional skills in assessment scales of speaking and aid learners and teachers in communicative classrooms. <s> BIB008 </s> Testing L2 Talk: A Review of Empirical Studies on Second-Language Oral Proficiency Testing <s> Paired Test Interaction and Assessment <s> Despite the popularity of task-based language teaching (TBLT) in foreign language (FL) education at elementary school, it remains unclear how young learners’ FL abilities can best be evaluated with tasks. The present study seeks to understand developmental differences in interactions among elementary-school students during task-based language assessment (TBLA) and aims to address the possibilities and limitations of introducing paired-TBLA among young FL learners. The participants were 32 fourth-grade students (ages 9–10) and 32 sixth-grade students (ages 11–12) and their teachers from a FL program where TBLT was mandated. The students were engaged in two sets of assessment tasks with their peers. It was found that the interaction patterns were different between the two grade levels. Compared with the sixth-grade dyads, the fourth-grade dyads showed less mutual topic development, used formulaic turn-taking more frequently, and had a harder time taking their partners’ perspectives during the tasks. Consequ... <s> BIB009
The interest in rating scales and the connections between assessment and interaction is a prevalent theme among our hits. An often cited article is BIB008 , involving the First Certificate in English. Using a discourse analytic approach informed by CA, along with quantitative methods, she identified three patterns of how test-takers orient to each other, which all signalled varying degrees of cooperation, mutuality, and equality. In terms of scores, the 'collaborative pattern' was deemed to be the most successful. Galaczi suggests that candidates with limited L2 skills are not particularly engaged in interactions with their interlocutors, a track explored further in Galaczi (2014) , where she recommends a broader view on interactional competence. Along similar lines, BIB009 found that fourth-graders used more formulaic/fixed turn allocation strategies than sixth-graders, concluding that turn allocation is an advanced interactional competence. Other studies that deal with the relationship between interaction and assessment in the paired format include BIB005 , who analyze sequences from the national test of English as a foreign language for ninth-graders in Sweden. The study adopts CA and compares the sequential analysis with assessment data, focusing on how students manage interactional trouble connected to the test tasks ('task-related trouble'). The findings indicate that certain task management strategies appeared to be rated less favorably than others, even though some of them (such as task abandonment and negotiation of understanding of the test task) were perfectly productive for the students themselves in terms of test-wiseness, a concept also discussed in work on language testing in general (see, e.g., BIB001 BIB002  ). The Interlocutor Yet other studies pay attention to the effect of the interlocutor on test performance, a topic generally explored through quantitative studies (e.g., BIB006 BIB004 ). In BIB007 study on the Cambridge Speaking Test, the author argues for a connection between pairing and performance if the candidates are at different proficiency levels and that gender and familiarity with the other candidate potentially plays a role for linguistic performance (but see Lazaraton's 2006 critique of the reliability of the findings of Norton's analyses). A radically different approach to analyzing the effect of pairing is BIB003 CA study of proficiency identities. The authors show that the candidates bring proficiency identities to the test and that these might be ratified, maintained, or modified depending on the interlocutor. This particular study demonstrates how proficiency should be considered a skill that test-takers dynamically construct in peer-peer interaction, rather than something only being displayed in interaction.