E P Ly N N

B A L Dw I N

Assessing multiple choice question (MCQ) tests -a mathematical perspective

Keywords: computer-based assessment, examinations, multiple choice question tests, penalizing guessing, test scores

A B S T R AC T The reasoning behind popular methods for analysing the raw data generated by multiple choice question (MCQ) tests is not always appreciated, occasionally with disastrous results.This article discusses and analyses three options for processing the raw data produced by MCQ tests. The article shows that one extreme option is not to penalize a student for wrong answers or for missing out questions, and the other extreme option is actually to penalize both aspects.The intermediate option of focusing on the number of questions actually attempted while penalizing wrong answers can be regarded as the fairest. In this case blind guessing will on average not help the student, although partial knowledge will lessen the negative impact on the final overall score. There are still many interesting challenges in designing techniques for MCQ tests.

Introduction

Assessment in higher education is always on the agenda, given its centrality to the awarding of degrees of one level or another. Every mark that a student gets for an item of coursework or an examination makes its contribution to the mark or grade for the module as a whole which, in turn, affects degree classification in some way. For example, classified undergraduate degrees in the UK are known as Honours degrees, and have four 'categories': first, upper second (2.1), lower second (2.2) and third.

Like us all, students wish to do as well as they can, and to achieve the best qualification that they can. In today's climate, in the UK at least, most employers state in their job advertisements that potential candidates will normally need an upper second (2.1) and so there is pressure for students to achieve this. With higher education being viewed more of a service to be bought by students and with us in universities being viewed as those who deliver that service, it is not surprising that every mark counts. This can be made particularly challenging by the way that these degrees are calculated.

In order to recommend a candidate for a degree award, the overall weighted mark average has to be calculated. Precisely how the final mark is weighted varies from institution to institution, but it is not uncommon to base it on the average mark from level 2 and the average mark from level 3 and to weight level 2 at one third and level 3 at two thirds based on the 'best' marks achieved in 80/100 credits out of 120 credits at each level.The decision-making of the board of examiners is usually made easy if the overall weighted average falls well within a range. For example, if a mark of 65 falls between 60 and 69, the examination board would normally recommend an upper second degree. Naturally enough, if a candidate has had some personal circumstances that have affected their academic performance, the decision-making is not so straightforward. However, at some point during the meeting of the board of examiners, we are faced with a candidate who has an overall weighted mark average of, say, 69.87. Depending on the discipline, institution or practices of the board of examiners in question, this will or will not be considered as a 'borderline' case. Some boards of examiners may rightly claim that as 69.87 is not a mark between 70 and 100 (the mark required for the award of a first class degree), the appropriate award should be an upper second (2.1). For others, a weighted mark average of 69.87 is considered so near as to be 'pretty much' a mark of 70. Such decision-making is known as 'academic judgement', which is exercised differently in different institutions, or indeed disciplines, for various reasons. However academic judgement is exercised, such seemingly small differences in the mark average can make the difference between a higher or a lower award which, in today's climate, can in one way or another affect our students.

To ensure that we make the most appropriate award for each student, we ask students to demonstrate their achievements by way of various assessments. Some would argue that this assesses not what they know, but what they can show us, which may be two very different things (Race, 2005: 67). Many would probably agree with [START_REF] Race | Making Learning Happen[END_REF] that 'assessment is the principal driving force for learning for so many learners' in higher education today and that this goes against what we claim institutions of higher education are about, that is, promoting and fostering learning itself. Many would also probably agree with Race (2005: 66) that we, like our students, are driven by results, given that our funding depends on showing that this learning has happened and that we have played our part in this. Whether or not this is the case, assessment is certainly at the top of the agenda in learning and teaching in higher education institutions, whatever the complex reasons, and we spend enormous amounts of time on making assessment 'valid, reliable, transparent and authentic' (Race, 2005: 74). Indeed, some might argue that we spend far more time on assessment-related matters than we do on learning and teaching itself.

For those of us with large classes of 300 plus (a class of 100 is now regarded as 'small'), the choice of the means of assessment appropriate to a particular course needs to be carefully considered if we wish to devote more of our valuable time to interacting with our students. Ideally the assessment approaches should not be selected purely on the basis of ease or speed of marking. There are two main types of assessment; normativereferenced and criterion-referenced. In addition, there is also self-referenced (ipsative) assessment, which is gaining in popularity in the more studentcentred (rather than teacher-centred) approaches that many claim to use in higher education [START_REF] Pihlajamaki | Self-, Peer-and Teacherassessment of Student Essays[END_REF][START_REF] Pitts | Testing, Testing …'; How Do Students Use Written Feedback?[END_REF]. Whether formative or summative in nature, assessment of, and feedback to, learners shapes how learners think and behave; this is further evidence of our need to take great care. Feedback plays a central role in self-regulation skills in learning, required when comparing performance against the goals that have been set (Nicol and Macfarlane-Dick, 2006). For a useful overview and in-depth discussion of assessment more generally, the reader is directed to [START_REF] Freeman | Planning and Implementing Assessment[END_REF], Brown and Glasner (1999) and [START_REF] Race | Making Learning Happen[END_REF].

One of the 'tools' in our 'toolbox' of assessment methods are objective tests. These fall into three main types, namely true/false, matching and multiplechoice questions (MCQs), in the context of which, say Freeman and Lewis (1998: 145), we must be clear what we mean by 'objective'. It is, they say, that marking can be done mechanically (more often than not by a computer, these days) as no judgement needs to be made. To call such a test 'objective' does not, however, mean that MCQ tests and the like assess candidates 'objectively', that is, without any prejudice or bias. Indeed, such a thing is not possible. No assessment method can be objective; all assessment methods are subjective to a greater or lesser extent. As markers, we can be influenced by gender, ethnicity and the like, and knowing, or not knowing, the identity of the students whose work we are marking (if students do or do not have their name on their work) affects the marks we award (Fleming, 1999, in Brown andGlasner, 1999). As Barrow (2006: 206) rightly says,'all assessment places students in a web of power'; given this, we need to think long and hard about how we use that power.

There is a wide range of views on the effectiveness of MCQ tests. According to Freeman and Lewis (1998: 147), MCQ tests can test only facts and are of less substance than other forms of assessment; these authors go on to say that such tests have the reputation, undeserved, of being 'easier' than other methods and also of being able to test only 'lower level' skills and abilities. Whatever their limitations, Brown and Glasner (1999: 11) claim that MCQ tests are 'much more sophisticated than many of us once believed', citing their use by the Open University (OU) in the UK and directing readers to the OU for examples of best practice. Race (2005: 86), too, supports this view, adding that if we design MCQ tests well, we can be as confident as we can be with any assessment, that such tests will tell us the extent to which students have made sense of what they have learned and that we can ensure that our marking will not be affected by illegible handwriting or that students will not be penalized for writing slowly. Indeed, [START_REF] Race | Making Learning Happen[END_REF] goes on to say that there are many more benefits from MCQ tests, among these that, by providing on-screen feedback immediately, students may avoid taking with them the errors from the earlier questions and that given the speed at which they get feedback, they can still remember why they thought and did what they thought and did, and can thus reflect on this. MCQ tests are, it seems, popular with learners although they may be perceived as being less fair [START_REF] Struyven | Students' Perceptions about Evaluation and Assessment in Higher Education: A Review[END_REF].

With the increasing use of computers to support our students' learning these days, many software packages now do much of the work for us in terms of helping us to design MCQ tests which we can use in our modules. We may ask students to carry these out in class, via, say, WebCT or Blackboard (most institutions in the UK now use one or other of these packages for, as a minimum, the 'basics' of the module content), or perhaps for self-assessment outside of the classroom (Blackboard, WebCT). Again, with the relative sophistication of the software these days, it is not too difficult -with, perhaps, a little help for those who are not so technically experienced -to set up such tests.

Naturally, the course organizer needs to build up a sufficiently large bank of questions so that even if students see these questions before the test, the large number of questions effectively ensures novelty each time students take the test. 41 20 39 3 8 14 35 18 41 20 39 3 8 14 18 35 41 20 39 3 8 14 18 20 41 35 39 3 8 14 18 20 35 41 39 3 8 14 18 20 35 39 MCQ tests, if appropriately designed, test not only the ability of students to memorize and recall knowledge but also test the ability to reason analytically with that knowledge. The example in Figure 1 shows how this is possible. Here the student is asked to determine the type of sorting algorithm that the diagram represents. This not only requires memorization of facts but the ability to appreciate how the relevant sorting algorithms work and to compare that knowledge with the information presented by the diagram. Questions on a segment of program code provide further examples of how MCQ tests can test not just powers of information retention and retrieval but also those of analysing and reasoning with knowledge in situations where context is important.

Admittedly, MCQ tests, unlike written examination papers which call for longer responses, do not show the steps taken by students to reach their answers. In addition, MCQ tests may not test the ability of students to create or synthesize new knowledge from previous items that have been learned and understood. It is for these reasons and others that MCQ tests are but one method of assessment and that we, as course organizers/assessors, need to exercise care to ensure that we are testing that which we wish to test. For example, a course in computer algorithms could include an MCQ test to examine students' knowledge of the subject (for example, linked list manipulation) and to exercise the ability to reason with that knowledge. However, such a course should also include exercises that enable a student to apply that knowledge to solve particular tasks (for example, application of linked lists to symbolic algebraic manipulation such as addition, multiplication or integration). Nonetheless, the nature of MCQ tests make them ideal as an assessment tool for first year university courses, where the nature of the material and large class sizes can really bring out the benefits.

Whilst the benefits and drawbacks of MCQ tests were discussed earlier in this article, the basic issues of how we can convert the raw data generated by an MCQ test into meaningful results are rarely raised. It is important to create a model that gets as close as possible to a student's reaction to, and processing of, the questions in an MCQ test, while not unduly penalizing the student. In addition, to what extent does guessing benefit the student at the expense of a fair assessment by an MCQ test of that student's abilities? The use of MCQ tests as an assessment tool is common in the higher education sector and is growing in popularity. It is argued here that course organizers/assessors who compile these tests in the educational environment may on occasion make serious errors when interpreting the raw data generated by MCQ tests if not aware of the issues raised within this article. These factors suggest that there is a need to consider the more formal mathematical aspects of this important topic if we are to ensure that the marks that we award are appropriate.

The basic equation

Let R be number of questions for which a student gives a right answer and K be the number of questions for which a student actually knows the right answer. The vital point is that R may be greater than K. This would typically be the case if the student has actually guessed the answers to the additional (RϪK) questions, because of lack of knowledge or application of appropriate reasoning. (Note, as a help to the reader, the symbols used in the equations in this and the following sections are brought together in the appendix at the end of this article.)

Assume that all questions carry equal weight and that for each question there are A possibilities. In addition, let Q be the total number of questions attempted out of an overall total of T questions. Now ((QϪK)/A) will be the number of questions that the student has -on average -got right by chance, since there are A possibilities for each question. Hence, the number (R) of questions for which a student gives a right answer is related to the number (K) of questions for which the student knows the right answer, by the following equation:

(1) or, rearranging,

(2)

The second equation ( 2) shows that the (QϪR) wrong results will penalize the student. Note, additionally, that in this equation, negative values of K can be forced to a floor value of zero, which is then the minimum mark a student can achieve.

Three options for multiple choice tests

Three options with differing approaches to penalizing guessing can be identified.These are the zero, intermediate and maximum penalty options.

Option 1 -zero penalty

Here the input marks, apart from scaling, can be regarded as raw data and are not subject to further processing (that is, they are not subject to any reduction).

In this case students can gain marks by guessing (that is, by luck), and, on average, 1/A of the guesses will be correct, even if the students have no knowledge of the actual topic being examined. Wrong results are not penalized. Thus in equation (1), if Kϭ0 (i.e. the student has no knowledge of the subject in which they are being assessed), the number R of questions for which the student gives the right answer, will still, on average, be (Q/A). This simple approach does not attempt to distinguish between answers resulting from genuine knowledge, and those resulting from lucky guesses. This scenario is different from a conventional written paper, where to gain any marks at all, some correct response -written or pictorial -is normally required of the student.

A typical exam rubric for this version of an MCQ test could read as follows:

Each question is provided with 4 answers, only one of which is correct. Answers are identified by the letters A, B, C and D. Work out which letter corresponds to the best answer and put its letter legibly in the box provided.

If K R Q R A ϭ Ϫ Ϫ Ϫ 1 R K Q K A + ϭ Ϫ
the letter is not legible, then no mark will be given. The zero penalty MCQ test option is, of course, suitable, and very popular, when MCTs are used in a self-test (didactic) mode, in which the student is learning, as opposed to being assessed.

Option 2 -penalty based on questions attempted

In this case, the number W of questions the student answers wrongly is given by:

(3) Equation ( 2) can be used to find K, the number of questions correctly reflecting the student's knowledge, in terms of W. This gives equation ( 4):

(4) This (intermediate) option is widely used when assessing MCQ tests, since while it penalizes wrong answers, it takes no account of the number of questions which the student has not attempted. With this intermediate option, a student attempting a given number X of questions, all of which are answered correctly, may get more marks than their colleague who attempts Y questions, where YϾX, but where still only X questions are answered correctly.The justification for this is that the 'Y' student has used more chances. This is emphasized in the table and the graph in Figures 2 and3, respectively. Both consider a scenario of 20 questions, with 4 choices per question -one choice of which is correct. The number (R) of correct questions has been multiplied by 5 to give a percentage mark.

When using the intermediate option 2, a possible rubric for an examination paper could read:

You will only be assessed on questions which you have attempted and to which you have provided a legible answer. It is recommended that you avoid guessing an answer to a question.

Option 3 -maximum penalty

A student can be further penalized by option 3 (maximum penalty) which not only considers the number of questions that were incorrectly answered, as in option 2 above, but also considers the number of questions that were not attempted. Thus questions not attempted are also assumed to be incorrectly answered.

K R W A ϭ Ϫ Ϫ 1 W Q R ϭ Ϫ
In equation ( 2), replace Q by T, the total number of offered questions, and K now becomes K M , the number of questions that it is assumed have been correctly answered as a result of the student's knowledge.

(5) This equation also represents the limiting case of equation ( 2), when all the questions have been attempted.

Another form of the Maximum Penalty Equation can be derived, by assuming M marks per question, and putting S ϭ K M M and S MAX ϭ TM. This gives the following equation:

(6)

Numerical and graphical comparisons

Table 1 considers a scenario of 20 questions, with 4 choices per questionone choice of which is correct. For convenience of presenting the numerical results in this article, the number of correct questions has been scaled by 5 to give a percentage mark. Both columns 1 and 2 can thus be regarded effectively as the same raw data but with different scaling. Note that usually one tries to avoid negative marks, so that K has not been allowed to go negative. The three options are represented as follows.

Option 1. With option 1 (zero penalty), there is no further processing as column 2 of Table 1 shows. Option 2. With the intermediate option (option 2), a student attempting a given number X of questions, all of which are answered correctly, may get more marks than their colleague who attempts Y questions, where YϾX, but where still only X questions are answered correctly.The justification for this is that the 'Y' student has used more chances. This is emphasized in Table 1 and Figure 2, respectively. For example, for the actual score of 8 questions answered correctly (but possibly including guesses) or 40 per cent, the intermediate approach gives marks of 33 per cent, 25 per cent and 20 per cent for Q ϭ 12, 17 and 20, respectively. Option 3. The fourth column also represents option 3 (maximum penalty) as the limiting case of option 2 (Q ϭ T ϭ 20), where all offered questions have been attempted. Figure 2 also shows that the operation area for MCQ tests is bounded by the triangular area formed by the zero and maximum penalty lines and the x-axis.

K T S S A S S A M MAX MAX MAX ϭ Ϫ Ϫ ( ) ( ) K R T R A M ϭ Ϫ Ϫ Ϫ 1

The influence of processing an MCQ test on a student's mark

The graph in Figure 3 is derived from the graph in Figure 2 and shows the differences in the scaled output mark (K) (that is, reading in the vertical direction) between the zero penalty approach and intermediate penalty approaches where the number of questions attempted is 4, 8, 12 and 16, respectively, with the limiting case of Q ϭ T ϭ 20 being equivalent to the maximum penalty approach.The maximum difference is as much as 25 per cent.

= 20 Q = 16 Q = 12 Q = 08 Q = 04

Figure 2 Graph comparing zero, intermediate and maximum penalty approaches

Figure 3 provides a stark illustration of how the mark that a student obtains for an MCQ test is strongly influenced by the way in which the educator processes the raw marks generated by the MCQ test.

Does the student benefit by guessing?

With Option 1, the answer is definitely yes for 'blind' guessing. However, on average the answer is no if the intermediate option 2 is used. This is borne out by equation ( 2) and Table 1. If Q, the total number of questions attempted, increases, but R, the actual score, stays the same, then the mark K actually given to the student will decrease. This is also summarized in Figure 4, which shows the number of questions attempted against the resultant mark K for different raw scores (R) where

Q = 04 Q = 08 Q = 20 Q = 16 Q = 12 Scaled raw data (%) R Scaled difference (%)
R can be regarded as the number of questions apparently correct. Note that R is expressed in terms of the number of questions answered correctly, irrespective of whether this is due to the student's actual knowledge or to guessing. The downward sloping lines for given values of R indicate diminishing returns for the student if wrong answers are given for further questions (QϪR) attempted.

Suppose a student has additional partial knowledge that improves the chances of an answer being correct from 1/A to 1/B (i.e. BϽA). For example, students may be able to narrow down their options from 1 in 4 choices (A ϭ 4) to 1 in 2 choices (B ϭ 2). In this case, in equation ( 4), the subtracted term (W/(AϪ1)) will be less in value if A is replaced by B, where BϽA. With partial knowledge, the negative impact on the final mark will, on average, be lessened. Naturally, this is only a very crude model of the situation, since students will not normally approach each question with the same degree of partial knowledge.

Guessing and the number of question choices

Does the effect of guessing depend on the number (A) of possible choices per question? So far we have assumed for each question, four choices (i.e. A ϭ 4) one of which is correct. From Figure 2 it can be seen that the maximum difference D MAX in marks between the zero and maximum penalty lines is given by vertical distance between the two lines from the crossing of the x-axis by the maximum penalty line. For A ϭ 4, this is 0.25 p.u. (per unit) or 25 per cent. This exercise can be repeated by re-plotting Figure 2 for different values of A (e.g. A ϭ 2 to A ϭ 10).

More generally, we can take equation ( 5), and put K M ϭ 0 and R ϭ D MAX. This gives equation ( 7):

(7) From this we can see that the effect of accounting for guessing, when using the intermediate option 2, becomes less as the number (A) of choices per question increases. Essentially, as the number of choices increases, the probability of picking the correct answer by chance diminishes. The graph and table in Figure 5 summarize this.

What is the optimum number (A) of choices for each question? Commonly, but not exclusively, three or four choices are used.The greater the number (A) of choices, the more time must be allocated to a student to complete a question, but the smaller will be the influence of blind guessing by the student. From the educator's perspective, the effort In the context of option 1, consider the following weighted cost function F, for which the term (␣A) indicates the time spent preparing the question, where ␣ is a weighting factor and the term (␤/A) indicates the effect of guessing, where ␤ is a further weighting factor.

(8)

If we assume that all variables and parameters in equation ( 8) are positive, then equation ( 8) has a minimum for: (9) Thus, if minimizing the effect of guessing is regarded as being 9 or 16 times more important than the initial work in preparing the question, the corresponding number (A) of choices should be 3 or 4. This fits in with normal practice. The emphasis on 'initial work' highlights one advantage for the teacher of building up and maintaining a sizeable bank of multiple choice questions. Of course, as has already been mentioned, another advantage of such a question bank is to ensure the effect of novelty in the questions for the student.

A ϭ ␤ ␣ F A A ϭ ϩ ␣ ␤ A 2

Future work

MCQ tests are very popular with students, especially when the tests are computer-based and they can thus receive feedback promptly.They are also popular with educators who, by saving on the time taken with marking, providing feedback and all the other tasks associated with assessment, can devote more resources to the actual process of teaching. The following issues show that there are still many challenges with setting and devising MCQ tests, however.

Revealing thinking patterns. Basic MCQ tests are said to test only factual knowledge, but by careful attention to the question, thinking processes could be deduced. Marks could then be allocated not only on the basis of whether the question was correctly or incorrectly answered, but also on how the student reached their conclusion. More work is needed here to formalize the process. Reassuring the student -good interfaces. A good computer interface can encourage and put students at their ease. A good layout of a printed version of an MCQ test can also be beneficial to the student. Reassuring the student -adjusting marking scheme. There are many possibilities. A basic one is to consider a compromise between the no-penalty option 1, which encourages students to supply answers, and option 2, which discourages guessing. This could be achieved by making the factor A less than the number of available question choices.

Conclusions

For a written paper, a student must have some knowledge of the subject, although a chance element still exists regarding exactly what that student actually chooses to put as the answer. However, the mark that a student obtains for an MCQ test is strongly influenced by the way in which the educator processes the raw marks generated.

Approaches to processing and analysing the raw data generated by MCQ tests that are used for assessment can be distinguished by the way the approaches deal with guessing by the student. One extreme imposes no penalty for wrong answers; in this case, even if the student has no knowledge of the subject being tested, they will get on average a mark of (100/A) per cent, where A is the number of choices per question.The other extreme penalizes both wrong answers and the lack of answers to questions and may be viewed as the option for which there is the least justification.

If we are to compensate for guessing, then the intermediate option will give more marks to each student than the maximum penalty option. The intermediate option, on average, penalizes blind guessing, although a partial knowledge lessens the negative impact on a student's final mark. With this option, the effect on a student's marks of accounting for guessing also becomes less as the number of choices per question increases, giving, for example, maximum absolute differences in marks of 33 per cent and 25 per cent for three and four question choices, respectively.

All three approaches obviously need to know the number (R) of right answers and the marks per question. In addition, the intermediate and maximum penalty options need to know the number (Q) of questions attempted and the total number (T) of questions offered in the MCQ test, respectively.

Multiple choice tests have gained popularity with students and educators, but there is considerable scope for further research, for example in techniques for reassuring and encouraging the student and techniques for revealing their thinking patterns.

Appendix. Symbols and abbreviations used in this article

41 Figure 1

 411 Figure 1 Typical multiple choice question on sorting algorithms The dark items with white lettering indicate the sorted items; the lighter items with black lettering indicate the unsorted items. The numbers 1 to 7 on the right-hand side indicate the particular stage of the sorting process. Which sorting algorithm is represented by the figure? A partition sort C insertion sort B bubble sort D selection sort

Figure 4 Figure 3

 43 Figure 4 Number (Q) of questions attempted against final score (K) for different numbers (R) of questions for which the correct answer is given (either from knowledge or by chance)

Figure 5

 5 Figure 5 Graph and table relating number (A) of choices to maximum possible absolute difference (D MAX ) in marks caused by accounting for guessing

  THERE IS NO PENALTY FOR A WRONG ANSWER.YOU ARE THEREFORE URGED TO ATTEMPT EVERY QUESTION. If you feel you do not know the answer to a question you should select the answer that you feel is the most likely.

Table 1 Comparing zero, intermediate and maximum penalty approaches

 1

	Option	1. Zero	2. Intermediate 2. Intermediate	3. Max.
		penalty			penalty
	Actual score	K ϭ R	Q ϭ 12	Q ϭ 17	Q ϭ T ϭ 20
	ϭ R	Score (%)	Score (%)	Score (%)	Score (%)
		ϭ 100*(R/T)	ϭ 100*(K/T)	ϭ 100*(K/T)	ϭ 100*(K/T)
	00	0	0	0	0
	01	5	0	0	0
	02	10	0	0	0
	03	15	0	0	0
	04	20	7	0	0
	05	25	13	5	0
	06	30	20	12	7
	07	35	27	18	13
	08	40	33	25	20
	09	45	40	32	27
	10	50	47	38	33
	11	55	53	45	40
	12	60	60	52	47
	13	65	-	58	53
	14	70	-	65	60
	15	75	-	72	67
	16	80	-	78	73
	17	85	-	85	80
	18	90	-	-	87
	19	95	-	-	93
	20	100	-	-	100

Total no. of questions ϭ Q ϭ T ϭ 20. Possible no. of answers per question ϭ A ϭ 4.

  ERIC SCHARF is a lecturer in the Department of Electronic Engineering, Queen Mary, University of London. He has been teaching topics dealing with software and microprocessors. He is an enthusiastic user of Multiple Choice Tests and has found that this enthusiasm is reflected in the positive approach taken by his students. Eric is research-active: he has successfully bid for, and led, a number of European Unionfunded research projects including, more recently, TORRENT (concerned with applying intelligent agent technology to foster consumer choice in residential access networks) and CoRAS (a platform for the risk assessment of security-critical systems). Address: Department of Electronic Engineering, Queen Mary, University of London, Mile End Road, London E1 4NS, UK. [email: e.m.scharf@elec.qmul.ac.uk] LYNNE P. BALDWIN is a senior lecturer in the Department of Information Systems and Computing at Brunel University, UK. She gained her PhD at Brunel University, and an MA in Language and Communication from the University of East Anglia, UK. Her research interests within the field of education and information systems are varied, although there is a strong emphasis on knowledge management, decision-making and related communication issues. Address: Department of Information Systems and Computing, Brunel University, Uxbridge, Middlesex UB8 3PH,UK. [email: Lynne.Baldwin@brunel.ac.uk]

	A	number of possibilities per MCT question
	B	inverse probability of partial knowledge of a question
	D MAX maximum difference in marks
	F	weighted cost function
	K	number of questions for which a student knows the right answer
	K M	number of questions for which a student knows the right answer
		(Maximum Penalty)
	M	marks per question
	Q	total number of MCT questions attempted by student
	R	number of questions for which a student gives a right answer
	S	product of K M and M
	S MAX	product of T and M
	T	total number of questions presented to a student in a MCT
	W	number of questions wrongly answered
	␣,␤	cost function factors
	MCQ Multiple Choice Question
	MCT Multiple Choice Test
	p.u.	per unit
	Biographical notes

Acknowledgements

I would like to thank students and colleagues in the Department of Electronic Engineering at Queen Mary, University of London for providing a suitable and fertile environment in which to apply multiple choice tests. I would also like to thank the reviewers of this article and, last but not least, my co-author, who encouraged me to bring this article to publication.


S T E P H A N Da H L

The student perspective on using plagiarism detection software

A B S T R AC T Recently there has been an increasing interest in plagiarism detection systems, such as the web-based Turnitin system. However, no study has so far tried to look at how students react towards those systems being used. This exploratory study examines the attitudes of students on a postgraduate module after using Turnitin as their standard way of submitting work and getting feedback. Overall, students reacted positively towards the system. However, the study also found evidence of a group of students who were less positive, which seemed to be a result of their insecurity about how to quote correctly.

Introduction

Over the last few years there has been an increased concern at the extent of academic plagiarism in academic life [START_REF] Szabo | Cybercheats: Is Information and Communication Technology Fuelling Academic Dishonesty?[END_REF]. The academic community has responded with a number of different strategies to counter the perceived rise in plagiarism. Some of these strategies were student-centred by, for example, raising student awareness about plagiarism [START_REF] Carroll | Deterring Student Plagiarism: Where Best to Start?[END_REF], clarifying what constitutes plagiarism [START_REF] Davis | Preventing Academic Dishonesty[END_REF] or by introducing honour codes [START_REF] Mccabe | New Research on Academic Integrity: The Success of "Modified" Honor Codes[END_REF]. Another response was the introduction of increasingly more sophisticated detection systems [START_REF] Frey | Chips and Cheating: Teachers Are Looking Out for Tech Assisted Dishonesty, even Using Software to Detect Plagiarism[END_REF]. One of the major innovations is the introduction of electronic plagiarism detection systems, such as Turnitin ® , a tool which allows the tutors to check for suspected plagiarism, as well as evolving into a more all-round electronic submission and grading tool.

Plagiarism is a somewhat ambiguous concept, and the exact definition of what plagiarism is remains vague in many areas [START_REF] Liddell | A Comprehensive Definition of Plagiarism[END_REF]. The finer details about what constitutes plagiarism and what not are contested, and in fact are often different from one subject area to another even within the same school or university. This ambiguity in defining plagiarism has been seen as a major cause of students (as well as faculty members) being confused over what exactly plagiarism is, and how to combat it. In fact at least one study finds that a carefully worded definition of plagiarism may well reduce the incidence of plagiarism substantially [START_REF] Brown | The Efficacy of Policy Statements on Plagiarism: Do They Change Students' Views?[END_REF]. For the purpose of this article, anything where a student incorporates a substantial, unacknowledged amount of material derived from the work (published or unpublished) of another student or source is seen as plagiarism [START_REF] Plagiarism | The Little Book of Plagiarism[END_REF]. However, irrespective of the finer details of any such definition, many educators perceive that plagiarism is steadily increasing, and that more students than ever plagiarize material from different sources, especially the Internet [START_REF] Underwood | Academic Offences and E-learning: Individual Propensities in Cheating[END_REF]. At the same time, many educators are unsure as to how to confront this 'tidal wave' of academic dishonesty: some educators seem to give up completely, while others have strong emotional reactions, ranging from feelings of anger to dismay [START_REF] Wood | Academic Original Sin: Plagiarism, the Internet, and Librarians[END_REF].

There are various estimates as to how much plagiarism is taking place. [START_REF] Hansen | Combatting Plagiarism[END_REF] suggests that 38 per cent of students copy from conventional sources and 40 per cent copy from Internet sources. [START_REF] Szabo | Cybercheats: Is Information and Communication Technology Fuelling Academic Dishonesty?[END_REF] in their study of self-reported plagiarism among science students report a rate of 32.2 per cent of students admitting unacknowledged copy and paste content from the Internet into their work. Other sources have indicated academic dishonesty rates as high as 75 per cent [START_REF] Eagle | Dealing With the Dark Side: The Link Between Academic Dishonesty and Future Work Practices[END_REF], with business students apparently more prone to academic dishonesty (and plagiarism) than students from other disciplines [START_REF] Smyth | Perceptions of Dishonesty among Two-year College Students: Academic versus Business Situations[END_REF].

Countering plagiarism by electronic means: what is being done?

For the most part, plagiarism prevention has so far relied on punitive measures [START_REF] Devlin | Policy, Preparation, and Prevention: Proactive Minimization of Student Plagiarism[END_REF], though the focus is increasingly shifting towards plagiarism prevention, for example by having clearer definitions of plagiarism [START_REF] Brown | The Efficacy of Policy Statements on Plagiarism: Do They Change Students' Views?[END_REF], coupled with effective plagiarism detection (Joint Information Systems Committee [JISC], 2005). Detection relies on either traditional search methods, especially Google and other search engines on the Internet, or on using software-based solutions such as PlagiServe or Turnitin. While there is little doubt that the threat of using electronic systems to detect plagiarism may prevent students from plagiarizing, the systems used to be fairly one-dimensional, in that they focused mainly on plagiarism detection. The same is true of most research published in the area (for example [START_REF] Guader | The Hunt for Plagiarism[END_REF]. While this research direction is certainly worthwhile, it tends to neglect how students feel about their work being systematically screened for plagiarism. However, as the detection systems become more integrated with the submission of assessments, and thus less of a standalone system, the relative importance of plagiarism detection may diminish overall.

In the early days of electronic plagiarism detection, there were standalone systems, doing little more than searching for similar phrases being used in websites and academic journals.This made detection fairly cumbersome, as it required individually uploading all assignments, and then reviewing the result once an originality report had been generated. Today, systems such as Turnitin make detecting plagiarism a lot more comfortable, with the system integrating into other university systems (such as WebCT) and providing easy to read, colour coded 'originality reports' and scores on each students work (see example screenshot, Figure 1).

However, such detection techniques are not without their critics. The systems have been criticized because of the lack of 'human' judgement, which in turn may make reliance purely on technical solutions unreliable [START_REF] Carbone | Turnitin.com: A Pedagogic Placebo for Plagiarism[END_REF], or because these systems store large amounts of student assessment for comparison (and profit) without the consent of, or at least a viable alternative for the students. This some argue may in itself be a copyright violation [START_REF] Foster | Plagiarism-Detection Tool Creates Legal Quandary: When Professors Send Students' Papers to a Database, Are Copyrights Violated?[END_REF].

In recent years, Turnitin has, however, shifted the focus from primarily detecting plagiarism to offering other services to both educators and students. The services available now include the possibility to deliver electronic feedback to the students, online grade books and attendance lists, and the possibility of (anonymous) peer-reviews of assignments by students. In addition to functional enhancements, Turnitin now seemingly offers the possibility of a complete electronic submission of assignments, with submission deadlines being enforced by the system itself as well as issuing digital receipts to students submitting work. This enhanced functionality has certainly done much to deflect from the single objective of catching plagiarism and for educators has made Turnitin at least potentially a very valuable tool overall, while at the same time making submissions and feedback via Turnitin seemingly more convenient for students. However, as much as the functionality and integration may be enhanced, it is still ultimately the students and educators who have to use the system. It seems important to see how students habitually using the system perceive Turnitin, especially given the perception of the system as primarily a plagiarism detection tool.

In summary, much of the available evidence suggests that education and detection can drive down plagiarism rates substantially [START_REF] Park | Rebels Without a Clause: Towards an Institutional Framework for Dealing with Plagiarism by Students[END_REF]. However, much of the research published on plagiarism and plagiarism detection so far has been focused on the need for such systems and their efficiency in detecting plagiarism [START_REF] Guader | The Hunt for Plagiarism[END_REF]. No research to date has addressed the issue of the reaction of students towards the systematic use of such detection systems.Therefore the primary aim of this study was to find out how a group of students feel about using electronic submissions, visible usage of plagiarism detection systems and how they perceived the service in comparison to more traditional methods (for example, handing in paper assignments).This article explores how a group of postgraduate students perceive the use of plagiarism detection software (Turnitin) in one module as a basis for further research.

This exploratory study examined the attitudes of postgraduate students on the Integrated Marketing Communication module, part of the MA in Marketing Communications at a large business school in London. The students on this module use Turnitin on a weekly basis to submit their assignments, obtain their grades and feedback as well as conducting regular peer-reviews of their peers' assignments. In addition to feedback and grades, students were also permitted to see their originality reports.

The students did not receive any specific training regarding Turnitin. However, they did receive a step-by-step instruction sheet on how to set up an account and submit work.

Methodology

The participants in this study were all students on the module. Before the beginning of a lecture the students were asked to fill out the study questionnaire. The students were advised that the questionnaire was voluntary, and all 24 students who were present at the time completed questionnaire. The questionnaires were anonymous and no incentive was given to complete the survey. Each questionnaire consisted of 22 statements to which the participants indicated their agreement or disagreement using a 5 point Likert scale, ranging from strongly agree (1) to strongly disagree (5).

The first seven questions related to how students perceived Turnitin in comparison to conventional methods of submissions. Questions 8-14 related to how students perceived the originality report and how they felt about plagiarism being tackled by using a system such as Turnitin. Questions 15-18 gauged the students' confidence in terms of referencing and finally, questions 19-22 asked for an overall evaluation of the students' experience and confidence in the system.

All items had the same polarity, with a lower score indicating strong agreement. Following the 22 statements, the students were also asked five open-ended questions regarding which aspect of Turnitin they liked most and why, which least and why, and how the system could be improved.

Results

Overall, students seemed to find Turnitin easy to use: the mean score for the question whether or not students found Turnitin easy to use was 1.67, with 11 students strongly agreeing, 11 students agreeing and only one student answering 'slightly disagree' to the question; one student was 'undecided' (see Table 1).

Similar to the first question, most students agreed or strongly agreed with the statement that Turnitin was a convenient way to submit work. Again, one student was undecided and another student slightly disagreed; however no student indicated a strong disagreement.

The third question asked whether students preferred using Turnitin to handing in the work at the student office. Of all the questions asked, this was the most positive question in terms of answers given by students: 17 students strongly agreed with this statement, five agreed and only two were undecided. No student disagreed or strongly disagreed with this statement. The fourth question asked whether or not students preferred to hand in paper assignments, without specifically asking for submission either via the tutor or via the student office (that is, in person at a central location where the students get a paper receipt for their work). The answers here were also very much in favour of electronic submission, with seven students strongly disagreeing with the statement, 11 disagreeing and only two students agreeing with this statement.

When asked whether students prefer to get feedback electronically, overall 78 per cent agreed or strongly agreed, with only one student disagreeing with this statement. Thus, the facility to give feedback seems to mostly be welcomed by the students. This outcome seemed slightly surprising, as the electronic feedback requires the student to read the feedback online rather than receiving a paper copy.

All students preferred to obtain their grades electronically via Turnitin (20 students agreed, or strongly agreed). This seems hardly surprising given the enthusiastic response towards receiving electronic feedback.

The last question in this section was interesting in that it showed a strong support for the adoption of Turnitin in more modules: 87.5 per cent of the students (21 students) would prefer to see Turnitin being adopted in other modules.

The next set of questions related to the core functionality of Turnitin as a tool to detect plagiarism and what effect this 'originality report' has on the students. As all students could readily access their originality reports from each assignment, it can be assumed that they were sufficiently familiar to comment on these.

In the first question of this set (question 8) it was interesting to see how positive the students were regarding their originality report: 83 per cent agreed or strongly agreed with the statement that they liked to see their report (see Table 2).

Overall the originality report appeared to be easy to understand for most students, although a relatively large number of students (seven) indicated that they are undecided on this topic, and two students disagreed with the statement. The students did not receive any formal training on how to use Turnitin, thus it may not be entirely surprising that the students felt slightly insecure about how to understand an originality report.

Some 62 per cent of the students (15 students in total agreed or strongly agreed and a further 5 students were undecided) were afraid that they would be accused of plagiarism although it is not true after using the system. While this appears to be a high number at first, it seems possible that this 'fear' translates into less plagiarizing, as students may as a result be extra careful when including material from other sources. The next question (question 11) tried to find out if a potential fear translated into more careful consideration when including third party material:The answers seem to suggest that the students are split halfway on this topic. some students are unsure some others seem to be quite confident in terms of including materials in their work. This seems indicative that the assumption that the mere presence of the system encourages students to critically evaluate the materials used may well be true.

Question 12 gauged whether students felt anxious about the originality report -again the students appeared either anxious, or not anxious at all. Similar to question 11, this may be because some students are more confident in the kind of materials they use (and how they use them) and subsequently are not anxious about their originality score.

About half of all students (13 students) were happy that Turnitin made plagiarism harder. This seems at first slightly surprising, though it may be a reflection of the fact that some students feel disadvantaged by other students 'getting away' with plagiarism. However, it is interesting to note that three students disagreed with the statement.

Most students seemed to see the systematic use of Turnitin as an indication of something being done about plagiarism.This may suggest that a more widespread use of the system may convey the message that plagiarism is taken seriously, although it appears that the use of Turnitin alone may not be sufficient to convince students of this. Possibly a more widespread introduction in other modules would enhance the perception.

The following four questions related to how confident students are with regards to plagiarism and citing correctly.

16 of the 24 students in total wanted to have more information about what plagiarism is. This seems to indicate that there is at least some uncertainty in this group of students as to what exactly is plagiarism. This high number of students is slightly surprising, as all students did receive a formal presentation at the beginning of their programme on plagiarism and plagiarism detection (see Table 3).

While overall there are some students who are confident about writing a bibliography, a significant number are not sure or are undecided about how to write a bibliography.

However, although some of the students were not confident about how to write a bibliography, the introduction of Turnitin seems to have had a positive effect in terms of students reading up on how to reference correctly, with just over half of students agreeing with this statement.

However, despite resources on referencing being readily available, only about half of the students felt confident about quoting other people's works correctly, which may indicate a further need to train the significant minority which seems unsure about how to quote. The last set of questions asked the students to evaluate the reliability of Turnitin both in terms of detecting the correct sources and the overall stability of the system.

Generally, the students rated Turnitin as very reliable: none of the students disagreed with the statement and only three students were undecided (see Table 4).

At least two students indicated that the originality report did not find the source the students used for their assignments. Of course this may be either because sometimes the originality reports pick up on some random sentences or frequently used phrases from other assignments stored in the database, or it can be because some sources may not be available to Turnitin because of technical limitations.

In terms of the originality reports there was less confidence in the system: although around half of the students found them generally accurate, 50 per cent of the students were undecided.

Finally, the majority of students felt safe that their assignments will not get lost when using Turnitin, a possible concern especially for students who are less experienced with computers.

In the open-ended feedback it was interesting that 22 answers stressed the convenience of the system in one form or another, for example by making it possible to submit without queuing at the student office or more generally in terms of the ease of use of the system. However, two students singled out the fact that something was done about plagiarism as the aspect they liked most about Turnitin.

In terms of negative points, the sometimes wrong (or at least confusing) originality report and slow network connections were the most often named, although some students chose not to indicate any negative points at all.

Although the dataset is very limited in terms of numbers, it is still interesting to highlight some interesting correlations that can be observed in the data set.

From the data in Table 5, there appears to be a group of students who find Turnitin both convenient and easy to use ( convenient/easy ϭ 0.668).This group of students, not surprisingly, also wants more modules to adopt Turnitin ( convenient/more modules ϭ 0.517 and easy to use/more modules ϭ 0.538). Students in this group are generally happy that something is being done about plagiarism ( convenient/happy plag ϭ 0.663 and easy to use/happy plag ϭ 0.530). Interestingly, those that find Turnitin easy or convenient to use are not necessarily the same students as those that prefer electronic feedback ( convenient/feedback elec ϭ 0.146 and easy to use/feedback elec ϭ 0.000).

At the same time, students who are happy that something is being done about plagiarism, maybe unsurprisingly, tend to be sure about how to quote ( happy plag/unsure quoting ϭ -0.545). However, students who were unsure of how to quote tended to prefer paper assignments ( unsure quoting/pref paper ϭ 0.517). Further, those students that were the most likely to prefer paper submissions were most likely to be anxious about their originality report ( anxious orig/pref paper ϭ 0.745). In turn, those students who were anxious about their originality reports also were very likely to be unsure of how to quote correctly ( anxious orig/unsure quoting ϭ 0.713). This could suggest that more information on how to quote correctly might overcome the feeling of anxiety for some students and at the same time might suggest that some students do not actively seek to plagiarize, but rather feel insecure about how to avoid it (and subsequently adopt a negative attitude towards plagiarism prevention).

Finally, interestingly there appears to be a correlation between finding the originality report easy to understand and reading up on how to reference correctly ( read how ref/orig easy ϭ 0.571). This may indicate that there is indeed a formative element in students seeing (and understanding) their originality reports, resulting in more attention being paid to correct use of referencing.

Discussion and implications

The findings of this small study seem to strongly support the more widespread adoption of Turnitin. Although there are some limitations of the study, such as the exploratory and small sample of postgraduate-only students, the results seem to be broadly positive for the use of Turnitin. While there are some concerns over the interpretation of originality reports, most students in this study were largely positive about Turnitin. Some students explicitly welcomed the introduction as a way to reduce plagiarism, though most students seemed to prefer the convenience of electronic submission in comparison to the more traditional submission via the student office.

Overall it appears that there are mainly two types of students: those who are confident about how to avoid plagiarism and how to quote correctly tend to be happy about the introduction of Turnitin. However, a different group of students tends to be less positive about Turnitin, though this seems mostly a result of their own lack of knowledge and insecurity about plagiarism and what is acceptable when quoting, rather than a malicious intent to plagiarize. This relative lack of students who deliberately set out to cheat is consistent with other studies (e.g. [START_REF] Macdonald | Plagiarism -A Complex Issue Requiring A Holistic Institutional Approach[END_REF], and underlines the fact that education together with effective detection is a viable way to cut plagiarism.

As such, the introduction of Turnitin (and therefore the visible use of detection systems) seems to have a noticeable effect on students. Firstly, the effect seemed formative, with at least some students seeking advice on how to reference correctly. Secondly, the effect seemed deterrent, as only two cases of plagiarism were detected during the semester.This is a surprisingly low rate if any of the plagiarism rates reported in the literature are to be applied (e.g. [START_REF] Eagle | Dealing With the Dark Side: The Link Between Academic Dishonesty and Future Work Practices[END_REF][START_REF] Szabo | Cybercheats: Is Information and Communication Technology Fuelling Academic Dishonesty?[END_REF]).Yet, the fact that two cases were found also highlights that even with visible detection methods in place plagiarism does still occur, thus the introduction of electronic detection is a tool to reduce, but not to eradicate, plagiarism, which again is consistent with the literature [START_REF] Carroll | Handling Student Plagiarism: Moving to Mainstream[END_REF].

As the students did not receive any special training on how to use Turnitin, it may be somewhat less surprising that some students seemed to be confused about the originality reports (though generally not about how to operate the website), and what they meant in terms of plagiarism. This seems to support the argument by [START_REF] Brown | The Efficacy of Policy Statements on Plagiarism: Do They Change Students' Views?[END_REF] that a clear definition of plagiarism will potentially reduce incidences of plagiarism.

Of course, this study was exploratory in nature, and therefore further research would be desirable to find out if the findings of this study can be replicated. Research with larger groups of students would be especially useful. In addition, research on students from different types of institutions and different degrees and on undergraduate as well as postgraduate courses would be desirable. Conversely, it would also be interesting to see if after receiving more training on how to reference correctly, students' perceptions of Turnitin would be more positive.

Overall, however, it was reassuring to find that the students reacted positively to the introduction of Turnitin. When considering that one of the arguments against the use of Turnitin is that it treats students as criminals [START_REF] Carbone | Turnitin.com: A Pedagogic Placebo for Plagiarism[END_REF], the reaction of students when asked directly seems reassuring for educators who are considering whether or not to use electronic plagiarism detection systems such as Turnitin. Also the research supported the view that few students actively set out to cheat, but rather that a combination of education and detection has the potential to cut down on plagiarismand that the students are very much in favour of this approach. The convenience of submitting electronically Convenience of doing it from almost everywhere It is electronic, no que (sic) at the student office submitting online sitting in our room easy to use, receipt, confident that work will not get lest comfortable it is a easy way to handing in my coursework and haven't wait in a long q. It is easy know my grades and feedback straightforward to use not having to queue up at the student office and having to deal with the rude staff there, flexible times I like the fact that I do not need to stand in endless queues to submit my work fast, easy entrance fast, easy and convenience I like the way to submit because it is easy sure that my assignment has sent as it shows on screen I don't have to go to university to hand in my works, I even don't have to print it out It is a very good system, especially plagiarism The quick way of submit It is much more convenient than submitting paper assignments Good method for submitting work When I have to submit my paper, it's easy to use Convenient to hand in assignment, student office will have more time to deal with other issues rather than submitting students' work Submit work online No long queue for assignment submission, easy to use, no paperwork Great that something is being done about cut and paste culture Forces students to work not copy. Good for honest students.

Appendix 1: Summary of questions and statistical data

Biographical note

STEPHAN DAHL is a Senior Lecturer in Marketing at Middlesex University Business School in London, where he is the programme leader for the MA in Marketing Communications. His research interests include health marketing and cross-cultural marketing. Dr Dahl's current research focus is on the role marketing communication can play in preventing health problems and how marketing communication can be used to help patients with chronic diseases. Address: Middlesex University Business School, The Burroughs, Hendon, London NW4 4BT, UK. [email: s.dahl@mdx.ac.uk] 191

Figure 1

 1 Figure 1 TurnitinUK Originality Report

  189

  D A H L : T U R N I T I N ® : P L A G I A R I S M D E T E C T I O N Mean Std. D Ra Min. Max.

  1

Appendix 2 :

 2 Answers to 'What do you like best about Turnitin'

  190 A C T I V E L E A R N I N G I N H I G H E R E D U C AT I O N 8(2)

Table 1 Questions relating to ease of use

 1

	Average		1.67		1.62			1.38				3.96		2.00					1.83				1.79
	Strongly	disagree	0%	(0)	0%	(0)		0%	(0)			29.2%	(7)	0%	(0)				0%	(0)			0%	(0)
	Disagree		4.2%	(1)	4.2%	(1)		0%	(0)			45.8%	(11)	4.2%	(1)				0%	(0)			4.2%	(1)
	Undecided		4.2%	(1)	4.2%	(1)		8.3%	(2)			16.7%	(4)	16.7%	(4)				16.7%	(4)			8.3%	(2)
	Agree		45.8%	(11)	41.7%	(10)		20.8%	(5)			8.3%	(2)	50%	(12)				50%	(12)			50%	(12)
	Strongly	agree	45.8%	(11)	50%	(12)		70.8%	(17)			0%	(0)	25%	(6)				33.3%	(8)			37.5%	(9)
	Statement		I find Turnitin	easy to use	I find Turnitin a	convenient way to submit	my assignments/coursework	I find Turnitin	more convenient than	handing in my work	at the student office	I prefer handing	in paper assignments	I prefer getting	my feedback electronically	rather than on a paper	based coursework	front sheet	I prefer to see my grades	electronically on Turnitin rather	than on a paper based	coursework front sheet	I would prefer it if more	modules would use	Turnitin for submitting	assignments/coursework
			1		2			3				4		5					6				7

(N) ϭ Number in brackets gives actual number of students

Table 2 Questions relating to originality report

 2

	Strongly disagree Average	2.04 4.2%	(1)	2.33 0%	(0)	2.33 4.2%	(1)		3.08 16.7%	(4)		2.83 8.3%	(2)		2.37 0%	(0)	2.13 0%	(0)
	Disagree	0%	(0)	8.3%	(2)	12.5%	(3)		20.8%	(5)		29.2%	(7)		12.5%	(3)	4.2%	(1)
	Undecided	12.5%	(3)	29.2%	(7)	20.8%	(5)		20.8%	(5)		16.7%	(4)		33.3%	(8)	33.3%	(8)
	Agree	62.5%	(15)	50%	(12)	37.5%	(9)		37.5%	(9)		29.2%	(7)		33.3%	(8)	33.3%	(8)
	Statement Strongly agree	I like to see my originality report 20.8%	(5)	I find the originality report easy 12.5%	to understand (3)	I am afraid that I may be accused 25%	of plagiarism based on the originality (6)	report although it is not true	Turnitin makes me unsure about 4.2%	what materials I can quote or use (1)	in my assignment	I get anxious about my originality 16.7%	score when I submit my work to (4)	Turnitin	I am happy that Turnitin makes 20.8%	plagiarism a lot harder (5)	Turnitin makes me feel as if 29.2%	(7) something is being done	about plagiarism	ϭ Number in brackets gives actual number of students
		8		9		10			11			12			13		14			(N)

Table 3 Confidence in citing

 3

	Average	2.75		2.50		2.58			2.46
	Strongly disagree	0%	(0)	4.2%	(0)	0%	(0)		0%	(0)
	Disagree	8.3%	(2)	12.5%	(3)	16.7%	(4)		20.8%	(5)
	Undecided	25%	(6)	45.8%	(11	29.2%	(7)		33.3%	(8)
	Agree	54.2%	(13)	29.2%	(7)	41.7%	(10)		29.2%	(7)
	Statement Strongly agree	12.5% I would prefer to get more	(3) guidance about what plagiarism is	8.3% I am confident when writing	(2) a bibliography	12.5% I have read up about how I should	(3) reference something correctly	after starting to use Turnitin	16.7% I feel confident about quoting other	(4) people's works correctly	ϭ Number in brackets (actual number of students)
		15		16		17			18		(N)

Table 4 Questions relating to reliability

 4

	Average	1.96		2.50		2.04		2.75
	Strongly disagree	0%	(0)	0%	(0)	0%	(0)	0%	(0)
	Disagree	8.3%	(2)	0%	(0)	4.2%	(1)	8.3%	(2)
	Undecided	33.3%	(8)	12.5%	(3)	50%	(12)	20.8%	(5)
	Agree	54.2%	(13)	70.8%	(17)	37.5%	(9)	37.5%	(9)
	Statement Strongly agree	4.2% Turnitin usually finds the sources	(1) I use for my assignments	16.7% Overall, Turnitin is reliable	(4)	8.3% Overall, Turnitin originality	(2) reports are accurate	33.3% I feel safe that my work will not	(8) get lost when submitting it via	Turnitin	ϭ Number in brackets (actual number of students)
		19		20		21		22			(N)

Table 5 Correlations of answers

 5

	1 .668 .469 Ϫ.272 .146 .201 .517 Ϫ.041 .138 Ϫ.250 Ϫ.431 Ϫ.421 .663 .573 Ϫ.306 ؊.075 Ϫ.030 .069 .323 .475 .196 .141	.668 1 .530 Ϫ.461 .000 .217 .538 .089 .117 Ϫ.118 Ϫ.298 -.374 .530 .381 Ϫ.118 Ϫ.060 Ϫ.122 Ϫ.019 .290 .381 .000 .140	.469 .530 1 Ϫ.638 .440 .622 .679 .049 Ϫ.165 Ϫ.179 Ϫ.263 -.396 .390 .289 Ϫ.219 Ϫ.053 Ϫ.252 .050 .082 .290 .233 .114	Ϫ.272 Ϫ.461 ؊.638 1 Ϫ.622 ؊.557 ؊.566 Ϫ.109 Ϫ.039 .396 .517 .745 Ϫ.377 -.206 .314 .038 .077 Ϫ.161 Ϫ.302 -.352 Ϫ.431 Ϫ.199	.146 .000 .440 Ϫ.622 1 .715 .502 .000 .274 Ϫ.099 Ϫ.234 Ϫ.442 .288 .310 Ϫ.416 Ϫ.059 .301 .331 .312 .405 .390 .418	.201 .217 .622 Ϫ.557 .715 1 .649 Ϫ.060 Ϫ.051 Ϫ.256 Ϫ.340 Ϫ.470 .351 .034 Ϫ.371 Ϫ.066 .066 .203 .157 .432 .514 .335	.517 .538 .679 Ϫ.566 .502 .649 1 .144 .251 Ϫ.264 Ϫ.487 Ϫ.606 .511 .349 Ϫ.453 .103 .269 .379 .100 .486 .193 .071	Ϫ.041 .089 .049 Ϫ.109 .000 Ϫ.060 .144 1 .537 .120 Ϫ.003 .007 .033 .049 .170 Ϫ.040 .136 .269 .319 .188 Ϫ.175 .104	.138 .117 Ϫ.165 Ϫ.039 .274 Ϫ.051 .251 .537 1 .157 Ϫ.249 Ϫ.195 .220 .473 Ϫ.418 .226 .571 .593 .394 .226 Ϫ.147 .260	Ϫ.250 Ϫ.118 Ϫ.179 .396 Ϫ.099 Ϫ.256 Ϫ.264 .120 .157 1 .518 .614 Ϫ.357 Ϫ.086 .032 Ϫ.367 .165 Ϫ.290 Ϫ.303 Ϫ.327 ؊.533 .027	Ϫ.431 Ϫ.298 Ϫ.263 .517 Ϫ.234 Ϫ.340 Ϫ.487 Ϫ.003 Ϫ.249 .518 1 .713 Ϫ.545 Ϫ.488 .552 Ϫ.437 Ϫ.038 ؊.569 Ϫ.244 Ϫ.255 Ϫ.347 Ϫ.266	Ϫ.421 Ϫ.374 Ϫ.396 .745 Ϫ.442 Ϫ.470 ؊.606 .007 Ϫ.195 .614 .713 1 Ϫ.510 Ϫ.398 .346 Ϫ.145 Ϫ.146 Ϫ.492 Ϫ.387 Ϫ.321 Ϫ.378 Ϫ.173	.663 .530 .390 Ϫ.377 .288 .351 .511 .033 .220 Ϫ.357 ؊.545 ؊.510 1 .592 Ϫ.257 .154 Ϫ.024 .341 .303 .520 .279 .405	.573 .381 .289 Ϫ.206 .310 .034 .349 .049 .473 Ϫ.086 Ϫ.488 Ϫ.398 .592 1 Ϫ.352 .294 .078 .487 .243 .275 Ϫ.100 .348	Ϫ.306 Ϫ.118 Ϫ.219 .314 Ϫ.416 Ϫ.371 Ϫ.453 .170 Ϫ.418 .032 .552 .346 Ϫ.257 Ϫ.352 1 Ϫ.300 Ϫ.376 Ϫ.322 .059 Ϫ.069 Ϫ.187 Ϫ.129	Ϫ.075 Ϫ.060 Ϫ.053 .038 Ϫ.059 Ϫ.066 .103 Ϫ.040 .226 Ϫ.367 Ϫ.437 Ϫ.145 .154 .294 Ϫ.300 1 .198 .611 Ϫ.208 Ϫ.105 Ϫ.191 .253	Ϫ.030 Ϫ.122 Ϫ.252 .077 .301 .066 .269 .136 .571 .165 Ϫ.038 Ϫ.146 Ϫ.024 .078 Ϫ.376 .198 1 .321 Ϫ.032 .297 Ϫ.065 .122	.069 Ϫ.019 .050 Ϫ.161 .331 .203 .379 .269 .593 Ϫ.290 ؊.569 Ϫ.492 .341 .487 Ϫ.322 .611 .321 1 .212 .045 .000 .421	.323 .290 .082 Ϫ.302 .312 .157 .100 .319 .394 Ϫ.303 Ϫ.244 Ϫ.387 .303 .243 .059 Ϫ.208 Ϫ.032 .212 1 .379 .459 .161	.475 .381 .290 Ϫ.352 .405 .432 .486 .188 .226 Ϫ.327 Ϫ.255 Ϫ.321 .520 .275 Ϫ.069 Ϫ.105 .297 .045 .379 1 .492 .252	.196 .000 .233 Ϫ.431 .390 .514 .193 Ϫ.175 Ϫ.147 ؊.533 Ϫ.347 Ϫ.378 .279 Ϫ.100 Ϫ.187 Ϫ.191 Ϫ.065 .000 .459 .492 1 .032	.141 .140 .114 Ϫ.199 .418 .335 .071 .104 .260 .027 Ϫ.266 Ϫ.173 .405 .348 Ϫ.129 .253 .122 .421 .161 .252 .032 1
	Convenient	Easy_to_use	Con_stdo	Pref_paper	Feedback_elec	Grades_elec	More_modules	Like_orig	Orig_easy	Afraid_false	Unsure_quoting	Anxious_orig	Happy_plag	Something_done	Guidance	Confid_biog	Read_how_ref	Con_quoting	Finds_source	Reliable	Orig_accurate	Feel_safe	Bold ϭ Correlation is

significant at the 0.01 level (2-tailed)

	Feel_safe
	Orig_accurate
	Reliable
	Finds_source
	Con_quoting
	Read_how_ref
	Confid_biog
	Guidance
	Something_done
	Happy_plag
	Anxious_orig
	Unsure_quoting
	Afraid_false
	Orig_easy
	Like_orig
	More_modules
	Grades_elec
	Convenient Easy_to_use Con_stdo Pref_paper Feedback_elec	ϭ Correlation is significant at the 0.05 level (2-tailed)
		Italics

D A H L : T U R N I T I N ® : P L A G I A R I S M D E T E C T I O N