S T E V E L Ov

Developing a quality assurance metric A panoptic view

Keywords: module evaluation methods, panoptic view, student

A B S T R AC T There are a variety of techniques that lecturers can use to get feedback on their teaching -for example, module feedback and coursework results. However, a question arises about how reliable and valid are the content that goes into these quality assurance metrics. The aim of this article is to present a new approach for collecting and analysing qualitative feedback from students that could be used as the first stage in developing more reliable quality assurance metrics. The approach, known as the multi-dimensional crystal view, is based on the belief that individuals have different views on the benefits that the embedded process in a system can have on the behaviour of the system. The results of this study indicate that in the context of evaluation and feedback methods, the multi-dimensional approach appears to provide the opportunity for developing more effective student feedback mechanisms.

Introduction

Student feedback and their evaluation of the teaching provision that they receive are crucial elements in the quality assurance process (Church, 2001). The Quality Assurance Agency (QAA) states that it is essential that there is a 'dialogue' between students and the people who teach them and supervise their work (QAA, 2004). They emphasize the importance of both qualitative and quantitative feedback from students as part of an overall strategy of 'enhancement and continuous development' on the part of teachers. Therefore, evaluation of teaching provision is a crucial part of our jobs as academics. Ramsden (1992) emphasized the importance of reflection for the teacher, and with the advent of the QAA review one can see that developing strategies for ascertaining effective and informative feedback from students on teaching provision has been taken on at an institutional as well as the individual teacher level.

There are a variety of techniques that we use to get feedback on our teaching: self-monitoring, audiotape/videotape, information from students (in the form of questionnaires), students' coursework results and exam results and outside observers (peer observation). Indeed, in recent years there have been several publications designed to provide 'best practice' guides for teachers and institutions (e.g. Learning and Teaching Support Network [LTSN] Assessment Guide, 2004;Evaluation Cookbook, 1999;Brennan and Williams, 2004). According to Fink (1995), the best feedback we can get on our teaching methods and assessment methods comes from students. They are in the unique position to carry out this evaluation as they experience first hand the effects of our teaching. Bearing this mind, the primary methods for collecting feedback from students come via questionnaires and interviews.

Questionnaires are the most common method used to get feedback on our teaching from students in the UK (Cowan et al., 2002). Typically, they contain questions relating to aspects of the module delivery (e.g. study guide for the module, printed handout, module resources), quality of feedback from the lecturer (e.g. in lectures, on coursework, to individual enquiries) and also some information from the students themselves (e.g. educational background, attendance at lectures). Questionnaires can be administered to students at any time but the main advantage of asking students to complete them at the end of term is that they can fully evaluate the learning activities they carried out during the module and evaluate the overall module. In addition, questionnaires have the benefit of anonymity on the part of the student, as they may otherwise be put off offering a criticism of the module for fear of 'staff reprisals '. However, Wilkinson (2003) points out that this approach has several weaknesses such as the superficial data it may provide from some students owing to the nature of the questions asked and the fact that some students may suffer from what she terms 'questionnaire fatigue' as a result of the number of questionnaires they are asked to complete each year.

Interviews, on the other hand, provide the opportunity for more in-depth feedback about the module and the academics involved in the teaching of this module. One drawback of this approach could be that students may feel awkward about the loss of anonymity (i.e. the student may be sitting in a one-to-one situation with the interviewer) in discussing aspects of the course and its delivery unless they have a good rapport with their teacher. An immediate solution to this would be to get a third party to conduct the interviews. One of the major advantages of the interview approach is that students can often highlight strengths or weaknesses of the course materials and module delivery that the interviewer had not anticipated beforehand. This would allow for further exploration of these topics in the course of the interview. The focus group is another technique that has been used to gather student feedback (Wilkinson, 2003). This approach provides the opportunity for a broad range of views to be gathered from students and discussed in depth. This benefit is acknowledged by Robson (1993) who stated that focus groups potentially offer more in-depth and qualitatively rich feedback from students in contrast to the information collected from questionnaires.

A key factor in the success of any of the methodological approaches highlighted above is the generation of appropriate information from students that can be subsequently used to improve their experience at the module level, programme of study level and ultimately at the university level. For example, how many departments evaluate the design of their module feedback forms on a regular basis to ensure that they provide students with the opportunity to comment on the major issues that have affected their perception of the module they have just studied? The aim of this article, therefore, is to present a new approach for collecting and analysing qualitative feedback from students (and staff), that could be used as the first stage in the development of a more formal feedback mechanism such as a focus group or questionnaire, to ensure all the major factors relating to quality assurance assessment are highlighted.

Our methodological approach

The novel approach that we are proposing, known as the multi-dimensional crystal view, is based on the belief that individuals have different views on the benefits that the embedded processes in a system have on the behaviour of the system. The views may arise from the particular perspective or past experiences of the stakeholders, but will, in any case, influence their attitudes when engaging with the process and, therefore, the expected functioning of the system. The approach, using simple matrix structures, attempts to capture and analyse these views in order to present a number of statements, or panoptic views, that may give insight on how effectively the system operates (Paul and Scoble, 2003;Scoble, 2002). Since this approach attempts to collate and compare qualitative information from a number of different sources, it is necessary, to make any of the views comparable, to have a methodology that narrows the collection of views to very specific parts and process. This is achieved by selecting a number of key processes and asking the stakeholders to assess their impact on key expected outcomes. In this approach the stakeholders are called 'points of observation' and the key processes are called 'drivers' while the key expected outcomes are called 'areas of interest'. This can be seen as a microanalysis of the interactions between processes and outcomes, or drivers and areas of interest, as viewed by each stakeholder, and can be compared with looking through a crystal and describing the reaction created when rays of light hitting from different directions meet somewhere within the bound crystal. Each stakeholder may look from a different angle of the crystal and what is seen is a particular combination of rays of light, or drivers and areas of interest. The panoptic view is achieved when these micro-analyses are aggregated into single statements that encompass and contrapose different perceptions of the systems' efficacy and efficiency.

A matrix tool for analysis

The use of a matrix as the analytical tool allows for an effective and 'tidy' collection of the stakeholders' view of the drivers and the areas of interests. Rows will identify the drivers, the key processes, while the columns will identify the areas of interest, the key outcomes. Each point of observation, stakeholder, will complete a matrix by giving a statement on their own view of how each driver impacts on each area of interest. The completed matrices can then be aggregated to create one single statement for each cell of the matrix that will encompass the views of each stakeholder. Insight and knowledge will then emerge from further interim aggregations concluding in one single statement on the behaviour of the system. The choice of such a tool is the ability to switch very easily between detail and summary while maintaining all the initial information. The matrix is also conducive to the use of colour coding to give a visual aid in the presentation of interim aggregations and final general statements (see Figure 1).

A C T I V E L E A R N I N G I N H I G H E R E D U C A T I O N 7(2) 132

The presentation tool: windows style and colour coding

The colour coding used for the presentation of the results of the aggregation allow for rapid interpretation. After each aggregation the cell containing the summary statement will be colour coded according to the overall judgement of the specific drivers on the area of interest. For example, in the case of the negative judgement from a point of observation on the impact of a driver on a specific area of interest, the cell containing the statement will be colour coded light blue. For simplicity there are only 3 colours: light blue for a negative impact, pale blue for a positive impact and no colour for no causal effect, in other words, no impact. The colour coding is also used when aggregating results. In case of aggregation of contradicting perceptions the cell will display an amount of colour proportional to the collated judgement. For example, a prevalence of light blue will indicate a prevalence of negative perceptions. A step-by-step process regulates the building of the matrix, the assemblage of the matrices for each point of observation and the aggregation process.

The process

The matrix is built step-by-step by first selecting the drivers, areas of interest and points of observation and by preparing the sample matrix by inserting the drivers as row headings and the areas of impact as column headings. The second step is to define the framework, or question table, for the collection of the information as a series of questions, one per cell, to pose to the individual selected or points of observation. Each question relates only to the row and column it refers to, i.e. to the interaction between a specific driver and a specific area of interest. The third step is to fill in each cell on each table, one per point of observation, according to the coordinates (driver and area of interest) with a statement of that particular interaction. The statements are a form of judgement of the consequence of the driver on the area of impact as perceived by the point of observation. Each cell is colour coded according to the judgement presented. The fourth step is to aggregate each point of observation matrix on each interaction into one summary matrix. The summary statement of each single matrix will be collated into a new summary statement in the new matrix. The cells in the new matrix will be colour coded according to the summary judgement presented.

The fifth step is to aggregate all summary statements on each of the remaining two dimensions. The first dimension to be considered is the areas of interest. The result is a table with one column summarizing for each driver its general impact on the system. Therefore, for each driver there will be only one summary statement that will encompass its impact on all areas of interest as viewed by the points of observation. Colour coding is applied according to the summary judgement. The same process is adopted for the aggregation of all drivers. The resulting table will have just one summary row that will encompass, for each area of interest, the impact of the drivers of the process as viewed by the points of observation. Colour coding is applied according to the summary judgement. The sixth and last step is to aggregate all summary statements of the impact of the drivers on the areas of interest. This will result in a single statement that generalizes, in the case study reported in this article, the impact of the drivers of the final year project process on areas of interest viewed by different points of observation. Colour coding is applied according to the summary judgement.

Matrix construction

The matrix is progressively constructed by selecting and inserting driver and area of interest in the rows and column of the matrix. Drivers are selected from the key processes of the Final Year Project module. In this model these were identified as:

• the impact of the project assigned to or selected by the student • the impact of the intermediate assessments and their feedback -tasks • the impact of the 1st supervisor and 2nd reader -supervision.

The drivers are the key processes that define the final year project module. In terms of the project definition, the first stage of the final year project process is to define the project that the student will undertake. In relation to this, students have two options; they can choose a project proposal put forward by a lecturer or they can submit their own project proposal and hope that they will find a supervisor who is willing to supervise this project. The second driver is the deliverables associated with this module. In this case, students have three pieces of coursework to submit. Task 1 requires students to submit a project definition report. In this report students are asked to define the project in terms of its aims, objectives, the methods they will use to carry out the project (e.g. requirements gathering, evaluation) and a project plan. For Task 2, students are required to create an A1-sized poster presentation. The student uses the poster to give a 10 minute presentation to their supervisor and second reader. They will then be asked questions in relation to this. The idea behind this is to give students midterm feedback on their project. Task 3 is the project report itself. This should be a report of approximately 10,000 words in length.

The final driver is supervision. The student has a supervisor whom they should meet with on a regular basis for guidance and advice on their project. In addition, the student has a second reader who marks their three pieces of coursework (apart from Task 1, the supervisor also marks the student's other two pieces of coursework). These are highlighted in Table 1.

Areas of interest are selected from the key outcomes of the Final Year Project module. In this model these were identified as:

• how the drivers impact on the quality of the project and consequential grade -quality/grade of the project • how the drivers impact on the learning outcomes -learning outcomes.

The areas of interest for the final year project were self-selecting. For students, and staff, the quality of the project is of the utmost importance as this will have a direct impact on the grade that they receive for their final year project. The other areas of interest are the learning outcomes. For a project to pass it must meet the learning requirements set out in the study guide. For a project to receive a good grade it must, therefore, not only meet these requirements but exceed them. Information on how students can achieve this are defined in the assessment criteria for each of the three assessed tasks. These are listed in Table 2.

Points of observation are selected from the key stakeholders of the Final Year Project module. In this model these were identified as:

• students' perception of the impact of the drivers on the areas of interest -students • supervisors' perception of the impact of the drivers on the areas of interest -supervisors.

The final dimension to this matrix is the points of observation. In the case of the study reported in this article, there are two points of observation: the students' and the academic members of staff perspectives.

Method of collection for the points of observation

A question table is used to guide the analyst through the collection of the different points of observation. For example, participants were asked questions such as, 'does choosing a project proposed by staff affect the final grade?' The points of observation were collected by personal interviews.

Each interviewee was asked to respond to the questions from the question table . A statement that summarises the interviewee's position is then entered in the corresponding cell and an appropriate colour coding is applied with the definitions given in Table 3.

A C T I V E L E A R N I N G I N H I G H E R E D U C A T I O N 7(2)
136

Data collection

For the purposes of the study reported here, three academic members of staff and three students were recruited to take part in the study. The students had just recently completed their final year project and received their final feedback for it. The questions were asked by the second author as she did not have any direct involvement with the final year project process. This seemed the experimentally valid option to take given what was mentioned in the introduction to the article about students (and perhaps even staff members) feeling uncomfortable about making their views known about a module to the module leader. All participants were informed that the data collection would be aggregated in the write up of the study and, in addition, that it would be anonymized. In addition, all participants were informed that they could receive a copy of the report once it had been written up or could contact the researchers if they had any subsequent questions in relation to the study.

Results of study

The results obtained from this methodological approach are presented in Table 4, which provides a 'panoptic view' by aggregating the responses from both sets of stakeholders (students and staff). From the results obtained from this approach, it appears that there are differences in student and staff perceptions, especially in relation to project supervision. Here, students feel it is important that members of staff have the appropriate experience to supervise their project effectively, whereas members of staff believe that it is the student's motivation to carry out the project and meet their deadlines that is the most important factor in conducting a successful project.

In addition, students believe that the feedback that they receive from their supervisors in relation to task 1 (project plan) and task 2 (poster presentation) is important for the successful development of their project. However, they also feel that is quite confusing if they appear to be getting contradictory feedback from their supervisors for these tasks. The students stated that it leaves them confused about which set of feedback comments they should follow. In relation to Task 3 (the final report) students were happy to get their grade and review the comments from their supervisor and second reader but felt that, by and large, these comments were redundant because they had already finished their project at this stage and could not make any changes to it.

Apart from assessing the actual content of the feedback, another important aspect to consider is the efficacy of this approach as a mechanism for Staff projects appear to be a lot more beneficial both for the quality of the final project and to help students achieve their learning outcomes, because of the knowledge staff have of the problem. However, students feel that a lack of ownership may demotivate students.

Project definition

Task 1 and task 3 appear to be the greatest contributors to the quality of the project. However, task 2 is felt by the students to be very time-consuming and not as focused as the other tasks. Task 3 is the biggest contributor to the achievement of the learning outcomes, while the other two tasks appear to help develop only project management skills.

Tasks

There are some reservations that by task 3 all learning outcomes should already have been achieved. Feedback is mainly seen as a checkpoint to review progress. However, students have reservations on the feedbacks from tasks 1 and 2 that could set a bias against the students when it comes to task 3. Feedback on task 3 appears to have no impact at all.

There is great discrepancy between the views of students and staff regarding project supervision. Students believe that, while expertise is important, the 1st supervisor's experience has the greatest positive impact on the quality of the project and the achievement of the learning outcomes. Staff believe that the experience and expertise of the supervisors Supervisors have very little impact and it is more the student's willingness to be involved that bears the best fruit.

generating more reliable and valid topics to be included in quality assurance mechanisms (such as questionnaires and semi-structured interviews) to obtain feedback from students. For example, when this matrix was compared to the formal feedback questionnaire for this module, it became apparent that this approach offered students the opportunity to provide more in-depth information on issues not addressed by the formal methods, such as perception of the quality of feedback provided by the supervisor and second reader.

In addition, the use of the colour coding scheme highlights the important areas of concern that could be included in any formal quality feedback methodology. Although this could be seen as a crude attempt at producing categorical data, the small data set has allowed for the collection of in-depth qualitative data.

Discussion

According to the QAA, it is important that teachers obtain feedback from students on their perception of course provision in order to continually enhance and develop their learning experience. The aim of this article was to adopt a novel approach for this evaluation and feedback process that could ultimately be used as the first stage in developing more reliable quality assurance metrics. At one level, in terms of evaluating our own final year project process, this new approach of eliciting feedback from students has highlighted one important area that needs to be addressed. This is the set of feedback comments that students receive from their supervisor and second reader, especially for Task 1 and Task 2. In order to rectify this potential area of confusion we propose that, in future, members of staff will produce an 'agreed' set of feedback comments in order to provide the student with a clear set of guidelines and suggestions on how they can develop their project to ensure that it meets the required learning outcomes. This is very salient as Gibbs (1993) stressed the importance of feedback for helping lecturers to improve students' educational experience.

In terms of a methodological approach to generating reliable and valid content for quality assurance metrics this approach has provided some indication that it has value. Heywood (2000) expressed concerns about questionnaire feedback from students being uninformative and not relevant to the particular module that the students were asked to assess. Our approach seeks to address this concern by obtaining in-depth information from students thereby providing a mechanism for the development of metrics that produce focused and informative feedback. Having said this, there are limitations to the work reported here that will have to be addressed in future work. In order to improve the ecological validity of the findings reported here, there is a need to test this approach on a more representative sample of the student population in our department. In addition, future developments in this work should also include a comparison of our approach with a more traditional approach of categorical data collection and analysis. This is an important comparison to make as it will also allow us to assess the methodological efficacy of this approach.

Overall, it can be said that this approach offers the potential for obtaining more informative feedback from students in relation to quality assurance issues. This can only be seen as yet another positive move in the iterative process of enhancing and developing the quality and provision of teaching and supervision that we offer our undergraduate students.

FigureFigure 1

 1 Figure 1 -The matrix

Table 1 Drivers of the Final Year Project process

 1

	Project definition
	(i) Staff proposal
	(ii) Student proposal
	Tasks
	(i) Project definition
	(ii) Poster presentation
	(iii) Final project report
	Supervision
	(i) Supervisor's experience
	(ii) Supervisor's expertise
	(iii) 2nd reader's experience
	(iv) 2nd reader's expertise

Table 2 Areas of interest for the Final Year Project process

 2

	Quality of	Project	Problem	Design	Evaluation
	project	management	solving	approach	of solution
	Project definition
	(i) Staff proposal
	(ii) Student proposal
	Tasks
	(i) Project definition
	(ii) Poster presentation
	(iii) Final project report
	Supervision
	(i) Supervisor's experience
	(ii) Supervisor's expertise
	(iii) 2nd reader's experience
	(iv) 2nd reader's expertise

Table 3 Definitions of cell coding

 3

	Positive effect -opinion cell coloured	pale blue
	Negative effect -opinion cell coloured	light blue
	No causal	no colour

Table 4 Panoptic view of student and staff responses

 4

Biographical notes

S T E V E L OV E works in the School of Information Systems, Computing and Mathematics at Brunel University. His main research interests include online learning and distance learning. He also undertakes research in interaction design, the use of mobile technologies and ubiquitous computing. Address: School of Information Systems, Computing and Mathematics, Brunel University, St John's Building, Kingston Lane, Uxbridge UB8 3PH, UK.

[email: steve.love@brunel.ac.uk] RO S A S C O B L E works in the School of Information Systems, Computing and Mathematics at Brunel University. Her main research interests include investigating performance indicators in higher education and research funding in higher education (with a particular focus on the Research Assessment Exercise). Address: School of Information Systems, Computing and Mathematics, Brunel University, St John's Building, Kingston Lane, Uxbridge UB8 3PH, UK. [email: rosa.scoble@brunel.ac.uk]


Keywords: heads of department, induction, new academics

A B S T R AC T This article focuses on the induction experiences of new academic staff and the role of their head of department in this process. Respondents reflected on personal experiences and their narratives give a fine-grained account of the same event from two contrasting perspectives. We expected to find that the heads would be key figures in the induction process, but we discovered a more complex situation in which contributions were largely hidden or indirect. We encountered many contradictions as each party recalled events. Meaningful communication had been sporadic at best, and professional and personal relationships were left undeveloped. In all cases, there was little genuine understanding of the potential of induction, and this was particularly evident in the lack of personal action displayed by the new academics. Some heads had developed a deeper theoretical position on induction but few of their ideas were realized in practice. We propose that this was mainly due to the heads' lack of experience and because induction outcomes were not systematically evaluated.

Contrasting views of induction

The experiences of new academic staff and their heads of department

Introduction

Starting an academic career or moving to a new academic job can be highly complex and demanding (Trowler and Knight, 2000). Academics have to find their place in a new community while preserving their individualism and autonomy, and co-operate with unfamiliar colleagues while maintaining their competitive edge. Furthermore, they need to balance their work and personal lives at a time of major upheaval, often having arrived in a new city or country, perhaps with the added responsibility of a family. Even experienced academics require support as not all their knowledge is transferable. New starters do not fully understand what their new department or university expects of them and because it takes time to become familiar with departmental cultures, they do not know what is acceptable or permissible in their actions and work. At this time of uncertainty, new starters seek to create a positive first impression as they try to lay the foundations of a new career that they hope will be both professionally rewarding and personally enjoyable.

It seems to us that induction seeks to support an academic's entry into their new organization and enable them to become a productive and longstanding member of their department and university. We have previously argued that new staff need to understand how they fit into their institution, what their institutional responsibilities might be and to have some sense of the many different ways that academic work is practised (Harland and Staniforth, 2000;Staniforth and Harland, 2003). Trowler and Knight (1999) define induction as 'professional practices designed to facilitate the entry of new recruits to an organization and to equip them to operate effectively within it ' (1999: 178). Formal activities typically include institution-wide orientation programmes, mentoring schemes and departmental handbooks. Programmes introduce the academic to the university's operating systems and, for those new to academia, may also include some support for learning how to teach. Although there are examples of university-wide programmes that look after the new starter throughout their first year (Staniforth and Harland, 2003), most are quite short and occur soon after arrival. Induction, however, also includes an academic's socialization into a department and university 'created by the discourses and practices of the community in which one works ' (Knight and Trowler, 1999: 23), with professional learning mediated through an active dialogue between the new academic and the social system that they find themselves in.

The most important social site for induction is the department and it has been suggested that a pivotal figure for the new academic is the department's head (Bensimon et al., 2000;Knight and Trowler, 1999;Sarros et al., 1997). Heads are not only in charge of recruitment but also have responsibility for managing, supporting and developing academic work through the different phases of induction. They also provide resource support and may take a mentoring role with respect to their new colleague. However, these roles are not straightforward. Heads are in a position of power with respect to new staff and act as gatekeepers in authorizing certain academic activities, such as workloads. They are also responsible for performance evaluation and tenure decisions. Tierney (2003) points out that all academics are expected to be experts on the one hand but novices in many of their interactions with heads of department (HoDs). This is an uneasy aspect of academic life, especially when it is recognized that the head may have little initial training or expertise in management or leadership, and will usually stand down after a limited term of office. Furthermore, the nature of academic work with its primary allegiances to a discipline may not lend itself readily to direct management. Yet a collegial approach, which might seem more appropriate, can have its own limitations as this requires consultation, persuasion, negotiation, consent, and ultimately the development of consensus. Despite these difficulties, HoDs as managers have to accept some responsibility for new academics.

In the present study we focus on the individual experiences of academics starting work at a university and the experiences of their HoD in this context.

Method

This case study was carried out in a research-led university in the UK. We asked respondents to reflect on their experiences of induction. We were seeking honest and open reflection on experiences and recognized that asking HoDs and new academics to talk about each other would be both ethically and methodologically problematic, particularly if the heads were still in a position of power with respect to the new academic. To overcome this we recruited our sample of heads from those who had recently completed their 'term of office'. We intended that this study design would allow for safe reflection on recent but past events, while permitting a more open response during semi-structured interviews. Special care has been taken throughout this study to ensure the complete anonymity of all who took part.

We identified HoDs who had appointed new staff and completed their term of office within the last two years. Seven former heads fitted these criteria, and six agreed to take part in the study. They had appointed fourteen academics between them of whom nine volunteered. The departments represented were all large. One was in commerce, three in science and two in humanities. The heads had all been of professorial rank and the nine academics all lecturers. Five of the heads were male and one was female. Of the new academics who took part in the research, four were male and five female. One head had appointed three, another two, and the remaining four heads had appointed one new starter each. All new starters were early career academics but only three were new to a university position, the rest having moved from other institutions.

Interviews generated information about induction through an interaction between the interviewer and respondent, allowing respondents the freedom to talk about their experiences (Holstein and Gubrium, 2004).

Questions were focused on the broad theme of 'the role of the head of department in induction' but our priority was to allow conversations to develop that were important to the respondents. Interviews lasted for up to one hour and were recorded and transcribed. Written transcripts and copies of field notes taken at the time were analysed independently by each researcher using grounded theory (Glaser and Strauss, 1967). Pseudonyms that reflect gender are used throughout. For HoDs and new academics the key is as shown in Table 1.

Results and discussion

We report our findings within three emergent themes:

The practice of induction

This theme examines conceptions of induction, the role of mentoring and the authority of the HoD.

Personal responsibility and communication

The second theme is concerned with who has responsibility for induction and the role of communication for those concerned. It also addresses the issue of personal relationships.

New heads and new academics

The final theme frames both HoDs and new academics as professional learners and looks at the role of evaluation in the success of induction.

The practice of induction

All HoDs talked about the importance of induction but displayed limited understanding of their own potential in the process. Heads described their main induction responsibility as protecting new starters from excessive workloads and from the 'university', and ensuring that induction protocols were in existence. New academic staff saw the role of their HoD in

A C T I V E L E A R N I N G I N H I G H E R E D U C A T I O N 7(2)
188 induction as important, but only in an indirect way. Heads provided leadership and good management but were generally not expected to take a personal role in induction. Most (but not all) of the new academics were naive about their own potential in induction, especially with respect to personal responsibility and developing social relationships.

It was clear that Robert and Mike, both science HoDs, had a low value for induction and they tended to take a 'hands-off approach'. For Mike, induction was something done centrally by the human resources department. Because their new academics, Jack and Paul, had both recently completed a long academic apprenticeship through their PhD studies, they were seen as bringing both expertise and cultural capital with them. Rosser (2003) uses the term 'anticipatory socialization' to describe the interactions that PhD students have with faculty before they become full academics. However, postgraduate experiences are inadequate as full preparation for academic life and there are many aspects of academic work that PhD students are never exposed to (Austin, 2002). Both Jack and Paul had a narrow conception of academic work centred first on their research field and then on their respective research groups, and they did not identify with either their department or the university. For Paul, it had not even occurred to him that he needed induction and when he was asked about a possible role for the HoD he thought that whatever it might be 'they should not be heavily involved '. Jill's views, as another science head, differed from those of Robert and Mike. She had a stronger value for induction, although she described her role in terms of putting in systems rather than being personally involved. She talked about induction being tailored to the individual with a focus on the department rather than the university and she had introduced a department mentoring scheme for new staff. Her main job was to protect them from what was going on in the university. Jill said that she was always accessible, that staff knew that they could bring concerns to her, and that you have to get to know staff before you can induct them. However, Hannah, her new staff member, gave a very different account:

And the head of department said . . . of course we give the academic staff a very good, thorough induction. And I was sitting there thinking: I've missed it. Where was it? When was it? And I said this to 'Neil' afterwards, the head of the research group, and he said: well you got the staff handbook. (Hannah, science) Both parties obviously had different expectations of what constituted induction. Hannah was also adamant that she had never spoken to her head of department and her reflections provide an image of how anonymous an academic can be. For Jill, lack of direct contact might not have been be an issue, as her mentoring scheme was designed to provide alternative support, however, Hannah's mentor had been away on study leave. Hannah, 'out of desperation', 'bombarded' the director of her teaching programme with questions but she felt guilty doing this.

All heads mentioned that they provided mentoring schemes for new staff to support professional development. These had no summative function related to employment but they all appeared to have a priority focus on research, rather than a wider conception of academic practice, or helping the mentee settle into their new job. Kathryn, an experienced researcher, was refused a mentor by Ray: I asked for a mentor and it kind of went all round the houses, but actually the answer was no. . . . So in this department I think the notion of mentor is very much related to your research. (Kathryn, humanities) When Andrew started to work with John, he saw his appointed mentor on a daily basis to talk about research and teaching. However, this did not last or fulfil the purposes of induction:

I had a period of utter depression, to be dramatic, of two months, where no one was knocking on my door, where there was none of the camaraderie that you had at other places, because where you are all teaching undergraduates, and its shit work, and you are all in it together, and I think here there was more an individualistic culture around research. People all locked in their offices or at home working. (Andrew, humanities) June was also assigned a mentor by John and this was the only example where mentoring worked well for induction, largely due to the skill and thoughtfulness of the mentor. In our study, the only HoD mentor was Jill and only for staff in her own research area. Mentoring is recognized as an important option for new starters and Bensimon et al. (2000) suggest department chairs should consider mentoring as part of their role. However, we can see tensions between the heads' hierarchical role in management and the need to avoid unequal and possibly exploitative mentoring relationships (see Knight and Trowler, 1999). Sarros et al. (1997) talk about role ambiguity as the head is supposed to both judge and support their staff member. In contrast, Moses (1989) suggests that heads can develop a relationship in the context of appraisal because academics understand that their HoDs only have 'positional authority', which they are typically prevented from using because of academic conventions. New academics avoided causing trouble, in part because they did not fully understand the risks and consequences of their actions.

John had appointed three academics during his tenure. His view of the head's role in induction was to devolve it to senior colleagues and, like Jill, to protect the new appointees from the centre of the university. We were left somewhat unsure of what John and Jill (and other heads) were protecting their staff from, as this was never made explicit during interviews, although the threat was obviously of concern. Paradoxically, John also felt that he needed to protect new staff from being exploited by other senior academics in the department, some of whom would have been former-HoDs. In this case protection was about new academics being taken advantage of through the unfair allocation of work. John suggested that it would be difficult for a new starter to refuse a request from a more senior member of staff. He recognized that dealing with senior staff was 'an important part of the induction process'.

Personal responsibility and communication

Our research showed that academics can successfully assume responsibility and take the initiative in actively co-constructing professional and socialization experiences.

Kathryn was one of the three academics who had taken up their first post. When she started, the most helpful person was not Ray, her HoD, but the department secretary. Kathryn held the view that it was quite acceptable for her to learn by her mistakes but also felt that it had been assumed that she would know what to do. Not knowing what was expected of her or what she had to achieve had been stressful. In contrast, Ray's other staff member, Rachel, was more experienced and active in her own professional learning. She described in great detail the activities that she undertook as part of her induction. Even when she took particular issues to Ray she believed that he, at best, could only point the way while she had to find things out for herself. Her independence and initiative were evident in many of her actions:

Probably in about the first five or six weeks, we actually had quite a lot of meetings and it was exceedingly enlightening experience for me to have a head of department that was always willing to meet with me and meet with me so often in the beginning weeks. (Rachel, humanities) Like Rachel, Amanda was also proactive in her relationship with her HoD. She not only contacted John for advice during the early part of her tenure but frequently informed him about her progress during her first year. This strategy gave her a sense of how well she was doing and it helped her to develop a more informal relationship with him. These types of conversation are at the heart of academic life and in particular academic leadership (Haigh, 2005;Ramsden, 1998) yet most academics and heads in our sample hardly spoke to one another. Our analysis showed that even brief or superficial transactions could be very important to the head and the new academic. For example, there are latent aspects of communication that may serve a variety of different purposes and there were instances when a single intervention by the HoD made a huge difference to the new staff member. One of these concerned June: . . . when I had a disaster John came up and said: this is what happened to me and don't worry, it's fine. You don't have to feel terrible, we still like you. Because you know, I think that was one of the lowest points of last year . . . and I think if he hadn't done that, I'd have probably just kind of committed suicide on the spot. (June, humanities) In contrast to Amanda, June had not bothered John at all and Andrew felt there were insurmountable barriers related to John's personality and leadership style that prevented him going to see him. Although Amanda developed a positive rapport with John, she compared his leadership to that of God and Tony Blair, the British Prime Minister, which suggested that she regarded him as occupying an exalted position. It was clear that 'personality' had been a major influence on her induction experiences.

The vastly different ways in which John was regarded by his three academics showed that some aspects of induction have to be tailored to the individual and that a head requires personal knowledge and understanding to be effective in any direct induction task. Heads and new academics simply need to get to know each other.

John wished that new academics would 'make efforts to integrate' and Andrew's depression (that he attributed to no one knocking on his door for two months) might have been ameliorated if he had knocked on someone else's door during this time. Having said this, if John had checked on Andrew he might have been able to offer the kind of support needed. Our data pointed to several explanations for poor communication between heads and their new appointees. For example, HoDs were extremely busy as they tried to fulfil many different roles and this was clearly recognized by most of the new academics who tended to 'get by' without bothering their HoD.

I didn't really like to bother the head of department. Well there are still things that baffle me about the university but I have not actually asked anybody. I tend to wait and see . . . if the answer emerges slowly. (June, humanities) Although all HoDs have a role in human resource management, John suggested that 'issues of power in relationships were a key concern' and hierarchical differences made establishing any kind of relationship problematic. The heads had all appointed their new staff and for a period were responsible for the quality of their work, could dictate the nature of that work and, in theory, could make decisions that impacted on employment. At the same time, the new academics described themselves in a manner that would suggest that they were 'self-employed'. New starters recognized that success in research and teaching was their responsibility and both HoDs and new staff tended to avoid any conflict in this uneasy relationship.

New heads and new academics

Induction was not a common event in the university we studied and as a consequence all heads were relatively inexperienced in this role. In many ways they had been in a similar position to their new staff as they were learning about new roles and responsibilities. Bob said that the many problems and tasks he had to deal with during his headship were unique and as a consequence, it was difficult to learn from experience. Mike suggested there was no institutional memory and that every new person (including himself) had to find out a lot of things for themselves. Ray took this observation one step further by proposing that one of his roles should have been to 'very carefully' induct the next HoD. The system of rotating heads may have advantages for departmental governance within a collegial framework, but in terms of developing certain types of professional practices it can be quite limiting. None of the heads in our study had undergone induction as they took up their post and we believe there may be conceptual resistance to professional education at this level. A recent Australian study of academic staff perceptions of management and leadership showed that while academics linked quality human resource management to efficient academic work, they did not believe this was central to a leader's role or that a leader might have development needs in this area (Marshall et al., 2000). Perhaps a recent experience of being inducted as a learner in their new job might have provided heads with more insight into the needs of their new academic staff.

Finally, none of our heads had any way of systematically evaluating the success or otherwise of their actions or induction arrangements. Most new academics were given no regular feedback, except for Amanda, and this was due to her own agenda setting in regular meetings. In the rather anonymous departmental cultures described in this study, it was hard to understand how any evaluative judgements could be made. Knight and Trowler (2000) argue that one of the most important attributes of a department manager is their capacity and willingness to analyse cultural contexts. In any systematic evaluation of the effectiveness of induction, including the HoD's contribution to this, each department would need to establish clear criteria for measuring desired outcomes.

Conclusions and implications

The data from our study revealed numerous examples of practice situations that seemed to disadvantage the new staff member. These included the time wasted on unnecessary activities and an unwarranted investment of intellectual and emotional energy in coping with aspects of the new job. If our study is representative of induction experiences within the broader university system, then we should be very concerned. There are several implications for induction theory and practice, which we feel would be worthy of further investigation.

Our original thought was that the HoD would be a key figure in induction and despite many new reservations, we still feel that they have an important role to play. Heads who took part in our study had low expectations for their personal involvement and typically saw their responsibility as implementing systems. However, some of their strategies were founded on nothing more than clichés. The interview transcripts were replete with examples such as 'new staff are encouraged to talk to each other'. Accordingly, we suggest that heads should systematically monitor the quality and effectiveness of induction processes and outcomes.

HoDs played a key role in protecting new staff from excessive workloads and 'the university' to enable research development. Protection from 'other work' is no doubt beneficial to establishing research, but protection from the university is likely to have long-term impact on the culture of an institution as many work patterns, behaviours and expectations are set early in a new job (Boice, 1992). A real concern is that new academics will become effectively 'invisible' because they do not feel that they have an institutional identity, or any expertise outside of their research and teaching that they can contribute to the wider university community. They may live up to the idea of being 'self-employed' in a university made up of a collection of separate 'businesses' (departments).

New academics had low expectations for induction and the majority, excluding Rachel, showed little proactive behaviour. One can understand why new academics are mostly concerned with the technical problems of practice but someone with experience can guide them to look beyond immediate interests. Such guidance should encourage the new academic to accept some responsibility for using their knowledge and expertise to both learn from and shape their new community. We would also argue that because induction is highly personal and contextual, it needs to be continually re-negotiated with the individual staff member over time. The process should also include support before the academic arrives on campus and extend beyond the first year (Bensimon et al., 2000;Boice, 1992). Most HoDs in our study saw induction as a brief event that started when the academic arrived.

Our data suggest that looking after a new staff member should be the responsibility of everyone in their community. To realize this would require a new epistemology predicated on the idea of induction as a collective social practice. In our study, human contact and everyday conversation were rare for all but one of our respondents. Perhaps the potential of such a simple act just gets overlooked in our hectic work lives and we might do well to re-evaluate how we engage with others.

Because of the fragmented nature of academic work, even if induction becomes embedded in the social fabric of a department, it will still be necessary to have someone with specialist professional skills and knowledge to ensure that it happens and that it is successful. Although our former-HoDs did not make such a good job of it first time round, given appropriate professional development support in 'how to' induct staff, they could effectively become induction 'experts'. They would have insight into the theories of induction and have knowledge of the university, the department and the HoD's requirements. They would also prevent potential conflicts of interest between a new academic and the current head.

Table 1 New academics and HoDs

 1

	Heads of department	New academic staff members
	John, humanities	Amanda	June	Andrew
	Bob, commerce	Stanley
	Jill, science	Hannah
	Robert, science	Jack
	Mike, science	Paul
	Ray, humanities	Rachel	Kathryn

Biographical notes

 Zealand. [email: tony.harland@stonebow.otago.ac.nz]


Keywords: institutional audit, quality improvement, standards, subject benchmarking

A B S T R AC T The introduction of subject benchmarking led to fears of increased external intervention in the activities of universities and a more restrictive view of institutional autonomy, accompanied by an undermining of the academic profession, particularly through the perceived threat of the introduction of a national curriculum for higher education. For this study, semi-structured interviews were conducted in one old and one new university with higher education professionals in chemistry, history and quality assurance, who were asked about their perceptions of subject benchmarking and its impact. The investigation did not bear out the fears articulated at the inception of subject benchmarking. Furthermore, the investigation showed that subject benchmarking was not perceived as having the principal characteristic normally associated with a benchmarking system, in that it was not perceived as leading to improvement. The article ends with suggestions for future research.

Introduction

Subject benchmarking was introduced into UK higher education (HE) in 2000 with the publication of the first Quality Assurance Agency (QAA) subject benchmark statements: these resulted from the report of the National Committee of Inquiry into Higher Education (NCIHE) Higher Education in the Learning Society (NCIHE, 1997), commonly referred to as the Dearing Report. It was the purpose of the NCIHE, chaired by Sir Ron Dearing:

To make recommendations on how the purposes, shape, structure, size and funding of higher education, including support for students, should develop to meet the needs of the United Kingdom over the next 20 years, recognising that higher education embraces teaching, learning, scholarship and research. (NCIHE, 1997: 3) translated crudely into rigidly defined objectives or targets'. Similarly, the recently imposed exposure to external scrutiny might be seen not only as breaking into the traditional territory of professional groups (Middlehurst, 1992: 24), but also as threatening to undermine the 'ideology of professionalism' (Kogan, 1986: 152), explained (Kogan, 1986: 152) as the assumption that, in schools, for instance, 'teachers are doing a good job '. Tapper and Salter (1998: 22) raise further objections, identifying three very significant consequences of the Dearing Report: it represents another move towards the creation of a national curriculum in HE; it offers a more restrictive interpretation of university autonomy; and it represents a threat to the growing diversity of British HE. In 2002, institutional audit replaced subject review as the QAA's quality assessment mechanism for HE in England (QAA, 2002a: 3), the QAA's (2002b: 3) Handbook for Institutional Audit: England specifying that one of the things on which audit teams will particularly focus is: 'the use made of external reference points, including . . . Subject benchmark statements '. Noting that subject review aimed to improve quality (Jowett, 2005: 82), Jowett (2005: 83) argues that, 'by including reference to Subject Benchmark Statements the new [institutional audit] process has included an element of defining frames of reference for a course, but avoided prescription of course outcome'. As expounded by Alstete (1996), benchmarking is used by HE institutions in order to provide information for quality improvement by searching for best practices. Jackson (1999: xi), however, highlights the attempt to 'reinterpret what has gone before [in HE] in terms of two different notions of benchmarking, namely: developmental benchmarkingpromoting best practice, and; regulatory benchmarking -assuring quality and standards. ' In Jackson's (1999: xi) view: 'Much of the conflict and confusion in discussions about the pros and cons of benchmarking for higher education relates to the inbuilt tension between these two different purposes. ' This article summarizes some of the findings of a small-scale study (Pidcock, 2004) investigating reactions to subject benchmarking during the transitional phase from the initial promulgation of the Dearing-inspired quality agenda for HE into the period introducing the first institutional audits. The study sought to gauge reactions both at the level of individual members of staff within academic disciplines and at institutional level. In particular, the perceived necessity of subject benchmarking was investigated, as was the extent to which benchmarking was seen as leading to improvements in the quality of degrees. Views were also sought on the motivation behind the introduction of subject benchmarking and on the consequences of subject benchmarking both for university professionals and for universities themselves.

Methodology

Seeking to understand respondents' perceptions of subject benchmarking and its impact, the study adopts a qualitative methodology: 'One facet of the distinction between quantitative and qualitative research is that the former is orientated to the specific concerns of the investigator and the latter to subjects' perspectives' (Bryman, 1988: 142).

Subject benchmarking was first undertaken in chemistry, history and law (QAA, 1998a: Part IV, 10): the study considers chemistry and history in order to provide exemplars of arts-based and science-based subjects. Reactions to subject benchmarking are investigated in two English universities: one pre-1992 university and one post-1992 university. For each institution, the primary source of data is a series of semi-structured interviews conducted between August 2002 and May 2003 with academic members of staff and with representatives of central quality services. Wragg (2002: 149) notes: 'A semi-structured interview schedule tends to be the one most favoured by educational researchers as it allows respondents to express themselves at length, but offers enough shape to prevent aimless rambling'. In all, 30 academics were interviewed, corresponding to approximately 30 per cent of the full complement, together with four members of institutional quality services. The questions from the interview schedule used with academic staff, and varied only slightly with quality administrators, are shown in Figure 1: some questions included Likert scales ranging from 'very good' to 'very bad' and 'other '. All but two of the interviews were tape recorded, producing transcripts offered to the respondents for checking, verification and amendment: in the two instances in which permission to record was denied, a written account based on notes was submitted in the same way. Additional data and insights were sought from documentary sources, an approach also affording triangulation of results (Marshall and Rossman, 1999: 194). All data gathering is firmly rooted in informed consent (Fontana and Frey, 1994: 372).

After an initial indexing of the data, analytic categories were developed by studying the texts (Arksey and Knight, 1999: 163). Given the relative brevity of the sections of text resulting from fairly short interviews, the coding was descriptive and interpretative, rather than pattern (Miles and Huberman, 1994: 57). The results of the analysis are presented largely discursively, though additional texture is provided by using simple counting techniques. Based on a small population (n = 34) selected by purposive sampling (Cohen and Manion, 1994: 89), the study makes no claim to generalizability, but does aim to achieve trustworthiness.

Findings

Differences between individual departments and the two universities studied were generally not great, and the small sample size inevitably means that the analysis of discrepancies remains heavily conjectural (Pidcock, 2004: 158). It was remarkable, however, that the strongest expressions of unrest tended to come from the younger members of academic staff, even though, overall, they tended to be positive about the introduction of subject benchmarking and threshold standards. One historian seems to encapsulate the dilemma of the tension between acceptance and independence by commenting on 'the realities of placing lines . . . which we have to meet, or boundaries over which we must hurdle . . . they're quite often false', and reflecting: 'as a new member of staff, I'm probably more likely to be courses? 6. Would you say that benchmarking has had an impact on your professional life? 7. In your view, is benchmarking leading to improvements in quality? 8. Do you think that benchmarking will encourage the dissemination of best practice? 9. In your view, was it necessary to introduce benchmarking? 10. In your view, why was benchmarking introduced? 11. Has benchmarking led to changes in the way you think about your own position as an academic? 12. Has benchmarking led to changes in the way you think about the position of universities in today's society? 13. Is there anything you would like to add concerning benchmarking or threshold standards? [eg. impact of new QAA methodology?]

P I D C O C K : W H A T I S T H E I M P A C T O F S U B J E C T B E N C H M A R K I N G ?
Many thanks for agreeing to take part in this project.

Would you like to check a transcript of our conversation?

Figure 1 Format of interview schedule used with academic staff accepting of these things than the older members of staff; and I am, probably; but. . . .' This raises interesting questions about the persistence of images of academic status and institutional autonomy arising in an elite HE system, but which have perhaps been superseded by the realities of a mass system of HE (cf. Halsey, 1992). In the following presentation, the results of the study are aggregated in the interests of setting out an overall view.

Quality assurance and enhancement

Generally speaking, the academics interviewed were at least used to the concepts of quality assurance and enhancement, even if their current involvement tended to be limited to student feedback forms and peer observation of teaching. Approximately one quarter of the academics interviewed situated quality assurance and enhancement in an accountability context. Some did this explicitly:

Well, clearly there is in the air, now, the notion of accountability for taxpayers' money, and therefore: there's a lot of money being spent, then one of our key activities will be monitored and vetted. I find it a little hard to disagree with that.

Some respondents indicated an accountability relationship implicitly: 'I think we owe our students a responsibility, which is to ensure that the programmes we deliver are of good quality'.

The introduction of subject benchmarking

Close knowledge of subject benchmarking was not widespread, with exactly one third of the academics interviewed claiming no familiarity with it. Additionally, there was an almost complete lack in both institutions, at subject, departmental or institutional level, of documentation relating to subject benchmarking.

A little over half of those asked rated the introduction of subject benchmarking as good or very good, with one third remaining neutral: there were no negative ratings. The one respondent replying 'very good' saw the system as a mechanism for facilitating the recognition of degrees: 'so, the benchmarking was basically saying: Right, now, anybody who claims to be a Bachelor in chemistry must understand this'. Ratings of 'good' included reasons such as that any chemist 'needs to know the basics' or that benchmarking 'should ensure that all of the [history] courses are meeting certain standards'. Some appreciated benchmarking 'because it's about trying to define what you think a graduate in the subject is', so that the benchmarking process had given rise to beneficial reflection on the nature of the subject; this view is not shared by all, however: 'it's not exactly helped departments to reflect; it's merely provided a template that you follow'. Several respondents expressed unease about a possible 'lack of flexibility' in the benchmarking system: 'My whole fear with anything, when you write down rules, is it can also become restrictive, in their turn'. One respondent took the view that an evaluation of benchmarking 'probably depends on how people apply it'. The major concern here was that benchmarking was 'getting close to a national curriculum for higher education', which was interpreted as a bad thing. By contrast, one respondent considered benchmarking to be good as there was a need for consistency across HE in what was being taught, this corresponding to there being 'accepted paradigms within subjects'. Asked about his description of this being 'almost like in a national curriculum', this respondent replied: 'I would say that it's a good thing '. Slightly under half of those asked rated 'good' the statements of performance criteria and/or threshold standard given in the benchmark document for their subject (nobody rated this 'very good'), and just under a quarter remained neutral; almost one seventh gave negative reactions (though none was classed as 'very bad'). Positive assessments in chemistry included: 'fine' and 'a worthy effort'; and in history: 'a fair enough statement' and 'a useable set of criteria'. Negative assessments, all occurring in chemistry, referred to the vagueness or subjectivity of information listed; alternatively: 'The attainment thing I find confusing. It isn't clear to me whether what it's trying to do is define First Class, IIi, IIii, Third Class degrees'. One respondent observed of threshold standards: 'I think, really, just as an idea it's good', but was 'disappointed' with the resulting documents overall: 'So, I would say bad'.

Subject benchmarking and change

It is perhaps in the area of benchmarking and change that the most striking findings emerged. Nearly three quarters of the respondents stated that benchmarking had not led academics to change their approach to their courses or institutions to change their approach to quality. A small number felt unable to reply; for the equally small number who answered 'yes', any changes had been mainly presentational. For half of the academics, benchmarking had had no impact on their professional lives: paperwork and administration formed the major reasons for answering 'yes' to this question (just over one third), one individual, for instance, referring to 'this deluge of stuff'. Among the few responses rated 'other', one academic reflected: 'My feeling is that, by creating a kind of, a standard approach, that it can only benefit institutions like ours, which maybe don't have a traditional reputation, being a new university'; this enhanced reputation then 'reflects on me, doesn't it?'

Subject benchmarking and improvement

The QAA's (QAA, 2002b: 1) Handbook for Institutional Audit: England states: 'The mission of the Agency is to promote public confidence that the quality of provision and standards of awards in higher education are being safeguarded and enhanced'. For almost half of the respondents, however, benchmarking was not leading to quality improvements; as one academic put it: 'the things that actually get altered are very much exterior gloss rather than genuine content or teaching methods'. Almost as many fell into the category 'other', generally because they did not know. Alongside a small number of somewhat speculative answers, of the kind: 'I would hope that it is', just two respondents identified a contribution to quality improvement. One of these highlighted the transparency of information provided for students and a rethinking of modules, programmes and assessment strategies, whilst nevertheless recognizing a possible alternative view 'that it's a paper exercise, and that you're ticking boxes in order to meet the demands of an outside agency, and then the internal agencies that respond to that. I think people have responded in different ways. ' The other positive response combined both of these possibilities: Indirectly, yes; because I think benchmarking is part of a whole process of what you might call a regulation culture: and whereas that can be rather stifling and bureaucratic and rigid, it does encourage people to reflect on what they do, and I think that sort of reflection is healthy. They will grumble about it and will often fill the forms in just wanting to get it off their back, but I think underneath they do reflect on what they do. And I think that's helpful.

In answer to the question as to whether subject benchmarking would encourage the dissemination of best practice (cf. QAA, 2002b: 12), responses were almost exactly equally distributed across 'yes', 'no' and 'other'. Those answering 'yes' did not do so, generally, with any strong endorsement of this aspect of benchmarking. Those who made explicit what they meant by 'best practice' mostly referred to issues concerned with teaching rather than to considerations of standards, course design or curricular content. Some thought that the dissemination would be achieved by the published benchmarking statements themselves. Negative responses tended to be rather firmer: 'I don't think it was ever intended to do that job'; 'I don't see how, because I think it's so lacking in detail.' For the dissemination of best practice, several respondents in this group referred to the Learning and Teaching Support Network and to conferences. In the category 'other', respondents tended to be unsure of the contribution of benchmarking or very guarded, offering comments such as 'I would hope so'.

Reasons for the introduction of subject benchmarking

Only one third of the respondents thought it had been necessary to introduce subject benchmarking. Some of those answering 'yes' referred to the great variety of institutions in HE, and one respondent formulated a different view encountered several times: 'If this process means that there is a minimum standard and that there is a set of aims that we're going about to achieve, then that's no bad thing, as far as I'm concerned.' A telling negative response came from a quality administrator: 'really, the position afterwards doesn't look very different from the position beforehand. It's difficult to imagine why one would have felt it was necessary, given that it doesn't seem to have had much impact.' Respondents (n = 34, though 4/34 offered no view) were then asked for their views as to why subject benchmarking had been introduced: the most frequently given explanations are shown in Figure 2. The figures indicate the number of times an item was mentioned.

As in Jowett's ( 2005) study of Subject Review, political pressure emerged as the predominant perceived reason for the introduction of subject benchmarking, most forcefully in the response: 'Well, I mean, to be brutally honest, Steve: benchmarks are about government watchdogging and not about academic standards. Aren't they? Let's be brutally frank! This is a political game, not an academic game'. Figure 2 shows eight references to standards (direct or indirect), and five to quality, though the distinction between quality and standards was far from clear. Indeed, one quality administrator stated: 'Definitely for the quality agenda. We're looking at standards, here.'

The perceived impact of subject benchmarking on individuals and institutions

Asked if benchmarking had led to changes in the way they thought about their own position as academics, over two thirds of academics answered 'no', and only one fifth 'yes', generally in terms of diminished autonomy. No clear pattern emerged among the quality administrators, who were asked about changes in the nature of the academic profession. Asked if benchmarking had led to changes in the way they thought about the position of universities in today's society, two thirds of the respondents answered 'no'. The small number answering 'yes' tended to refer to reduced independence or undermined professionalism: 'makes it more like a school than an institution of higher education'.

At the end of the interview, some respondents did take up the invitation to make additional comments, not infrequently returning, directly or indirectly, to a point made earlier; others raising issues not germane to the study. Respondents were then asked if they thought that the change in QAA methodology would make any difference to the status of subject benchmarking. One third of the respondents provided no data answering this question clearly, and a further third did not know. Two respondents thought that the advent of Institutional Audit would produce no change in the status of subject benchmarking. Five thought that benchmarking would become more important, though one of those five was rather muted: 'It may do. In some ways, one hopes that the benchmarking statements become more central '. In the view of one quality administrator, the new QAA methodology would give benchmarks more significance 'because they are coming as a package, now, rather than individually . . . they are being seen as a whole package alongside other quality and standards initiatives. They are being foregrounded more'. A further five respondents thought that benchmarking would become less important. One chemist, for instance, took a view diametrically opposed to that just quoted, since benchmarking was now 'one of a raft of things . . . that you're supposed to have taken into account when designing a course. . . . And therefore their impact as the major influence on curriculum, I think has been diluted, as a consequence of that' (emphasis in original). The strongest reaction to this question came from a quality administrator, who immediately commented on the changes in QAA methodology: 'Well, of course, they greatly reduce [the status of subject benchmarks]'.

Discussion

The interviews revealed a broad awareness and acceptance of quality assurance principles. Underlying this was often a commitment to public accountability and so an acknowledgement of the need to maintain high standards, though there were frequent instances of academics objecting to the excessive bureaucratic burden of quality assurance (cf. Morley, 2003: 55). However, there was very little to indicate the kind of outlook adopted by Trow (1994: 15): 'bureaucratic institutions and their mechanisms are the alternative to a relationship of trust between the state and universities'. Nor did the idea of quality assurance seem to undermine perceptions of professionalism, also described by Trow (1994: 27): 'almost everything in a university depends on the inner motivations of teachers'.

Whilst just over half of the respondents gave a positive evaluation of the introduction of benchmarking, almost nobody indicated having changed anything other than presentation as a result of subject benchmarking. One senior chemist, for instance, thought that benchmarking was 'probably good' at the national level, but stated: 'At [name of university], we easily met all the levels, so it is of relatively little value'. This lack of impact suggests the underlying interpretation that benchmarking was viewed as good essentially because of its potential when applied elsewhere. The same senior chemist observed: 'So many places can call themselves universities that it may be of some value to define minimum standards'.

Reactions of this kind need to be assessed in context. Both universities studied are well established and well regarded, and it is possible that attitudes in less mature institutions may differ. At the same time, however, there is evidence that, although HE is changing: 'Things are different but standards are not worse' (Harvey, 2002: 21). Support for this view emerges from the QAA's (QAA, 2003b: 4) own report on Subject Review 1993-2001, which records that, after 2904 Subject Review reports, 'HE provision in England and Northern Ireland was generally found to be of very high quality: the overwhelming majority (99 per cent) of subject review visits resulted in the provision being approved in the first instance'.

The QAA (2003b: 4) overview report further states: 'A new era of quality assurance, including a stronger emphasis on the maintenance and enhancement of academic standards, has already begun'. Part of the associated quality assurance framework is benchmarking, one of a number of tools being adopted from industry and business by higher education (Weeks, 2000: 59). Price (1996: 3) records: 'By common consent [benchmarking] is usually acknowledged as rooted in Japanese manufacturing philosophies of continually seeking inspiration and example from other organizations. Its adoption into western management practice is normally credited to Rank Xerox during the mid 1980s'. Benchmarking as understood by Xerox is: 'A continuous, systematic process of evaluating companies recognized as industry leaders, to determine business and work processes that represent "Best Practices" and establish rational performance goals' (Coopers &Lybrand andCBI, 1993: 4). For Camp (1989: 12): 'Benchmarking is the search for best practices that lead to superior performance '. Spendolini (1992: 15) emphasizes: 'Perhaps the simplest one-phrase response to the definition question would include some reference to "learning from others" '. In Alstete's (1996: 20) view: 'fundamentally, benchmarking involves analyzing performance, practices, and processes, within and between organizations and industries, to obtain information for selfimprovement' (emphases removed). A review of the literature (Pidcock, 2004) shows that benchmarking is normally conceived as being a dynamic process of enhancement and continuous development. Indeed, this widespread view prompted objections to the QAA's benchmarking proposals: 'It was . . . pointed out that in normal usage "benchmarking" was a term used to denote a means for continuous improvement of quality and standards; and there was a danger that the proposals . . . would lead to stasis' (QAA, 1998b: 18). In its own 'response and proposals' (QAA, 1998b: 19) in answer to such specific issues, this misgiving is not addressed. Significantly, in the study, almost nobody thought that benchmarking was leading to quality improvements and only one third of the respondents saw a link between benchmarking and the dissemination of best practice. On the basis of the fieldwork, therefore, it is not easy to assess the constructive contribution of subject benchmarking to the QAA's 'new era of quality assurance'.

Nevertheless, some respondents took a view similar to that articulated by a senior academic: '[The benchmark statement] would help people who've perhaps not taught a Bachelor's degree before: if they planned to start teaching chemistry Bachelor degrees, they must be careful not to leave out things which are contained within there'. By contrast, this respondent continued: 'But a university that has a reasonable reputation for the subject will, you know, it's almost inconceivable that they're not already doing everything that's in the [benchmark]'. A related reaction is recorded by Hargreaves and Christou (2002: 188): 'To comply with the QAA regulatory framework at Sheffield Hallam University, explicit use of the benchmark statements is expected by course teams in the design and review of courses'. Additionally, discipline audit trails have now been dropped from Institutional Audit (Baty, 2005: 7). Considerations of these kinds invite reflection as to whether the subject benchmark statements, rather than being a separate element of the standards infrastructure (Wright and Williams, 2001: 11), might not be better located as a sub-set of the QAA's (2000) Code of Practice for the Assurance of Academic Quality and Standards in Higher Education. Section 7: Programme Approval,Monitoring and Review. According to the QAA's (2003a: 10) Institutional Audit: England -Key Features of the First Audits:

In general . . . the [audit] reports suggest that the eight institutions have responded appropriately to the introduction of the FEHQ and subject benchmark statements. . . . All of the institutions are found to have given consideration to the purpose of these external reference points and undertaken some initial work to address them.

Unfortunately, there is no explanation or analysis of these comments. In the benchmarking study, neither institution visited was able, at the time of the fieldwork, to provide any substantial documentation relating to subject benchmarking. It did emerge, however, that the new university studied, 'as part of its internal review [of] quality management procedures, became very keen that people took account of benchmarking'. Indeed, a course document within one of its departments had adopted almost verbatim significant sections of the corresponding benchmark statement. One of the departments in the old university, anticipating a QAA visit,'used the benchmarking as a Bible [respondent laughs]'.

Almost nobody in the study identified subject benchmarking with a national curriculum for HE, though those academics who established the link all did so with strong disapproval: 'That's my only concern about it: that it's the first step on a dreadful route to national curriculum'.

Overall, reactions to subject benchmarking tended to be somewhat subdued. As one academic put it: 'you do find yourself trying to jump through these little hoops that are set, just to get the documentation in order, which, in a way, detracts from the teaching and detracts from other aspects of my job, as well'.

Additional insights into the impact of benchmarking may emerge from the report in the series Outcomes from Institutional Audit announced by the QAA (2005: 3): 'A later paper in the Outcomes . . . series will address the use of subject benchmark statements . . . in institutions' internal quality and academic standards arrangements'. Similarly, a more manifest contribution to the identification and dissemination of best practice may result from the review of the published benchmark statements programmed to start in April 2005 (QAA, 2004: 12). It is clear, however, that such matters continue to arouse strong feelings, as when Furedi (2005: 54) writes:

In the case of the PGCHE [the element of socialisation involved in all forms of accreditation] is almost exclusively about socialising academics into the ethos of the audit culture that dominates the campus. It is about indoctrinating new lecturers into values of a conformist orientation towards teaching. Academics learn to talk the lingo of best practice, benchmarking and summative assessment.

A challenge confronting QAA benchmarking is the UK's involvement in the Bologna Process (Williams, 2004: 10), especially in view of the 'increasing interest in recent years to apply benchmarking as a tool for quality improvements within higher education in Europe' (Hämäläinen et al., 2003: 5). The QAA's (1999: para. 6.2) own advisory group on multidisciplinary and modular programmes states: 'The Advisory Group . . . recommends that the standards of higher education in the UK could be better assured through the adoption of an enhancement rather than referent benchmarking process and philosophy. ' However, Cuthbert (2002: 39) records that this examination of multi-subject courses was abandoned because 'the QAA could not accommodate a fundamental challenge to its subject-based philosophy'. Consequently, not only does the problem of multidisciplinary and modular programmes remain unresolved, but also, at a European Network for Quality Assurance in Higher Education workshop on 'Benchmarking in the Improvement of Higher Education', the QAA presented the task of subject benchmarking groups as, firstly: 'to create reference points, which help define the nature of awards in the subject' (Crozier, in Hämäläinen et al., 2003: 23); and, secondly: 'to formulate or articulate the minimum requirements or expectations of performance for an award in the subject' (Crozier, in Hämäläinen et al., 2003: 24). However, the Burslem Report (HEFCE, 2005: 12), in seeking to improve the Quality Assurance Framework, emphasizes '[t]he wish to develop quality enhancement aspects'. It remains to be seen whether Neave (2005: 19) is correct to describe the Bologna Process as generating a 'field of strife . . . in such issues as modes of accreditation, the criteria of quality and its benchmarking '. In Scotland, of course, the QAA (2003c) has adopted a model of enhancement-led institutional review (ELIR), in which enhancement is defined as 'taking deliberate steps to bring about continuous improvement in the effectiveness of the learning experience of students' (QAA, 2003c: 5) and in which subject benchmarking information is listed as one of the common reference points most likely to be included in ELIR (QAA, 2003c: 5). Yet this approach is not without its problems: 'There is a certain amount of evidence that some of those involved in the institutions in this first round of ELIR visits were unsure as to where the emphasis lay or should lie as between assurance and enhancement' (Swinfen, 2004: 9). In the UK, the particular interest HE professionals show in enhancement activity is very clear (Lawton, 2003: 19;2004: 15). Unfortunately, the nature of improvement in HE remains an unresolved issue (Houston and Studman, 2001: 475). Williams (2002: 1) points out that 'neither "assurance" nor "enhancement" carries a simple definition '. Indeed, Morley (2003: 13) suggests that 'the concept of improvement in quality assurance can often seem under-theorized in the context of sophisticated analyses in the academy of measurement techniques'. Rather than settling for the comfortable vagueness of continuous improvement simply being 'always trying to do things better' (Williams, 2002: 1), there is a need for a differentiated conceptualization of enhancement if techniques are to be devised for the investigation and evaluation of improvement. Future research could profitably analyse the perceptions and insights of HE professionals in order to generate a taxonomy and shared vocabulary of enhancement which could then, in turn, inform the daily activities of those same professionals and foster further discussion and refinement. In a wider context, the development of theoretical models of quality enhancement may prove helpful in 'the bitter but unavowed struggle between individual higher education systems in Western Europe' (Neave, 2005: 19).

Moreover, a rich field for research is the student perspective. Haller (1995: 26) demonstrates that, 'students' evaluated importance of [quality] criteria, as well as their perceptions, changed during the period of the service process'. Material constraints on the benchmarking study discussed here made it impossible to include students: indeed, Hill et al. (2003: 15) indicate that student perceptions of quality generally are under-researched. At the time of the fieldwork described, however, the UK National Union of Students (NUS) did not have a policy on subject benchmarking (NUS: private communication). With the advent of top-up fees and the proposal in the White Paper The Future of Higher Education (DfES, 2003: para. 4.33) to award the university title on the basis of taught degree awarding powers, student perceptions of subject benchmarking become an even more significant area for investigation. Margham and Jackson (1999: 103) argue that quality and standards 'are often used interchangeably to refer to the value and reputation of an institution's programmes of study '. If, however, quality and standards are considered to enjoy a symbiotic relationship (Pidcock, 2004), each having a beneficial impact on the other, then the distillation of student views on the strengths and weaknesses of subject benchmarking could assist all involved to make a valuable contribution to the enrichment and enhancement of the student experience.

PFigure 2

 2 Figure 2 Perceived causes of the introduction of subject benchmarking

P I D C O C K : W H A T I S T H E I M P A C T O F S U B J E C T B E N C H M A R K I N G ?

Acknowledgements

I gratefully acknowledge help and advice given in the development of this article by Dr Hugh Busher of the Graduate School of Education at the University of Leicester.