Developing a Practical Framework for Information Literacy Program Evaluation
Purpose. The Purdue University Libraries, like many academic libraries, face increased expectations for demonstrating their value and impact. This has not only led to an expectation of the increased use of metrics to demonstrate impact, but also a more fundamental imperative that libraries more clearly articulate their contributions to educational and research outcomes of their campus communities (value) At Purdue, the provost implemented a new program review process in July 2015, while the libraries were simultaneously going through the process of developing a new mission statement for its information literacy program. This statement was developed through a broad collaborative process within the libraries and with external campus stakeholders. These two developments led the libraries to launch a project to advance an outcomes based, mission centric framework for evaluating its information literacy programming that can be sustained over time. The project to develop this framework was predicated on being able to answer the following question, derived from the program mission statement: "Does the Purdue University Libraries' information literacy programming empower diverse learners to use information to learn in transformative ways; lead to the discovery of new knowledge; and foster academic, personal and professional success?" This question not only needs to be answered, but needs to be answered on an ongoing basis to communicate the programming impact to external stakeholders. To be effective, sustainable, and practical, it also needs to be uncomplicated and integrated into regular workflows. Methods. The methods for developing this framework consist of four steps: (1) focus groups with librarians to collaborate on gaining a more comprehensive understanding of existing assessment practices, as well as their perceptions of challenges and opportunities in assessing information literacy programs, (2) analysis of focus group findings, characterizing current assessment practices and identifying where outcomes based assessment is already occurring, (3) a gap analysis, comparing focus group findings to the information literacy mission statement, and (4) development of recommendations with measures/indicators to address gaps and develop a comprehensive framework for program evaluation. This paper reports on the first three steps, concluding with suggestions for further development of the evolving framework. Findings. The assessment practices identified in the analysis of the focus group discussions suggested that librarians assessed how students critically used information to learn more than the other dimensions of the Purdue Libraries' information literacy mission statement: research based programming, empowerment of diverse learners, enabling the creation of new knowledge, and fostering academic, personal, and professional success. The findings suggest next steps in the development of the framework, including: (1) developing guidelines for collecting assessment data gathered by librarians for use in programmatic assessment, (2) determining assessment strategies for the libraries and allocating resources, and (3) providing professional development and incentives for librarians to create assessment strategies related to all aspects the mission statement. Practical Implications/Value. We expect that the results of this project will contribute to the body of knowledge in library assessment by presenting a framework for the outcomes based evaluation of information literacy program evaluation that is based on a strategic perspective on the program, but 2016 Library Assessment Conference 606 that also builds upon existing practices and capacity within the organization. Introduction The Purdue University Libraries, like many academic libraries, face increased expectations for demonstrating their value and impact. Because of external drivers related to value, the campus, and thus the libraries, focuses on the undergraduate learning experience and demonstrating the value of these efforts. This imperative is leading the libraries to more clearly articulate their contributions to campus educational and research outcomes, transform their approach to making these contributions, and use evidence to demonstrate impact to stakeholders. Using an approach that recognizes the importance of instructional and research engagement in the broader context of the campus community, the Purdue Libraries moved from ad hoc approaches to instruction to increasing integration into the formalized structures of instruction on campus. This reflects a move away from "one shot" instruction to deeper engagement in courses and curricula and systematic involvement in campus curricular transformation. At the same time, articulation of the libraries' instructional program has become increasingly structured. This resulted in the development of a new information literacy mission statement, intended to better articulate the goals and potential impact of libraries instruction to the campus: Purdue University Libraries' researchbased information literacy programming empowers Purdue's diverse communities of learners to use information critically to learn and to create new knowledge, fostering academic, personal, and professional success. (https:/www.lib.purdue.edu /infolit/mission) Although aligned with the libraries' strategic plan, information literacy programming was largely a bottom up process of building structure around existing, successful practices that were the outgrowth of a focus on engagement and partnerships. This led to an ad hoc approach to information literacy assessment, inconsistently carried out at the individual student level with little programmatic evaluation beyond the requirements of external reporting such as ARL Statistics. It was clear that improving program evaluation and assessment would be necessary to continue to develop and improve the libraries' information literacy efforts. At the same time, the provost implemented a new program review process. These developments led the libraries to launch a project to develop an outcomes based, mission centric framework for evaluating its information literacy programing that can be sustained over time. The project to develop this framework was predicated on this question, derived from the program mission statement: Does the Purdue University Libraries' information literacy programming empower diverse learners to use information to learn in transformative ways; lead to the discovery of new knowledge; and foster academic, personal and professional success? This question needed to be answered for two reasons: (1) so that the libraries would continuously evaluate and communicate their effectiveness and programmatic impact to external stakeholders; and (2) so that the libraries' faculty and staff could improve their practices through evidence. A framework approach was chosen so the practices could be coordinated from the top down to meet organizational needs while still honoring existing practices and providing space for innovation and customized practice. To use a music metaphor, we were not attempting to compose a symphony but were rather trying to develop jazz charts a structure that coordinates the ensemble but still showcases creativity and spontaneity. The intention of the project was to develop and implement a framework for programmatic evaluation that met the internal and external evaluation needs of the organization as a whole. At the same time, it allowed individual practitioners to develop innovations that could be piloted and eventually integrated into the larger framework, or simply used to improve individual practice. Literature Review Academic libraries have a long history of collecting data for assessment. Input data such as gate counts and circulation statistics sufficed for reporting purposes in the past. But there is widespread acknowledgement that input data do not capture the breadth or depth of how libraries influence students and faculty. Librarians, individual libraries, and professional associations are developing new assessment strategies, but there is no agreement on a method or model for assessing academic libraries. Koltay postulated that "impact assessment is a field in its infancy for research libraries."1 Professional Bracke, Maybee, and Weiner 607 associations have programs to support libraries in their efforts. The Association of Research Libraries Statistics and Assessment program (http:/www.arl .org/focus areas/statistics assessment) the American Library Association Libraries Matter: Impact Research (http:/www.ala.org/research /librariesmatter/ the Association of College and Research Libraries VALUE of Academic Libraries initiative (http:/www.acrl.ala.org/value/ and much of the focus of SLA (https:/www.sla.org/career center/helping organizations succeed/ http: www.sla.org/wp content/uploads/2014/03/FT SLA Report.pdf reflect this. Demonstrating impact should be communicated to constituencies, but a study of members of the Association of Research Libraries found that few reported assessment data on their websites.2 Accountability to the institutional administration is one purpose of programmatic assessment. But there are other benefits to assessment, such as program improvement and the development of teaching theory.3 Individual librarians assess information literacy and student learning at the individual or class level. However, only a few libraries addressed the information literacy program as the unit of measure or evaluated the impact of the information literacy program in the literature. The institutional mission should be the foundation for library assessment.4 According to ACRL, a mission statement and assessment that includes program performance are best practices for information literacy programs. Program evaluation should include measurement of the progress of meeting the program's goals and objectives; and assessment of integration with