Tools for Formative Evaluation: Gathering the Information Necessary for Program Improvement.
New Extension educators experience a steep learning curve when attempting to develop effective Extension programs. Formative evaluation is helpful to new, and experienced, Extension educators in determining the changes necessary for making programs more effective. Formative evaluation is an essential part of program evaluation. However, its use has been overlooked by Extension educators due to overemphasis on outcomes evaluation for accountability. Extension educators must develop evaluation tools with questions appropriate for determining program weaknesses and strengths and identifying changes necessary for improvement. This article describes how to develop formative evaluation questions for program improvement. Introduction Program accountability and improvement are two major functions of Extension evaluation. Program improvement is considered one of the most important tasks of evaluation. However, this function of evaluation has been overlooked by some Extension educators due to their overemphasis on accountability. This situation can be attributed to the increased demand for accountability in Extension services (Hachfeld, Bau, Holcomb, Craig, 2013) The program improvement function of evaluation is called formative evaluation (Scriven, 1991) The intent of formative evaluation is to "help form or shape the program to perform better" (Rossi, Lipsey, Freeman, 2004, p. 34) As the name indicates, formative evaluation makes Extension educators more informed about the changes needed to improve their programs. Formative evaluation helps Extension educators determine what worked well and what went wrong in their educational programs. Also, formative evaluation helps Extension educators determine the changes K. S. U. Jayaratne Associate Professor and State Leader for Program Evaluation Department of Agricultural and Extension Education North Carolina State University Raleigh, North Carolina jay_jayaratne@ncsu.e du needed to further improve their programs. Cooperative Extension services in the United States are experiencing more resource limitations (Monroe, McDonell, Hermansen Baez, Long, Zipperer, 2007) and are being forced to optimize their program effectiveness. Under current resource limitations, there is no room for programs to fail. For Extension educators operating under this demanding situation, it is critical to determine all necessary improvements at the pilot stage of programs and to make necessary changes before the next round of programming. This approach prevents potential program failures and increases the cost effectiveness of Extension. Extension educators are having to develop more effective programs than ever before and should be prepared to better their programs continuously by determining needed improvements. This continuous improvement process contributes to the effectiveness of Extension programs and maximizes program outcomes and impacts. Also, this process contributes to improving the professional outlook of Extension educators and to achieving client satisfaction. The latter factor is a desirable condition for gaining public support for Extension programs. Normally, new Extension educators experience a steep learning curve when developing Extension programming. They have to learn educational programming within a very short period of time and without failing. New Extension educators can accomplish this task if they pay due attention to formative evaluation and use formative evaluation as a helpful tool in determining the changes needed to further improve their educational programs. The purpose of this article is to discuss how to develop a formative evaluation tool useful for gathering the information needed for program improvement. Information Needed for Formative Evaluation All the information needed for formative evaluation can be categorized into three broad groups: (a) negative factors associated with the program, (b) positive factors associated with the program, and (c) the changes needed for further improvement of the program. Gathering Information About Negative Factors If an Extension educator is aware of the weaknesses, problems, and barriers associated with a program, he or she can find alternatives to overcome those issues in the next round of programming. For this reason, it is important to ask questions to identify weaknesses, problems, and barriers associated with a program. The following questions can be posed to program participants to gather this information: What do you consider to be the most significant weakness of the program? Did you meet your learning expectations from this program? (yes or no) If you did not meet your learning expectations, why not? Tools of the Trade Tools for Formative Evaluation: Gathering the Information Necessary for Program Improvement JOE 54(1) (c)2016 Extension Journal Inc. 1 Gathering Information About Positive Factors If an Extension educator is aware of the strengths of an educational program, he or she can capitalize on those strengths when building future programs. The following questions are helpful for gathering information about program strengths: What do you consider to be the most significant strength of the program? Which part of the program did you like the most? Gathering Information for Making Needed Changes Program participants are the best group from whom to solicit suggestions for further improvement of any educational program. Adult audiences, especially, are capable of identifying what needs to be done to improve a program. For this reason, Extension educators can gather useful information from participants for further improvement of their programs. The following questions can be used to gather such information: What do you suggest could be done to improve this program? What would be the best way to spread the word about this program to potential participants in future? In addition to the aforementioned questions, provide participants with scaled items for assessing their satisfaction with the program (see Figure 1) Figure 1. Scaled Items for Assessing Participants' Satisfaction with Program Please circle the appropriate number for your level of response. How satisfied are you with Not Satisfie d Somewhat Satisfied Satisfied Very Satisfie d the relevance of information to your needs? 1 2 3 4 the presentation quality of the instructor(s) 1 2 3 4 the subject matter knowledge of the instructor(s) 1 2 3 4 the training facilities? 1 2 3 4 the overall quality of the workshop? 1 2 3 4 Tools of the Trade Tools for Formative Evaluation: Gathering the Information Necessary for Program Improvement JOE 54(1) (c)2016 Extension Journal Inc. 2 Practical Application To design more useful evaluation tools, Extension educators should include formative evaluation questions in addition to outcome evaluation questions on evaluation instruments. In assessing Extension programs, questions such as those suggested above can be used to gather formative information for making changes necessary for program improvement. Such questions help Extension educators determine the strengths and weaknesses associated with their programs. When Extension educators are aware of the strengths of their programs, they can capitalize on those strengths to overcome weaknesses, thereby building better programs for the future. Additionally, when Extension educators have information that helps them identify changes needed for program improvement, they can make informed decisions about necessary program revisions. References Hachfeld, G. A. Bau, D. B. Holcomb, C. R. Craig, J. W. (2013) Multiple year Extension program outcomes and impacts through evaluation. Journal of Extension [Online] 51(1) Article 1FEA2. Available at: http:/www.joe.org/joe/2013february/a2.php Monroe, M. C. McDonell, L. Hermansen Baez, L. A. Long, A. J. Zipperer, W. (2007) Building successful partnerships for technology transfer. Journal of Extension [Online] 45(3) Article 3TOT6. Available at: http:/www.joe.org/joe/2007june/tt6.php Rossi, P. H. Lipsey, M. W. Freeman, H. E. (2004) Evaluation: A systematic approach (7th ed. p.34. Thousand Oaks, CA: Sage Publications. Scriven, M. (1991) Evaluation thesaurus (4th ed. Newbury Park, CA: Sage Publications. Copyright (c) by Extension Journal, Inc. ISSN 1077 5315. Articles appearing in the Journal become the property of the Journal. Single copies of articles may be reproduced in electronic or print form for use in educational or training activities. Inclusion of articles in other publications, electronic sources, or systematic large scale distribution may be done only with prior electronic or written permission of the Journal Editorial Office, joe ed@joe.org. If you have difficulties viewing or printing this page, please contact JOE Technical Support Tools of the Trade Tools for Formative Evaluation: Gathering the Information Necessary for Program Improvement JOE 54(1) (c)2016 Extension Journal Inc. 3