Most work sites engage in some form of health promotion programming designed to improve worker health and reduce health care costs. Although these programs have typically been delivered through combinations of seminars and print materials, workplace health promotion programs are increasingly being delivered through the Internet. The purpose of this research was to evaluate the effectiveness of a Web-based multimedia health promotion program for the workplace, designed to improve dietary practices, reduce stress, and increase physical activity.t tests. Using a randomized controlled trial design with pretest-posttest comparisons within each group, 419 employees of a human resources company were randomly assigned to the Web-based condition or to a condition that provided print materials on the same topics. All subjects were assessed at pretest and posttest through an online questionnaire containing multiple measures of health behavior and attitudes. The test period was 3 months. Questionnaire data were analyzed mainly by analysis of covariance and 1,415 = 7.104, P = .008) and Dietary Stage of Change , but there were no significant group differences on the five other dietary measures. Both groups also showed improvement from pretest to posttest on most dietary measures, as indicated by significant t tests. Within the Web-based group, dosage analyses showed significant effects of the number of times the subject accessed the program on measures of Dietary Self-Efficacy , Attitudes Toward a Healthful Diet , and Dietary Stage of Change . No significant differences were found between the two groups on measures of stress or physical activity, although t tests of pretest-posttest changes indicated that both groups improved on several of these measures. The Web-based group gave significantly higher ratings to the program materials than the print group on all health topics and in their overall evaluation . Retention rates were good for both groups—85% for the Web-based group and 87% for the print group. Subjects using the Web-based program performed significantly better than the print group on Attitudes Toward a Healthful Diet (F The Web-based program was more effective than print materials in producing improvements in the areas of diet and nutrition but was not more effective in reducing stress or increasing physical activity. The higher ratings given to the Web-based program suggest that workers preferred it to the print materials. Both groups showed numerous pretest-posttest improvements in all health topics, although such improvements might be attributable in part to a Hawthorne effect. Results suggest that a multimedia Web-based program can be a promising means of delivering health promotion material to the workforce, particularly in the area of diet and nutrition. As shown in the first three rows of Subjects also rated the extent which the materials (Web-based or print) were “engaging and appealing,” and “easy to access and understand.” The last three rows of The direct comparisons of the Web group and print group showed that the Web-based program performed significantly better than the print materials on measures of Attitudes Toward a Healthful Diet and Dietary Stage of Change, and the dosage analysis indicated significant effects of the Web-based nutrition/weight management segment on Dietary Self-Efficacy as well. In addition, the Web-based program performed significantly better than the print materials on virtually all dimensions of program evaluation. However, on outcome measures of stress and physical activity, there were no significant differences between those who received the Web-based program and those who received the print materials.Of particular interest was the finding that both the Web group and the print group showed significant improvements on multiple measures in each topic area  from pretest to posttest, suggesting that either (1) both interventions were effective in improving health practices and attitudes or (2) the improvements were the spurious result of some other force or event, perhaps an overall social desirability effect, a Hawthorne effect, or a function of most subjects in both groups coming to the field test with substantial motivations to improve their health. Because it was originally thought that the print materials would form a relatively weak intervention compared to the Web program, a no-treatment control was not included in the design; thus, it is difficult to determine which interpretation is the correct one. It is possible that the improvements occurred as a result of a secular or environmental change affecting both groups, such as an easing of the workload across the workforce (unlikely) or the implementation of another health promotion program in all three offices (this was not the case). It is also possible that both groups exhibited a quasi-placebo effect: wanting to improve their health practices and given some impetus to do so (the surveys and programs), they reported improvements across most of the measures, but the improvements were not a function of the interventions. However, the fact that the improvements occurred consistently in the predicted direction across the majority of these measures, including reductions in reported weight from pretest to posttest, argues against this interpretation.Perhaps the most plausible explanation is that most of the participants in both groups entered the study with considerable motivation to make improvements in their health practices and attitudes, and both the Web-based program and the print materials contained sufficient information and guidance to help participants make the desired improvements. Indeed, for both groups, the pretest motivation items for improving dietary practices and for increasing activity exceeded 4.0 on a 5-point scale, indicating that both groups were indeed motivated to make positive changes in diet and activity upon entering the field test. Although the print group was originally conceived as a control group, it seems likely that the scope and quality of the print materials—five colorful booklets that included tracking forms and logs, totaling 73 pages—were such that the print materials were effective interventions in themselves and constituted a substantial rival to the Web program. As noted earlier, the effectiveness of print materials for changing health behaviors and outcomes is supported in the literature -19. In cOn the other hand, the print materials offered other advantages: users had all the material in hand from the start, with no need to access the Web or navigate the program, and they could carry one or more of the booklets virtually anywhere they went. The apparent efficacy of the print materials suggests that increasing the number of printable downloads in the Web program would improve its effectiveness.The findings from the dosage analysis further support the view that the effects of the Web program were real and not a result of social desirability or other nonprogram effects, at least in the dietary topic area. On multiple measures, the Web program effects were a positive linear function of the number of times users accessed the program: as exposure to the program increased, so did the size of the improvements. As noted in the presentation by Cook et al (see Multimedia Appendix 4), the frequency with which users accessed the Web-based program was lower than anticipated and was especially low for the modules on stress and physical activity, a finding which suggests, along with the dosage analysis, that frequency of access might be related to program effects. Although the causal direction is unclear, taken together these findings suggest that procedures designed to increase the frequency of user access  might improve both adherence and efficacy.determinants of behavior [The results of this study are generally congruent with findings from other evaluations of Web-based health interventions, although this Web-based program addressed a wider variety of health topics than most previous interventions, which have typically focused on a single health topic. The meta-analysis of Web-based versus non-Web-based interventions conducted by Wantland and associates concluded that effect size comparisons across the 22 studies showed improvement in knowledge and/or behavior change health outcomes for individuals using Web-based interventions . Howeverbehavior . A studybehavior .However, although the studies of Web-based weight management programs reviewed by Wantland et al did not generate evidence of strong effects, recent evaluations of other Web-based weight management programs -28 have In some contrast to the relatively positive findings from this study and others on Web-based dietary and weight management programs, studies of Web-based programs aimed at stress management and increasing physical activity are less numerous and, as indicated by the Wantland review , somewhaIt should be emphasized that Web-based programs vary widely not only in content but also in the degree to which they are science-based , media-rich, and interactive. Web-based programs containing questionable information or lacking accepted theoretical foundations are unlikely to be effective ,29.There are substantial differences in the level of effort and costs of implementing and distributing Web- and print-based approaches. Providing access to the Web program required only a single email message sent instantaneously to all participants, whereas the print materials had to be mailed individually (through company mail) to all participants located in three different offices across the United States. Providing a Web-based program to users is also probably less costly than providing print materials, except perhaps for very small workforces. Indeed, once built, Web-based programs can be delivered to millions of users in a very cost-effective manner, reaching audience sizes unattainable by the traditional workplace health promotion programs.Limitations of the study include the following: (1) all dependent measures were self-reports; (2) the test period was limited to three months; and (3) a no-treatment control condition was not included. Future research on the efficacy of Web-based health promotion interventions should include the collection of physical/biological measures and health care utilization data, should assess participants for longer periods or time, and should include a no-treatment control group.


Although the health communication program feedback cycle is frequently referenced, the steps for moving between or within the sections of the model in a public health environment are rarely described. We detail the process by which the Texas Tobacco Research Consortium implemented the stage of "assessing effectiveness and making refinement" and expanded it to include a program assessment feedback model.Tools were developed to move the consortium through five stages of the expanded program assessment feedback model: 1) formulate research questions using logic models to identify key evaluation items, 2) format data displays from multiple data sources to address research questions, 3) use a facilitated group process to present and review research findings, 4) prepare group recommendations, and 5) involve local partners to translate recommendations into practice.The process allowed us to sift through a large volume of information and prepare data-based program recommendations. A Web-based reporting system provided timely access to community-based program activity data and process indicators that, when linked to logic models, provided actionable items for program improvement. Partnerships among researchers and state and local practitioners created the conditions for implementing the recommendations.Program changes included revisions to program materials, target audiences, and evaluation instruments for a community-based tobacco-cessation campaign. The systematic approach allowed translation of research into practice and should be applicable to other areas of population-based health promotion. Public health promotion programs should be dynamic. They should change based on the findings of previous program activities and emerging research. To effect these changes, public health leaders should collaborate with academic partners to link evaluation findings with local program revisions through a continuous health communication program process, or feedback cycle , which iThe feedback cycle has been frequently referenced ,3 and usThe Texas Tobacco Research Consortium was established in 2000 with funding from the Texas Tobacco Settlement, a settlement between Texas and the tobacco industry in 1998 . UniversIn this article, we expand on the fourth step of the feedback cycle: assessing effectiveness and making refinements. The five stages of the expanded program assessment feedback model are as follows: 1) formulate research questions using logic models to identify key evaluation items, 2) format data displays from multiple data sources to address research questions, 3) use a facilitated group process to present and review research findings, 4) prepare group recommendations, and 5) involve local partners to translate recommendations into practice.We describe the supporting processes, including use of logic models and group facilitation, in selecting, formatting, and interpreting data to improve state and local programs. We use the Texas tobacco-cessation program implemented by the Texas Tobacco Research Consortium to illustrate how data collected from state agencies and multiple academic partners can be used to improve program performance. The expanded program assessment feedback model that evolved is available from the University of Texas Web site .Guide to Community Preventive Services   in MarchThe capacity to generate community-based program activity reports was necessary to create an effective feedback cycle. Research indicates that frequency and encouragement of feedback are positively associated with computer-based monitoring practices . The repWe initially developed logic models for each program goal for the purpose of designing survey instruments . The logThe first four steps in the program assessment feedback model occurred during face-to-face meetings of consortium members, who included researchers and health department professionals. Typically held every other month, the 4-hour meetings were meticulously planned with process and outcome agendas . Each coThe tobacco-cessation logic model developed by the consortium  includesSeveral weeks before the consortium meeting that focused on these two questions, researchers were asked for one- to three-page data displays that addressed the questions. One data display showed the relationship between the launch dates of mass media campaigns and county and state Quitline call volume as reported by the American Cancer Society . AnotherThe two data displays cover five mass media campaigns that ran in Texas during 2002 through 2004: the Great American Smoke-Out (November 2002 and November 2003), Quit and Win (April 2003 and April 2004), and Yes You Can! (January 2004). The Great American Smoke-Out is an annual event sponsored by the American Cancer Society . Quit an.A skilled facilitator, knowledgeable about the data, helped guide consortium meetings. Based on timed agendas, members presented their findings. After considering multiple data sources and linking results to the cessation logic model, consortium members generated recommendations on ways to sustain and improve tobacco-cessation programsWhen they examined the data on Quitline call volume, number of tobacco-cessation activities, and run dates of the various mass media campaigns, for example, consortium members noted early spikes in call volume during October and November 2002 when three campaigns advertising the Quitline and the need to stop smoking were aired simultaneously . AlthougConsortium members also noted that the highest level of community-based tobacco-cessation program activity in the pilot county occurred in April 2004. During spring 2004, a mass media campaign and community-based programs in the pilot county focused on an antismoking campaign to counter tobacco industry advertisements urging people to smoke and included efforts to publicize free nicotine replacement therapy (NRT), the Quitline number, and the opportunity to win a prize for giving up smoking through a local Quit and Win contest.The second highest level of community-based tobacco-cessation activity in the pilot county occurred in November 2003, during the Great American Smoke-Out. This, however, did not coincide with an increase in call volume. In April and May 2004, the pilot county experienced a spike in call volume that was independent of state call volume and paralleled an increase in local program activity. Findings from a follow-up survey of smokers who entered the Quit and Win contest revealed that more than 90% of the smokers who registered for the contest did not call the Quitline, and fewer than 25% were aware of the free NRT . The surDuring the first years of the consortium (2000 through 2002), researchers included tobacco-cessation program recommendations in their end-of-year reports to the state health department. The recommendations, a few of which were included in the report to the state legislature, were all too often embedded in lengthy reports, written in terminology appropriate for research publications, and effectively lost to local communities. Review, discussion, and development of recommendations during group meetings proved to be important at state and local levels. The process not only created increased awareness of the findings but also assisted with the implementation of program changes and promoted the cross-referencing of local findings with national guidelines .About twenty recommendations were generated and grouped into three categories: change mass media messages or delivery channels, expand community awareness, and modify evaluation. Recommendations included actions for program developers, community partners, and evaluators at multiple levels and are detailed on the Web site .Our approach to working with community-based partners has grown out of experience. Participatory processes are often cited as the solution — and a necessity — for the challenge of engaging local community support . ParticiCommunity partners reviewed the recommendations and provided more suggestions for what was needed to increase calls to the Quitline. At one meeting, we discussed new ways to deliver program messages, identified better ways to work together, and initiated study groups to carry out projects in the pilot county. The first study group selected research on smoke-free restaurants as a way to reduce both smoker and nonsmoker exposure to tobacco smoke. A participatory process created a consultation calendar in which local partners identified topics they believed would be most beneficial in their communities. Online newsletters provided additional information and resources to the communities ..Community leaders and contractors now meet to review selected data and recommendations from content experts. Examples of key questions posed in local group consultations include the following: What needs to stay the same? What needs to change? How do we use this information to create change? The framework is used to guide the discussion and is supplemented by worksheets designed to help analyze program messages, target audience, and delivery channels and formulate recommendations for improving the frequency and intensity of messagesThe process used to expand the fourth stage of the health communication program feedback cycle evolved during a 5-year period. The process may work for others who manage a dynamic and diverse body of public health research with local implications. Following are key lessons learned during the process:Create a common understanding of which data are being collected and why. Program logic models are tools that allow researchers and practitioners to formulate common research questions and identify the most relevant data. The wealth of data and cross-tabulations allow for a wide range of research questions. Focus on one or two key research questions within the group, such as: How effective were program activities in achieving the intended outcomes? What did we learn from the campaign? How can we strengthen implementation strategies to increase tobacco cessation using the evidence available?Develop a Web-based program activity reporting system, such as PMATS, for maintaining timely access to process data that can be used to identify actions for program improvement. A Web-based program, along with a central database and report archives, creates organizational memory and a data-retention system.Format selected data related to the research question so that key stakeholders can easily understand it. Typically, a wealth of information is embedded within various databases — the limited amount extracted for local program planning should not interfere with professional publications and basic research reports.Interpret findings together. Use facilitated face-to-face meetings to review the data. Plan meetings well in advance, and alert all parties to the kind of information that will be needed as well as the format desired for delivering the information. Develop recommendations as a group.Invest in the implementation and communication of recommendations at all levels. Work collaboratively to identify how the recommendations relate to program content, implementation, and evaluation. Communicate progress on the recommendations.Create a knowledge-management system that includes knowledge creation, retention, transfer, and use.The expanded program assessment feedback model is another tool for public health promotion and is not limited to tobacco-cessation programs. Effective use of the model depends on the timely availability of locally relevant data and a knowledge-management, retention, and transfer process. When linked to a knowledge-creation system, such as a public health research consortium, and an organization dedicated to improving performance by leveraging current and future knowledge, data-based program development can become a reality.


Chronic disease programs in state public health agencies across the United States are increasingly taking action to integrate activities across single-disease program lines. The perceived benefits of program integration are the motivating force behind these actions, but there is little documentation about how to integrate programs, what the benefits are to program integration, and what barriers exist.Public health agencies recognize benefits from chronic disease program integration because of the potential for efficient use of staff, funds, and surveillance and intervention efforts. Recent reductions in state and federal funding for chronic disease prevention have increased the need for partnerships to make achievement of program objectives possible, but there are constraints imposed by 1) funding that is specific to a disease or an organization  and may be unavailable to integrated programs; 2) barriers in the way agencies and organizations operate; and 3) program accountability that is not flexible enough to enable chronic disease integration. Despite these barriers, public health organizations see value in program integration, and there is a growing determination among public health professionals and policy makers to coordinate and link chronic disease public health programs.strategic alignment of resources for meaningful change. Another definition of program integration is state of combination or the process of combining into completeness and harmony , and 3) concern that program identity, status, constituents, and focus on program outcomes or achievements may be lost (Unpublished data from a survey of state chronic disease and tobacco control program managers identified a variety of barriers to program integration that included 1) categorical funding and accountability, 2) competition between organizations for funding  respecting program identity; 2) preserving successful interventions and categorical program expertise; 3) helping sustain constituencies; 4) maintaining accountability to categorical program funding agencies; 5) acknowledging categorical program priorities, responsibilities, capacity, and successes; and 6) encouraging cooperation, coordination, and collaboration among program agencies. Integration efforts should not compromise categorical program goals.Expected benefits to participating program partners should be clearly stated. All involved parties should  understand the benefits of the integration effort and how it will advance accomplishment of individual program goals. The planning process should identify and document mutual opportunities for integrative program development and policy advocacy work.Program efficiency should be a primary objective of integration processes, and the effectiveness of categorical program partners should not be compromised. Efficient program integration actions should leverage human resources, use time and dollars wisely, avoid duplication of effort, and build on common program interests and objectives. These initiatives should minimize work added for program staff. The benefits of integration initiatives should be greater than the sum of individual program contributions.Improved population health is the most important guiding principle for program integration. Integration activities should work toward improved health of common populations, address crosscutting issues , enhance the ability of programs to address diversity, reduce health disparities, be measurable, and use evidence-based or best-practice intervention methods.Integration outcomes should be defined, monitored, and evaluated, and adjustments should be made to ensure continuous quality improvement of programs. The impact of individual and combined programs should be evaluated by using qualitative and quantitative methods when possible, and successful integration efforts should be publicly recognized by organizational leaders.Successful integration initiatives engage key stakeholders, are meaningful and respectful of individual organizations, create and sustain strong internal and external partnerships, respect and value all perspectives, address needs and fears associated with integration, promote shared decision making, establish clear roles, share vision and accountability, and build internal capacity and staff competencies.Integration initiatives require strong visionary leaders, a supportive organizational environment that guarantees dedicated resources and infrastructure, and communication strategies to reduce or remove barriers to the progress and effective oversight of institutional programs.To stimulate chronic disease program integration initiatives in states, nationally, and for public health professionals, we suggest recommendations for three major public health sectors, SHAs, NACDD, and CDC, to help assure the success of chronic disease program integration initiatives.Engage leadership at SHAs by 1) convening cross-program meetings, 2) defining the purposes of program integration, 3) identifying through collaboration linkage points across programs, 4) establishing and measuring program outcomes, 5) securing organizational endorsement of activities, and 6) keeping leaders informed about integration progress.Develop crosscutting epidemiology and surveillance programs by 1) improving data collection and analysis, 2) integrating mapping of disease burden and risk factors, 3) supporting a multiprogram epidemiology workforce, 4) sharing technology resources, 5) packaging data reports to include multiple program areas, and 6) developing data sources for integrated public health information systems.Leverage use of information technology by 1) collecting integration data, 2) assuring compatibility of administrative and management systems across programs and organizations, 3) sharing program information, and 4) developing collaborative work plans and population-based data repositories.Build state and local partnerships by 1) strengthening professional relationships across programs, government and nongovernment organizations, and lay and professional groups; 2) striving for mutual benefits to individual programs; 3) including nontraditional partners; 4) establishing communication networks for staff and community partners; and 5) sharing information for mutual program development opportunities.Develop integrated state plans by 1) establishing program integration as a normative and priority process; 2) guiding integration efforts; 3) making use of common and specialized data sources; 4) assuring that performance measures are identified, data are collected, and activities are realigned for integrated programming; and 5) convening internal and external partners regularly to monitor the integration progress.Engage management and administration by 1) assessing readiness and organizational support for integration; 2) dedicating staff time to program integration; 3) conducting assessments to determine common linkages and program gaps; 4) planning and implementing strategies to engage partners, stakeholders, and program staff; 5) negotiating, tracking, and evaluating changes in financial management practices; 6) reorganizing as necessary to add or realign staff, budget, and program activities; and 7) engaging in joint problem solving among partners and stakeholders.Implement integrated interventions by 1) developing interventions that identify specific targets for change, 2) assuring implementation of evidence-based strategies, 3) focusing on integration benefits and results, 4) identifying ways to share workload and resources, and 5) reviewing activities regularly for efficiency and effectiveness.Evaluate integration initiatives by 1) determining the success of integrated program implementation, 2) measuring crosscutting performance outputs and outcomes, and 3) making programmatic adjustments based on evaluation findings.Box 2. Recommendations for State Health Agency Actions to Support Integration of Chronic Disease ProgramsEngage SHA leadership.Develop crosscutting epidemiology and surveillance programs.Leverage the use of information technology.Build state and local partnerships.Develop integrated state plans.Engage management and administration.Implement integrated interventions.Evaluate integration initiatives.Many chronic disease prevention professionals responsible for implementation of program integration initiatives are members of NACDD. This national association of chronic disease professionals can play an important role in advancing program integration, and we make the following recommendations for NACDD to assist with chronic disease program integration.Develop and disseminate tools by providing templates for integrated plans, promoting integrated program and surveillance models, and assisting with development of partner databases.Provide educational opportunities by advancing program integration methods through workshops and program sessions at national chronic disease conferences and NACDD member teleconferences and meetings.Provide outreach and recruit national partners by engaging national public health organizations focused on prevention and control of chronic diseases and related risk factors.Reach beyond chronic disease programs by advocating program integration across state and federal categorical chronic disease programs and cooperating with state and federal agencies. Advocate leadership support by communicating with state and federal public health leaders and enlisting their support for chronic disease program integration. Assure continuity of integration initiatives by convening CDC and state program managers to encourage and advance planning, implementation, and evaluation of integration initiatives.Box 3. Recommendations for NACDD Action to Support Integration of Chronic Disease Programs.Develop and disseminate program integration tools.Provide educational opportunities.Provide outreach and recruit national partners.Reach beyond chronic disease programs.Advocate leadership support.Assure continuity of integration initiatives.CDC can play a significant role in the advancement of chronic disease program integration. The following recommendations for CDC to assist chronic disease program integration are offered with caution. National categorical programs must continue to set their goals, serve as effective program advocates, and provide technical assistance where needed.Establish common terminology for improved communication with SHAs and categorical programs to guide program integration.Modify state request for application (RFA) guidelines with wording that encourages program integration, provides flexibility, and creates an environment for SHAs to integrate programs creatively.Develop performance standards and evaluation tools to assure development of quality program integration plans and universal evaluation tools that measure the value-added impact of program integration.Build CDC staff capacity by implementing effective integration activities by establishing performance expectations, providing training, identifying performance incentives, and making program integration a part of institutional processes within NCCDPHP.Develop integration tools by building integrated management information systems, crosscutting program evaluation tools, and integrated surveillance and evaluation plans and by documenting case studies about promising integration practices.Allocate integration-specific resources by identifying 1) an internal program integration champion, 2) model integration strategies, 3) a clearinghouse for effective integration methods, and 4) communication channels among state and national partners to support program integration.Box 4. Recommendations for CDC Action to Support Integration of Chronic Disease ProgramsEstablish common language for SHAs and categorical disease programs.Review and modify RFA guidelines to encourage and make integration possible.Develop performance standards and evaluation tools appropriate to program integration.Build staff capacity.Develop integration tools.Designate integration-specific resources.Implementing the recommendations suggested in this essay should assist national- and state-level chronic disease programs achieve well-integrated programs. These recommendations can be summarized by the following actions: 1) engage the support of key partners and stakeholders, 2) educate and involve program staff and leaders about the value of integration activities, 3) plan for integration, 4) implement best practices and proven approaches, and 5) evaluate integration activities.A strong commitment is needed from public health leaders and staff, and community partners if chronic disease program integration is to become a reality.


As more integrative medicine educational content is integrated into conventional family medicine teaching, the need for effective evaluation strategies grows. Through the Integrative Family Medicine program, a six site pilot program of a four year residency training model combining integrative medicine and family medicine training, we have developed and tested a set of competency-based evaluation tools to assess residents' skills in integrative medicine history-taking and treatment planning. This paper presents the results from the implementation of direct observation and treatment plan evaluation tools, as well as the results of two Objective Structured Clinical Examinations (OSCEs) developed for the program.The direct observation (DO) and treatment plan (TP) evaluation tools developed for the IFM program were implemented by faculty at each of the six sites during the PGY-4 year (n = 11 on DO and n = 8 on TP). The OSCE I was implemented first in 2005 (n = 6), revised and then implemented with a second class of IFM participants in 2006 (n = 7). OSCE II was implemented in fall 2005 with only one class of IFM participants (n = 6).Data from the initial implementation of these tools are described using descriptive statistics.Results from the implementation of these tools at the IFM sites suggest that we need more emphasis in our curriculum on incorporating spirituality into history-taking and treatment planning, and more training for IFM residents on effective assessment of readiness for change and strategies for delivering integrative medicine treatment recommendations. Focusing our OSCE assessment more narrowly on integrative medicine history-taking skills was much more effective in delineating strengths and weaknesses in our residents' performance than using the OSCE for both integrative and more basic communication competencies.As these tools are refined further they will be of value both in improving our teaching in the IFM program and as competency-based evaluation resources for the expanding number of family medicine residency programs incorporating integrative medicine into their curriculum. The next stages of work on these instruments will involve establishing inter-rater reliability and defining more clearly the specific behaviors which we believe establish competency in the integrative medicine skills defined for the program. In 2000, a set of suggested curriculum guidelines  for famiBeginning in 2003, six residency programs joined with the University of Arizona Program in Integrative Medicine to collaboratively develop a four-year combined family medicine/integrative medicine residency program. The two goals of the program are to develop and implement an accredited model for a four year Integrative Family Medicine (IFM) training program which combines training in Integrative Medicine with conventional Family Medicine residency training, and to train physicians capable of practicing healing oriented medicine within the context of the whole person  and with the ability to employ all appropriate therapeutic options, both conventional and alternative.The six sites participating in the IFM Program are the University of Arizona, Beth Israel/Albert Einstein College of Medicine, Maine Medical Center, Middlesex Hospital (University of Connecticut), Oregon Health Sciences University, and the University of Wisconsin. The Accreditation Council of Graduate Medical Education Family Medicine Residency Review Committee (RRC) awarded this pilot program experimental status in 2003. IFM residents from participating sites are identified in the early part of their PGY-2 year, and begin the integrative medicine component of the training experience in January of that year. Each of the six sites has a faculty member trained in integrative medicine who serves as the mentor for the resident's clinical experience during the program.The core elements of the IFM program are outlined in Table Several sets of competencies in integrative medicine, developed for use in other settings, were reviewed as we began to describe the required competencies for the IFM program.  In an itThe most significant challenge to reaching consensus was deciding whether a number of the communication and patient-centered care competencies were already being adequately taught and measured in standard Family Medicine residency training, and thus should be omitted from the specific competencies for this program; or whether these areas – so central to the vision of integrative medicine – should be included here, despite the obvious overlap with the conventional family medicine curriculum. A decision was made to include both the more general communication competencies and the integrative medicine-specific competencies for this first round of evaluation. These competencies were then mapped to the six Accreditation Council for Graduate Medical Education competency domains: patient care, medical knowledge, practice based learning and improvement, interpersonal and communication skills, professionalism, and system-based practice. The IFM competencies and their relationship to the ACGME competencies are depicted in Table Once competencies were identified, evaluation strategies specific to the IFM program were developed based on faculty experience, a review of the evaluation literature, recommendations from the ACGME Outcomes Project, and adopWe developed a direct observation checklist that delineates the specific behaviors expected of IFM participants in the direct care of patients. Behaviors included in that instrument are reflective of program competencies and move across the ACGME General Competencies. The tool builds on a number of other previous efforts in assessing competence in graduate medical education, including the widely-used Mini-CEX.  The toolEach resident is evaluated approximately semi-annually; scores for each resident are compared to expected outcomes based on year of residency and then compared across years of training to determine professional growth and change. To emphasize the formative aspect of the methodology, each direct observation experience is followed by a debriefing session, which incorporates both (a) review of the behaviors observed by faculty and (b) reflection about the encounter experience. We believe such reflection is critical to good practice of integrative medicine. In the Debrief component of the Direct Observation evaluation, the resident is asked to describe: (a) How did you feel during the encounter? (b) What, if anything, did this encounter teach you about yourself? and (c) Reflecting back, is there anything else you could have done to enhance healing during this encounter?Similarly, we developed a treatment plan evaluation tool that delineates the specific behaviors and language expected of IFM participants when developing and modifying evidence-based integrative treatment plans for specific patients. The same rating scale is used for the behaviors, which again are linked to the IFM program competencies as well as the ACGME General Competencies. Two treatment plans from each resident are evaluated annually. These scores are shared with the resident during a debriefing session.The OSCE component of our evaluation strategy has been implemented via two OSCE experiences: the first, OSCE I, takes place in the middle of the PGY-3 year, and the second, OSCE II, in the middle of the PGY-4 year. OSCE I examines integrative medicine history-taking skills; OSCE II assesses the development of an integrative medicine treatment plan and the effective communication of that plan to the patient. For OSCE I, residents are presented with a patient experiencing migraines; for OSCE II, with a patient at risk for cardiovascular disease. Each OSCE represents an encounter with a single Standardized Patient. We do recognize that using a single patient raises potential problems with content specificity and with attaining a generalizable estimate of clinical performance; however for this initial round of OSCE implementation logistical constraints limited us to a single patient encounter. We hope to develop multiple OSCE stations for future iterations of this process.The data from the first implementation of OSCE I (2005) were reported elsewhere , and ledThe direct observation (DO) and treatment plan (TP) evaluation tools were implemented by faculty at each of the six sites during the PGY-4 year (n = 11 on DO and n = 8 on TP). The OSCE I was implemented first in 2005 (n = 6), revised and then implemented with a second class of IFM participants in 2006 (n = 7). OSCE II was implemented in fall 2005 with only one class of IFM participants (n = 6). This pilot study was approved by the University of Arizona Institutional Review Board.Data from the initial implementation of these tools were then examined (a) to attempt to evaluate the effectiveness of our teaching and (b) to begin to evaluate the actual tool for its utility in our evaluation process. If residents performed poorly on any competencies across IFM sites and evaluation strategies, this might represent a deficiency in our teaching program. Contrastingly, if particular behaviors could not be evaluated consistently across sites (a given competency was repeatedly rated as DNO), this might mean that the competency (a) was not clearly defined or stated or (b) that the competency is not one well-suited to the type of evaluation  for which it was proposed. Each of these findings would be used to suggest changes to the next iteration of this evaluation tool. Descriptive statistics were used to describe the data collected from the DO and TP tools.In order to evaluate the effectiveness of teaching and to determine areas of weakness within the training program, scores for individual competencies on the Direct Observation and Treatment Plan Evaluation Tools across participants/program sites were examined. We hypothesized that such weakness would emerge as consistently low scores across sites for a given item; for the purposes of this analysis, we considered items where the mean score was 2.5 (83%) or below out of a possible three. We then examined the data across sites for behaviors that observers consistently (>= 50% of the time) identified as "unable to evaluate" or "did not observe" in order to evaluate the utility of the evaluation tools. The results of the Direct Observation evaluation are listed in Table Results from the OSCE I (2005) were previously reported . ResultsOutcomes from OSCE I (2006) also suggest that although we can assume basic communication skills as a prerequisite already established in our IFM participants by the middle of PGY-3 year, we still have significant work to do in the area of comprehensive history-taking, both conventional and integrative. Perhaps most surprising is that only 57% of participants asked about current and past use of CAM, only 57% adequately screened for depression, and only 57% recognized that both stress and neck pain could be contributing to the migraines in the OSCE patient. Only 29% inquired in depth on how the migraines might be affecting work or personal relationships for the patient.OSCE II was designed to assess knowledge base and treatment planning and communication skills. OSCE II (2005) results Table  suggest American Family Physician reviews, [Two fundamental goals directed our development of competency-based evaluation tools for family physicians training in integrative medicine. Our immediate goal was to develop an effective evaluation strategy for the participants in the Integrative Family Medicine pilot program to meet internal evaluation needs as well as those of the Family Medicine Residency Review Committee (RRC). Our long-term goal is to contribute measurement tools for broad use in Family Medicine Residency programs. As more integrative medicine educational content is integrated into conventional family medicine teaching, the need for effective evaluation strategies grows. Current curriculum being introduced into family medicine residency training includes the use of botanicals and nutritional supplements, the use of specific dietary strategies such as the anti-inflammatory diet and the use of mind-body approaches. Evidence-based discussion of these areas regularly appears in reviews,  and the reviews, However there are no competency-based evaluation tools that effectively assess competencies in evidence-based integrative medicine for use in residency programs. Our intention is to share our experience in developing and utilizing tools that may be modified for use in residency programs.We found the direct observation checklist a useful tool, particularly when incorporating a debriefing session following the observation. The addition of this element allowed us to re-emphasize the "personal process" dimension of the integrative interview which incorporates an emphasis on reflection and countertransference as clinical skills essential to the integrative approach. The limitation, as with all such tools, relates to the difficulty of standardizing across observers and sites the exact behaviors that must be observed to demonstrate competency in a certain area. We plan to establish better face validity for the next iteration of these tools by defining exact behaviors more clearly using input from experts in the field of integrative medicine outside of the IFM faculty. Whereas the six IFM programs all have an "expert" faculty member who can rely on their own experience in assessing competency in herbal prescribing, or incorporation of mind-body strategies into treatment planning, other residencies might lack such expertise among the faculty. Therefore defining more clearly the behaviors which represent competency will be critical to the potential broader use of this instrument in non-IFM residencies.The reliability of these tools also needs to be established if they are to enter wider use in residency programs. At our next faculty meeting, we plan to hold an "observation rating session" in which the six faculty in the program all observe one resident in a patient encounter and rate their performance using the DO tool. We will then use this data to measure the inter-rater reliability for this tool. We are also planning a similar process to establish inter-rater reliability for the treatment plan evaluation tool during the coming year.Our initial OSCE experience revealed that we were measuring basic patient-centered communication skills which more properly belong in the domain of conventional family medicine. IFM participants in the initial OSCE I (2005) confirmed that by the PGY-3 year these skills have been thoroughly mastered and should be considered prerequisites for the IFM training rather than objectives for the program. Our second iteration of the OSCE I (2006) focused more narrowly on integrative medicine history-taking – competencies 6 through 14 specifically – and was much more effective in delineating strengths and weaknesses in our residents' performance. We are hopeful that as we develop new OSCEs more focused on these specific integrative medicine competencies that these might be useful, either wholly or in part, for residencies to incorporate into their ongoing OSCE assessment strategies.Recognizing the infancy of both the IFM program and these evaluation strategies – and the fact that we do not know if areas of weakness in performance identified are a function of actual resident performance or of a weakness in the design of our tools – there were two areas of relatively weak performance which appeared both on the DO/TP assessment and on the OSCEs. First, as discussed above, there is an apparent pattern of weakness in the area of how treatment recommendations are communicated to patients – i.e. how often the obstacles to implementation are discussed adequately, and how effective residents are in assessing readiness for change. Second is the fact that apparently spirituality is often not discussed with patients by IFM participants. Since both of these findings appear in both evaluation settings, our hypothesis is that these do represent real weaknesses in performance rather than weaknesses in our measurement strategies. Of course it is also possible that due to our small "n" and variability across sites, these results represent evaluation artifact or confounding rather than actual weakness in performance and/or curriculum. We plan to more actively address both of these areas in our curriculum in the coming year, and our hope is that this will lead to improvement in competency scoring on the next round of DO/TP and OSCE evaluation scheduled for 2007 and to a change in how our rising PGY-4s perform in these areas.The Integrative Family Medicine program has created a unique need and opportunity to develop specific integrative medicine competencies for residents as well as the tools to measure them. To date, OSCEs exploring the expanded integrative history and treatment plan have been developed, assessed and modified. These skills have been further assessed with a direct observation tool and a treatment plan review tool. As these tools are refined further they are likely to be of value to the expanding number of family medicine residency programs incorporating integrative medicine into their curriculum.The next stages of work on these instruments will involve establishing inter-rater reliability by having faculty independently rate the same clinical encounter, and establishing face validity more clearly by defining more clearly, using opinion solicited from experts in the field of integrative medicine outside of the IFM faculty, the specific behaviors which we feel represent competency in the integrative medicine skills we have defined for the program.The author(s) declare that they have no competing interests.All of the authors participated in conceiving the evaluation process and tools described in the article. BK coordinated the writing of the manuscript and assisted in the analysis of the evaluation data. MK was responsible for analysis of the evaluation data and contributed to the writing of the methods, results and discussion sections. VM oversees the IFM program and contributed to the writing of the Background and Discussion sections. MH, CS, PL and SH were responsible for collecting the evaluation data on the participants and contributed significantly to the writing of the article. All authors read and approved of the final manuscript.The pre-publication history for this paper can be accessed here:


Through well-planned health promotion and education programming, federally funded projects are often leaders in the development and implementation of sound public health practices. In public health practice, program evaluations are used to 1) determine whether program objectives related to health status have been achieved, 2) improve program implementation, 3) increase community support, 4) contribute to a scientific base, 5) provide accountability to community members and other stakeholders, and 6) guide program-specific policy decisions  oversees all requests for review and approval under the PRA . The age9 months . HoweverAlthough the time involved in obtaining OMB clearance can pose a considerable challenge to evaluators, the process can also significantly improve the quality of the data collection. Program planners and evaluators should work together to focus their efforts and use ethical data-collection methods to obtain useful and necessary data. Such cooperation improves the program results while decreasing the public's burden.During the clearance process, OMB requires detailed descriptions of the following :The reasons the data collection is necessaryThe purpose and use of the information that will be obtained from the data collectionThe use of improved information technology to reduce burdenEfforts to identify the duplication and use of similar informationThe possible impact of the data collection on small businesses or other small entities The consequences of collecting the data less frequently than plannedSpecial circumstances relating to Title 5, Part 1320.5 ("Controlling Paperwork Burdens on the Public") of the Code of Federal Regulations (CFR) guidelines about federal information collectionsFederal Register notice and evidence of efforts to consult with individuals other than those in the agency collecting the informationComments in response to the An explanation of any payments or gifts to respondentsConfidentiality assurances provided to respondentsJustification for sensitive questionsAn estimate of annualized cost and burden hoursAnnualized government costs An explanation for any program changes or adjustmentsA plan for tabulation and publication and a time scheduleAny reasons that an OMB expiration date might be inappropriate to post on instrumentation Exceptions to certification for the PRA submissionCareful attention to the requirements for OMB clearance enhances program efficiency and improves evaluation processes and outcomes. The requirements are consistent with academic and professional recommendations that program planning be carried out with evaluation in mind; the two processes need to be simultaneous for best results ,11.www.whitehouse.gov/omb/inforeg/infocoll.html, which guides program planners and evaluators through the process.The general public recognizes that the OMB review and approval process is a positive procedure that holds the federal government accountable for the public's burden to provide data. Federal agencies that are striving to provide evidence that tax dollars are being well spent consider OMB compliance to be a difficult but necessary obstacle to overcome. The PRA, which is enforced by the OMB, is a reasonable compromise between reducing paperwork burdens on the public and maximizing the benefits of data collection to ensure that well-planned public health programs have meaningful evaluations. Knowledge about and compliance with the PRA requirements are the essential component of the compromise, improving taxpayer satisfaction and government accountability. Fortunately, all necessary information about the PRA and OMB is available online from


Web-based behavior change interventions often include rich media . The rationale for using rich media includes the need to reach users who are not inclined or able to use text-based website content, encouragement of program engagement, and following the precedent set by news and sports websites. We describe the development of a bandwidth usage index, which seeks to provide a practical method to gauge the extent to which websites can successfully be used within different Internet access scenarios . We conducted three studies to measure bandwidth consumption. In Study 1, we measured the bandwidth usage index for three video-rich websites . We then estimated the number of concurrent users that could be accommodated by each website under various Internet access scenarios. In Study 2, we sought to validate our estimated threshold number of concurrent users by testing the video-rich smoking cessation website with different numbers of concurrent users. In Study 3, we calculated the bandwidth usage index and threshold number of concurrent users for three versions of the smoking cessation website: the video-rich version (tested in Study 1), an audio-rich version, and a Web-enabled CD-ROM version in which all media-rich content was placed on a CD-ROM on the client computer. In Study 1, we found that the bandwidth usage index of the video-rich websites ranged from 144 Kbps to 93 Kbps. These results indicated that dial-up modem users would not achieve a “good user experience” with any of the three rich media websites. Results for Study 2 confirmed that usability was compromised when the estimated threshold number of concurrent users was exceeded. Results for Study 3 indicated that changing a website from video- to audio-rich content reduced the bandwidth requirement by almost 50%, but it remained too large to allow satisfactory use in dial-up modem scenarios. The Web-enabled CD-ROM reduced bandwidth requirements such that even a dial-up modem user could have a good user experience with the rich media content. We conclude that the bandwidth usage index represents a practical tool that can help developers and researchers to measure the bandwidth requirements of their websites as well as to evaluate the feasibility of certain website designs in terms of specific use cases. These findings are discussed in terms of reaching different groups of users as well accommodating the intended number of concurrent users. We also discuss the promising option of using Web-enabled CD-ROMs to deliver rich media content to users with dial-up Internet access. We introduce a number of researchable themes for improving our ability to develop Web-based behavior change interventions that can better deliver what they promise. The Internet holds great promise as a delivery channel for programs designed to help people change their behaviors ,2. For eIn this report we will review some of the forces that encourage the development of rich media websites, but we will also examine in considerable detail the factors that temper unbridled enthusiasm for this trend. We believe that any reasoned analysis of the use of rich media website content should also consider the possible barriers to its use. As a key part of our analysis, we will describe the development and testing of a practical index for gauging the amount of bandwidth that a Web-based program requires. We hope to highlight the ways that bandwidth constraints associated with rich media restrict both program reach as well as the number of concurrent end users who can successfully use a Web-based program. We also discuss ways that behavioral research can help elucidate how and when to use different website program ingredients, including, for example, the use of rich media. The remainder of this section describes the trends encouraging the use of rich media and introduces key technology issues that set the stage for the presentation of our investigation.broadband Internet access both in the United States  usually measured in bits or bytes per second” [Most users and program developers are familiar with the term  second” . The phyBandwidth constraints have significant practical implications because, when exceeded, they can greatly increase end user frustration and seriously degrade interaction with a website. Consider, for example, that usability research confirms that users have extremely limited patience when waiting for a Web page to download . The lonThere are at least four potential delivery bottlenecks that can interact in a cumulative fashion to impede the acceptable flow of content from the Internet to the end user . The siz                    Bottleneck A is defined as the capacity of the host to serve (deliver) the program content. Throughput can be undermined at this point if the program content is being served by an inefficient or overworked server. Fortunately, current servers are sufficiently robust that they become a throughput bottleneck only in scenarios when they are called upon to manage responses and generate logic to build and delivery dynamic Web pages for a very large number of concurrent users. Moreover, there are sophisticated hardware and software solutions for handling this type of demand. For example, major commercial Internet content providers typically use a server farm configuration in which one group of servers manages applications (program logic), other servers process database activities, and yet other servers manage the delivery of media assets  . Howeverbroadband connections enable home users to download significantly greater amounts of data per unit time from the Internet.Bottleneck B describes the connection between the Internet and the home user. It is possible that the user’s computer can be underpowered to manage Web-based programs with rich media. But a critically important factor at this point involves the “last mile” challenge ,42 , can be used to define the minimum bandwidth requirements for any given program and can be used to estimate the number of concurrent users that can be accommodated (have a good user experience) under varying Internet access scenarios .We determined the BUI for three different video-rich Web-based behavior change programs: (1) the 1-2-3 SmokeFree smoking cessation program  Figure , (2) a p                    Specifically, we measured the amount of data received by an end user (client computer) from a server (host computer) that delivered representative content from each of the three rich media Web-based programs. The 13-minute test followed a consistent protocol described generically in It is important to note that the end user in this test accessed a representative portion of the behavior change program content hosted on the server. We defined “representative portion” as a mixture of Web pages which included text, graphics, and rich media content (audio and video) used in a manner that simulated a typical use case. The index or gauge of data throughput was calculated as the total number of bits received by the client computer from the server divided by the duration of the test period to yield an average expressed in Kbps.We used the BUI calculations to predict the number of concurrent users who could have a satisfactory experience using Web-based behavior change programs under a number of different Internet access scenarios.throttle a network computer (host) so that it would allow only 768 Kbps throughput . Next, we used that network computer to host the video-rich 1-2-3 SmokeFree Web-based smoking cessation program tested in Study 1. We stationed each of five end user participants  in separate offices where they could receive study instructions. Each end user participant used a separate workstation connected to the company intranet, and each was asked to clear the browser cache  an “audio-rich” version in which program content had been adapted such that video content was replaced by audio-only content, and (2) a Web-enabled CD-ROM condition in which all media-rich content was placed on a CD-ROM that was located on the client computer. All program content and behavior was otherwise identical to that found in the video-rich version of the smoking cessation program tested in Study 1. We followed the test protocol described in idealized throughput of various methods of accessing the Internet, ranging from the slowest 28.8 Kbps dial-up modem to an extremely fast 100-baseT corporate network [The BUI for three video-rich Web-based behavior change programs ranged from 144 Kbps for the smoking cessation website to 93 Kbps for the eldercare website . By divi network , to calcIn terms of Bottleneck A , Table 2In terms of Bottleneck B describing end user access to the Internet, data displayed in In terms of Bottleneck C, Finally, considering Bottleneck D, data presented in When 5 participants accessed the video-rich version of the smoking cessation program using a maximum download bandwidth of 768 Kbps, all users reported that they had a good experience with the program. The situation changed dramatically when 6 participants concurrently accessed the same program from within the same network environment—in this test case, all users reported that they had an unsatisfactory experience using the website, with reports of long delays in page loading, videos not playing, and so forth. These results confirmed the validity of the threshold numbers described in Results of the third study are presented in In this report we have described a series of tests of the bandwidth usage index (BUI) that confirm the value of measuring the actual data throughput associated with delivering Web-based behavior change programs that have varying amounts of rich media. We believe that the BUI or other similar throughput measures can provide designers with important guidance about how much rich media to include in their Web-based programs depending upon the intended audience and the intended number of concurrent users.While technology changes at a very rapid rate, it is nonetheless also true that program designers need to take into careful consideration the significant delays associated with the diffusion of these changes to the large installed base of computer users. This challenge is confronted when considering end user software and hardware . It is also highly relevant when considering types of Internet access which, in turn, affect the choice of program ingredients. These considerations may be even more salient when a Web-based intervention is targeted to groups that are not as far ahead as others on the adoption curve—as in the limited broadband penetration in rural  and diffDelivering digital content is fraught with tradeoffs regarding costs, reach, and scalability. If a Web-based program uses rich media content, then the calculations we describe in this report indicate that the program will not be usable for approximately half of US Internet users who still use dial-up modems. Even if you target a website to users whose broadband Internet access provides them with more than sufficient bandwidth to receive rich media programming, the hosting of that rich media content to thousands of concurrent users has substantial costs associated with using content delivery network solutions .lean                    media websites that use vector-based animated graphics to offer interactivity and graphical content without excessive bandwidth consumption. One vector graphics tool to consider in this regard is Macromedia Flash (for its vector graphics capabilities rather than its video capabilities). Flash uses a client-side browser plug-in that has a very large installed base: version 2.0 is on 98.3% of current computers, whereas the most recent version 7.0 is on 90.0% of current computers [Future research should consider developing omputers . Alternaomputers . The accResearchers and developers of Web-based behavior change programs should consider using intelligent adaptive designs that “sniff out” user’s bandwidth transparently and then tailor program content according to the bandwidth characteristics of each user’s Internet connection. Using this approach would require program designers to consider how best to deliver equivalent content using different media—a topic with both technical and behavior change complexities that have received little empirical study to date.Website designers should make greater use of Web-enabled CD-ROMs to deliver rich media to the very large base of Internet users who do not have broadband access. Programs of this type would still use centralized functions of a Web-based program , but they would access rich media located on each user’s computer. In this scenario it should be noted that video and audio would not need to be as compressed as for Web delivery. As a result, rich media components could be longer and larger , and their enhanced quality could result in better acceptability and improved impact.Plans to market a Web-based program need to consider the bandwidth demands that might accompany program recruitment announcements or delivery scenarios that encourage concurrent usage. Consider, for example, the possible stress on a smoking cessation website tied to an event that occurs on a single date  or when an announcement is issued to all employees of a large corporate client. Similarly, consider the scenario in which scores of users are assembled in community computer centers or corporate training rooms to access rich media Web-based programs. Announcements via mass media, email as well as URL publishing, can generate large demand spikes. Resulting peaks of Internet traffic, especially when programs deliver rich media content, can compromise the functioning of any Web-based program.There are some noteworthy limitations to the design of the current study. The tests that validated the BUI threshold (number of concurrent users who would have a satisfactory user experience) were not exhaustive. More users could have been included to provide further confidence in the results we report. Nor was it possible for us to qualify the use of BUI by quantifying the peaks and valleys of available bandwidth in different real-world circumstances . It is also important to acknowledge the subjective nature of the “satisfactory user experience” criterion we used. This measure was based upon individual user reactions when accessing program content in a realistic manner. We attempted to operationally define our use cases to be similar to what would typically be experienced by users of these programs.Our investigation does, however, have a number of strengths. For example, we were able to measure the bandwidth use of a number of Web-based behavior change programs that contained rich media. In addition, we were able to test the impact on bandwidth use of three variations of one of these rich media behavior change programs, which enabled us to recommend the approach of using Web-enabled CD-ROM design to extend the reach of rich media Web programs to the largest possible audience of potential users. Finally, while the BUI is not a precise tool, it nonetheless describes a process that others can replicate in order to evaluate their own Web-based programs. We believe that important implications can be drawn from using the BUI to calculate estimates of the maximum number of concurrent users by Internet access type (Tables 2 and 3).Because they attract relatively meager numbers of concurrent users, typical randomized controlled trials of Web-based programs provide an inadequate test of their ability to deliver content in a scalable manner. In some ways, it is as if the airworthiness of a modern airliner is tested with a light load of passengers and fuel in only ideal flying conditions without ever being tested with a full load in windy, stormy conditions. But website scalability is a typical rationale invoked in support of using the Internet as a delivery channel and in justifying the considerable development expenses. We believe it is essential to acknowledge that Web-based programs need to be designed to fit the goals for which they are intended and that their bandwidth requirements need to be considered, measured, and reported in order to gauge how well they will likely operate under their intended use case scenarios.


The Steps to a HealthierUS Cooperative Agreement Program (Steps Program) enables funded communities to implement chronic disease prevention and health promotion efforts to reduce the burden of diabetes, obesity, asthma, and related risk factors. At both the national and community levels, investment in surveillance and program evaluation is substantial. Public health practitioners engaged in program evaluation planning often identify desired outcomes, related indicators, and data collection methods but may pay only limited attention to an overarching vision for program evaluation among participating sites.We developed a set of foundational elements to provide a vision of program evaluation that informs the technical decisions made throughout the evaluation process. Given the diversity of activities across the Steps Program and the need for coordination between national- and community-level evaluation efforts, our recommendations to guide program evaluation practice are explicit yet leave room for site-specific context and needs. Staff across the Steps Program must consider these foundational elements to prepare a formal plan for program evaluation. Attention to each element moves the Steps Program closer to well-designed and complementary plans for program evaluation at the national, state, and community levels. The Steps to a HealthierUS Cooperative Agreement Program (Steps Program) enables funded communities to implement chronic disease prevention and health promotion efforts to reduce the burden of diabetes, obesity, asthma, and related risk factors — physical inactivity, poor nutrition, and tobacco use. The Steps Program funds communities in three categories: state-coordinated small cities or rural areas, large cities or urban areas, and tribes or tribal entities. In fiscal year 2003, the Department of Health and Human Services (HHS) distributed $13.6 million to 12 programs representing 24 communities . In fiscal year 2004, HHS distributed $35.8 million to increase support to existing communities and fund an additional 10 programs representing 16 communities . To date, the Steps Program includes 40 communities nationwide.In addition to fiscal resources, HHS provides oversight and technical expertise to support evidence-based program planning and implementation, disease and risk factor surveillance, and program evaluation. The allocation of resources to surveillance and evaluation meets the recommended 10% of total program dollars in the majority of funded communities . DiseasePublic health practitioners engaged in program evaluation planning often identify a stream of program outcomes, related indicators, and data collection methods but may pay only limited attention to developing an overarching vision for program evaluation among participating sites. Because of the need for coordination between national and community-level evaluation efforts, recommendations to guide program evaluation practice must be explicit; however, they must also be flexible enough to accommodate the diversity of programmatic activities and community-specific needs. It is important to remember that "the term evaluation does not imply a particular type of a study design" ; the eleFramework for Program Evaluation in Public Health to inform evaluation planning; 4) seek cultural competence throughout the evaluation process; 5) prepare a program logic model as a platform for evaluation planning, implementation, and use of findings; 6) identify the purposes of the evaluation; 7) identify intended users and uses of the evaluation; 8) identify key evaluation questions; 9) attend to process and outcome evaluation; and 10) maximize the use of existing surveillance systems for outcome measurement. For the Steps Program, considering these foundational elements is necessary to prepare a formal plan for program evaluation. The We have developed a set of foundational elements for program evaluation planning, implementation, and use of findings to clarify expectations for national and community-specific practice. For the Steps Program, these elements include the following: 1) distinguish between research and program evaluation; 2) define program evaluation; 3) use the A key misunderstanding about program evaluation is that it must follow an academic or epidemiologic research model. For community-based public health programs, this model is typically difficult to implement, if not impossible or inappropriate. Although often considered the "gold standard" for public health research, the "use of randomized control trials to evaluate health promotion initiatives is, in most cases, inappropriate, misleading and unnecessarily expensive" . ResearcDefinitions of program evaluation vary by field of practice and approach. However, they typically include some reference to the "systematic investigation of the worth or merit of an object" . For theFramework for Program Evaluation in Public Health (Framework) is a practical, nonprescriptive tool designed to summarize and organize essential elements of program evaluation. The document is a practical, nonprescriptive tool designed to summarize and organize essential elements of program evaluation. The document  recommends the following six steps for program evaluation: 1) engage stakeholders, 2) describe the program, 3) focus the evaluation design, 4) gather credible evidence, 5) justify conclusions, and 6) ensure use and share lessons learned  disease and risk factor surveillance via the BRFSS and YRBSS; 2) systematic program monitoring to assess implementation and early outcomes; 3) assessment of longer-term outcomes at the community level; and 4) strategic coordination with national-level evaluation activities. Stakeholders prioritized evaluation questions on the path to a complete evaluation and implementation plan based on these pillars. Thus, systems are in place to provide a steady stream of credible information for each community. The constant exchange of information among these communities led to practical and cost-effective methods for demonstrating progress toward program goals, identifying opportunities for program improvement, and sharing lessons learned to enhance chronic disease prevention and health promotion efforts statewide. For this program, the six steps and related standards have proven to be an effective approach to better integrating disease and risk factor surveillance, program, and evaluation planning with an emphasis on stakeholder participation across four distinct communities in New York.Framework. However, the authors do not explicitly define cultural competence. As a starting point for practice across the Steps Program, cultural competence in program evaluation "involves a set of academic or interpersonal skills that allow individuals to increase their understanding and appreciation of cultural differences and similarities within, among, and between groups"  Steps Program Office (SPO) and Division of Adult and Community Health, the Inter-Tribal Council of Michigan (ITCM) developed a culturally appropriate strategy for participation in the BRFSS. The approach draws on localized definitions of culturally competent practice and related ethics to meet the information needs of decision makers at the national level and among tribes involved. For the majority of these tribes, inclusion in the sampling frame was actively voluntary. Tribal members submitted their phone numbers via boxes at each site. Tribes offered modest incentives for participation. The approach resulted in approximately 7800 phone numbers. The ITCM provided these numbers, without identifiers, for sampling and data collection. In this community, ongoing dialogue, respect for tribal values, and flexibility resulted in full participation in the data collection requirements of the Steps Program. Stakeholders at the national and community levels maximized an opportunity to improve surveillance and evaluation practice and ensure culturally competent service  to tribal members.Although cultural competence may look different at the national and community levels and vary in concept and practice across the Steps Program, it is critical to the ultimate usefulness of an evaluation. Appropriate attention to cultural competence throughout the evaluation process reflects and affirms principles, ethics, and standards for program evaluation in public health. Moreover, it ensures a consequential role for diverse program stakeholders and compels greater use of the evaluation for decision making.theory of change . Inputs include program resources . Outputs are the direct products of program activities . Outcomes are the results, effects, or benefits of public health programs; they are the changes that occur for individuals, groups, families, households, organizations, or communities during or after the program . It is important to understand the difference between outputs and outcomes. Outputs relate to "what we do" as public health practitioners, and outcomes refer to "what difference is there" because of these efforts . Sound eThe purpose of the evaluation differs from the purpose of the program; articulating the purpose of the evaluation will “prevent premature decision making regarding how the evaluation should be conducted” . ProgramAn explicit statement of the purpose of the evaluation adds clarity and focus to the study and enhances usability among stakeholders with limited knowledge of program evaluation. With the purposes of the evaluation agreed upon, subsequent pieces of the study fall into place more easily . Stakeholders in the Steps Program delineated the following purposes of the national evaluation early in the planning process: assess the merit or worth of the Steps Program or key efforts; document program processes and progress toward intended outcomes; identify opportunities for ongoing program improvement; demonstrate accountability for resources to key stakeholders; and identify opportunities for ongoing program development and improvement.The goal of utilization-focused evaluation is intended use by primary intended users . PrimaryThe key to designing an evaluation that best meets stakeholder needs is precisely defining the questions that the study is expected to answer . EvaluatOften, evaluation questions are implicitly understood by program staff and consultants closest to the study and therefore are not included as an explicit component of the evaluation plan. However, a participatory approach to evaluation requires stakeholder involvement in the identification of evaluation questions. For the Steps Program, individuals responsible for program evaluation at the national, state, and community levels are encouraged to discuss, prioritize, and articulate key evaluation questions to ensure that evaluation practice and products are meaningful to all stakeholders.Public health programs should be evaluated in terms of their processes and outcomes. While outcome evaluation is used to assess whether a program works, it cannot typically demonstrate why or how the program works (or does not work). Knowledge of why or how a program creates change is as relevant in public health programs as information about whether a desired change occurred . ProcessFor example, Boston Steps includes programmatic activities in seven neighborhoods: Dorchester, Hyde Park, Jamaica Plain, Mattapan, Roxbury, South Boston, and South End/Chinatown. A cornerstone of the program, NeighborWalk, is an evidence-based community walking initiative. The Boston Public Health Commission, in collaboration with the Harvard Prevention Research Center, uses three tools to document implementation and participation. One, to document community participation in the program, walk coordinators citywide submit a weekly summary of activities . Two, NeighborWalk participants volunteer to complete enrollment and exit questionnaires that capture demographic information about themselves and their relevant health-related behaviors. Three, walk coordinators complete brief narratives to document and describe perceived successes, barriers, and practical suggestions to improve NeighborWalk before the next cycle of activities. Each of the tools offers unique data for program monitoring and informed decision making . This investment in process evaluation allows staff to quickly assess whether the program is reaching its intended audience, describe actual characteristics of the program in diverse neighborhoods, and identify improved pathways to influence health-related behaviors in these communities.The definition and application of "process," "outcome," and "impact" evaluation vary in public health and beyond. PRECEDE–PROCEED, a prominent model for program planning, implementation, and evaluation in public health, positions impact evaluation to measure short-term effects defined as knowledge, skills, and behavior. Outcome evaluation measures health or quality of life . Yet, inSurveillance is the "ongoing, systematic collection, analysis, and interpretation of outcome-specific data for use in planning, implementing, and evaluating public health practice" . MonitorHealthy People 2010 objectives and specific program outcomes. Whenever possible, the Steps Program uses data from these sources for national- and community-level program planning and evaluation purposes.The release of Indicators for Chronic Disease Surveillance  providesHealthy People 2010, Indicators for Chronic Disease Surveillance, and community-specific selections drawn from local stakeholder priorities and needs. Steps to a Healthier Colorado enhanced existing surveillance systems to collect data in funded communities and ensure quality information to assess progress toward short-term, intermediate, and long-term outcomes. The BRFSS includes adults aged 18 years and older, the YRBSS includes youth in grades 9 through 12, and the state-based Child Health Survey includes children aged 1 to 14 years. These surveys provide data for both strategic program planning and the tracking of progress toward desired health outcomes. To maximize use for program evaluation, staff enhanced each survey to include additional short-term and intermediate measures relevant to community-based programming and objectives.For example, Steps to a Healthier Colorado includes programs in Mesa, Pueblo, Teller, and Weld counties. The Colorado Department of Public Health and Environment coordinates surveillance and evaluation activities with staff in each of these communities. Program staff identified outcomes for measurement from three primary sources: Monitoring and evaluation are included among essential public health services as important components of efforts to promote continuous quality improvement of public health systems and related programs. The information presented here makes visible the overarching direction of evaluation practice across the Steps Program, including attention to the intended use of findings for accountability and continued program development. Foundational elements for program evaluation planning, implementation, and use of findings highlighted here illustrate the commitment of the Steps Program to improving public health, not only through service to communities but also through careful and appropriate documentation of program implementation and outcomes to provide practice-based evidence for decision making now and in the future.


HIV infection rates have reached epidemic proportions amongst injecting drug users (IDUs) in Iran. Although a number of community-based interventions have being implemented in the country, there is little information on the risk behaviors of IDU participants in these programs. This cross-sectional report aimed to compare the risk behaviors of injecting drug users with differential exposure rates to an HIV outreach program in Tehran, Iran. Results indicated that shared use of needle/syringe in the past month was significantly lower among IDUs who received estimated ≥ 7 syringes per week than those who did not . While the effectiveness of this outreach program needs further evaluation through a longitudinal investigation, our preliminary findings suggest that the outreach program in Tehran may have been beneficial in reducing direct sharing among those who received more than several needles/syringes from the program. Injecting drug use, to date, has been reported as the main route of HIV transmission in Iran ,2 and reIn October 2004, one year after the establishment of the outreach program, a convenience sample of drug users (consisting 213 IDUs and 85 non-IDUs) was recruited at the drop-in centre, and at parks and streets in the area. Active drug users were approached by an ex-user staff member of the NGO for recruitment and were then interviewed by a male researcher not affiliated with the Persepolis Society. In the questionnaire-based interview, participants were asked about their demographics and HIV risk characteristics, as well as their contact with the outreach program, estimated length of contact, and the total number of syringes they received from the program.Of 131 male injecting drug users who ever received free needles/syringes from the program, data of the 105 male IDUs with a complete set of variables for multivariate analysis were considered for this report and then categorized based on the rate of exposure to the program. The sub-sample was divided into two groups of IDUs who received few needle/syringes (< 7 syringes per week) (37/105 = 35%) and those who received estimated ≥ 7 syringes per week during their participation in the program (68/105 = 65%). While adjusted for demographics and basic drug use characteristics, a logistic regression analysis was performed to assess the association between having received ≥ 7 syringes per week from the program and the related behavioral variables (Table). The research protocol was approved by the Ethical Committee of the Iranian National Center for Addiction Studies at Tehran University of Medical Sciences in Iran and by the Committee for Research on Human Subjects at Kyoto University in Japan.The median age of IDUs in this sub-sample was 32.0 years and 67% had educational levels of junior high school or more. Up to 61% were jobless at the time of interview and 52% have ever been married. Among socio-demographic characteristics, only being recruited from the drop-in centre and having lived for 6 years or more in that area were associated with having received ≥ 7 syringes per week from the program in the bivariate analysis.P = 0.000) (not in the Table). Also, shared use of needles/syringes in the past month was significantly lower among those IDUs who received ≥ 7 syringes per week than those who did not . However, there was no difference between two groups in the use of a shared cooker at last injection, condom use during last sex, levels of HIV/AIDS knowledge, and a history of previous HIV testing (Table).While 18.9% (7/37) of those who received few needles/syringes from the program reported having injected using a shared needle or syringe at last injection, none of the 68 IDUs who received ≥ 7 syringes per week from the program shared a needle/syringe at last injection  declare that they have no competing interests.MV and BN coordinated the study and collected the data in the field. SZ was responsible for the statistical analysis and writing the paper. MOK and SMR provided assistance in analyzing and interpreting the data. MK and MMG secured funding and supervised the collaborative socio-epidemiologic study between Japan and Iran. All authors read and approved the final manuscript.


Preventing Chronic Disease (PCD) address various aspects of program evaluation, such as planning, methods, approaches, stakeholder involvement, and the use of program evaluation findings to guide the direction of future programs.As the burden of chronic diseases in the United States continues to increase, greater efforts are being made to identify and implement interventions that successfully reduce disease risk, improve access to high-quality health care, and create sustainable health-promotion programs that ultimately improve health status and quality of life . IdentifThe articles presented in this issue have three primary goals: 1) to provide the reader with practical examples of program evaluation that can be immediately applied in other settings; 2) to carefully discuss the way program missions and objectives, stakeholder interests, evaluation theory, and evaluation methods are considered when conducting, analyzing, and reporting the status of program outcomes; and 3) to candidly explore the use of evaluation frameworks, logic models, and organizational strategic planning to increase capacity for routinely monitoring program outcomes at the national, state, and community levels.Chronic diseases such as diabetes, cardiovascular diseases (particularly heart disease and stroke), and cancer are among the most prevalent and costly of all health problems -6. More More recent public health discussions about the role of social determinants and health disparities among women and racial and ethnic minorities in the United States help illustrate the complex and dynamic aspects of chronic diseases. The discussions also emphasize the dynamic interactions between individuals and their social and physical environments . AddressThe Centers for Disease Control and Prevention (CDC) is dedicated to helping Americans live long, healthy, and satisfying lives . The orgThrough its relationship with collaborating partners, the CDC is able to provide national leadership in health promotion and disease prevention by 1) conducting public health surveillance, epidemiologic studies, and behavioral interventions; 2) disseminating guidelines and recommendations for public health interventions; and 3) helping state health departments build their capacity to prevent chronic diseases . The CDCEvaluation of national, regional, state, and community programs remains a priority of the CDC, which uses various tailored program evaluation activities designed to meet their stakeholders’ needs and capacities. In addition, demand is increasing for 1) a formal evaluation infrastructure for regularly assessing the effectiveness of public health programs ; 2) the The demand for accountability is not new to the CDC. In 2002, Milstein et al explained:With demands for accountability and results at a high level, the CDC faced the problem that many of its programs involved collaborative, multifaceted initiatives with communities across the nation and around the world. Engaging these community partners required complex approaches melding policy, structural, and individual change that were 1) implemented differently in different contexts and 2) hard to measure feasibly and consistently. Furthermore, the ultimate outcomes of interest, such as reductions in hypertension, HIV infection, obesity, or violence, were ones that might take years to materialize. The CDC remained committed to showing that its efforts as an agency were worthwhile. Yet understanding the precise effects of a single program under these circumstances proved to be an extraordinary challenge. decision and accountability, utilizations focused, client centered and responsive, case study, and outcomes monitoring and value added are a few of the evaluation approaches that have met the high program evaluation standards of usability, feasibility, propriety, and accuracy  diabetes module were used to evaluate progress toward achieving Healthy People 2010 targets. Evaluators compared 2003 data with Healthy People 2010 targets and 2000 baseline rates. The degree to which the DDT and its partners achieved the Healthy People 2010 targets is discussed, as are challenges and important factors to consider when selecting and monitoring these national objectives. Areas for future research and evaluation are also discussed.In Mukhtar et al's original research article on mixed-method approach, in this evaluation effort. Specifically, they use quantitative program performance data to identify high- and low-performance WISEWOMAN sites and use qualitative interviews, observations, and focus groups to understand underlying strategies for implementing the interventions. The authors conclude with a discussion about the relevance of using a mixed-method approach to conduct evaluation of community-based interventions.In their original research article, Besculides et al describe an evaluation approach that identifies best practices in implementing lifestyle interventions for women in the WISEWOMAN program . The autHypertension is the leading cause of stroke, coronary artery disease, heart attacks, and heart and kidney failure in the United States . ProgramIn Houston et al's community case study, the authors share with readers the evaluation of South Carolina's Diabetes Prevention and Control Program and the Diabetes Today Advisory Council's effort to conduct a 1-day conference for people with diabetes. The conference offered educational sessions on improving diabetes self-management practices . AuthorsIdeally, program evaluation should be considered at the inception of any public health program. Balamurugan et al explain that the effectiveness of programs in underserved rural areas of Arkansas was impeded because of the lack of advance evaluation planning . The autFramework for Program Evaluation in Public Health  clearance . FederalSteps to a HealthierUS Cooperative Agreement Program  focuses on chronic disease prevention and health promotion efforts to reduce the burden of diabetes, obesity, asthma, and related risk factors. In their article, MacDonald et al describe the need for the Steps Program to coordinate national and community evaluation efforts  initiative . AuthorsPractical Program Evaluation Assessing and Improving Planning, Implementation, and Effectiveness by Huey-Tsyh Chen (This issue includes Lavinghouze's book review  of Practsyh Chen . LavinghThe diverse nature of evaluation efforts undertaken by the CDC and its many partners highlights the interest and commitment to designing, implementing, and evaluating high-quality chronic disease prevention and control activities that are responsive to target audience and stakeholder needs. The use of evaluation is being integrated into the accountability movement and is embedded in a consumer-oriented public health ideology . It is b