Environmental Protection Agency (EPA) Officeof Research and Development, a subcommittee of the Board of ScientificCounselors Executive Committee conducted an independent and open peerreview of the Endocrine Disrupting Chemicals Research Program (EDC ResearchProgram) of the U.S. EPA. The subcommittee was charged with reviewingthe design, relevance, progress, scientific leadership, and resourcesof the program. The subcommittee found that the long-term goalsand science questions in the EDC Program are appropriate and representan understandable and solid framework for setting research priorities, representinga combination of problem-driven and core research. Long-termgoal (LTG) 1, dealing with the underlying science surroundingendocrine disruptors, provides a solid scientific foundation for conductingrisk assessments and making risk management decisions. LTG 2, dealingwith defining the extent of the impact of endocrine-disrupting chemicals (EDCs), hasshown greater progress on ecologic effects of EDCscompared with that on human health effects. LTG 3, which involves supportof the Endocrine Disruptor Screening and Testing Program of theU.S. EPA, has two mammalian tests already through a validation programand soon available for use. Despite good progress, we recommend thatthe U.S. EPA  At the request of the U.S. Environmental Protection Agency (EPA) Officeof Research and Development (ORD), the Board of Scientific Counselors (BOSC) ExecutiveCommittee  organizeThe subcommittee was charged with reviewing the design, relevance, progress, scientificleadership, and resources of the program  and provIn the early 1990s scientists began to synthesize information about thepotential impacts of endocrine-mediated toxicity on humans and wildlife, arrivingat the hypothesis that weakly endocrine-active compoundsin the environment were having significant adverse effects on public andenvironmental health . In respA recent publication identifies 10 key areas of uncertainty that must beaddressed to determine the significance of endocrine disruptors as publicand ecologic health threats . The relThe peer-reviewed blueprint for the U.S. EPA EDC Research Program was publishedin 1998 as the ORD Research Plan and took into considerationthe advice provided to OPPTS on the implementation of the legislationby an independent expert advisory panel . Five yeThe purpose of the MYP is to provide a framework that integrates researchacross the laboratories and centers of the ORD to produce scientificallycredible results in accordance with the Government Performance andResults Act  goals anIn addition to the MYP, which identifies the EDC research directions forall the laboratories and centers of the ORD, some ORD organizationshave developed their own implementation plans. For example, the NationalRisk Management Research Laboratory (NRMRL) of the ORD has developeda Risk Management Evaluation , and theThe subcommittee reviewed materials sent by the U.S. EPA, including theResearch Plan , the MYPThe U.S. EPA staff provided an overview of the EDC Research Program andthe LTGs, and poster sessions were presented and discussed by intramuraland extramural researchers, program and regional office scientists, andgrantees. Poster sessions were followed by presentations by representativesfrom program and regional offices who spoke to the relevanceof the research program. The meeting included an opportunity for publiccomment. At the conclusion of the meeting, the subcommittee presenteda draft oral report of its findings.The subcommittee organized the review based on the three LTGs presentedin the MYP , commentLTG 1 strives to provide the science underlying the effects, exposure, riskassessment, and risk management of EDCs. The goals set forth in theResearch Plan and the MYP to address the underlying science needs forrisk assessment and management of EDCs continue to be appropriate. Theresearch and implementation plans to achieve these goals are wellfounded and provide a logical framework for attaining the Research Plangoals.LTG 1 is well designed and takes advantage of existing core competenciesin reproductive toxicology, mechanistic toxicology, ecotoxicology, riskassessment, and risk management methodology to address these questions. Thecapabilities of these scientists are unique in breadth, depth, andscope within the federal government. No other federal agency isequipped to provide answers regarding both risk assessment and managementof EDCs. Thus, the outcomes of the Research Plan continue to provideessential, fundamental scientific support for other regulatory andresource management agencies, both federal and state, as well as externalinvestigators and industry.The EDC Program has relied on the Science to Achieve Results (STAR) grantsprogram to conduct research and provide expertise to achieve program-relatedoutcomes related to LTG 1. STAR grant recipients have contributedimportant findings on many topics including interspecies differencesin steroid receptors, avian and invertebrate models for EDC evaluations, andthe effects of multiple EDC exposures. The Research Planhas also utilized and relied on the skills and abilities of scientistsfrom other federal agencies to complement some activities in this researcharea. The STAR program, therefore, is an essential element in theEDC Program to continue to meet its goals outlined in the Research Plan. Oneexample of the plan’s reliance on extramural expertiseis in avian toxicology. The expertise to conduct avian toxicology studiesin the laboratory or in the field does not reside within the U.S. EPA/ORD.The science conducted under LTG 1 is unique and provides the foundationrequired for future risk assessment and risk management activities legislativelymandated to the U.S. EPA. Work on development of models  is ongoing, with appropriate endpoints to help identify and evaluate uncertainties for human and ecologicrisk assessment of EDCs. The end points chosen are relevant for theecologic risk assessment process. Model compounds include traditionalorganochlorine pesticides and industrial compounds, positive controls (estrogensand androgens), current use pesticides , andthyroid-active agents. The chemicals chosen are timely and importantrelative to exposure, and low-dose effects and latent effects arebeing addressed in a rigorous manner. Thus, a good deal of high-qualitydata are being developed under LTG 1, providing the foundation for environmentalrisk assessments and risk management of EDCs.The Research Plan has developed critical and relevant information on modeof action, interspecies differences, multiple chemical exposures, criticallife stages, dose–response characteristics, effects atmultiple levels of biological organization, linkages among assessmentend points, and low-dose effects of EDCs. All of these findings are requiredfor the appropriate evaluation and risk assessments of EDCs.The scientific expertise available in ORD and some of the external STARrecipients are a primary strength for LTG 1. The research program structureand implementation is logical and well designed, adding strengthsin this key area. Additionally, the models that are being characterizedto evaluate EDCs under LTG 3 will complement the on-going effortsin LTG 1.The EDC Program and scientists within ORD provide strong leadership inthe design and execution of research efforts in this area. The modelsthat have been designed or modified to evaluate endocrine disruption willprovide the data required to develop risk assessments. The modelshave been the subject of harmonization efforts to be compatible with Organisationfor Economic Cooperation and Development (OECD) guidelinesfor toxicity testing.The risk management program provides important information for the regulatedcommunity with regard to identifying and prioritizing EDCs of concernand leadership in developing risk management approaches.The complexity of endocrine systems, in conjunction with the diversityof potentially endocrine-active chemicals makes the evaluation of combinedeffects of EDCs a daunting task. Even with a solid approach, goodlaboratory models, and adequate funding, it is likely that it will besome years for ORD to fully evaluate this question.Scientific expertise for the areas of human and aquatic  species is very good. However, this same strengthis not apparent in the area of wildlife toxicology, and much of theexperimental research and expertise comes in the way of STAR grant recipients. Itis certainly advantageous to utilize the expertise of thesescientists from outside the agency; however, more expertise in thearea of wildlife toxicology within the agency will be required to fullyattain the goals within this program and meet the exact needs of regulatoryconcern. In addition the evaluations of EDCs on wildlife withina risk assessment paradigm, including evaluation of uncertainties, wouldalmost certainly require full-time U.S. EPA personnel. Because ofthe complexities in extrapolating among the many species in the environmentthat may be affected by endocrine disruptors, it will be importantfor ORD to continue to collaborate with other federal, academic, andnongovernmental organizations, and industry partners to better characterizethe range of variability among species.The model and framework for development of critical information on EDCsfor risk assessment is well established and making progress. Effortsshould now focus on development of risk assessment paradigms for EDCsand application of the research findings. A major challenge for risk assessmentwill be to settle on definitions of what constitutes an adverseeffect and what constitutes a biological indicator. This challengedoes not reside exclusively within ORD, but ORD research will be neededto support decisions in this area.The development of analytical methods for detection and quantitation ofEDCs is a significant challenge and was not clearly identified as anannual performance goal or annual performance measure for this research. Thismay slow the progress of the risk management program. In addition, studiesconducted by investigators outside the U.S. EPA have reportedthe use of predictive tools that can be used to prioritize the focusof selective water and wastewater treatment technologies. These findings, aswell as plans for future research regarding natural processesin sediments, could be integrated into this work.LTG 2 seeks to determine the extent of the impact of endocrine disruptorson humans, wildlife, and the environment. The subcommittee evaluationof the Research Plan and the MYP found that the goals and science questionsare appropriate and represent an understandable and solid frameworkfor setting research priorities for endocrine disruptors. The ResearchPlan has stood the test of time and is appropriately reflectedin the MYP for LTG 2.The presentations and posters under LTG 2 represented primarily issuesof environmental and human exposures to actual and suspected EDCs, andthe spectrum of effects that might be produced from those exposures. Thereis obvious overlap with the other long-term goals. For example, oneof the key science questions of the MYP for LTG 2 is to determinehow and to what degree human and wildlife populations are exposed to EDCs . This qua) study chemicals with known EDC activity and b) evaluate the endocrine activity of emissions and releases from differentsources followed by attempts to identify the chemicals responsiblefor the observed activity. Both approaches are needed, but the U.S. EPAshould not lose sight of the goal of determining chemical classes ofinterest and the sources of EDCs.In the case of environmental releases and ecologic effects, the U.S. EPAhas taken two approaches: The U.S. EPA research program relevant to the science questions containedin LTG 2 has been productive, of high quality, and relevant to themission of the U.S. EPA. Available resources have been used efficiently, andboth the intramural and extramural investigators demonstrate ahigh degree of enthusiasm for the projects. In general, greater progresshas been made on ecologic effects of EDCs compared with human healtheffects, although several appropriate human health projects are underway.Many of the models required for studying fish and invertebrate effectsof EDCs have been developed, modified, and/or applied. Ongoing studieshave set appropriate priorities for determining sources of EDC exposuresincluding concentrated animal feedlot operations, combustion processes, andpulp mills. The ecologic studies have effectively coupled fieldstudies, biomarker measurements, analytical chemistry, laboratorystudies with whole organisms, and hormone-responsive cells with effluentssuspected of possessing hormonal activity.U.S. EPA scientists have made good decisions on how to best use genomics, witha good balance between molecular toxicology and effects on aquaticspecies and experimental animals. The information generated fromthese studies should produce important data that address critical knowledgegaps.Interactions between ORD and the regional offices on EDC issues are strong, effective, andfrequent and provide a good model for the U.S. EPAto use in other areas. The research within LTG 2 is consistent with theoverarching Research Plan developed by the U.S. EPA and other agenciesin 1998, with research priorities reflecting the high-priority goalsidentified by that plan.Although priorities for chemicals studied have been appropriate, the U.S. EPAshould strive to continue to improve its interactions with otheragencies with a strong interest in the EDCs, such as the Centers forDisease Control and Prevention (CDC), National Institute of EnvironmentalHealth Sciences (NIEHS), National Toxicology Program, National Institutefor Occupational Safety and Health (NIOSH), and U.S. Food andDrug Administration (FDA). These collaborations will facilitate identifyingnew sources of environmental and human exposures to EDCs. Moreover, theU.S. EPA should mine data made available from the High ProductionVolume program  and workThe epidemiology studies represent an important component of the EDC Programrelevant to LTG 2. The U.S. EPA is encouraged to continue thesestudies and to use the exposure results to set priorities for future epidemiologystudies so that duplicative studies are kept to a minimum. TheU.S. EPA should continue to investigate the common ground betweenecologic and human health, as no other agency’s mission providesthis opportunity.It will be important for the U.S. EPA to take a leadership role in theapplication of “omics” technologies to address many ofthe science questions critical for evaluating environmental and humanhealth effects of EDCs. Although there was evidence of considerable progressin this regard, future development of this approach will requirea strong commitment to a systems biology approach and computationaltoxicology as well as effective interactions with those generating muchof the basic data.The screening and testing program of the U.S. EPA was established to complywith the The recommendations of EDSTAC have served as the basis for the U.S. EPAEDSP . ORD hasThe research on screening and testing is essential to the mission of theU.S. EPA and to the mandates given to the U.S. EPA under the The program plan with respect to LTG 3 exceeds the explicit recommendationsof EDSTAC and takes advantage of improvements in the science, especiallyin the realm of computational biology. The U.S. EPA has recentlylaunched a national, multilaboratory computational toxicology programthat stands to contribute significantly to endocrine disruptor screening, particularlythrough the development of quantitative structure–activityrelationship (QSAR) models.The progress on LTG 3 within the EDC Program has been excellent. Two mammaliantests have already been through a validation program administeredby the OECD. These should be available for use by the EDSP very soon. Developmentof the other two tests recommended by EDSTAC is in progress, andpublications emanating from this work indicate that the workis on track.There has been significant progress within ORD and its scientific partnersin the development and validation of several relevant bioassays importantfor the screening and testing requirements for LTG 3. The uterotrophicassay for estrogenic effects and the Hershberger assay for androgeniceffects have been the subject of multilaboratory, multinationalvalidation programs coordinated by OECD, with considerable guidancefrom ORD scientists. The pubertal male and female assays, which evaluatethe attainment of puberty in rodents and are semiapical in that theyassess the integrated function of a number of mechanisms of action , are still under developmentvia ORD research programs. ORD included the completion of theseas short-term goals in its MYP.In vitro assays for androgen receptor (AR) and estrogen receptor (ER) binding havebeen developed and validated. Moreover, in vitro AR- and ER-dependent transactivation assays in transformed cell linesare now available, and these use both stable and transiently transfectedcells. These assays, coupled with ongoing studies in other laboratories, suggestthat this important screening component of LTG 3 is nearlycomplete. In addition, an in vitro assay for determining the effects of EDCs on steroidogenesis has beendeveloped, and the approach will be capable of measuring both modulationof steroidogenic gene expression and activity. This bioassay seemshighly promising and requires further validation using more extensivesets of test EDCs and possibly development of alternate cell lines todetermine possible intercellular differences in response to EDCs. Excellentprogress has also been made on validation of short-term in vivo assays for the determination of estrogenic/antiestrogenic and androgenic/antiandrogenicchemicals, using the rat as a model.The EDC research is mechanistically driven, which provides a solid scientificfoundation for the test methods that are developed. Because ofthis mechanistic focus, it is highly likely that the methods developedwill be valid, broadly applicable, and easily interpreted.Clear goals are articulated for the development of screening and testingmethods for endocrine disruption. U.S. EPA research is well coordinatedwith other federal agencies and with international efforts on thestandardization and validation of endocrine-screening assays. ORD hasbeen highly responsive to the needs of the EDSP  and has a) a substantial commitment by the U.S. EPA staff to refine and troubleshotassays, and b) a negative effect on other core research activities that are the responsibilityof the U.S. EPA staff. The subcommittee recommends that therebe a mechanism in place to ensure the timely transition of protocolsto the OPPTS.The major challenge that ORD has faced is handing off its research to theprogram offices so that validation and implementation can occur ina timely manner. Much of the delay in validation and regulatory acceptance, however, isbecause this process takes place largely outside theagency. The transfer of protocols to contract laboratories has been problematic. Thishas led to Research within NHEERL has contributed to basic understanding of the toxicresponses to estrogens, antiandrogens (within the Reproductive ToxicologyDivision) and thyroid toxicants , which in turn has led directly to the development ofimproved methods for EDC detection. This research is diffuse and is occurringin multiple divisions within NHEERL; many of the accomplishmentsin these areas have been difficult to capture in the list of annualperformance goals. The subcommittee recommended that the U.S. EPA tryto summarize this research and its relevance to EDC identification insubsequent reports and revisions of the MYP.ORD is beginning to develop core competencies in genomics and QSAR methods, bothof which hold promise in EDC identification. Because these areasare so data intensive, it will be important for ORD to train or hireexperts in bioinformatics to work with the life sciences experts alreadyon staff.The EDC Program has enjoyed outstanding leadership since its inceptionin 1995. The EDC Program scientists consult with and provide technicalassistance to other U.S. EPA program offices, other federal agencies, andwithin the broader scientific community. EDC scientists are engagedin intramural and extramural research within ORD, and program scientistshave provided exemplary leadership in the field at the nationaland international level. The EDC scientists are at the forefront of researchin this field in EDC screening and testing methodologies for mammalianand ecologic tests, source identification, effects on wildlife, andecologic health.The EDC Program is unique in that no other U.S. federal agency has an EDCprogram with such broad responsibilities. The EDC Program is not justan umbrella for a series of independent project but a fully integratedprogram across all the laboratories and centers (with the exceptionof the Homeland Security Research Center). The program is nationallyand internationally recognized as a multidisciplinary set of researchprojects for both human health and wildlife and cuts across the riskassessment/risk management paradigm.The EDC Program was projected to have an average annual budget of $12 million. Thisfigure includes the STAR grants program, which averages $4 millionin years when it is funded. In actuality, theaverage annual budget from fiscal year (FY) 2003–2005 has rangedfrom $12.7 million enacted in 2003 to the FY 2005 requestof $8.0 million, which includes approximately 55 full-time equivalentpersonnel per year. The EDC Program director does not have directaccess to human or financial resources to carry out the objectivesof the program. Instead, the director must negotiate with the divisionheads of the laboratories and centers of ORD to use the time and effortof scientists with the needed expertise.The laboratories and center that contribute resources to the EDC Programare NRMRL, National Center for Environmental Research (NCER), NERL, andNHEERL. Although the total budget for the program has decreased sinceFY 2003 (from $12.7 million to an $8.0 million requestfor FY 2005), the percentage of resources provided by each of theselaboratories has been relatively stable (except for NCER) over thepast 2 years and is in proportion to the number and extent of tasks thatthey perform for the EDC Program across the long-term goals. The STARgrants program adds significant value to the research portfolio ofthe EDC Program. The research sponsored by the STAR program assists infilling identified research gaps, brings in research expertise that isnot found among intramural scientists, and assists the ORD in respondingto new issues that the laboratories and centers may not be able toreadily address.The manner in which this program is funded, though indirect and possiblycumbersome, does not appear to hinder the quality of the research beingdone in the program. It is apparent that the EDC Program directorhas had success in convincing division directors to loan scientist timeto the EDC Program. It does make it more difficult, however, for theprogram director to do forward planning or to plan to investigate emergingissues. ORD has been very astute in leveraging the resources ofthe EDC Program by collaborating with other federal agencies. The amountof research done by the EDC Program has been expanded by collaborationwith agencies such as NIOSH, NIEHS, and the National Cancer Institute; however, thefragmentation of scientists’ time without compensationraises concern about whether the productivity  of these scientists is negatively impactedby participation in the EDC Program.a) hiring additional personnel to share the workload of the participatinglaboratories; b) elevating the position of the EDC Program director to the level of thelaboratory/center directors; and c) giving the EDC Program director budget authority. These actions wouldallow the program director to negotiate for needed research expertisefrom a position of strength and allow the program director to enhancethe laboratories that participate in EDC Program research. The ORD hasplans to enact the latter two of these ideas in the near future forall newly hired national program directors.The situations cited above (insufficient funding and the mechanism usedto provide resources to the EDC Program) can be remedied by several coursesof action: The goals and science questions are appropriate and represent an understandableand solid framework for setting research priorities for EDCs. TheResearch Plan was formalized in 1998 after a series of workshops, interagencyconsiderations, and meetings that embraced all relevant stakeholders. Theprogram is nationally and internationally recognizedas a multidisciplinary set of research areas for both human health andwildlife and cuts across the risk assessment/risk management paradigm. Keyresearch areas are closely aligned to the LTGs and annual performancegoals. The EDC Program is a combination of problem-driven and coreresearch that has stood the test of time.The subcommittee is favorably impressed with the quality and relevanceof the work and the progress to date, although we recognize that muchremains to be done. The annual performance goals are highly ambitiousand it should be recognized that progress on those goals will, in mostcases, continue well past the initial timelines. We are impressed withthe enthusiasm of the investigators and their commitment to addressingthe difficult and controversial issues that surround endocrine-disruptorresearch.One of the main functions of the U.S. EPA is risk assessment and risk managementof chemicals in commerce and the environment. LTG 1 providesa solid scientific foundation for conducting risk assessments and makingrisk management decisions related to endocrine disruptors. The researchthat falls under this goal covers the major questions in the keyareas of the risk paradigm. Although there are many challenges to fullydefining the nature of possible biological effects and the extent ofexposure, the research being carried out under this long-term goal willput the U.S. EPA in a strong position to make scientifically groundeddecisions.The research program relevant to the science questions contained in LTG 2 hasbeen productive, of high quality, and relevant to the mission ofthe U.S. EPA. Available resources have been used efficiently, resultingin a high degree of enthusiasm for the projects by both the intramuraland extramural investigators. In general, greater progress has beenmade on ecologic effects of EDCs compared with human health effects, althoughseveral appropriate human health projects are underway.The progress on LTG 3 within the EDC Program has been excellent. Two mammaliantests have already been through a validation program administeredby OECD. These should be available for use by the EDSP very soon. Developmentof the other two tests recommended by EDSTAC is in progress, andpublications emanating from this work indicate that the work isprogressing appropriately on track. ORD has articulated clear goals forthe development of screening and testing methods for endocrine disruptionand is fulfilling those goals in an admirable fashion. The researchis directly relevant to legislation that the U.S. EPA administersand is serving the program offices well. Research of the U.S. EPA iswell coordinated with other federal agencies and with international effortson the standardization and validation of endocrine-screening assays.a) strengthening their expertise in wildlife toxicology; b) expediting validation of the EDSTAC tests; and c) taking a leadership role in the application of omics technologies toaddress many of the science questions critical for evaluating environmentaland human health effects of EDCs. In addition, support from U.S. EPAmanagement, multidisciplinary intramural research spanning ORD andother U.S. EPA entities, extramural grants, and continued interagencycollaborations , especially with regard toidentifying new sources of environmental and human exposure, will becritical. To date, the EDC Program has been very astute in leveragingits resources by collaborating with other federal agencies and has extendedits research program as a result of these collaborations. The continuationof the external grants program, in particular the STAR grantsprogram, is vital, as it provides a mechanism for the U.S. EPA to moreefficiently evaluate new technologies and innovations for use in therisk assessment and risk management arenas.Despite good progress, the future success of the U.S. EPA in meeting thespecified goals of the EDC Program will depend on a number of factors, including


Little is known about how sexual and reproductive (SRH) health can be made accessible and appropriate to adolescents. This study evaluates the impact and sustainability of a competitive voucher program on the quality of SRH care for poor and underserved female adolescents and the usefulness of the simulated patient (SP) method for such evaluation.28,711 vouchers were distributed to adolescents in disadvantaged areas of Managua that gave free-of-charge access to SRH care in 4 public, 10 non-governmental and 5 private clinics. Providers received training and guidelines, treatment protocols, and financial incentives for each adolescent attended. All clinics were visited by female adolescent SPs requesting contraception. SPs were sent one week before, during (with voucher) and one month after the intervention. After each consultation they were interviewed with a standardized questionnaire. Twenty-one criteria were scored and grouped into four categories. Clinics' scores were compared using non-parametric statistical methods (paired design: before-during and before-after). Also the influence of doctors' characteristics was tested using non-parametric statistical methods.Some aspects of service quality improved during the voucher program. Before the program started 8 of the 16 SPs returned 'empty handed', although all were eligible contraceptive users. During the program 16/17 left with a contraceptive method (p = 0.01). Furthermore, more SPs were involved in the contraceptive method choice . Shared decision-making on contraceptive method as well as condom promotion had significantly increased after the program ended.Female doctors had best scores before- during and after the intervention. The improvements were more pronounced among male doctors and doctors older than 40, though these improvements did not sustain after the program ended.This study illustrates provider-related obstacles adolescents often face when requesting contraception. The care provided during the voucher program improved for some important outcomes. The improvements were more pronounced among providers with the weakest initial performance. Shared decision-making and condom promotion were improvements that sustained after the program ended. The SP method is suitable and relatively easy to apply in monitoring clinics' performance, yielding important and relevant information. Objective assessment of change through the SP method is much more complex and expensive. Nicaragua has one of the highest adolescent fertility rates of Latin America, with 119 births annually per 1000 young women aged 15–19. By the age of 19, 45% of the girls are either pregnant or have already given birth. High fertility rates are associated with low socio-economic status and low educational attainment [Major reasons for the low use of contraceptive methods among adolescents are lack of access to information and services on sexual and reproductive health (SRH) and a low quality of care. While obstacles to appropriate care have been documented extensively -5, thereEvidence of interventions that have succeeded in improving the quality of SRH care for adolescents in existing health centres in Latin America is scarce. Indications existed that the competitive voucher program for sex-workers in Managua, Nicaragua, succeeded in improving the quality of care for these women ,9, howevThe evaluation showed that voucher receipt increased use of SRH care among female adolescents  and of contraceptives and condoms in specific groups . FurtherThe objective of the simulated patient (SP) study reported here was to assess whether the technical quality, the content of the communication and the general treatment were improved by the voucher program, and whether any change was sustained after the intervention ended. The second aim of this exploratory study was to evaluate the usefulness of the SP method to evaluate such changes. SPs were sent to the participating clinics, before, during and after the intervention to request a contraceptive method. This paper reports on their experiences with the health care and our experiences with the SP method.The intervention took place in Managua, the capital of Nicaragua, one of the poorest countries of Latin America. Primary health services in Managua consist of public health centres run by the Ministry of Health, municipal public health centres, private clinics, and clinics run by non-governmental organisations (NGOs). Most participating clinics are staffed by two doctors, and in most larger clinics two doctors were allocated to receive adolescents.and follow-up visit was US$ 4.56. Adolescents were free to redeem their voucher in a clinic of their choice, and the clinics received reimbursement for each adolescent consultation with a voucher. The voucher program started with four clinics and new clinics were added periodically.Over fifteen months, 28,711 vouchers were distributed to adolescents aged between 12 and 20, at markets, outside schools and door-to-door in disadvantaged neighbourhoods. The only selection criterion was age. The vouchers gave free of charge access to SRH care in any of the four public, five private or ten NGO clinics contracted by ICAS, for one consult and a follow-up visit. The selection of clinics was based on suitability and proximity to the areas where vouchers were distributed. Identified clinics were invited to participate, and the price per consultation was negotiated based on their customary fees. The average price negotiated per consultation Vouchers were valid for three months and 20% of the vouchers distributed to girls were redeemed (boys 6%). This is a relatively high redemption rate considering the limited period over which the vouchers were valid and that a large percentage of the receivers was not yet sexually active. Among sexually active girls, 51% used their voucher, while among girls who were not yet sexually active, use was only 14% .The voucher program addressed various aspects of quality of care. Doctors had to complete a standardized clinical form based on 'best practice' protocols, which guided them during each consultation. The form was designed to ensure that every adolescent was asked about their sexual activity; their need for information; their need for contraceptives; and was given a package with two condoms plus health education material on adolescence and STIs. Contraceptives, drugs for syndromic treatment of STIs and laboratory tests for pregnant girls were made available through the program. It was assumed that the competitive nature of the voucher program would prompt providers to improve the quality of services to attract more voucher users . DoctorsIn the SP method patients are 'standardized' making it possible to compare the performances of different doctors or of one doctor at different moments. The use of the SP method in developed countries has proven to be reliable, valid, feasible and acceptable -15. The All 19 participating clinics signed a contract of agreement that included information that the quality of care would be evaluated on a regular basis. They were not informed specifically on the methods to be used. The week before the intervention started in a certain clinic, a SP without a voucher was sent. During the course of the program, SPs were sent with a voucher to monitor the quality of care provided under the program. One to two months after the program ended, SPs without a voucher visited each clinic again.The SPs were female adolescents active in NGOs involved in the voucher distribution, and university students participating in social work activities. They were contracted on a day-to-day basis and were paid a days' wage and enough money to pay for the consultation fees, if needed, and transportation costs. In total, 17 SPs took part in the survey. Their mean age was 18.7 years (range 16–22 years) and seven had secondary school education and 10 studied at the university. The mean number of doctors visited was 3.5 (range 1 – 12).All SPs received a short training on the task to be performed. The objective of the study was explained and the girls were instructed to present a story that over the past three months they had formed a relationship with a boyfriend and were presently using withdrawal or periodic abstinence, but did not want to risk pregnancy. Furthermore, they had to ask for information on HIV/AIDS. All were told that they were free to accept or reject any information or services offered to them. If necessary, they were advised to decline vaginal examination by saying that they were currently menstruating.The SPs were not accompanied to the clinic. If needed, the SP would make an appointment. After the consultation, the SPs were immediately interviewed by one of the two doctors of the research team at the ICAS office, using a standardized questionnaire. The answers were recorded by the interviewer on the questionnaire. The study was approved by the ethical review committee of ICAS.The questionnaire contained 35 open-ended questions and included commonly used elements for evaluating quality of family planning services: choice of contraceptive methods, information given to users, technical competence, interpersonal relations, mechanisms to encourage continuity and appropriate organisation of services . QuestioThe 21 criteria for evaluation are based on the program objectives, and were extensively discussed by the research team and defined before the analysis was performed. They are grouped into four categories. Two categories relate to contraception: A. method choice and B. continuity in use; category C. relates to prevention of STIs; and D. to the organisation of the services. The last category is composed of a selection of criteria that have been found to influence patients' perception of the quality of care, but do not relate directly to the doctors performance. A maximum score of 10 points could be attained if there was a positive response for all criteria in one category. The addition of the scores of the first three categories serves as an indicator of the quality of the consultation. For more details on the components of each category see Tables Information on date of interview, clinic, and doctor visited was disconnected from the file, and records were put in a random order. Codification was done by one person, and checked by another. All data were entered twice in Epi-info 6.04 by two different data processors. Stata 7.0  was used for further analysis.The response to each criterion is tabulated in table To get an indication as to whether doctors' characteristics influence their performance while counselling for contraception and STI/HIV, the sum scores of the three categories reflecting the quality of the consultation are calculated, at the three relevant time-points. Because the doctors are not necessarily the same at the different time-points, only comparisons per time-point are made. Per time-point, the non-parametric Mann Whitney test is used to assess the influence of doctors' gender and the Kruskal-Wallis test to assess the influence of the age group and the type of clinic on their responses. Since doctors had to complete a medical form for all patients with a voucher, this included the SPs. These forms were used to evaluate the SPs' performance and doctors' registration.before the intervention. In one clinic, the SP was received by the counsellor instead of the doctor, one was sent too late and treated according to the voucher protocol, and one is missing for unknown reasons. During the intervention, 17 clinics were visited by 30 SPs with a voucher. One of the clinics was excluded due to administrative problems and one resigned from further participation. After the pilot, one clinic was temporarily closed and another permanently closed. Eighteen SPs visited the remaining 15 clinics.All clinics received SPs. In total 65 visits were made to 16 clinics before, 17 during and 15 after the intervention. Seventeen SPs visited 16 of the 19 participating clinics 18%) of the doctors worked in a public facility; 19 (58%) in an NGO clinic and 8 (24%) in a private clinic. Sixty-one percent are female and 39% younger than 35 years, broadly conforming to the gender and age profile of primary health care doctors in Nicaragua.Thirty-three of the 40 doctors who participated in the voucher program received at least one SP. Their characteristics are similar to the entire group of participating doctors. Six . However, the number of barriers to use diminished. Barriers related to unnecessary diagnostic tests disappeared during the voucher program , but returned after the program.There was considerable variation in the methods recommended by the doctors. Some doctors promoted the condom for its safety or dual protection, where other doctors advised that condoms were unsafe, boyfriends would resist using them, or warned about possible allergic reactions. The same variation was seen in methods such as the intra-uterine device and monthly injectables. Before and after the program, some doctors recommended natural methods like periodic abstinence as best method for adolescents. The quality of the explanations given improved during the voucher program and remained better after the program ended .Before the intervention started, most SPs indicated that it was the doctor who decided on the type of contraceptives to be used. Patients with a voucher reported more frequently that they made the choice jointly with the doctor . The percentage of SPs reporting shared decision-making remained relatively high after the intervention (p = 0.03).Although all SPs were eligible contraceptive users, before the program eight of the 16 SPs left the consultation room without a contraceptive method or prescription. Six of these eight SPs were told to come back on the first day of their next menstrual cycle. One doctor preferred to wait for the results of unnecessary laboratory tests  and one advised periodic abstinence (the 'rhythm method').During the voucher program, 94% of the girls left the clinic with a contraceptive method . This improved practice continued after the program ended (p = 0.10), but the frequency of follow-up appointments decreased .Although the percentage of doctors discussing STI/HIV-risks during a consultation increased, a considerable number of patients did not receive the desired information. More than 10% of the SPs in each round felt the doctor was in a hurry to end the consult. Some felt pressured to the extent that they did not ask their question on STIs.Although in less than half of the consultations before, during and after the intervention SPs were satisfied with the explanations provided, patients seen after the intervention more frequently received correct information than before  and condom promotion increased from 53% to 93% .There was considerable variation in the content of the information on prevention on STIs/HIV. Some doctors gave good and reliable information where others confined themselves to observations like: "One has to know the person with whom one is with"; "The only way to prevent AIDS is not to sleep with someone who has AIDS"; "One should not use clothes from other persons"; or "If someone's partner is sweating and having weight loss he might have AIDS".No clear differences in the organisational aspects of the clinic could be established, except that privacy during consultation appeared better respected after than before and during the intervention (p = 0.08).Patients with a voucher were not always treated according to the contractual agreements. SPs reported that receptionists sometimes completed considerable parts of the medical forms and that other patients were favoured in the waiting room.Experiences of the SPs with vouchers suggest that some private clinics tried to make as much money as possible out of the program by erroneous declarations, and by selling or withholding the materials provided by the program.The gender of the doctor was strongly associated with the results. Female doctors scored better than male doctors before, during and after the intervention. Male doctors improved by a greater degree during the voucher program. However, these changes were not sustained except for the category on STI/HIV prevention (not shown). Elderly doctors had the lowest initial scores, which improved during the program, but still remained the lowest. Doctors younger than 35 improved most during the voucher program. In contrast to the analyses mentioned above, individualdoctors arehere the unit of analysis. The results are reflected in first part of table Comparing the public, NGO and private clinics, the NGO clinics performed best before the intervention. During the intervention, the improvements observed were more pronounced among the public and private clinics, that is, the clinics with weakest initial performance. The performances became of a comparable quality during the intervention, but the improvements were not sustained after the intervention ceased . Judged from the medical records of patients who went with a voucher to the clinic, the SPs performed their role correctly.SP studies designed to measure differences usually limit themselves to record doctors' actions because inter-observer variability for criteria related to doctor-patient interaction is much higher ,14,21,23We cannot exclude that the simulated patients were treated differently because of their somewhat higher level of education and higher age than the average voucher redeemer. However, accessing sexual and reproductive health care is difficult for many adolescents in Nicaragua, even for those who attended secondary school. Because the same group of girls participated in the pre- during and post assessment, the changes observed do not relate to their background.Significant changes were foremost observed in criteria directly influenced by the program, such as how a decision on type of contraceptive method was taken and the improved delivery of methods. This may indicate that these criteria were the only ones influenced by the program. Because we compared clinics and not individual providers and the sample size was modest, it is also possible that less pronounced effects, or effects with lower inter-observer and/or inter-provider agreement, may not have been detected.The voucher program had a substantial impact on the number of women that left the clinic with a contraceptive method. This is in accordance with the programs instruction to give any women who requested a contraceptive method, a one month supply of that method with clear instructions on when to start (first day of the next menstruation) and to use condoms before. Patients were asked to return during the month following their menstruation and to relate their experiences. In this return visit, still covered by the voucher, patients could then receive a three months supply.Clinics and doctors with weakest initial scores showed biggest improvements, suggesting they made serious efforts to comply with the program requirements. Unfortunately, most changes did not sustain beyond the program. Furthermore, even during the program there is room for further improvement. A longer intervention period and specific strategies to enhance quality improvements might realize more substantial and more sustainable changes. The voucher program carries this possibility. On one hand, the participating doctors were enthusiastic about the program, appreciated the experience, and were interested to improve their communication and other skills . On the Another strategy that could be combined with the voucher program is explicit use of the criteria for 'adolescent friendliness' . RecentlPreparing and sending SPs, and recording their experience is the simplest part of the SP method. Their experiences provide valuable information about the actual performance of health services. The SP method, when openly employed, can furthermore be a useful support for regular feedback and identification of further training needs. It becomes more complicated when the experiences of SPs are to be converted into indicators to assess and evaluate changes in quality of care in a scientifically correct manner. This methodological challenge is insufficiently emphasized in the literature. Although the SP method is particularly recommended for evaluating programs with a focus on the interaction between patients and doctors, difficulties and solutions related to standardized assessment of the inherently subjective patient's opinion are not reported. This is an interesting field for future research in view of the importance of patients' perception in health care use.Assessing quality improvements -including through the SP method- is complicated and requires highly trained human resources, time and financial resources. Although many health care interventions in developing countries aim to achieve quality improvements, programs are seldom supplied with sufficient funds to make such valuable and important evaluations possible.The adolescent voucher program seemed to have succeeded in increasing the accessibility of contraceptives through health services, and had the greatest impact on the quality of care in clinics with lowest initial quality levels. This is an important achievement, because if unwanted and untimely pregnancies are to be prevented, adolescents who engage in sexual relations need access to information and contraceptive methods.Although in Nicaragua wide agreement exists that health care services should respond to these needs -as is explicitly outlined in norms and guidelines of the Ministry of Health  – implemThe SP method proved to be a valuable and feasible method for program monitoring in a developing country. The results of this pilot study suggest that scaling up of the voucher program could be an effective strategy to make SRH care accessible, acceptable and appropriate for underserved teenagers.AIDS Acquired Immune Deficiency Syndrome.DFID The British Department for International DevelopmentFP Family PlanningHIV Human Immune Deficiency VirusICAS instituto CentroAmerica de Salud, Central American health instituteNGO non-governmental organisationsRTI Reproductive tract infectionSP Simulated patientSRH Sexual and reproductive healthSTI Sexually transmitted infectionsThe author(s) declare that they have no competing interests.LE M was involved in all stages of the process from conception and design of the survey, the analysis and interpretation of data. AC G was involved in the conception and design, the analysis and interpretation of the data, participated in writing and revising the paper critically for substantial intellectual content until the final approval. ADM K was involved in the statistical analysis and interpretation of the data, participated in writing and revising the paper critically for substantial intellectual. JA K gave support in the development of the research, the analysis and interpretation of the data, and contributed substantially in preparing the manuscript, revising it critically for substantial intellectual content. All authors read and approved the final manuscript.The pre-publication history for this paper can be accessed here:


It is a time of introspection for the Superfund Basic Research Program, auniversity-based grants program established in 1987. While maintainingthe program’s premise of supporting basic research for practicalapplication to address the nation’s problems associatedwith hazardous waste sites, the program is continuously evolving anddeveloping new approaches to address these problems. A course was setto allow the Superfund Basic Research Program to award grants annuallyand to diversify funding mechanisms. With this announcement of the resultsof the most recent competition, the program is now poised to meetthese goals.We are pleased to announce that NIEHS has made awards to six programs addressingcomplex issues associated with hazardous waste sites throughinterdisciplinary, multi-project team approaches. Included in these sixawards are one new grantee, the University of Iowa, and five granteesthat have been part of the program in the past: Columbia University, MichiganState University, the University of California at Berkeley, theUniversity of North Carolina at Chapel Hill, and the University ofWashington.In their new research endeavors, these grantees are using integrated researchmodels to understand how contaminants are transformed as they movethrough soils, sediments, and groundwater and how they interact withecosystems and ultimately effect human health. These robust researchefforts are augmented with graduate and postdoctoral training in interdisciplinaryenvironmental research and with community outreach activities. Eachgrantee is also required to engage proactively in researchtranslation to other federal agencies, industry, and the community atlarge.In addition to these six awards, the program is also pleased to announcethat it is initiating an Individual Research Project Program, whichwill support single investigator–initiated research. This initiativewill allow the program to respond quickly to emerging issues andto fulfill identified unmet needs.For interested investigators who are not already part of this vital andexciting program but would like to be, please contact program staff aboutfuture opportunities. It is not too early to start the planning process! Thenext solicitation is scheduled for release in August; subsequentsolicitations will be announced annually in August.We are available to assist you as you begin to conceptualize a future programat your university!http://www-apps.niehs.nih.gov/sbrp/For more information on the Superfund Basic Research Program see: William Suk, Ph.D., Director. | suk@niehs.nih.govClaudia Thompson, Ph.D. | thomps14@niehs.nih.govBeth Anderson | tainer@niehs.nih.gov


The field of conservation policy must adopt state-of-the-art program evaluation methods to determine what works, and when, if we are to stem the global decline of biodiversity and improve the effectiveness of conservation investments. For far too long, conservation scientists and practitioners have depended on intuition and anecdote to guide the design of conservation investments. If we want to ensure that our limited resources make a difference, we must accept that testing hypotheses about what policies protect biological diversity requires the same scientific rigor and state-of-the-art methods that we invest in testing ecological hypotheses. Our understanding of the ecological aspects of ecosystem conservation rests, in part, on well-designed empirical studies. In contrast, our understanding of the way in which policies can prevent species loss and ecosystem degradation rests primarily on case-study narratives from field initiatives that are not designed to answer the question “Does the intervention work better than no intervention at all?”Millennium Ecosystem Assessment. While the biological chapters are rife with data and empirical studies, thePolicy Responses volume [When it comes to evaluating the success of its interventions, the field of ecosystem protection and biodiversity conservation lags behind most other policy fields . None, however, focus and elaborate on the key feature of the evaluation process whose absence is glaringly obvious and whose adoption would do the most good for distinguishing cause and effect in conservation initiatives: the measurement ofcounterfactual outcomes  on one or more outcomes.Unfortunately, rigorous measurement of the counterfactual in the conservation literature is nonexistent. Consider some of the best-known conservation interventions—protected areas. Are such areas generally effective in protecting habitats and species? Based on observations that ecosystem conditions inside of protected areas are better than outside of protected areas  or managIn their study of protected areas in Africa, Struhsaker et al.  write, “Policy Responses volume of theMillennium Ecosystem Assessment [One of the “Main Findings” in theessment  is that When evaluating the effect of a conservation intervention, we must worry about confounding effects—effects that are contemporaneous with the intervention and could plausibly affect the outcome and thereby mask the intervention's effect. Examples of confounding effects include historical trends, unrelated programs or policies, and unobserved environmental and social characteristics. As in all scientific research, confounding effects are addressed through baselines, measures of covariates, and control groups . BaselinOne potential confounder deserves mention because of its widespread, and apparently not well-understood, effects on our ability to make inferences about program effectiveness: endogenous selection. Current analyses typically do not consider the implications of why an area was picked for an intervention and another was rejected, or why some individuals “volunteered” and others did not. In any non-randomized program, characteristics that influence the outcome variable also often influence the probability of being selected into the program. Failure to address the issue of endogenous selection can lead to biased estimates of a program's effectiveness.To better understand the problem of endogenous selection and the need for baselines, covariates, and controls, consider a currently popular conservation intervention: direct incentives in the form of Payments for Environmental Services (PES) ,18. PESThere are, however, two problems with this conclusion: it assumes that the past perfectly predicts the future and that “volunteers” represent the general population. If these assumptions are invalid, we cannot infer the deforestation rate in the absence of the program: the counterfactual is missing. With respect to the first assumption, there are good reasons to believe that past trends are not representative of future ones. Perhaps government subsidies that promote deforestation also declined around the same time that the payment program was initiated.and non-participants would avoid the assumption that the past perfectly predicts the future, but one still must ask, “Why did some landowners choose to participate and others did not?” For example, suppose one observes that forest cover on participating lands is much higher on average than that on non-participating lands. Can one conclude the program is effective? No. Participating landowners may be much more likely to have a pro-environmental ethic or low returns in alternative uses of the land.  design and from one that does not is enormous.Although randomized experiments are possible, they can be challenging to implement and evaluate in many circumstances . For exaThe lack of randomization does not imply that conservation scientists cannot rigorously evaluate and learn from conservation interventions. In thousands of evaluations in other disciplines, analysts use an array of quasi-experimental methods, grounded in theory and statistics, to isolate the causal effect of the intervention. Here we briefly describe three of the most popular quasi-experimental methods: (a) “natural” experiments, (b) instrumental variables, and (c) matching. Baker  providesClosest in spirit to randomization is the method of natural experiments, which refer to situations where nature (or chance) creates “treatment” and “control” units . In consexogenous variation in a conservation intervention. In the only published application of IVM in the conservation arena, Edmonds [endogenous formation of local management groups and found that communities with such groups extract less fuelwood from forests, on average. In general, good instrumental variables are difficult to find. Using IVM typically requires a mix of clear theoretical intuition, good quality secondary data, and a solid grasp of field conditions.In the absence of random events, researchers can apply instrumental variable methods (IVM). Instrumental variables represent a source ofEdmonds  considerThe method of matching is similar to IVM, but applies a different logic: areas (or landowners) that are in a conservation program are matched to otherwise “very similar” areas (or landowners) that are not in the program. These non-participating areas provide estimates of the counterfactual outcomes. Perhaps the best-known and most used matching method is propensity score matching . PropensGiven the billions of dollars invested in conservation initiatives and research in the past two decades, one may wonder why careful empirical studies and compelling data are lacking . Encouraging participants, including local actors, to agree on a set of explicit objectives to evaluate may be difficult in many conservation contexts.At the very least, we must use the principles of evaluation to assess the potential for bias in making inferences about program effectiveness.Third, conservation researchers are unaware of state-of-the-art empirical program evaluation techniques and the biases in current analyses. Donors and government agencies that fund conservation projects typically know little about program evaluation methods, and the practitioners who implement the projects typically lack incentives for careful analysis and falsification of hypotheses. Thus there is neither funding, nor a demand for funding, to conduct more careful analysis of interventions.Fourth, many believe that rigorous evaluations of effectiveness are expensive and thus would divert scarce conservation funds toward “non-essential” investments. In contrast, researchers and practitioners in other policy fields have demonstrated that randomized experimental methods can be implemented in the context of small pilot programs or policies that are phased in over time. The difference between what one can learn from a pilot initiative that uses an experimental  design and from one that does not is enormous.Fifth, the nature of biodiversity conservation can make evaluations more difficult than in other fields. Where outcomes are local, strong and complex spillover effects can occur. Enforcement and cheating can be difficult to verify. Property rights are often unclear in low-income nations and so the effects of interventions are complex both cross-sectionally and in time-series. Biological outcomes often respond slowly to interventions (wildlife stocks), and only time-series identification can be used for many problems.Sixth, many conservation interventions are short-term projects. The benefits of a careful evaluation, however, will largely be realized after the project ends and will accrue to the global conservation community. Field personnel are thus better off investing their time and resources in actions that will yield benefits to them rather than to the larger conservation community.Seventh, program evaluation methods require data. In other fields of policy analysis, researchers have longstanding national surveys and historical relationships with government agencies and field practitioners that generate substantial datasets for research. Most conservation interventions, particularly in low-income nations, are framed as independent projects that “test” an idea in one or several locations. Data collection in these locations is often poor or non-existent, with little or no planning for data collection in control “non-project” locations. Furthermore, we can comprehensively link programs to changes in behaviors and conservation success only when we combine data on ecological, geographic, socio-economic, demographic, and institutional measures. Given the disciplinary biases about appropriate scale and methods for data collection, we rarely find such transdisciplinary efforts.Finally, on a related point, credible estimates of conservation success depend on the ability to vary (or isolate) policy interventions in simple ways across space and time. We are well aware that within the same ecosystem, heterogeneity in institutions, income opportunities, access to markets, and other socio-economic characteristics can lead to different reactions to a given intervention. However, if every village or household is exposed to a different intervention , we are left with few observations for each intervention and thus cannot make any inferences about effectiveness.some policy interventions should be conducted in this manner to allow practitioners and decision makers to make inferences about their effectiveness. An evaluation may not be able to address the full range of questions, but addressing a tractable subset of questions may be far more productive, particularly given that reliable knowledge obtained from narrow studies may ultimately inform broader policy questions. Where it is impossible to use experiments, analysts must creatively use quasi-experimental methods to characterize the counterfactual and attribute cause to outcomes. At the very least, we must use the principles of evaluation to assess the potential for bias in making inferences about program effectiveness.We are not proposing that all policy interventions be uniformly applied across space and time, but we are arguing thatIn the field of program evaluation, one lesson is paramount: you cannot overcome poor quality with greater quantity.What is the most effective way to slow deforestation? How can we reduce poaching of protected species in low-income nations? Does conservation education lead to changes in behaviors that affect biodiversity? We know little about the answers to these questions, but finding the answers is crucial to stemming the global decline of biodiversity and improving the effectiveness of conservation investments.In the field of program evaluation, one lesson is paramount: you cannot overcome poor quality with greater quantity. We cannot learn from thousands of projects if none of these projects is designed in a way that permits an evaluation of its effectiveness. The results from a handful of well-designed individual tests can provide much more useful guidance than thousands of well-intentioned but poorly designed projects. In a field that takes the design and implementation of its initiatives seriously, we should expect to see greater value placed on evaluating the effectiveness of these initiatives.Non-governmental organizations and governments can use the results of evaluations to focus their limited budgets on those programs that are most effective. Kremer  providesRandomized policy experiments are often no more expensive than traditional “pilot” studies: the former simply builds program evaluation ideas into the project design. When randomization is not feasible and practitioners do not have the statistical skills to use appropriate quasi-experimental designs, we suggest a four-tiered rule for effectiveness evaluations see. Each tiAs noted in the introduction, we are not advocating that every conservation intervention be evaluated with an experimental or quasi-experimental design, or that every project collect data on outcomes and covariates from treatment and control units before and after the intervention. We are merely advocating that some of the hundreds of millions of dollars that are invested each year in biodiversity conservation initiatives be spent in this manner. The fate of the world's ecosystems and species depends on it.Does reducing the cost of schooling increase student attendance?  Initiated in the 1990s, the Mexican PROGRESA program provides cash grants to families if their children attend school regularly and receive preventative health care. The program was phased in randomly across villages. Analysts observed an average increase in enrollment of 3.4% for all students in grades 1 through 8, and 14.8% among girls who had completed grade 6. Using these same data, more sophisticated analyses were also done (“What would happen if the payments increased?”). In part, these clear and credible estimates of PROGRESA's effect led the Mexican government to expand the program, and other nations in Latin America to introduce similar programs.Counterfactual: The outcome that would have happened if there had been no conservation intervention.Endogenous: Used to describe a variable in a model or system that is causally dependent on other variables in the model or system.Exogenous: Used to describe a variable in a model or system that is causally independent of other variables in the model or system.Selection bias: Bias in estimating a program's effect that occurs when the participant and control groups differ from each other because of factors that also affect the program's outcomes. Such differences often arise when program units  volunteer to participate in the program or are purposively inducted into the program. As a result, outcome differences between the participant and control groups may arise from differences between the groups rather than the program itself.Can cash incentives encourage upland farmers to forgo clear-cutting of forests?A Vietnamese professor designed a quasi-experiment in which forest owners were offered cash to adopt sustainable forest management. Full randomization was considered difficult and costly. However, analysis of initial surveys showed that factors that affect forest use—e.g., availability of family labor, distance to roads—also affect decisions to participate, suggesting evidence of selection bias. Thus a control group of 50 households from a different upland community was chosen such that the characteristics of interest were balanced  between “cash treatment” and the control groups. The control group received the forest management training, but not the cash payment. The ongoing experiment, while small and far from perfect, shows how an understanding of proper evaluation techniques can avert potential analytical pitfalls in the design stage .Does listing and funding under the U.S. Endangered Species Act affect species recovery?The evidence marshaled to date for and against the effectiveness of the U.S. Endangered Species Act suffers from a problem common in analyses of biodiversity protection measures: the absence of data on what would have happened without the act. Statistical matching methods can be used to select control groups of species and thereby estimate how species listed and funded under the act would have fared had they not been listed or funded. The control groups must be similar in characteristics that can plausibly affect both listing/ funding and recovery . The analysis offers new insights and a methodology to guide evaluation of the effectiveness of non-randomized regulatory approaches to biodiversity protection .Do protected areas improve health and income of local people?Most answers to this question are based either on ex ante predictions from historical use patterns and strong assumptions, or ex post analyses that often prove only that the poor live near protected areas. Because national parks are not randomly sited, we can expect selection bias in interpreting the impact of parks on local people. An ongoing evaluation tracks health and livelihood outcomes of 1,000 households that traditionally have used resources around four new national parks in Gabon and 1,000 households that live outside the influence of the same parks. The simple selection of control households will go a long way to making a meaningful contribution in the debate over the effects of protected areas on local people .Consider ecological and socioeconomic factors that covary with the program.Guess-estimate the direction of potential bias in interpreting intervention effectiveness.Construct simple control groups (those that do not receive the intervention).Collect data on outcomes and key inputs before and after interventions.


Popliteal artery aneurysms representing 80% of peripheral artery aneurysms rarely rupture (a reported incidence of 0.1–2.8 %) and second commonest in frequency after aorto-iliac aneurysms. They usually present with pain, swelling, occlusion or distal embolisation and can cause diagnostic difficulties. We report a 78 year old man who was previously admitted to hospital with a pulmonary embolus secondary to deep venous thrombosis. He was heparinized then warfarinised and was readmitted with a ruptured popliteal aneurysm leading to a large pseudo aneurysm formation. The pulmonary embolus had been due to popliteal vein thrombosis and propagation of the clot. A thorough review of literature identified only one previously reported case of ruptured popliteal artery aneurysm and subsequent large pseudo aneurysm formation. We feel it is important to exclude a popliteal aneurysm in a patient with DVT. This may be more common than the published literature suggests. Popliteal artery aneurysms even in busy vascular units are infrequent and difficult to manage. Ruptured popliteal aneurysms are extremely rare and are said to have varied presentations and can cause diagnostic dilemma. We present a 78 year old man who presented with a ruptured popliteal aneurysm associated with deep vein thrombosis and pulmonary embolism. A through literature search identified only one previously reported case of ruptured popliteal artery aneurysm with large psuedoaneurysm formation with deep vein thrombosis .A 78 year old Caucasian male presented to our casualty 6 weeks following admission elsewhere with a history of swelling of the right leg and a pulmonary embolus. He had undergone VQ scanning with a proven diagnosis of pulmonary embolus and was therefore anticoagulated with heparin and warfarin.On admission he had swelling of the right knee and lower thigh with loss of sensation on the dorsolateral aspect of the right foot. Clinical examination revealed a large mass in the right popliteal fossa. The initial diagnosis was that of deep venous thrombosis and imaging was undertaken and the patient referred for a vascular opinion.On vascular review, the patient had a pulsatile mass comparable with a popliteal artery aneurysm of 12 cms diameter on the right and 6 cms on the left. His foot was warm and well perfused and his INR was 2.6. He underwent a duplex scan of the lower limb arteries which revealed ectatic iliacs and an abdominal aorta of 3.4 cms in diameter. A further CT arteriogram  . Deep veUltrasound examination is of limited value in a case of suspected rupture; however it does help in confirming the presence of an aneurysm. CT scan with or without angiography is the investigation of choice to confirm popliteal artery rupture . It can It is stated that ruptured popliteal artery aneurysms are best treated by ligation and continuity established preferably by autologous saphenous vein graft or by synthetic grafts when the patients have unsuitable veins. More recently this condition has been managed by percutaneous endovascular treatment in medically unfit patients .Since there is no evidence to show that routine screening of popliteal arteries will detect asymptomatic aneurysms  and in a1. Popliteal artery aneurysms very rarely rupture (a reported incidence of 0.1–2.8 %).2. Ruptured popliteal aneurysms have varied presentations and can cause diagnostic dilemma.3. A high index of suspicion is needed in a patient presenting with signs and symptoms of deep vein thrombosis or a massively swollen leg with impending compartment syndrome or anaemia with out any clear diagnosis4. Inappropriate management of patients with deep venous thrombosis from unrecognised arterial aneurysms is associated with unacceptable morbidity and mortality.


The program and the arrangement for a versatile, computer-controlledflow injection analysis system is described. A resident program (which can be run simultaneously and complementary to any other program) controls  a pump and a pneumatic valve (emptying and filling position). The system wasdesigned to be simple and flexible for both research and routine work.


We appreciate the letter from Janus of CropLife America commenting on that group’s assessment of the database and our article  from itsFor example, in Table 3 of our article , we presThe U.S. EPA has gone to great lengths to make ToxRefDB and its development as transparent as possible. Three manuscripts and data sets have been published to date , 2009b, in vitro data coming from the ToxCast research program (We recognize the complexity of pathologic progression to cancer. The end point progression scheme we presented  included program . This ap program .We agree that it is important to incorporate pharmacokinetics and metabolism, including chemical detoxification and activation, into predictive toxicology efforts. However, this issue is outside the scope of our article  and is bTwo additional papers on multigenerational reproductive and prenatal developmental toxicity studies in ToxRefDB have been recently published , again wWe view ToxRefDB as a valuable resource to the scientific community and one in which the U.S. EPA, stakeholders, and other interested parties can work together to ensure the success of ToxRefDB and the larger ToxCast effort.


Workplace interventions or programs to improve RTW are difficult to design and implement given the various individuals and environments involved, each with their own unique circumstances. Intervention mapping provides a framework for designing and implementing complex interventions or programs. The objective of this study is to design a best evidence RTW program for occupational LBP tailored to the Ontario setting using an intervention mapping approach.Despite over 2 decades of research, the ability to prevent work-related low back pain (LBP) and disability remains elusive. Recent research suggests that interventions that are focused at the workplace and incorporate the principals of We used a qualitative synthesis based on the intervention mapping methodology. Best evidence from systematic reviews, practice guidelines and key articles on the prognosis and management of LBP and improving RTW was combined with theoretical models for managing LBP and changing behaviour. This was then systematically operationalized into a RTW program using consensus among experts and stakeholders. The RTW Program was further refined following feedback from nine focus groups with various stakeholders.A detailed five step RTW program was developed. The key features of the program include; having trained personnel coordinate the RTW process, identifying and ranking barriers and solutions to RTW from the perspective of all important stakeholders, mediating practical solutions at the workplace and, empowering the injured worker in RTW decision-making.Intervention mapping provided a useful framework to develop a comprehensive RTW program tailored to the Ontario setting. Back pain continues to be the leading cause of morbidity and lost productivity in the workplace ,2. Despiparticipatory ergonomics and return-to-work (RTW) coordination whose primary focus is the workplace, may hold promise in reducing disability and improving RTW following an episode of LBP. In these studies an ergonomist and/or occupational physician coordinate RTW by identifying injured worker and workplace barriers to RTW. This is followed by a meeting at the workplace with the injured worker and workplace parties with the goal of identifying solutions to the identified RTW barriers and devising a RTW plan. This approach demonstrated a two fold improvement in RTW compared to clinical interventions.Studies in Quebec  and the Recent systematic reviews ,9 have aEven within one jurisdiction, workplace RTW interventions are complex to design and implement. This is because of the many different workplace settings and stakeholders that exist, each with their own unique circumstances and the potential to impact RTW. Personnel used to implement and coordinate RTW interventions also differ depending on setting. Some settings use occupational physicians and ergonomists where others use nurses, health & safety consultants, RTW coordinators and/or RTW specialists. All these factors make designing workplace RTW interventions challenging.Intervention mapping is a methodology used for designing and implementing complex interventions or programs. It has been used for over 20 years for systematically designing multifaceted programs involving numerous interventions directed at various individuals and environments . AlthougThe purpose of this study was to design a detailed workplace RTW program tailored to the Ontario setting using intervention mapping. The aim of RTW program is to reduce the duration of time off work and improve the sustainability of RTW following work-related LBP disability.The study design is a qualitative synthesis using the intervention mapping methodology as described by Bartholomew et al . There acore processes [what does a work supervisor have to do to facilitate RTW in a worker with LBP? What are the determinants that will impact the supervisor's ability to facilitate successful RTW? What needs to change at the level of the supervisor in order to facilitate successful RTW?Within each step of intervention mapping, specific tasks are performed and questions answered which guide the decision making process. These tasks are accomplished systematically using rocesses . Core prBest evidence was identified and comprised of systematic reviews ,15-24, cAlthough presented as steps, the intervention mapping process is iterative, rather than linear, with the ability to move between steps and tasks as additional insight is gained during the process. Below is an outline of each of the intervention mapping steps.The objective of the needs assessment was to establish the rationale to improve RTW for occupational LBP in Ontario. It was also to assess the feasibility of performing the intervention mapping process at the Toronto Western Hospital. This is a University of Toronto teaching hospital and provides assessment and treatment, including workplace interventions for injured workers. 	The needs assessment was conducted by the principal investigator (CA) and co-investigators  by examining trends in the duration of lost time claims in Ontario and meeting with Directors of the RTW Branch of the WSIB. The feasibility was assessed following discussions with the Directors of Research and Clinical Care at the hospital. The needs assessment also established the population of interest, which was the adult worker who suffers an occupational LBP episode and remains off work (acute and sub-acute LBP).Following the needs assessment, an intervention mapping team was assembled consisting of three researchers, three RTW coordinators (RTWc), a behavioural therapist, an occupational physician, a WSIB physician and two physiotherapists. The team members were selected based on their experience in work-related disability and RTW and the ability to commit to the time obligations of the project. The intervention mapping team met biweekly for approximately nine months. Using a group discussion format the team worked collaboratively through the remaining intervention mapping steps and core processes.what do they (stakeholders) have to do to facilitate RTW and what are the expected outcomes for these actions? Each performance objective was then matched with modifiable determinants that act as barriers or facilitators for achieving the objective. Determinants were grouped into three broad categories: cognitive-behavioural , knowledge and skills/self-efficacy. For the injured worker for example, one performance objective is the worker attempts to RTW on modified duties [The first task for the intervention mapping team was to use the core processes outlined above and list all important stakeholders that can impact RTW.	This was followed by listing performance objectives and expected outcomes for each identified stakeholder. Performance objectives are necessary activities that each stakeholder should perform to aid RTW. For example, for each stakeholder the intervention mapping team would answer the questions; d duties ,27. Feard duties , the lacd duties  was consThe goal of step 2 was to identify for each important stakeholder all potential barriers and facilitators to RTW and their corresponding change and/or learned objectives. Examples of the injured worker and workplace matrices are summarized in Tables In step 3, the intervention mapping team generated a list of possible interventions that were matched to each change and/or learned objective listed in step 2. Evidence from the literature , large employers, WSIB adjudicators and case managers, union representatives, RTW coordinators, physicians, chiropractors and physiotherapists and health and safety consultants. At each focus group session a preliminary draft the RTW program was presented. A focus group moderator detailed each step and asked for feedback on appropriateness and feasibility of the RTW program and how to best modify the program to maximize success of implementation. Each focus group session was audio recorded and transcribed. Summaries of the transcriptions were performed independently by two researchers  then synthesized via consensus. The synthesized feedback was then presented to the intervention mapping team. Using the core processes the intervention mapping team further refined the RTW program based on the synthesized feedback from the focus groups. The final step by step RTW program is outlined in Addition file Step 5 – Planning for program implementation – involves repeating step 2 using performance objectives specific for program implementation (rather than design) and Step 6 – Evaluation of the program – involves testing the designed program in a real world setting. These will be performed in future studies.Intervention mapping Ethics approval was obtained from the University Health Network Research Ethics Review Board.The intervention mapping team identified the following important stakeholders in RTW: 1) workplace stakeholders  2) the injured worker 3) health care providers 4) WSIB adjudicator and nurse case manager and 5) family and social network including co-workers. RTW stakeholders are individuals or groups of individuals who have the capacity to either hamper and/or facilitate RTW.The end result of the intervention mapping process was a comprehensive 5 Step RTW program.  for the injured worker's job, if available, is requested from the WSIB Case manager. The PDA outlines the injured worker's pre-accident work duties.The treating health care provider(s) are contacted and again asked to identify and rank barriers to RTW from their perspective. A copy of a Functional Abilities Form is obtained from the principal health care provider . In Ontario, the Functional Abilities Form is completed by the attending health care provider and outlines any restrictions for the injured worker when returning to work. This form is used to assist the employer in providing the injured worker with suitable work based on functional abilities.The supervisor and key decision makers  are contacted by the RTW coordinator and asked to identify and rank barriers to RTW. The injured worker's pre-injury job performance is discussed as well as the workplace's willingness to engage in RTW coordination including work modifications. The Physical Demands Analysis and the Functional Abilities Form are reviewed for any other potential barriers to RTW. A date is then set for a workplace meeting with injured worker, union representative, RTW coordinator and workplace parties.Prior to the meeting, the RTW coordinator outlines the ground rules for the meeting and emphasizes the importance of privacy and confidentiality and that only issues related to facilitating RTW will be discussed.The RTW coordinator facilitates and engages communication between each stakeholder member, addressing the prioritized barriers to RTW. Each party is asked to come up with solutions to these barriers and then rank them based on importance and feasibility. The RTW coordinator facilitates consensus around practical solutions for RTW, who will be responsible for implementing these solutions, and a time line for implementation. Shared responsibility for the implementation of the agreed upon solutions is emphasized. A final RTW plan and tentative RTW date is mediated by the RTW coordinator.Following the meeting, the injured worker and workplace decision makers tour the worksite and discuss agreed upon modifications and accommodations. The work demands are assessed and any perceived safety issues from the worker's perspective addressed. It is the responsibility of the workplace and the injured worker that work is conducted in a safe manner. The agreed RTW plan is reaffirmed. It is important that the injured worker feels empowered by the RTW process and that her/his reasonable concerns are addressed.The RTW coordinator writes a report outlining the agreed RTW plan. A copy of the report is provided to the injured worker and workplace representatives. A copy of the report is also sent to the third party payer (WSIB) and the health care providers for approval.At each step of the RTW program the RTW coordinator maintains ongoing contact with the injured worker and provides reassurance ,26. If uThe RTW coordinator contacts the workplace and the injured worker to determine whether the injured worker has returned to work at the agreed upon time, and performs the agreed upon duties. If necessary the RTW coordinator provides ongoing reassurance, positive reinforcement and education on self-management skills .Contact with the injured worker and workplace is maintained by the RTW coordinator following RTW and adjustments to the RTW plan are made to accommodate new information or overcome new barriers. The timing and intensity of the follow-up is tailored to the needs of the injured worker and workplace. The aim of the RTW coordinator is also to educate the injured worker and workplace on how to they can work together to resolve outstanding issues and future conflict. When issues are deemed too difficult or the parties have failed to resolve them then the RTW coordinator will intervene at the request of the workplace and/or injured worker.The injured worker is discharged from the RTW program when, from the perspective of the injured worker and the workplace there are no significant barriers preventing sustainable RTW.During the initial follow-up contact, an assessment is made from the perspective of the injured worker and the workplace party (supervisor), on whether each agreed solution in the RTW plan was implemented fully, partially or not at all. Satisfaction with the RTW plan among stakeholders is also assessed using a 5-point Likert scale ranging from very satisfied to very dissatisfied. This assessment is administered face to face at the workplace or by e-mail, fax or phone.A final report is sent to the third party payer (WSIB), injured worker, the workplace and health care providers.In this study, we describe the step-by-step development of a comprehensive RTW program for occupational LBP. To our knowledge this is the first study to outline the design of a comprehensive RTW program for occupational LBP using an intervention mapping approach. Designing RTW programs are inherently challenging because of the complex and multi-faceted nature of RTW.Intervention mapping provides a very useful framework to systematically guide us through this complexity. Of particular importance is the input from stakeholders who provide practical strategies to improve RTW. Another important feature of the intervention mapping process is the ability to tailor the interventions within the RTW program to the needs of various stakeholders and environments. A major drawback to the intervention mapping approach is that it is very time consuming and resource intensive.The impetus for this study was two previous studies, one from Quebec  and the Although the focus of our RTW program is the workplace, similar to the Quebec and The Netherland studies, our RTW program differs in that it identifies and addresses RTW barriers beyond the workplace. These include communication barriers with the third party payer or health care practitioner(s) and potential underlying psychosocial barriers involving the injured worker and his/her social environment. To address these other barriers, the individual(s) coordinating and implementing the RTW program, such as a RTW coordinator will require diverse skills, including the ability to engage in cognitive behavioural treatment (or coaching) or have access to other qualified professionals who can perform these activities. In addition to developing a comprehensive RTW program, another important contribution of this study is that we have outlined the roles and responsibilities involved in RTW coordination. RTW coordination can be performed by the insurer, health care provider  or disability manager provided they have the skills and knowledge to perform the duties outlined in the RTW program. RTW coordination does not necessarily mean a comprehensive intervention is needed. Sometimes all that may be required is a simple phone call. But someone needs to identify that a lack of communication exists and that a phone call is needed. 	Most injured workers return to work without the need of assistance. Among those who remain off work for reasons beyond the physical impairment of the injury, barriers are usually organizational and psychosocial in nature. When there is a discrepancy between the injury and ability to work then RTW coordination can be useful.In our setting, RTW coordinators have a health care background  with basic skills in ergonomic assessment and modifications. They also have informal training in cognitive behavioural coaching provided by cognitive behavioural therapists and psychologists who are members of the team. It is expected that in a sub acute LBP population, extensive knowledge in these areas is not often necessary. Physical therapists have been trained to perform basic cognitive behavioural therapy and can obtain similar outcomes compared to psychologists . In termThere are several important strengths of this study. We used a comprehensive and systematic approach (intervention mapping) in the design of the RTW program. Although this approach is novel in the occupational setting intervention mapping has been used extensively in the design of complex community health programs for over 20 years . In addiPotential weaknesses of the study include the design of the program within a hospital academic setting and using hospital based RTW coordinators who typically manage a more chronic population. This would potentially narrow the generalizability and applicability of our RTW program. Although designed in a hospital setting the draft RTW program was refined and contextualized with consultation and feedback from nine different stakeholder groups outside the hospital setting. In addition, the evidence and theories that formed the basis of the interventions within the RTW program focused on sub-acute LBP and not chronic pain. The RTW program in its design can be tailored to the various settings and injured worker populations. The underlying principles of the RTW program likely apply regardless of duration of symptoms, location of injury or setting.During the design of the RTW program, the WSIB in Ontario was under going a change in their service delivery model with associated restructuring of their policies and procedures. This may also limit its applicability. However, the principals of RTW program should still apply although the delivery of the RTW program may have to be modified. Moreover, employers and other third party payers can use the RTW program to develop their own RTW program, adapting it to their setting and aligning it with the new WSIB service delivery model.The subjective nature of interpretation of the evidence, theories and experiences of stakeholders may result in a different RTW program depending on the make-up and biases of the intervention mapping team members. However, the final RTW program and essential elements are consistent with current high quality systematic reviews and other published workplace RTW programs. Finally the effectiveness of the RTW program has not yet been evaluated.Ideally preventing work-related LBP is preferable to attempting to prevent LBP disability after an injury but this is not always feasible or effective ,62. HoweWe have a developed a 5 step RTW program to improve RTW in Ontario for injured workers with sub-acute LBP using an intervention mapping approach. The next step will be to evaluate the RTW program. We plan to pilot test the RTW program in Ontario workplaces. This will be followed by an evaluation of the effectiveness and cost-effectiveness of the RTW Program. We also plan to compare the RTW program to those implemented in the Quebec and The Netherland studies. Finally, we plan to further develop and test the methodology used in this study to adapt the RTW program to other jurisdictions in Canada and around the world.The authors declare that they have no competing interests.CA conceived and coordinated the study and drafted the manuscript. DC participated in the design and coordination of the study and helped draft the manuscript. IS participated in implementation of the study and assisted in drafting the manuscript. SS facilitated the focus group sessions and lead the synthesis of the qualitative data. EB participated in the design, writing the ethics application and drafting the manuscript. SE mapped out procedures performed by RTW coordinators and participated in intervention mapping team meetings. HH participated in the intervention mapping team meetings and provided technical guidance. BB assisted in interpretation of the data. PC assisted in the design of the study and helped write the manuscript. All authors read and approved the final manuscript.The pre-publication history for this paper can be accessed here:Step 2. Matrix for workplace: What the workplace needs to do to improve RTW?. the table describes a matrix where performance objectives of the injured worker are matched to corresponding determinants. The body of the matrix outlines what need to be learned or changed in order to achieve the performance objectives.Click here for fileStep 3. Intervention methods and strategies for the Workplace. the table describes the translation of learned and change objectives for the workplace into interventions and practical strategiesClick here for fileStep 4. Operationalize RTW Workplace Interventions into RTW Program. The table outlines a comprehensive step by step RTW ProgramClick here for file