Infectious Disease Modeling: Opportunities to Improve Coordination and Ensure Reproducibility

Outbreaks of infectious diseases—such as Ebola, Zika, and pandemic influenza—have raised concerns from Congress about how federal agencies use modeling to, among other things, predict disease distribution and potential impacts. In general, a model is a representation of reality expressed through mathematical or logical relationships. Models of infectious diseases can help decision makers set policies for disease control and may help to allocate resources.

GAO was asked to review federal modeling for selected infectious diseases. This report examines (1) the extent to which HHS used models to inform policy, planning, and resource allocation for public health decisions; (2) the extent to which HHS coordinated modeling efforts; (3) steps HHS generally takes to assess model development and performance; and (4) the extent to which HHS has addressed challenges related to modeling. GAO reviewed documents and interviewed HHS officials, state officials, and subject matter experts. GAO identified practices commonly used to assess infectious disease model performance and reviewed 10 selected modeling efforts to see if they followed these practices.

Within the Department of Health and Human Services (HHS), the Centers for Disease Control and Prevention (CDC) and the Office of the Assistant Secretary for Preparedness and Response (ASPR) used models to inform decision-making during and after outbreaks of Ebola, Zika, and pandemic influenza. These agencies' modeling efforts informed public health planning, outbreak response, and, to a limited extent, resource allocation. Four CDC centers perform modeling.

HHS agencies reported using multiple mechanisms to coordinate modeling efforts across agencies, but they do not routinely monitor, evaluate, or report on the extent and success of coordination. Consequently, they risk missing opportunities to identify and address modeling challenges—such as communicating clearly, and obtaining adequate data and resources—before and during an outbreak. As a result, agencies may be limiting their ability to identify improvements in those and other areas. Further, there is potential for overlap and duplication of cross-agency modeling efforts, which could lead to inefficiencies.

CDC and ASPR generally developed and assessed their models in accordance with four steps GAO identified as commonly-recognized modeling practices: (1) communication between modeler and decision maker, (2) model description, (3) verification, and (4) validation. However, for four of the 10 models reviewed, CDC did not provide all details needed to reproduce model results, a key step that lets other scientists confirm those results. GAO found that CDC's guidelines and policy do not address reproducibility of models or their code. This is inconsistent with HHS guidelines and may jeopardize the reliability of CDC's research.

This report also identifies several modeling-related challenges, along with steps agencies have taken to address them.

GAO recommends that HHS (1) develop a way to routinely monitor, evaluate, and report on modeling coordination efforts across multiple agencies and (2) direct CDC to establish guidelines to ensure full reproducibility of its models. HHS agreed with GAO's recommendations.

Today’s globalized economy and transportation systems allow infectious  diseases to spread more rapidly than ever. Notable outbreaks include  novel coronavirus beginning in 2019, Zika virus disease (Zika) in 2015,  Ebola virus disease (Ebola) in 2014, and H1N1 pandemic influenza in  2009. Disease outbreaks can cause catastrophic harm to the United  States, disrupt economic and social systems, and kill, sicken, and  traumatize people on a massive scale. For example, approximately 1  billion people worldwide get sick annually from zoonotic pathogens— pathogens that can spread from animals to humans—of which,  approximately 15 million people die. Such outbreaks are on the rise. The  latest example is the novel coronavirus disease which had, as of May 6,  2020, caused approximately 250,000 deaths worldwide and sickened  approximately 3,600,000 people. In the United States, the virus had  caused approximately 63,000 deaths, and sickened approximately  1,200,000 people. The situation has heightened U.S. attention to potential  future infectious disease threats and raised questions about the nation’s  preparedness and response capabilities. It has also raised concerns among some members of Congress about how federal agencies predict  the spread of emerging infectious diseases, in particular through the use  of modeling.

A model is a simplified representation of reality expressed through  mathematical or logical relationships. Modeling is widely used in fields as  diverse as engineering, finance, meteorology, and wildlife management.  In public health, infectious disease modeling can help decision makers by  predicting the social and economic effects of an intervention and  informing spending for preparedness and response, among other things.  It can answer public health questions that other methods cannot, whether  for practical, ethical, or financial reasons. However, because models  simplify reality, they may give misleading answers if the underlying data  or assumptions are flawed or not fully understood by decision makers.  Further, some real-world systems can be difficult to model because of  their inherent complexity, scale, or randomness. For these and other  reasons, researchers must carefully design, interpret, and communicate  the results of models that may be used to support public health decisions.  Understanding where and when infectious disease outbreaks may occur  can provide information—in near real time—to decision makers who help  set disease control policies and allocate resources.

You asked us to examine how federal agencies have used models to  inform decision-making in recent infectious disease outbreaks, and the  limitations and challenges in developing and using models. This report  examines (1) the extent to which the Department of Health and Human  Services (HHS) has developed or used models to inform public health  planning, policy, and resource allocation for Ebola, Zika, and pandemic  influenza; (2) the extent to which HHS coordinated its modeling efforts for  selected infectious diseases; and (3) steps HHS took to develop and  assess the performance of its models for the selected infectious diseases  and steps it applied to a selection of infectious disease models. It also (4)  describes the extent to which HHS has addressed challenges related to  modeling for selected infectious diseases.

In our review, we focused on HHS because of its leadership in scientific  and technical issues related to infectious disease modeling, role in  infectious disease outbreak preparedness and response activities, and  use of infectious disease modeling for policy and regulatory activities.  Within HHS, we identified four agencies—the Centers for Disease Control  and Prevention (CDC), Office of the Assistant Secretary for Preparedness  and Response (ASPR), National Institutes of Health (NIH), and Food and  Drug Administration (FDA)—that may develop or use infectious disease  models. It is important that these agencies coordinate with one another  and with other relevant external entities to avoid the overlap and  duplication of modeling efforts across agencies and to share new ideas  and advances in modeling that might lead to new insights. We focused on  three infectious diseases in our review: Ebola, Zika, and pandemic  influenza. We selected these diseases based on their inclusion on the  National Institute of Allergy and Infectious Diseases’ Emerging Infectious  Diseases/Pathogens list and consulted with agency officials and five  infectious disease modeling experts for input on the selection of diseases  in our review. We selected the experts based on our background  research and input from agency officials (additional details on expert  selection methodology can be found in appendix I).

To examine the extent to which HHS has conducted modeling to inform  public health planning, policy, and resource allocations for selected  infectious diseases:

We interviewed agency personnel, including agency officials and staff  who develop and use models, referred to here as “modelers,” and  reviewed agency documents and reports to determine how or why the  agencies develop or fund models; determine the types of models used  and the questions they are addressing; or obtain a general description  and specific examples of how these agencies use models to inform  planning, policy, and resource allocation.

We interviewed NIH officials about funding for research related to  modeling for the selected diseases.

We interviewed officials from five state health departments—selected  based on a review of a CDC draft report on model usage, on the level  of influenza activity that states experienced, and geographic variation  by U.S. region—about their experiences using CDC-developed  modeling tools for influenza response.

For context on and examples of the types of modeling that CDC and  ASPR conducted, we reviewed documents CDC and ASPR officials  provided to us or cited in our interviews. (For a bibliography of models  reviewed, see appendix II.) We did not include FDA and NIH in this  portion of the review, because FDA has a limited role in modeling, and  NIH funds, rather than conducts, modeling.

To examine the extent to which HHS agencies coordinated their modeling  efforts for the selected infectious diseases:

We interviewed agency officials and reviewed documents related to  coordination and collaboration, including memoranda of  understanding between agencies, to identify the nature and extent of  coordination and collaboration across HHS agencies that conduct or  fund modeling. We compared these actions to six of the eight leading  collaboration practices we identified in our prior work based on their  relevance to the coordination efforts we reviewed (see appendix I). In  this report, and in our past work, we define coordination broadly as  any joint activity that is intended to produce more public value than  could be produced when organizations act alone.

To examine steps HHS took to develop and assess the performance of  models for selected diseases and the steps it applied to a selection of  infectious disease models:

We identified steps that infectious disease modelers generally  consider when developing and assessing the performance of models  from a synthesis of information gathered from interviews with agency  officials, interviews with additional relevant experts, and reviews of  documents. From these sources, we also gathered information on  how these assessments may impact the use of models for public  health decision-making.

We reviewed information regarding steps taken to develop and  assess the performance of models, for a non-probability sample of  models in published papers or memos, including seven models  prepared by CDC (two each for Ebola and Zika, and three for  pandemic influenza); and three prepared by ASPR (one for each  disease). We compared the steps taken in the development and  assessment of the performance of these models to the commonly- considered steps we identified as described above and followed up  with agencies to confirm our determinations and gather information on  why some steps were not taken.

To describe the extent to which HHS has addressed challenges related to  modeling for selected infectious diseases, we took the following steps:

We interviewed selected experts regarding modeling-related  challenges. We also interviewed agency officials, including modelers,  and selected experts, regarding challenges and limitations related to  modeling; steps they’ve taken to address the challenges; and whether  these challenges can be addressed or are ongoing.

We reviewed documents and reports from agencies and other  sources such as the National Science and Technology Council report,  “Towards Epidemic Prediction: Federal Efforts and Opportunities in  Outbreak Modeling” to identify challenges related to modeling and  steps taken or recommended, if any, to alleviate these challenges.

We conducted this performance audit from May 2018 to May 2020 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Public Health Agency Roles in Infectious Disease Outbreaks and Response

In the United States, HHS is the lead federal agency responsible for  public health. Its responsibilities include preparing for, mitigating,  responding to, and recovering from public health emergencies. Within  HHS, ASPR and CDC prepare for and respond to infectious disease  outbreaks.

ASPR leads and coordinates national preparedness and response to  outbreaks in the United States. It also coordinates and supports  advanced research and development, manufacturing, and procurement  and deployment of medical countermeasures, such as vaccines, drugs,  therapies, and diagnostic tools that can be used in the event of a potential  public health emergency to protect the public from harm.

CDC monitors and responds to outbreaks by, among other things,  studying the link between infection and health; monitoring and reporting  cases of infection; and providing guidance to the public, travelers, and  health care providers. During public health emergencies, CDC may  operate an Emergency Operations Center (EOC) for monitoring and  coordinating its response to emergencies—including infectious disease  outbreaks of Ebola, Zika, and pandemic influenza—in the United States  and abroad. The EOC staff helps with directing specific incident  operations; acquiring, coordinating, and delivering resources to incident  sites; and sharing incident information with the public.

Other agencies perform additional work related to infectious diseases. For  example, FDA monitors and protects the blood supply, and NIH makes  grant awards that support research related to diseases and modeling.

ASPR, CDC, and FDA have different approaches to modeling. In the  cases of Zika, Ebola, and pandemic influenza, CDC and ASPR are two  key agencies that conduct federal infectious disease modeling efforts. As  of February 2020, ASPR had a centralized modeling unit staffed by about  nine people, who are a mix of federal and contract employees, according  to ASPR officials. At CDC, however, modeling is decentralized and  integrated into the individual centers that make up the agency. Some  staff work full time on modeling, while others spend part of their time on  other tasks. In addition, some of CDC’s modeling efforts are conducted  externally. According to CDC, approximately 70 staff members  participated in modeling studies, as of October 2018. Of those staff,  CDC’s Health Economics and Modeling Unit employed about 10  modelers who have worked on Ebola and other diseases. For Zika, CDC  officials responding to Zika said most modeling work was done by one  modeler in CDC’s Division of Vector-Borne Diseases, a part of the  National Center for Emerging and Zoonotic Infectious Diseases. CDC  influenza officials said influenza modeling is conducted by six or seven  members of CDC’s Influenza Division. Agency infectious disease  modeling activities are not limited to Ebola, Zika, or pandemic influenza.

Agency efforts to protect the nation from disasters and emergencies can  be organized into two elements: preparedness and response. Infectious  disease modeling is one tool used to inform a wide range of decisions  related to outbreak preparedness and in response to an outbreak. In the  context of infectious disease outbreaks, ASPR and CDC perform work on  preparedness and response. For example, ASPR leads the Public Health  Emergency Medical Countermeasures Enterprise (PHEMCE), an  interagency group that helps develop medical countermeasures—FDA- regulated products including drugs, or devices that may be used in the  event of a potential public health emergency to protect the public from  harm. CDC may activate its EOC to assist with the response during an  outbreak. For example, during the 2014-2016 West Africa Ebola  outbreak, CDC activated its EOC in July 2014 to help coordinate  activities. CDC personnel were deployed to West Africa to assist with  response efforts, including surveillance, data management, and  laboratory testing.

Since the 1980’s, emerging infectious diseases have resulted in more  recurrent disease outbreaks, causing an increasing number of human  infections. Emerging infectious diseases have at least one of the following  characteristics: they are newly recognized, have emerged in new areas,  are newly affecting many more individuals, or have developed new  attributes. Some of these diseases—including Ebola and Zika—are  zoonotic pathogens, meaning they spread from animals to humans.  Zoonotic pathogens can be carried from an animal to a human by another  animal, such as a mosquito, chicken, or bat, which is known as a vector.  Such pathogens sicken approximately 1 billion people annually.

According to the World Health Organization, Ebola causes an acute,  serious illness, which is often fatal if untreated. Ebola is introduced into  human populations through close contact with the blood and other bodily  fluids of infected animals. Humans spread Ebola through direct contact  with the bodily fluids of infected individuals or objects contaminated with  these fluids. Ebola symptoms include fever, muscle pain, vomiting,  diarrhea, impaired kidney and liver functioning, and, in some cases,  internal and external bleeding. There have been five Ebola outbreaks  since 2014, including the 2014-2016 West Africa outbreak which caused  more than 28,600 cases and 11,325 deaths. Since 2018, there has  been an ongoing outbreak in the Democratic Republic of the Congo.  Figure 1 provides a timeline of Ebola outbreaks since 2014.

Zika is a virus that is primarily transmitted through mosquito bites. It can  cause symptoms such as fever, rash, conjunctivitis (red eyes), and joint  and muscle pain. It can also be transmitted from mother to child during  pregnancy, or around the time of birth, or from person to person through  sexual contact or blood transfusion. Many infected people do not have  symptoms or will only experience mild symptoms. The Zika outbreak that  began in 2015 affected individuals infected with the virus in ways that had  not been seen with previous outbreaks of the disease. Specifically, during  the 2015-2016 outbreak, Zika infection in pregnant women was linked to  microcephaly and other severe brain defects, according to CDC. CDC  officials said this was the first time in more than 50 years that an  infectious pathogen has been identified as the cause of birth defects. Zika  was also linked to other problems, such as miscarriage, stillbirth, and  Guillain-Barré syndrome, an uncommon disorder affecting the nervous  system. In the Western Hemisphere, the first cases of locally- transmitted Zika were confirmed in Brazil in May 2015. In December  2015, locally-transmitted Zika was reported in Puerto Rico. On January  22, 2016, CDC activated its Emergency Operations Center to respond to  outbreaks of Zika occurring in the Americas and to increased reports of  birth defects and Guillain-Barré syndrome in areas affected by Zika.  Within the continental United States, the first locally-transmitted cases  were confirmed in Florida in June 2016. The World Health Organization  declared Zika a Public Health Emergency of International Concern from  February to November 2016.

In the spring of 2009, a novel influenza virus emerged, known as  influenza A (H1N1)pdm09. According to CDC, it was detected first in the  United States and quickly spread across the world, causing a pandemic  or global outbreak of a new influenza A virus. This new virus contained a  combination of influenza genes not previously identified in animals or  people. The virus was very different from other H1N1 viruses circulating  at the time, so seasonal influenza vaccines offered little cross-protection  against infection with the new H1N1 virus, according to CDC. A vaccine  against the new virus was produced, but it was not available in large  quantities until late November—after the peak of illnesses during the  second wave in the United States. CDC activated its EOC on April 22,  2009, to manage the H1N1 response. From April 12, 2009, to April 10,  2010, CDC estimated there were about 60.8 million cases, 274,304  hospitalizations, and 12,469 deaths in the United States due to the new  H1N1 virus. According to CDC, few young people had any existing  immunity—as detected by antibody response—to the virus, but nearly  one-third of people over 60 years old had antibodies against it, likely from  exposure to an older H1N1 virus.

Multiple strains of influenza can infect humans, including strains that  originate in animals. According to CDC, human infections with an Asian  lineage avian influenza A (H7N9) virus were first reported in China in  March 2013. During an epidemic that lasted from October 1, 2016,  through September 30, 2017, the World Health Organization reported 766  human infections with H7N9 virus, making it the largest H7N9 epidemic.  From 2013 to December 7, 2017, there were 1,565 humans infected with  Asian lineage H7N9 reported by the World Health Organization.  According to CDC, while the risk posed by H7N9 virus to the public’s  health was low, the agency was concerned about its pandemic potential.

Agencies use infectious disease models to answer a variety of public  health questions, including those related to outbreak preparedness and  response. A model is a physical, mathematical, or logical representation  of a system, phenomenon, or process that allows a researcher to  investigate that system, phenomenon, or process in a controlled way. For  example, the classic Susceptible-Infected-Recovered or “SIR” model  divides a population into three categories: 1) susceptible to the disease,  S; 2) infected and infectious, I; and 3) recovered or removed from the  infected or susceptible population, R. This model uses equations to  determine how many people move between these three categories. The  equations contain parameters—numerical descriptors of the disease  based, for example, on experiment, expert opinion, or statistics of an  ongoing or past outbreak. The equations allow the researcher to estimate  how many people are or could be affected by the disease. For example,  for past Ebola outbreaks, models estimated that after 40 days, about 44  percent of the population in close contact with infected individuals was  susceptible to infection, 31 percent was infected, and 22 percent was  recovered. Based on these parameters, equations for transfer between  categories, and underlying demographics of the community, an  epidemiologist could use the model to estimate how many people within a  given town could be susceptible, infected, or removed from the categories  of susceptible or infected (due to death or recovery and immunity).  Based on model estimates and if a vaccine was available, CDC officials  said the decision maker could plan for a specific number of vaccine kits  and additional medical staff and supplies to treat infected patients.

Models can also help agency officials anticipate future outbreaks, forecast  the spread or severity of a disease, and predict the effects and costs of  different intervention options. After an outbreak, models can help sort out  what happened, what drove the outbreak, and how it compared to past  outbreaks. Other tools are available to accomplish some of these tasks,  but models are particularly useful when existing data are not sufficient to  answer a given question, or when agencies need to integrate data from  disparate sources.

Infectious disease models can be put into two broad categories:

Statistical models. This type of model identifies relationships or  patterns that can be used to describe what is occurring or predicts  what may occur in the future based on what has occurred in the past.  Statistical models tend to use a large amount of data, such as past  observed events, to forecast future events, such as disease  occurrence, but do not require a fundamental understanding of  biological processes or human behavior. They can predict outcomes  when causes are not known or understood and when scientific  understanding of a disease is limited. They tend to use large amounts  of data on past events to forecast future events. Statistical models do  not provide full explanations about an infectious disease but may be  used when epidemiologists have all or most of the data needed to test  a hypothesis. Several benefits can be derived from statistical  modeling, including the ability to control for multiple factors that might  impact the outcome reviewed, and the ability to isolate the potential  effect of infectious disease factors on a particular outcome.

Mechanistic models. Mechanistic models rely heavily on scientific  evidence and theory related to infectious diseases, and the  understanding of disease dynamics or human behavior from prior  knowledge—such as biological processes or interactions between  people—to represent known processes. They use basic infectious  disease science to inform public health guidance and provide insights  into outbreak emergence, spread, and control. For example,  population-based models can simulate the course of an epidemic by  dividing the population into different categories, such as susceptible,  infected, and recovered. Mechanistic models can project the likely  course of disease transmission, calculate and predict the effect of  proposed interventions, and take into account variable conditions,  such as human behavior.

Both statistical and mechanistic models can range from simpler to more  complex. A simpler model may, for example, have fewer parameters  (inputs) or equations than a more complex model. According to CDC  modelers and an expert, a simpler model may be run with a variety of  software, ranging from spreadsheet software to more sophisticated  software, whereas more complex models are usually run using  sophisticated statistical or mathematical programming languages. As a  model becomes more complex, it can become harder to describe,  recreate, and understand its internal functioning.

Modeling is identified as a beneficial tool in various national plans for  disease response and biodefense. These plans do not define the extent  to which modeling should occur or how models should be developed for  policy, resource allocation, or planning purposes. See table 1 for  examples of relevant national plans.

HHS Has Used Infectious Disease Models to Help Inform Policy and Planning

Use of Models to Inform Planning and Policy Decisions

CDC and ASPR use models primarily to answer questions from decision  makers. CDC and ASPR officials told us, and documents show, that  modeling is one source of information that may inform such decisions,  along with sources such as expert opinion, surveillance, other prior work  on the disease, and an official’s own knowledge.

CDC modelers and officials said there is no “rule” as to when to use  models, and in some situations, it may not be considered useful. For  example, CDC did not use modeling when issuing a travel notice for an  Ebola outbreak in specific provinces in the Democratic Republic of the  Congo, officials said. Instead, CDC based the travel order on an analysis  that considered disease incidence and prevalence, public health  infrastructure, and the availability of therapeutics, among other things.  Similarly, CDC officials responding to Ebola said modeling may be  undesirable when it would take too long to engage the necessary external  subject matter experts or when modeling would detract from responding  to a disease.

CDC and ASPR modelers use models for a variety of purposes. CDC  officials said modeling is done differently for each disease, and the  amount and type of modeling varies across CDC centers, in part because  some centers have less capacity to conduct modeling than others.  According to a CDC internal report, the most frequent uses of infectious  disease modeling at CDC are: guiding preparedness and response efforts;  conducting economic analyses to evaluate the benefits of public  health actions, thereby reducing illness and deaths from infectious  diseases; understanding pathogen biology, disease transmission, and  estimating disease burden; and assessing the effect of interventions and prevention strategies.

ASPR modelers and officials said models have provided information  about topics such as:  resources, including protective equipment, needed to help respond to  an Ebola outbreak;  the number of therapeutics and vaccine doses needed to respond to  Ebola, both in Africa and domestically; expected U.S. demand for Zika diagnostics; and  the number of vaccine doses needed to mitigate the spread of  pandemic influenza.

ASPR modelers and officials said modelers tend to serve in a broad role  that can include modeling, data analysis, or other tasks. For example,  officials said a modeler could provide a team with day-to-day analytic  support and not necessarily spend time developing models or use them.  Additionally, ASPR maintains a Visualization Hub that can be used for  outbreak planning and response, including outbreaks of pandemic  influenza and other emerging infectious diseases (see fig. 2).

CDC and ASPR modelers and officials said they generally initiate  modeling in response to questions from decision makers. The modelers  then work closely with epidemiologists and other subject matter experts to  answer the questions. Modeling, according to CDC officials, may be used  by individuals or groups within centers, such as division directors,  branches, or teams to influence decisions. Who answers a particular  question depends, according to ASPR modelers and officials, on the  decision maker. Sometimes questions asked will not be within their  mission—modelers may suggest such questions be sent to a more  relevant agency or part of HHS. CDC and ASPR have modeled to answer  a variety of public health questions relevant to Ebola, Zika, and pandemic  influenza, and, at times, the results helped inform policy and planning  decisions. Modelers and officials provided the following examples:

Planning: ASPR modelers and officials said the bulk of the agency’s  modeling is related to the planning, development, and deployment of  medical countermeasures. For example, these modelers and officials  said many clinical trials for vaccines and therapeutics were planned  during the 2014-2016 Ebola outbreak response. As a part of these  planning activities, ASPR modelers said modelers developed  forecasts of future trajectories of disease incidence under a variety of  conditions. These forecasts indicated a significant likelihood the  disease incidence in Sierra Leone could decrease to a level that  would significantly reduce the success of the trials, according to  modelers. Additionally, at the beginning of the 2014-2016 Ebola  outbreak response, CDC modelers received modeling questions  related to the resources needed to effectively limit the spread of the  disease, according to CDC documentation. CDC used models to  predict the number of Ebola cases that could be expected over time  with and without disease interventions such as Ebola treatment units,  community care centers, and safe burials. On the basis of this  information and other factors, including a United Nations document on  Ebola needs, CDC leadership and other U.S. government officials  recommended a rapid increase in Ebola response aid, according to  CDC documentation. According to CDC documentation, later  analyses demonstrated that this increase helped to greatly reduce the  actual number of cases, compared to the likely number if prompt  action had not been taken. Additionally, in response to the H7N9  influenza outbreak in 2017, ASPR modeled to determine when doses  of influenza vaccine should be delivered and how many doses should  be administered in order to mitigate a domestic outbreak. This model  found that having a vaccine stockpile could be helpful in preventing  disease and that a slow effort to administer an H7N9 vaccine could  reduce the vaccine’s usefulness.

Policy: During the Zika outbreak, CDC modelers and officials said  they modeled to determine the potential effectiveness of using  pesticides to remove insects from aircraft, trains, or ships. According  to modelers and agency officials, the issue arose as concern about  Zika virus grew, including from other countries and U.S. agencies, like  the Department of Transportation and Department of Defense. The  model indicated that humans are more likely than insects to transport  Zika on airplanes, and officials therefore concluded that the use of  pesticides on airplanes would not be an effective intervention.  According to CDC modelers and officials, this modeling resulted in an  additional sentence being added to World Health Organization policy,  which stated that pesticide use was not expected to be effective.

The extent of modeling conducted for Ebola, Zika, and pandemic  influenza varied according to the question being asked, along with other  factors as follows:

Type of question: CDC and ASPR have used models to answer  such questions as who should be prioritized for vaccination or  treatment, how transmissible a disease is, and how effective certain  interventions are likely to be, according to modelers and agency  officials. For example, ASPR modelers and officials said they modeled  to help estimate the resources needed to respond to an Ebola  outbreak; the number of therapeutics and vaccine doses needed to  respond to Ebola, both in Africa and the U.S; and the expected U.S.  demand for Zika diagnostics. One ASPR official said that, during the  2009 pandemic influenza outbreak, modeling questions were used to  provide decision makers with information on what might happen in a  given situation. For example, models were used to provide information  related to decisions on early vaccine distribution and how this  intervention could affect the potential mortality rate.

Time to model: How soon decision makers needed information also  influenced the extent to which CDC and ASPR modeled. For example,  if decision makers needed an answer in a week, modelers would  inform the decision makers about how much of the answer they could  provide within that time frame, ASPR modelers said. Similarly, CDC  modelers and officials said that, in one instance, modelers had only  12 hours to provide decision makers with information. Even estimating  the time needed to develop and conduct modeling could represent an  additional challenge, according to CDC modelers responding to Zika.  According to a CDC article on modeling to inform responses to novel  influenza viruses, the amount of time required to develop and execute  a model can vary from less than a week to more than a month.  Agency officials concurred with these time frames.

Personnel and data availability: The availability of qualified  personnel was also a factor that affected how much modeling  agencies conducted for the selected diseases. For example, CDC  modelers and officials said the agency’s Division of Vector-Borne  Diseases has focused its resources in other areas, such as building  the capacity of states to address vector-borne diseases, and therefore  had not invested in individuals with the right skill sets to conduct  modeling for the Zika outbreak response. As a result, the division  had to call on the three or four CDC modelers from outside of the  division who were available to assist with the Zika outbreak response,  which limited the amount of modeling that could be performed. Data  challenges can also limit the types of modeling conducted. For  example, when modeling for Zika, ASPR modelers said they used  available information, but data quality and availability limited their  ability to model. More data typically become available as an outbreak  progresses, but models may be most helpful at the beginning of an  outbreak when critical decisions need to be made (see fig. 3).

CDC and ASPR do not keep a list of all modeling conducted, and we  therefore cannot quantify the extent of their efforts in terms of a number of  models. ASPR modelers and officials said modeling is typically one small  aspect of the way the agency carries out its mission. One ASPR official  said models are never the sole source of information for decision-making.

According to NIH officials, NIH does not conduct or fund internal modeling  for decision-making purposes. NIH’s Fogarty International Center has  conducted self-initiated, internal modeling to answer questions generated  from research, and from ideas from Center-held workshops. Two NIH  institutes—the National Institute of General Medical Sciences and the  National Institute of Allergy and Infectious Diseases—along with NIH’s  Fogarty International Center have awarded grants for external modeling  research for our selected diseases. However, NIH officials said these  efforts were intended to advance science, not for policy or outbreak  response.

Use of Models to Inform Resource Allocation Decisions

CDC and ASPR modelers and officials said they considered modeling  results to a limited extent when making decisions about resource  allocation. While modeling can help determine the amount of particular  resources needed during an infectious disease outbreak, CDC modelers  and officials said it is not central to their resource allocation planning. For  example, CDC modelers and officials noted that while a model could  inform a decision maker about how many diagnostic testing supplies  would be needed based on the range of predicted cases, this would be  one input among many into the decision. Decision makers would also  consider whether there are other diagnostic test supplies for similar  diseases that could be used, the extent of laboratory testing capacity, or  the longevity of those supplies.

Models can be used to help plan for the cost of interventions by  determining the numbers or types of interventions that can be used during  a response to an infectious disease outbreak, according to CDC modelers  and officials. It can also help decision makers recognize gaps in their  ability to implement resource allocation decisions, according to CDC  officials. For example, CDC leadership described how modeling input  requirements spurred analysis of the factors limiting hospitals’ use of  ventilators during a pandemic influenza outbreak. This work, according  to CDC officials, helped determine the number of ventilators that should  be included in the national stockpile. While modeling results are important  to consider during a public health event, ASPR officials and modelers said it is also important to consider concrete financial estimates based on  prior experience and whether recommended medical interventions or  countermeasures are available or effective.

For example, ASPR modelers and officials have occasionally been asked  to analyze costs for medical countermeasures, but modelers and officials  said that few medical countermeasures typically meet the requirements of  decision makers, and existing medical countermeasures are typically  unavailable for use in a response. ASPR modelers and officials noted that  the usefulness of modeling to the decision maker in these instances is  limited. In the event that they were asked to model for such questions,  ASPR modelers and officials said time would also be a limiting factor in  their analysis.

CDC has also developed models to inform decision-making at the state  level, specifically to assist state and local public health agencies in  developing outbreak response plans. A professional organization of  epidemiologists we contacted expressed some concerns with limitations  of CDC models, specifically noting that state and local officials viewed  CDC models as lacking the level of refinement needed for their state- and  local-level planning needs. To follow up, we interviewed officials from a  non-generalizable selection of five states based on their reported use of  CDC models, the level of selected disease activity in the state, and  geographic variation. Two of the five state health departments we  contacted reported using one of CDC’s models for Ebola, Zika, or  pandemic influenza. These two states confirmed that the usefulness of  the CDC FluSurge pandemic influenza model was limited by unrealistic  assumptions or a lack of predictive capability, but added that the models  were useful to them when considering how to allocate resources or  otherwise prepare for a severe pandemic. Officials from one state health  department told us they had similar concerns with the CDC Ebola model  regarding an unrealistic overestimate of the potential cases, but added  that it was useful for informing staff allocation planning as part of their  overall response.

Officials from another state health department told us they used CDC’s  Zika modeling results that indicated how many emergency room visits  they could expect and what symptoms it would take to confirm a Zika  infection. At the time, state officials said, commercial testing for Zika was  not available, so this modeling was very helpful to health officials looking  to recommend who hospitals should test based on the presence of Zika  symptoms. State health department officials added that many other  factors are considered when deciding on resource allocation, such as  local leadership and willingness to embrace the public health response.

Agencies Coordinate Infectious Disease Modeling Efforts but Do Not Fully Monitor, Evaluate, and Report on Coordination

The four HHS agencies that work on infectious disease modeling reported  using multiple mechanisms to coordinate their efforts. However, they do  not routinely monitor these efforts, evaluate their effectiveness, or report  on them to identify areas for improvement.

HHS Agencies Coordinate Infectious Disease Modeling Efforts in Multiple Ways

The four HHS agencies that work on infectious disease modeling—ASPR,  CDC, FDA, and NIH—reported using multiple mechanisms to varying  extents to coordinate such efforts. For example:

Emergency Operations Center (EOC). During the response to an  outbreak, CDC activates its EOC—a temporary, formal organizational  structure for coordinating expertise within CDC and among agencies.  The four HHS agencies—ASPR, CDC, FDA, and NIH—used EOCs to  coordinate modeling efforts during responses to Ebola, Zika, and  pandemic influenza outbreaks. For example, during the 2015-2016  Zika outbreak, CDC’s EOC served as the command center for  monitoring and coordinating the response by bringing together CDC  scientists with expertise in areas such as arboviruses (the category  that includes Zika), reproductive health, birth defects, and  developmental disabilities. CDC modelers and officials told us that  they had weekly strategy meetings and briefings with response  leadership within the EOC where they discussed which modeling  questions to prioritize. In general, CDC modelers in the EOC were  expected to coordinate with modelers from other agencies within and  outside of HHS—such as ASPR, FDA, NIH, and the Department of  Homeland Security—to produce timely estimates of cases,  hospitalizations, and deaths. These estimates can inform response  leadership and enable them to assess the speed and impact of the  geographic spread of the pandemic. Modelers in the EOC also  provide support to decision makers as they examine the potential  effects of various response options. These options include when and  how to deploy Strategic National Stockpile assets, such as influenza  antiviral drugs and mechanical ventilators. We found the use of  EOCs to be consistent with leading collaboration practices we have  previously identified, such as defining and articulating a common  outcome.

Public Health Emergency Medical Countermeasures Enterprise  (PHEMCE). The four HHS agencies also participated in PHEMCE, a  federal interagency body formed by HHS in 2006 that coordinates the  development, acquisition, stockpiling, and recommendations for use  of medical products that are needed to effectively respond to a variety  of high-consequence public health emergencies. PHEMCE is led by  ASPR and also includes partners at the Departments of Defense,  Veterans Affairs, Homeland Security, and Agriculture. PHEMCE’s  2017-2018 strategy and implementation plan, its most recent,  identified Ebola, pandemic influenza, and emerging infectious  diseases more broadly as high-priority threats. PHEMCE leadership  could ask modelers to address questions related to these infectious  diseases, according to ASPR modelers and officials. According to  ASPR officials, such questions tend to support larger response- related efforts, and modeling results are often incorporated into final  reports and products. According to ASPR officials, as of February  2020, the PHEMCE structure has been updated and it is unclear how  modeling fits into the new structure. We found that coordination  through PHEMCE is consistent with leading collaboration practices  such as establishing mutually-reinforcing or joint strategies.

Working groups. Modelers with the four HHS agencies have  participated in working groups related to infectious disease modeling  (see table 2). The use of working groups and similar bodies is  consistent with leading collaboration practices that we have previously  reported as useful for enhancing and sustaining interagency  collaboration, such as identifying and addressing needs by leveraging  resources. For example, CDC and ASPR modelers participated in  the National Science and Technology Council’s Pandemic Prediction  Forecasting Science and Technology Working Group, which facilitates  coordination among numerous federal agencies. In 2016, this group  produced a report that identified challenges in outbreak prediction and  modeling for federal agencies and offered recommendations for  federal actions to advance the development and effective application  of outbreak prediction capabilities.

Description  This interagency working group, directed by the National Science and Technology  Council, is responsible for analyzing the state of infectious disease modeling and  prediction, and facilitating coordination among numerous federal agencies.  According to CDC modelers and officials, as of October 2018, the charter for this  group is no longer active, and it meets on a voluntary, ad hoc basis.  According to CDC officials, this group connects modelers by holding seminars,  managing an email list, and arranging for members to peer review one another’s  models. This group had over 160 participants from various centers across CDC,  as of June 2019.  During the 2014-2016 Ebola and 2015-2016 Zika outbreaks, the Department of  Health and Human Services’ (HHS) Office of the Assistant Secretary for  Preparedness and Response (ASPR) established temporary modeling  coordination groups that brought together government agencies and academics to  share early modeling results and discuss pressing questions that could be  answered through modeling, according to ASPR modelers and officials. A wide  range of entities participated in these groups, including the four HHS agencies,  other federal agencies such as the Departments of Defense and Homeland  Security, universities, and foreign entities, such as the World Health Organization  and the United Kingdom. According to ASPR modelers and officials, there are no  plans to convene modeling coordination groups unless there is an ongoing  infectious disease outbreak.

Joint model development. ASPR and CDC modelers jointly  developed some modeling products during outbreak responses. For  example, during the 2014-2016 Ebola response, ASPR and CDC  developed a model to estimate future numbers of Ebola patients  needing treatment at any one time in the United States. According to  a publication describing the model, policymakers have used it to  evaluate responses to the risk for arrival of Ebola-infected travelers,  and it can be used in future infectious disease outbreaks of  international origin to plan for persons requiring treatment within the  United States. Building these positive working relationships can help  bridge organizational cultures by building trust and fostering  communication, which facilitates collaboration and is vital in  responding to emergencies. For example, in our 2011 report, we  found that, through interagency planning efforts, federal officials built  relationships that helped facilitate the federal response to the H1N1  influenza pandemic. Similarly, HHS officials said that federal  coordination during the H1N1 pandemic was much easier because of  these formal networks and informal relationships built during  pandemic planning activities and exercises.

Memoranda of understanding. The four HHS agencies have  entered into various agreements through memoranda of  understanding in order to define their relationships for coordinating  infectious disease modeling (see table 3). Generally these  memoranda were between individual agencies rather than  department-wide. We found that the use of memoranda of  understanding was consistent with leading collaboration practices,  such as agreeing on roles and responsibilities. Our prior work found  that agencies that articulate their agreements in formal documents  can strengthen their commitment to working collaboratively.  Similarly, CDC modelers and officials said that written agreements  can reduce the possibility of misunderstandings or disagreements and  help ensure that participants have a mutual understanding of  collaboration goals. For example, in the absence of such written  agreements, the potential for duplication is increased because  agencies could be working on similar types of models without one  another’s knowledge.

Table 3. Selected Examples of Memoranda of Understanding for Coordinating on Infectious Disease Modeling  Collaborating agencies   The Office of the Assistant  Secretary for Preparedness  and Response (ASPR) and  Centers for Disease Control  and Prevention (CDC)

ASPR and the Food and  Drug Administration (FDA)

Description  From 2013 to 2018, CDC and ASPR had a memorandum of understanding to promote collaboration,  provide expertise, and facilitate data and information exchange related to infectious disease modeling.  This agreement expired in 2018. ASPR modelers and officials told us that, as of August 2019, it had not  been updated, and there were no plans to do so. Despite this, according to CDC modelers and officials,  the substance of the agreement is still being followed. CDC modelers and officials told us they continue  to collaborate with ASPR modelers on the development of models that address questions of mutual  interest. For example, for the ongoing Ebola response, CDC modelers and officials said they have kept  ASPR informed on modeling efforts, and ASPR shares data on vaccine production that is included in  one of the models.  ASPR and FDA have a memorandum of understanding to promote collaboration and enhance  knowledge and efficiency by providing for the sharing of information and expertise. This memorandum  was in place from 2012 to 2017, and was then renewed in 2019. It remains valid unless modified by  consent of both parties or terminated by either party immediately upon written notice in the event that a  federal statute is enacted or a regulation is issued by a federal partner that materially affects the  memorandum. According to FDA modelers and officials, the agreement facilitates collaboration related  to FDA’s Medical Countermeasure Initiative and FDA’s role in supporting the HHS-led Public Health  Emergency Medical Countermeasures Enterprise (PHEMCE). FDA modelers and officials told us that  the agreement supports the frequent, ongoing collaborations between FDA and ASPR, including  collaboration related to preparedness for emerging infectious diseases. However, FDA modelers and  officials said, while no specific steps have been taken with regards to collaborating on infectious  disease modeling under the agreement, modeling assistance could be provided in the future, if needed.

Description  From 2013-2018, ASPR had a memorandum of understanding with NIH’s Models of Infectious Disease  Agent Study program to (1) enable Models of Infectious Disease Agent Study program researchers to  work with ASPR as part of public health preparedness and response activities, (2) share data and  information, and (3) support model development and use in the HHS modeling hub. This agreement  has expired. ASPR modelers and officials told us that, as of August 2019, it has not been updated, and  there were no plans to do so.  Since 2015, CDC has had a memorandum of understanding with NIH’s Models of Infectious Disease  Agent Study program, to promote collaboration and facilitate the exchange of data, tools (models),  methods, and information. It was set to expire in February 2020.  From 2013 to 2018, ASPR had separate memoranda of understanding with the Departments of  Defense and Homeland Security to promote collaboration, provide expertise, and facilitate data and  information exchange. The goals of the collaboration in both agreements were to explore ways to,  among other things: share analytical approaches and efforts, such as modeling and simulation tools, in  support of public health preparedness and response activities; provide personnel as needed to facilitate  analytical efforts; and share data and information. These goals were similar to those laid out in the  agreement between CDC and ASPR. These agreements expired in 2018. ASPR modelers and officials  told us that, as of October 2019, they have not been updated, and there were no plans to do so.

Forecasting competitions. CDC and NIH have sponsored formal  forecasting competitions to improve modeling for Ebola, Zika, and  seasonal influenza. According to a report from the National Science  and Technology Council, controlled, multi‐center modeling contests  and projects generate valuable insights. For example, they often show  that simpler models perform as well as more complex models and that  ensemble models, which combine the results of multiple models to  predict an outcome, perform better than an individual model. Such  competitions are consistent with a leading collaboration practice we  previously reported: identifying and addressing needs by leveraging  resources. In this case, such leveraging allowed CDC and NIH to  obtain additional benefits and insights on models that may not  otherwise be available. These modeling competitions can therefore  help the HHS agencies better prepare for future outbreaks through  coordination with participants. The following are examples of  forecasting competitions sponsored by CDC or NIH:

Ebola competition. NIH’s Fogarty International Center held an  Ebola forecasting competition from August to December 2015,  related to the 2014-2016 West African Ebola outbreak, to compare  the accuracy of predictions from different Ebola models, among  other things. According to NIH modelers and officials, lessons  learned from the challenge were that (1) with regard to short-term  incidence predictions, ensemble estimates were more consistently  accurate than predictions by any individual participating model; (2)  as expected, more accurate and granular epidemiological data  improved forecasting accuracy; (3) the availability of contextual  information, including patient-level data and situational reports, is  important for accurate predictions; (4) the accuracy of forecasting  was not positively associated with more complex models; and (5)  coordination of modeling teams and comparison of different  models is important to ensure robustness of predictions.  According to NIH officials, based on these lessons and in  response to the most recent Ebola outbreak, NIH has established  a coordination group to share information about modeling and  data sharing for this particular outbreak and a formal model  comparison is underway under World Health Organization  leadership.

Aedes (Zika) competition. In 2019, CDC hosted a forecasting  competition related to using models to predict the presence of  Aedes mosquitoes, which is a vector for the Zika virus. Evaluating  these models can, according to CDC, help clarify model accuracy  and utility, the seasonal and geographical dynamics of these  mosquitoes, and key directions for future research. According to  CDC documentation, these advances can contribute to improved  preparedness for arboviral invasion in the United States and in  other regions where Aedes suitability may be limited and  changing. CDC plans to evaluate forecasts for this competition in  early 2020, as soon as final surveillance data for 2019 are  available.

FluSight (seasonal influenza) competition. CDC holds an  annual seasonal influenza forecasting competition—known as  FluSight—to facilitate efforts to engage external researchers to  improve the science and usability of seasonal influenza forecasts.  The results of the competition are evaluated by the CDC Influenza  Division, which works with state and local partners to determine  whether the results are useful to them and if there are other  metrics, milestones, or targets that would be more helpful in  making public health decisions. According to CDC officials in  February 2020, the results from the FluSight competition are not  directly incorporated into pandemic influenza forecasting because  the most accurate seasonal influenza forecasts would not  necessarily be the most accurate pandemic influenza forecasts.  According to these officials, the overall lessons learned from the  FluSight competition relate to how to quantify, visualize, and  communicate model results and model accuracy, as well as the  value of forecast ensembles to summarize multiple models. CDC  officials said these lessons are incorporated into pandemic  influenza forecasting plans.

Coordination with academic and other modelers. CDC  coordinated infectious disease modeling efforts with academic and  other modelers through various means, including the following:  Intergovernmental Personnel Act agreements. CDC has used  agreements under the Intergovernmental Personnel Act of 1970 to  collaborate with external experts on modeling efforts. For  example CDC’s Division of Vector-Borne Diseases had an  agreement from 2014 to 2017 to assign a CDC official to the  Harvard T.H. Chan School of Public Health. The agreement was  to help CDC integrate with a larger modeling community and  provide the Harvard School of Public Health with expertise in  arboviral diseases and applied public health.

Vector-Borne Disease Centers of Excellence. CDC has funded  the Vector-Borne Disease Centers of Excellence, which are  engaged in modeling-specific projects. In 2017, CDC established  five universities as regional centers of excellence to help prevent  and rapidly respond to emerging vector-borne diseases across the  United States. According to CDC, the goals of the centers are to  build effective collaboration between academic communities and  public health organizations at federal, state, and local levels for  surveillance, prevention, and response, among other things.

Support for other governmental entities. CDC has coordinated  with other entities—such as state and local officials—to provide  modeling tools, estimates of case counts, or effects of  interventions during the Ebola, Zika, and pandemic influenza  outbreaks. For example, CDC developed pandemic influenza  models for state and local health departments to use in influenza  pandemic planning activities. The tools are available on the CDC  pandemic influenza website and from ASPR’s emergency  preparedness information portal. As previously discussed, officials  from two of the states we spoke with said they generally were  unaware of the availability of the models. According to CDC  modelers and officials, these models were developed in the mid- 2000s for pandemic influenza planning and remain useful but had  not been a priority to update because they have not received a  request to do so.

Informal collaboration. CDC has engaged in a range of informal  collaborations related to infectious disease modeling. According to  CDC modelers and officials, modelers often develop relationships  through conferences or other contacts. For example, CDC  modelers and officials said they informally collaborated on Ebola  modeling needs with academic institutions, as well as modelers  and analysts in the World Health Organization and other U.S.  government agencies, such as the Federal Emergency  Management Agency. For example, CDC modelers and officials  told us that model estimates produced under collaboration with  academics helped inform decisions about how many beds to be  ordered and delivered on the ground in West Africa during the  2014-2016 Ebola Outbreak. Similar to the forecasting  competitions described above, such informal coordination  mechanisms are consistent with the best practice of identifying  and addressing needs by leveraging resources, thus obtaining  additional benefits that may not be available if they were working  separately. For example, we have previously reported that  informal collaboration mechanisms—such as building  relationships between key personnel and soliciting input for  research projects—can provide the opportunity to leverage  expertise.

HHS Agencies Do Not Fully Monitor, Evaluate, and Report on Coordination Efforts

CDC and ASPR modelers and officials did not routinely monitor, evaluate,  and report on coordination efforts for infectious disease modeling. While  CDC did conduct after-action reviews for Ebola and Zika, which included  a review of modeling efforts, such reviews are not routine outside of a  response and do not examine modeling coordination between agencies.  ASPR modelers and officials told us they saw no reason to monitor  coordination efforts under the memorandum of understanding with CDC  because such memoranda outline expectations rather than requirements.  However, we have found that agencies that create a means to monitor,  evaluate, and report the results of collaborative efforts can better identify  areas for improvement. We have previously reported that progress  reviews or after action reviews can be useful mechanisms for monitoring,  evaluating, and reporting on collaborative efforts. For example, we  previously reported that, to monitor, evaluate, and report on the status of  achieving the Healthy People 2010 objectives, HHS held progress  reviews in which the federal agencies with lead responsibilities for a focus  area reported on the progress towards achieving the objectives. During  these reviews, the participating agencies discussed the data trends,  barriers to achieving the objectives, strategies undertaken to overcome  barriers, and alternative approaches to attain further progress. By holding  similar progress reviews in which CDC and ASPR evaluate and report on  coordination efforts for infectious disease modeling, these agencies could  be better positioned to identify and address challenges prior to infectious  disease outbreaks occurring, which could lead to improved responses.  Further, there is the potential for overlap and duplication of modeling  efforts across agencies, which may not be identified if coordination efforts  are not effectively being monitored, and which could lead to inefficiencies.

The memorandum of understanding between CDC and ASPR had  expired in 2018. Agency officials told us they had no plans to review or  update the agreement. According to ASPR modelers and officials, the  agreement has not been updated because it was not a priority and the  substance of the expired agreement is being followed. However, without  an active agreement in place that clearly defines the goals of the  collaborative effort and the roles and responsibilities of participants, a lack  of understanding and agreement becomes more likely, particularly as  agencies’ priorities evolve over time. Our prior work on leading  collaboration practices found that agencies that articulate their  agreements in formal documents can strengthen their commitments to  working collaboratively, and that such agreements are most effective  when they are regularly reviewed and updated.

Further, we found that the memorandum of understanding between ASPR  and CDC was not fully implemented when it was active. For example,  according to this agreement, CDC was to appoint a designee to  participate in a steering committee related to modeling within HHS.  However, ASPR modelers and officials told us that this steering  committee was never formed because of changing leadership and priorities. They told us that HHS does not have any intention to form such  a steering committee in the future. However, our past work shows  creating a steering committee or other similar coordination mechanism  could help facilitate monitoring of coordination efforts.

We similarly found that other memoranda of understanding related to  infectious disease modeling were not fully implemented. For example,  although ASPR had a 2013-2018 memorandum of understanding with  NIH’s Models of Infectious Disease Agency Study program, ASPR  modelers and officials said they rarely use models funded by NIH,  including those funded through the program. In particular, ASPR  modelers and officials recalled only using one such model in recent years.  That model, known as “FluTE,” is an influenza model that was used as  part of a larger study on vaccine availability. However, ASPR modelers  faced challenges in using this model. Specifically, these ASPR modelers  and officials said the FluTE model initially was not compatible with  ASPR’s computer system, so software engineers had to modify the  source code to resolve the compatibility issue. The model did not have  documentation describing its parameters, according to ASPR modelers  and officials, so they had to read through the model’s source code to  understand them. Similarly, regarding a separate agreement between  ASPR and FDA, FDA modelers and officials said that, while there is  ongoing information sharing, no specific steps have been taken with  regard to collaborating on infectious disease modeling under the  agreement. However, these modelers and agency officials said that  modeling assistance could be provided in the future, if needed.

CDC and ASPR Generally Followed Identified Practices for Infectious Disease Modeling, but CDC Has Not Fully Ensured Model Reproducibility

We identified four elements of practices for developing and assessing  models: (1) communication between decision maker and modeler, (2)  description of the model, (3) verification, and (4) validation. We  determined that CDC and ASPR generally followed these GAO-identified  practices for 10 models we reviewed. However, for four of the 10  models, CDC modelers did not provide all of the details needed in the  verification steps to reproduce their model results, which is inconsistent  with HHS guidelines on transparency and reproducibility.

CDC and ASPR Generally Followed Identified Modeling Practices but Did Not Always Fully Assess Model Performance

According to our interviews with agency modelers and experts, along with  our review of selected literature, there are no documented standards that  prescribe the steps agencies must or should follow when developing and  assessing models. However, based on our interviews and review, we  identified four broad elements of the modeling process that modelers  generally consider. They are:  1  communication between modelers and officials to refine questions to  be addressed by the model, such as geographic spread of the  disease and total cases of the disease;  2  description of the model, including detailed descriptions of  assumptions and data sources used;  4  validation.

Figure 4 outlines the model development and assessment process.

Based on our assessment of 10 selected models, we found that CDC and  ASPR generally took steps that corresponded to our four elements, and  agency modelers generally agreed with our assessment of each model.  See table 4 for more information on the elements. See appendix III for a  list of models we reviewed and a complete list of the steps we identified  that make up each element.

Communication between modeler and decision maker. In all 10  agency models we reviewed, we found that agencies took all the steps  we identified for communication between decision maker and modeler. In  some cases, these steps were formalized, while in others they were  informal. For example, CDC modelers responding to Ebola ensured  communication with decision makers by following a memo template they  developed, which has a section requiring modelers to communicate key  aspects of their model. These modelers noted, however, that they would  not follow all the steps in their memo template for models developed  during an outbreak because of time constraints. CDC modelers  responding to pandemic influenza noted they do not have formal best  practices for communication about key model aspects to decision makers,  and a CDC modeler responding to Zika highlighted the role of CDC’s  Emergency Operations Center (EOC) in communication between decision  makers and modelers, which is activated only during a response. ASPR  modelers noted that—as a best practice—they hold a discussion for all  new models, in which decision makers describe what they are looking for  and modelers describe what they can provide.

Description of the model. In nine of the 10 models we reviewed,  modelers took all steps we identified for describing their model type,  inputs, outputs, assumptions, and limitations. In one case, ASPR’s  “flumodels” package, the agency did not carry out the step of describing  the model’s limitations. ASPR modelers told us they did not do so  because they expected the model’s intended users—primarily federal  public health modeling experts—would understand the limitations of their  model, an assumption we find reasonable.

Verification. In six of 10 models reviewed, we found agency modelers  followed most of the steps we identified for model verification. However,  in four of the seven CDC models reviewed, CDC did not publish the  model’s code, a part of model reproducibility and a model verification  step. We examine CDC’s policy and efforts on reproducibility in more  detail below.

Validation. For four of the 10 models we reviewed, agencies performed  few validation steps. In all three CDC pandemic influenza models we  reviewed, and the ASPR Zika model, sensitivity analysis was the only  validation step performed. CDC influenza modelers said they did not  perform other validation steps because of a lack of comparable external  models or applicable data which could be used for other types of model  validation. For example, they said they could not validate their models  using real-world data because they made projections for scenarios that  did not come to pass (e.g., an unmitigated pandemic influenza outbreak).  They said they have continued to look for comparable models that could  be used to cross-validate their model estimates. ASPR modelers  responding to the Zika outbreak also did not have access to comparable  external models or applicable data to confirm their model projections, but  have since attempted to validate their model. For the other six models we  reviewed, agencies carried out most but not all validation steps. For  example, CDC modelers responding to Zika also said they did not  perform cross-validation (comparison of different model results to each  other) for their Zika model because of a lack of comparable models.  However, these ASPR and CDC Zika modelers said they have attempted  to validate their model since its publication as new data emerges, and we  found this occurred.

Assessing Model Validity  Assessing model validity means determining  whether a model is sufficiently accurate for its  purpose. Several methods are available,  including the following:

Modelers can compare the results of the  model against real-world data the model  was designed to predict.  If there are no such data, another method  is to determine how much the model  projections change in response to  changes in input data. This is known as  model sensitivity analysis.

Modelers can also withhold a part of the  available data in building the model and  then confirm the model can reproduce the  withheld data.  real-world data is to run the model along  with a separate, independent model using  the same input data, and comparing the  outputs.

CDC modelers and ASPR modelers responding to Zika followed identified  practices and validated their model projections for the Zika outbreak,  although their efforts yielded mixed results for model performance. CDC  modelers responding to Zika attempted to estimate whether there was an  enhanced risk of microcephaly in infants born to expectant mothers  infected with Zika. Using data available during the initial stage of the  outbreak, they calculated the enhanced risk to be between 0.88 and 13.2  percent if the mother was infected in the first trimester. In two subsequent  studies using later data on the actual incidence of microcephaly as a  result of the outbreak, other researchers found the enhanced risk was  within the bounds of CDC modelers’ earlier projections: a 10 percent  enhanced risk in one study and an 8.3 percent enhanced risk in the other.  In the second case, ASPR modelers attempted to estimate potential new  cases of Guillain-Barré syndrome, a rare disorder in which the body’s  immune system attacks part of its own nervous system, in places  burdened by Zika infection. Their initial projections were that there would  be between 191 and 305 new cases in Puerto Rico, a three- to five-fold  increase above the number normally expected. ASPR modelers  attempted to verify these results themselves and found that the incidence  did increase, but only two-fold, to 123 new cases. through independent performance  evaluations. For example, agencies  sometimes host modeling competitions, in  which independent modelers compare the  predictive performance of multiple models  under controlled conditions using  standardized data. The National Institutes  of Health hosted an Ebola forecasting  competition in 2015, and the Centers for  Disease Control and Prevention (CDC)  launched its FluSight competition in 2013.

The Challenge of Modeling During an  Outbreak.  Early in the 2014-2016 Ebola outbreak,  Centers for Disease Control and Prevention  (CDC) officials faced the challenge of  answering questions with limited data and  time. In order to estimate the potential number  of future cases and to aid in planning for  additional disease-control efforts, CDC  developed EbolaResponse, an Excel  spreadsheet-based model that could forecast  how interventions would impact the outbreak.  Using EbolaResponse, CDC predicted in early  September 2014 that 1.4 million cases of  Ebola could occur in Liberia and Sierra Leone  by January 2015, if the world health  community did not increase interventions.  These estimates included a correction factor  intended to account for the underreporting of  cases and that, according to officials, was to  represent model uncertainty. Partly because  of these estimates of rapidly increasing cases,  CDC and others increased intervention by  sending more treatment units, personnel, and  medical supplies in late 2014.  EbolaResponse was created to model the  effects of intervention, and it later turned out  to be unreliable for the 4-month forecast that  CDC used to support its request for increased  intervention. Independent analysis found that  the model could forecast cases up to a month  ahead well but could not provide any measure  of uncertainty. Furthermore, the model was  unable to make accurate forecasts much  beyond 3 months, a limitation that was  common among the models used during the  outbreak. CDC later reported that roughly  8,500 cases, or 34 percent of the corrected  EbolaResponse prediction of 25,000 cases,  occurred in Liberia by the end of January  2015.

We also found that CDC and ASPR modeling approaches varied  somewhat, while generally remaining within the bounds of our identified  practices. For example, all the agency modeling groups reviewed their  model assumptions, but they also varied in whether this review was  formal or informal and internal or external. CDC modelers responding to  Ebola use a formal internal peer review process during non-outbreak  periods, as well as a detailed checklist to ensure communication with  decision makers, full consideration of model inputs and outputs,  quantification of model uncertainty, and validation of the model. By  contrast, CDC modelers responding to Zika told us they do not have a  formal system for evaluating their models, and instead rely on their own  review of model assumptions. ASPR and CDC pandemic influenza  modelers told us their modeling approach also relied on peer review, but  the review was done by external experts; informally for ASPR and  formally for CDC pandemic influenza modelers.

There are several reasons agency modeling approaches can vary.  According to agency modelers, agency modeling practices can be  influenced by the availability of time, data, and comparable models. For  example, CDC pandemic influenza modelers and officials said they follow  a shortened process when facing time constraints by documenting model  development in a journal publication after the model has already been put  to use. Similarly, CDC modelers responding to Ebola noted that, during a  response, a lack of time may mean models are not reviewed through  CDC’s formal clearance process; instead, a more informal review of  model results may occur.

CDC and ASPR modelers also described variation in the complexity of  the models they use. They said they sometimes use both simple and  complex models for the same disease and during the same outbreak.  CDC modelers and officials responding to Ebola said that they preferred  models run in spreadsheet programs for their transparency and  communicability, whereas CDC influenza modelers mostly use dedicated  statistical software programs to run models and spreadsheets for  communicating with state and local health departments. ASPR modelers  develop more complex prediction models so that they can be reused to  answer more than one question, as opposed to models run in  spreadsheet programs that are designed to answer one question.

Experts and agency modelers generally agreed that infectious disease  models should not be more complex than is necessary to answer the  questions they were developed to address. A simpler model may be run  on a variety of software programs, ranging from spreadsheet programs to  specialized programming languages that can do statistical analysis. One  downside of models run in spreadsheet programs, according to CDC  influenza modelers, is that it is harder to conduct quality control  measures. Two experts we spoke to, along with CDC Zika modelers, also  expressed concerns with reliability and reproducibility of models run in  spreadsheet programs.

CDC Has Not Fully Implemented a Policy to Ensure Model Reproducibility

Since 2002, HHS agencies responsible for disseminating influential  scientific, financial, or statistical information have been required to ensure  methods used to develop this information are “reproducible.” A 2019  report from the National Academies of Sciences, Engineering, and  Medicine noted that the scientific enterprise depends on the ability of the  scientific community to scrutinize scientific claims and to gain confidence  over time in results and inferences that have stood up to repeated testing.  As part of this process of scrutiny, a study’s data and code should be  made available so that the study is reproducible by others. The National  Academies report defines reproducibility as obtaining consistent  computational results using the same input data, computational steps,  methods, code, and conditions of analysis. Reproducibility is specifically  addressed earlier in this section in our discussion of model verification, a  step that requires making code available for independent review.

HHS requires its component agencies to either follow HHS department  guidelines on reproducibility or to ensure their own guidelines include a  high degree of transparency about the data and methods used to  generate scientific information. HHS guidelines require that, in a  scientific context, agencies identify the supporting data and models for  their published scientific information and provide sufficient transparency  about data and methods that an independent reanalysis could be  undertaken by a qualified member of the public. When asked whether  CDC has specific policies related to reproducibility that would have  applied to provision of model code in their published scientific research,  CDC referred to its guidelines developed in response to the 2002 HHS  Guidelines. However, CDC guidelines do not contain any reference to  reproducibility, models, or provision of model code. CDC guidelines for  review of scientific information provided to the public focus on  completeness, accuracy and timeliness, data management and analysis,  clarity and accuracy of presentation, and validity of interpretation of  findings. CDC’s policy on public health research and non-research data  management and access does not make any reference to reproducibility  or model code. This lack of reference to reproducibility in CDC’s  guidelines and policies is not in accordance with HHS guidelines.

Our review found four instances in which CDC modelers did not provide  model code when they published their models. CDC modelers said in  some instances, issues with publication formats made the code difficult to  share, they did not have time to produce a user-friendly version of the  code, or they would share the code upon request.

By contrast, ASPR modelers provided code for every model within our  review when they published their models. While neither agency cited a  specific HHS policy that required them to share model code, ASPR  modelers noted that their internal peer review process typically includes  sharing model source code with other modelers within PHEMCE. In our  review of HHS guidelines and agency-specific guidance for these HHS  guidelines, we found that, of three published agency guidance, two  require reproducibility, or transparency for the methods used in the  reports they issue to the public. Of these agencies, CDC was the only one  that did not explicitly require transparency or reproducibility.

The National Academies report noted that researchers have to be able to  understand others’ research in order to build on it. This report also notes  that the ability of qualified third parties to reproduce a model using  published code is important because it can reveal mistakes in model  code, which can lead to serious errors in interpretation and reported  results. If researchers do not share an important aspect of their study,  such as their model code, it is difficult to confirm the results of their  research and ultimately produce new knowledge. One agency official  acknowledged the importance of releasing model code, noting that HHS  could benefit by ensuring policies across the agency are consistent  regarding reproducibility and transparency in modeling. By not specifically  addressing reproducibility in their policy on dissemination of scientific  information, CDC risks undermining the reliability of the scientific  information they disseminate to the public.

Modelers Faced Several Challenges and Have Worked to Address Them

Based on our review of documents and reports from agencies, as well as  expert and agency interviews, we identified three categories of challenges  that CDC modelers and officials and ASPR modelers faced when  modeling for Ebola, Zika, and pandemic influenza, along with steps they  took to address the challenges. The categories are data, resources, and  communicating results.

According to a 2016 report from the National Science and Technology  Council (NSTC), obtaining timely and accurate data and information has  long been a major challenge to an effective response during an infectious  disease outbreak. One expert described reliable data as a modeler’s  most limited resource. Until data of sufficient quality and quantity are  available and usable, the predictive value of models will be limited.

Agency modelers and officials provided examples of data-related  challenges, which we categorize as follows:

Data Access. Public health data, according to one expert, often has  access restrictions. For example, ASPR modelers said their ability to  access data during the 2014-2016 Ebola outbreak was reduced by a  need to enter into agreements with data-owning countries in order to  obtain patient data. Modelers said there were agreements between  CDC and data owners, but further agreements would have been  required for ASPR to obtain data because the agreements did not  authorize CDC to share data with its partners. In addition to the  example above, the lack of data sharing agreements during the 2014- 2016 Ebola outbreak response led to modeling projects being  delayed, according to a CDC publication. ASPR modelers said their  inability to obtain data without a data-sharing agreement made it  challenging for them to developing a current, reliable estimate of  Ebola incidence before modelers could start creating future estimates  of disease incidence. They said that, as a result, they instead  developed a statistical model, which provided less reliable estimates  of future numbers of disease cases than they would have preferred.  Modelers said they worked to address this challenge by obtaining  data and indirect information through personal relationships with other  modelers. In addition to the example provided above, CDC modelers  and officials responding to Ebola described experiencing data access  challenges.

Data availability. Without sufficient data, models may be unable to  identify an epidemic’s key drivers, which could result in misdirected  intervention efforts. For example, ASPR modelers noted that during  the 2015-2016 Zika outbreak response, there were substantial limits  on available data, and data that were available could be unreliable  and delayed. They said it was very difficult, and in many cases  effectively impossible, to determine the accuracy of forecasting  models for the evolving Zika outbreak. In addition, CDC officials and  modelers responding to Ebola, Zika, and influenza described  encountering limits on available data as an ongoing challenge. Steps  that modelers said they have taken to address data availability  challenges include designing models to use a minimum amount of  data, building trust and communication with stakeholders who might  be able to provide additional data, and updating data systems to  provide all available information. According to CDC modelers, data  availability will likely continue to pose a challenge to public health  responses.

Data collection. There is limited manpower during an infectious  disease outbreak response, which can limit the health care system’s  ability to collect data, according to CDC modelers and officials  responding to Ebola and ASPR modelers. ASPR modelers said if a  provider has to fill out a time-consuming form, then they will be  delayed in treating the next patient. In order to address this challenge,  CDC modelers and officials and ASPR modelers said data requesters  should ask for the minimum amount of data needed. For example,  CDC modelers and officials said they focus on understanding what  data are essential, how they are collected, and the policy implications  of reporting those data. A 2016 NSTC report recommended the  federal government address this challenge by identifying questions  likely to arise during an outbreak response, in order to help define and  prioritize data collection and modeling goals.

Data quality. Experts said creating models with low-quality data can  result in inaccurate models that may not provide clear answers to  decision maker questions. For example, CDC modelers and officials  responding to the 2015-2016 Zika outbreak said the data quality  varied, based on many factors such as surveillance systems that were  doing different things and defining reporting Zika cases differently,  and the availability of diagnostic testing. Because of data quality  concerns, there were questions about whether modeling could be  conducted, but through discussions modelers and agency officials  said they were able to address challenges. To address such  challenges, CDC modelers and officials responding to Zika said they  worked to improve public data sharing, sent an official to the Pan- American Health Organization to help interpret data and understand  the outbreak from an international perspective, and used modeling  methods appropriate for data with high levels of uncertainty. In  addition to the example provided above, CDC modelers and officials  responding to Ebola, ASPR modelers, and experts described  experiencing data quality challenges.

Data integration. CDC modelers and officials responding to Ebola  and Zika also faced the challenge of integrating multiple data sets,  which may not be standardized or in a readily usable form. For  example, CDC modelers and officials responding to Zika found it  challenging to integrate data as the definition of the disease was  refined over time. As the definition got more specific and monitoring  systems became available, it was hard to establish data trends, these  officials said. Further, there were variations in who would be tested,  with all people who exhibited symptoms being tested in some areas,  and only pregnant women in others, and also when data would be  placed into a combined form and reported to state, national, or  international officials, according to these officials. This integration  issue may have complicated efforts to conduct modeling such as  determining the risk of microcephaly in infants over time. In order to  address this challenge, Zika modelers said they set up an online data  repository to, among other things, standardize shared data.

CDC modelers and officials responding to Ebola and Zika, along with  experts, said finding staff with sufficient training to support modeling  during an infectious disease outbreak represented an ongoing  challenge. For example, CDC modelers responding to Zika said it can  be difficult to find modelers with both an epidemiological background and  skills in coding and mathematics. Modelers and agency officials said  those who had the correct skills were in high demand, and it was difficult  to fully engage them in the Zika outbreak response. They said they could  have conducted more modeling or completed modeling efforts more  rapidly if they had had access to more modelers with the right skills. To  address this challenge, modelers participate in trainings on how to  communicate what models can and cannot do, participate in working  groups that support modeling efforts, employ the Intergovernmental  Personnel Mobility Act Program, maintain collaborations with external  partners, and host students and researchers. ASPR modelers said they  faced personnel challenges in their modeling efforts but that they were  wide-ranging and not specific to Ebola, Zika, or pandemic influenza.

According to a 2016 NSTC report, time constraints make it challenging for  researchers to keep up with scientific literature during an outbreak. CDC  influenza modelers said they faced this challenge and that they conduct  weekly searches for new influenza publications, which normally identify  about 150 publications each week. To address this challenge, modelers  said they conduct literature searches, share the responsibility of reviewing  publications and informing others of their content, talk to experts, and  attend conferences. Modelers said this challenge was more easily  addressed than others.

Communicating model results can be difficult and, as modelers and  agency officials pointed out, decision makers will not give credence to  results from a model they do not understand. Model results, according to  CDC influenza modelers, are often nuanced and complicated, and  officials have to think about what pieces of information are the most  important to convey to a decision maker, the public, or health officials.  Furthermore, as one expert noted, the complexities of modeling can get  lost in translation, especially with the media, which may focus on only a  worst-case scenario. When modeling for infectious diseases,  appropriately communicating complex information has been described as  a constant challenge, and CDC influenza modelers described it as their  biggest challenge. CDC influenza modelers particularly noted the  challenge of communicating uncertainty. CDC influenza and ASPR  modelers said if decision makers did not understand the models, they  could misunderstand the results, which, according to ASPR modelers,  could lead to errors in decision making. CDC modelers and officials  responding to Ebola and Zika, CDC influenza modelers, ASPR modelers,  and experts described experiencing challenges communicating model  results to decision makers.

Clear communication may help prevent misunderstandings. For example,  one review article said officials may not understand what models can and  cannot do before an epidemic, and modelers may not be fully aware of a  decision maker’s needs. An expert said there is a need to constrain the  use of models intended to inform decisions so that the model does not  over- or under-influence a decision maker. And, according to ASPR  modelers, decision makers sometimes want a model to make a decision  for them, although models can only inform the decision making process.

They said this is less of a problem during an outbreak response, when  decision makers know they have to act based on incomplete information.

Some steps officials described taking to address communication  challenges were similar across CDC and ASPR officials. For example,  CDC modelers and officials and ASPR modelers said they took steps to  improve communication, such as working to develop relationships outside  of an outbreak and to improve how data are visualized. For example,  ASPR modelers and officials said they provided decision makers with a  website that displays an interactive influenza model known as ShinyFlu.  The website lets users adjust a model to see how its results could change  based on its inputs used. However, modelers said this only works if the  decision maker is willing to engage with data.

Other steps to address communication challenges were not discussed by  all modelers we spoke to. For example, ASPR modelers said that, when  they use models with high uncertainty, they do additional research to  assess and communicate how a model could be misrepresenting a real- world problem. Additionally, CDC modelers responding to Zika and CDC  influenza modelers said they sometimes use the language of weather  forecasting—which provides information on the risk of an event occurring  over a specified period of time—to help communicate model outcomes.

For all 10 of the models we reviewed, modelers communicated all the  information they had agreed to provide to decision makers, including  information about model uncertainty. Agency modelers and officials said  they provided this information through discussions with decision makers  and by showing decision makers the results of multiple modeling  situations to convey uncertainty.

Infectious disease modeling is one tool that can provide decision makers  with valuable information to support outbreak preparedness and  response. In particular, modeling can help answer questions that are  difficult to address in other ways because of practical, ethical, or financial  reasons. Federal agencies have recognized the importance of modeling.  CDC and ASPR reported using it to inform policy and planning questions  and, to a more limited extent, to inform planning and the use of resources.

HHS agencies that work on infectious disease modeling—ASPR, CDC,  FDA, and NIH—reported using multiple mechanisms to coordinate their  modeling efforts, including working groups, memoranda of understanding,  and coordination with academic and other external modelers. The use of  these mechanisms was consistent with many leading collaboration  practices, such as defining and articulating a common outcome and  addressing needs by leveraging resources. However, HHS does not  routinely monitor and evaluate its coordination efforts, as called for by  another leading collaboration practice, which limits the department’s  ability to identify areas for improvement. Further, there is the potential for  overlap and duplication of modeling efforts across agencies, which may  not be identified if coordination efforts are not effectively being monitored,  and could lead to inefficiencies. By holding progress reviews in which  CDC and ASPR evaluate and report on coordination efforts for infectious  disease modeling, these agencies could be better positioned to identify  and address challenges prior to infectious disease outbreaks, which could  lead to improved response efforts.

CDC and ASPR modelers generally followed GAO-identified modeling  practices, with the notable exception of model verification. Specifically,  CDC did not make model code available to others for four of the seven  CDC models we reviewed. HHS does not have a policy that requires its  agencies to share model code, but it does require its component agencies  to either follow its guidelines or ensure that their own guidelines include a  high degree of transparency to facilitate reproducibility by qualified third  parties. Without sharing code and other important information, CDC  cannot ensure that its models are reproducible, a key characteristic of  reliable, high-quality scientific research.

In order to facilitate HHS infectious disease modeling efforts, we are  making two recommendations.

The Secretary of Health and Human Services should develop a  mechanism to routinely monitor, evaluate, and report on coordination  efforts for infectious disease modeling across multiple agencies.   (Recommendation 1)

The Secretary of Health and Human Services should direct CDC to  establish guidelines that ensure full reproducibility of CDC’s research  by sharing with the public all permissible and appropriate information  needed to reproduce research results, including, but not limited to,  model code.   (Recommendation 2)

We provided a draft of this report to the Department of Health and Human  Services (HHS) for review and comment. In its comments, reproduced in  appendix IV, HHS agreed with our recommendations and noted that it  was developing a process to coordinate its infectious disease modeling  efforts across its components.

With regard to our second recommendation—that HHS should direct CDC  to establish guidelines that ensure the full reproducibility of CDC’s  research by sharing all permissible and appropriate information needed to  reproduce research results, including, but not limited to, model code— HHS’s comments indicated that CDC believes it has already completed  actions to implement this recommendation.

For example, the HHS comments state that CDC has established  policies such as “Public Access to CDC Funded Publications” and  “Policy on Public Health Research and Nonresearch Data  Management and Access” that ensure that results are made  available to the public, as appropriate. However, as we state in  our report, these policies do not contain any reference to  reproducibility, models, or provision of model code and therefore  do not fully address our recommendation.

CDC also said in the HHS comments that its methods—including  its practice of providing a copy of model code upon request—are  in line with standard practice in the scientific community and peer- reviewed journals. However, in the four instances we identified  where CDC modelers did not share code, code being available  upon request was only one of the reasons cited. Further, this  practice is inconsistent with those of the other HHS agencies we  reviewed, and may limit the ability of external researchers to  confirm the results of CDC’s research and ultimately produce new  knowledge.

As noted in our report, by not specifically addressing reproducibility in its  policies on access to data and publications, CDC risks undermining the  reliability of scientific information disseminated to the public. Therefore,  we did not change our recommendation in response to HHS’s comments.  We did, however, revise our report to include information on other HHS  agency policies related to reproducibility.

HHS also provided technical comments, which we incorporated as  appropriate.

As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  appropriate congressional committees, the Secretary of Health and  Human Services, and to other interested parties. In addition, this report  will be available at no charge on the GAO website at http://www.gao.gov.

If you are your staff have questions about this report, please contact  Timothy M. Persons, Chief Scientist, at (202) 512-6888 or  personst@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  Key contributors to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

In conducting our review of infectious disease modeling by the  Department of Health and Human Services (HHS) agencies, our  objectives were to (1) examine the extent to which HHS has used various  types of models to inform policy, planning, and resource allocation for  public health decisions for selected infectious diseases, (2) examine the  extent to which HHS coordinated their modeling efforts for selected  infectious diseases, (3) examine the steps HHS generally took to develop  and assess the performance of its models for the selected diseases and  steps it applied to a selection of infectious disease models, and (4)  describe the extent to which HHS has addressed challenges related to  modeling for selected infectious diseases.

For purposes of this review, we focused on HHS because of its focus on  scientific and technical issues related to disease modeling, role in  infectious disease outbreak preparedness and response activities, and  use of modeling for policy and regulatory issues related to disease. Within  HHS, we identified four agencies—HHS’s Office of the Assistant  Secretary for Preparedness and Response (ASPR), the Centers for  Disease Control and Prevention (CDC), National Institutes of Health  (NIH), and Food and Drug Administration (FDA)—which may develop or  use infectious disease models.

To inform all four objectives, we selected three naturally-occurring  infectious diseases that have pandemic or epidemic potential—Ebola  virus disease (Ebola), Zika virus disease (Zika), and pandemic  influenza—to use as examples of broader infectious disease modeling  efforts. We selected these diseases based on document review, their  inclusion on NIH’s pathogen priority list, modeling being conducted by  HHS agencies, and interviews with experts that we selected based on  their experience with infectious disease. Based on these steps, the team  selected diseases that fit into one of the three categories on NIH’s  pathogen priority list: the disease (1) can be transmitted easily from  person to person, resulted in a high mortality rate and had the potential  for major public health impact, might cause social disruption, and may  require special action for public health preparedness (Ebola), (2) was  moderately easy to disseminate, and required specific enhancements for  diagnostic capacity and enhanced disease surveillance (Zika), or (3) was  an emerging pathogen that could be engineered for mass dissemination  in the future because of availability, ease of production and  dissemination, and have the potential for high morbidity and mortality  rates and major health impacts (pandemic influenza).

HHS Use of Models to Inform Policy, Planning, and Resource Allocation Decisions

To examine the types of models developed by HHS agencies to inform  policy, planning, and resource allocation decisions, we reviewed  documents from 2009—the year of the last pandemic influenza outbreak  in the United States—to April 2019 to identify examples of models  developed by the agencies for the three selected diseases. For context  on and examples of the types of modeling that CDC and ASPR have  conducted, we reviewed published articles that CDC and ASPR officials  and experts provided to us or cited during the course of our review, such  as articles identified during interviews which we later obtained. We also  obtained selected internal memoranda, when available, that described  models used in the Ebola virus outbreak. We did not include FDA and  NIH in this review because FDA has a limited role in modeling, and NIH  generally funds, rather than conducts, modeling. This review yielded  articles and memoranda describing about 60 CDC and ASPR models.  See appendix II for a bibliography of model publications reviewed. We  then categorized the models using categories derived from a federal  working group report to characterize the types of modeling conducted and  the purpose of the modeling, when that purpose was identified.

To analyze each study, one analyst initially coded each study, and each  classification was then independently reviewed to verify that it had been  correctly classified and to resolve any categorization discrepancies. We  used these categories to describe types of modeling efforts undertaken  by HHS agencies. Because we focused on studies published between  2009 and 2019, our findings are not generalizable to models that were  developed outside of that time period. Additionally, because we relied on  agency officials or reviews of relevant agency documents and  publications to identify studies, we may not have captured all studies  relevant to our scope. Further, because CDC and ASPR modelers and  officials said that they do not publish every model they conduct, our  review was not intended to develop an inventory of the modeling  conducted during the time period. Therefore, we were unable to  determine the extent to which the models we identified represented  agency modeling efforts as a whole.

To describe the extent of model use for public health decision making, we  interviewed officials from HHS agencies identified as decision makers for  conducting the response to these selected diseases—CDC, ASPR, and  FDA—and officials who conducted the modeling. We also interviewed  two NIH institutes and one center about funding for research related to  modeling for the selected diseases. Additionally, we conducted semi- structured interviews of officials from five states concerning their use of  models prepared by HHS agencies for decision making, among other  topics. We selected these states based on a review of a CDC draft report  on states’ use of CDC models, on the level of influenza activity  experienced by states, and consideration of geographic variation by U.S.  region. During our review, we sought to identify the common types of  decisions that could be informed by models, as well as the considerations  that could impact the extent to which a decision maker requests and uses  models for specific types of decisions. Based on interviews with agency  officials and our review of HHS models we identified examples of models  that were used to make specific decisions during response and non- response times. Because we relied on officials to describe the extent to  which models inform decision making, we may not have captured all  relevant instances when models for the selected infectious diseases  informed decision makers.

To examine coordination and collaboration across HHS agencies, we  reviewed documents describing HHS agencies’ collaboration and  coordination mechanisms such as Memoranda of Understanding,  descriptions of Emergency Operations Center procedures, and after- action reports following infectious disease outbreaks. We also conducted  interviews with and requested information from HHS officials, asking them  to provide information on their efforts to coordinate their infectious  disease modeling activities. In this report, and in our past work, we define  coordination broadly as any joint activity that is intended to produce more  public value than could be produced when organizations act alone. We  compared these actions to relevant selected collaboration leading  practices: define and articulate a common outcome; establish mutually reinforcing or joint strategies;  identify and address needs by leveraging resources; agree on roles and responsibilities; establish compatible policies, procedures, and other means to operate  across agency boundaries; and develop mechanisms to monitor, evaluate, and report on results.

Because we judgmentally selected a group of experts and diseases, the  results of our review cannot be generalized to HHS coordination efforts  for other infectious diseases. However, our assessment of collaboration  and coordination activities did cover modeling efforts for the three  selected diseases.

Developing Infectious Disease Models and Assessing Their Performance

To identify steps that are generally considered when modelers develop  infectious disease models and assess their performance, we conducted  semi-structured interviews with relevant experts from academia and other  organizations and CDC and ASPR officials, and reviewed literature  identified by experts.

We used a snowball sampling approach to identify relevant experts and  groups. We initially identified five infectious disease modeling experts  through informal conversation with individuals working in the field,  infectious disease modeling experts known through GAO work, as well as  a review of websites, publications, and grants funded by NIH. Using a  snowball sampling approach, we reviewed key literature related to the  steps generally taken to develop models and assess their performance,  consulted with infectious disease modeling experts, and interviewed  agency officials to identify relevant groups, as well as individual experts,  who could convey to us the steps generally taken during infectious  disease modeling.

Through literature searches, the team identified literature from public  health journals or other major sources. The team applied personal  background and knowledge in public health, infectious disease modeling,  and statistics to help identify key sources. For the selected literature, we  reviewed references and used a snowball approach to identify further  relevant studies. Finally, we reviewed CDC guidance on decision making  for data access and long-term preservation as it related to documentation  standards.

Based on our review of identified literature, we developed a data  collection instrument to assess the extent to which CDC and ASPR used  the steps for infectious disease model development identified by experts  and in the literature. Through this data collection instrument, we  gathered information about the elements of developing and assessing  model performance and the steps that could be taken within each  element. In order to develop the data collection instrument, based on our  review of literature, we mapped out steps to develop and assess model  performance, and developed broad categories of assessment elements.  Within each assessment element, we included steps modelers could take  as a part of each assessment element. For example, the data collection  instrument included items that recorded model verification steps that  might have been taken by modeler(s) within the broader model  verification element. The instrument was reviewed by internal  stakeholders, who provided feedback on its content. Prior to sending the  data collection instrument to the agency, we filled in information on  verification steps taken for each of the 10 selected models, based on  provided model documentation to reflect steps we determined modelers  took as a part of the model development and assessment process. In  order to provide officials with this information, two analysts reviewed each  model’s documentation, with one analyst providing an initial coding of the  model and the other reviewing and verifying the first analyst’s findings.  This method was first tested on one of the 10 selected models by two  analysts independently coding information from the model’s  documentation into the data collection instrument and then reviewing  coding choices to reconcile any differences found. We then sent the  instruments with filled-in information to CDC and ASPR modelers to  receive their feedback concerning the steps taken to develop models and  assess their performance, provide any missing information, and resolve  any ambiguities. See Appendix III for a list of the 10 selected models  reviewed and steps to develop and assess model performance included  in the data collection instrument. The data collection instrument was  intended to record whether a specific step had been taken, but did not  assess the quality of the modeling steps.

In order to determine steps CDC and ASPR took to develop and assess  its models, we selected a non-generalizable sample of 10 models for  review in our data collection instrument that demonstrated steps that HHS  agencies took to develop models and assess their performance. The  model selection process described above informed our selection of  infectious disease models. To be selected for inclusion in our non- generalizable sample, the model had to be (1) developed by CDC, or  ASPR officials or contractors; (2) developed to answer a question about  Ebola, Zika, or pandemic influenza; and (3) used to inform public health  decision makers during an outbreak or for preparedness activities. We  selected 10 models that differed in form and answered different types of  questions, which included studies prepared during both outbreak  preparedness and response times, and covered topics such as the impact  of vaccination programs on deaths and hospitalization. For Ebola and  Zika, we focused on review of selected papers or memos produced since  2014 in order to capture the time period following the 2014-2016 Ebola  and 2015-2016 Zika outbreaks. For pandemic influenza, we focused on  papers and memos produced since 2009, when the H1N1 pandemic  occurred in the United States. Because we selected from a group of  models identified by HHS modelers and officials for Ebola, Zika, and  pandemic influenza, the results of our review cannot be generalized to  other diseases outside of the scope of this report. Furthermore, we  requested models that informed public health decision making, and did  not consider models that were not used for this purpose. Because we  reviewed a non-generalizable sample of 10 models, the results of our  review cannot be generalized to a larger population of models prepared  by HHS agencies.

To identify challenges associated with modeling for the selected  infectious diseases, we reviewed documents and reports to identify  modeling challenges and steps to address those challenges, and  interviewed agency officials and modelers, and experts identified through  the previously-described snowball sampling methodology. We used semi- structured interview protocols that included open-ended questions about  challenges associated with infectious disease modeling and limitations  associated with model development. Not all officials and experts we  interviewed provided comments on every challenge or limitation. In  addition, because we judgmentally selected a group of experts and  diseases, the results of our review cannot be generalized to all infectious  disease modeling efforts.

We conducted this performance audit from May 2018 to May 2020, in  accordance with generally accepted government auditing standards.  These standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe the  evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Bibliography of Selected Model Publications Reviewed

Carias, Cristina, et al. “Preventive Malaria Treatment for Contacts of  Patients with Ebola Virus Disease in the Context of the West Africa 2014- 15 Ebola Virus Disease Response: An Economic Analysis.” The Lancet  Infectious Diseases, vol. 16, no. 4 (April 2016): pp. 449-458.

Christie, Athalia, et al. “Possible Sexual Transmission of Ebola Virus— Liberia, 2015.” Morbidity and Mortality Weekly Report, vol. 64, no. 17  (May 8, 2015): pp. 479-481.

Martin I. Meltzer, et al. “Estimating the Future Number of Cases in the  Ebola Epidemic - Liberia and Sierra Leone, 2014-2015.” Morbidity and  Mortality Weekly Report, vol. 63, no. 3 suppl. (September 26, 2014): pp.  1-14.

Meltzer, Martin I., et al. “Modeling in Real Time during the Ebola  Response.” Morbidity and Mortality Weekly Report, vol. 65, no. 3 suppl.  (July 8, 2016): pp. 85-89.

Rainisch, Gabriel, et al. “Estimating Ebola Treatment Needs, United  States.” Emerging Infectious Diseases, vol. 21, no. 7 (July 2015): pp.  1273-1275.

Rainisch, Gabriel, et al. “Regional Spread of Ebola Virus, West Africa,  2014.” Emerging Infectious Diseases, vol. 21, no. 3 (March 2015): pp.  444-447.

Undurraga, Eduardo A., Cristina Carias, Martin I. Meltzer, Emily B. Kahn.  “Potential for Broad-Scale Transmission of Ebola Virus Disease during  the West Africa Crisis: Lessons for the Global Health Security Agenda.”  Infectious Diseases of Poverty, vol. 6, no. 159 (2017).

Washington, Michael L., Martin I. Meltzer. “Effectiveness of Ebola  Treatment Units and Community Care Centers Liberia, September 23- October 31, 2014.” Morbidity and Mortality Weekly Report, vol. 64, no. 3  (January 30, 2015): pp. 67-69.

Adamski, Alys, et al. “Estimating the Numbers of Pregnant Women  Infected with Zika Virus and Infants with Congenital Microcephaly in  Colombia, 2015–2017.” Journal of Infection, vol. 76 (2018): pp. 529-535.

Dirlikov, Emilio, et al. “Guillain-Barré Syndrome and Healthcare Needs  during Zika Virus Transmission, Puerto Rico, 2016.” Emerging Infectious  Diseases, vol. 23, no. 1 (January 2017): pp.134-136.

Ellington, Sascha R., et al. “Estimating the Number of Pregnant Women  Infected With Zika Virus and Expected Infants With Microcephaly  Following the Zika Virus Outbreak in Puerto Rico, 2016.” JAMA  Pediatrics, vol. 170, no. 10 (2016): pp. 940-945.

Grills, Ardath, et al. “Projected Zika Virus Importation and Subsequent  Ongoing Transmission after Travel to the 2016 Olympic and Paralympic  Games—Country-Specific Assessment, July 2016.” Morbidity and  Mortality Weekly Report, vol. 65, no. 28 (July 22, 2016): pp.711-715.

Johansson, Michael A., et al. “Zika and the Risk of Microcephaly.” The  New England Journal of Medicine, vol. 375 (July 7, 2016): pp.1-4.

Johnson, Tammi L., et al. “Modeling the Environmental Suitability for  Aedes (Stegomyia) aegypti and Aedes (Stegomyia) albopictus (Diptera:  Culicidae) in the Contiguous United States.” Journal of Medical  Entomology, vol. 54, no. 6 (November 7, 2017): pp. 1605-1614.

Mitchell, Patrick K. et al., “Reassessing Serosurvey-Based Estimates of  the Symptomatic Proportion of Zika Virus Infections.” American Journal of  Epidemiology, vol. 188, no. 1 (January 2019): pp. 206-213.

Mier-y-Teran-Romero, Luis, Mark J. Delorey, James J. Sejvar, Michael A.  Johansson. “Guillain-Barré Syndrome Risk Among Individuals Infected  with Zika Virus: a Multi-Country Assessment.” BMC Medicine, vol. 16, no.  67 (2018).

Mier-y-Teran-Romero, Luis, Andrew J. Tatem, Michael A. Johansson.  “Mosquitoes on a Plane: Disinsection Will Not Stop the Spread of Vector- Borne Pathogens, a Simulation Study.” PLoS Neglected Tropical  Diseases, vol. 11, no. 7 (July 3, 2017).

Reefhuis, Jennita, et al. “Projecting Month of Birth for At-Risk Infants after  Zika Virus Disease Outbreaks.” Emerging Infectious Diseases, vol. 22,  no. 5 (May 2016): pp. 828-832.

Russell, Steven, et al. “Detecting Local Zika Virus Transmission in the  Continental United States: A Comparison of Surveillance Strategies.”  PLoS Currents Outbreaks (November 22, 2017).

Watts. Alexander G., et al. “Elevation as a Proxy for Mosquito-Borne Zika  Virus Transmission in the Americas.” PLoS ONE, vol. 12, no. 5 (May 24,  2017).

Atkins, Charisma Y., et al. “Estimating Effect of Antiviral Drug Use during  Pandemic (H1N1) 2009 Outbreak, United States.” Emerging Infectious  Diseases, vol. 17. no. 9 (September 2011): pp. 1591-1598.

Biggerstaff, Matthew, et al. “Estimates of the Number of Human Infections  With Influenza A(H3N2) Variant Virus, United States, August 2011–April  2012.” Clinical Infectious Diseases, vol. 57, suppl. 1 (2013): pp. S12-S15.

Biggerstaff, Matthew, et al. “Estimating the Potential Effects of a Vaccine  Program Against an Emerging Influenza Pandemic—United States.”  Clinical Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S20-S29.

Carias, Cristina, et al. “Potential Demand for Respirators and Surgical  Masks during a Hypothetical Influenza Pandemic in the United States.”  Clinical Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S42-S51.

Cauchemez, Simon, et al. “Role of Social Networks in Shaping Disease  Transmission during a Community Outbreak of 2009 H1N1 Pandemic  Influenza.” Proceedings of the National Academy of Sciences of the  United States, vol. 108, no. 7 (February 15, 2011): pp. 2825-2830.

Dawood, Fatimah S., et al. “Estimated Global Mortality Associated with  the First 12 Months of 2009 Pandemic Influenza A H1N1 Virus  Circulation: a Modelling Study.” The Lancet Infectious Diseases, vol. 12  (September 2012): pp. 687-695.

Fung, Isaac Chun-Hai, et al. “Modeling the Effect of School Closures in a  Pandemic Scenario: Exploring Two Different Contact Matrices.” Clinical  Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S58-S63.

Iuliano, A. Danielle, et al. “Estimates of Global Seasonal Influenza- Associated Respiratory Mortality: a Modelling Study.” The Lancet, vol.  391, no. 10127 (March 31, 2018): pp. 1285-1300.

Jain, Seema, et al. “Hospitalized Patients with 2009 H1N1 Influenza in the  United States, April–June 2009.” The New England Journal of Medicine,  vol. 361, no. 20 (November 12, 2009): pp. 1935-1944.

Kostova, Deliana, et al. “Influenza Illness and Hospitalizations Averted by  Influenza Vaccination in the United States, 2005–2011.” PLoS ONE, vol.  8, no. 6 (June 19, 2013).

Lafond, Kathryn E., et al. “Global Role and Burden of Influenza in  Pediatric Respiratory Hospitalizations, 1982–2012: A Systematic  Analysis.” PLoS Medicine, vol. 13, no. 3 (March 24, 2016).

Meltzer, Martin I., Nancy J. Cox, Keiji Fukuda. “The Economic Impact of  Pandemic Influenza in the United States: Priorities for Intervention.”  Emerging Infectious Diseases, vol. 5, no. 5 (September-October 1999):  pp. 659-671.

Meltzer, Martin I., et al. “Estimates of the Demand for Mechanical  Ventilation in the United States during an Influenza Pandemic.” Clinical  Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S52-S57.

O’Hagan, Justin J., et al. “Estimating the United States Demand for  Influenza Antivirals and the Effect on Severe Influenza Disease during a  Potential Pandemic.” Clinical Infectious Diseases, vol. 60, suppl. 1 (2015):  pp. S30-S41.

Presanis, Anne M., et al. “The Severity of Pandemic H1N1 Influenza in  the United States, from April to July 2009: A Bayesian Analysis.” PLoS  Medicine, vol. 6, no. 12 (December 8, 2009).

Reed, Carrie, et al. “Estimates of the Prevalence of Pandemic (H1N1)  2009, United States, April-July 2009.” Emerging Infectious Diseases, vol.  15, no. 12 (December 2009): pp. 2004-2007.

Reed, Carrie, Martin I. Meltzer, Lyn Finelli, Anthony Fiore. “Public Health  Impact of Including Two Lineages of Influenza B in a Quadrivalent  Seasonal Influenza Vaccine.” Vaccine, vol. 30 (2012): pp. 1993-1998.

Reed, Carrie, et al. “Estimating Influenza Disease Burden from  Population-Based Surveillance Data in the United States.” PLoS ONE,  vol. 10, no. 3 (March 4, 2015).

Rolfes, Melissa A., et al. “Annual Estimates of the Burden of Seasonal  Influenza in the United States: A Tool for Strengthening Influenza  Surveillance and Preparedness.” Influenza and Other Respiratory  Viruses, vol. 12 (2018): pp. 132-137.

Russell, K., et al. “Utility of State-Level Influenza Disease Burden and  Severity Estimates to Investigate an Apparent Increase in Reported  Severe Cases of Influenza A(H1N1) pdm09 – Arizona, 2015–2016.”  Epidemiology and Infection, vol. 146 (June 14, 2018): pp. 1359-1365.

Shrestha, Sundar S., et al. “Estimating the Burden of 2009 Pandemic  Influenza A (H1N1) in the United States (April 2009–April 2010).” Clinical  Infectious Diseases, vol. 52, suppl. 1 (2011): pp. S75-S82.

Tokars, Jerome I., Melissa A. Rolfes, Ivo M. Foppa, Carrie Reed. “An  Evaluation and Update of Methods for Estimating the Number of Influenza  Cases Averted by Vaccination in the United States.” Vaccine, vol. 36  (2018): pp. 7331-7337.

Appendix III: Ten Selected Infectious Disease Models and Questions from Data Collection Instrument

Appendix III: Ten Selected Infectious Disease  Models and Questions from Data Collection  Instrument Document describing model   Meltzer, Martin I., Charisma Y. Atkins, Scott Santibanez, Barbara Knust, Brett W. Petersen,  Elizabeth D. Ervin, Stuart T. Nichol, Inger K. Damon, Michael L. Washington. Estimating the  Future Number of Cases in the Ebola Epidemic–Liberia and Sierra Leone, 2014-2015, MMWR.  Volume 63, Number 3, September 26, 2014.  Rainisch, Gabriel, Manjunath Shankar, Michael Wellman, Toby Merlin, and Martin I. Meltzer.  Regional Spread of Ebola Virus, West Africa, 2014. Emerging Infectious Diseases. Volume 21,  Number 3, March 2015.  Asher, Jason. Forecasting Ebola with a Regression Transmission Model. Epidemics. Volume  22, 2018.

Ellington, Sascha R., Owen Devine, Jeanne Bertolli, Alma Martinez Quiñones, Carrie K.  Shapiro-Mendoza, Janice Perez-Padilla, Brenda Rivera-Garcia, Regina M. Simeone, Denise J.  Jamieson, Miguel Valencia-Prado, Suzanne M. Gilboa, Margaret A. Honein, Michael A.  Johansson. Estimating the Number of Pregnant Women Infected With Zika Virus and Expected  Infants With Microcephaly Following the Zika Virus Outbreak in Puerto Rico, 2016. JAMA  Pediatrics. Volume 170, Number 10, October 2016.

Johansson, Michael A., Luis Mier-y‐Teran-Romero, Jennita Reefhuis, Suzanne M. Gilboa, and  Susan L. Hills. Zika and the Risk of Microcephaly. New England Journal of Medicine. Volume  375, Number 1, July 7, 2016.  Dirlikov, Emilio, Krista Kniss, Chelsea Major, Dana Thomas, Cesar A. Virgen, Marrielle  Mayshack, Jason Asher, Luis Mier-y-Teran-Romero, Jorge L. Salinas, Daniel M. Pastula, Tyler  M. Sharp, James Sejvar, Michael A. Johansson, Brenda Rivera-Garcia. Guillain-Barré  Syndrome and Healthcare Needs during Zika Virus Transmission, Puerto Rico, 2016. Emerging  Infectious Diseases. Volume 23, Number 1, January 2017.  Biggerstaff, Matthew, Carrie Reed, David L. Swerdlow, Manoj Gambhir, Samuel Graitcer, Lyn  Finelli, Rebekah H. Borse, Sonja A. Rasmussen, Martin I. Meltzer, Carolyn B. Bridges.  Estimating the Potential Effects of a Vaccine Program against an Emerging Influenza  Pandemic—United States, Clinical Infectious Diseases. Volume 60, Issue Supplement 1, 2015.  Carias, Cristina, Gabriel Rainisch, Manjunath Shankar, Bishwa B. Adhikari, David L. Swerdlow,  William A. Bower, Satish K. Pillai, Martin I. Meltzer, Lisa M. Koonin. Potential Demand for  Respirators and Surgical Masks during a Hypothetical Influenza Pandemic in the United States.  Clinical Infectious Disease. Volume 60, Issue Supplement 1, 2015.  Reed, Carrie, Frederick J. Angulo, David L. Swerdlow, Marc Lipsitch, Martin I. Meltzer, Daniel  Jernigan, and Lyn Finelli. Estimates of the Prevalence of Pandemic (H1N1) 2009, United States,  April–July 2009, Emerging Infectious Diseases. Volume 15, Number 12, December 2009.  Asher, Jason, Matthew Clay. Deterministic compartmental models for influenza with mitigations.  R: “flumodels” package. Version: 1.0.7, April 24, 2017.

GAO Review of Model Assessment Steps for Selected Agency Models

Purpose: The Government Accountability Office has been asked by the  Congress to review the Department of Health and Human Services’  agency efforts to model infectious disease. As part of our methodology,  we selected and reviewed published papers and internal memoranda  from the sources provided to us. We reviewed these sources to describe  the steps taken to describe, verify, validate, and communicate results of  these modeling efforts. The purpose of this inquiry is to provide the  authors of the selected papers the opportunity to confirm, clarify, or  provide additional information in the table below.

Instructions: In the table below, we have two sets of columns: one set  indicating GAO’s assessment of whether the document contained  information about a step being taken. The second set of columns is for  the authors of the selected paper to fill out. If you agree with information  in the GAO columns, please indicate your concurrence in the Reviewer  Comments column. Otherwise, please provide information accordingly.

If a step is marked “Step taken” please review the entries we have made  in the GAO Reviewer Comments column for accuracy and completeness  and indicate your concurrence in the Reviewer Comments column.  Please also provide additional supporting documentation if available. For  any steps that were taken, but where we indicated either “not taken” or  “not enough information to determine” in our review, please provide a  description of the actual steps and any documentation you may have. If a  step was not taken, please provide an indication as to why that step was  not taken and, if possible, please provide supporting documentation. For  example, if limited data availability impacted the ability to conduct a model  validation step(s), then please include this information in the appropriate  table cells.

Model Verification (Internal Validation, Internal Consistency, Technical Validity)

10  Independent expert (internal or external) review of key programming  11  Debugging tests and checks for coding accuracy  12  Model’s code or Excel spreadsheet is available  13  Test model assumptions (i.e. confirming model assumptions are  reasonable and appropriate for question), for example:

Distributional assumptions about model residuals

Form of the model  14  Model handling of input data/parameters is verified as correct (i.e. as  intended by developers)

16  Sensitivity analysis (assessing impact of assumption/parameter  uncertainty on output or model form)  17  Cross validation or between model comparisons: Compare results to  other models that address the same problem  18  External validation: Compare model results to actual event data  19  Predictive validation: Compare model predictions for future events to  actual outcomes.

21  Modelers supply customer with agreed upon information, which may  vary depending on the model  22  Modeler provides customer with clear information on uncertainty in  model results, such as inclusion of standard errors or confidence  intervals, or qualitative explanations of uncertainty in the model results  Assessment Steps Question: Do you think that the assessment  elements identified in the table above sufficiently reflect the steps that  should generally be taken to develop and assess the performance of  models? Would you remove any steps, add any steps, or make any other  adjustments to these steps in order to consider them best practices in  assessing performance of models, generally? Please explain.

Appendix IV: Comments from the Department of Health and Human Services

Appendix V: GAO Contacts and Staff Acknowledgments

In addition to the contact named above, the following individuals made  contributions to this report: Sushil Sharma (Assistant Director), Charlotte  E. Hinkle (Analyst-in-Charge), Sam Amrhein, Breanne Cave, Jehan  Chase, Carol A. Gotway Crawford, Justin Cubilo, Karen Doran, Nancy  Fasciano, Douglas G. Hunker, Dennis Mayo, Anika McMillon, Sarah  Resavy, Edward Rice, Ben Shouse, Amber Sinclair, Walter Vance, Sarah  Veale, and Richard Zarrella.

Outbreaks of infectious diseases—such as novel coronavirus and pandemic flu—have raised concerns about how federal agencies use modeling to predict a disease’s course. Models can help decision makers set disease control policies and allocate resources. If models are unsound, they may not produce the reliable predictions needed to make good decisions.

We examined how Health and Human Services, which includes the Centers for Disease Control and Prevention, uses and assesses models. We recommended that HHS improve coordination of modeling across agencies and ensure models are reproducible, which helps build confidence in their results.