Aviation Security: TSA Coordinates with Stakeholders on Changes to Screening Rules but Could Clarify Its Review Processes and Better Measure Effectiveness

On December 25, 2009, while on a flight from Amsterdam to Detroit, a person attempted to detonate explosives hidden in their underwear. This person was not included in the government's consolidated database of known or suspected terrorists at the time. In response, in 2010, TSA began identifying passengers who are not known or suspected terrorists, but who TSA determined should receive enhanced screening. Specifically, TSA identifies passengers for enhanced screening through the application of screening rules, which TSA develops by considering current intelligence and other factors. TSA refers to these rules and lists as Silent Partner and Quiet Skies.

The TSA Modernization Act includes a provision for GAO to review the current oversight mechanisms and effectiveness of Silent Partner and Quiet Skies. This report examines the extent to which TSA has (1) coordinated with relevant DHS and TSA stakeholders to review passenger screening rules; and (2) assessed the effectiveness of these rules. GAO analyzed TSA documents, including standard operating procedures, and interviewed senior DHS and TSA officials involved in managing and overseeing the programs.

The Transportation Security Administration (TSA) coordinates reviews of its intelligence-based screening rules known as Silent Partner and Quiet Skies. Specifically, TSA's Intelligence and Analysis office (I&A) coordinates quarterly rule reviews and notifies Department of Homeland Security (DHS) and TSA stakeholders of rule changes. According to stakeholders, these review processes provide a good mechanism for program oversight. TSA has established guidance for rule changes that involve TSA stakeholders reviewing rules in advance of their implementation. In some instances, TSA uses an alternate process, allowed by guidance in exigent circumstances, where rule changes go into effect before some stakeholders review them. However, agency guidance does not define the conditions for using the standard or exigent processes. Further, TSA officials do not document which review process—standard or exigent—they use for each rule change. Clarifying guidance and documenting which review process is used could improve transparency and better ensure screening rule changes are adequately reviewed.

TSA tracks some data on rule implementation, but has not identified a means to comprehensively measure rule effectiveness. TSA officials explained that they had not yet fully assessed the rules' effectiveness because it was difficult to measure. Silent Partner rules identify passengers for enhanced screening on inbound flights to the United States. Quiet Skies rules—a subset of the Silent Partner rules—identify passengers for enhanced screening on subsequent domestic and outbound flights. TSA officials said that the one method they had used to assess effectiveness was to count Quiet Skies passengers who were later added to the government's watchlist of known or suspected terrorists. However, because this analysis was limited to Quiet Skies, it excluded 93 percent of the screening rules, making it difficult to interpret what the results indicate about effectiveness. TSA has access to data, such as the outcomes of enhanced screening of Silent Partner and Quiet Skies passengers, that could be explored to better assess rule effectiveness. Exploring additional data sources could help TSA refine and supplement their existing efforts to measure program effectiveness.

GAO is making three recommendations. DHS should (1) clarify the criteria for exigent and standard rule review procedures; (2) document which review process is used for each new rule or rule change; and (3) explore additional data sources for measuring rule effectiveness. DHS concurred with these recommendations.

On December 25, 2009, while on a flight from Amsterdam to Detroit, a  person attempted to detonate explosives hidden in their underwear. This  individual was not at the time in the Terrorist Screening Database—in  general, the government’s consolidated watchlist of known or suspected  terrorists. Following the attempted attack, the Transportation Security  Administration (TSA) sought new ways to identify and designate  passengers for enhanced screening like the person responsible for the  2009 attempted attack. Enhanced screening generally includes a pat- down and an explosives trace detection or physical search of the interior  of the passenger’s accessible property, electronics, and footwear at the  airport security checkpoint.

In 2010, TSA began using risk-based factors to create screening rules to  identify potentially higher-risk passengers and designate them for  enhanced screening. Based on threat intelligence, TSA may create a rule  that, for example, targets passengers in a specific age range, traveling on  a flight originating from a particular country. TSA implements these rules  through two programs—the Silent Partner program and the Quiet Skies  program.

The Silent Partner rules identify passengers for enhanced screening on  inbound flights to the United States. The Quiet Skies rules are a subset of  the Silent Partner rules and identify passengers for enhanced screening  on subsequent domestic and outbound flights for a limited period of time  or number of flights. Passengers who match the rules are, in general,  included on a Silent Partner or Quiet Skies List. TSA’s passenger  prescreening program—Secure Flight—uses the lists to designate the  passengers for enhanced screening prior to boarding an aircraft.  According to senior officials at TSA’s Federal Air Marshal Service, in  March 2018 the agency began prioritizing the deployment of air marshals  on flights with Quiet Skies List matches to observe the individuals and  provide an on-board security presence.

The TSA Modernization Act, enacted in October 2018, imposed  requirements related to the oversight of the Silent Partner and Quiet  Skies programs. Specifically, the Act provides that TSA’s Intelligence  and Analysis (I&A) office is to identify and review TSA’s passenger  screening rules, in coordination with other TSA and Department of  Homeland Security (DHS) stakeholders, within 60 days of enactment and  every 120 days thereafter. It also requires TSA I&A to notify these  stakeholders within two days of implementing a new rule or making  changes to an existing rule.

The Act includes a provision for GAO to study whether the rules are  effective in mitigating potential threats to aviation security; and whether,  and if so how, TSA coordinates with DHS regarding any proposed change  to a rule. This report examines the extent to which TSA has (1)  coordinated with relevant DHS entities to review passenger screening  rules and (2) assessed the effectiveness of its passenger screening rules.

To examine the extent to which TSA coordinated with relevant DHS and  TSA entities to review passenger screening rules, we reviewed  documentation of steps TSA has taken to coordinate reviews of the rules  since October 2018. Such documentation included memos signed by the  TSA Assistant Administrator of Intelligence and Analysis confirming  quarterly reviews conducted in fiscal year 2019, meeting minutes for  quarterly rule review meetings in fiscal year 2019, TSA I&A notifications  of rule changes for the period October 2018 through May 2019, and a list  of Quiet Skies and Silent Partner rules as of August 2019. We also  reviewed program standard operating procedures (SOP). To understand  the rule review and approval process, we interviewed TSA I&A officials  responsible for reviewing and approving Silent Partner and Quiet Skies  rules and DHS and TSA officials in each office with a specific role in  overseeing the program: TSA’s Civil Rights and Liberties, Ombudsman,  and Traveler Engagement; DHS’s Office for Civil Rights and Civil  Liberties; TSA’s Chief Counsel; DHS’s Office of the General Counsel;  TSA’s Privacy Office; DHS’s Privacy Office; the Federal Air Marshal  Service, and DHS’s Traveler Redress Inquiry Program. We compared  documentary and testimonial evidence of the current coordination efforts  between TSA I&A and stakeholders to the requirements in the TSA  Modernization Act; DHS and TSA SOPs; the Quiet Skies Implementation  Plan and action memo; a Standards for Internal Control in the Federal  Government principle related to implementing control activities; and  characteristics of homeland security risk management described in DHS’s  Risk Management Framework.

To examine the extent to which TSA has assessed the effectiveness of its  passenger screening rules we analyzed documentation of TSA’s  assessments of the Silent Partner and Quiet Skies rules including TSA  I&A’s quarterly operational statistics for the first quarter of fiscal year  2019. We interviewed TSA I&A officials who manage the Silent Partner  and Quiet Skies programs about any efforts to assess rule effectiveness.  We also interviewed DHS and TSA stakeholders mentioned above to  obtain their views on the effectiveness of the rules. In addition, we  interviewed U.S. Customs and Border Protection (CBP) officials about  how that agency has assessed the effectiveness of its rules-based  program. Lastly, we compared TSA I&A’s efforts to TSA’s Quiet Skies  Implementation Plan and performance management practices identified in  our prior work and OMB guidance.

We conducted this performance audit from April 2019 to November 2019  in accordance with generally accepted government auditing standards.

Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

TSA began implementing its Secure Flight program in 2009 to identify  passengers who may pose security risks before boarding an aircraft. The  program requires U.S. and foreign commercial aircraft operators traveling  to, from, within or overflying the United States, as well as U.S.  commercial aircraft operators with international point-to-point flights, to  collect information from passengers and transmit it electronically to TSA.  This information includes personally identifiable information, such as full  name, gender, date of birth, passport information (if available), and  certain non-personally identifiable information, such as itinerary  information and the unique number associated with a travel record  (record number locator).

The Secure Flight program matches the passenger-provided personally  identifiable information against federal government watchlists and other  information to determine if passengers may pose a security risk and to  assign them a risk category. Since January 2009, Secure Flight has  matched passengers to two subsets of the Terrorist Screening  Database—the No Fly List, composed of individuals who should be  precluded from boarding an aircraft or entering the sterile area of a U.S.  airport, and the Selectee List, composed of individuals who should  receive enhanced screening prior to boarding an aircraft or entering an  airport sterile area. The risk categories are not specifically communicated to the air carriers, but for each passenger Secure Flight  provides responses to air carriers commensurate with the risk levels  identified (e.g., an air carrier will receive a response of “inhibited” if the  passenger was identified as being in the highest-risk category, or the  boarding pass printed for a high-risk passenger will identify that  passenger as a selectee for enhanced screening at the security  checkpoint).

In April 2011, in response to the December 25, 2009 attempted attack,  TSA also began matching passengers to a third subset of the Terrorist  Screening Database—the Expanded Selectee List—to designate known  or suspected terrorists not otherwise included on the No Fly or Selectee  Lists as selectees for enhanced screening. The Expanded Selectee List,  in general, includes all records in the Terrorist Screening Database with a  full name (first name and surname) and full date of birth not otherwise  included on the No Fly or Selectee Lists. The Secure Flight system, which  also screens passengers against the Silent Partner and Quiet Skies Lists,  among others, results in passengers receiving one of four prescreening  outcomes:

Low risk (expedited screening). Passengers who are eligible for  expedited screening, such as those with TSA Pre®,

Unknown Risk (standard screening). Passengers who warrant  standard screening,

High Risk (enhanced screening). Passengers who receive enhanced  screening such as a pat down and explosives trace detection,  because they have been identified as matches to government  watchlists, including the Selectee, Expanded Selectee, Silent Partner  and Quiet Skies Lists, or

Highest Risk (denied boarding). Passengers who are not permitted to  board a commercial aircraft, such as passengers who are on the No  Fly List or the Centers for Disease Control and Prevention Do Not  Board List (see fig. 1).

Secure Flight also randomly identifies passengers for enhanced  screening. Although subject to the same screening measures as high risk  passengers, they have not been determined to be high risk. Similarly,  individuals included on the Silent Partner and Quiet Skies Lists have not  been determined to be of high risk, but rather have been identified using  rules based on current intelligence and other factors that may indicate an  elevated risk.

TSA leverages CBP information and targeting capabilities to create the  Silent Partner List. Specifically, TSA leverages (1) data CBP collects  regarding passengers traveling internationally (such as citizenship,  passport country of issuance, and address information), and (2) CBP’s  Automated Targeting System. CBP uses the Automated Targeting  System to identify potentially high risk passengers arriving or departing  the United States by comparing passenger information with law  enforcement, intelligence, and other enforcement data using risk-based  targeting scenarios and assessments.

Analysts within TSA I&A’s Threat Analysis Division review intelligence to  identify factors that may indicate elevated passenger risk. TSA works with  CBP to create Silent Partner and Quiet Skies rules in the Automated  Targeting System based on these factors. The system returns information  on passengers who match with the rules and are scheduled to fly on U.S.- bound flights. TSA then omits any individuals on the Silent Partner  cleared list (i.e. travelers exempted from further enhanced screening  based on a specific rule) before placing the remaining passengers on the  Silent Partner List. The Secure Flight program designates passengers  who are on the Silent Partner List as selectees for enhanced screening  for a particular international flight.

In April 2012, TSA’s Quiet Skies List became fully operational. The Quiet  Skies List is a subset of passengers on the Silent Partner List.  Specifically, TSA identifies certain Silent Partner rules that warrant  continued enhanced screening for passengers’ subsequent domestic or  outbound travel after arriving in the United States. Passengers identified  via these rules—the Quiet Skies rules—comprise the Quiet Skies List.  Passengers matched to the Quiet Skies List are designated as selectees  and receive enhanced screening on any subsequent domestic flights for a  designated period of time, or for a designated number of flights,  whichever comes first. After the designated time period has elapsed (or  number of flights is flown), passengers’ names and identifying information  are moved to a cleared list.

Pursuant to the TSA Modernization Act, TSA I&A is to identify and review  its Silent Partner and Quiet Skies screening rules, in coordination with  DHS and TSA stakeholders, every 120 days and provide notification to  these stakeholders no later than two days after making a change to a  rule. Table 1 lists the DHS and TSA stakeholders TSA I&A must  coordinate with under the Act.

TSA Coordinates with Stakeholders as Required, but TSA Guidance Is Not Clear About Criteria for Review of Rule Changes TSA I&A Coordinates Quarterly Rule Reviews and Notifies Oversight Offices of Rule Changes

According to DHS and TSA officials, TSA has coordinated quarterly rule  review meetings with DHS and TSA stakeholders since the inception of  the Silent Partner and Quiet Skies programs. We reviewed  documentation of the reviews that occurred from December 2018 through  March 2019. The quarterly review meetings are called for in DHS’s  Automated Rule Review SOP and its Quiet Skies Implementation Plan.

Pursuant to the TSA Modernization Act, TSA I&A is to identify and review  its screening rules in coordination with DHS and TSA stakeholders every  120 days—or at least three times a year. TSA I&A officials stated that  they plan to continue convening four times a year because, given the  difficulty of scheduling these large meetings, it will help them ensure they  meet the 120 day requirement. Since October 2018, TSA I&A has also  included representatives of DHS’s Traveler Redress Inquiry Program and  the Federal Air Marshal Service in these quarterly review meetings, as  required by the Act. Officials from these offices told us in August 2019  that they are still determining their role in the rule review process, but  expect the coordination to be beneficial.

DHS and TSA SOPs set forth the process for the quarterly review  meetings. TSA I&A and stakeholder officials stated that the process  generally happens as described in the SOP. Two weeks prior to the  meeting, TSA I&A sends out materials including a list of new rules, rule  changes, archived (discontinued) rules, and the rationale and links to the  underlying intelligence supporting each rule change. According to TSA  officials, TSA and DHS stakeholders review the rules from their particular  areas of expertise. For example, TSA Chief Counsel officials reported that  they review rules and the supporting intelligence to ensure that the rules  meet legal sufficiency standards. A TSA Privacy official stated that they  review rules and the supporting intelligence to ensure rules do not violate  passengers’ rights. All stakeholders review the rules to ensure they are  based on current intelligence that identify specific threats. If a  stakeholder finds that there is insufficient current intelligence to support  the rule, TSA I&A officials stated that they would modify it to ensure it is  tailored to current intelligence or archive a rule when the intelligence- based threat is no longer relevant. For example, during the March 2019  quarterly review meeting TSA I&A officials discussed archiving a Silent  Partner rule due to insufficient current intelligence to support it. According  to TSA I&A officials, the rule was archived in April 2019.

TSA I&A officials and stakeholders generally agreed that the quarterly  reviews provide a good mechanism for oversight of both programs.  Stakeholders told us these meetings provide a forum to discuss the scope  of the rules and whether or not they were supported by current  intelligence or if they are sufficiently specific. For example, a TSA  stakeholder questioned the basis for a rule that identified a particular  travel pattern as a high risk factor. As a result, TSA I&A officials reviewed  the intelligence and revised the rule.

TSA I&A officials stated that since enactment of the TSA Modernization  Act in October 2018, they have also notified DHS and TSA stakeholders  within two days of making changes to a rule. We reviewed the eight  notifications that TSA I&A sent to stakeholders regarding rule changes  during the period from October 2018 through May 2019. These  notifications detailed changes to rules, new rules, and rules that were  archived. DHS and TSA stakeholders we spoke with said that the two day  notifications are helpful in keeping them informed in between quarterly  meetings. In addition, stakeholders said it allowed them to proactively  reach out to TSA I&A to ask questions and share more timely feedback  about rule changes. TSA I&A has implemented the two day notifications  and other steps required in the TSA Modernization Act, but TSA I&A’s  Standard Operating Procedures have not yet been updated to reflect  these changes. TSA I&A officials stated that they have plans to do so in  fall 2019.

TSA Has a Standard and Expedited Rule Review Process, but TSA Guidance Is Unclear about Criteria for Each Process

TSA I&A’s standard operating procedures establish two situation- dependent processes for reviewing and approving rule changes, as  shown in figure 2. First, under standard circumstances, TSA I&A’s  standard operating procedures detail a four-part vetting process by which  TSA I&A drafts support for the rule change and it is subsequently  approved by TSA Chief Counsel, the TSA I&A Assistant Administrator,  and ultimately TSA senior leadership. TSA procedures specify that in  standard circumstances, all rule changes are to be supported and  approved in writing prior to implementation. Specifically, TSA I&A is to  draft a memo with the nature of the threat and how all components of the  rule address the concerns from intelligence reporting. The memo, along  with all pertinent intelligence sources, is then required to be routed  through TSA Chief Counsel and TSA leadership for intelligence, legal,  and policy review. TSA’s April 2012 Quiet Skies Implementation Plan specified that the Chief Counsel’s review is to ensure that the proposed  rule targets the threat presented in the assessment, the assessment  properly documents the reasons for the recommendation, and the  recommendation is in compliance with relevant legal authorities,  regulations, and DHS policies. Upon approval, the memo is referred to  TSA senior leadership—the TSA Administrator or TSA Deputy  Administrator—for final written approval. Following this, the rule change  can be implemented.

A second process, called exigent, is also briefly described in the SOPs. In  exigent circumstances—circumstances requiring immediate action—the  TSA I&A Assistant Administrator or his or her designee may direct that  the rule be implemented immediately without a signed decision memo.  The signed memo is still required, but can be drafted, reviewed, and  approved after the change is implemented. TSA I&A officials stated that  the exigent process entails verbal direction to implement a rule.

It is unclear if TSA I&A has followed the exigent rule review process in  standard circumstances because the SOP is unclear on the criteria for  each process. TSA’s SOP states that the exigent review process may be  used “if TSA determines that exigent circumstances require immediate  implementation of a Silent Partner rule.” However, the SOP does not  clarify who or which office within TSA makes this determination or what  types of circumstances would be appropriately characterized as exigent.

TSA I&A officials told us that exigent circumstances were very rare. They  estimated that in the last 3 years exigent circumstances had occurred  once. Yet, the same officials also estimated that they implemented  approximately 90 percent of the rule changes following verbal approval  from either TSA or I&A leadership and drafted the required memos after  the fact. This indicates that TSA I&A officials have not followed the  standard review process when implementing rule changes in  circumstances they regard as standard, and the process followed  appears to be closer to what would occur in exigent circumstances.  These TSA I&A officials explained that drafting and processing the  approval memo after they implement a rule change allows them to more  quickly respond to changing intelligence. TSA’s SOP provides flexibility  for this in exigent circumstances. However, given the absence of clarity in  the SOP about when the exigent process is to be used and who is to  make that decision, it is unclear whether or not TSA I&A used the exigent  review process—a process which is not, initially, contingent upon TSA’s  legal review or I&A’s written support—in circumstances that DHS and  TSA leadership who oversee the program would regard as standard.

According to Standards for Internal Control in the Federal Government,  management should implement control activities through policies by, for  example, documenting responsibilities in policies and periodically  reviewing policies and procedures for continued relevance and  effectiveness. As TSA I&A updates its Silent Partner and Quiet Skies  SOPs in fall 2019, clarifying the criteria for standard and exigent rule  review procedures would provide greater assurance that screening rule  changes are reviewed as intended.

TSA I&A officials further told us that they do not document or otherwise  have a way of determining what proportion of rule changes have been  reviewed in accordance with the standard process versus the exigent  process because they had not identified a need to do so. According to the  2012 TSA memo establishing Quiet Skies as a permanent program, at  the program’s outset a working group of DHS and TSA stakeholders  identified the need for transparency as the first of seven key areas of  consensus. Further, DHS’s Integrated Risk Management Framework  establishes transparency and documentation as important characteristics  of homeland security risk management. Documenting which review  process TSA I&A uses for each rule change could improve transparency.

TSA Tracks Some Data on Rule Implementation, but Has Not Comprehensively Assessed Effectiveness TSA Has Monitored List Size and Number of Rule Matches, but Has Not Identified a Means to Comprehensively Measure Rule Effectiveness

TSA I&A monitors some operational data on its passenger screening  rules. For example, TSA I&A officials track the number of individuals on  the Silent Partner and Quiet Skies Lists, and the number of Silent Partner  and Quiet Skies rules triggered by the passengers’ travel. TSA I&A  officials stated that rule matches and list size are helpful for oversight  purposes because they allow TSA I&A to monitor for Secure Flight  system errors. Officials identified one example in which a Secure Flight  software update created a system error that prevented 808 passengers  from being moved to the Quiet Skies cleared list after a designated  number of flights. According to the officials, monitoring list size and the  number of rules triggered by passengers’ travel allowed them to identify  and correct this error within 10 days of identifying the system error.

TSA I&A has not identified a means to comprehensively measure rule  effectiveness. TSA I&A officials explained that they would find it helpful to  demonstrate the effectiveness of the program, but had not yet done so  because it was difficult to measure. TSA I&A officials reported that the  approach they have used was to count the number of Quiet Skies  passengers who were later identified as a known or suspected terrorist  and added to the Terrorist Screening Database. TSA I&A officials  reported that in January 2019 they reviewed all Quiet Skies passengers  from January 2014 through July 2018 to determine how many were  subsequently added to the Terrorist Screening Database. However,  because it included Quiet Skies only, this analysis excluded about 93  percent of the rules. TSA officials reported that it is not feasible to do a  similar analysis for Silent Partner rules because of the higher numbers of  rules and matches and the difficulty matching Silent Partner rules to data  in the Terrorist Screening Database. Further, TSA officials noted that  without comparable information on the rate that non-Quiet Skies  passengers were added to the Terrorist Screening Database during that  time period, it is difficult to interpret what the results indicate about rule  effectiveness.

TSA’s April 2012 Quiet Skies Implementation Plan established that TSA  would continually evaluate the performance of the rules in the Silent  Partner and Quiet Skies programs. Further, GAO and the Office of  Management and Budget have previously identified useful practices to  enhance performance management and measurement processes. GAO  has previously reported that measuring performance allows organizations  to track the progress they are making toward their goals and gives  managers critical information on which to base decisions for improving  their progress. Office of Management and Budget guidance has also focused specifically on common challenges associated with measuring  effectiveness, including data availability and identifying measurable  outcomes for a program. This guidance suggests using a variety of  approaches such as outlining short-term milestones, identifying target  outcomes, and using proxy measures to assess these programs.

Assessing the effectiveness of Silent Partner and Quiet Skies rules may  be difficult, but I&A could explore using other data sources to assess  program effectiveness in addition to further developing their consideration  of Terrorist Screening Database additions. For example, TSA I&A could  consider analyzing TSA data on the outcomes of the enhanced screening  of Silent Partner and Quiet Skies passengers at passenger security  checkpoints. CBP officials said that they review secondary inspection  results to help them assess CBP’s rules-based program. TSA I&A  officials noted that they were considering this measure and would need to  determine what comparison group would make sense, and if they want to  focus on specific screening outcomes versus all outcomes.

TSA I&A could also consider using the results of air marshals’ monitoring  of Quiet Skies passengers. According to senior Federal Air Marshal  Service officials, the service—with a budget of approximately $780 million  for fiscal year 2019—began deploying air marshals on as many flights as  possible with Quiet Skies passengers in March 2018. According to  TSA’s Privacy Impact Assessment for Silent Partner and Quiet Skies and  a Federal Air Marshal Service official, after air marshals complete a flight  with a Quiet Skies List match, they file a report saying either “nothing to report” or, if they observe that the individual was involved in a security  incident or suspicious activity, they will describe this in an after-action  report. TSA I&A officials told us that while they have seen individual  after-action reports, they do not review them regularly. These after-action  reports are another source of information TSA I&A could consider using  to gauge program effectiveness.

Given the TSA resources being devoted to the enhanced screening and  in-flight monitoring of many passengers matching the Silent Partner and  Quiet Skies Lists, and the burden on the traveling public, it is important  that TSA understand the value of its screening rules programs. Exploring  additional data sources—such as checkpoint screening results and  Federal Air Marshal Service after-action reports—could help TSA refine  and supplement their existing efforts to measure program effectiveness.

The attempted attack of December 25, 2009, highlighted the unknown  threats to U.S. civil aviation. TSA has created the Silent Partner and Quiet  Skies Lists to help address these unknown threats by ensuring that  certain potentially higher risk passengers receive enhanced screening  when traveling to, from, or within the United States. TSA created an  oversight process that was further bolstered by the TSA Modernization  Act, and DHS and TSA officials we met with generally regard the process  as effective. However, TSA SOPs are not clear about when it is  appropriate for TSA to use an expedited review process and they do not  document which review process they used. The lack of clear SOPs  inhibits program oversight. By establishing clear criteria for and  documentation of each review process, TSA could increase transparency  and ensure rule changes are reviewed as intended. Moreover, TSA has  not identified a means to comprehensively measure the effectiveness of  its Silent Partner and Quiet Skies rules. Exploring additional data  sources—such as checkpoint screening results and Federal Air Marshal  Service after-action reports—could help TSA refine and supplement their  existing efforts to measure program effectiveness.

We are making the following three recommendations to TSA:  The Administrator of TSA should clarify the criteria for exigent  circumstances and standard rule review procedures; (Recommendation  1)

The Administrator of TSA should document which rule review process  TSA I&A uses (exigent or standard) for each new rule or rule change;  (Recommendation 2)

The Administrator of TSA should explore additional data sources  measuring the effectiveness of Silent Partner and Quiet Skies rules.  (Recommendation 3)

We provided a draft of our report to DHS for comment. In written  comments, which are included in appendix I, DHS concurred with our  three recommendations and described steps they plan to take to address  them. DHS also provided technical comments, which we have  incorporated, as appropriate.

We are sending copies of this report to the appropriate congressional  committees and to the Acting Secretary of Homeland Security. In  addition, this report is available at no charge on the GAO website at  http://gao.gov.

If you or your staff have any questions concerning this report, please  contact me at (202) 512-8777 or russellw@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff who made significant  contributions to this report are listed in Appendix II.

Appendix I: Comments from the Department of Homeland Security

Appendix II: GAO Contacts and Staff  Acknowledgments Error! No text of specified style in document.

William Russell, (202) 512-8777 or russellw@gao.gov.

In addition to the contact named above Claudia Becker, Assistant  Director; Imoni Hampton, Analyst-in-Charge; Melissa Greenaway, John  De Ferrari, Michele Fejfar, Eric Hauswirth, Tom Lombardi, and Kevin  Reeves made key contributions to this work.

Airline passengers may be selected for enhanced screening even if they’re not known or suspected terrorists. In 2010, TSA began using rules it developed based on current intelligence to identify such passengers.

Department of Homeland Security and TSA stakeholders are to review rule changes before TSA implements them. However, in exigent circumstances, TSA can implement rule changes using a more streamlined process.

We found that TSA was frequently using this streamlined review process and the criteria for doing so wasn’t clear. To ensure transparency, we recommended TSA clarify its criteria for using the streamlined process.


FDA Drug Approval: Application Review Times Largely Reflect FDA Goals

Before a drug can be marketed in the United States, FDA must determine that the drug is safe and effective for its intended use through a review of evidence that a drug sponsor—the entity seeking to market the drug—submits in an NDA. The review is conducted by one of FDA's divisions (17, at the time of GAO's review) that each specialize in a specific group of drug products, such as hematology products. NDA reviews are complex, and may involve not only an initial review, but also reviews of resubmissions if the initial review does not result in approval. Under FDA's PDUFA commitments, FDA's goal is to complete reviews of 90 percent of NDAs within specific time frames linked to key features of the NDAs.

GAO was asked to examine NDA review times across FDA's divisions. In this report, GAO examines (among other things) differences between FDA divisions in the key features of the NDAs they review and initial review times, as well as the extent to which key NDA features contribute to these differences.

GAO analyzed data from FDA's Center for Drug Evaluation and Research regarding 637 NDAs submitted from fiscal years 2014 through 2018. These data also included biologic license applications submitted to the center. GAO excluded NDAs that were withdrawn by the applicant before FDA completed a review, as well as NDAs for which FDA had not completed a review by March 31, 2019. GAO also interviewed FDA officials about the agency's review process and these review times.

The Department of Health and Human Services provided technical comments on a draft of this report, which GAO incorporated as appropriate.

Four key features of new drug applications (NDA) are linked to the time the Food and Drug Administration (FDA) takes to complete initial reviews of NDAs. Three key NDA features determine the time frames for initial review that would meet FDA's goals under the Prescription Drug User Fee Act (PDUFA) and its reauthorizations, which authorize FDA to collect user fees from drug sponsors:

Whether or not the NDA qualifies for the priority review program, which is generally an expedited program for drugs that provide significant therapeutic improvements in the prevention, diagnosis, or treatment of a serious condition when compared to available drugs. The PDUFA goal for review of a priority NDA is 4 months less than for an otherwise similar standard NDA, for which the goal is to complete the review in 10 months.

Whether or not the NDA involves a new molecular entity (an active ingredient that has not been previously marketed or approved in the United States). The PDUFA goal for review of an NDA with a new molecular entity is 2 months longer than for an NDA without one.

Whether or not the applicant submits a major amendment (additional or new information, such as a major new clinical study) while the NDA is under review. The PDUFA goal for a review of an NDA may be extended by 3 months if the applicant submits a major amendment.

The fourth key NDA feature is whether or not it qualified for one or more of three other expedited programs for drugs intended to treat serious or life-threatening conditions.

GAO's analysis of 637 NDAs submitted from fiscal years 2014 through 2018 indicated that the proportion of NDAs with these key features differed among FDA review divisions. For example, 6 percent of the NDAs reviewed by the dermatology and dental division had a priority designation, compared to 56 percent for the anti-infective division. FDA has reported that some divisions, such as the oncology divisions, generally regulate products for conditions that are more likely to be serious or life-threatening, and, therefore, those products may be more likely to qualify for priority designation and other expedited programs.

GAO found that FDA's divisions differed in the average number of days they took to complete an initial review of NDAs, and these differences largely reflected the key features of the NDAs they reviewed. GAO's analysis shows that the time FDA took to complete an initial review of NDAs was affected by (1) the target time frame for completion of the review under the agency's PDUFA goals, (2) the number of expedited programs for which the NDA qualified, and (3) the division performing the review. GAO also found that the target time frame for review was largely responsible for differences in initial review times. Specifically, NDAs with key features that resulted in shorter target time frames for review under FDA's PDUFA goals had shorter initial review times. Controlling for the effects of these target time frames and the number of expedited programs for which the NDA qualified, GAO found that most of the divisions' average review times were similar to (within 2 weeks of) each other.

Getting safe, effective drugs to market in the United States typically  involves a lengthy drug development process. Before the drug can be  marketed, the Food and Drug Administration (FDA), an agency within the  Department Health and Human Services, must determine that the drug is  safe and effective for its intended use through a review of evidence that a  drug sponsor submits in a new drug application (NDA). The review is  conducted by one of the agency’s Center for Drug Evaluation and  Research (CDER) divisions, each of which specialize in a specific group  of drug products, such as hematology or neurology.

Four key features of NDAs are linked to drug development and review  processes. FDA’s goal for completing its initial review of certain standard  NDAs is 10 months after initial receipt, but some NDAs are subject to a  different initial review goal based on each of three key features of the  NDA:

FDA may designate NDAs for priority review when they are for drugs  that provide a significant improvement in safety or effectiveness for  treatment of a serious condition when compared to available drugs;  FDA’s goal is to review priority NDAs more rapidly than standard  ones.

FDA’s goal includes extra time if the NDA involves a new molecular  entity—an active ingredient that has not been previously marketed or  approved for use in the United States.

FDA may extend its goal if the applicant submits substantial additional  information while the NDA is under review.

A fourth key feature of NDAs is whether they qualify for one or more of  FDA’s expedited programs, which are intended to help reduce the  development or review time needed to bring a drug to market. NDAs for  therapies intended to treat serious or life-threatening conditions may  qualify for one or more of these programs.

NDA reviews are complex and may take multiple review cycles. While  conducting its initial review, FDA may determine that it needs additional  information or further evidence, and in such cases, the agency can end  the initial review with a letter to the applicant describing specific  deficiencies. The applicant can respond in a resubmission, initiating a  new cycle of review.

Because drug sponsors must collect evidence to demonstrate the safety  and effectiveness of new drugs, and these efforts represent a major  component of drug development time and cost, the amount and nature of  the evidence needed can be an important determinant of when and  whether new therapies become available to the public. The issue of what  constitutes sufficient evidence to support NDAs has been debated by  FDA, the scientific community, industry, and others. FDA has typically  required NDAs to include safety and effectiveness evidence for new  drugs from two adequate and well-controlled clinical trials. However,  under certain circumstances, drug sponsors can use different sources of  evidence to show that a new drug is safe and effective for its intended  use. The 21st Century Cures Act (Cures Act), enacted in 2016, directed  FDA to evaluate and facilitate the use of these different sources of  evidence by FDA reviewers and drug sponsors to inform the agency’s  assessment of drug safety and effectiveness.

FDA has published evidence showing that review times differ between  divisions. In light of this evidence, you asked us to examine NDA review  times across FDA’s divisions, as well as FDA’s use of certain tools to  inform the agency’s assessment of drug safety and effectiveness. This  report examines  1.  differences between FDA divisions in the proportion of NDAs they  review with key features;  2.  differences between FDA divisions in the time taken to complete initial  reviews and the extent to which the key NDA features contribute to  these differences; and  3.  actions FDA has recently taken to evaluate and facilitate the use of  different sources of evidence to support NDAs.

To address our first two objectives, we analyzed data from FDA regarding  637 NDAs—the NDAs that were initially submitted from fiscal years 2014  through 2018. (Appendix I provides a detailed description of the  methodologies we used to analyze these data.) Our examination  excluded NDAs that were withdrawn by the applicant before FDA  completed an initial review, as well as NDAs for which FDA had not  completed a review by March 31, 2019. For some analyses, we also  excluded five NDAs, as described in appendix I, bringing the count of  NDAs to 632. We reviewed the reliability of the data by conducting a  series of electronic and logic tests to identify missing data or other  anomalies and worked with FDA to correct information when we identified  discrepancies. We determined that the data were sufficiently reliable for  our purposes. In addition, although our focus was on initial review times,  we also collected information about total review times—review times  across all completed cycles of review; appendix II includes information  about these total review times. Finally, we interviewed FDA officials about  the agency’s review process and these review times.

To examine recent FDA actions to evaluate and facilitate the use of  different sources of evidence to support NDAs, we identified initiatives  FDA started implementing as a result of the enactment of the Cures Act in  late 2016. We focused our analysis on initiatives that (1) involve NDAs  reviewed by CDER divisions and (2) could affect a variety of diseases  and populations. We spoke with FDA officials and reviewed FDA  documentation to learn about the initiatives and determine steps FDA has  taken to implement the initiatives. We also interviewed two stakeholder  groups (Pharmaceutical Research and Manufacturers of America and  Biotechnology Innovation Organization) that represent drug sponsors  about the initiatives.

We conducted this performance audit from July 2018 to February 2020 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

In 1992, the Prescription Drug User Fee Act (PDUFA) was enacted, in  part, to provide additional funds for FDA to support the process of  reviewing NDAs. PDUFA authorized FDA to collect user fees from drug  sponsors to supplement its annual appropriation for salaries and  expenses. PDUFA has been reauthorized every 5 years since 1992;  most recently PDUFA VI reauthorized the prescription drug user fee  program from fiscal year 2018 through fiscal year 2022. As part of each  reauthorization process, FDA identifies goals in a commitment letter to  Congress. In general, these goals identify a percentage of certain types of  applications that FDA is expected to review within specified time frames,  including goals for the time the agency takes to complete reviews of  different types of NDAs upon initial submission and resubmission. For  example, in its commitment letters for PDUFA V and VI, FDA committed  to completing its initial review of 90 percent of priority NDAs that involve  previously marketed or approved active ingredients within 6 months of  receipt.

As previously noted, four key features of NDAs are linked to drug  development and review processes. For initial NDA reviews, the time  frames for FDA’s review that would meet its PDUFA V and VI  commitments—its PDUFA goals—vary and are linked to three key  features of the NDA. (See table 1.) The target time frame for the initial  review of any specific NDA under these user fee commitments reflects  the goals associated with all three of the key features.

The fourth key feature of NDAs is whether they qualify for one of FDA’s  expedited programs. Whether designated as priority or standard, FDA  may determine that NDAs for drugs intended to treat serious or life- threatening conditions qualify for development and review under one or  more expedited programs. These programs confer specific benefits with  the potential to help reduce the development or review time needed to  bring a drug to market.  For example, some expedited programs provide  for more intensive drug development guidance from FDA officials or allow  the applicant to submit completed sections of the NDA for review before  submitting the entire application. FDA’s expedited programs include  accelerated approval, breakthrough therapy designation, and fast track  designation. (See table 2.)

NDAs must include substantial evidence of a drug’s effectiveness, which  is typically drawn from clinical trials. In traditional clinical trials, patients  receiving a new drug are often compared with patients receiving a  placebo or a different drug. To maximize data quality, these clinical trials  are usually randomized (patients are randomly assigned to either the  group receiving the new drug or a comparison group) and double-blinded  (neither the patients nor the investigators know who is receiving a  particular treatment). According to FDA, although this type of study  design is often the most powerful tool for evaluating the safety and  effectiveness of new drugs, many traditional clinical trials are becoming  more costly and complex to administer. Additionally, according to FDA,  many new drugs are not easily evaluated using traditional approaches.  For example, drugs intended for patients with rare diseases are difficult to  evaluate due to the limited number of patients affected by the disease  and available for study.

The Cures Act was enacted on December 13, 2016, to accelerate the  discovery, development and delivery of new treatments—including  drugs—for patients. Among other things, the Cures Act includes  provisions for FDA to evaluate and facilitate the use of evidence from  sources other than traditional clinical trials to support safety and  effectiveness determinations for new drugs. For example, FDA was  directed to evaluate the potential use of evidence based on data that is  routinely collected outside of traditional clinical trials from sources such as  electronic health records, medical claims data, and disease registries;  evidence from such data sources is referred to as real-world evidence. In  the commitment letter associated with PDUFA VI, which was enacted on  August 18, 2017, the agency agreed to certain goals relating to the use of  real-world evidence in regulatory decision-making and also agreed to  certain activities intended to facilitate the development and application of  an additional source of evidence known as model-informed drug  development. Although these nontraditional sources of evidence were  included in NDAs prior to the enactment of the Cures Act and PDUFA VI,  at the time this legislation was enacted, most of them were not widely  used. For example, according to FDA officials, the NDAs that included  real-world evidence were generally for drugs to treat oncology diseases  or rare diseases.

FDA Divisions Differ in Proportions of NDAs Reviewed with One or More Key Features

Our analysis of the 637 original NDAs submitted from fiscal years 2014  through 2018 indicates that divisions differed in the proportions of NDAs  they reviewed that had any one of three key features that are linked to  time frames for initial review under FDA’s PDUFA goals. As examples:

6 percent of the NDAs reviewed by the dermatology and dental  division had a priority review designation, while 56 percent of the  NDAs reviewed by the anti-infective division had a priority review  designation;

4 percent of the NDAs reviewed by the anesthesia, analgesia, and  addiction division involved a new molecular entity, while 52 percent of  the NDAs reviewed by the neurology division involved one; and

None of the NDAs reviewed by the transplant and ophthalmology  division involved a major amendment, while 36 percent of the  applications reviewed by the gastroenterology and inborn errors  division involved one.  (See fig. 1. App. IV provides more detailed information about differences  between divisions in the number and proportion of NDAs with these key  features.)

We also found differences between divisions in the proportion of NDAs  that they reviewed under an expedited program—the fourth key feature of  NDAs. For example, none of the NDAs reviewed by the metabolism and  endocrinology division qualified for one or more expedited programs,  while 52 percent of the NDAs reviewed by the antiviral division qualified  for one or more expedited programs. (See fig. 2. App. V provides more  detailed information about differences between divisions in the number  and proportion of NDAs that qualified for one or more expedited  programs.)

It is not unexpected that divisions differ in the proportion of their  applications with key features linked to FDA’s time frames for review or  qualification for expedited programs because the divisions are  responsible for different products. For example, some divisions, such as  the oncology divisions, regulate products for conditions that are more  likely to be serious or life-threatening, and therefore the NDAs reviewed  by these divisions are more likely to qualify for priority review designation  and expedited programs, compared with other divisions, such as the  dermatology and dental division.

FDA Divisions Vary in Their Initial Review Times for NDAs, Largely Due to PDUFA Goals

Our analysis of review times for the 637 original NDAs submitted from  fiscal years 2014 through 2018 shows that FDA divisions differed in the  number of days they took to complete their initial reviews. For example,  the median time taken to complete an initial review of an NDA by the anti- infective division was about 2 months faster than the median time taken  by the gastroenterology and inborn errors division. (For more information  about initial review times, see app. VI.)

We found, however, that these differences in initial review times largely  reflected key features of the NDAs reviewed by the divisions, particularly  those features linked to FDA’s time frames for review under its PDUFA  goals. We analyzed initial review times using a statistical regression with  two variables reflecting key features of the NDAs—target time frame for  review of the application under FDA’s PDUFA goals (in days, from FDA’s  receipt of the NDA to FDA’s targeted date for completion of the initial  review) and number of expedited programs (0, 1, or 2 or more)—along  with division as independent variables. We found that each of these  variables was a significant determinant of initial review times. Specifically,  our regression analysis shows that on average

The shorter the target time frame for initial review of the NDA under  FDA’s PDUFA goals, the shorter the initial review, and this target time  frame was responsible for the majority of variation in initial review  times.

The greater the number of expedited programs for which the NDA  qualified, the shorter the time FDA took to complete the initial  review.

Controlling for the effects of these key NDA features, however, we found  that most of the divisions’ average review times were similar to (within 2  weeks of) each other. In contrast, the hematology and oncology divisions  reviewed applications a bit more rapidly—about 2 or 3 weeks faster—than  other divisions. Figure 3 illustrates the results of our analyses. The  panel on the left shows the variation in the divisions’ actual average  review times. The panel on the right shows the estimated average review  times, after accounting for key application features, that is, what the  review times would have been if each division had reviewed equal  numbers of applications with these key features.

We asked FDA officials what might contribute to somewhat faster review  times by the hematology and oncology divisions, and FDA officials told us  that a number of variables could have contributed to these differences.  For example, the officials told us that applicants differ in their level of  experience, which can affect the quality of the NDA or the speed of  response to FDA’s requests for information; applications differ in  complexity; and the oncology and hematology divisions could differ from  others in their risk/benefit considerations. As previously noted, some  divisions, such as the oncology divisions, regulate products for conditions  that are more likely to be serious or life-threatening compared with other  divisions, such as the dermatology and dental division, and risk/benefit  considerations can differ across conditions that vary in how serious or life- threatening they are. For example, the potential benefits of drugs that  carry substantial risks for dangerous side effects would likely be weighed  differently if the drug is intended to address a life-threatening illness for  which there is no other treatment than if the drug is intended to address  an illness that is not life-threatening or for which there is an alternative  treatment.

FDA Is Implementing Initiatives to Evaluate and Facilitate the Use of Different Evidence Sources to Support NDAs

FDA has several initiatives underway to evaluate and facilitate FDA  review divisions’ and drug sponsors’ use of evidence derived from  sources other than traditional clinical trials to support NDAs. (See table 3  for a description of these different evidence sources and each initiative.)

According to FDA officials, implementing these initiatives can help ensure  that when drug sponsors utilize these sources of evidence in NDAs, the  evidence is of sufficient quality to be used in regulatory decision-making  and that there is consistency across FDA review divisions in their  evaluation of the evidence. FDA officials also said that although complex  innovative trial designs might replace traditional clinical trials as evidence  in NDAs, real-world evidence is more likely to be used to supplement  clinical trial data.

Although the initiatives are not restricted to any particular type of disease  or patient population, according to FDA officials, some initiatives may be  more relevant for certain types of diseases or patient populations than  others. For example, according to FDA officials:  real-world evidence may be most relevant for diseases that have  outcomes that are consistently collected in the health care system.  clinical outcome assessments (one aspect of patient-focused drug  development) may be most relevant for diseases that are chronic,  symptomatic, or affect functioning and activities of daily living.  complex innovative trial designs may be most relevant for situations in  which the population size is small or limited, such as pediatric  populations, or where there is an unmet medical need, such as rare  diseases.

Our review of FDA documentation and interviews with FDA officials show  that FDA has taken steps to implement each of these five initiatives.  These steps include conducting public workshops with key stakeholders,  issuing guidance for industry and FDA staff, initiating pilot programs, and  developing FDA staff capacity, including by providing training and other  educational resources. (See table 4 for examples of key activities by  initiative.) These and future planned activities—including issuing  additional guidance and revising relevant FDA policies and procedures— are intended to address deliverables for FDA to accomplish through 2021  that are outlined in the Cures Act and the PDUFA VI commitment letter.  According to FDA officials, the agency intends to meet these deliverables,  though, according to these officials, some of the activities implemented  under the initiatives, such as certain pilot programs, will likely extend  beyond 2021.

Although implementation is still in progress for all of the initiatives, FDA  officials reported some outcomes. For example, since the launch of the  model-informed drug development pilot program, the agency has  received two NDA supplements that incorporated model-informed drug  development concepts discussed during pilot program meetings.  Additionally, officials told us there has been a recent increase in  investigational new drug submissions utilizing complex innovative trial  designs. FDA officials also reported an increase in biomarker  submissions under the drug development tool qualification program, and  continued growth of the clinical outcome assessment qualification  program. FDA expects that fully implementing the initiatives will lead to  further increases in the use of evidence from sources other than  traditional clinical trials.

We provided a draft of this report to the Department of Health and Human  Services for review and comment.  The department provided technical  comments, which we incorporated as appropriate.

As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date.  At that time, we will send copies of this report to the  appropriate congressional committees, the Secretary of the Department  of Health and Human Services, and other interested parties. In addition,  the report will be available at no charge on the GAO website at  http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-7114 or dickenj@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix VII.

Appendix I: Methodology for Data Analyses

To determine (1) how Food and Drug Administration (FDA) divisions differ  in the proportion of new drug applications (NDA) they review with key  features linked to review time goals and expedited programs and (2) how  FDA review divisions differ in the time taken to complete initial reviews  and the extent to which key features of NDAs contribute to those  differences, we analyzed data from FDA. We also interviewed FDA  officials about the data and their review processes.

We obtained data regarding all NDAs submitted to FDA’s Center for Drug  Evaluation and Research (CDER) from fiscal years 2014 through 2018.  These data included information about features that distinguish NDAs  from one another, including which division was responsible for the review.  The data also included information through March 31, 2019, about the  dates when FDA received and completed a review of each NDA, along  with the target dates for completion of review under FDA’s goals in  commitment letters associated with the Prescription Drug User Fee Act  (PDUFA) reauthorizations for fiscal years 2013 through 2017 (PDUFA V)  and fiscal years 2018 through 2022 (PDUFA VI).

To ensure meaningful analysis of review times, we excluded NDAs for  which FDA had not completed an initial cycle of review. Of 686 NDAs  submitted in fiscal years 2014 through 2018, the applicant withdrew 10  NDAs prior to completion of FDA’s initial review and 39 NDAs were still  under FDA review as of March 31, 2019, leaving 637 NDAs for which  FDA had completed an initial review.

To assess the reliability of these data, we conducted a series of electronic  and logic tests to identify missing data or other anomalies. These  analyses were informed by our review of relevant documentation and  interviews with knowledgeable FDA officials. As part of our assessment of  reliability, we worked with FDA to identify and correct information about  certain NDAs in a small number of instances in which we identified  discrepancies. Using these methods, we determined that the remaining  data were sufficiently reliable for our purposes. Unless otherwise  specified, the results we present are statistically significant at the 0.05  level.

To determine how FDA divisions differ in the proportion of NDAs they  review with key features linked to FDA’s time frames for initial reviews  and expedited programs, we conducted a series of chi-square tests  comparing the distributions of the 637 NDAs with and without specific  features across divisions. These key features included: whether the NDA had a priority review designation (a designation  applied by FDA if the product would provide a significant therapeutic  improvement in the safety and effectiveness of the prevention,  diagnosis, or treatment of a serious condition when compared to  available drugs) or instead had a standard designation; whether the NDA did or did not involve a new molecular entity—an  active ingredient that had not previously been marketed or approved  for use as a drug in the United States, whether the NDA did or did not involve a major amendment (a  submission, while a pending NDA is under FDA review, of additional  information that may include a major new clinical safety or efficacy  study report or major new analyses of studies, among other things);  and whether the NDA did or did not qualify for an expedited program  (accelerated approval, breakthrough therapy designation, or fast track  designation), programs intended to help reduce the time involved in  developing or reviewing certain drugs that have the potential to treat  serious or life-threatening conditions.  (See table 5 for relevant statistics from these chi-square tests.)

To determine how FDA review divisions differ in the time taken to  complete initial reviews, we conducted a preliminary regression analysis  of 637 NDAs with the number of days an FDA division took to complete  its initial review as the dependent variable and division as a single  independent variable. We defined the time to complete a review as the  number of days from FDA’s receipt of the NDA to the agency’s  completion of the initial review by taking regulatory action.

To determine the extent to which key NDA features contributed to  differences between divisions in the time taken to complete initial reviews,  we conducted a multiple regression analysis of the number of days FDA  took to complete its initial review with division as an independent variable,  along with two other independent variables to control for the key NDA  features:

Target time frame for initial review of the NDA under FDA’s  PDUFA goals. Three key NDA features are linked to time frames for  FDA’s initial review under its PDUFA goals—whether the NDA was  priority or standard, did or did not involve a new molecular entity, and  did or did not involve a major amendment. To control for these three  features simultaneously, we counted the number of days from FDA’s  receipt of the NDA until FDA’s target date for completion of the initial  review under FDA’s PDUFA goals, and used that variable—the target  time frame for review under FDA’s PDUFA goals—as an independent  variable. We identified five NDAs for which FDA’s review time was  exceptionally long in comparison to the target time frame for review  under its PDUFA goals, and we asked FDA officials about them. FDA  officials stated that these reviews were substantially delayed because  of complicated manufacturing site issues, complicated legal and  regulatory issues, or emerging public health issues requiring last  minute advisory committee meetings—conditions that we deemed  sufficiently unusual to exclude these five NDAs from further statistical  analyses of review times.

Number of expedited programs for which the NDA qualified.

Another key NDA feature is whether it qualified for one or more  expedited programs, programs with the potential to help reduce the  development or review time needed to bring a drug to market. We  controlled for this feature by including number of expedited programs  (0, 1, or 2 or more) as an independent variable in our multiple  regression analysis.

Thus, we tested the effect of division on initial review times for 632 NDAs  while controlling for the target time frame for review under FDA’s PDUFA  goals and qualification for expedited programs. (See tables 6 and 7 for  relevant statistics from this multiple regression analysis.)

Our multiple regression analysis allowed us to test a specific hypothesis  about the effect of division on review times, namely, whether divisions  differed in their review times after controlling for the key features of NDAs.  This regression analysis did not test a model of review times—that is, we  did not attempt to identify all variables that affect review times, nor did we  seek to identify the specific set or combination of variables within our data  that had maximum explanatory power. Our analyses indicated that  variation remained in initial review times, even after we controlled for  these variables. It is important to note that an array of factors might be  expected to influence review times, including not just those factors that  were captured in our analysis, but also factors such as state of the  science and quality of the application.

With data from 632 NDAs distributed unevenly across 15 divisions,  meaningful tests of additional variables or their interactions were not  possible. Nonetheless, we conducted exploratory analyses that included  other potentially relevant variables in addition to the target time frame for  review under FDA’s PDUFA goals, number of expedited programs, and  division. In separate regression analyses, we examined (a) the fiscal year  in which FDA received the NDA and (b) whether the application was a  BLA, an NDA based on information from studies conducted by the  applicant, or an NDA based on at least some information from studies not  conducted by or for the applicant. We did not find evidence of a  consistent effect of either of these additional factors on review times, but  in light of the number of NDAs, we cannot exclude the possibility that one  or more of these factors affects review times. In a third exploratory  analysis, we examined the outcome of the initial review—(a) approval; (b)  tentative approval, which FDA grants if the NDA meets requirements for  approval, but cannot be approved due to a patent or exclusivity period for  a listed drug; or (c) issuance of a letter to the applicant called a complete  response letter, in which FDA describes the specific deficiencies the  agency identified and recommends ways to make the application viable  for approval. This analysis suggested that NDAs that were approved for  marketing at the end of the initial cycle of review were reviewed slightly  faster on average than other NDAs, but this result should be viewed with  caution because a small number of NDAs with certain initial review  outcomes were distributed unequally. For example, very few of the NDAs  (11) reviewed through one or more expedited programs resulted in  tentative approval.

Appendix II: Total Times Taken by FDA Divisions to Review New Drug Applications Received in Fiscal Years 2014 through 2018

The Food and Drug Administration’s (FDA) Center for Drug Evaluation  and Research (CDER) divisions differed in the total number of days they  took to complete reviews of 637 new drug applications (NDA) submitted  from fiscal years 2014 through 2018 and completed by March 31, 2019.  (See fig. 4.) Importantly, these times reflect differences associated with  the number of completed review cycles, FDA’s target time frames for  review under its goals in commitment letters associated with the  Prescription Drug User Fee Act (PDUFA) reauthorizations for fiscal years  2013 through 2017 (PDUFA V) and fiscal years 2018 through 2022  (PDUFA VI), and number of expedited programs.

Number of review cycles. The number of cycles of review to which  the NDAs we examined were subject was largely dependent on  factors that were not under FDA’s control, namely, the applicant’s  actions and timing. When a cycle of review ends with an FDA action,  that action can be (a) approval, which allows the applicant to market  the drug, (b) tentative approval, which FDA grants if the NDA meets  requirements for approval, but cannot be approved due to a patent or  exclusivity period for a listed drug, or (c) issuance of a letter to the  applicant called a complete response letter, in which FDA describes  the specific deficiencies the agency identified and recommends ways  to make the application viable for approval. The applicant may  respond to either tentative approval or a complete response letter by  resubmitting a revised application, triggering a new cycle of review; it  is up to the applicant to decide whether to resubmit the application. In  addition, NDAs that were submitted earlier in time would have a  greater chance of being resubmitted and reviewed by March 31, 2019,  than applications submitted later in time. The number of completed  review cycles ranged from one to four cycles:

637 NDAs went through a completed first (initial) cycle review;

99 of those 637 NDAs went through a completed second cycle  review;

20 of those 99 NDAs went through a completed third cycle review;

3 of those 20 NDAs went through a completed fourth cycle review.

Target time frames for review. Review times reflect differences in  time frames for review under FDA’s PDUFA goals. The target time  frames for review ranged from less than 6 months to 15 months for  the first cycle and from less than 2 months to 9 months for later cycles  of review.

Number of expedited programs. These review times also reflect  differences associated with the number of FDA’s expedited programs  for which NDAs qualified. In general, these expedited programs are  designed to help reduce the development or review time needed for  drugs intended to treat serious or life-threatening conditions.

Appendix III: Requests for Breakthrough Therapy and Fast Track Designations, Fiscal Years 2013 through 2018

Two of the Food and Drug Administration’s (FDA) expedited programs for  new drugs intended to treat serious or life-threatening conditions— breakthrough therapy designation and fast track designation—must be  requested by the drug sponsor. These programs are intended to help  reduce the development or review time needed to bring a drug to market  by offering benefits such as more intensive drug development guidance  from FDA officials or by allowing the applicant to submit completed  sections of the NDA for review before submitting the entire application.  The request is normally made while the drug sponsor is conducting  clinical trials or when seeking FDA’s permission to collect clinical trial  data, although the request may also be made when submitting a new  drug application (NDA) or while the NDA is under review.

FDA’s Center for Drug Evaluation and Research (CDER) divisions are  responsible for determining whether requests qualify for these expedited  programs based on evidence the drug sponsors provide in support of the  requests. To qualify for breakthrough therapy designation, the drug  sponsor must present preliminary clinical evidence involving one or more  clinically significant endpoints that indicate that the drug may demonstrate  substantial improvement over available therapies. To qualify for fast track  designation, the drug sponsor must either provide evidence  demonstrating the drug’s potential to address unmet need or document  that the drug is designated as a qualified infectious disease product. FDA  may grant or deny the request, or the drug sponsor may withdraw the  request before FDA renders a decision. If FDA grants the designation, the  drug sponsor may subsequently withdraw from the designation, or FDA  may rescind either designation if the drug no longer meets the qualifying  criteria.

We obtained data regarding all requests for breakthrough therapy and  fast track designations submitted to CDER from fiscal years 2013 through  2018. These data included information about which division was  responsible for the review and the outcome of the request—whether it  was granted or denied or whether the drug sponsor withdrew the request  before FDA reached a decision. To assess the reliability of these data, we  conducted a series of electronic and logic tests to identify missing data or  other anomalies. These analyses were informed by our review of relevant  documentation and interviews with knowledgeable FDA officials. Using  these methods, we determined that the data were sufficiently reliable for  our purposes. We examined these data to determine whether there were  any material differences between divisions in the frequency of possible  outcomes. Our analyses focused on the outcomes and did not allow us to  determine whether divisions differed in their application of the stated  criteria.

Breakthrough therapy designation. We found few differences across  divisions in the frequency of the possible outcomes of requests for  breakthrough therapy designation:

Of 634 requests for breakthrough therapy designation (including nine  requests submitted with or after the NDA submission), 39 percent  were granted, 48 percent were denied, and 13 percent were  withdrawn by the drug sponsor before FDA reached a decision.

Divisions differed widely in the number of requests for breakthrough  therapy designation they received, from 0 for the nonprescription drug  division to 102 for one of FDA’s two oncology divisions.

With two exceptions, the numbers of these requests that were  granted, denied, or withdrawn for each division were similar to what  would be expected based on the overall frequency of the possible  outcomes. Requests to the hematology division were withdrawn more  frequently than requests to other divisions (32 percent) and that  division denied requests less frequently (17 percent) than other  divisions. The neurology division denied more (81 percent), and  granted fewer (13 percent), requests for breakthrough therapy  designation than other divisions.

Within the time period we studied, the drug sponsor withdrew from  breakthrough therapy designation after it was granted in six cases and  FDA rescinded the designation in 14 cases.

Fast track designation. Similarly, we found few differences across  divisions in the frequency of the possible outcomes of requests for fast  track designation:

Of 965 requests for fast track designation (including 35 requests  submitted with or after the NDA submission), 71 percent were  granted, 24 percent were denied, and 5 percent were withdrawn by  the drug sponsor before FDA reached a decision.

Again, divisions differed widely in the number of requests for fast track  designation they received, from 2 for the nonprescription drug division  to 133 for the neurology division.

The numbers of these requests that were granted, denied, or  withdrawn for each division were generally similar to what would be  expected based on the overall frequency of the possible outcomes,  although the anti-infective division granted more (91 percent), and  denied fewer (6 percent), requests for fast track designation than  other divisions.

Within the time period we studied, no drug sponsor withdrew from fast  track designation after it was granted, nor did FDA rescind any such  designation.

Appendix IV: New Drug Applications with Key Features Linked to Time Frames for Review, Fiscal Years 2014 through 2018

Pursuant to the Prescription Drug User Fee Act (PDUFA) and its  subsequent reauthorizations, the Food and Drug Administration (FDA)  collects user fees from drug sponsors to supplement its annual  appropriation for salaries and expenses. As part of each reauthorization  process, FDA identifies goals in a commitment letter to Congress,  including goals for the time the agency takes to complete reviews of  different types of drug applications upon initial submission and  resubmission. In general, these goals identify a percentage of certain  types applications that FDA is expected to review within specified target  time frames. For initial NDA reviews—reviews of the NDA as originally  submitted—FDA’s target time frames for review that would meet its  PDUFA goals vary and are linked to three key NDA features that reflect  the drug or the applicant’s action: (1) whether or not the application  receives priority review designation, which indicates that the drug could  provide significant therapeutic improvements in the safety and  effectiveness of the prevention, diagnosis, or treatment of a serious  condition when compared to available drugs; (2) whether or not the  application involves a new molecular entity—an active ingredient that has  not been previously marketed or approved for use in the United States;  and (3) whether or not the applicant submitted a major amendment while  the NDA was pending, that is, while under FDA’s review. The target time  frame for review for any specific NDA reflects all three of these features.  Reviews are conducted by one of the agency’s Center for Drug  Evaluation and Research (CDER) divisions, each of which specialize in a  specific group of drug products, such as hematology or neurology.

As shown in table 8, divisions differed in the numbers and proportions of  NDAs they reviewed that had the features linked to time frames for review  under FDA’s PDUFA goals.

Appendix V: New Drug Applications That Qualified for Expedited Programs, Fiscal Years 2014 through 2018

The Food and Drug Administration (FDA) may determine that NDAs for  drugs intended to treat serious or life-threatening conditions qualify for  one or more expedited programs. These programs confer specific  benefits with the potential to help reduce the development or review time  needed to bring a drug to market, for example, some expedited programs  provide for more intensive drug development guidance from FDA officials  or allow the applicant to submit completed sections of the NDA for review  before submitting the entire application. FDA’s expedited programs  include accelerated approval, breakthrough therapy designation, and fast  track designation. Reviews are conducted by one of the agency’s Center  for Drug Evaluation and Research (CDER) divisions, each of which  specialize in a specific group of drug products, such as hematology or  neurology.

As shown in table 9, divisions differed in the proportions of NDAs they  reviewed that qualified for expedited programs.

Appendix VI: Times Taken to Complete Initial Reviews of New Drug Applications Received from Fiscal Year 2014 through 2018

The Food and Drug Administration’s (FDA) Center for Drug Evaluation  and Research (CDER) divisions differed in the total number of days they  took to complete initial reviews of new drug applications (NDA) received  from fiscal years 2014 through 2018 and completed by March 31, 2019.  (See fig. 5.) These review times reflect differences associated with FDA’s  target time frames for initial review under its goals in commitment letters  associated with the Prescription Drug User Fee Act (PDUFA)  reauthorizations for fiscal years 2013 through 2017 (PDUFA V) and fiscal  years 2018 through 2022 (PDUFA VI). These target time frames for  review are linked to specific features of the NDA and ranged from less  than 6 months to 15 months for the initial review. These review times also  reflect differences associated with the number of expedited programs for  which NDAs qualified.

Appendix VII: GAO Contact and Staff Acknowledgments

John E. Dicken, (202) 512-7114 or dickenj@gao.gov.

In addition to the contact named above, William Hadley (Assistant  Director), Geri Redican-Bigott (Assistant Director), Aubrey Naffis (Analyst- in-Charge), and Kristen Joan Anderson made key contributions to this  report. Also contributing were Sam Amrhein, Todd D. Anderson, Leia  Dickerson, Kaitlin Farquharson, Rich Lipinski, and Ethiene Salgado- Rodriguez.

Before a drug can be marketed in the United States, the FDA must determine it is safe and effective. The company or entity seeking to market the drug must submit evidence for review. FDA’s goal is to complete 90% of these reviews within deadlines that vary depending on the drug. Some drugs—ones that may provide significant improvement over what’s available—receive priority designations for expedited reviews.

We reviewed 637 new drug applications submitted from FY 2014-2018 and found the FDA met its goals. Differences in review times among FDA’s 17 divisions were largely driven by the characteristics of the applications being reviewed.