Critical Infrastructure Protection: Key Pipeline Security Documents Need to Reflect Current Operating Environment

More than 2.7 million miles of pipeline transport natural gas, oil, and other hazardous liquids needed to operate vehicles and heat homes, among other things, in the United States.

Responsibility for safeguarding these pipelines is shared by TSA, within the Department of Homeland Security (DHS); PHMSA, within the Department of Transportation (DOT); and pipeline operators. TSA oversees the security of all transportation modes, including pipelines. PHMSA oversees pipeline safety. DHS and DOT signed a MOU on their roles across all transportation modes in 2004. In 2006, TSA and PHMSA signed an annex to the MOU (MOU Annex) to further delineate their pipeline security-related responsibilities.

The TSA Modernization Act includes a provision for GAO to review DHS and DOT roles and responsibilities for pipeline security. This report addresses, among other things: (1) the extent the MOU Annex delineates TSA's and PHMSA's pipeline security roles and responsibilities; and (2) the extent TSA has communicated federal incident response procedures for pipeline breaches to stakeholders. GAO reviewed the MOU annex and related documents and TSA's Pipeline Security and Incident Recovery Protocol Plan, and interviewed officials from PHMSA, TSA, and four pipeline associations.

The memorandum of understanding (MOU) Annex signed by the Transportation Security Administration (TSA) and Pipeline and Hazardous Materials Safety Administration (PHMSA) in 2006 delineates their mutually agreed-upon roles and responsibilities for pipeline security, but has not been reviewed to consider pipeline security developments since its inception. As a result, the annex may not fully reflect the agencies' pipeline security and safety-related activities. Efforts to update the annex were delayed by other priorities. As of June 2019, there are no timeframes for completion. By developing and implementing timeframes for reviewing the MOU Annex and updating it, as appropriate, TSA and PHMSA could better ensure any future changes to their respective roles and responsibilities are clearly delineated and updated on a regular basis.

TSA's Pipeline Security and Incident Recovery Protocol Plan, issued in March 2010, defines the roles and responsibilities of federal agencies and the private sector, among others, related to pipeline security incidents. For example, in response to a pipeline incident, TSA coordinates information sharing between federal and pipeline stakeholders and PHMSA coordinates federal activities with an affected pipeline operator to restore service. However, TSA has not revised the plan to reflect changes in at least three key areas: pipeline security threats, such as those related to cybersecurity, incident management policies, and DHS's terrorism alert system. By periodically reviewing and, as appropriate, updating its plan, TSA could better ensure it addresses changes in pipeline security threats and federal law and policy related to cybersecurity, incident management and DHS's terrorism alert system, among other things. TSA could also provide greater assurance that pipeline stakeholders understand federal roles and responsibilities related to pipeline incidents, including cyber incidents, and that response efforts to such incidents are well-coordinated.

GAO is making five recommendations, including that: (1) TSA and PHMSA develop and implement a timeline for reviewing and, as appropriate, updating the 2006 MOU Annex; and (2) TSA periodically review, and as appropriate, update its 2010 pipeline incident recovery plan. DHS and DOT concurred with these recommendations.

More than 2.7 million miles of pipeline transport and distribute the natural  gas, oil, and other hazardous liquids that U.S. citizens and businesses  depend on to operate vehicles and machinery, heat homes, generate  electricity, and manufacture products. The interstate pipeline system runs  through remote, as well as highly populated urban areas, and is generally  considered to be resilient and versatile. However, it is also vulnerable to  accidents, operating errors, and malicious physical attack. Some  pipelines are also vulnerable to aging infrastructure. In addition, pipelines  increasingly rely on sophisticated networked computerized systems and  electronic data, which are vulnerable to cyber attack or intrusion.

Many pipelines transport volatile, flammable, or toxic products. As  demonstrated by the September 2018 explosion of a Merrimack Valley,  Massachusetts natural gas distribution pipeline system, the potential  consequences of a catastrophic event on life, property, the economy, and  the environment resulting from a natural disaster, operational accident, or  from a successful physical or cyber attack on a pipeline are high. A  minor pipeline system disruption could result in commodity price  increases while prolonged pipeline disruptions could lead to widespread  energy shortages. Further, disruption of any magnitude may affect other  domestic critical infrastructure and industries that are dependent on  pipeline system commodities.

Responsibility for safeguarding the nation’s pipeline systems from such  catastrophic events is shared by the Department of Homeland Security’s  (DHS) Transportation Security Administration (TSA), the Department of  Transportation’s (DOT) Pipeline and Hazardous Materials Safety  Administration (PHMSA), and pipeline owners and operators. TSA is  responsible for security in all modes of transportation, which includes the  physical security and cybersecurity of the nation’s pipeline system.  PHMSA is responsible for overseeing the safety of the nation’s pipeline  system. In September 2004, DHS and DOT entered into a memorandum  of understanding (MOU) regarding their respective roles across all modes  of transportation. In August 2006, TSA and PHMSA signed an Annex to  the MOU (MOU Annex) to further delineate lines of authority and  responsibility between the agencies on pipeline and hazardous materials  transportation security. The MOU Annex recognizes TSA as the lead  federal entity for transportation security, including hazardous materials  and pipeline security, and PHMSA as responsible for administering a  national program of safety in natural gas and hazardous liquid pipeline  transportation, including identifying pipeline safety concerns and  developing uniform safety standards. Private sector pipeline operators are  responsible for implementing asset-specific safety standards and  protective security measures.

In September 2018, we issued an update to the information security high- risk area that identified actions needed to address cybersecurity  challenges facing the nation. For example, challenges we identified  included protecting the cybersecurity of the nation’s critical infrastructure,  which includes pipeline systems. We last reported on pipeline security in  December 2018. TSA concurred with all ten of our recommendations, and  we will continue to monitor the status of implementation.

The TSA Modernization Act includes a provision for us to conduct a study  regarding the roles and responsibilities of DHS and the DOT for pipeline  security. We briefed your offices on our preliminary results on March 29,  2018. This report addresses the following questions: (1) To what extent  does the 2006 Annex to the MOU between DHS and DOT delineate  TSA’s and PHMSA’s responsibilities for pipeline security? (2) How do  TSA and PHMSA communicate their roles and responsibilities related to  pipeline safety and security and what are industry stakeholder views on  the clarity of the communication? (3) To what extent has federal incident  response processes and procedures for pipeline security breaches been  communicated to stakeholders?

To identify the extent to which the 2006 Annex to the MOU between DHS  and DOT delineates TSA and PHMSA responsibilities for pipeline  security, we reviewed relevant TSA and PHMSA documents including the  2006 MOU Annex, action plans for implementing provisions of the MOU  Annex, and documents related to the agencies’ process for revising the  Annex. We assessed TSA and PHMSA efforts to revise the 2006 MOU  Annex against relevant standards in the Standards for Internal Controls in  the Federal Government and project management guidance related to  periodically reviewing policies and developing project timelines with  milestone dates. We also reviewed relevant laws, regulations, and  statements of Executive Branch policy, including presidential directives. In  addition, we conducted semi-structured interviews with TSA and PHMSA  officials to obtain their perspectives on respective pipeline security roles  and responsibilities.

To identify how TSA and PHMSA communicate their roles and  responsibilities related to pipeline safety and security, we reviewed TSA  and PHMSA documents that describe each agency’s respective pipeline  security and safety programs such as TSA’s Pipeline Security Guidelines,  and PHMSA’s pipeline safety regulations and advisory bulletins. We  conducted interviews with TSA and PHMSA officials to identify the types  of activities they conduct to communicate and clarify their respective roles  and responsibilities to stakeholders. To assess industry stakeholder views  on the clarity of the communication, we interviewed representatives of the  four of the five major associations with ties to the pipeline industry.

To identify the extent to which federal incident response processes and  procedures for pipeline security breaches have been communicated to  stakeholders, we reviewed TSA’s Pipeline Security and Incident Recovery  Protocol Plan against criteria outlined in the Standards for Internal Control  in the Federal Government related to periodic review of policies,  procedures, and related control activities. To determine the extent that the  plan remains current and reflects relevant federal laws and policies, we  reviewed federal laws related to critical infrastructure protection that had  been enacted since TSA issued the plan in March 2010, and federal  incident management policies referenced by the plan including, the  Federal Emergency Management Agency’s (FEMA) National Response  Framework (NRF) and National Incident Management System (NIMS).  We also interviewed TSA, PHMSA, and association officials to gather  their perspectives on pipeline incident response processes and  procedures for responding to pipeline security breaches.

For each objective, we interviewed representatives of four of five major  associations with ties to the pipeline industry: the American Petroleum  Institute (API), the American Gas Association, the Interstate Natural Gas  Association of America, and the American Public Gas Association. While  the information gathered during association interviews cannot be  generalized to all pipeline operators, it provides a range of perspectives  on a variety of topics relevant to the 2006 MOU Annex.

We conducted this performance audit from January to June 2019 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

The national pipeline system consists of more than 2.7 million miles of  networked pipelines transporting natural gas, oil, and other hazardous  liquids. Natural gas and hazardous liquid pipelines—primarily buried  underground in the continental United States—run under remote and  open terrain, as well as densely-populated areas. There are three main  types of pipelines based on the types of materials transported:

Hazardous liquid: About 216,000 miles of hazardous liquid pipeline  transport crude oil, diesel fuel, gasoline, jet fuel, anhydrous ammonia,  and carbon dioxide.

Natural gas transmission and storage: About 319,000 miles of  pipeline—mostly interstate—transport natural gas from sources to  communities.

Natural gas distribution: About 2.2 million miles of pipeline—mostly  intrastate—transport natural gas from transmission sites to  consumers.

Figure 1 depicts the network of hazardous liquid and natural gas  transmission pipelines in the United States.

More than 3,000 pipeline companies operate the nation’s pipeline  systems, which can traverse multiple states and the U.S. borders with  Canada and Mexico. Many pipeline systems are comprised of the  pipelines themselves, as well as a variety of facilities, such as storage  tanks, compressor stations, and control centers. Most pipeline systems  are monitored through automated industrial control systems or  Supervisory Control and Data Acquisition (SCADA) systems using remote  sensors, signals, and preprogramed parameters to activate and  deactivate valves and pumps to maintain flows within established  tolerance levels.

Pipeline accidents can occur from a variety of causes, including third- party excavation, corrosion, mechanical failure, control system failure,  and operator error. Natural forces, such as floods and earthquakes, can  also damage pipelines. Although pipeline releases have caused relatively  few fatalities, a single pipeline accident can be catastrophic in terms of  public safety and environmental damage. Figure 2 shows notable  pipeline accidents since September 2010.

According to TSA, pipelines are also vulnerable to physical attacks by  crude or unsophisticated tactics, such as rudimentary explosives, arson,  or equipment sabotage—largely due to their stationary nature, the  volatility of transported products, and the dispersed nature of pipeline  networks spanning urban and outlying areas. Threats to the nation’s  pipeline systems include sabotage by activists, physical attack by  terrorists, and cyber attack or intrusion by nations. In October 2016,  environmental activists forced the shutdown of five crude oil pipelines in  four states: Minnesota, North Dakota, Montana, and Washington State.  Further, in January 2019, the Director of National Intelligence stated that  China has the ability to launch cyber attacks that have caused localized,  temporary disruptive effects on critical infrastructure—such as disruption  of a natural gas pipeline for days to weeks—in the United States.

Key Critical Infrastructure Protection Guidance and Presidential Directives

Federal policy and public-private plans establish the roles and  responsibilities for the protection of critical infrastructure, including  pipelines. These policies and public private plans include Presidential  Policy Directive /PPD-21 (PPD-21) and the National Infrastructure  Protection Plan (NIPP). PPD-21, issued in February 2013, was developed  to advance a national unity of effort to strengthen and maintain secure,  functioning, and resilient critical infrastructure, which includes pipelines.  PPD-21 reflects an all-hazards approach to protecting critical  infrastructure, by accounting for the protection of critical infrastructure  from natural or manmade threats or incidents. Examples of threats or  incidents include natural disasters, cyber incidents, industrial accidents,  pandemics, acts of terrorism, sabotage, and destructive criminal activity  targeting critical infrastructure. PPD-21 also identifies the 16 critical  infrastructure sectors and assigns roles and responsibilities for each  sector among nine designated federal sector-specific agencies as shown  in Figure 3.

While PPD-21 identifies the critical infrastructure sectors and assigns  responsibility for each sector’s sector-specific agency, the NIPP outlines  critical infrastructure stakeholder roles and responsibilities regarding  critical security and resilience. The NIPP describes a voluntary  partnership model as the primary means of coordinating government and  private sector efforts to protect critical infrastructure. As part of the  partnership structure, the designated sector-specific agencies serve as  the lead coordinators for security programs of their respective sector. For  example, DHS and DOT are designated as co-sector-specific agencies  for the transportation systems sector, which includes pipelines. Each  sector also has a government coordinating council, consisting of  representatives from various levels of government, and many have a  sector coordinating council (SCC) consisting of owner-operators of these  critical assets or members of their respective trade associations. For  example, the Transportation Government Coordinating Council has been  established, and the Pipeline Modal SCC has also been established to  represent pipeline operators.

Pipeline Stakeholder Roles and Responsibilities

Protecting the nation’s pipeline systems is a responsibility shared by both  the federal government and private industry. As a result, several federal  departments, agencies, and the private sector have significant roles in  pipeline safety and security. The entities primarily responsible for  pipeline safety and security are included below.

Transportation Security Administration (TSA). TSA has primary  oversight responsibility for the physical security and cybersecurity of  transmission and distribution pipeline systems. Within TSA, the Policy,  Plans, and Engagement’s Pipeline Security Branch is charged with  overseeing its pipeline security program. Pursuant to the Implementing  Recommendations of the 9/11 Commission Act of 2007 (9/11  Commission Act), TSA’s Pipeline Security Branch issued voluntary  Pipeline Security Guidelines in 2011, and released revised guidelines in  March 2018. Further, in accordance with the 9/11 Commission Act,  TSA’s Pipeline Security Branch also identifies the top 100 critical pipeline  systems in the nation. TSA also ranks the relative risk among these top  100 systems. Additionally, the Pipeline Security Branch is responsible for  conducting voluntary security reviews, which assess the extent to which  these 100 pipeline systems are following the intent of TSA’s Pipeline  Security Guidelines.

Pipeline and Hazardous Materials Safety Administration (PHMSA).  PHMSA, within DOT, is responsible for regulating the safety of hazardous  materials transportation and the safety of pipeline systems, some aspects  of which may relate to pipeline security. PHMSA develops regulations  for domestic interstate and intrastate natural gas and hazardous liquid  pipelines. Its regulatory programs are focused on ensuring safety in the  design, construction, operation, and maintenance of pipelines. Under  PHMSA’s pipeline safety program, pipeline operators have primary  responsibility for ensuring the integrity of their pipelines. PHMSA and  some state pipeline safety offices are responsible for conducting  inspections to oversee operators’ compliance with federal pipeline safety  regulations and other federal requirements. Inspectors from PHMSA’s  five regional offices and states are responsible for inspecting nearly 3,000  companies that operate 2.7 million miles of pipelines.

Private sector. Although TSA has primary federal responsibility for  overseeing interstate pipeline security, private sector and publicly-owned  pipeline operators are responsible for implementing asset-specific  protective security measures. As we previously reported, since the  September 11th terrorists attacks, operators have increased their  attention on security by incorporating security practices and programs into  their overall business operations. Pipeline operators’ interests and  concerns are primarily represented by five major trade associations—the  Interstate Natural Gas Association of America, American Gas  Association, American Public Gas Association, American Petroleum  Institute (API), and Association of Oil Pipe Lines. According to TSA  officials, pipeline operators, and association representatives, these  associations have worked closely with the federal government on a  variety of pipeline security-related issues, including collaborating on  TSA’s voluntary standards and information sharing.

MOU Annex Delineates Pipeline Security Roles and Responsibilities But Has Not Been Reviewed to Consider Pipeline Security Developments Since 2006

MOU Annex Delineates Pipeline Security and Safety Roles and Responsibilities

The MOU Annex delineates TSA and PHMSA mutually agreed-upon  pipeline security roles and responsibilities, consistent with their respective  missions, and acknowledges that both agencies benefit by sharing each  other’s expertise, among other things. Specifically, the MOU Annex  identifies 11 program areas, where TSA and PHMSA agreed to  coordinate their respective roles and responsibilities. The first program  area for example, calls for both agencies to coordinate efforts to identify  critical infrastructure, and to share relevant data and observations found  during respective safety inspections and security assessments. Another  program area addresses coordination in developing transportation  security standards, regulations, guidelines, or directives. The MOU Annex  further provides that TSA and PHMSA are to seek early and frequent  coordination in developing such standards, regulations, guidelines, or  directives. They are also to review the adequacy of existing standards in  the private and public sector, and identify any gaps that should be  addressed through rulemaking, guidelines, or directives, among other  items. For a complete listing of the MOU Annex’s 11 program areas,  including TSA and PHMSA roles and responsibilities and agreed-upon  actions, see appendix I.

TSA and PHMSA Do Not Have Timeframes for Reviewing the MOU Annex to Assess Pipeline Security Roles and Responsibilities

TSA and PHMSA have both noted various developments that have  occurred since 2006 that may affect their roles and responsibilities related  to pipeline security. However, the MOU Annex has not been updated  since its inception in 2006 to consider incorporating these changes which  includes subsequently issued presidential directives, the establishment of  the Cybersecurity Infrastructure and Security Agency (CISA), and  distinctions between current TSA and PHMSA current inspection  operations. As a result, the Annex is not current and may not fully reflect  the agencies’ pipeline safety and security-related activities. For example,  Homeland Security Presidential Directive/HSPD-7 (HSPD-7), which is  cited as an underlying authority in both the 2004 MOU and 2006 MOU  Annex was revoked and replaced by PPD-21 in 2013. According to  PPD-21, the directive advances a national unity of effort to strengthen  and maintain secure, functioning, and resilient critical infrastructure by,  among other things, refining and clarifying critical infrastructure-related  functions, roles, and responsibilities across the federal government. PPD- 21 further provides, however, that plans developed pursuant to HSPD-7  shall remain in effect until specifically revoked or superseded. According  to TSA and PHMSA officials, statements of Executive Branch policy  including presidential directives such as PPD-21 include changes that  could impact their pipeline security and safety roles and should be  considered in any future revisions to the MOU Annex.

Further, PHMSA officials also told us that TSA and PHMSA’s roles and  responsibilities in identifying critical infrastructure should be reviewed  given the establishment of the CISA in November 2018. CISA, formerly  the DHS National Protection and Programs Directorate, is responsible for,  among other things, coordinating a national effort to secure and protect  against critical infrastructure risks. These responsibilities include  coordinating with sector-specific agencies to carry out its cybersecurity  and critical infrastructure activities. TSA and PHMSA officials stated that  they have closely coordinated in identifying critical infrastructure when  responding to past national emergencies. For example, TSA identified  and provided PHMSA with information on the pipelines that supplied fuel  to specific airports during the hurricane seasons in 2017 and 2018.  However, PHMSA officials stated that both TSA and PHMSA should  consider reviewing how these types of efforts may need to be coordinated  with CISA in the future and whether any adjustments to respective roles  and responsibilities in the MOU Annex are needed. In addition,  representatives from all of the industry associations that we interviewed  stated that the agreement should be revised to consider how the  establishment of CISA may impact current TSA and PHMSA pipeline  security roles and responsibilities. TSA officials stated that they do not  believe that the establishment of CISA impacts TSA’s roles and  responsibilities for identifying pipeline critical infrastructure. While CISA  may or may not have impacts on TSA and PHMSA’s pipeline security  roles, reviewing the MOU Annex in light of new developments, such as  the CISA, would allow the TSA and PHMSA to determine whether  updates are necessary.

TSA and PHMSA officials stated that distinctions in current inspections  and enforcement operations necessitate a revision to the MOU Annex.  The MOU Annex states that agencies are to explore opportunities for  collaboration in inspection and enforcement activities. According to TSA  and PHMSA officials, they have since explored the possibility for  conducting joint activities and found that distinctions in their respective  operating environments and roles and responsibilities do not allow for  joint inspection and enforcement activities. For example, PHMSA  conducts physical inspections of facilities to assess pipeline operators’  compliance with pipeline safety regulatory requirements and relies on a  range of enforcement activities, such as civil penalties to ensure that  pipeline operators correct safety violations and prevent safety problems.  TSA, however, conducts voluntary security assessments of pipeline’s  corporate security programs and critical facilities and relies on pipeline  operators’ willingness to participate and implement recommended  changes to improve pipeline security. As a result, TSA and PHMSA  officials stated that pipeline operators are reluctant to participate in a  voluntary assessment that might include PHMSA inspectors because they  represent a regulatory agency. TSA, PHMSA and industry association  representatives we interviewed agreed that the annex should be updated  to accurately reflect current distinctions in the agencies’ roles and  responsibilities and their respective operating environments.

PHMSA officials stated that they had planned to review the MOU Annex  in 2018 to assess current roles and responsibilities and determine  whether any updates to the MOU Annex were needed, but efforts were  delayed because of competing priorities such as addressing the aftermath  of major hurricanes in 2017 and 2018. Specifically, TSA and PHMSA had  agreed to an initial list of timeframes for reviewing the MOU Annex and  these timeframes called for the agencies to complete the MOU Annex  revision in 2018. However, as of March 2019, TSA and PHMSA have yet  to complete the review and although both agencies stated that the review  is ongoing, neither agency could provide updated timeframes for  completion. Furthermore, while the Annex recognizes that TSA and  PHMSA may propose agreed-upon amendments or modifications to the  agreement, it does not call for regular or periodic reviews to identify  whether any updates or revisions are needed and, as appropriate,  implemented.

TSA and PHMSA officials, as well as the industry association  representatives we interviewed all reported that the MOU Annex helped  to coordinate pipeline security and safety efforts because: (1) it is a  signed written agreement that can be readily consulted; (2) it  memorialized respective TSA and PHMSA roles and responsibilities for  government leaders and staff at the time; and (3) it can be modified or  amended as needed.

Standards for Internal Control in the Federal Government states that  periodic review of policies, procedures, and related control activities  should occur to determine their continued relevance and effectiveness in  achieving identified objectives or addressing related risks. In addition,  documentation of any changes made as a result of such reviews, such as  changes to an entity’s roles and responsibilities or in technology, should  occur to ensure that such controls are clear over time as staff change  within an organization. Standards for project management state that  managing a project involves, among other things, developing a timeline  with milestone dates to identify points throughout the project to reassess  efforts under way to determine whether project changes are necessary.  By developing and implementing mutually agreed upon time frames for  reviewing the annex and updating it, as appropriate, TSA and PHMSA  could better ensure that the roles and responsibilities for TSA and  PHMSA remain current. Additionally, including a provision in the annex  for periodically reviewing for needed updates would help ensure the  agreement consistently reflects relevant and updated information on TSA  and PHMSA’s roles and responsibilities.

TSA and PHMSA Communicate Their Roles through Guidelines and Other Methods, and Selected Industry Stakeholders Reported the Agencies’ Roles Are Clear

TSA and PHMSA have communicated their respective pipeline safety and  security roles and responsibilities by issuing pipeline security guidance  and safety regulations, issuing a joint advisory bulletin, and maintaining  informal contacts with pipeline stakeholders when conducting outreach  activities, pipeline security assessments, or safety inspections.

TSA security guidelines. TSA’s Pipeline Security Branch first issued  its voluntary Pipeline Security Guidelines in 2011, and revised them in  March 2018. The guidelines include TSA’s recommendations for  pipeline industry security practices, such as establishing a corporate  security program, conducting security vulnerability assessments, and  identifying critical facilities. The guidelines also recommend facility  security and cybersecurity measures, which serve as the basis for the  pipeline security assessments conducted by TSA’s Pipeline Security  Branch.

PHMSA regulations. PHMSA’s Office of Pipeline Safety issues and  enforces intrastate and interstate regulations covering aspects of  pipeline safety, including the design, construction, operation and  maintenance, and spill response for hazardous liquid and gas pipeline  facilities, including liquefied natural gas facilities.

Advisory bulletins. PHMSA also issues advisory bulletins to  communicate safety-related conditions to pipeline operators, and can  issue advisory bulletins in coordination with TSA to notify pipeline  operators of a security incident. Such bulletins may include identifying  the affected operators, describing the threat, and providing  information on federal resources for assistance. For example, in  response to physical intrusions on pipelines and a coordinated  campaign by domestic saboteurs, and to remind pipeline operators of  the importance of safeguarding and securing their pipelines from  physical and cyber intrusion or attack, PHMSA, in coordination with  TSA, issued an advisory bulletin in 2016. The bulletin also included  a brief discussion of TSA’s and PHMSA’s roles on pipeline safety and  security.

Forums and routine interactions with operators. TSA and PHMSA  officials also reported that they communicate their agencies’  respective roles and responsibilities for pipeline safety and security to  stakeholders when conducting general outreach, information sharing  efforts, or inspections or assessments. TSA and PHMSA officials  noted that these activities provide opportunities for agency officials  and pipeline stakeholders to clarify their roles and responsibilities  should pipeline operators have questions. Examples of such  community outreach activities include attending meetings of the Oil  and Natural Gas subsector SCC or the Pipeline Modal SCC, and  TSA’s annual International Pipeline Security Forum. TSA officials  also said that TSA’s monthly and quarterly unclassified threat  briefings provided TSA officials and pipeline stakeholders the  opportunity to discuss and clarify their roles and responsibilities.  Additionally, TSA produces classified and unclassified threat  assessments on physical and cyber threats to pipelines, which  according to agency officials can help to clarify TSA’s security role.  Finally, TSA and PHMSA officials said that pipeline security  assessments and safety inspections and other enforcement activities  that the agencies regularly conduct are also opportunities to  communicate their roles and responsibilities. For example, TSA  officials reported that should an operator ask for assistance regarding  a safety issue while TSA staff was conducting a security review, TSA  staff would be able to refer the operator to PHMSA to address the  issue. Similarly, PHMSA officials stated that inspectors would refer an  operator to TSA or its pipeline security guidelines should the operator  have questions regarding, for example, what security measures to  implement.

The representatives of the four pipeline associations we interviewed  reported that TSA and PHMSA had clearly communicated their respective  roles and responsibilities to pipeline stakeholders. Specifically, all of the  association representatives said that their membership understood that  TSA is responsible for pipeline security matters and PHMSA is  responsible for pipeline safety matters. For example, one industry  association representative stated that they had contacted their members  to determine whether they were unclear regarding TSA’s and PHMSA’s  respective roles and responsibilities and that members reported the roles  were clear to them. Further, another association representative reported  that the initial security reviews and outreach efforts that TSA conducted  after the pipeline security program was created helped pipeline operators  to understand that its role was to oversee pipeline security. In addition,  all of the association representatives we interviewed stated that the MOU  Annex helped ensure that TSA and PHMSA understood and respected  each other’s roles and responsibilities. As a result, according to the  association representatives, their pipeline operator membership had not  experienced challenges associated with overlapping or duplicative efforts  on the part of TSA and PHMSA pipeline safety or security programs.

TSA Communicated Pipeline Incident Response Protocols in Its 2010 Plan, but Has Not Updated the Plan to Address Changes in Key Areas

TSA Has Established a Pipeline Incident Response Protocol Plan That Communicates Agencies’ Roles and Responsibilities During Pipeline Incidents

In accordance with the 9/11 Commission Act, TSA issued its Pipeline  Security and Incident Recovery Protocol Plan in March 2010. The plan’s  stated intent is to establish a comprehensive interagency approach to  counter risks, coordinate federal agencies’ actions, and minimize the  consequences of incidents involving pipeline infrastructure as well as  recovery time from them. The plan also defines the roles and  responsibilities of federal agencies; tribal, state, and local governments;  and the private sector during a pipeline incident. It also defines the  measures they may take related to pipeline infrastructure security  incidents. According to the plan TSA, PHMSA, the Department of Energy  (DOE), and the Federal Bureau of Investigation (FBI) have principal roles  in pipeline incident response, while other agencies such as the U.S.  Coast Guard, the Federal Emergency Management Agency (FEMA), and  the National Transportation Safety Board (NTSB) have supporting roles.  The following are examples of agencies’ roles and responsibilities in each  of the plan’s three response phases.

Prevention/protection. TSA is responsible for monitoring pipeline  owner and operators’ implementation of its pipeline security  guidelines, and PHMSA is responsible for enforcing its pipeline safety  regulations. TSA, in addition to the FBI, is responsible for assessing  the credibility of any physical or cyber threat information it receives  and sharing any intelligence related to pipeline security with pipeline  owners and operators.

Response. TSA is responsible for coordinating information sharing  between federal agencies and pipeline stakeholders, and PHMSA is  responsible for coordinating federal agency activities with the affected  pipeline operator and state pipeline safety agency. The plan also  states that the FBI is responsible for investigating attempted or  successful attacks on pipeline infrastructure including those that are  believed to have a nexus to terrorism.

Recovery. PHMSA is primarily responsible for working with the  pipeline operator, along with other supporting federal agencies, to  facilitate service restoration. DOE is responsible for monitoring flows  of throughput in the affected pipeline system or systems, assessing  regional, national, and global impacts of an incident on energy  infrastructure throughout all three phases.

Appendix I provides more details on key federal agencies’ and pipeline  operators’ roles and responsibilities, as well as the actions they may take  in response to an incident as detailed in the plan.

TSA Has Not Updated Its Incident Response Plan to Address Changes in Pipeline Security Threats, Technology, and Federal Laws and Policies

TSA’s plan states that it will be updated periodically to address changes  in pipeline security threats, technology, and federal laws and policies.  Further, Standards for Internal Control in the Federal Government states  that periodic review of policies, procedures, and related control activities  should occur to determine their continued relevance and effectiveness in  achieving identified objectives or addressing related risks. In addition,  internal control standards also states that changes in an entity’s programs  or activities, organizational structure, personnel, or technology can affect  the operating environment and management can respond by revising  internal controls on a timely basis to ensure effectiveness. However, TSA  has not reviewed or revised its 2010 plan to ensure it addresses changes  in at least three key areas: cybersecurity-related laws and policies,  incident management policies, and DHS’s terrorism alert system as  described below.

TSA’s 2010 plan includes some discussion of cyber threats and refers  operators to guidance they may use to better secure their SCADA and  control systems. However, the plan does not identify the cybersecurity  roles and responsibilities of federal agencies that are identified in the  plan, such as DOE, Federal Energy Regulatory Commission (FERC), or  the FBI, or discuss the measures these agencies should take to prevent,  respond to, or support pipeline operators following a cyber incident  involving pipelines.

TSA’s 2010 plan also has not been updated to reflect current  cybersecurity incident response guidance. In December 2016, DHS  issued its National Cyber Incident Response Plan (NCIRP). The NCIRP  is to be the primary framework for stakeholders, including pipeline  operators, to understand how federal departments and agencies provide  resources to support response operations for a significant cyber  incident. NCIRP identifies the FBI and the National Cyber Investigative  Joint Task Force as responsible for investigating reported cyber  incidents. NCIRP also identifies the National Cybersecurity and  Communications Integration Center (NCCIC), an agency within DHS, as  responsible for providing technical assistance to affected entities, such as  pipelines, to mitigate vulnerabilities and reduce impacts of cyber  incidents. NCCIC is also to share information across the public and  private sectors to protect against similar incidents in the future.

In addition, NCIRP provides guidance detailing when and to which federal  agencies or entities the public should report a cyber incident. These  include the FBI, the National Cyber Investigative Joint Task Force, U.S.  Secret Service, and NCCIC. For example, NCIRP states that any  cybercrime—including computer intrusions or attacks, theft of trade  secrets, criminal hacking, terrorist activity, espionage, sabotage, or other  foreign intelligence activity—is to be reported to FBI field offices’ cyber  task forces. However, TSA’s plan does not include this information or  describe what measures, if any, the agencies with pipeline-related roles  and responsibilities listed in NCIRP are to take in response to a pipeline  cyber incident.

Moreover, the 2010 plan does not account for other agencies whose roles  and responsibilities are related to critical infrastructure, such as pipelines  and cybersecurity. Specifically, the plan does not account for the role of  NCCIC, which was established in 2009. In addition, TSA’s 2010 plan  does not account for CISA’s role in cyber threat response activities or  how it may affect other agencies’ roles and responsibilities for pipeline  incident response.

TSA officials acknowledged that reviewing and, as appropriate, revising  the plan would be beneficial to ensuring the plan addresses current  pipeline security threats, technology, and federal laws and policies. They  stated TSA had not updated the plan to include cybersecurity response  protocols because an overarching cybersecurity response protocol for all  critical infrastructure sectors—not just pipelines—should first be  developed. According to TSA officials, developing a pipeline cybersecurity  response protocol would require a whole-of-government approach, as  well as coordination with private sector and input from many sectors  because of the challenges and complexity of critical infrastructure  cybersecurity in general.

However, through NCIRP, DHS provided a cybersecurity response  protocol across all critical infrastructure sectors in December 2016.  Further, NCIRP states that public and private sector entities should  consider creating an operational cyber incident response plan to further  organize and coordinate their efforts in response to cyber incidents.  Therefore, TSA could potentially provide such an operational cyber  incident response plan for the pipeline sector in its plan.

TSA could also better ensure that pipeline operators understand how  federal agencies may provide support in response to a cyber incident by  periodically reviewing and, as appropriate, revising the plan to include its  cyber incident response plan. Representatives of the four pipeline  associations we interviewed told us that their membership more clearly  understood federal agencies’ roles and responsibilities related to physical  incidents than to cybersecurity. For example, for physical incidents the  representatives stated that their members clearly understood that they  are to first notify local first responders (often through the emergency 911  system) and appropriate state or federal regulators, and are to contact  either the National Response Center or TSA’s Transportation Security  Operations Center (TSOC), depending on the nature of the incident.

However, they stated that they did not believe all of their members clearly  understand that they are to report any actual or suspected cyber incidents  that could impact pipeline industrial control systems or other information  technology-based systems to the NCCIC. All of the association  representatives told us that the process for reporting a cyber incident is  less clear because, in part, of the large number of federal agencies with a  cybersecurity-related role. One of the representatives also attributed the  lack of clarity to the reorganization of NCCIC, and the establishment of  CISA. Further, all of the representatives we interviewed indicated that  clarifying the cybersecurity roles and responsibilities of DOE, Federal  Energy Regulatory Commission (FERC), and TSA would, among other  things, improve operators’ ability to appropriately report and respond to a  cyber incident.

TSA also has not updated the plan to address changes in federal incident  management and response policies that have occurred since the plan  was developed in 2010. The plan states that it is to be consistent with the  National Response Framework (NRF) and the National Incident  Management System (NIMS) incident command system procedures.  The NRF was first issued in 2008 and described the roles, responsibilities  and coordinating structures for delivering core capabilities during incident  response. According to FEMA, it revised the NRF in 2013 and 2016 to  reflect lessons learned from real world events and other experiences  since the framework was first developed. Likewise, NIMS was developed  in 2004 as a comprehensive, national approach to incident management  that was to be applicable at all jurisdictional levels and across functional  disciplines, such as law enforcement, public health, or public works.  According to FEMA, it revised NIMS in 2017 to reflect and incorporate  policy updates and lessons learned from exercises and real-world  incidents. The revision was also intended to clarify that NIMS applies to  all stakeholders with incident management roles, and to enhance  guidance on information management processes, data collection plans,  social media integration, and the use of geographic information systems.  TSA officials acknowledged the benefit of periodically reviewing, and if  necessary, revising the plan to reflect FEMA’s revisions to NIMS or the  NRF, but had not done so because of competing priorities.

TSA has also not updated the plan to address changes DHS made to its  terrorist alert system in 2011. Consistent with the 9/11 Commission Act,  the plan describes actions that federal agencies can take at each color- coded level of the Homeland Security Advisory System to ensure the  increased security of pipeline infrastructure. For example, under the  protect/prevent phase, the plan states that when there is a high risk of a  terrorist attack (i.e., red: severe condition) and threat is general and not  specific to pipelines, TSA and PHMSA are to coordinate to identify the  potential for any related or cascading events that may impact the pipeline  sector. However, if there is a specific threat to pipelines, TSA, in  collaboration with pipeline operators, is to identify any immediate  protective measures that pipeline operators are to implement. TSA is also  to ensure pipeline operators have the information necessary to implement  these measures, and, if necessary, to issue security directives.

In 2011, DHS replaced the four color-coded alert system of the Homeland  Security Advisory System with the National Terrorism Advisory System,  which has only two alert levels (elevated threat and imminent threat).  TSA issued revised protective measures that pipeline operators are to  take under either threat condition in April 2011 and March 2018.  However, TSA has not updated the plan to communicate the actions  federal agencies can take at either level of the National Terrorism  Advisory System to ensure the increased security of pipeline  infrastructure.

TSA officials acknowledged that periodically reviewing and, as  appropriate, revising the plan would help to clarify federal agencies’ roles  and responsibilities for addressing pipeline security. TSA officials reported  that they have not updated the plan since 2010 because they faced  competing priorities. However, as described earlier, TSA’s incident  response plan was developed to provide a comprehensive interagency  approach to important activities such as countering risks, coordinating  federal agencies’ actions and minimizing the consequences of incidents  involving pipeline infrastructure. Further, the plan itself states that it will be  updated periodically to address changes in pipeline security threats,  technology, and federal laws and policies.

By periodically reviewing and, as appropriate, revising its Pipeline  Security and Incident Recovery Protocol Plan, TSA could better ensure  that the plan addresses all possible and relevant threats to pipeline  systems, such as cybersecurity, and fully incorporates relevant changes,  such as those related to incident management and DHS’s terrorism alert  system. By doing so, TSA could also provide greater assurance that  federal agencies understand the actions they are to take to prevent,  respond to, or recover from a physical or cyber incident.

TSA and PHMSA share responsibility for safeguarding the nation’s  pipeline systems from catastrophic events. While the 2006 MOU Annex  delineates TSA’s and PHMSA’s mutually agreed-upon pipeline security  roles and responsibilities, it has not been reviewed since its inception to  consider pipeline security developments. By developing and  implementing a mutually agreed upon timeline with timeframes for  reviewing the annex and as appropriate, updating it, TSA and PHMSA  could better ensure that their roles and responsibilities are properly  documented and updated in a timely manner to remain current.  Furthermore, by revising the MOU Annex to include a provision for  periodically reviewing the annex for needed updates, TSA and PHMSA  could better ensure the agreement consistently reflects relevant and  updated information on their roles and responsibilities.

Similarly, TSA’s Pipeline Security and Incident Recovery Protocol Plan— which defines the roles and responsibilities of federal agencies; tribal,  state, and local governments; and the private sector for responding to a  pipeline incident—also has not been updated to reflect changes in federal  laws or policies since the plan was issued in 2010. By periodically  reviewing and, when appropriate, updating its Pipeline Security and  Incident Recovery Protocol Plan, TSA could better ensure that the plan  addresses and fully incorporates changes relevant to cybersecurity,  incident management and DHS’s terrorism alert system, among others.  By doing so, TSA could also better ensure that federal agencies’ actions  are well coordinated in response to a pipeline-related physical or cyber  incident, and that pipeline stakeholders understand federal agencies’  roles and responsibilities in preparing for, responding to, or supporting  pipeline operators to restore service after a pipeline-related physical or  cyber incident.

We are making a total of five recommendations including three to TSA  and two to PHMSA:

The TSA Administrator should work with the PHMSA Administrator to  develop and implement a timeline with milestone dates for reviewing  and, as appropriate, updating the 2006 MOU Annex.  (Recommendation 1)

The PHMSA Administrator should work with the TSA Administrator to  develop and implement a timeline with milestone dates for reviewing  and, as appropriate, updating, the 2006 MOU Annex.  (Recommendation 2)

The TSA Administrator, in consultation with the PHMSA Administrator  should revise the 2006 MOU Annex to include a provision requiring  periodic reviews of, and as appropriate, corresponding updates to the  Annex.(Recommendation 3)

The PHMSA Administrator, in consultation with the TSA Administrator  should revise the 2006 MOU Annex to include a provision requiring  periodic reviews of, and as appropriate, corresponding updates to the  Annex.(Recommendation 4)

The TSA Administrator should periodically review, and as appropriate,  update the 2010 Pipeline Security and Incident Recovery Protocol  Plan to ensure the plan reflects relevant changes in pipeline security  threats, technology, federal law and policy, and any other factors  relevant to the security of the nation’s pipeline systems.  (Recommendation 5)

We provided a draft of this report to DHS and DOT. DHS and DOT  provided written comments which are reproduced in appendices III and IV  respectively. We also provided draft excerpts of this product to the  American Petroleum Institute (API), the American Gas Association, the  Interstate Natural Gas Association of America, and the American Public  Gas Association. For those who provided technical comments, we  incorporated them as appropriate.

With regard to our first recommendation, that TSA work with the PHMSA  to develop and implement a timeline with milestone dates for reviewing  and, as appropriate, updating the 2006 MOU Annex, DHS stated that  TSA will work with PHMSA to develop and implement a timeline with  milestone dates for reviewing and updating, as appropriate, the 2006  MOU Annex.  DHS estimated that this effort would be completed by  August 31, 2019. This action, if fully implemented, should address the  intent of this recommendation.

With regard to our second recommendation, that PHMSA work with TSA  to develop and implement a timeline with milestone dates for reviewing  and, as appropriate, updating the 2006 MOU Annex, DOT concurred and  stated it would provide a detailed response within 180 days of the  issuance of this report.

With regard to our third recommendation, that TSA, in consultation with  PHMSA, revise the 2006 MOU Annex to include a provision requiring  periodic reviews of, and as appropriate, corresponding updates to the  Annex, DHS stated that TSA will, in consultation with PHMSA, revise the  2006 MOU Annex to include a provision requiring periodic reviews of, and  as appropriate, corresponding updates to the Annex. DHS estimated that  this effort would be completed by March 31, 2020. This action, if fully  implemented, should address the intent of this recommendation.

With regard to our fourth recommendation, that PHMSA, in consultation  with TSA, revise the 2006 MOU Annex to include a provision requiring  periodic reviews of, and as appropriate, corresponding updates to the  Annex, DOT concurred and stated it would provide a detailed response  within 180 days of the issuance of this report.

With regard to our fifth recommendation, that TSA periodically review,  and as appropriate, update the 2010 Pipeline Security and Incident  Recovery Protocol Plan to ensure the plan reflects relevant changes to  pipeline security threats, technology, federal law and policy, and any  other factors relevant to the security of the nation’s pipeline systems,  DHS concurred and estimated that TSA will complete its first review by  December 31, 2019. DHS further stated that it will establish a timeline for  updating the plan should the review determine that an update is  necessary. This action, if fully implemented, should address the intent of  this recommendation.

We are sending copies of this report to the appropriate congressional  committees; the Secretary of Homeland Security, Secretary of  Transportation; and other interested parties. In addition, the report will be  available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  William Russell at (202) 512-8777 or russellw@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this report. Key contributors to this report are  listed in appendix V.

Appendix I: 2006 Memorandum of Understanding (MOU) Program Areas and Accompanying Text

The Transportation Security Administration (TSA) and Pipeline and  Hazardous Materials Safety Administration (PHMSA),”the parties”,  recognize that the following program areas are important to the  development and deployment of an enhanced security strategy for the  transportation of hazardous materials by all modes, including pipeline.

Appendix II: Summary of Key Federal Agencies’ and Pipeline Operator’s Roles and Responsibilities

This appendix summarizes the roles and responsibilities of key federal  agencies as well as the actions that they may take in response to an  incident as detailed in Transportation Security Administration’s (TSA)  2010 Pipeline Security and Incident Recovery Protocol Plan. A summary  of pipeline stakeholder’s roles, responsibilities, and examples of actions  that may be taken during each incident response phase is presented  below.

Prevention/Protection. During the prevention/protection phase,  pipeline operators are to use TSA’s pipeline security guidance and the  Pipeline and Hazardous Materials Safety Administration’s (PHMSA)  safety regulations as the framework to prepare and prevent against an  incident. TSA is responsible for monitoring pipeline owners and  operators’ implementation its security guidelines, and PHMSA is  responsible for enforcing its safety regulations. The plan also states  that during this phase TSA is to assume a primary role for ensuring  federal agencies’ actions are coordinated through protective security  advisors (PSAs). In addition, the Federal Bureau of Investigation  (FBI) is responsible for assessing the credibility of a known threat,  preparing and implementing a preliminary investigative plan, and, if  necessary, disseminating public safety notifications. The Department  of Energy (DOE) is responsible for assessing and monitoring pipeline  systems for supply shortages.

The prevention/protection section of the plan also describes how  agencies are to share and assess threat information. For example, the  plan states that TSA, PHMSA, or any federal agency that receives  threat information regardless of the source, must immediately notify  the FBI. It also states that if the FBI receives intelligence about a  pipeline threat, it is to share this information with TSA. TSA is then to  notify the pipeline operator and, if necessary, provide  recommendations for additional protective measures.

Finally, the prevention/protection section of the plan defines actions  various agencies can implement during a heightened security threat  level to increase protection from a potential attack. For example,  when there is a high risk of a terrorist attack (i.e., red: severe  condition) and threat is general and not specific to pipelines, TSA and  PHMSA are to coordinate to identify the potential for any related or  cascading events that may impact the pipeline sector. If there is a  specific threat to pipelines, TSA is, in collaboration with pipeline  operators, to identify any immediate protective measures that ought to  be taken by pipeline operators, and ensure pipeline operators have  the information necessary to implement them, and, if necessary issue  security directives.

Response. According to the plan, pipeline owners or operators are to  notify local first responders and state regulators through the  emergency 911 system. After the pipeline operator has notified local  government, they are to contact the National Response Center (NRC)  if the incident results in an unintentional release or causes significant  damage. As we previously reported, pipeline operators are also  requested to report any physical security incident that is indicative of a  deliberate attempt to disrupt pipeline operations or activities that could  be considered precursors to such an attempt to TSA’s Transportation  Security Operations Center (TSOC). Once TSA has been notified of  an incident by a pipeline operator, its Pipeline Security Branch is to  monitor the incident, notify relevant federal agencies, and, if deemed  appropriate, activate the Interagency Threat Coordination Committee  (ITCC). PHMSA may also deploy on-scene pipeline inspectors and  investigators which are to among other things, coordinate federal  agencies’ activities with the affected pipeline operator and state  pipeline safety agency, provide subject matter expertise to the  incident command, and direct safe restoration of pipeline facilities and  services.

The plan also states that, during the response phase, responsibility for  investigating the incident falls to NTSB or the FBI depending on  whether the incident is determined to be the result of criminal activity.  The FBI is solely responsible for investigating any pipeline security  incident that appears to be an intentional criminal act. For example,  if the incident were suspected to be the result of terrorist attack, the  National Joint-Terrorism Task Force would conduct an investigation of  the attack, and if appropriate, with assistance from other FBI assets.  If, however, the incident resulted in fatalities, substantial property  damage, or significant injury to the environment, NTSB would have  responsibility for investigating the incident, and may issue safety  recommendations to help prevent future accidents.

Recovery. When response activities are complete, PHMSA is to have  primary responsibility for overseeing pipeline operators’ safe  restoration of service with TSA and other federal agencies serving  primarily in support roles. PHMSA, for example, is to work with the  owner/operator to facilitate restoration of service by, among other  things, providing technical oversight, advice, and guidance to  owner/operators; coordinating recovery activities with state pipeline  safety agency, and evaluate whether to a special permit is necessary  to facilitate an expedited restoration of services. Meanwhile, DOE is to  continue to assess the impacts of an incident on energy infrastructure,  and advise federal, state, tribal, and local authorities on priorities for  energy restoration, assistance, and supply.

Appendix III: Comments from the Department of Homeland Security

Appendix IV: Comments from the Department of Transportation

Appendix V: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Ben Atwater, Assistant Director  and Michael C. Lenington, Analyst-in-Charge, managed this assignment.  Nanette Barton, Eric Hauswirth, Susan Hsu, and Thomas Lombardi also  made significant contributions to this report.

More than 2.7 million miles of pipeline transport the natural gas, oil, and other hazardous liquids the nation needs. The Departments of Homeland Security and Transportation share responsibility for safeguarding these pipelines along with pipeline operators.

In 2010, DHS's Transportation Security Administration issued a plan to coordinate pipeline security incident responses among government agencies and with the private sector.

However, TSA has not updated this plan since its issuance, so it doesn't fully reflect developments in key areas, such as cybersecurity.

We recommended that TSA periodically review and update this plan.


Veterans Health Administration: Past Performance System Recommendations Have Not Been Implemented

VHA anticipates that it will provide care to more than 7 million veterans in fiscal year 2019. The majority of veterans using VHA health care services receive care in one or more of the 172 medical centers or at associated outpatient facilities. VHA collects an extensive amount of data that can be used to assess and manage the performance of medical centers. Many measures are publicly reported on VA web pages, allowing veterans the ability to compare medical centers' quality of care.

GAO was asked to assess VHA's management of medical center performance. This report examines (1) the tools VHA uses to assess medical center performance; (2) VHA's use of medical center performance information to assess medical center directors; and (3) the extent to which VHA has evaluated the effectiveness of the SAIL system.

GAO reviewed VHA policies, guidance, and performance information for medical centers and their associated directors. GAO also interviewed officials from VHA as well as from four VA medical centers, selected for variation in performance and geographic location.

Department of Veterans Affairs' (VA) Veterans Health Administration (VHA) officials told GAO they primarily use the Strategic Analytics for Improvement and Learning (SAIL) system to assess VA medical center performance. SAIL includes 27 quality measures in areas such as acute care mortality and access to care. VHA officials use SAIL to calculate and assign each medical center an annual star rating of 1 (lowest) to 5 (highest) stars as an assessment of overall quality. For the 146 medical centers that received star ratings in fiscal year 2018, the distribution of star ratings was as follows: 6 percent, 1 star; 24 percent, 2 stars; 38 percent, 3 stars; 19 percent, 4 stars; and 12 percent, 5 stars. Although the specific medical centers within each star-rating category could change from year to year, GAO found that the fiscal year 2018 star ratings for 110 of the 127 medical centers (87 percent) that received star ratings in fiscal year 2013 did not differ by more than 1 star from their fiscal year 2013 rating.

Changes in VHA Strategic Analytics for Improvement and Learning Star Ratings, Fiscal Year 2013 Compared to Fiscal Year 2018

GAO found that VHA's appraisal process for assessing medical center director performance relies heavily on medical center performance information, including SAIL. For example, the most heavily weighted appraisal element (40 percent of the overall rating) is made up entirely of medical center performance information.

SAIL was evaluated in 2014 and 2015, but VHA has not assessed the recommendations from those evaluations, or taken action on them. The evaluations, which found issues related to the validity and reliability of SAIL and its star ratings for measuring performance and fostering accountability, together included more than 40 recommendations for improving SAIL. The findings are similar to concerns expressed by officials GAO interviewed from VHA, networks, and medical centers about SAIL's effectiveness and how it is currently being used to assess medical center performance. VHA officials told GAO the findings and recommendations of the previous SAIL evaluations were not assessed because the evaluation reports were not widely distributed within VHA due to leadership turnover, as well as attention that was diverted to other concerns such as extensive wait times for medical appointments. Without ensuring that the recommendations resulting from these previous evaluations are assessed and implemented as appropriate, the identified deficiencies may not be adequately resolved, and VHA's ability to hold officials accountable for taking the necessary actions may be diminished.

GAO recommends that the Under Secretary for Health: (1) assess recommendations from previous evaluations of SAIL for implementation; and (2) implement, as appropriate, recommendations resulting from the assessment. VA concurred with GAO's recommendations and identified actions it is taking to implement them.

The Department of Veterans Affairs’ (VA) Veterans Health Administration  (VHA) anticipates that it will provide care to more than 7 million veterans  in fiscal year 2019. The majority of veterans utilizing VHA health care  services receive care in one or more of VA’s 172 medical centers or their  associated outpatient facilities. VHA collects an extensive amount of data  that can be used to assess and manage the performance of its medical  centers, including data on patient outcomes, access to care, and the  patient experience. Many measures are publicly reported and  summarized on VA web pages, allowing veterans the ability to review and  compare medical centers’ quality of care. VHA’s Strategic Analytics for  Improvement and Learning (SAIL) system consolidates, summarizes, and  provides tools for interpreting medical center performance information.  VHA designed SAIL to provide internal benchmarking of medical center  performance and to promote high quality health care delivery across its  system of regional networks and medical centers.

We and others have expressed concerns about VHA’s management of its  health care system, including VHA’s ability to effectively provide and  monitor access to quality and timely health care to veterans. These  concerns contributed to our decision to add VA health care to our High- Risk List in 2015, and to its continued inclusion in our 2017 and 2019  updates. You asked us to assess VHA’s management of network and  medical center performance as part of a broad-based management  review of VHA. This report examines:  1.  the tools VHA uses to assess and manage medical center  2.  VHA’s use of medical center performance information to assess the  performance of its network and medical center directors; and  3.  the extent to which VHA has evaluated the effectiveness of the SAIL  system.

To examine the tools VHA uses to assess and manage medical center  performance, we reviewed VHA policies and related documents that  describe performance measures and other information VHA officials use  to assess, monitor, compare, and manage performance across its  medical centers. Additionally, we interviewed officials from VHA’s Office  of Reporting, Analytics, Performance, Improvement and Deployment, who  are responsible for determining and reporting on medical center  performance. We also reviewed documents and interviewed officials from  four VA medical centers to obtain information on the tools they use to  monitor and manage performance: Nebraska-Western Iowa Health Care  System (Omaha, Neb.); New York Harbor Health Care System (New  York, N.Y.); Tennessee Valley Healthcare System (Nashville and  Murfreesboro, Tenn.); and VA Central California Health Care System  (Fresno, Calif.). We selected these medical centers for variation in  geographic location, medical center complexity level, quality (indicated by  SAIL star ratings for fiscal years 2016 and 2017), and directors’ individual  performance ratings. We also interviewed officials from the four regional  networks that oversee these four selected medical centers. Information  obtained from these selected networks and medical centers cannot be  generalized. Our scope was focused on examining the tools used to  assess medical center performance as a whole; we did not specifically  examine all tools that can be used to monitor and assess performance for  specific programs or health conditions.

To examine VHA’s use of medical center performance information to  assess the performance of its network and medical center directors, we  reviewed relevant VHA documents, including the performance plan  templates used to evaluate network and medical center directors for fiscal  years 2016 through 2018. In addition, we interviewed officials from the VA  and VHA offices that oversee human resource efforts and executive  performance management—VA’s Corporate Senior Executive Service  Management Office and VHA’s office of Workforce Management and  Consulting. We also interviewed officials from the networks and medical  centers in our review to obtain their perspectives on VHA’s performance  assessment process.

To determine the extent to which VHA has evaluated the effectiveness of  the SAIL system, we reviewed prior reports on VHA quality of care data,  including SAIL. We also interviewed officials from VHA’s Office of  Reporting, Analytics, Performance, Improvement and Deployment; Health  Information Management; Office of Internal Audit and Risk Assessment;  and the Assistant Deputy Under Secretary for Health for Integrity. In  addition, we interviewed officials from our selected networks and medical  centers to obtain their perspectives on SAIL’s effectiveness in assessing  medical center performance. We evaluated VHA’s actions in the context  of relevant federal standards for internal control.

We conducted this performance audit from October 2017 to April 2019 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

VHA began using the SAIL system in 2012 to measure, evaluate, and  benchmark the quality, efficiency, and productivity of medical centers, and  to highlight successful strategies of high-performing medical centers.  SAIL includes 29 performance measures (27 quality measures and two  measures of overall efficiency and capacity) in areas such as acute-care  mortality, access to care, and employee satisfaction. (See appendix I for  the full list of SAIL measures.) SAIL is a diagnostic tool that allows VHA to  assess medical centers’ performance relative to their peers, and  determine how much absolute improvement they have made in the past  year based on relevant clinical data. VHA publishes SAIL results quarterly  to provide information to network and medical center officials regarding  improvement opportunities at each medical center. SAIL data are also  available on VHA’s intranet site. VHA staff can view a wide range of  detailed information about their medical center, compare performance to  other medical centers, and (for those staff with medical-record-level  access) view information on patients with a particular medical condition.

Network and Medical Center Director Performance Appraisal Process

VHA conducts annual performance appraisals for all network and medical  center officials. The appraisal process begins when officials from VHA’s  office of Workforce Management and Consulting transmit a performance  plan template to the network directors. The template identifies  performance priorities and expectations for the upcoming appraisal period  and criteria to be used to measure performance outcomes and ratings for  each performance element. Network directors use the template to  develop performance plans that include targets and time frames—the  schedule of when performance targets are to be achieved during the  year—with each of the medical center directors in their network.  According to VA policy, performance plans resulting from the template  should be finalized within 30 days of the start of the appraisal period.  After expectations have been set for a medical center director, the  director, in turn, sets performance expectations for the department heads  within the medical center.

VHA Primarily Uses SAIL and Its Associated Star Ratings to Assess and Manage Medical Center Performance

VHA Primarily Uses SAIL and Its Star Ratings to Assess Medical Center Performance

VHA officials told us they primarily use the SAIL system to assess the  performance of medical centers. Specifically, VHA uses SAIL data to  calculate and assign each medical center an annual star rating of 1  (lowest) to 5 (highest) stars as an assessment of overall quality. SAIL  documentation states that the goal of the star ratings is for low-performing  medical centers to learn from the best practices of high-performing ones,  although all medical centers have the opportunity to improve. VHA  applies a weighting and calculation methodology to each of SAIL’s 27  quality measures to determine a single composite score for each medical  center. The scores are then ranked and grouped by percentile and the  associated medical centers are assigned initial star ratings based on their  relative ranking. For example, the lowest performing 10 percent of  medical centers as determined by SAIL’s 27 quality measures are  assigned a 1-star rating, and the next lowest performing 20 percent of  medical centers are assigned a 2-star rating. (See fig. 1.)

After the initial star rating is determined by SAIL measures each year,  VHA officials can make changes to the rating if a medical center meets  certain conditions. For example, SAIL documentation states that a  medical center that initially received a 5-star rating will be reduced to a 4- star rating if it has a high mortality rate. In addition, VHA officials told us  they can decide to increase a 1-star medical center’s rating to a 2-star  rating if the medical center outperforms the bottom 10 percent of U.S.  hospitals in certain criteria as measured by external systems such as the  Centers for Medicare & Medicaid Services’ Hospital Compare website.

We found that the percentage of medical centers that received a final 1- star rating ranged from 4 percent to 10 percent from fiscal years 2013  through 2018. VHA officials publish the final annual star ratings for each  medical center both internally and externally. See figure 2 for the number  of medical centers that received each final star rating for fiscal years 2013  through 2018.

Although the specific medical centers within each star-rating category  could change from year to year, we found that the fiscal year 2018 star  ratings for 110 of the 127 medical centers (87 percent) that received star  ratings in fiscal year 2013 did not differ by more than 1 star from their  fiscal year 2013 star rating. For example, eight of the 10 1-star medical  centers in fiscal year 2013 received either a 1- or 2-star rating in fiscal  year 2018. (See fig. 3.) In addition, 44 of the 127 medical centers had the  same rating in fiscal year 2018 as they did in fiscal year 2013. At the end  of the 6-year period of our review, only one medical center differed by  more than 2 stars from its fiscal year 2013 star rating, decreasing from 5  stars to 2.

VHA Uses Tools from the SAIL System to Manage Medical Center Performance

VHA officials told us they use SAIL tools on VHA’s intranet when  conducting site visits to medical centers and for other performance  management efforts. The SAIL system includes several performance  management tools that present data in greater detail than SAIL’s  quarterly data release and enable officials to identify areas for  improvement. VHA, network, and medical center officials we interviewed  mentioned three in particular:

Opportunity matrix – This matrix shows how a medical center ranks  compared to others on all SAIL performance measures based on  quarterly data. Each performance measure is labeled by quintile, with  the first quintile comprising the top 20 percent of medical centers and  the fifth quintile comprising the bottom 20 percent. Officials told us  they use this tool to focus improvement efforts by examining specific  measures for which a medical center needs improvement.

Geometric control charts – These charts, referred to as G-Charts,  allow officials to monitor on a daily basis what VHA considers to be  rare occurrences. For example, one G-Chart allows VHA to monitor  patient safety indicators that contain information on occurrences of  specific medical conditions, such as cardiac arrest, pneumonia, and  sepsis. Medical center officials can use these charts to examine the  occurrence of events over time, analyze patient-level data, and  quickly detect changes in the frequency of these events. Other events  that the charts allow VHA to monitor include inpatient complications  and deaths.

Symphony action triggers – Symphony is an online tool that tracks  over 100 performance measures daily, related to medical center  access, outcomes, and productivity, and includes an early warning  system to notify network and medical center officials of results that  require attention. Officials can use Symphony to view patient-level  information to understand the details of particular events and  determine solutions.

VHA officials also told us that they use these tools to manage medical  center performance as part of their ongoing support of lower performing  medical centers. Specifically, officials who oversee SAIL identify lower  performing medical centers using SAIL and conduct site visits as part of  VHA’s Strategic Action for Transformation initiative. This initiative utilizes  a four-tiered, escalating approach based on the severity of concern at a  medical center. In order of increasing severity, the four levels are watch,  high-risk, critical, and VA receivership. One-star medical centers are  automatically placed on the high-risk list, along with some 2-star medical  centers with decreasing performance. If performance continues to  decrease, medical centers are considered critical, and can be escalated  to VA “receivership,” at which point VHA officials may step in to correct  ongoing problems and replace network or medical center leadership  officials. As of January 2019, VHA officials told us no medical center had  entered VA receivership since the initiative began. VHA officials told us  that they may also conduct site visits or hold calls with medical center  leadership by request, although their focus is on lower performing medical  centers.

In addition to the SAIL tools, which report data on performance measures  across the entire medical center, VHA officials told us that they may also  use other data sources as part of medical center performance  management. For example, several program offices—such as primary  care, mental health, and surgery—have dashboards that track  performance and quality of care specific to those offices. In addition, VA’s  Inpatient Evaluation Center focuses on mortality data, including estimates  of expected patient mortality.

VHA’s Appraisal Process for Assessing Network and Medical Center Directors’ Performance Relies Heavily on Medical Center Performance Information

We found that VHA relies heavily on medical center performance  information to assess the performance of its network and medical center  directors. VA’s Senior Executive Service Part V. Performance Appraisal  System handbook states that directors are assessed using five appraisal  elements established by the Office of Personnel Management: (1) Results  Driven, (2) Leading People, (3) Leading Change, (4) Business Acumen,  and (5) Building Coalitions. The five elements are included in VHA’s  performance plan template, which forms the basis for network and  medical center directors’ performance plans. The handbook designates a  relative weight for each element used to calculate a director’s rating. (See  fig. 4.) The handbook states that a director is rated in each element on a  scale of level 1 to level 5, with 5 being the highest level. Each rating is  then multiplied by the weight for its corresponding element, and the  results are added to generate a summary score. According to the  handbook, the summary score is used to identify potential recipients of  pay increases and monetary awards.

The most heavily weighted appraisal element in the handbook, Results  Driven, represents 40 percent of a director’s overall performance and is  based entirely on medical center performance information. Specifically,  for fiscal year 2018, SAIL results comprised 25 percent of the overall  rating and included measures such as patient mortality, length of stay,  and readmissions. Other medical center performance information  comprised the remaining 15 percent of the overall rating. (See fig. 5.)

Medical center performance information is also used when assessing  directors’ performance across other appraisal elements. For example, in  VHA’s fiscal year 2018 performance plan template, the Leading Change  appraisal element included the implementation of suicide prevention  initiatives, using medical center performance in the SAIL mental health  domain as criteria. In addition, the Leading People element included  performance information from VA’s All Employees’ Survey, which  included medical center staff.

Although medical center performance information plays a prominent role  in the performance assessment process, VHA officials told us that there  are other considerations that may result in medical center directors  receiving a rating that is higher than that indicated by the star rating of the  medical center. For example, VHA officials told us that when calculating a  medical center director’s rating for the Results Driven element, they  consider whether the medical center’s overall performance improved or  deteriorated compared to the previous year’s performance. These  officials also stated that they take into consideration the length of a  director’s tenure, such as cases where a director started at a low- performing medical center partway through the rating year and would not  have had sufficient time to improve the medical center’s performance  from the previous year.

In our review, we also found that the release of VHA’s performance plan  template is often delayed, which can limit its effectiveness as a means of  assessing directors’ performance. Specifically, in fiscal years 2016, 2017,  and 2018, VHA released the performance plan template to network  directors in November or December, close to the end of the first quarter of  the performance appraisal period. Directors at two of the medical  centers in our review expressed frustration with the delay and not having  a full year to meet performance expectations, but directors at the two  other medical centers stated that they find the process clear and are able  to anticipate performance expectations. Officials from VHA’s office of  Workforce Management and Consulting, which sends out the template,  told us that they have been working in recent years to shorten the  template’s development and review process within VHA; however, the  delays may continue because of late changes from VA or the Office of  Personnel Management. In our December 2016 review of human  resource management practices at VHA, we also reported on delays in  the release of VHA’s performance plan template. We reported that the  delay limited medical center officials’ ability to use the template as a tool  to align expectations and performance, which is inconsistent with leading  practices on employee performance management. We recommended that  VHA accelerate its efforts to develop a modern, credible, and effective  performance management system, including the timely release of the  performance plan template. VA partially concurred with our  recommendation and has made limited progress in implementing it. As of  December 2018, this recommendation remains open and we reiterate the  need for VHA to implement it.

VHA Has Not Assessed for Implementation Previous Recommendations Made to Ensure SAIL’s Effectiveness in Assessing Medical Center Performance

Although SAIL is used in the assessment of both medical centers’ and  directors’ performance, VHA officials have not assessed and  implemented as appropriate the recommendations from previous  evaluations of the SAIL system to ensure its effectiveness. This is  inconsistent with federal standards for internal control, which state that  management should remediate identified internal control deficiencies on a  timely basis. This remediation may include assessing the results of  reviews to determine appropriate actions, and, once decisions are made,  completing and documenting corrective actions on a timely basis.

VHA officials told us that since it was established in 2012, there have  been two evaluations of SAIL:

The first evaluation was an internal review, which VHA officials told us  was completed in February 2014 and submitted to the director of  VHA’s Office of Analytics and Business Intelligence and reviewed by  the then Under Secretary for Health and Principle Deputy Under  Secretary for Health. The internal review, which had 22  recommendations, found issues related to the validity and reliability of  SAIL as a tool for measuring performance and fostering  accountability. For example, it included a recommendation that VHA  no longer use aggregate star ratings for accountability, or for  presenting medical center quality and efficiency information to  stakeholders. Rather, the recommendation called for VHA to work to  identify valid and reliable approaches for presenting this information.

The second evaluation was an external review, which VHA officials  told us was submitted to the Office of the Principal Deputy Under  Secretary for Health in April 2015. The external review included 19  recommendations for short- and long-term improvements to SAIL,  such as a recommendation to examine the potential for misclassifying  medical centers—i.e., assigning star ratings that do not reflect medical  centers’ pattern of performance on the underlying measures. The  review noted two ways such misclassification could occur: (1) two  medical centers with summary scores that are close together could  receive different star ratings, or (2) two medical centers with widely  different summary scores could receive the same star rating.

The findings of the previous SAIL evaluations are similar to concerns that  officials from the four networks and four medical centers in our review  expressed about SAIL’s effectiveness, including whether the star ratings  were an accurate reflection of medical center performance. For example, officials from one medical center told us that, because the mortality  measure has a higher weight relative to other SAIL measures, it can  amplify the importance of a small difference between medical centers.  As a result, a 1-star medical center may appear to be performing  much more poorly on this measure than it is in practice; and officials from two medical centers told us that the length-of-stay  measure may not be an accurate reflection of quality of care, as there  are valid clinical reasons why some veterans need a longer length of  stay that may not be reflected in the diagnostic and procedure codes  for that stay. Therefore, the difference in performance on the length of  stay measure between two medical centers may be the result of how  data were entered into the medical record and coded, rather than  actual differences in quality of care.

In addition, VHA officials also expressed concerns about SAIL and how it  is currently being used to assess medical center performance. For  example, VHA officials who oversee SAIL told us it was designed to be an  internal performance improvement tool, but is now also being used as a  performance accountability tool. The external review included a  recommendation that VHA consider whether the primary purpose of SAIL  is improvement or accountability, as SAIL would need to be redesigned to  do both. One VHA official told us that SAIL is being used in punitive ways  through the Strategic Action for Transformation initiative. For example, at  one medical center, officials told us that they received a letter from VHA’s  Executive in Charge about the medical center’s low performance only a  few months after its star level increased from 1 to 2 stars. Officials said  the letter warned them that medical center leadership could be removed if  performance does not improve. Medical center officials described this as  counterproductive for their improvement efforts, as it was demoralizing  while not identifying any specific areas for improvement.

VHA officials confirmed that, other than their routine reviews to determine  the need for annual adjustments to SAIL measures and other minor  adjustments to the system, they have not assessed or implemented as  appropriate the recommendations from the internal and external SAIL  evaluations. In addition, although the Under Secretary for Health received  a response to the internal review’s recommendations from an individual  program office, VHA officials told us no action was taken on the response  or to formally assess the recommendations from the internal review.  Officials noted that two reasons for the lack of action taken to assess  recommendations for implementation were leadership turnover and  attention diverted to other issues, such as concerns about extended wait  times for medical appointments at VHA medical facilities. In addition,  officials stated that the evaluations were not widely distributed within  VHA. As a result, officials we spoke with from several VHA offices were  unaware that SAIL had ever been evaluated. To address the federal  internal control standard for timely remediation of identified deficiencies,  federal agencies assign responsibility and authority for carrying out and  documenting corrective actions. VHA officials told us they did not  formally assign responsibility to an office to assess recommendations  from previous evaluations of SAIL. As a result, when the officials who  received both evaluations left VHA, there were no other individuals or  offices responsible for ensuring that recommendations were acted on.

VHA officials who oversee SAIL told us that they are planning to use the  2015 external review as part of their plans to make changes to SAIL and  its measures. However, there is no documentation available describing  the planned changes to SAIL or how those planned changes will  incorporate the results of the external review. If changes made to SAIL  run counter to the evidence, it could potentially diminish the integrity of  the system to effectively evaluate performance.

VHA primarily uses the SAIL system to assess and compare the  performance of medical centers. Veterans can also view SAIL data to  compare medical center performance when making health care decisions.  Officials from the networks and medical centers in our review expressed  concerns about how SAIL is being used and whether star ratings are an  accurate reflection of medical center quality. SAIL has been evaluated  twice, and both evaluations have found similar concerns with SAIL.  However, VHA has yet to use the results of those evaluations to address  identified concerns and make evidence-based improvements to the SAIL  system. Specifically, VHA has not taken action to ensure that officials  assess the recommendations from SAIL evaluations, document their  decisions, and implement recommendations as appropriate. If changes to  SAIL are implemented without this assessment of existing evaluations,  VHA may make changes that run counter to the evidence, potentially  diminishing the integrity of the system to effectively evaluate  performance.

We are making the following two recommendations to VA:

The Under Secretary for Health should assess recommendations from  two previous evaluations of SAIL. This assessment should include the  documentation of decisions about which recommendations to  implement and assignment of officials or offices as responsible for  implementing them. (Recommendation 1)

The Under Secretary for Health should implement, as appropriate,  recommendations resulting from the assessment of the two previous  SAIL evaluations. (Recommendation 2)

We provided VA with a draft of this report for review and comment. VA  provided written comments, which are reprinted in appendix II. In its  written comments, VA concurred with both of the report’s  recommendations, and identified actions it is taking to implement them.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of Veterans Affairs, the Under Secretary for  Health, and other interested parties. In addition, the report is available at  no charge on the GAO web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-7114 or at draperd@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix III.

Appendix I: VHA Strategic Analytics for Improvement and Learning (SAIL) Performance Measures

2    Risk adjusted complication Index  In‐Hospital risk adjusted mortality (SMR) Measure    30‐day risk adjusted mortality (SMR30)

Desired  Direction of  Measure  Lower   3    Severity adjusted average length of stay (ALOS)    %Acute admission reviews met InterQual criteria    %Acute continued stay reviews met InterQual criteria  Inpatient core measures mean percentage (ORYX)

HEDIS outpatient core measure mean percentage (chart abstract)

HEDIS outpatient core measure mean percentage (population based)  5    HCAHPS score (patient rating of hospital)

Rating of primary care provider    Rating of specialty care provider    Care Transition (inpatient)

Stress discussed (PCMH)  2    Best Places to Work score    Hospital-wide all conditions 30-day readmission rate  5    Timely Appointment, Care and Information – PCMH    Timely Appointment, Care and Information – SC    Same Day Appointment When Needed – PCMH    Call center speed in picking up calls  3    Mental health population coverage    Mental health continuity of care    Mental health experience of care  2    Stochastic frontier analysis (= 1/SFA)

The acronyms VHA used in the table are as follows: SMR=standard mortality ratio;  HEDIS=Healthcare Effectiveness Data and Information Set; HCAHPS= Hospital Consumer  Assessment of Healthcare Providers and Systems; PCMH=patient-centered medical home;  ACSC=ambulatory care sensitive conditions; SC=specialty care.

Appendix II: Comments from the Department of Veterans Affairs

Appendix III: GAO Contact and Staff Acknowledgments

Debra A. Draper, (202) 512-7114 or draperd@gao.gov.

In addition to the contact named above, Janina Austin, Assistant Director;  Sarah Harvey and Malissa G. Winograd, Analysts-in-Charge; Jennie F.  Apter; Frederick Caison; and Alexander Cattran made key contributions  to this report. Also contributing were Vikki Porter and Jennifer Whitworth.

VA Health Care: Actions Needed to Improve Oversight of Community- Based Outpatient Clinics. GAO-18-375. Washington, D.C.: Apr. 12, 2018.

Veterans Health Care: Additional Actions Could Further Improve Policy  Management. GAO-17-748. Washington, D.C.: Sept. 22, 2017.

Veterans Affairs: Improved Management Processes Are Necessary for IT  Systems That Better Support Health Care. GAO-17-384. Washington,  D.C.: June 21, 2017.

VA Health Care: Improvements Needed in Data and Monitoring of Clinical  Productivity and Efficiency. GAO-17-480. Washington, D.C.: May 24,  2017.

Veterans Health Administration: Management Attention Is Needed to  Address Systemic, Long-standing Human Capital Challenges.  GAO-17-30. Washington D.C.: Dec. 23, 2016.

Veterans Health Care: Improvements Needed in Operationalizing  Strategic Goals and Objectives. GAO-17-50. Washington D.C.: Oct. 21,  2016.

VA Health Care: Processes to Evaluate, Implement, and Monitor  Organizational Structure Changes Needed. GAO-16-803. Washington  D.C.: Sept. 27, 2016.

The Veterans Health Administration ranks medical centers on overall quality from 1 star to 5 stars. We found that these rankings are based on a performance measurement system that may not effectively measure quality of care.

Two previous reviews of VHA's system—one internal VHA review and one review from a private consulting organization—recommended that VHA make significant changes to it, but it has neither assessed those recommendations nor implemented them. In this report, we recommended that VHA use those previous reviews to improve their system.

VA health care has been on our High Risk List since 2015.