Data Center Optimization: Agencies Report Progress, but Oversight and Cybersecurity Risks Need to Be Addressed

In December 2014, Congress enacted federal IT acquisition reform legislation that included provisions related to ongoing federal data center consolidation efforts. OMB's Federal Chief Information Officer launched DCOI to build on prior data center consolidation efforts; improve federal data centers' performance; and establish goals for inventory closures, cost savings and avoidances, and optimization performance.

The 2014 legislation included a provision for GAO to annually review agencies' data center inventories and strategies. This report addresses (1) agencies' progress and plans for data center closures and savings; and (2) agencies' progress against OMB's June 2019 revised data center optimization metrics. To do so, GAO assessed the 24 DCOI agencies' data center inventories as of August 2019, reviewed their reported cost savings documentation, evaluated their data center optimization strategic plans, and assessed their progress against OMB's established optimization targets. GAO also compared OMB's revised metrics to key characteristics of an effective performance measure.

The 24 agencies participating in the Office of Management and Budget's (OMB) Data Center Optimization Initiative (DCOI) reported progress toward achieving OMB's fiscal year 2019 goals for closing unneeded data centers. As of August 2019, 23 of the 24 reported that they had met, or planned to meet, their fiscal year closure goals, and would close 286 facilities in doing so (see figure). Agencies also reported plans to close at least 37 of the remaining data centers.

OMB issued revised guidance in June 2019 that narrowed the scope of the type of facilities that would be defined as a data center. This revision eliminated the reporting of over 2,000 facilities government-wide. OMB had previously cited cybersecurity risks for these types of facilities. Without a requirement to report on these, important visibility is diminished, including oversight of security risks.

The 24 DCOI agencies have reported a total of $4.7 billion in cost savings from fiscal years 2012 through 2019. Of the 24 agencies, 23 reported in August 2019 they had met, or planned to meet, OMB's fiscal year 2019 savings goal of $241.5 million. One agency did not complete a plan, but planned to do so in the future. Agencies also reported plans to save about $264 million in fiscal year 2020.

The 24 agencies reported progress against OMB's three revised data center optimization metrics for virtualization, advanced energy monitoring, and server utilization. For a new fourth metric (availability), the data were not sufficiently reliable to report on because of unexpected variances in the information reported by the agencies. As of August 2019, eight agencies reported that they met all three targets for the metrics GAO reviewed, five met two targets, and six met one target. In addition, one agency had not established any targets, and four agencies reported that they no longer owned any data centers.

While the three revised metrics' definitions included the key characteristics of being clearly defined and objective, none included statistical universe parameters that enable determinations of progress. Specifically, these metrics call for counts of the actual numbers of (1) virtualized servers, (2) data centers with advanced energy metering, and (3) underutilized servers; but the metrics did not include a count of the universe of all servers and all data centers. Accordingly, percentages cannot be calculated to determine progress–for example, the number of virtualized servers may increase, but if the universe of servers increases at a higher rate, then progress would actually be negative.

To improve DCOI reporting and performance, GAO is making four recommendations to OMB, and four to three selected agencies. The three agencies agreed with the recommendations while OMB did not state whether it agreed or disagreed. GAO continues to maintain that the four recommendations to OMB are warranted.

As federal agencies have modernized their operations, put more of their  services online, and improved their information security profiles, their  need for computing power and data storage resources has grown.  Accordingly, this growing demand has led to a dramatic increase in the  number of federal data centers and a corresponding increase in the costs  for their operation.

To reduce data center duplication and costs, the Office of Management  and Budget’s (OMB) Federal Chief Information Officer (CIO) launched  two initiatives. The first initiative, started in 2010, was the Federal Data  Center Consolidation Initiative (FDCCI), which aimed to reduce the  number of data centers that were outdated or duplicative. The second  initiative—the Data Center Optimization Initiative (DCOI)—was  announced in August 2016 and superseded the previous initiative. DCOI  shifted the focus to optimizing agencies’ remaining data centers by  requiring, among other things, that agencies consolidate inefficient  infrastructure, optimize existing facilities, and transition to more efficient  infrastructure, such as cloud services.

Congress has recognized the importance of reforming the government- wide management of information technology (IT) and, in December 2014,  enacted Federal Information Technology Acquisition Reform provisions  (commonly referred to as FITARA) as a part of the Carl Levin and Howard  P. ‘Buck’ McKeon National Defense Authorization Act for Fiscal Year  2015. Among the requirements related to federal data center  consolidation, the act required:

Covered departments and agencies (agencies) to report annually to  OMB about federal data center inventories and strategies to achieve  consolidation, including yearly calculations of investments and cost  savings.

OMB to develop goals for the amount of planned cost savings and  optimization improvements that agencies are to achieve through  FDCCI. OMB is to make the goals publicly available and compare  progress against the goals.

In addition to these requirements, FITARA included a provision for GAO  to annually review and verify the quality and completeness of federal data  center inventories and consolidation strategies submitted by covered  agencies. This report addresses (1) agencies’ progress on data center  closures and the related savings that have been achieved, and agencies’  plans for future closures and savings and (2) agencies’ progress against  OMB’s data center optimization targets.

To review closures to date and plans for future closures, we obtained and  analyzed August 2019 data center inventory documentation from the 24  DCOI agencies. We compared information on these agencies’ completed  and planned data center closures for fiscal year 2019 to the closure  targets that the agencies identified in their DCOI strategic plans. We  determined the number of data centers that had been closed in fiscal year  2019 by counting the agencies’ reported closures in their August 2019  inventory submissions to OMB.

We identified future closures by counting any data centers that the 24  agencies reported as planned closures in their inventories, as of August  2019 through fiscal year 2023. OMB’s guidance for developing agencies’  DCOI strategic plans required agencies to report cumulative numbers for  their planned and achieved data center closures; as a result, we  calculated agencies’ fiscal year 2019 targets from the data reported in  DCOI plans.

To verify the quality, completeness, and reliability of the agencies’ data  center inventories, we compared the information on completed and  planned data center closures to similar information reported on OMB’s IT  Dashboard and in agencies’ DCOI strategic plans. We determined that  the data were sufficiently reliable to report on agencies’ consolidation  progress and planned closures.

However, we also identified changes in OMB’s guidance regarding which  data centers agencies were required to report. These changes limited the  extent to which we could compare the currently reported number of data  centers with the number of data centers that agencies reported in  previous years.

To evaluate agencies’ progress in, and plans for, achieving data center  cost savings, we reviewed August 2019 cost savings and avoidance  documentation that the 24 DCOI agencies submitted in response to  OMB’s March 2013 PortfolioStat and August 2016 and June 2019 data  center initiative memorandums. This documentation included the  agencies’ quarterly reports of cost savings and avoidances posted to their  websites and discussed in their DCOI strategic plans.

We determined the cost savings achieved by adding agencies’ reported  savings and avoidances from the start of fiscal year 2012 through August  2019, as found in the August 2019 quarterly reports posted to the  agencies’ digital services websites. We identified planned savings by  totaling the agencies’ projected savings and avoidances from fiscal years  2019 through 2020, as reported in their DCOI strategic plans.

To assess the quality, completeness, and reliability of each agency’s data  center consolidation cost savings information, we reviewed each agency’s  August 2019 quarterly cost savings report and DCOI strategic plan for  errors and missing data, such as missing cost savings information. In  addition, we compared agencies’ reported cost savings and avoidances  with data from our most recently issued report on data center  consolidation. Further, we obtained written responses from agency  officials regarding the steps they took to ensure the accuracy and  reliability of their cost savings data. In taking these steps, we determined  that the data were sufficiently reliable to report on agencies’ data center  consolidation cost savings information.

To assess agencies’ progress against the targets for OMB’s data center  optimization metrics, we obtained the September 2019 data center  optimization progress information—both planned and achieved—for 20 of  the 24 DCOI agencies, as reported on the IT Dashboard. We then  evaluated the extent to which the reported progress met the planned  optimization performance targets that OMB set for each agency.

To assess the reliability of the agencies’ information about their progress  in optimizing their data centers, as shown on OMB’s IT Dashboard, we  reviewed the information for errors or missing data and compared  agencies’ optimization progress information across multiple reporting  quarters to identify any inconsistencies in their reported progress. We  also discussed with agency officials the steps they took to ensure the  accuracy and reliability of their reported progress. We determined the  data were sufficiently complete and reliable to report on agencies’  progress information for three of the four metrics: virtualization, advanced  energy metering, and server utilization.

However, for the fourth metric—data center availability—our analysis  identified variances in how agencies reported their data. Because of  these variances and the impact they had on the reported information, we  determined that the data for the availability metric were insufficiently  reliable for us to report on agencies’ progress.

To assess whether OMB’s four optimization performance metrics met key  characteristics of an effective performance measure, we identified  appropriate principles from the GAO Standards for Internal Control in the  Federal Government (commonly referred to as the Green Book), which  describes characteristics of effective performance measures. The Green  Book provides an overall framework for establishing and maintaining an effective internal control system that includes monitoring through  performance measures. We compared each OMB optimization  performance metric, as defined in the revised DCOI guidance and  reported on OMB’s IT Dashboard, to the criteria we identified from the  Green Book to determine the extent to which each metric met each  characteristic. Appendix I provides greater details regarding our  objectives, scope, and methodology.

We conducted this performance audit from April 2019 to March 2020 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

According to OMB, federal agencies reported that they operated 432 data  centers in 1998, 2,094 in July 2010, 5,607 in August 2016, and 5,916 in  August 2018. As previously mentioned, operating such a large number  of centers has been, and continues to be, a significant cost to federal  agencies. For example, in 2007, the Environmental Protection Agency  (EPA) estimated that the annual cost for electricity to operate federal  servers and data centers across the government was about $450 million.

Further, according to the Department of Energy (Energy), a typical  government data center has 100 to 200 times the energy use intensity of  a commercial building. However, in 2009, OMB reported server  utilization rates as low as 5 percent across the federal government’s  estimated 150,000 servers. These factors contributed to OMB  recognizing the need to establish a coordinated, government-wide effort  to improve the efficiency, performance, and environmental footprint of  federal data center activities.

Subsequently, OMB launched the Federal Data Center Consolidation  Initiative in 2010 to reduce the growing number of federal data centers  and we have reported extensively on federal agencies’ efforts to  implement the initiative’s requirements. Among other things, OMB  required agencies to consolidate inefficient infrastructure, optimize  existing facilities, improve their security posture, and achieve cost  savings. For example, each agency was required to maintain a complete  inventory of all data center facilities owned, operated, or maintained by or  on its behalf, and measure progress toward defined optimization  performance metrics on a quarterly basis as part of its data center  inventory submission.

IT Acquisition Reform Law Enhanced Data Center Consolidation and Optimization Efforts

Recognizing the importance of reforming the government-wide  management of IT, Congress enacted FITARA in December 2014.  Among other things, the law required agencies to:

Submit to OMB a comprehensive inventory of the data centers owned,  operated, or maintained by or on behalf of the agency.

Submit, by the end of fiscal year 2016, a multi-year strategy to  achieve the consolidation and optimization of the agency’s data  centers. The strategy was to include performance metrics that were  consistent with the government-wide data center consolidation and  optimization metrics.

Report progress toward meeting government-wide data center  consolidation and optimization metrics on a quarterly basis to OMB’s  Administrator of the Office of Electronic Government.

In addition, according to FITARA, the Office of Electronic Government at  OMB was to:

Establish metrics applicable to the consolidation and optimization of  data centers (including server efficiency), ensure that information  related to agencies’ progress toward meeting government-wide data  center consolidation and optimization metrics was made available to  the public in a timely manner, review agencies’ inventories and  strategies to determine whether they were comprehensive and  complete, and monitor the implementation of each agency’s strategy.

Develop and make publicly available not later than December 19,  2015, a goal broken down by year for the amount of planned cost  savings and optimization improvements that were to be achieved  through the FDCCI; and, for each year thereafter until October 1,  2020, compare reported cost savings and optimization improvements  against those goals.

OMB Established DCOI to Provide Oversight of FITARA Data Center Consolidation and Optimization Requirements

In August 2016, OMB issued Memorandum M-16-19, which established  DCOI and included guidance on how to implement the data center  consolidation and optimization provisions of FITARA. The memorandum  directed each agency to develop a DCOI strategic plan that defined its  data center strategy. Among other things, this strategy was to include a  timeline for agency consolidation and optimization activities, with an  emphasis on cost savings and optimization performance benchmarks that  the agency could achieve between fiscal years 2016 and 2018. For  example, each agency was required to develop cost savings targets due  to consolidation and optimization actions and report any realized cost  savings. OMB required each agency to publicly post its DCOI strategic  plan to its agency-owned digital strategy website.

In addition, OMB’s memorandum included a series of performance  metrics in the areas of data center closures, cost savings, and  optimization progress. The guidance further noted that agency progress  was to be measured by OMB on a quarterly basis, using agencies’ data  center inventory submissions and OMB-defined closures, cost savings,  and optimization targets.

Further, the memorandum stated that OMB was to maintain a public  dashboard (the IT Dashboard) to display government-wide and agency- specific data center consolidation and optimization progress. In this  regard, OMB began including such progress information on the IT  Dashboard in August 2016.

GAO Previously Made Recommendations on Agencies’ Consolidation and Optimization Efforts

Since the enactment of FITARA in December 2014, we have reviewed  and verified the quality and completeness of each covered agency’s  inventory and DCOI strategy annually. We have also published reports  documenting the findings from each of these reviews. In addition, we  have examined and reported on agencies’ efforts to optimize their data  centers, as well as the challenges encountered and successes  achieved. As of December 2019, 75 of the 117 recommendations from  these reports had not been fully addressed. The results and  recommendations of our previous reviews are detailed in appendix II.

OMB Updated DCOI in 2019 and Revised the Definition of a Data Center

In June 2019, OMB issued a memorandum, M-19-19, that updated DCOI  and redefined a data center as a purpose-built, physically separate,  dedicated space that meets certain criteria. The memorandum also  revised the priorities for consolidating and optimizing the federal data  centers. Specifically, OMB directed agencies to focus their efforts on  their tiered data centers and to stop reporting on spaces not designed to  be data centers (i.e., non-tiered data centers) as part of their inventory.  The guidance outlined a process by which agencies could request, and  OMB would approve, that these facilities be dropped from reporting.

The guidance also noted that OMB would set agency-specific data center  closure and cost savings targets in collaboration with each agency and in  alignment with that agency’s mission and budget. In addition, OMB  described criteria for designating certain data centers as mission critical  facilities, which would be exempt from new agency-specific closure  targets. Those mission critical designations are to be assumed to be  granted unless OMB specifically overturns them.

OMB’s revised June 2019 DCOI guidance also directed agencies to stop  reporting on spaces not designed to be a data center as part of their  inventory, and to focus their efforts on their remaining purpose-built data  centers. This is a change from the previous DCOI guidance, which  required agencies to report on a much wider range of facilities.

OMB’s new memorandum also replaced the previous optimization metrics  with revised measures that focused on (1) reporting the number of  agencies’ virtualized hosts, underutilized servers, and data centers with  advanced energy metering; and (2) the percentage of time that data  centers were expected to be available to provide services. In contrast to  the previous DCOI guidance, the new memorandum did not specify  government-wide performance targets for the optimization metrics, such  as setting a target for server utilization of 65 percent for all agencies.  Instead, OMB worked with agencies to establish agency-specific targets  that were also identified in agency DCOI strategic plans and on the IT  Dashboard. In addition, the guidance described how agencies could apply  for an optimization performance exemption for data centers where typical  optimization activities (consolidation of data collection, storage, and  processing to a central location) were technically possible but increased  the response time for systems beyond a reasonable limit.

Agencies Have Continued to Close Data Centers and Achieve Cost Savings, but Oversight and Cybersecurity Risks Need to be Addressed

As in previous years, the 24 agencies participating in DCOI continued to  report progress in closing unneeded data centers and achieving related  additional cost savings. The agencies reported closing a total of 102 data  centers in fiscal year 2019, as of August 2019, and reported plans to  close an additional 184 data centers by the end of fiscal year 2019.  According to agencies’ data center inventories, almost all of the 24  agencies met or planned to meet their fiscal year 2019 closure targets. In  addition, agencies reported that their DCOI-related activities had either  achieved, or planned to achieve, the $241.5 million in total planned  savings for fiscal year 2019. However, recent OMB DCOI policy changes  will reduce the number of data centers covered by the policy and both  OMB and agencies may lose important visibility over the security risks  posed by these facilities.

Almost All 24 Agencies Met, or Planned to Meet, OMB’s Fiscal Year 2019 Targets for Data Center Closures

For fiscal year 2019, 23 of the 24 agencies reported that they met or  planned to meet their fiscal year data center closure targets, as  established under OMB’s June 2019 guidance. Of those 23 agencies:  three agencies reported that they did not have any agency-owned  data centers and had a target of zero closures; these agencies were  listed on the IT Dashboard as having completed their closure efforts;  five agencies were not expected to close any of their operating data  centers during the fiscal year, and their target was zero;

13 agencies reported meeting or exceeding their target closures by  two agencies—the Departments of Defense (Defense) and Veterans  Affairs (VA)—reported closing a number of data centers and had  additional closures planned that were expected to meet their  respective fiscal year targets.

In addition, one agency—the Office of Personnel Management (OPM)— did not submit a DCOI strategic plan and, consequently, did not report a  data center closure target.

Table 1 details, for each of the 24 agencies, the number of data centers  open at the start of fiscal year 2019, the agency’s fiscal year 2019 closure  target, the number of data centers closed, and the number planned for  closure during the remainder of the fiscal year, as of August 31, 2019.

Agencies reported a total of 102 fiscal year 2019 data center closures  through August 31, 2019, with an additional 184 planned closures by the  end of that fiscal year. Figure 1 aggregates this information to show  agencies’ overall fiscal year 2019 progress against the reported total  number of federal data centers.

In regard to the remaining data centers, as of August 2019, 12 of the 24  agencies reported plans to close 37 data centers in fiscal year 2020 and  beyond. Specifically, 10 agencies reported plans to close 31 additional  data centers in fiscal year 2020. Further, two agencies—Energy and the  Social Security Administration (SSA)—reported plans to close a total of  five data centers in 2021, and one agency—the Department of Homeland  Security (DHS)—reported plans to close one data center in 2022. Based  on our past work reviewing agencies’ DCOI strategic plans, this total  number of planned closures is likely to increase when agencies submit  their annual DCOI strategic plans in the spring of 2020.

However, the ability to track agencies’ progress against their goals is  hampered because the agencies are not reporting their planned and  achieved closures on a fiscal year basis, and in one case, the agency had  not submitted a plan. As of September 2019, neither the agencies’  strategic plans nor the IT Dashboard provided a specific breakdown of the  planned and achieved closures for each fiscal year. OMB’s guidance on  DCOI strategic plans only requires reporting cumulative numbers, and  staff in OMB’s Office of the Federal CIO confirmed that the IT Dashboard  is now intended to report agencies’ cumulative numbers of actual and  planned data center closures, rather than numbers broken out by fiscal  year. This lack of visibility into exactly how many closures the agencies  expect to achieve every fiscal year jeopardizes OMB’s and Congress’  ability to effectively oversee agencies’ data center consolidation efforts.

OMB’s Policy Changes Will Reduce Oversight of Certain Key Data Centers

In August 2016, OMB expanded its definition of a data center to include  many smaller facilities that OMB cited as consuming significant amounts  of resources. Specifically, OMB included rooms with at least one server,  providing IT-related services, and categorized data centers into two  groups: tiered (which had to meet specific characteristics defined by  OMB) and non-tiered. We previously reported that, based on this  definition, as of August 2018, the 24 agencies planned to have a total of  4,907 operating data centers at the beginning of fiscal year 2019.

However, OMB’s June 2019 revised DCOI reporting requirements further  changed the definition of a data center, including no longer requiring  agencies to report most of the facilities previously categorized as non- tiered data centers. As noted previously, OMB directed agencies to stop  reporting on spaces not designed to be data centers as part of their  inventory. As a result, agencies are no longer required to report on about  2,000 facilities, some of which are considerable in size and will continue  to operate. Based on OMB’s revised definition of a data center, agencies  revised their data center inventory counts and now reported 2,727  operating data centers at the beginning of fiscal year 2019.

Specifically, our analysis identified 20 data centers of more than 1,000  square feet that agencies had previously reported as planned for closure,  but will not be reported under the current definition. In addition, our  analysis found 260 data centers over 1,000 square feet, previously  categorized as non-tiered, that agencies plan to continue operating, but  which will no longer be reported as part of DCOI. This includes SSA,  which plans to no longer report on, but to continue operating, five data  centers that are each over 8,000 square feet. Similarly, the Department of  State (State) plans to no longer report on, but to keep operating, two  facilities that are each at least 10,000 square feet in size.

Further, many of the smaller facilities that are now exempt from DCOI  reporting represent what OMB has said in the past are the types of data  centers that should be included in DCOI because of the risks they posed.  Specifically, in its 2016 guidance memorandum, OMB stated that these  smaller facilities posed a cybersecurity risk, and consequently, identified  them as data centers that needed to be included in consolidation efforts  under DCOI. In particular, OMB called out server rooms and closets as  security risks that should be targeted for closure. However, while OMB’s  2019 guidance noted the need to address security at these locations and  encouraged agencies to continue working to consolidate and optimize  them, there is no requirement for agencies to continue to track and report  on their progress in closing these smaller facilities.

In July 2019, we found that IT systems supporting federal agencies, such  as those found in the government’s data centers, are inherently at risk.  Specifically, we reported that because these systems can be highly  complex and dynamic, technologically diverse, and often geographically  dispersed, these factors increase the difficulty of protecting their security.  Since each physical location represents a potential access point to an  agency’s interconnection with other internal and external systems and  networks, each location also poses a risk as a point of potential attack.  We also noted that IT systems are often riddled with security  vulnerabilities—both known and unknown. Cybersecurity vulnerabilities,  such as unsecured access points, can facilitate security incidents and  cyberattacks that disrupt critical operations; lead to inappropriate access  to and disclosure, modification, or destruction of sensitive information;  and threaten national security, economic well-being, and public health  and safety. Because of OMB’s decision to remove these types of data  centers from DCOI reporting, agencies may lose track of the security  vulnerabilities that these facilities present due to the consequent  reduction in overall visibility and oversight into all data centers.

In its June 2019 guidance, OMB also outlined a process by which  agencies could request, and OMB approve, that specific facilities be  removed from reporting. As part of this process, agencies were allowed to  identify data centers to be removed in one reporting period and then  actually remove them in the next, unless OMB provided a written denial  within 30 days of the original request. Similarly, agencies could request  an exemption for mission critical facilities from their closure target; that  request also allows 30 days for OMB to object to the request before an  agency should consider the request approved.

However, there is currently no documentation of OMB’s decisions on  requests to remove specific data centers from reporting, or to exempt the  data centers from closure targets because the facility is mission critical.  Although an agency’s data center inventory included fields for  documenting OMB’s decisions with regard to potential exemptions to  optimization, there is no requirement or mechanism to document OMB’s  approval that a data center could be dropped from reporting or exempt  from closure. There is also no mechanism that would allow a third party to  determine whether OMB is providing any denials within the 30 days  specified in the DCOI guidance. Staff in OMB’s Office of the Federal CIO  acknowledged that someone without access to OMB’s repository of  agencies’ data center inventories could not determine whether OMB  completed its review within the required time period.

We recognize that OMB’s data center definition and reporting revisions  are an effort to focus agency closure and optimization efforts on certain  types of facilities. However, OMB’s own past guidance has acknowledged  the security risks posed by the types of facilities that agencies can now  exclude from DCOI. While agencies are best positioned to determine  whether these locations should be closed or optimized, it is important that  these facilities, previously covered by DCOI, continue to be reported on  quarterly, regardless of whether they are subject to closure or  optimization. Further, the lack of transparency into OMB’s approval  process for removing certain facilities from reporting due to a lack of  documentation hinders its ability to understand how and why those  decisions are made. This, in turn, jeopardizes OMB’s and Congress’  ability to effectively oversee agencies’ data center consolidation and  optimization efforts.

Almost All DCOI Agencies Met, or Planned to Meet, OMB Fiscal Year 2019 Cost Savings Targets, with More Savings Planned in 2020

Since 2013, federal agencies have been required to report on data center  cost savings. In this regard, OMB provided guidance regarding how  agencies were to report cost savings and avoidances. Specifically, it  required agencies to report both data center consolidation cost savings  and avoidances, among other areas, as part of a quarterly data collection  process known as the integrated data collection.

FITARA also called for each agency to submit a multi-year strategy for  achieving the consolidation and optimization of data centers that includes  year-by-year calculations of investment and cost savings through fiscal  year 2018, which has now been extended to 2020. In addition, OMB’s  June 2019 memorandum, M-19-19, noted that agency-specific targets  would be set in collaboration with each agency and aligned to that  agency’s mission and budget.

In their fiscal year 2019 DCOI strategic plans, agencies identified a  collective goal of achieving $241.5 million in savings. As of August 2019,  the 24 DCOI participating agencies had collectively identified in their  quarterly reports to OMB a total of $202.36 million in data center-related  cost savings for fiscal year 2019, with an additional $39.14 million  expected to be realized in the remaining month of the fiscal year.  Specifically, 18 agencies reported that they had met or exceeded their  cost savings targets, including seven agencies that did not have a cost  savings target and did not report achieving any cost savings. Further, 12  agencies reported plans to achieve about $264 million in data center- related cost savings for fiscal year 2020.

Five agencies that had cost savings targets—the Departments of  Agriculture (Agriculture), Commerce (Commerce), DHS, and State; and  the National Aeronautics and Space Administration (NASA)—reported  that they had not yet met their targets, but planned to do so. Additionally,  as noted previously, OPM had not submitted its DCOI strategic plan as of  August 2019 and, therefore, did not identify cost savings targets for fiscal  year 2019 and beyond.

Table 2 provides a breakdown of each agency’s planned and achieved  cost savings for fiscal year 2019, as of August 2019, and planned savings for fiscal year 2020, according to their DCOI strategic plans and quarterly  reporting.

Agencies that did not report achieving any cost savings provided a variety  of reasons for why they had not done so. For example, officials in the  Department of Veterans Affairs’ (VA) Office of the CIO reported 12 data  center closures, but said they did not report any achieved cost savings  because the majority of those data centers were within multi-use facilities  that were still owned and maintained by the agency. However, according  to VA’s DCOI strategic plan, the agency plans to achieve cost savings in  fiscal year 2020 because it expects to stop leasing two data centers,  which is expected to reduce data center spending.

In addition, officials from three agencies—the Department of Housing and  Urban Development (HUD), the General Services Administration (GSA),  and the United States Agency for International Development (USAID)— reported that they did not have any agency-owned data centers and had  limited opportunity to achieve cost savings related to closing and  optimizing their data centers. According to OPM officials, the agency did  not have a savings target due to the lack of a fiscal year 2019 DCOI  strategic plan, which was attributed by the officials to an oversight that  resulted from changes in OPM CIO leadership at the time the plan was  due. The officials reported that the agency continued to execute on a plan  that was already in place and they did not anticipate any meaningful  changes in the agency’s DCOI strategy for 2020. The officials said they  expect OPM to submit its fiscal year 2020 strategic plan on time in April  2020.

Overall, the 24 participating DCOI agencies have reported a total of $4.7  billion in cost savings and avoidances from fiscal years 2012 through  2019. We have previously stressed that identifying and reporting the  savings resulting from agencies’ data center consolidations was an  important indicator for monitoring the progress of DCOI. Until OPM  submits a plan that identifies its cost savings targets to OMB, the  agency’s ability to plan how to achieve DCOI’s expected benefits will be  limited. In addition, until the five agencies that still expect to achieve  savings establish and meet their cost savings targets, DCOI may not  deliver the expected financial benefits.

Agencies Reported Progress against DCOI’s Revised Optimization Metrics, but Metrics Lacked Appropriate Information on Performance Parameters

FITARA required OMB to establish metrics to measure the optimization of  data centers, including server efficiency, and to ensure that agencies’  progress toward meeting those metrics is made available to the public.  Pursuant to this requirement, OMB has used several different sets of  performance measures that have changed over time. Most recently, and  as previously noted, OMB issued revised DCOI guidance in June 2019  that defined a set of three revised and one new data center optimization  metrics to replace the five previous metrics. According to the OMB  memorandum that published these changes, the current metrics were  intended to focus optimization efforts in key areas where agencies can  make meaningful improvements and achieve further cost savings through  optimization. Table 3 provides a description of the four data center  optimization metrics and how each metric is to be calculated.

According to the June 2019 revised DCOI guidance, agencies are to  focus their optimization efforts on their remaining open, agency-owned,  tiered data centers. OMB also included in the guidance its plans to work  with the agencies to set agency-specific optimization performance targets  for each fiscal year. According to staff in OMB’s Office of the Federal CIO,  these targets are to be established by fiscal year and progress toward  meeting them is expected to be provided via the IT Dashboard.

For three of OMB’s June 2019 optimization metrics, 19 of the 24 DCOI  agencies reported progress in meeting OMB’s fiscal year 2019 data  center optimization targets identified on the IT Dashboard. Specifically,  as of September 2019:

11 reported that they had met their target for virtualization,

11 reported that they had met their advanced metering target, and

18 reported that they had met their server utilization target.

Of the remaining five agencies, OPM had not submitted a DCOI strategic  plan as of September 2019 and consequently, did not have established  optimization targets or a basis to measure and report optimization  progress. The remaining four agencies—the Department of Education  (Education), HUD, GSA, and USAID—reported that they did not have any  agency-owned data centers in their inventory and, therefore, the  optimization metrics were not applicable. In addition, Justice had not  established a target for the server utilization metric and, therefore, did not  have a basis to measure and report progress. Figure 2 summarizes the  DCOI agencies’ progress in meeting each optimization target, as of  September 2019.

Of the 19 agencies with a basis to report against OMB’s optimization  targets, eight agencies—Energy, DHS, the Department of the Interior,  State, NASA, NSF, NRC, and SSA—reported meeting three targets as of  September 2019. Also, five agencies reported that they had met two  targets, and six agencies reported meeting one target.

Table 4 lists the DCOI agencies and their status on meeting their OMB  optimization performance targets.

Of the current DCOI metrics, as shown in table 4, agencies reported  greater success in meeting their agency-specific optimization targets than  we had reported in our previous reviews, as detailed in appendix II. As of  September 2019, the IT Dashboard reported that four agencies had fully  completed their overall DCOI optimization efforts for all of their data  centers and had no further work to do. The IT Dashboard further reported  that another four agencies had met their optimization targets for fiscal  year 2019.

However, eight agencies had not met their fiscal year 2019 virtualization  target. The reasons agencies provided for not meeting the target varied.  For example, officials in the Department of Agriculture’s Office of the CIO  reported that the department did not meet the virtualization target  because the closure date for one of its data centers was moved to fiscal  year 2020, which resulted in fewer virtualized hosts for 2019 under OMB’s  new definition. Additionally, although EPA did not meet its virtualization  target, its DCOI strategic plan described the agency’s intention to meet its  goals by expanding its virtualization strategy agency-wide, which would  increase the agency’s virtualization performance.

In addition, OMB required agencies to report the number of agency- owned data centers with advanced energy metering. As of September  2019, of the 19 agencies with the basis to report, eight reported that they  did not reach their target for having such metering in their data centers.  For example, officials at the Department of Veteran Affairs reported that  they did not meet their advanced energy metering target due to difficulties  in getting a contract in place to install the metering.

Further, for the new availability metric, there were unexpected variances  in how agencies reported information—thus rendering the data for this  metric unreliable. Specifically, according to OMB’s quarterly reporting  instructions, agencies were to report the number of hours, in the 3-month  reporting period, that each data center was expected to be available to  provide services. However, several agencies reported information based  on annual, instead of quarterly, calculations. In addition, Department of  Agriculture officials stated that, for one data center, they reported the total  number of availability hours for multiple instances where they provided  data center services to other agencies. Based on the various instances of  erroneous agency reporting that we identified, we determined that the  data for this metric was not sufficiently reliable for us to use.

When the problems with these data were brought to agencies’ attention,  many agreed that their reporting needed to be updated; in some cases,  the agencies updated their information, but not in time for it to be  analyzed and addressed in this report. Based on our discussions with  agencies, we will continue to monitor their progress in improving the  accuracy of their reporting for this metric through our follow-up efforts for  this report, as well as our future mandated reviews of DCOI progress.

Additionally, and as mentioned previously, Justice had not established a  target for server utilization. Officials in the department’s Justice  Management Division stated that this was due to OMB’s issuing the  revised DCOI guidance and metrics in June 2019. Once they can track  server utilization for a few reporting periods, the officials stated that the  agency will finalize its definition for underutilized severs and establish an  appropriate target for the metric.

Overall, while agencies reported more success in meeting the current  optimization metrics, most agencies did not meet all of their metric targets  for fiscal year 2019. Until these agencies take the steps necessary to  meet their optimization targets, it is unlikely that these agencies will  achieve the expected benefits of optimization and the resulting cost  savings. Given that our April 2019 report included recommendations for  all of the agencies except Commerce that missed an optimization target  to take action to meet the data center optimization metric targets  established under DCOI by OMB, we are not making new optimization- related recommendations to those agencies.

OMB’s New Optimization Metric Definitions Lack Key Characteristics of Effective Performance Measures

GAO’s Green Book provides the standards for internal control in the  federal government and an overall framework for establishing and  maintaining an effective internal control system. Such a control system  addresses, in part, the attainment of a federal entity’s objectives, which is  accomplished through monitoring specific performance measures. Such  monitoring is also expected to assess the quality of performance over  time.

In addition, the Green Book discusses the importance of clearly defining  an entity’s objectives in order to determine what is trying to be achieved  and to establish related performance measures. According to the Green  Book, the controls represented by an agency’s performance metrics  should include these key characteristics. The controls should be:

Clearly defined in measurable terms that are easily understood.

Objective and free of bias, rather than subjective.

Defined by appropriate parameters that allow for evaluating  performance.

Understood by all levels of the organization, including what is being achieved with the metric, who is primarily responsible for achieving the metric, how the metric will be achieved, and when the metric will be achieved.

Aligned with internal and external requirements, including  applicable legislation, regulations, and standards.

We found that all four of OMB’s current optimization performance metrics  met three of these five characteristics—that is, each was clearly defined,  objectively measurable, and aligned with internal and external  requirements. However, the performance metrics did not fully meet the  two other characteristics—namely they did not include appropriate  performance parameters and did not fully include all the information that  would allow them to be understood at all levels of the organization.

Table 5 provides our assessment of the extent to which the OMB metrics  aligned with the characteristics of an effective metric. In addition,  appendix III provides additional detail of our assessment of the  characteristics of each metric.

While all four of OMB’s metrics met three of the five characteristics of an  effective metric, none of the metrics addressed the fourth characteristic of  providing appropriate performance parameters. Specifically, none of the  metrics included statistical universe parameters that would enable a  determination of progress against goals. For example, the virtualization  metric requires an agency to report the number of its virtual hosts, but  does not relate that to the overall number of servers and mainframes at  the agency. As a result, the metric does not indicate whether an agency’s  reported number of virtual hosts is almost all of that agency’s servers and  mainframes, or very few. Similarly, the server utilization metric identifies  how many underutilized servers an agency has, but does not give the  context of how that relates to the agency’s total population of servers. In  both these cases, percentages cannot be calculated to determine  progress. For instance, while the number of an agency’s virtualized  servers may increase, if the universe of servers were to increase at a  higher rate, then progress would actually be negative. In the June 2019  DCOI revised guidance, OMB acknowledged removing targeted averages  for its metric targets. However, by doing so, OMB also removed important  information that provided a relative sense of the progress indicated by the  data.

Further, the lack of performance parameters in defining the metrics had  an impact on OMB’s public reporting of agencies’ progress. The IT  Dashboard displays agencies’ consolidation and progress information  through a DCOI Optimization Summary that displays data about the  number of agency data center facilities, achieved and planned closures,  achieved and planned IT cost savings, and progress of the current  performance metrics against the related targets. However, the IT  Dashboard does not provide important information, such as in which fiscal  year the targets are to be achieved and how the metric information being  reported relates to an agency’s operations. For example, the IT  Dashboard reports the number of servers and mainframes serving as  virtual hosts in agency-managed data centers, but does not provide the  total number of servers and mainframes to give the context of how well  agencies are managing the number of their virtual hosts.

Staff in OMB’s Office of the Federal CIO stated that the lack of  performance parameters for the metrics is due to OMB and the agencies  needing time to collect baseline data before making changes to the  metrics. However, until OMB addresses missing information from the  optimization metric definitions, the metrics will lack important and  meaningful information about agencies’ DCOI performance that would  assist OMB and Congress in their oversight roles. In addition, unless  OMB takes action to update the metrics’ definitions to include missing key  metric characteristics, agencies’ reporting may not provide an accurate  view of their data center optimization progress. Further, without this  information on the IT Dashboard, Congress lacks the information needed  to inform its decision making and oversight responsibilities.

Federal data center consolidation efforts have been underway since  2010, and agencies continue to report progress towards meeting their  goals for data center closures and achieving related savings. Specifically,  almost all of the 24 DCOI agencies met, or planned to meet, their goals  for data center closures in fiscal year 2019. Additionally, in fiscal year  2019, almost all of the agencies met or planned to meet their $249 million  total savings target. Agencies’ efforts in both respects have made an  important contribution to achieving the overall goals of DCOI. However,  agencies’ annual closure goals are not currently reported in their DCOI  strategic plans or tracked on the IT Dashboard, requiring us to manually  calculate those targets. Unless agencies’ annual closure goals are fully  reported and tracked, oversight of DCOI will be hampered. Further, the  six agencies without plans to meet their fiscal year data center closure or  cost savings targets will continue to be challenged to realize the full  benefits of DCOI.

As part of the 2019 changes to DCOI, OMB significantly reduced the  scope of what is considered a data center, and, in doing so, excluded  about 2,000 smaller facilities that were previously reported by agencies in  2018. While OMB previously acknowledged that these types of facilities  inefficiently consume resources and pose security risks, agencies are no  longer required to report these locations in their inventories. Further, there  is currently no documentation of OMB’s decisions on agency requests to  remove data centers from reporting, or to exempt mission critical data  centers from closure targets. By no longer reporting key facilities as part  of DCOI and by not documenting decisions on which facilities are exempt  from DCOI, oversight of agencies’ consolidation and optimization efforts  may be impaired, and agencies may remain exposed to the related  vulnerabilities.

Agencies’ progress against OMB’s three revised metrics was mixed, and,  for one new metric, agencies reported data that varied so widely, we  concluded the data for this metric were not sufficiently reliable for us to  report on. However, in comparing OMB’s four metrics against the  characteristics of an effective metric, we most notably found that none of  the metrics included appropriate performance parameters for evaluating  agencies’ progress against goals. Metrics that include more robust and  informative agency performance data can play an important role in both  achieving the optimization goals and mission of DCOI and allowing for  stronger oversight of those efforts.

In addition to reiterating our prior open recommendations to the agencies  in our review regarding their need to meet DCOI’s closure and savings  goals and optimization metrics, we are making a total of eight new  recommendations—four to OMB and four to three of the 24 agencies.  Specifically:  The Director of the Office of Management and Budget should (1) require  that agencies explicitly document annual data center closure goals in their  DCOI strategic plans and (2) track those goals on the IT Dashboard.  (Recommendation 1)

The Director of the Office of Management and Budget should require  agencies to report in their quarterly inventory submissions those facilities  previously reported as data centers, even if those facilities are not subject  to the closure and optimization requirements of DCOI. (Recommendation  2)

The Director of the Office of Management and Budget should document  OMB’s decisions on whether to approve individual data centers when  designated by agencies as either a mission critical facility or as a facility  not subject to DCOI. (Recommendation 3)

The Director of the Office of Management and Budget should take action  to address the key performance measurement characteristics missing  from the DCOI optimization metrics, as identified in this report.  (Recommendation 4)

The Secretary of Agriculture should take action to achieve its data center- related cost savings target established under DCOI by OMB.  (Recommendation 5)

The Secretary of Commerce should take action to achieve its data center- related cost savings target established under DCOI by OMB.  (Recommendation 6)

The Secretary of Commerce should take action to meet its data center  optimization metric targets established under DCOI by OMB.  (Recommendation 7)

The Administrator of the National Aeronautics and Space Administration  should take action to achieve its data center-related cost savings target  established under DCOI by OMB. (Recommendation 8)

We provided a draft of this report to OMB and the 24 agencies for their  review and comment. In response, of the seven agencies to which we  made recommendations, five agencies stated that they agreed with the  recommendations and two agencies did not state whether they agreed or  disagreed with the recommendations.

In addition, of the 18 agencies to which we did not make  recommendations, three agencies stated that they concurred with the  information presented in the report, three other agencies did not state  whether they agreed or disagreed with the report, and 12 agencies stated  that they had no comments on the report. Further, four agencies provided  technical comments on the report, which we incorporated as appropriate.

Of the agencies to which we made recommendations, five agreed with  the recommendations.

In an email, a Director for Strategic Planning, Egovernment, and  Audits in the Office of the CIO at Agriculture stated that the  department agreed with our recommendation to achieve its data  center-related cost savings target established under DCOI and that it  planned to meet the cost savings target in 2020. Agriculture also  included technical comments, which we have incorporated as  appropriate.

In written comments, Commerce agreed with our recommendations to  achieve its data center-related cost savings target established under  DCOI and to meet its data center optimization metric targets  established under DCOI by OMB. The department also described  actions that they planned to take in order to address the  recommendations. Commerce’s comments are reprinted in appendix  IV.

In written comments, DHS agreed with our recommendation to  achieve its data center-related cost savings target established under  DCOI. Further, the department stated that, in its November 2019  DCOI data submission, it reported $354.97 million in cumulative DCOI  cost savings through fiscal year 2019. Subsequent to reviewing our  draft report, the department provided documentation of the savings  claimed in their response. In reviewing this data, we confirmed that  these cumulative savings included the $33.8 million savings the  department had planned for fiscal year 2019. As a result, we consider  our recommendation to have been addressed and therefore removed  it from the final report. DHS also provided technical comments, which  we have incorporated as appropriate. DHS's comments are reprinted  in appendix V.

In written comments, NASA agreed with our recommendation to  achieve its data center-related cost savings target established under  DCOI and described actions that the agency planned to take to  address the recommendation. NASA stated that it expects to  complete these actions by March 31, 2020. NASA's comments are  reprinted in appendix VI.

In written comments, OPM agreed with our recommendation to  develop and submit to OMB a complete DCOI strategic plan.  Subsequent to reviewing our draft report, OPM informed us that the  agency had published its fiscal year 2019 plan, and that the agency  was on track to meet the OMB reporting deadline for fiscal year 2020.  We confirmed that OPM's fiscal year 2019 strategic plan was  published and publicly available through the agency's website. As a  result, we consider our recommendation to have been addressed and  therefore removed it from the final report. OPM's comments are  reprinted in appendix VII.

In addition, two agencies did not state whether they agreed or disagreed  with their recommendations.

In an email, a GAO liaison on OMB’s Ethics Team provided an  annotated copy of our draft report. In OMB’s comments in that copy of  the draft, OMB did not agree or disagree with our recommendations.  However, OMB took issue with the report’s findings that the removal  of facilities from DCOI oversight posed cybersecurity-related risks  represented by those facilities. OMB’s comments further  recommended that we remove references to cybersecurity from our  report’s title and from the body of the report.

In raising these objections, OMB’s comments stated that DCOI is  focused on consolidating and optimizing the federal data center  portfolio and that cybersecurity is not a primary driver of the initiative.  OMB added that DCOI was never designed to track or directly  address cybersecurity risks. Specifically, OMB’s comments took issue  with our finding that data centers not tracked within DCOI are at a  greater risk for a cybersecurity incident. These comments noted that  many other laws, policies, and procedures directly deal with the  cybersecurity posture of all federal IT systems, and that OMB’s DCOI  guidance does not affect the applicability of those requirements. The  comments also acknowledged that, while past DCOI guidance has  stated that the reduction of data centers may improve the  cybersecurity posture of federal agencies, this was because agency  CIOs could better allocate constrained resources across a smaller  portfolio of devices.

We agree that agencies are subject to numerous cybersecurity  requirements external to DCOI. We also agree that a reduced portfolio  of data centers may improve the cybersecurity of an agency.  However, our report focuses on OMB’s recent DCOI policy changes  that allow agencies to stop tracking and reporting on over 2,000 data  centers. In this discussion, we cite our July 2019 report which found  that, facilities such as these, represent a potential access point to an  agency’s systems and networks and pose a risk as points of potential  attack. OMB’s policy changes do not require agencies to continue to  close these points of access, nor do they yield the smaller portfolio of  devices that OMB referenced in its comments on our draft report.

Our report notes that OMB’s policy change to remove those data  centers from DCOI reporting may contribute to agencies losing track  of the security vulnerabilities that those facilities present because  DCOI has provided a mechanism for ongoing visibility and oversight  of these facilities separate from the federal government’s  cybersecurity framework. As such, we maintain our report accurately  characterizes the increased potential for cybersecurity risk that could  be posed by these now-unreported physical locations. We also affirm  that our related recommendation to OMB to require agencies to report  in their quarterly inventory submissions, those facilities previously  reported as data centers, even if those facilities are not subject to the  closure and optimization requirements of DCOI, is still appropriate.

In written comments, State did not say whether it agreed or disagreed  with our recommendation to achieve its data center-related cost  savings target established under DCOI by OMB. Subsequent to  reviewing our draft report, the department informed us of $61.1 million  in fiscal year 2019 optimization and consolidation cost savings and  avoidances, an amount in excess of its $58.9 million fiscal year 2019  target, and provided documentation to support this claim. The  department also stated that this information would be reported in the  department's annual DCOI strategic plan update in the second quarter  of fiscal year 2020. In reviewing the documentation provided by the  department, we confirmed State’s reported $61.1 million in fiscal year  2019 savings. As a result, we consider our recommendation to have  been addressed and therefore removed it from the final report. State's  comments are reprinted in appendix VIII.

Further, of the 18 agencies to which we did not make recommendations,  three agencies agreed with the information presented in the report.

Via emails, audit liaisons in the Office of the CIO at Justice, the Office  of the Assistant Secretary for Policy at Labor, and the Office of  Congressional and Legislative Affairs at VA agreed with the findings in  the draft report.

In addition, three agencies did not state whether they agreed or disagreed  with the report.

In written responses, Defense and USAID did not state whether they  agreed or disagreed with the draft report. The agencies' responses  are reprinted in appendices IX and X respectively.

In an email, an audit liaison in the OIG-GAO Audit Liaison Office at  Interior did not state whether the department agreed or disagreed with  the draft report. The department also provided technical comments,  which we have incorporated as appropriate.

Finally, 12 agencies stated that they had no comment on the report.

In written responses, HUD and SSA stated that they had no  comments on the draft report. The agencies' responses are reprinted  in appendices XI and XII respectively.

We also received emails from officials of Education, Energy, HHS,  Transportation, Treasury, EPA, GSA, NSF, NRC, and SBA, which  stated that the agencies had no comment on the report. EPA also  provided technical comments, which we have incorporated as  appropriate.

We are sending copies of this report to interested congressional  committees, the Director of OMB, the secretaries and heads of the  departments and agencies addressed in this report, and other interested  parties. In addition, the report will be available at no charge on GAO’s  website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact  me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix XIII.

Appendix I: Objectives, Scope, and Methodology

This report addresses (1) agencies’ progress on data center closures and  the related savings that have been achieved, and agencies’ plans for  future closures and savings and (2) agencies’ progress against the Office  of Management and Budget’s (OMB) data center optimization targets.

To address the first objective, for data center closures, we obtained and  analyzed August 2019 data center inventory documentation from the 24  departments and agencies (agencies) that participate in OMB’s Data  Center Optimization Initiative (DCOI). To determine data center closures  to date, we totaled their reported closures for fiscal year 2019 through  August 31, 2019, and, to identify future closures, we totaled their reported  planned closures for fiscal years 2019 through 2022. We also compared  agencies’ completed and planned closures to the planned fiscal year  2019 consolidation goals, as documented in their DCOI strategic plans.  OMB’s guidance for developing agencies’ DCOI strategic plans required  agencies to report cumulative numbers for their planned and achieved  data center closures; as a result, we calculated agencies’ fiscal year 2019  targets from the data reported in DCOI plans.

To verify the quality, completeness, and reliability of each agency’s data  center inventory, we compared information on completed and planned  data center closures to similar information reported on OMB’s IT  Dashboard—a public website that provides information on federal  agencies’ major IT investments. We also checked for missing data and  other errors, such as missing closure status information. In some cases  identified, we followed up with agency officials to obtain further  information. We determined that the data were sufficiently complete and  reliable to report on their consolidation progress and planned closures.

For cost savings and avoidances, we obtained and analyzed  documentation from the 24 DCOI agencies. This documentation is  required by OMB’s March 2013, August 2016, and June 2019  memorandums and included the agencies’ quarterly reports of cost  savings and avoidances posted to their digital services websites and their  DCOI strategic plans. To determine cost savings achieved, we totaled  agencies’ reported savings and avoidances from the start of fiscal year  2012 through August 2019, as found in the August 2019 quarterly  reports posted to the agencies’ digital services websites. To identify  future planned savings, we totaled the agencies’ projected savings and  avoidances from fiscal years 2019 through 2020, as reported in their  DCOI strategic plans.

To assess the quality, completeness, and reliability of each agency’s data  center consolidation cost savings information, we used the latest version  of each agency’s quarterly cost savings report and DCOI strategic plan as  of August 31, 2019. We also reviewed the quarterly reports and DCOI  strategic plans for errors and missing data, such as missing cost-savings  information. In addition, we compared agencies’ cost savings and  avoidances with data from our most recent data center consolidation  report. Further, we obtained written responses from agency officials  regarding the steps they took to ensure the accuracy and reliability of  their cost savings data. As a result, we determined that the data were  sufficiently complete and reliable to report on agencies’ data center  consolidation cost-savings information.

For our second objective, we analyzed the September 2019 data center  optimization progress information of the 20 DCOI agencies. This  progress information was obtained from the IT Dashboard. We then  compared the agencies’ current optimization progress information to  agencies’ fiscal year 2019 optimization targets, as documented on the IT  Dashboard.

In addition, to assess the reliability of agencies’ optimization progress  information on OMB’s IT Dashboard, we reviewed the information for  errors or missing data, such as progress information that was not  available for certain metrics. We also compared agencies’ optimization  progress information across two reporting quarters to identify any  inconsistencies in agencies’ reported progress. We also followed up with  the agencies to understand the steps they took to insure that what they  reported to OMB was accurate and reliable. We determined that the data  were sufficiently complete and reliable to report on agencies’ progress  information for virtualization, advanced energy metering, and server  utilization.

However, for the fourth metric—data center availability—our analysis  identified variances in how agencies reported their data. According to  OMB’s quarterly reporting instructions, agencies were to report the  number of hours, in the 3-month reporting period, that each data center  was expected to be available to provide services. Instead, several  agencies reported information based on annual, instead of quarterly,  calculations. In addition, Department of Agriculture officials stated that, for  one data center, they reported the total number of availability hours for  multiple instances where they provided data center services to other  agencies. Because of these variances and the impact they had on the  reported information, we determined that the availability metric data were  insufficiently reliable to report on agencies’ progress.

To assess whether OMB’s new performance metrics met key  characteristics of an effective performance measure, we adapted  principles from the Green Book that described characteristics of effective  performance measures. The Green Book provides an overall framework  for establishing and maintaining an effective internal control system that  includes monitoring through performance measures. We then compared  each OMB optimization performance metric, as defined in the revised  DCOI guidance and reported on OMB’s IT Dashboard, to the criteria we  identified from the Green Book to determine the extent to which each  metric met each characteristic.

We conducted this performance audit from April 2019 to March 2020 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: GAO Previously Made Recommendations on Agencies’ DCOI- related Efforts

Since the enactment of FITARA in December 2014, we have reviewed  and verified the quality and completeness of each covered agency’s  inventory and Data Center Optimization Initiative (DCOI) strategy  annually. Accordingly, we have published reports documenting the  findings and recommendations from each of these reviews. In addition,  we have examined and reported on agencies’ efforts to optimize their  data centers, as well as the challenges encountered and successes  achieved. As of December 2019, 75 of the 117 recommendations from  these reports had not been fully implemented.

In a report that we issued in March 2016, we noted that agencies had  reported significant data center closures—totaling more than 3,100  through fiscal year 2015—but fell short of the Office of Management and  Budget’s (OMB) fiscal year 2015 consolidation goal. Agencies also  reported significant consolidation cost savings and avoidances—totaling  about $2.8 billion through fiscal year 2015. However, we pointed out that  many agencies lacked complete cost savings goals for the next several  years despite having closures planned.

In addition, we reported that 22 agencies had made limited progress  against OMB’s fiscal year 2015 data center optimization performance  metrics, such as the utilization of data center facilities. Accordingly, we  recommended that the agencies take actions to complete their cost  savings targets and improve optimization progress. As of December  2019, 17 of the 32 recommendations from this report had yet to be fully  addressed.

In May 2017, we reported that the agencies continued to report significant  data center closures—totaling more than 4,300 through August 2016— with more than 1,200 additional centers planned for closure through fiscal  year 2019. The agencies also reported achieving about $2.3 billion in  cost savings through August 2016. However, agencies’ total planned cost  savings for fiscal years 2016 through 2018 were more than $2 billion less  than OMB’s fiscal year 2018 cost savings goal of $2.7 billion.

In addition, our May 2017 report identified weaknesses in agencies’ DCOI  strategic plans. Of the 23 agencies that had submitted their strategic  plans at the time of our review, seven had addressed all of the five  required elements of a strategic plan, as identified by OMB (such as  providing information related to data center closures and cost savings  metrics). The remaining 16 agencies that submitted their plans either  partially met or did not meet the requirements.

Given these findings, we recommended that OMB improve its oversight of  agencies’ DCOI strategic plans and their reporting of cost savings and  avoidances. We also recommended that 17 agencies complete the  missing elements in their strategic plans, and that 11 agencies ensure the  reporting of consistent cost savings and avoidance information to OMB.  As of December 2019, five of the 30 recommendations had not been fully  addressed.

In a subsequent report that we issued in August 2017, we noted that 22 of  the 24 agencies required to participate in the OMB DCOI had collectively  reported limited progress against OMB’s fiscal year 2018 performance  targets for the five optimization metrics. Specifically, for each of the five  targets, no more than five agencies reported that they had met or  exceeded that specific target.

In addition, we noted in the report that most agencies had not  implemented automated monitoring tools to measure server utilization, as  required by the end of fiscal year 2018. Specifically, four agencies  reported that they had fully implemented such tools and 18 reported that  they had not done so. Two agencies did not have a basis to report on  progress because they did not have any agency-owned data centers.

Accordingly, we recommended that OMB formally document a  requirement for agencies to include plans, as part of existing OMB  reporting mechanisms, to implement automated monitoring tools at their  agency-owned data centers. We also recommended that the 18 agencies  without fully documented plans take action within existing OMB reporting  mechanisms to complete plans describing how they intended to achieve  OMB’s requirement to implement automated monitoring tools at all  agency-owned data centers by the end of fiscal year 2018. As of  December 2019, two of the 19 recommendations had been fully  addressed.

In May 2018, we noted that the 24 agencies participating in DCOI  reported mixed progress toward achieving OMB’s goals for closing data  centers by September 2018. Over half of the agencies reported that they  had either already met, or planned to meet, all of their OMB-assigned  closure goals by the deadline. However, four agencies reported that they  did not have plans to meet all of their assigned goals and two agencies  were working with OMB to establish revised targets. With regard to  agencies’ progress in achieving cost savings, 20 agencies reported  planned and achieved savings that totaled $1.62 billion for fiscal years  2016 through 2018. However, this total was approximately $1.12 billion  less than OMB’s DCOI savings goal of $2.7 billion.

In addition, the 24 agencies continued to report limited progress against  OMB’s five data center optimization targets, with one agency meeting four  targets, one meeting three targets, six meeting either one or two targets,  and 14 meeting none of their targets. Further, as of August 2017, most  agencies were not planning to meet OMB’s fiscal year 2018 optimization  targets.

Because we had previously made a number of recommendations to OMB  and the 24 DCOI agencies to help improve the reporting of data center- related cost savings and to achieve optimization targets, we did not make  new recommendations in our May 2018 report, but indicated that we  would continue to monitor the agencies’ progress toward meeting OMB’s  DCOI goals.

Most recently, in April 2019, we reported that the 24 DCOI agencies  continued to report mixed progress toward achieving OMB’s goals for  closing data centers and realizing the associated savings by September  2018. Thirteen agencies reported that they had met, or had plans to  meet, all of their OMB-assigned closure goals by the deadline. However,  11 agencies reported that they did not have plans to meet their goals.

In addition, 16 agencies reported that they had met, or planned to meet,  their cost savings targets, for a total of $2.36 billion in cost savings for  fiscal years 2016 through 2018. This is about $0.38 billion less than  OMB’s DCOI savings goal of $2.7 billion. This shortfall is the result of five  agencies reporting less in planned cost savings and avoidances in their  DCOI strategic plans as compared to the savings targets established for  them by OMB. Three agencies did not have a cost savings target and did  not report any achieved savings.

Regarding data center optimization, the 24 agencies reported limited  progress in fiscal year 2018 against OMB’s five optimization targets. In  this regard, 12 agencies reported that they had met at least one target,  while 10 reported that they had not met any of the targets. Two agencies  stated that they did not have a basis to report on progress as they did not  own any data centers.

Further, 20 agencies did not plan to meet all of OMB’s fiscal year 2018  optimization goals. Specifically, only two agencies reported plans to meet  all applicable targets, while six reported that they did not plan to meet any  of the targets.

As a result of these findings, we recommended that 22 agencies take  actions to meet the data center closure, cost savings, and optimization  performance metrics targets, as appropriate. As of December 2019, none  of the 36 recommendations had been fully addressed.

Appendix III: Detailed Analysis of Optimization Metrics

As noted previously in this report, the Office of Management and Budget  (OMB) issued revised Data Center Optimization Initiative (DCOI)  performance metrics in June 2019 as part of its revised DCOI guidance.  According to OMB, the four current data center optimization metrics were  intended to focus targeted improvements in key areas where agencies  can make meaningful improvements and achieve further cost savings  through optimization. OMB’s intent was to avoid using averages for  metrics and instead identify metrics where agencies could demonstrate  continuous improvement beyond the performance period of the June  2019 memorandum. OMB stated this would provide a more accurate  measure of the agencies’ data center performance.

GAO published the Green Book, which provides the standards for internal  control in the federal government and an overall framework for  establishing and maintaining an effective internal control system. Such a  control system addresses, in part, the attainment of a federal entity’s  objectives, which is accomplished through monitoring specific  performance measures. Such monitoring is also expected to assess the  quality of performance over time.

In addition, the Green Book discusses the importance of clearly defining  an entity’s objectives in order to determine what is to be achieved and to  establish related performance measures. According to the Green Book,  the controls represented by an agency’s performance metrics should  include several key characteristics.

Clearly defined in measurable terms that are easily understood.

Objective and free of bias, rather than subjective.

Defined by appropriate parameters that allow for evaluating  performance.

Understood by all levels of the organization, including what is being achieved with the metric, who is primarily responsible for achieving the metric, how the metric will be achieved, and when the metric will be achieved.

Aligned with internal and external requirements, including  applicable legislation, regulations, and standards.

We compared each OMB optimization performance metric, as defined in  the revised DCOI guidance and reported on OMB’s IT Dashboard, to the  key effective metric characteristics identified in the Green Book. In  assessing each of the OMB metrics against the key characteristics, we  assigned one of three categories:

Met. The metric definition aligned with the characteristics of an  effective metric.

Partially met. The metric definition aligned with some, but not all, the  characteristics of an effective metric.

Not met. The metric definition did not align with the effective metric  characteristics.

OMB’s virtualization metric counted the number of servers and  mainframes serving as a virtual host in an agency-managed data center.  We found that the virtualization metric met three characteristics, met two  of four parts of one characteristic, and didn’t meet one. Table 6 provides  our evaluation of the extent to which this OMB metric aligns with key  characteristics of an effective metric.

OMB’s advanced energy metering metric counted the data centers with  advanced energy metering covering the majority of their floor space. We  found that the advanced energy metering metric met two characteristics,  met three of four parts of one characteristic, and did not meet two. Table  7 provides our evaluation of the extent to which this OMB metric aligned  with key characteristics of an effective metric.

OMB’s server utilization metric counts the number of underutilized  production servers in federal data centers. We found that the  underutilized servers metric met three characteristics, met two of four  parts of one characteristic, and did not meet one. Table 8 provides our  evaluation of the extent to which this OMB metric aligned with key  characteristics of an effective metric.

OMB’s data center availability metric calculated the ratio of uptime (when  the data center services were available) to unexpected downtime  (unplanned service outages) in data centers. We found that the data  center availability metric met two characteristics, met two of four parts of  one characteristic, and did not meet two. Table 9 provides our evaluation  of the extent to which the OMB metric aligned with key characteristics of  an effective metric.

Appendix IV: Comments from the Department of Commerce

Appendix V: Comments from the Department of Homeland Security

Appendix VI: Comments from the National Aeronautics and Space Administration

Appendix VII: Comments from the Office of Personnel Management

Appendix VIII: Comments from the Department of State

Appendix IX: Comments from the Department of Defense

Appendix X: Comments from the U.S. Agency for International Development

Appendix XI: Comments from the Department of Housing and Urban Development

Appendix XII: Comments from the Social Security Administration

Appendix XIII: GAO Contact and Staff Acknowledgments

In addition to the contact named above, individuals making contributions  to this report included Dave Hinchman (Assistant Director), Justin Booth  (Analyst-in-Charge), Lamis Alabed, Chris Businsky, Nancy Glover, Gina  Hoover, and Jonathan Wall.

The Office of Management and Budget has been working with federal agencies to reduce the number of outdated or duplicative federal data centers. In fiscal year 2019, agencies closed 102 centers, and planned to close 184 more.

OMB also narrowed the definition of a data center, recategorizing about 2,000 facilities and excluding them from federal reporting requirements. But many of these facilities—access points into federal IT systems—will continue operating. Each one is a potential target for cyberattacks.

We recommended that OMB require agencies to continue to report on these facilities to ensure effective cybersecurity oversight.