Federal Research: DOD's Use of Study and Analysis Centers

For decades, the government has contracted and entered into agreements to sponsor academic, nonprofit, or private organizations to operate FFRDCs. DOD military departments and other DOD components sponsor 10 FFRDCs to help develop innovative solutions to diverse national security threats. Five FFRDCs—referred to as S&A Centers—aim to provide independent analyses to support DOD policy development. Federal regulation and DOD guidance specify sponsors' oversight activities, including the establishment, use, and review of FFRDCs.

A Senate Armed Services Committee report included a provision that GAO review DOD's use of FFRDCs. This report describes, among other objectives: (1) DOD obligations (in dollars) to DOD's FFRDCs from fiscal years 2013 through 2018; (2) factors that led DOD to use S&A Centers for research; and (3) how DOD used this research. GAO analyzed obligation data for DOD's 10 FFRDCs. GAO focused further review on DOD's five S&A Centers that primarily provide studies and analysis. GAO analyzed sponsoring agreements, comprehensive reviews, and 22 S&A Center research projects selected based on factors such as obtaining a mix of project costs, and interviewed DOD and FFRDC representatives.

From fiscal years 2013 through 2018, the Department of Defense (DOD) obligated about $2 billion annually to 10 DOD-sponsored Federally Funded Research and Development Centers (FFRDC), excluding obligations related to two intelligence programs and capital equipment costs (such as antenna or radar systems). Of these obligations, roughly $400 million annually went to a subset of five FFRDCs called Study and Analysis (S&A) Centers.

Note: Obligation amounts were not adjusted for inflation and totals may be affected by rounding.

a Numbers in parentheses refer to the number of FFRDCs within each category.

DOD primarily cited strategic relationships between the sponsor (the agency responsible for the overall use of the FFRDC) and the FFRDC and the core competencies of the FFRDC as factors when sponsoring S&A Centers and initiating projects. For example:

Strategic relationships. The Army determined that an S&A Center was uniquely qualified to conduct a research project that required knowledge of defense planning scenarios, noting that awarding the project to an industry contractor would have given that contractor a competitive advantage.

Core competencies. The Center for Naval Analyses has core competencies in Navy policy, strategy, and doctrine, among other things.

S&A Centers perform hundreds of research projects annually on behalf of DOD, and DOD reported using them to inform decisions, shape guidance, and identify opportunities to improve efficiency. For example, one S&A Center's study on the causes of weapons system cost overruns found DOD needed to re-examine its assumptions when estimating program cost, schedule, and performance. DOD officials told us the study contributed to policy, process, and training updates.

To aid in its efforts to develop innovative solutions to immediate and  diverse threats to national security, the Department of Defense (DOD)  utilizes 10 Federally Funded Research and Development Centers  (FFRDC). Five of these FFRDCs are Study and Analysis (S&A) Centers  that deliver independent and objective analysis and advice to DOD in  support of policy development. As described in the Federal Acquisition  Regulation (FAR), FFRDCs meet special, long-term research or  development needs of sponsoring agencies (i.e., the agencies  responsible for the overall use of the FFRDC) in areas integral to their  missions that cannot be met as effectively by existing in-house or non- FFRDC contractor resources. DOD sponsors FFRDCs by establishing  contracts or other agreements with entities to operate, manage, and/or  administer the FFRDCs. These entities can be nonprofit, university  affiliated, or private industry organizations. FFRDCs have been in place  for decades. There is congressional interest in understanding how  FFRDCs are used for research and the value DOD obtains from the  research they perform.

The Senate report accompanying a bill for the John S. McCain National  Defense Authorization Act for Fiscal Year 2019 included a provision for us  to review DOD’s use of FFRDCs. This report describes: (1) DOD  obligations to DOD-sponsored FFRDCs from fiscal years 2013 through  2018; (2) factors that led DOD to use S&A Centers for research; (3) how  DOD uses and assesses the outcomes of S&A Center research; and (4)  DOD and S&A Center policies and practices related to conflicts of  interest.

For the purposes of reporting on DOD obligations to DOD-sponsored  FFRDCs, we included DOD’s 10 FFRDCs; for subsequent objectives, we  focused our review on DOD’s five S&A Centers, which commonly deliver  studies and analyses in the form of research reports. To determine how  much DOD obligated to DOD-sponsored FFRDCs from fiscal years 2013  through 2018, we collected information and data on fiscal years 2013  through 2018 obligations to the 10 FFRDCs. To assess the reliability of  the obligations data, we compared data provided by primary sponsors  and FFRDC representatives to DOD obligations data, checked the data  for inconsistencies, interviewed knowledgeable DOD officials and FFRDC  representatives, and obtained additional information from officials and  representatives, as necessary. Based on these steps, we determined the  data were sufficiently reliable for the purposes of providing information on  obligations to DOD’s FFRDCs.

To identify the factors that led DOD to use S&A Centers for research, we  reviewed relevant FAR provisions related to FFRDCs as well as the DOD  instruction and guidance for working with FFRDCs. We collected and  analyzed the last three comprehensive reviews for each S&A Center  FFRDC and reviewed sponsoring agreements which relate to establishing  the relationship between DOD and the FFRDC. We also selected a  nongeneralizable sample of 22 research projects across the five S&A  Centers from fiscal years 2013 to 2017 (the most recent year available at  the time of our selection) to provide illustrative examples of selected S&A  Center research projects. Our selection was based on factors such as  obtaining a mix of project costs and sources of work. For each research  project, we collected and reviewed research project deliverables (for  example, white papers or formal reports, such as studies), and relevant  contract documentation, including task orders and contract modifications,  to identify potential factors that led DOD to initiate selected research  projects. For a list of the 22 projects, see appendix I. We also interviewed  DOD primary sponsor officials and FFRDC representatives to gain  insights into the processes related to requesting a project and the factors  that led DOD to use the FFRDCs for research.

To determine how DOD uses and assesses the outcomes of S&A Center  research, we reviewed DOD guidance related to the review of FFRDCs  (including DOD’s 2011 FFRDC Management Plan and DOD Instruction  5000.77). We also reviewed comprehensive reviews and performance  evaluation questionnaires for each of the five S&A Centers. We  interviewed DOD officials and FFRDC representatives to gain insights  into how S&A Center research was used by DOD for each of the 22  selected research projects. Further, we requested and analyzed  information provided by DOD primary sponsor officials and FFRDC  representatives on S&A Center recommendations made to and  implemented by DOD.

To identify and describe DOD and S&A Center policies and practices  related to conflicts of interest, we reviewed relevant FAR provisions, DOD  instruction and guidance related to conflicts of interest, and S&A Center  corporate-wide conflict of interest policies. We also interviewed DOD  officials and S&A Center representatives about the policies and  procedures, including whether conflicts of interest had been identified at  each of the S&A Centers between fiscal years 2013 to 2017 (the most  current data available at the time we selected projects) and approaches  used to address conflicts of interest provisions in DOD policy. To enhance  our understanding of the FFRDC efforts, we obtained information on how  private entities that provide analytical studies and other consulting  services to the federal government address conflict of interest concerns  from a trade association representing some of these organizations.

We conducted this performance audit from October 2018 to December  2019 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

FFRDCs arose from partnerships between the federal government and  academic researchers and scientists during World War II. Those  partnerships were later restructured into federal research centers to retain  scientists, and they became known as FFRDCs by the mid-1960s. Since  that time, FFRDCs have continued to perform tasks including technical  studies and analyses, research and development, and systems  engineering on behalf of federal agencies, such as DOD. In sponsoring  an FFRDC, agencies draw on academic and private sector resources that  can contribute to an agency’s ability to accomplish tasks that are integral  to the mission and operation of the sponsoring agency. FFRDCs may be  operated, managed, and/or administered by a university or consortium of  universities, other nonprofit organizations, or a private industry firm as an  autonomous organization or as a separate unit of a parent organization.

As of May 2019, federal agencies sponsored a total of 42 FFRDCs, 10 of  which are sponsored by DOD. These 10 DOD-sponsored FFRDCs can  be divided into three categories:

S&A Centers: These centers deliver independent and objective  analyses and advise in core areas important to their sponsors in  support of policy development and decision-making, among other  things.

Research and Development Laboratories: These laboratories  conduct research and development, focusing on the development and  prototyping of new technologies and capabilities to meet DOD needs.  For example, these laboratories engage in research programs that  emphasize the evolution and demonstration of advanced concepts  and technology, and transfer new technology to the private sector.

Systems Engineering and Integration Centers: These centers  meet long-term technical and engineering needs to ensure complex  systems meet operational requirements. Among other things,  Systems Engineering and Integration Centers assist with testing  system performance, development and acquisition of system  hardware and software, integration of new capabilities, and the  continuous improvement of system operations and logistics. Table 1  lists the 10 DOD-sponsored FFRDCs.

As shown in table 1, each of the 10 DOD-sponsored FFRDCs is managed  by a specific military department or organization within DOD—referred to  as the FFRDC primary sponsor. More broadly, the Office of the Under  Secretary of Defense for Research and Engineering oversees and  manages DOD’s FFRDC program.

DOD’s relationships with FFRDCs are defined through sponsoring  agreements between the primary sponsor (i.e., the DOD organization  responsible for the overall use of the FFRDC) and the FFRDC parent  organization. According to the FAR and DOD instruction, sponsoring  agreements define the FFRDC’s purpose and mission and may not  exceed 5 years in duration. DOD’s instruction also states that sponsoring  agreements are to establish conditions under which DOD may award an  FFRDC contract and describe the overarching requirements for operation  of the FFRDC. For example, the DOD instruction states that sponsoring  agreements are to describe constraints on the FFRDC parent  organization that are necessary to preserve the integrity of the FFRDC,  such as provisions to prevent the occurrence or appearance of  organizational or personal conflicts of interest that may undermine the  independence, objectivity, or credibility of the FFRDCs. The DOD  instruction also states that sponsoring agreements will preclude FFRDCs  from performing commercial work. In this regard, the FAR provides that  sponsoring agreements are required to address whether or not the  FFRDC may accept work from other entities and if so, the procedures to  be followed and the limitations as to the work that can be accepted.  Further, the DOD instruction and the FAR provide that sponsoring  agreements will generally preclude FFRDCs from competing with any  organization in response to a formal request for proposals other than the  operation of the FFRDC. After the primary sponsor identifies the need for  FFRDC work, and has defined FFRDC core competencies, roles, and  responsibilities in the sponsoring agreement, the primary sponsor awards  a noncompetitive contract to the FFRDC to support the sponsor’s  research requirements, such as addressing national security issues and  systems development.

Prior to extending a contract or sponsoring agreement for an FFRDC, the  FAR requires that the primary sponsor conduct a comprehensive review  of the use and need for the FFRDC at least every 5 years. The FAR  describes elements of what the comprehensive review should include,  such as examination of the sponsor’s special technical needs and mission  requirements performed by the FFRDC and assessment of the efficiency  and effectiveness of the FFRDC in meeting the sponsor’s needs. The  FAR further requires that the head of the sponsoring agency approve  continuing or terminating sponsorship based on the results of the  comprehensive review.

FFRDCs initiate work on specific projects at the request of “work  sponsors,” or the entities that request the services of the FFRDC. Work  sponsors can be the primary sponsor of the FFRDC or another entity.  When initiating work at FFRDCs, the primary sponsor determines whether  to approve research projects for the FFRDC before projects are placed on  contract. Approval of research projects is based on the determination that  work proposed is appropriate for the FFRDC and consistent with the  FFRDC’s core competencies as documented in the sponsoring  agreement. Additionally, the primary sponsor ensures FFRDC work  efforts do not exceed available resources. Among other things, FFRDC  work sponsors identify project requirements, propose an appropriate  research design, confirm the work is appropriate and consistent with  FFRDC core competencies, identify the source of project funding, and  monitor the progress of the work to ensure FFRDC performance is  satisfactory and meeting desired requirements.

In some instances, S&A Centers serve only a specific military department  or office, while in other cases an FFRDC may serve a range of DOD  entities. For example, RAND Arroyo Center broadly supports the analytic  requirements of the Army in order to provide timely advice to help senior  Army leadership make informed policy choices. Accordingly, the RAND  Arroyo Center sponsoring agreement with the Department of the Army  provides that the scope of RAND Arroyo Center work is to support Army  sponsors throughout the Army requiring comprehensive analytical  support. In contrast, the Institute for Defense Analyses (IDA) and RAND  National Defense Research Institute serve DOD more broadly on national  security issues. For example, according to IDA’s sponsoring agreement  with DOD’s Office of the Under Secretary of Defense for Acquisition and  Sustainment, the primary mission of IDA is to assist the Office of the  Secretary of Defense and other Defense organizations in addressing  important national security issues, particularly those requiring scientific  and technical expertise.

DOD manages the overall level of FFRDC work using a metric known as  staff years of technical effort (STE), which is roughly equal to the work of  one employee working for 1 year. Congress typically sets an annual  limitation on the STE that may be funded for DOD FFRDCs to support  non-intelligence programs on behalf of the agency (hereafter, Defense  STE). Between fiscal years 2013 to 2017, Congress established an  annual ceiling of 5,750 Defense STE available to DOD, of which 1,125  could be allocated to S&A Centers. In fiscal year 2018, Congress raised  the ceiling on Defense STE to 6,030; however, the limit on S&A Centers  remained unchanged. In managing Defense STE, DOD:  consolidates annual Defense STE requirements for each fiscal year  based on projected primary sponsor requirements and submits STE  requirements to Congress; establishes Defense STE allocations for each DOD-sponsored  FFRDC and provides associated funding limitations to each primary  sponsor; monitors Defense STE usage and associated obligations; and provides an annual report to Congress at the end of each fiscal year  outlining the Defense STE funded and associated DOD funds  obligated for each FFRDC.

In addition to Defense STE, FFRDCs may support DOD intelligence  activities under the Military Intelligence Program and the National  Intelligence Program. Oversight for STE usage for these programs is  provided by the Office of the Under Secretary of Defense for Intelligence  and Office of the Director of National Intelligence, respectively. Military  Intelligence Program and National Intelligence Program STE funding may  not be used to support Defense STE requirements.

In October 2008, we reported that Congress implemented the Defense  STE ceiling during the 1990s in response to concerns that DOD was  inefficiently using its FFRDCs. In addition, we found that STE ceilings  aimed to ensure that FFRDC work was appropriate and that resources,  which were limited, were being used on DOD’s highest priorities. In  December 2018, we reported that officials in the Office of the Secretary of  Defense’s Studies and FFRDC Management Office stated that the ceiling  significantly constrains the use of DOD’s FFRDCs and that DOD  customer demand for FFRDC services is significantly greater than the  annual ceiling set by Congress. Further, officials indicated at that time  that FFRDC-related work must be deferred to later years when the limits  are reached, since there are no other legally compliant alternatives  capable of fulfilling these requirements. We did not make any  recommendations related to this issue.

Following the completion of FFRDC work, the primary sponsor, with  assistance from the work sponsor, reviews FFRDC performance in written  assessments via questionnaires. In addition, the primary sponsor  assesses FFRDC performance annually, addressing the technical quality,  responsiveness, value, and timeliness of the work performed. Some of  the information from the annual reviews may be used in support of the  comprehensive review, such as to demonstrate the efficiency and  effectiveness of the FFRDC in meeting the primary sponsor’s needs.

DOD Obligated about $3 Billion per Year to DOD-Sponsored FFRDCs from Fiscal Years 2013 through 2018

From fiscal years 2013 through 2018, total DOD obligations to the 10  DOD-sponsored FFRDCs generally increased annually from about $2.7  billion in fiscal year 2013 to approximately $3.2 billion in fiscal year 2018.  Approximately 70 percent of total annual DOD obligations to DOD- sponsored FFRDCs between these fiscal years went to support non- intelligence programs and were comprised of DOD obligations associated  with utilized Defense STE, or Defense STE obligations. Specifically, DOD  Defense STE obligations ranged from about $1.9 billion in fiscal year  2013 to $2.2 billion in fiscal year 2018, with S&A Centers representing  approximately 18 percent of these obligations. In addition to DOD  Defense STE obligations, about 30 percent of total DOD obligations to  DOD-sponsored FFRDCs between fiscal years 2013 through 2018 went  towards other FFRDC-related activities and costs, such as intelligence  program activities through the Military Intelligence Program and National  Intelligence Program and capital equipment costs. Figure 1 shows DOD  obligations by fiscal year to DOD-sponsored FFRDCs.

For fiscal years 2013 to 2018, the FFRDCs we reviewed in-depth—DOD’s  S&A Centers—collectively accounted for about 18 percent of DOD  Defense STE obligations annually, whereas Research and Development  Laboratory FFRDCs and Systems Engineering and Integration Centers  accounted for 27 and 55 percent, respectively (see figure 2).

DOD Defense STE obligations to S&A Centers rose from about $320  million in fiscal year 2013 to approximately $380 million in fiscal year  2018 totaling about $2.3 billion during this period. Within each S&A  Center, obligations remained relatively constant over the 6 years, with  obligations for some FFRDCs higher than obligations for others. For  example, on average DOD obligated about $134 million annually to IDA  between fiscal years 2013 through 2018, whereas DOD obligated  approximately $39 million annually to RAND Arroyo Center during this  timeframe.

DOD Defense STE obligations to S&A Centers were almost entirely  awarded to support research projects requested by DOD. In some cases,  work was done in response to congressional direction. For example,  RAND Project Air Force (PAF) initiated a fiscal year 2017 independent  review and assessment of the Ready Aircrew Program to respond to  requirements outlined in the National Defense Authorization Act of Fiscal  Year 2017. Overall, according to information provided by DOD sponsors  and FFRDC representatives, between fiscal years 2013 through 2017,  S&A Centers began work on about 600 research projects annually on  behalf of DOD, with about 93 percent of these projects initiated at the  request of DOD. The dollar value of these S&A Center projects ranged  from about $2,000 to $11 million between fiscal years 2013 through  2017.

DOD Reported It Primarily Considered Strategic Relationships and FFRDC Core Competencies When Sponsoring S&A Centers and Initiating Projects

Sponsoring agreements note and primary sponsors reported in  comprehensive reviews that S&A Centers are utilized because of DOD’s  strategic relationships with FFRDCs. As described in the FAR, FFRDCs  meet special, long-term research or development needs of the sponsoring  agencies. Sponsoring agreements with S&A Centers outline the  importance of strategic relationships that have helped these FFRDCs to  develop and maintain in-depth knowledge of their sponsors’ and users’  programs and operations.

In our review of S&A Center sponsoring agreements and comprehensive  reviews, we identified that strategic relationships between sponsors and  S&A Centers are generally characterized by the stability of long-term  capabilities in subject areas important to DOD, access to sensitive and  proprietary data and information, and objectivity in the form of freedom  from conflicts of interest. These documents also indicate that strategic  relationships enable S&A Centers to maintain in-depth knowledge of work  sponsor programs and operations. For example, in the 2015 sponsoring  agreement between the Army and RAND Arroyo Center, the sponsoring  agreement states that both the Army and RAND Arroyo Center share a  strategic relationship, and that the RAND Arroyo Center is structured to  maintain strong analytic expertise related to Army policy and operations.  In addition, the sponsoring agreement outlines the importance of RAND  Arroyo Center’s continuity of expertise to the Army, long-term research  efforts, and high-quality staff.

Office of the Under Secretary of Defense for Acquisition and Sustainment  (OUSD(A&S)) officials told us that S&A Centers are oftentimes chosen to  perform work for DOD due to unique long-term strategic relationships with  sponsors for independent and knowledgeable expertise within core  competencies to address sponsors’ specific analytic requirements. In  some cases, these strategic relationships date back to World War II.  Regarding these strategic relationships, OUSD(A&S) officials also told us  the primary sponsor has a degree of control over an FFRDC’s business  affairs that can limit the risks of organizational conflicts of interest at  FFRDCs.

DOD also cited strategic relationships between DOD and S&A Centers as  a reason for using S&A Centers when initiating projects we reviewed. For  example:

Prior to initiating a 2016 assessment of the impact of long-term fiscal  trends on Army capabilities, the Army determined RAND Arroyo  Center was uniquely qualified to conduct the research because the  project required knowledge of defense planning scenarios that would  have given an industry contractor a competitive advantage, potentially  leading to a conflict of interest. The Army also identified RAND Arroyo  Center’s long-standing expertise on security cooperation when  requesting a fiscal year 2013 study on assessing value in Army  security cooperation as a reason RAND Arroyo Center was uniquely  suited to complete the study.

Navy primary sponsor officials identified the long-term relationship  between CNA, the FFRDC, and the Navy, which has led to broad  subject-matter expertise in naval matters, as a reason they used CNA  for the fiscal year 2016 study on the assessment of the effects of  possible policy changes to a career track program for military officers  trained to work with other military services. CNA leadership chose two  researchers to lead the effort, one of which had prior experience in  this area.

An OUSD(A&S) official cited RAND National Defense Research  Institute’s (NDRI) longstanding portfolio on military workforce issues  as a reason for using RAND NDRI for a fiscal year 2017 study on the  military’s 40-year pay table. An official told us that RAND NDRI’s prior  work in this area would allow for a quicker response and more in- depth analysis to respond to the work request.

In addition to the strategic relationships, sponsoring agreements and  comprehensive reviews cited FFRDC core competencies as key factors in  establishing and continuing relationships with S&A Centers, which is  consistent with provisions outlined in DOD Instruction 5000.77. The DOD  instruction states that FFRDCs maintain long-term competencies and  capabilities to meet DOD needs that cannot be met by government or  other private sector resources as effectively, and these competencies  derive from the sponsor’s analytical requirements. In general, core  competencies include expertise in engineering, research and  development, and analysis, and are further described in FFRDC  sponsoring agreements and comprehensive reviews. For example:

The Navy 2015 comprehensive review of CNA states that CNA  satisfies the Navy’s need for highly specialized skills and  competencies in Navy warfighting and warfighting support— particularly research staff from CNA’s studies and analyses division— to accomplish their operational missions.

The 2019 sponsoring agreement between DOD’s OUSD(A&S) and  IDA outlined the need for technical and analytical support, citing IDA’s  four core competencies as the scope of work of the FFRDC: systems  and capabilities evaluations, technology assessments, force and  strategy assessments, and resource and support analyses.

The Army 2010 and 2014 comprehensive reviews of RAND Arroyo  Center stated that RAND Arroyo Center has currency in all requisite  Army proficiencies, provides a multidisciplinary research process that  integrates and applies competencies with an assurance of  consistently high quality, and also has the ability to apply  competencies with expedience when an Army request for analytic  support requires a quick response.

OUSD(A&S)’s sponsoring agreement with RAND NDRI defines RAND  NDRI’s research capability and core competencies such as, but not  limited to, global and national security, defense acquisition,  intelligence, and system risk management as means to satisfy  essential needs of the FFRDC’s work sponsors for policy research  and analysis.

Primary sponsor officials we spoke with also told us that FFRDC staff  skills and knowledge related to FFRDC core competencies are important  to DOD. For example:

Navy officials said CNA is uniquely suited to perform work for the  Navy due to CNA’s core competencies relating to maritime defense  analysis and how those competencies align with Navy goals and  requirements.

Army officials told us that RAND Arroyo Center staff has extensive  background knowledge and analytical skills relating to reserve affairs,  manpower policy, and war game analysis, among other things, in  providing work for the Army.

Air Force officials told us that RAND PAF has robust knowledge of Air  Force processes and maintains top staff and researchers in each core  competency.

OUSD(A&S) officials also told us that sponsors and FFRDCs have a  relationship in which sponsors rely on FFRDCs for independent and  knowledgeable expertise within their core competencies to address  sponsors’ analytic requirements.

As shown in figure 4, DOD’s S&A Center primary sponsors identified 3 to  15 core competencies in their sponsoring agreements with each S&A  Center.

DOD cited FFRDCs’ core competencies as factors that contributed to  using S&A Centers when initiating projects we reviewed, as provided by  DOD instruction. For example:

When initiating a fiscal year 2016 CNA assessment on the effects of  possible policy changes to a career track program for military officers  who are trained to work with other military services, DOD’s Office of  the Under Secretary of Defense for Personnel and Readiness cited  CNA’s core competencies of analysis of maritime resources; maritime  program planning; and maritime policies, strategies, and doctrines as  justification for using CNA to perform the work, among other things.

In initiating a fiscal year 2014 IDA analysis on satellite ground control,  DOD’s Office of the Deputy Assistant Secretary of Defense for Space  and Intelligence cited IDA’s core competencies related to technology,  such as systems and capabilities evaluations, as justification for using  IDA for the research.

DOD Uses S&A Center Research in a Variety of Ways and Takes Some Steps to Assess the Value of Research and Centers

DOD Uses S&A Center Research to Inform Decisions, Shape Guidance, and Identify Potential Efficiencies

DOD reports that it uses studies and analyses to inform decision-making;  shape guidance, policies, and training; and identify opportunities to save  time and money.

Inform decision-making. For example, a 2016 study conducted by  the RAND Arroyo Center on linking Army cost and performance found  that the Army needed an updated tool to inform more strategic  allocation of its resources. Among other things, the study contributed  to updated strategies to measure the Army’s performance regarding  force structure and readiness as well as the cost implications for these  activities. According to an Army official, the study contributed to the  development of updated Army metrics for cost and other performance  indicators. In another example, a 2013 research project conducted by  RAND NDRI on effectiveness measures of a DOD program to reduce  the threat from infectious diseases and biological weapons developed  and recommended two sets of metrics to improve program evaluation  efforts. According to OUSD(A&S) officials, DOD used the  recommended metrics to develop program performance measures.

Shape guidance, policies, and training. For example, a 2013 study  conducted by RAND NDRI on the root causes related to DOD  weapons programs cost overruns found, among other things, that  DOD needed to re-examine its assumptions when estimating a  program’s cost, schedule, and technical performance. OUSD(A&S)  officials told us the study contributed to DOD’s decision to update its  policy, processes, management practices, and training curriculum so  as to improve estimates. In another example, a 2013 study conducted  by RAND Arroyo Center on the value of security missions conducted  by the Army’s geographically aligned forces found that the use of  these forces improved the efficiency of security planning and  preparation and recommended a range of process and planning  improvements for the Army. According to an Army official, the Army  used several of the recommendations to update guidance for  preparation and planning for future missions involving regionally  aligned forces.

Identify opportunities to improve efficiency. For example, a 2013  study conducted by RAND Arroyo Center on marketing and resources  needed for Army recruiting efforts identified strategies aimed at  optimizing the Army’s annual spending, estimated at nearly $1 billion  for recruiters, enlistment bonuses, and television advertising. An Army  official said that the Army has used the recruiting tool developed by  RAND Arroyo Center for this study to make decisions and the Army  estimates the tool can reduce costs by potentially hundreds of millions  of dollars annually. In another example, DOD reported in its 2015  comprehensive review of RAND PAF that a 2010 study conducted by  RAND PAF on aircraft maintenance at centralized repair facilities  found that these facilities should be consolidated. According to the Air  Force primary sponsor, this study helped the Air Force make  decisions that led to saving up to $300 million annually as well as  saving time on aircraft inspections.

DOD Has Taken Steps to Assess the Value of S&A Center Research and the Centers

In terms of assessing the outcomes of research, we found that DOD  primary sponsors took steps to assess the value of S&A Center research  and the centers. The DOD instruction requires that primary sponsors  assess the efficiency and effectiveness of the FFRDC in meeting DOD  needs in comprehensive reviews, including a review and summary of  FFRDC accomplishments and their effectiveness utilizing factors such as  quality and timeliness of the work produced and value of projects  assessed. Additionally, the DOD instruction provides that the factors of  technical quality, responsiveness, value, and timeliness be addressed in  annual performance reviews. DOD’s FFRDC Management Plan—which preceded the DOD instruction and was in effect until the DOD instruction  became effective in January 2018—also required primary sponsors to  annually assess the value of FFRDC performance, among other factors,  and include summaries of these annual assessments in comprehensive  reviews. Primary sponsors generally assess the value of S&A Center  research through annual performance reviews (through performance  evaluation questionnaires to solicit feedback from work sponsors) and  comprehensive reviews.

To monitor the execution of research projects, primary sponsors regularly  solicit work sponsor input regarding S&A Centers’ performance, including  the value, technical quality, responsiveness, and timeliness of the work  performed. Time frames for soliciting this input vary by primary sponsor  but most do this annually. These questionnaires include one or more  sections for work sponsors to add comments about S&A Center work and  allow work sponsors to rate S&A Center performance. Some of these  questionnaires use a numerical scale. For example, the Air Force  questionnaire sent to RAND PAF work sponsors asks respondents to rate  project value using a scale from 1 through 10, with 1 indicating “very  poor” and 10 “very good.” The OUSD(A&S) questionnaire sent to IDA  work sponsors asks respondents to rate the value of IDA’s work and  results using a scale from 1 through 5, where 1 symbolizes either  “strongly agree” or “outstanding performance” and 5 symbolizes “strongly  disagree” or “poor performance.”

FFRDC primary sponsors conduct comprehensive reviews at least every  5 years to, among other things, identify the accomplishments made by  each FFRDC. In August 2014, we reported that DOD officials described  the comprehensive review process as an opportunity to take a broad  assessment of the FFRDC and its key competencies beyond the annual  assessments of FFRDCs. Included in these comprehensive reviews is a  summary of FFRDC accomplishments and effectiveness in meeting work  sponsors’ needs since the last comprehensive review.

In our examination of the most recent comprehensive reviews for each of  the five S&A Centers, we found that the comprehensive reviews  summarize the results from the performance evaluation questionnaires  and assessed the value of the research in varying ways. For example, the  Army questionnaire to RAND Arroyo Center work sponsors assessed  value in terms of whether a project was worth the investment monetarily.  OUSD(A&S) questionnaires sent to work sponsors assessed the value of  IDA work in relation to whether the results were useful, consistent with the  level of effort, and if IDA brought competence, expertise, and helpful  perspectives to the issues. The Army reported in the 2014 comprehensive  review of RAND Arroyo Center that between fiscal years 2010 through  2013, work sponsors provided overwhelmingly positive results that RAND  Arroyo Center performance was “worth the level of effort.” DOD’s Office  of the Under Secretary of Defense for Acquisitions, Technology, and  Logistics—RAND NDRI’s primary sponsor prior to the DOD  reorganization in 2018—reported in the 2014 comprehensive review of  RAND NDRI that in fiscal year 2013, work sponsors provided  overwhelmingly positive results that RAND NDRI performance provided  long-term value.

We also found that comprehensive reviews included anecdotal examples  of how DOD used S&A Center research. For example, the Army 2014  comprehensive review of RAND Arroyo Center highlighted 53 of 114  research projects completed between fiscal years 2010 through 2013 to  demonstrate how RAND Arroyo Center work met Army research  requirements. Likewise, the Air Force primary sponsor’s 2015  comprehensive review of RAND PAF highlighted 28 of 207 research  projects completed between fiscal years 2010 and 2014 to demonstrate  how the Air Force leveraged RAND PAF work to improve efficiency in the  department. An Air Force official told us that RAND PAF, and not the Air  Force, collected 28 project examples for the purposes of the  comprehensive review.

DOD Does Not Track Whether S&A Center Recommendations Have Been Implemented, but Recently Took Steps Intended to Improve Insights

Another potential way to assess the outcomes of research is to track to  what extent a research project’s recommendations were implemented,  and how. Neither DOD nor primary sponsors currently track the  implementation of S&A Center research project recommendations. While  primary sponsors are not tracking recommendations, in 2015 one of the  S&A Centers—RAND PAF—began tracking recommendations made to  the Air Force. According to a RAND PAF representative, the tracking  system captures the issue, approach, conclusions, opportunities, and  outcomes for each completed project. A RAND PAF representative told  us that tracking recommendations is useful for demonstrating the value  that RAND PAF provides the Air Force. In April 2019, a Navy official told  us that the Navy is working on a database to track CNA reports, including  recommendations, report topic, work sponsor, and project funding, among  other things, to prevent duplication of requests. The Navy official said this  effort is expected to be completed in 2019. Both OUSD(A&S) and Army  officials told us that while they do not currently track recommendations,  they are considering doing so as part of their oversight efforts. Further,  Army officials told us that it is important for the sponsor that implements  the recommendations to track how and whether that information was  used.

While tracking recommendations is useful according to some primary  sponsors, some DOD officials cautioned that tracking recommendations  would not provide insights into the overall value across all S&A Center  research. DOD officials told us that recommendations are only one  potential outcome of S&A Center research and that the value of a study  may not be specifically linked to a recommendation. For example, Navy  officials said that CNA’s projects may present the Navy with options and  associated courses of action rather than formal recommendations, and  DOD officials also told us that S&A Center work can provide value to  DOD that is not always represented by recommendations, such as  presentations or research aimed at contributing to the understanding of a  particular issue, but without specific recommendations.

In February 2019, DOD’s Office of the Under Secretary of Defense for  Personnel and Readiness issued a memorandum related to the oversight  of the Personnel and Readiness Studies and Analysis program. The  memorandum tasked the program director with developing a studies and  analysis program framework that improves accountability for project  results and the implementation of study recommendations. Personnel and  Readiness also issued a template “action memo” providing for an  executive summary of completed projects as well as implementation  plans delineating recommendations made, implementation approach, and  plan of action for each recommendation. According to a senior Personnel  and Readiness official, work sponsors with reports that were completed or  published since September 2018 are subject to these actions. This official  noted that the purpose is to increase accountability of the Personnel and  Readiness staff regarding the use of FFRDCs and to develop an overall  picture of the value proposition of FFRDC research. It is too soon to tell to  what extent these memorandums will affect DOD’s insights on its  implementation of S&A Center recommendations.

DOD and the S&A Centers We Reviewed Have Conflict of Interest Policies and Practices

Regulation Requires FFRDCs to Operate Free from Conflicts of Interest

What are Conflicts of Interest?  A Personal Conflict of Interest exists when an  individual employed by an organization is in a  position that could materially influence  research findings or recommendations and  may lack objectivity due to their financial  interests, personal activity, or relationships.  An Organizational Conflict of Interest exists  when, because of other interests or  relationships, an entity is unable or potentially  unable to render impartial assistance or  advice to the government or the entity might  have an unfair competitive advantage.

The Federal Acquisition Regulation (FAR) requires an FFRDC to conduct  its business in a manner befitting its special relationship with the  government and to be free from conflicts of interest. To perform its  responsibilities to the sponsoring agency, an FFRDC and its employees  have access beyond that which is common in a normal contractual  relationship, including access to sensitive and proprietary data and  information, equipment, and property. To accomplish this, the FAR and  DOD instruction state that an FFRDC must be free from conflicts of  interest and fully disclose financial and outside interests to the sponsoring  agency. Conflicts of interest can be personal or organizational. Personal  conflicts of interest can be, but are not limited to, financial interests of the  employee or close family members, other employment, gifts, consulting  relationships, other forms of research funding or support, investment in  the form of stock or bonds ownership, real estate, or business ownership.  Additionally, the DOD instruction outlines steps FFRDC parent  organizations should take to prevent and mitigate conflicts of interest.  These steps include, but are not limited to, having procedures in place to  screen employees for potential conflicts of interest; requiring disclosure of  financial and other interests that might affect the employee’s objectivity;  establishing policies and procedures to protect proprietary, privileged, and  sensitive information from disclosure; and reporting any conflicts of  interest to the applicable contracting officer or contracting officer’s  representative and the primary sponsor as soon as it is identified. See  figure 5 for DOD’s conflict of interest elements outlined in the DOD  instruction that primary sponsors are to require from FFRDC parent  organizations.

All Five S&A Centers Have Conflict of Interest Policies and Practices

Each of the five S&A Centers we reviewed has corporate-wide conflict of  interest policies and practices which incorporate various key elements of  the DOD instruction. For example, all S&A Center policies we reviewed  have measures that require personnel to protect proprietary, privileged,  and sensitive information. S&A Center representatives told us they  undertake various approaches in practice that meet key elements in the  DOD instruction in order to ensure they operate in the public interest with  objectivity and independence. For example:

Reviewing all personnel annually or on a task-by-task basis for  conflicts of interest. Generally, representatives we spoke with from the  five S&A Centers address conflicts of interest annually or task-by- task, which is an option outlined in the DOD instruction. For instance,  RAND representatives said they perform task-by-task, instead of  annual, conflict of interest reviews because staff do not know which  projects they will be working on during the year. In addition, IDA and  RAND representatives told us they screen employees upon hire as  well as when an employee initiates a new project, and both IDA and  RAND have automated their screening processes. IDA  representatives explained that their automated tool screens personnel  at the initiation of each project, by including, for example, a process to  determine if staff assigned to a project have any affiliations with  industry or companies and competitors in the particular field of study.  If staff or members of their households do have affiliations, IDA may  issue a waiver if the financial interest (such as but not limited to  stocks, stock options, and bonds) in a single company is below  $15,000, the threshold for disclosure outlined in the DOD instruction.  IDA representatives also told us that IDA staff are required to self- report any changes to previous financial interest disclosures during  the year. In another example, RAND representatives said their  automated conflict of interest tool screens for conflicts of interest by  comparing areas of work RAND performs to similar areas in the  private sector. Additionally, the system will identify any staff that have  not submitted a conflict of interest statement within a year.

Providing initial and annual conflict of interest training for all  personnel. S&A Center representatives told us that they perform  training related to or specifically covering conflicts of interest in  varying ways. IDA’s corporate-wide conflict of interest policy includes  initial and annual conflict of interest training elements, as outlined in  the DOD instruction. For example, IDA’s policy states that all  employees are to participate in conflict of interest training upon initial  hire, and in annual refresher training thereafter. The other four S&A  Centers did not explicitly include annual conflict of interest training in  corporate-wide policies, but representatives told us they provide  annual ethics training, which includes training on conflicts of interest,  to their employees. For example, CNA representatives told us they  provide ethics and conflicts of interest training to staff, which is  required by their contract with the Navy. CNA representatives told us  that if CNA staff do not complete the required training, the staff will be  blocked from accessing CNA’s time card system and will not receive  pay until the training is complete. In another example, RAND  representatives told us they have annual training that covers ethics,  conflicts of interest, and culture and discrimination issues for newly  hired staff.

Representatives from each of the S&A Centers told us they attempt to  mitigate potential conflicts of interest as soon as the potential conflicts  become known and before they become a reportable conflict. For  example, CNA representatives told us that in one instance, a CNA  employee’s spouse worked for the Navy and CNA mitigated this potential  conflict by transferring the employee to another project where the  relationship did not pose a potential conflict. In another example, when a  RAND employee inherited stock in the middle of a project, a potential  conflict of interest was mitigated by the employee selling the inherited  stock. In another instance, a RAND employee was initially staffed to a  project related to an area of work a spouse worked on commercially, and  RAND mitigated the potential conflict by recusing the employee from the  project.

We provided a draft of this report to DOD for review and comment. In its  comments, DOD concurred with our findings. DOD also provided  technical comments, which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional  committees and the Secretary of Defense. In addition, the report is  available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-4841 or LudwigsonJ@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix II.

Appendix I: Selected Study and Analysis Center Research Projects

Table 2 provides detailed information on the 22 projects we selected for  review.

Appendix II: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Janet McKelvey (Assistant  Director), Andrew Burton (Analyst-in-Charge), Mallory Bryan, Lisa Fisher,  and Jordan Kudrna made key contributions to this report. Additional  assistance was provided by Marie Ahearn, Pete Anderson, Jenny  Chanley, Joseph Cook, Julia Kennon, Tind Shepper Ryen, and Roxanna  Sun.

DOD sponsors research and development activities at 10 Federally Funded Research and Development Centers that provide innovative solutions to national security threats. Half of those are Study and Analysis Centers, which provide research that contributes to policy development.

We found that from fiscal years 2013-2018, DOD committed about $400 million annually to the Study and Analysis Centers. DOD reported using the Centers’ studies to inform policy and process decisions. For example, one study on the cause of weapons systems cost overruns found DOD needed to change its approach to estimating program cost, schedule, and performance.


Electronic Health Records: VA Needs to Identify and Report Existing System Costs

VA provides health care services to approximately 9 million veterans and their families and relies on its health information system—VistA—to do so. However, the system is more than 30 years old, is costly to maintain, and does not fully support exchanging health data with DOD and private health care providers. Over nearly 2 decades, VA has pursued multiple efforts to modernize the system. In June 2017, the department announced plans to acquire the same system—the Cerner system—that the Department of Defense is implementing. VA plans to continue using VistA during the department's decade-long transition to the Cerner system.

GAO was asked to summarize its report that is being released today which discusses, among other things, (1) the extent to which VA has defined VistA and (2) the department's annual costs to develop and sustain the system.

In preparing the report on which this testimony is based, GAO analyzed documentation that defines aspects of VistA and identifies components to be replaced; and evaluated the reliability of cost data, including funding obligations associated with the development and sustainment of VistA for fiscal years 2015, 2016, and 2017.

The Department of Veterans Affairs (VA) has various documents and a database that describe parts of the Veterans Health Information Systems and Technology Architecture (VistA); however, the department does not have a comprehensive definition for the system. For example, VA has identified components that comprise VistA, identified interfaces related to the system, and collected system user guides and installation manuals. VA has also conducted analyses to better understand customization of VistA components at various medical facilities. Nevertheless, the existing information and analyses do not provide a thorough understanding of the local customizations reflected in about 130 versions of VistA that support health care delivery at more than 1,500 sites. Program officials stated that they have not been able to fully define VistA due to the decentralization of the development of the system for more than 30 years. Cerner's contract to provide a new electronic health record system to VA calls for the company to conduct comprehensive assessments to identify site-specific requirements where its system is planned to be deployed. Three site assessments have been completed and additional assessments are planned. If these assessments provide a thorough understanding of the 130 VistA versions, the department should be able to define VistA and be better positioned to transition to the new system.

VA identified costs for VistA and its related activities adding up to approximately $913.7 million, $664.3 million, and $711.1 million in fiscal years 2015, 2016, and 2017, respectively—for a total of about $2.3 billion over the 3 years. However, of the $2.3 billion, the department was only able to demonstrate that approximately $1 billion of these costs were sufficiently reliable.

In addition, the department omitted VistA-related costs from the total. The lack of a sufficiently reliable and comprehensive total cost for VistA is due in part to not following a well-documented methodology that describes how the department determined the costs for the system. As a result of incomplete cost data and data that could not be determined to be sufficiently reliable, the department, legislators, and the public do not have a complete understanding of how much it has cost to develop and maintain VistA. Further, VA lacks the information needed to make decisions on sustaining the many versions of the system.

In its report being issued today, GAO is recommending that VA develop and implement a methodology for reliably identifying and reporting the total costs of VistA. The department agreed with the recommendation.

Thank you for the opportunity to participate in today’s hearing regarding  the Department of Veterans Affairs’ (VA) health information system—the  Veterans Health Information Systems and Technology Architecture  (VistA)—which has been essential to the department’s ability to deliver  health care to veterans. This technically complex system has been in  operation for more than 30 years, is costly to maintain, and does not fully  support exchanging health data with the Department of Defense (DOD)  and private health care providers.

VA has initiated a major program to replace the VistA electronic health  record (EHR) with a commercial-off-the-shelf (COTS) product. The  department plans to start deploying its new EHR system in March 2020.  However, VA sites are to continue using VistA until they receive the new  system during a phased transition over the next 10 years.

We recently reviewed key aspects of VistA in response to a request from  the House Committee on Veterans’ Affairs. We examined, among other  things, the extent to which VA has defined VistA and the department’s  annual costs to develop and sustain the system.

At your request, my testimony for this hearing summarizes the findings  discussed in our report on VistA, which is being released today. More  detailed information on our objectives, scope, and methodology for that  work can be found in the issued report.

We conducted the work on which this statement is based in accordance  with generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.

VA’s mission is to promote the health, welfare, and dignity of all veterans  in recognition of their service to the nation by ensuring that they receive  medical care, benefits, social support, and lasting memorials. In carrying  out this mission, the department manages one of the largest health care  delivery systems in the United States that provides enrolled veterans with  a full range of services. These services may include primary care; mental  health care; and outpatient, inpatient, and residential treatment. The  Veterans Health Administration (VHA), one of the department’s three  major components, is responsible for overseeing the provision of health  care at all VA medical facilities.

Information technology (IT) is widely used and critically important to  supporting the department in delivering health care to veterans. As such,  VA operates and maintains an IT infrastructure that is intended to provide  the backbone necessary to meet the day-to-day operational needs of its  medical centers and other critical systems supporting the department’s  mission. The infrastructure is to provide for data storage, transmission,  and communications requirements necessary to ensure the delivery of  reliable, available, and responsive support to all VA staff offices and  administration customers, as well as veterans. The Office of Information  and Technology (OIT) is responsible for managing the majority of VA’s IT- related functions. The office provides strategy and technical direction,  guidance, and policy related to how IT resources are to be acquired and  managed for the department.

VA provides health care services to approximately 9 million veterans and  their families and relies on its health information system—VistA—to do  so. VistA has been essential to the department’s ability to deliver health  care to veterans. It was developed based on the collaboration between  staff in the VA medical facilities and VHA IT personnel. Specifically,  clinicians and IT personnel at the various VA medical facilities  collaborated to define the system’s requirements and, in certain cases,  carried out its development and implementation. As a result of these  efforts, the system has been in operation since the early 1980s.

VistA supports a complex set of clinical and administrative capabilities. It  is comprised of an architecture that ties together servers and personal  computer workstations with various applications within VA facilities and the supporting infrastructure, such as data centers, storage, and  messaging technologies. The core system and database code are  programmed in the MUMPS programming language. Among other  things, VistA contains an EHR for each patient and supports clinics and  medical centers.

In addition, the system provides functionality beyond the EHR and  exchanges information with many other applications and interfaces. For  example, the system also provides the functionality of a time and  attendance program, asset management system, library, and billing  system, among other things.

Users interact with VistA through a number of interfaces that connect  stored health data. These interfaces enable the system to communicate  (send or exchange data) with other VA systems, as well as with other  federal agencies (e.g., DOD), health information exchange networks, and  COTS products. According to OIT officials, applications either interface  with VistA directly through a messaging protocol or extract data from the  system via a reporting mechanism.

The Computerized Patient Record System is a graphical user interface to  VistA that runs on workstations, laptops, and tablets and enables the  department to support clinical workflows. Specifically, the Computerized  Patient Record System enables the department to create and update an  individual EHR for each VA patient. Among other things, clinicians can  order lab tests, medications, diets, radiology tests, and procedures;  record a patient’s allergies or adverse reactions to medications; request  and track consults; enter progress notes, diagnoses, and treatments for  each encounter; and enter discharge summaries.

According to VHA officials, there are also more than 100 COTS products  that interface with VistA. In addition to these commercial products,  medical equipment or devices at local facilities may also require  interfaces to the system, and these vary on a site-by-site basis.

VA Has about 130 Different Versions of VistA

Over the last several decades, VistA has evolved into a technically  complex system that supports health care delivery at more than 1,500  locations, including VA Medical Centers, outpatient clinics, community  living centers, and VA vet centers. Customization of the system by local  facilities has resulted in about 130 clinical versions of VistA—referred to  as instances.

According to the department, no two VistA instances are identical.  Further, each instance is comprised of over 27,400 routines (executable  modules of code), which are logically grouped into products or modules.  VistA products or modules can also be comprised of one or more  software applications that support health care functions, such as  providing care coordination and mental health services. The department  reported that there are approximately 140 to 200 products or modules  that comprise the system.

The 130 clinical instances of VistA are operated from four regional VA  data centers. Users interact with the system through the Computerized  Patient Record System. Aggregated clinical data from every instance of  the system are located on servers hosted at VA’s National Data Center.

Over time, VA has identified the need for enhancements and  modifications to VistA in order to ensure that the system keeps up with  current technology and health care delivery. However, according to the  department, the system has become difficult and costly to maintain. This  is a result of, for example, being programmed in MUMPS, a language for which there is a dwindling supply of qualified software developers. It is  also due to years of decentralized customization of the system by staff  members who were permitted to develop and implement applications at  the local level.

OIT and VHA Share Responsibilities for VistA

OIT and VHA serve as the technical and functional leaders, respectively,  for the department’s health care delivery and, together, they have worked  to develop and maintain VistA for decades. Specifically, OIT is  responsible for managing the majority of VA’s IT-related functions. The  office provides strategy and technical direction, guidance, and policy  related to how IT resources are to be acquired and managed for the  department.

According to the department, OIT’s mission is to collaborate with its  business partners (such as VHA) and provide a seamless, unified veteran  experience through the delivery of state-of-the-art technology. The  Assistant Secretary for Information and Technology/Chief Information  Officer (CIO) serves as the head of OIT and is responsible for providing  leadership for the department’s IT activities.

The CIO also advises the Secretary regarding the execution of VA’s IT  systems appropriation, consistent with the Federal Information  Technology Acquisition Reform Act. For fiscal year 2019, the  department has been appropriated $4.1 billion for IT. According to VA’s  budget documentation, about $1.2 billion of this amount is intended to  support IT staffing and associated costs for approximately 8,100 full-time  employees.

VHA provides information and expertise to OIT to support the  department’s health-related information systems. For example, VHA  officials help identify clinical and business needs used to inform IT  requirements development. The Under Secretary for Health is the head of VHA and is supported by the Principal Deputy Under Secretary for  Health, four Deputy Under Secretaries for Health, and nine Assistant  Deputy Under Secretaries for Health.

VA Has Begun to Acquire a New EHR System

After nearly 2 decades of pursuing multiple efforts to modernize VistA, in  June 2017, the former VA Secretary announced that the department  planned to acquire the same EHR system that DOD is acquiring—Cerner  Millennium. According to the department, it has chosen to acquire this  product because Cerner Millennium should allow VA’s and DOD’s patient  data to reside in one system, thus, potentially reducing or eliminating the  need for manual and electronic exchange and reconciliation of data  between two separate systems.

Accordingly, the department awarded an indefinite delivery, indefinite  quantity contract to Cerner Corporation in May 2018 for a maximum  amount of $10 billion over 10 years. Cerner is to replace the 130  instances of VistA with a standard COTS system to be implemented  across VA. This new system is to support a broad range of health care  functions including acute care, clinical decision support, dental care, and  emergency medicine. When implemented, the new system will be  expected to become the authoritative source of clinical data to support  improved health, patient safety, and quality of care provided by VA.

The Electronic Health Record Modernization (EHRM) program is  responsible for managing the Cerner contract implementation. For fiscal  year 2019, the program was appropriated about $1.1 billion for planning  and managing the transition from VistA to Cerner.

Further, the department has estimated that an additional $6.1 billion in  funding, above the Cerner contract amount, will be needed to fund  additional project management support supplied by outside contractors,  government labor costs, and infrastructure improvements over the 10- year contract period.

VA plans to deploy the new EHR system at three initial operating  capability sites within 18 months of October 1, 2018, with a phased  implementation of the remaining sites over the next decade. Each VA  medical facility is expected to continue using VistA until the new system  has been deployed. The three initial deployment sites, located in the  Pacific Northwest, are the Mann-Grandstaff, American Lake, and Seattle  VA Medical Centers and related clinical facilities that operate the same  instances of VistA. These are the first locations where the system is  expected to “go live.”

The task order to deploy the Cerner system at the three initial sites  provides a detailed description of the steps Cerner needs to take in order  to reach initial operating capability at the Mann-Grandstaff site in March  2020, and at the Seattle and American Lake sites in April 2020. According  to the schedule, the initial operating capability sites are expected to be  operational by July 2020.

VA Has Undertaken Efforts to Define VistA, but Additional Work Remains

In order to maintain internal control activities over an IT system and its  related infrastructure, organizations should be able to define physical and  performance characteristics of the system, including descriptions of the  components and the interfaces. Further, consistent with GAO’s Cost  Estimating and Assessment Guide, a comprehensive system definition  should identify customization and the environment in which the system  operates. While defining a complex IT system can be challenging,  having an adequate understanding of its characteristics will better position  the organization to comprehensively project and account for costs over  the life of a system or program as well as identify specific technical and  program risks. Definition of VistA remains important because VA plans to  continue using the system during the department’s decade-long transition  to the Cerner system.

VA maintains multiple documents and a database that describe parts of  VistA, including various components and interfaces. However, despite  these existing sources, OIT officials acknowledged that there is no  comprehensive definition of the VistA system. Consequently, VA has  completed a number of efforts to better define VistA and understand the  environment in which it operates and additional work is planned in the  future.

Specifically, VA has documented descriptions of the system, including the  components that comprise it. These descriptions are documented in  multiple sources: the VA Monograph, VA Systems Inventory, and VA  Document Library.

The VA Monograph is a document maintained by OIT that provides  an overview of VistA and non-VistA applications used by VHA.  According to VHA officials, the VA Monograph is the primary  document that describes the components of the system. The  Monograph describes VistA in terms of modules. For modules  identified, including VistA modules, information such as the  associated business functions, VA Systems Inventory identification  number, and a link to the VA Document Library for additional technical  information are provided.

The VA Systems Inventory is a database maintained by OIT that  identifies current IT systems at the department, including systems and  interfaces related to VistA. For systems identified, the database  includes information such as the system name, the system status (i.e.,  active, in development, or inactive), and related system interfaces.

The VA Document Library is an online resource for accessing  documentation (i.e., user guides and installation manuals) on the  department’s nationally released software applications, including  VistA.

VA has taken additional steps to further define the system. For example,  EHRM program officials recognized the need to further understand the  customization of VistA components at the various medical facilities and  have conducted analyses to do so. These analyses include:

Variance analysis: As part of its VistA Evolution program, which  has focused on standardizing a core set of VistA functionality, the  department implemented a process to compare the instances of VistA  installed at sites to the Enterprise Standard version. The results of  this analysis allowed the department to assess the criticality of each  variance, which is expected to help with VA’s transition to the Cerner  system.

Module analysis: EHRM program subject matter experts undertook  an analysis that involved reviewing and assessing capabilities  provided by VistA modules. This analysis enabled department officials  to determine whether the capability provided by a VistA module could  be provided by the Cerner system, or whether another COTS solution  would be required to support this function going forward.

Visual mapping: EHRM program officials also directed an analysis  that involved developing a notional visual mapping of VA’s health care  applications, components, and supporting systems within the health  delivery environment. The results of this analysis provided a  description of the current state of one instance of VistA and the VA  health environment, which is intended to inform the department of  possible opportunities for business process and IT improvements as it  proceeds with the Cerner acquisition.

Nevertheless, even with these analyses, VA has not yet fully defined  VistA, including, for example, identifying performance characteristics of  the system and describing the environment in which it operates. The  department’s three sources that describe VistA and the additional  analyses undertaken do not provide insight into site specific  customizations of the system. For example, the VA Monograph does not  include information on module customization at local facilities. In addition,  according to OIT officials, the systems inventory does not reflect  differences among the 130 different instances of VistA and does not take  into consideration regional and local customizations of related  components. Further, the visual mapping analysis noted that there was  not full insight of the intertwined structure of data and applications or the  various local customizations of VistA.

EHRM program officials stated that they have not been able to fully define  VistA and understand all local customizations due to the decentralization  of the development of the system and its evolution over more than 30  years. They explained that VistA’s complexity is partly due to the various  instances of the system, compounded by local customizations, which  have resulted in differences in VistA instances operating at various  facilities.

According to EHRM program documentation, Cerner’s contract calls for  the company to conduct comprehensive assessments to capture the  current state of technical and clinical operations at specific facilities, as  well as identify site-specific requirements where the Cerner system is  planned to be deployed. As of June 2019, Cerner had completed site  assessments for the three initial operating capability sites in the Pacific  Northwest and had planned additional assessments at future deployment  sites. The initial site assessments included, among other things, an  assessment of the unique VistA instances and the environment in which  the system operates. The continuation of planned site assessments  should provide a thorough understanding of the 130 VistA versions, help  the department better define VistA, and position it for transitioning from  VistA to Cerner’s COTS solution.

VA Identified Total VistA Costs of about $2.3 Billion between 2015 and 2017, but Could Not Sufficiently Demonstrate the Reliability of All Data and Omitted Other Costs

When using public funds, an agency must employ effective management  practices in order to let legislators, management, and the public know the  costs of programs and whether they are achieving their goals. To make  those evaluations for a program or for a system as large and complex as  VistA, a complete understanding of the system and reliable cost  information is required. By following a methodology and utilizing reliable  data, an agency can ensure that all costs are fully accounted for, which in  turn, better informs management decisions, establishes a cost baseline,  and enhances understanding of a system’s performance and return on  investment.

Fundamental characteristics of reliable costs are that they should be  accurate (unbiased, not overly conservative or optimistic), well- documented (supportable with source data, clearly detailed calculations,  and explanations for choosing a particular calculation method), credible  (identifying any uncertainty or biases surrounding data or related  assumptions), and comprehensive (costs are neither omitted nor double  counted). Identification of VistA’s costs remains important because VA  plans to continue using the system during the department’s transition to  the Cerner system over the next decade.

VA identified costs for VistA and its related activities adding up to  approximately $913.7 million, $664.3 million, and $711.1 million in fiscal  years 2015, 2016, and 2017, respectively—for a total of about $2.3 billion  over the 3 years. However, the department could not sufficiently  demonstrate the reliability of certain costs that were identified. In addition,  VA identified other categories of VistA-related costs, but omitted these  costs from the total.

VA Did Not Sufficiently Demonstrate the Reliability of Data for All VistA Costs

Of the $2.3 billion total costs for VistA, VA demonstrated that only  approximately $1 billion of these costs were reliable. Specifically, OIT  officials identified VistA-related costs within seven categories. The  officials were able to sufficiently explain why these categories were  included in the development and sustainment costs for VistA and how  they were documented by the department; the officials also presented  detailed source data for our examination. As a result of our review, we  determined that the cost data for these seven categories were accurate,  well-documented, credible, and comprehensive and, thus, sufficiently  reliable.

Table 1 provides a summary of the program costs identified for VistA by  OIT and VHA for fiscal years 2015 through 2017 that we determined to be  reliable.

As shown in the table, VA identified costs for the following seven  categories for fiscal years 2015 through 2017:

VistA Evolution – The VistA Evolution program costs were  associated with VistA strategy, system design, product development,  and program management. These costs totaled approximately $549.6  million.

Interoperability – The Interoperability program focused on sharing  electronic health data between VA and non-VA facilities, including  private sector providers and DOD. For example, interoperability  costs were associated with architecture, strategy, the Interagency  Program Office, product development, and program management.  These VistA-related costs totaled approximately $140.2 million.

Virtual Lifetime Electronic Record (VLER) Health – This program  focused on streamlining the transition of electronic medical  information between VA and DOD. These VistA-related costs were  associated with product development and program management and  totaled approximately $81.2 million.

Contracts – Contract costs for VistA Evolution included VHA’s  obligations associated with workload management, change  management, clinical requirements, and clinical interoperability.  These VistA-related costs totaled approximately $202.8 million.

Intergovernmental personnel acts – Intergovernmental personnel  acts are agreements for the temporary assignment of personnel  between the federal, state, and local governments; colleges and  universities; Indian tribal governments; federally funded research and  development centers; and other eligible organizations. These costs  accounted for VHA’s need to use outside experts from approved  entities for limited periods of time to work on VistA Evolution  assignments. The total VistA-related costs were approximately $2.4  million.

Memorandums of understanding – According to VHA,  memorandums of understanding are agreements used by the  administration to obtain the services of personnel between VA entities  for VistA-related activities. These agreements accounted for  approximately $2.3 million.

Pay – Costs in this category included salaries for VHA staff who  worked on VistA-related projects as well as travel, training, and supply  costs associated with employment. These costs totaled approximately  $34.1 million.

However, VA was not able to sufficiently demonstrate the reliability of  approximately $1.3 billion in costs related to VistA. Specifically, OIT  officials identified the additional legacy VistA costs that generally fell into  three categories:

Legacy VistA: Infrastructure, hosting, and system sustainment –  Legacy VistA costs are generally related to the maintenance of fully  operational items, such as VistA Imaging and Fileman—two key  components related to VistA’s operation. The costs also included  obligations for costs related to hosting health data in both VA and  non-VA facilities. The OIT officials and subject matter experts  estimated these total costs to be approximately $343 million during  fiscal years 2015 through 2017.

However, we were not able to determine the reliability of these costs  because, for example, source data were not well documented;  changes in the cost information provided to us during our review  indicated that the cost data may not be credible; and subject matter  experts were unclear about how to separate VistA costs from non- VistA costs.

Related software – Related software costs are associated with the  software supporting, or closely integrated with, VistA that were  identified by EHRM officials, yet not tracked directly for one of the  VistA-related programs. Both OIT and VHA identified software  licensing costs as VistA-related obligations. The EHRM program  reported these costs to be approximately $389 million in total during  fiscal years 2015 through 2017.

However, we were not able to determine the reliability of the costs in  this category for a variety of reasons, including that source data were  not well documented. In addition, VA officials were not clear regarding  how the total amounts in each category should be divided between  OIT and VHA. Given this confusion, we were not able to determine if  the costs were fully accurate or credible.

OIT personnel (pay and administrative) – According to EHRM  officials, OIT does not track labor costs by program. Instead, the  department provided estimations of the amount of salaries paid to OIT  government staff working on activities such as VistA Evolution,  program management, and overall support of VistA and related  applications. OIT personnel costs were estimated by the EHRM  program office to be approximately $544 million total during fiscal  years 2015 through 2017.

However, we were not able to determine the reliability of costs in this  category because assumptions made for estimating the personnel  and salary costs were not well documented and could not be verified.

VA Omitted Certain Costs from the Total Cost of VistA

In addition, VA omitted certain VistA costs from the total costs identified  for fiscal years 2015, 2016, and 2017. Specifically, VA omitted the  following costs:

Additional hosting – OIT officials stated that additional costs related  to hosting health data by an outside vendor, as well as hosting backup  VistA instances at each of the medical center sites, should also be  included in the total costs for VistA; however, VA omitted these costs  from the total for fiscal years 2015 through 2017. Specifically,  according to the officials, calculating costs for these hosting activities  requires subject matter experts to identify equipment, space, utilities,  and maintenance costs for resources allocated specifically for VistA.  However, the department has not yet developed a methodology to  calculate the costs. The officials said they were working on identifying  a reliable approach for calculating these costs in the future.

Data standardization and testing – OIT officials stated that  additional costs related to work on clinical terminology mapping and  functional testing were not included in the total costs for VistA for  fiscal years 2015 through 2017. This work related to mapping existing  clinical data to national standards and making updates to VistA or the  Joint Legacy Viewer and included mapping data and building test  scripts and reports. OIT officials noted that this work had been  critical to the VistA Evolution program, but they did not provide actual  cost data in this category.

The lack of sufficiently reliable and comprehensive costs indicates that  the department is not positioned to accurately report the annual costs to  develop and sustain VistA. This is due in part to VA not following a well- documented methodology that describes how the department determined  the total costs for the system. In lieu of a methodology, OIT officials said  that leadership and staff from the program took efforts to identify and  track the cost components and contracts associated with the system.  However, they noted that costs associated with VistA were not all clearly  labeled as VistA in an IT system and it was necessary to estimate other  costs. The officials were also unable to verify how VistA-related costs  were separated from other department costs in all areas and subject  matter experts were not consistently familiar with the estimation methods  employed and how VistA was defined for the purposes of calculating  costs. Further, VA officials noted that they were still working on the best  approach to identifying and calculating omitted costs.

Without documenting the methodology for what costs are to be included  and how they were identified and calculated, VA’s total does not  accurately reflect the development and sustainment costs for VistA. As a  result, the department, legislators, and the public do not have the  comprehensive, reliable information needed to understand how much it  actually cost to develop and maintain the system. Further, VA does not  have the reliable information needed to make critical management  decisions for sustaining the many versions of VistA over the next 10 years  until the Cerner system is fully deployed.

Implementation of GAO’s Recommendation Could Help Ensure VA Reliably Reports VistA Costs

In our report, we are making a recommendation for VA to improve its  reporting of VistA’s costs. Specifically, we are recommending that the  department develop and implement a methodology for reliably identifying  and reporting the total costs of VistA. The methodology should include  steps to identify the definition of VistA and what is to be included in its  sustainment activities, as well as ensure that comprehensive costs are  corroborated by reliable data. In written comments on a draft of the report,  the department agreed with the recommendation and stated that it will  provide the actions it plans to take to address this recommendation within  180 days.

In conclusion, although VA is not likely to be positioned to retire VistA for  at least another 10 years, the department lacks the comprehensive and  reliable cost information needed to make critical management decisions  for sustaining the system. As the department continues to work toward  acquiring a new electronic health record, it will be important for VA to take  actions to address our recommendation for improving the reporting of  VistA costs. Doing so is essential to helping ensure that decisions related  to the current system are informed by reliable cost information and that  there is an accurate basis for reporting on the return on its investment for  replacing VistA.

Chair Lee, Ranking Member Banks, and Members of the Subcommittee,  this completes my prepared statement. I would be pleased to respond to  any questions that you may have.

If you or your staffs have any questions about this testimony, please  contact Carol C. Harris, Director, Information Technology Management  Issues, at (202) 512-4456 or harriscc@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this testimony statement. GAO staff who made key  contributions to this testimony are Mark Bird (Assistant Director),  Rebecca Eyler, Jacqueline Mai, Monica Perez-Nelson, Scott Pettis,  Jennifer Stavros-Turner (Analyst in Charge), and Charles Youman.

This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.

The VA’s health information system is more than 30 years old and is costly to maintain. Over nearly 2 decades, VA’s multiple modernization efforts have continually fallen short.

In June 2017, VA announced it would buy the same system DOD is implementing. VA plans to continue using its current system during a decade-long transition.

This testimony, which is based on a report being released today, discusses the current system and its related costs.

We found that VA lacked insight into the total costs for the system. This could make it more difficult to make decisions about sustaining the system throughout the transition.