Federally Funded Research and Development Centers: Improved Oversight and Evaluation Needed for DOD's Data Access Pilot Program

FFRDCs provide federal agencies with research and development functions, technical systems engineering capabilities, and policy development and decision-making studies, among other services. The Federal Acquisition Regulation states that FFRDCs have a special relationship with DOD, which can give FFRDCs access to sensitive data beyond what would commonly be shared with contractors. The National Defense Authorization Act for Fiscal Year 2017 directed DOD to establish a 3-year pilot program that allows FFRDCs streamlined access to sensitive data maintained by DOD. It also included a provision for GAO to report on the pilot program within 2 years of implementation. This report addresses the extent to which (1) FFRDCs are using the pilot program, (2) DOD put procedures in place to protect data accessed, and (3) DOD is evaluating the pilot program.

GAO reviewed DOD guidance and FFRDC processes, pilot reports for January 2018 through September 2019, and DOD's plans and efforts for evaluating the pilot program. GAO also selected a nongeneralizable sample of six projects—at least one from each FFRDC with an enrolled project as of December 2018—for further review. In addition, GAO assessed the pilot program against leading practices for pilot design.

The Department of Defense (DOD) launched a 3-year pilot program in December 2017 to enable a streamlined process to share certain sensitive data, such as data collected from its contractors, with its Federally Funded Research and Development Centers (FFRDC). At times, FFRDCs need to access such data to support DOD. The pilot was intended to reduce the burden on FFRDCs to seek permission from hundreds of contractors to access information needed for their research. Six of DOD's 10 FFRDCs have taken part in the pilot, enrolling a combined total of 33 projects, as shown in the table.

DOD officials and FFRDC representatives reported that the streamlined process made the use of sensitive data feasible. As a result, FFRDCs with completed projects in GAO's sample indicated they were able to provide more robust analyses or insights to DOD.

DOD guidance for the pilot program established procedures to protect sensitive data. But GAO found that DOD did not incorporate all of the details of the required protections into its agreements with FFRDCs. Further, GAO found that not all FFRDCs were performing annual certification of financial disclosure forms, as required by its agreements with DOD. DOD does not have a process to ensure that all the protections pertaining to FFRDCs' streamlined access to sensitive data are being followed. Without a process that defines roles and responsibilities, DOD cannot ensure that FFRDCs adhere to the protections.

DOD developed goals for the pilot program and outlined what information was to be obtained for each participating project, actions that are consistent with GAO's leading practices for pilot design. However, DOD has not developed a plan for evaluating the program nor has it consistently collected information on about a third of the pilot projects. Leading practices for pilot design call for an evaluation plan, which should include an assessment methodology and identify responsibilities as to how the evaluation will be conducted. Without an evaluation plan and a mechanism to collect information on pilot projects, DOD will not be positioned to identify the effectiveness of the pilot program and benefit from lessons learned. Such information will be useful as Congress considers the path forward after the pilot ends in December 2020.

GAO is making six recommendations, including that DOD take steps to ensure data protections are in the agreements and followed, collect information on projects, and evaluate the pilot. DOD agreed with the recommendations.

Federally Funded Research and Development Centers (FFRDCs) provide  federal agencies with research and development functions, technical  systems engineering capabilities, and policy development and decision- making studies, among other things. FFRDCs historically have assisted  the Department of Defense (DOD) in assessing individual programs or  identifying trends among the department’s weapon system acquisitions.  At times, FFRDC employees have needed to access certain sensitive  data, such as proprietary information from DOD’s contractors, to  accomplish these tasks. The Federal Acquisition Regulation states that  FFRDCs have a special relationship with federal agencies, to include  DOD, and that the department may give FFRDCs access to sensitive  data beyond that which DOD would commonly share with contractors.  DOD currently has 10 FFRDCs under contract.

Despite this special relationship, in practice, providing access has been  complicated in certain circumstances. The Trade Secrets Act prohibits  unauthorized disclosure of certain confidential information (which includes  sensitive data) by government personnel. To allow FFRDCs access to  sensitive information, until recently, DOD generally had FFRDC  researchers sign nondisclosure agreements with the original owners of  the sensitive data (defense contractors). According to DOD officials, this  practice effectively limited FFRDCs’ ability to perform certain work  because some large data repositories—such as the Cost Assessment  Data Enterprise—include sensitive data that DOD has collected from  hundreds of contractors as part of its acquisition contracts. In 2016, DOD  requested legislative authority to allow DOD personnel to share sensitive  data with FFRDCs for the purposes of performing work that would benefit  from access to large data repositories.

The National Defense Authorization Act for Fiscal Year 2017 directed  DOD to establish a 3-year pilot program permitting DOD personnel to  disclose sensitive information to its FFRDCs for the sole purpose of  performing administrative, technical, or professional services for the  department. After 3 years, for this authority to continue, Congress will  need to take action to extend the pilot or make the authority permanent.  The Act also contained a provision for GAO to review DOD’s pilot  program activities and report to Congress within 2 years of initiation of the  pilot. This report addresses the extent to which (1) FFRDCs are using the  pilot program, (2) DOD put procedures in place to protect sensitive data  accessed in the pilot program, and (3) DOD is evaluating the pilot  program.

To determine how extensively the pilot program is used, we obtained and  reviewed reports required by the pilot to identify the number of FFRDCs  and projects participating in the pilot from January 2018 (when the DOD  sponsors started reporting on the pilot) through September 2019. In  addition, we obtained information from DOD primary sponsor officials—in  the offices that oversee FFRDCs—and representatives from all 10 DOD  FFRDCs about factors that influenced participation in the pilot. We also  examined documentation from a nongeneralizable sample of six projects  enrolled in the pilot during its first year. We selected these projects using  pilot reports for January through December 2018—the most current  information available at the time of our selection. We included at least one  project from each FFRDC with a pilot project, with a focus on selecting  completed projects. We also interviewed FFRDC project representatives  for all six selected projects. Further, we obtained the perspectives of DOD officials involved with three of the completed projects. We used the  results from these six projects to better understand the operation of the  pilot program and its potential benefits.

To describe the procedures put in place to protect sensitive data  accessed using the pilot program, we obtained information about the  DOD pilot and FFRDCs’ processes as established under the pilot to  ensure protection of sensitive data, such as researcher training and other  required certifications. We also interviewed officials from DOD’s  Laboratories and Personnel Office within the Office of the Under  Secretary of Defense (OUSD) for Research and Engineering, which is  responsible for managing the pilot, primary sponsors, and the offices that  manage department-wide data repositories about the processes and any  potential risks of sharing access to this data with FFRDCs. Primary  sponsors include the OUSDs for Acquisition and Sustainment and for  Research and Engineering; Departments of the Air Force, Army, and  Navy; and the National Security Agency. Selected data repositories  include the Cost Assessment Data Enterprise, Defense Acquisition  Management Information Retrieval/Defense Acquisition Visibility  Environment, and Defense Contract Management Agency’s Industrial  Analysis Group. In addition, we interviewed representatives from all 10  DOD FFRDCs.

To determine the extent to which DOD is evaluating the pilot program, we  obtained documentation and interviewed officials from the Laboratories  and Personnel Office about plans and efforts, if any, to evaluate the pilot  program. We assessed DOD’s plans and efforts against GAO’s leading  practices for pilot design—which include activities such as establishing  objectives and having an assessment plan—and discussed the leading  practices with officials from the Laboratories and Personnel Office.

We conducted this performance audit from February 2019 to February  2020 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

DOD’s guidance states that its FFRDCs are created to (1) provide  strategic value through independent, intellectually rigorous, relevant, and  timely products and services; and (2) support the department’s goals of  long-term improvement in operations and enhanced national security.  They are managed by various military departments or divisions within the  department, called primary sponsors. FFRDCs are operated by  universities, other not-for-profit or nonprofit organizations, or private  firms—called parent organizations—under long-term contracts. They  provide special research and development services that generally cannot  be readily satisfied by government personnel or private contractors. For  example, the Lincoln Laboratory develops key radar and electronic  warfare technologies for integrated air and missile defense systems. The  Software Engineering Institute provides cybersecurity solutions for  defense entities.

DOD’s FFRDCs are grouped into three categories: research and  development laboratories, study and analysis centers, and systems  engineering and integration centers. DOD oversees 10 FFRDCs (see  table 1). According to the Director of Laboratories and Personnel within  the OUSD for Research and Engineering, he took over responsibility for  managing FFRDCs in July 2018, following a reorganization of OUSD for  Acquisition, Technology and Logistics.

DOD and each FFRDC have a sponsoring agreement, which is a stand- alone, bilateral, written agreement between the primary sponsor and the  parent organization. It must be approved by the Under Secretary of  Defense for Research and Engineering prior to award of an FFRDC  contract and is incorporated into the contract. According to DOD’s  guidance, the sponsoring agreement defines the FFRDC’s purpose and  mission, establishes the conditions under which DOD may award an  FFRDC contract, and describes the overarching requirements for  operation of the FFRDC. For example, the primary sponsor must include  provisions in the sponsoring agreement to prevent real or perceived  organizational and personal conflicts of interest. As part of that, sponsors  are to require FFRDC parent organizations to establish and maintain  policies and procedures to protect information, such as sensitive data,  from disclosure and provide training that covers ethics and conflicts of  interest. We reported in December 2019 that representatives from the  five study and analysis center FFRDCs said they provide annual training  covering ethics and conflicts of interest for all personnel.

DOD may use FFRDCs to perform work that is closely associated with  the performance of inherently governmental functions or that is critical to  maintaining control of the department’s missions and operations. Work  could include activities such as support for financial analyses, policy  development, acquisition planning, source selection, and contract  management. In the course of performing work, FFRDCs may need  access to acquisition data collected from DOD’s prime contractors and  program offices. FFRDCs may obtain these data through DOD personnel,  government databases, or directly from prime contractors. Government- held data may be stored and managed in department-wide databases or  by individual program offices. For example, the Cost Assessment Data  Enterprise is a web application that allows users access to various reports  that include information such as major defense acquisition programs’  cost, software, and technical data.

On December 21, 2017, DOD issued implementing guidance that marked  the launch of its 3-year pilot program. According to DOD officials, prior  to the start of the pilot program, FFRDC researchers needed to obtain  permission from each data owner (e.g., DOD prime contractor or  supplier), typically by signing a nondisclosure agreement. According to  DOD officials that requested the authority to allow FFRDCs to have  increased access, one of the purposes of the pilot was to allow for a  streamlined nondisclosure agreement process. Under the pilot program,  FFRDC researchers no longer have to obtain nondisclosure agreements  with each data owner.

To participate in the pilot, the FFRDC and DOD sponsor must first take  steps to ensure certain protections are in place to protect against  unauthorized disclosure or use of the data being accessed. For example,  according to the statute, in order to be eligible, participating FFRDCs and  its personnel (FFRDC researchers) had to agree to be subject to and  comply with appropriate ethics standards and requirements applicable to  government personnel, including the Ethics in Government Act of 1978,  the Trade Secrets Act, and the Procurement Integrity Act. After the  protections are in place, the FFRDC and DOD sponsor can enroll  individual projects in the pilot program. Per the implementing guidance,  the FFRDCs and DOD sponsors agree to collect and provide information  about the enrolled projects. For example, DOD sponsors must provide the  Laboratories and Personnel Office quarterly updates on a project’s  progress obtaining data and, once the project is complete, information on  the results of its access to sensitive data under the pilot program.

Over Half of DOD’s FFRDCs Used the Pilot for A Small Percentage of Projects and Reported Benefits of Participating

As of September 2019, Six of 10 DOD FFRDCs Participated in the Pilot and Enrolled 33 Projects

Six of 10 DOD FFRDCs elected to participate in the pilot program during  its first 21 months (figure 1).

According to FFRDC representatives, the decision regarding whether an  FFRDC would participate in the pilot program primarily depended on two  factors: (1) the data needs of the FFRDC’s projects and (2) the ability of  FFRDCs to access necessary data without the pilot program.  Representatives from the six participating FFRDCs told us they elected to  participate because they required access to sensitive data and, in some  cases, lacked viable options for obtaining that data. For five of these  FFRDCs, representatives said their researchers had identified specific  projects for which they were interested in using the pilot to gain access to  data sources with sensitive data from numerous contractors.

Representatives from the four nonparticipating FFRDCs said that the  existing processes the FFRDCs have in place provide the access they  need for their projects. For example,

Lincoln Laboratory representatives said their researchers are often  working with an individual program or working to advance a specific  technology; therefore, their work is generally with a limited number of  contractors. In cases where they have needed access to sensitive  data to do this work, they have executed a blanket nondisclosure  agreement with their primary sponsor and, in some cases, have  executed more tailored nondisclosure agreements with companies  when obtaining information directly from a defense contractor.

Aerospace has a blanket nondisclosure agreement with the Air Force  Space and Missile Systems Center and the center included a  provision in its contracts that requires its prime contractors to directly  share information with the FFRDC.

A Center for Communications and Computing representative said  existing processes already provide the access they need for their  projects. According to the FFRDC’s sponsor, its work focuses more  on technological development rather than acquisitions analysis.

Participating FFRDCs reported 33 projects enrolled in the pilot program  from January 2018 through September 2019. Pilot projects represented  about 1.5 percent of these FFRDCs’ total number of projects as of June  2019. Of the projects enrolled in the pilot program, 11 were complete  and 10 were ongoing as of the end of September 2019. In addition, 11  projects enrolled in the pilot initially, only to realize they did not require  access to the requested data and thus were removed. One project was  put on hiatus pending a decision about whether it will continue. Table 2  summarizes the status of the projects in the pilot program as of  September 2019.

According to DOD officials, the fiscal year 2018 reorganization of OUSD    for Acquisitions, Technology and Logistics into two offices, coupled with  changes in leadership, shifted attention away from the pilot program  design and implementation. For example, an official from the Acquisition  and Sustainment office also told us it missed an opportunity to conduct  outreach with its FFRDC because the office did not hold its biannual  meetings in 2017 or 2018 due to the reorganization. In these biannual  meetings, he explained, they would have discussed the department’s  future research priorities and how the pilot program may have helped.  This official—who was involved in the pilot’s implementation—also noted  that the shift in attention meant they did not engage with the offices that  maintain the data repositories as fully as they would have liked. We found  it took sponsors and FFRDCs from a few weeks to 7 months to resolve  questions about pilot program requirements and update the FFRDCs’  sponsoring agreements to incorporate the pilot protections. During that  time, FFRDCs were unable to move forward with certain analyses for  their proposed projects.

DOD Officials and FFRDC Representatives Reported Benefits from Using the Pilot

Of the six projects we selected for further review (shown in table 3), four  have been completed and FFRDCs reported benefits from their pilot  program participation. The two remaining projects are on hiatus or  removed.

DOD officials and representatives from the four completed projects  shared with us the following benefits:

Systems and Analyses Center assessment of the U.S. munitions  defense industrial base capacity. Portions of the research required  access to sensitive data about the availability and production levels of  manufacturing parts for a large number of contractors and suppliers.  The FFRDC researchers used these data in their microlevel  assessments of the manufacturing capacity and supply chain  resiliency of the U.S. defense munitions industrial base. They said  they were able to provide DOD’s Industrial Policy office with a more  complete picture by combining these microlevel analyses with broader  analyses of employment trends and economic outputs. A DOD  industrial policy official who requested the work also said that the  analysis enabled her office to meet an executive branch reporting  requirement, which DOD did not have the manpower to conduct.

National Defense Research Institute support for analysis of  munitions industrial base. FFRDC researchers worked in  collaboration with government officials to perform analyses on the  adequacy of the munitions and missiles industrial base using  government-held data from prime contractors and subcontractors. For  example, the researchers supported working groups examining  propulsion and chemicals in munitions and provided analysis for a  report to Congress on solid rocket motors. The DOD official that  requested the work and National Defense Research Institute  representatives said that, without the pilot, the FFRDC would not have  been able to access the data used to support DOD in these efforts.  The official also noted that in this case the FFRDC helped fill a gap in  DOD’s workforce to meet a congressional reporting requirement.

National Security Engineering Center and Software Engineering  Institute analysis of software acquisitions practices. FFRDCs  supported a Defense Innovation Board study that aimed to identify  correlations between software complexity, cost, and schedule  evolution. FFRDC researchers’ access to and use of the data  provided important insights about the quality and reliability of the  department’s data. Specifically, DOD gained further insight into the  kinds of software data the department holds and the significant gaps  that would need to be addressed to improve overall DOD-held data  quality. The Defense Innovation Board’s report included findings  related to the quality of the software data accessed and analyzed by  these two FFRDCs.

Project AIR FORCE assessment of contractor risk. According to  FFRDC representatives and an Air Force official involved in the work,  the pilot program facilitated the FFRDC’s access to sensitive data  held by the Defense Contract Management Agency that researchers  used to identify early indicators of contractor performance risks. In  response to the results of this work, the Air Force has funded a follow- on project to further research the potential of data analytics to provide  early indicators of challenges in contract execution, according to an  Air Force official involved with project.

In addition to the benefits at the project level, several DOD sponsor  officials and FFRDC representatives also noted the benefits of using the  streamlined nondisclosure agreement process to accomplish their work.  According to several DOD sponsor officials and FFRDC researchers we  spoke to, completing the requested analysis without the pilot program  would have required individual nondisclosure agreements with hundreds  of individual contractors and suppliers. Systems and Analyses Center  representatives said this would have been essentially impossible, and  would have prevented researchers from completing important parts of the  analyses. In another case, a Software Engineering Institute  representative told us that, before the pilot, their team could not access  software data when attempting to complete a 2017 project involving DOD  software costs and production time frames. For that project, the DOD  organization responsible for the data repository had recommended  researchers send out a data request letter to each of the contractors with  data in the system. Researchers sent out roughly one hundred requests  to contractors for permission, but received no responses. They pointed  out this was in part because contractors have no incentive to respond to  an FFRDC’s request for access to their data. As a result, Software  Engineering Institute was unable to use updated data for the 2017 report.

While several sponsoring agency officials noted benefits of using the data  for analyses to inform key program decisions, they also noted that a  causal relationship between the pilot program activities and acquisition  process improvements would be hard to establish, in part due to the  length of time needed for projects to effect change. DOD officials  responsible for two completed projects examining the munitions industrial  base said they expect the analyses performed will lead to improved  acquisition processes but that it would take many years to see the  benefits. Specifically, they said the FFRDCs’ work helped identify areas  for improvement in the department’s budget and acquisition strategy to  better signal future demand to its lower tier munitions industrial base  suppliers. In addition to noting these expected improvements, several  DOD officials also acknowledged that expanding access of sensitive data  to more people increases the potential for unauthorized use or disclosure  but said that the pilot program put in place important protections to help  mitigate these risks.

DOD’s Pilot Established Protections for Accessing Sensitive Data but Did Not Establish Procedures to Verify Compliance

DOD’s guidance to implement the pilot program outlined protections the  FFRDCs must agree to, in order to guard against unauthorized disclosure  or use of sensitive data, and required that these protections be  incorporated into the sponsoring agreements between the FFRDCs and  the DOD sponsor. However, we found some instances where details of  the required protections were not incorporated into the agreements. We  also found that the Laboratories and Personnel Office, which is  responsible for managing the pilot program, does not have a procedure to  verify whether protections were implemented, in part because it has not  developed a process for doing so. Table 4 explains the protections.

Some of these protections were already part of the FFRDCs’ business  operations, while others are new. For example, the prohibition on their  use of sensitive data to compete against a third party was already a  fundamental aspect of FFRDCs’ role in supporting DOD. Similarly,  participating FFRDC representatives told us that certain protections, such  as implementing nondisclosure agreements and training, required only  small adjustments to their existing procedures. However, the pilot’s  financial disclosure program, annual certifications by parent  organizations, and instructions for researchers to notify contracting  officers of employment offers when supporting source selection decisions  were new and specific to the pilot, according to a DOD official involved in  the pilot’s implementation.

We found that not all the details of protections were incorporated into the  sponsoring agreements we reviewed. According to DOD’s implementing  guidance, to participate in the pilot program, FFRDCs must agree to and  follow these protections, which are to be incorporated into FFRDCs’  sponsoring agreements. We found that all six participating FFRDCs’  sponsoring agreements were updated and that most of the protections  were incorporated. However, none included the instructions for FFRDC  personnel involved in source selections to notify contracting officers if  they are contacted about employment by an entity whose proposal is  being evaluated and recuse themselves. We also found that the  sponsoring agreements omitted one of the three officials that should be  notified in the event of a Trade Secrets Act violation. These details were  also not included in the templates DOD provided sponsors to use when  updating FFRDC sponsoring agreements. When we raised these gaps  to the attention of the DOD office responsible for managing the pilot  program, the officials we spoke with were unaware of these omissions.

In addition, the Laboratories and Personnel Office has not taken steps to  ensure that another protection—the certification of the annual review of  financial disclosure forms—has occurred, even though it was  incorporated into the sponsoring agreements. The implementing guidance  states that FFRDCs’ parent organizations must certify the annual review  of financial disclosure forms and archive these forms for 6 years.  However, only two FFRDC parent organizations provided us with this  certification. According to representatives from parent organizations of  the other four FFRDCs, the review of financial disclosures is generally  performed as part of their conflict of interest programs. They review the  disclosures on an annual or rolling basis when researchers are assigned  to new projects but had not certified, as the sponsoring agreements  require, that they have taken this step for the pilot program. We found that  the Laboratories and Personnel Office had not taken steps to verify  FFRDC parent organization compliance with this protection, such as  collecting or reviewing the certification. When we raised this gap to the  attention of the DOD office responsible for managing the pilot program,  the officials were unaware of the missing annual certifications. By not  ensuring the annual review is occurring, DOD has limited information  about FFRDCs’ adherence to this pilot program protection.

The pilot’s implementing guidance also states that, before government  personnel provide access to sensitive data, the FFRDCs and researchers  must have addressed these protections. However, the Laboratories and  Personnel Office has not taken steps to ensure it is done. In our review of  the six specific projects, we found that different people were checking that  some of the protections were in place. For example,

For two of the six projects, a primary sponsor official had a copy of the  FFRDC addendum, and collected and reviewed the nondisclosure  agreements and certifications of financial disclosure for individual  researchers on each project.

For a third project, a DOD official in the office that requested the  project told us she confirmed that FFRDC researchers working on the  project were part of the pilot program and told the official from the  data repository that he could share information with the researchers.

For the remaining three projects, representatives for a data repository  that provided researchers with data access told us they confirmed that  the addendum was incorporated into the sponsoring agreement and  that researchers had the individual protections, such as a  nondisclosure agreement, in place before providing access to the  data.

Standards for internal controls in the federal government state that  responsibilities for control activities, such as sponsors ensuring the  protections are incorporated into the agreements and that FFRDCs are  following these protections, should be documented through policy and  procedures. Without a process that includes clearly defined roles and  responsibilities to ensure the protections are followed, DOD cannot  ensure that its goal to safeguard sensitive data is achieved.

DOD Is Collecting Some Pilot Information but Lacks a Plan for Evaluation DOD Collects Information from FFRDCs Quarterly but Does Not Ensure Comprehensive Reporting

DOD established what information sponsors must collect about the  projects enrolled in the pilot in its implementing guidance to sponsors and  notified FFRDCs about these responsibilities. The requirements include: pre-action information to be collected when the project is enrolled in  the pilot, which includes basic details about the project, the data  required, and planned analysis; quarterly status updates, which include progress obtaining access  to sensitive data and any challenges or barriers to access; and post-action information regarding the results of pilot access when  the project’s analyses are completed, which includes a summary of  how the pilot supported FFRDC research, and any benefits accrued to  DOD from pilot participation.

The implementing guidance also instructs sponsors to collect information  about the project’s results again 6 months after the project is completed.

The Laboratories and Personnel Office, which is responsible for  managing the pilot program, sends an email quarterly requesting that  sponsors submit information. However, we found that the Laboratories  and Personnel Office did not receive pre-action information from 11 of the  33 projects in the pilot program. For example:

Systems and Analyses Center has not submitted pre-action  information for six of its eight projects. An official from the Systems  and Analyses Center’s primary sponsor office told us that in his view,  pre-action information can be obtained by other means, and he had  not requested it.

Project AIR FORCE and Arroyo Center have submitted project pre- action information for five projects to the FFRDCs’ primary sponsors:  Air Force and Army, respectively. However, the primary sponsors  have not provided this information to the Laboratories and Personnel  Office. An Air Force official explained that he gets a request from the  Laboratories and Personnel Office for the quarterly reports, but not  the pre-action information, and thus had not provided it.

In addition, the office did not collect a quarterly report for three projects in  the pilot, and, as of September 2019, two completed projects had passed  the 6-month post-completion time frame and only one had submitted  post-action information.

These gaps in reporting have occurred because the Laboratories and  Personnel Office is not monitoring the project information it receives to  ensure sponsors are submitting all required reporting. DOD’s  implementing guidance states that primary sponsors will collect and  submit this information for each project enrolled in the pilot program.  Further, GAO’s leading practices for pilot design state, among other  things, that a well-designed pilot program should have a clear approach  to gathering information for the purpose of supporting the future  evaluation of the pilot and tracking the pilot program’s implementation and  performance. Consistency in collecting pre-action, quarterly, and post- action reports is important because each contains different information,  which could be useful for the department to track the pilot program’s  progress and in an evaluation of the pilot program. For example, without  the pre-action information from Systems and Analyses Center, Arroyo  Center, and Project AIR FORCE, the Laboratories and Personnel Office  will not have general descriptions of their pilot projects or information  about the kinds of data these FFRDCs initially planned to access. Without  complete information, DOD will not be able to effectively evaluate the pilot  program and inform future decisions about the program’s status.

DOD Established Pilot Goals but Does Not Have a Plan to Evaluate Its Results

We found that DOD followed some but not all of the leading practices for  evaluating its pilot program. According to GAO’s leading practices for pilot  design, a well-developed and documented pilot program can help ensure  that agency assessments produce information needed to make effective  program and policy decisions. Such a process enhances the quality,  credibility, and usefulness of evaluations, in addition to helping to ensure  that time and resources are used effectively. Five leading practices form a  framework for effective pilot design and evaluation. (See figure 2.)

We found DOD generally addressed the first of the leading practices of  pilot design—establish objectives—by establishing goals for the pilot  program, summarized below.

Make sensitive data previously restricted or unavailable available for  analysis.

Use sensitive data in accordance with the FFRDC contract.

Document risks or costs of FFRDC access to sensitive data.

Gain analytic value from FFRDC access to sensitive data.

Demonstrate benefits to government from sharing sensitive data.

Inform future actions for making FFRDC access to sensitive data  permanently available.

However, DOD has not fully addressed the other leading practices.  Specifically, we found that the Laboratories and Personnel Office does  not have: a plan that (1) includes an assessment methodology to ensure DOD is  collecting the correct information to evaluate whether the pilot has met  the department’s goals and (2) defines how DOD will use the  information collected to evaluate the implementation and performance  of the pilot program, when the evaluation will take place, and by  whom; a plan for identifying or documenting lessons learned; and a plan for gathering input from stakeholders, such as DOD sponsors,  FFRDCs, and officials from DOD’s data repositories, for the pilot  program’s evaluation.

According to officials involved in its implementation, DOD did not consider  creating such plans when developing the pilot program. The pilot program  guidance, however, stated that information collected would be used for  the department to assess the ongoing efficacy of the pilot program and  GAO’s evaluation. These officials explained that, when the pilot’s  guidance was formulated, the department was in the process of  reorganizing the former Acquisition, Technology, and Logistics Office and  they pointed out that GAO was to do the assessment of the pilot  program. They said the reason the program collected information—such  as quarterly reports—was to inform our review. Thus, they had no plan to  assess the information collected and no plans to talk to stakeholders or to  collect and share lessons learned.

While our review occurred during pilot implementation, an evaluation of  the pilot conducted by DOD after more projects are completed would  provide an opportunity to identify lessons learned and gather valuable  input from stakeholders—such as the offices that manage the data  repositories and the sponsors requesting the projects. We found cases  where FFRDC researchers had problems accessing data and where  gathering this input from stakeholders involved with pilot projects would  have been useful for DOD. For example, some FFRDC researchers  described barriers when trying to gain access to certain government- and  department-wide databases. In one case, Project AIR FORCE  researchers reported not being able to access information in the  Electronic Document Access database and various databases containing  contractor performance information because researchers lacked military  or government email addresses. Further, some of the databases that  FFRDC researchers and a DOD official said would be useful are not  owned by DOD. We found that guidance for one such database explicitly  prohibits disclosure of contractor evaluation data to any contractor or non- government entity. In addition, the researchers were able to gain only  partial access to DOD’s Acquisition Information Repository (a database  containing acquisition documents for DOD’s major weapons programs)  and, as a result, were unable to access individual documents, such as  program assessment reports. They told us the repository is set up such  that the researchers must request access to individual documents directly  from document owners, who set permissions when uploading documents,  rather than from a central source that can grant access across the  repository. Without further evaluation of the pilot, DOD is missing an  opportunity to benefit from gathering input from its stakeholders and  identifying lessons learned, such as learning and understanding more  about these barriers to accessing certain databases.

There is still time for DOD to develop an evaluation plan with elements  described in our leading practices. The pilot program ends December 21,  2020. Our review comes at a time when 11 of the overall 33 projects have  been completed; therefore, information exists to report on outcomes.  Officials and representatives from the Laboratories and Personnel Office  and participating sponsors and FFRDCs expressed a continued need for  access to the sensitive data. Without an evaluation plan, DOD will have  difficulty determining the effectiveness of the pilot to meet its goal of  accruing more analytic value for the department while also safeguarding  sensitive data.

The FFRDC pilot program has already provided DOD with some benefits,  as a few FFRDCs have reported success in completing analysis that  would not have been possible without it. However, in implementing the  pilot, DOD has room for improvement. A key control of this pilot that  provides access to sensitive data is ensuring protections are in place to  prevent improper disclosure. Another control is to establish a process to  ensure these protections are followed, yet the responsible office within  DOD has not done so. Further, despite the fact that the pilot is past the  midpoint of implementation, this office still has an opportunity to develop a  plan on how to evaluate it. But to do this, it must develop a mechanism to  ensure it is collecting complete information on the pilot activities. Ensuring  comprehensive reporting and implementing a well-developed evaluation  plan will help DOD understand and articulate the benefits the department  has accrued because of FFRDC’s access to sensitive data. Further,  through identifying lessons learned and obtaining stakeholder input, the  Laboratories and Personnel Office has an opportunity to better  understand the challenges FFRDCs and the department face when  attempting to access and use sensitive data included in government- and  department-wide databases. Such an evaluation could help inform  Congress’ decision whether to extend, make FFRDC access permanent,  or end the pilot.

We are making the following six recommendations to the Department of  Defense:  The Under Secretary of Defense for Research and Engineering should  direct the Laboratories and Personnel Office to take steps to ensure that  the details of the pilot program’s data protections are incorporated into the  existing agreements. (Recommendation 1)

The Under Secretary of Defense for Research and Engineering should  direct the Laboratories and Personnel Office to take steps to ensure that  the FFRDCs and sponsors are implementing the pilot program’s  protections for sensitive data. (Recommendation 2)

The Under Secretary of Defense for Research and Engineering should  direct the Laboratories and Personnel Office to establish a monitoring and  oversight mechanism to ensure that primary sponsors submit complete  information on pilot projects, as required by DOD’s guidance for the pilot  program. (Recommendation 3)

The Under Secretary of Defense for Research and Engineering should  direct the Laboratories and Personnel Office to develop a plan that  outlines the methodology by which DOD will assess the pilot and how and  when information collected will be analyzed to evaluate the pilot program.  (Recommendation 4)

The Under Secretary of Defense for Research and Engineering should  direct the Laboratories and Personnel Office to develop a plan to identify and  evaluate lessons learned from the pilot program. (Recommendation 5)

The Under Secretary of Defense for Research and Engineering should  direct the Laboratories and Personnel Office to develop a plan for obtaining input  from stakeholders on the pilot program. (Recommendation 6)

We provided a draft of this product to DOD for comment. DOD provided a  letter response, reproduced in Appendix I. DOD agreed with our  recommendations and described actions that it intends to take in  response. We also provided excerpts of this product to FFRDCs for  comment, of which three provided technical comments that we  incorporated as appropriate.

We are sending copies of this report to the appropriate congressional  committees and the Secretary of Defense. In addition, the report is  available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-4841 or makm@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in Appendix II.

Appendix I: Comments from the Department of Defense

In addition to the contact named above, Tatiana Winger (Assistant  Director), Leslie Ashton (Analyst-in-Charge), Evan Nemoff, Tanya Waller,  Jenny Chanley, Laura Greifner, Christine Pecora, and Roxanna Sun  made key contributions to this report.

DOD sponsors research and development activities at 10 Federally Funded Research and Development Centers that provide innovative solutions to national security threats.

These centers sometimes need to access sensitive DOD data (such as proprietary information from DOD’s contractors), which can be a long, time-consuming process. DOD began a pilot in 2017 to streamline these centers’ access to sensitive data.

We found that some centers reported easier access to sensitive data during the pilot. However, DOD has not consistently collected information on the pilot or developed a plan to evaluate it.