National Science Foundation: Cost and Schedule Performance of Major Facilities Construction Projects and Progress on Prior GAO Recommendations

NSF supports the design, construction, and operations of major facilities projects–science and engineering research infrastructure such as telescopes and research vessels that typically have construction costs of at least $70 million and may take many years to design and construct. The agency oversees the performance of each project against an authorized total project cost and schedule. NSF currently has four projects under construction at a combined authorized cost of $1.6 billion and two additional projects in design. Prior GAO reports reviewed NSF's cost estimating and schedule policies, as well as project management expertise of its oversight workforce.

Senate Report 114-239 and House Report 114-605 included provisions for GAO to review NSF's major facilities projects. Among other objectives, this report (1) describes the cost and schedule performance of NSF's ongoing major facilities projects and (2) assesses the extent to which NSF addressed prior GAO recommendations related to its management of major facilities. GAO analyzed NSF policies and documents for projects in design and construction, interviewed agency officials, and compared NSF's processes to best practices identified in prior GAO work.

Since GAO's March 2019 report on the status of its major facilities projects, the National Science Foundation (NSF) had no increases to the authorized total project costs or schedules for its four projects under construction (see figure):

The Daniel K. Inouye Solar Telescope was on track to be completed within its $344.1 million cost and June 2020 completion date.

NSF was evaluating options for reducing the scope of the Vera C. Rubin Observatory (previously the Large Synoptic Survey Telescope), which it believed might be necessary to keep the project within its $473 million cost and October 2022 completion date.

Construction of a second Regional Class Research Vessel began in September 2019 and was anticipated to begin on a third and final vessel in March 2020 at a combined cost of $365 million.

The Antarctic Infrastructure Modernization for Science entered the construction phase in February 2019 at a cost of $410.4 million.

NSF fully implemented two of the six prior GAO recommendations including revising policies for estimating the costs of major facilities projects and revising the Vera C. Rubin Observatory's schedule to better meet best practices. NSF took steps to address but has not fully implemented the remaining four recommendations on the agency's oversight of major facilities.

NSF agreed with and has taken initial steps to address four open recommendations from GAO's prior work, including to revise policies for developing schedules and to ensure the sharing of lessons learned for major facilities projects. NSF needs to complete additional steps to fully address the recommendations.

The National Science Foundation (NSF) supports the design,  construction, and operations of various major facilities projects—science  and engineering research infrastructure such as telescopes and research  vessels that have a construction cost of at least $70 million. These  projects are designed in collaboration with the scientific community in  order to respond to scientific needs. NSF uses cooperative agreements  and contracts for the projects throughout their life cycles, including the  design, construction, and operations stages. Award recipients of the  cooperative agreements and contracts, which may include universities,  nonprofit associations, and companies, manage the projects’ day-to-day  construction and operations activities. NSF typically funds construction  efforts for major facilities projects through its Major Research Equipment  and Facilities Construction (MREFC) account. In fiscal years 2018 and  2019, NSF received appropriations of $183 million and $296 million,  respectively, for the MREFC account.

In June 2018, we reported on NSF’s procedures for estimating  construction costs and developing schedules of major facilities projects.  We found that NSF’s procedures met many best practices for cost  estimating but not those for developing project schedules. We  recommended that NSF revise its policies for estimating and reviewing  the costs and schedules of major facilities projects to better incorporate  best practices in GAO’s cost and schedule guides. In March 2019, we  reported on several aspects of NSF’s management of major facilities  projects and made further recommendations. For example, we found that  NSF took some steps to assess project management expertise among its  staff, but did not take certain additional steps. We recommended that  NSF assess the agency’s major facilities oversight workforce to identify  any project management competency gaps, develop a plan to address  any gaps and time frames for doing so, and monitor progress in closing  them. NSF concurred with the recommendations in both the 2018 and  2019 reports.

In its fiscal year 2020 budget request, NSF proposed funding mid-scale  research infrastructure projects—projects costing between $20 million  and $70 million—through the MREFC account. NSF has reported a  significant community demand for mid-scale projects and has identified  mid-scale research infrastructure as one of NSF’s 10 Big Ideas for its  future investments, as the scientific community is increasingly relying on  such infrastructure and facilities projects to provide innovative  approaches for solving the community’s most pressing problems.  However, NSF did not previously have a formal mechanism to fund and  centrally manage a program for these projects outside of its research  directorates, which support research and education in a wide range of  science and engineering disciplines. The American Innovation and  Competitiveness Act of 2017 required NSF to develop a strategy to  support mid-scale research infrastructure projects. In response, the  National Science Board (NSB) issued a report confirming the need for a  program to fund mid-scale projects and recommended that NSF consider  funding mid-scale projects through its MREFC account as one option.

Senate Report 114-239 and House Report 114-605, issued in 2016,  included provisions for us to review projects within NSF’s MREFC  account, which now includes mid-scale projects as well as major facilities  projects. This report, our third in response to the Senate and House  report provisions, (1) describes the cost and schedule performance of  NSF’s major facilities projects in construction and the status of projects in  design since issuance of our 2019 report; (2) assesses the extent to  which NSF has implemented prior GAO recommendations for improving  its oversight of major facilities; and (3) identifies the steps NSF has taken  to make awards to and provide guidance on oversight for mid-scale  research infrastructure projects.

To describe the cost and schedule performance of NSF’s major facilities  projects in construction and the status of projects in design since our  2019 report, we reviewed project documents and NSF’s written  responses to our questions about projects which were under  construction—the Daniel K. Inouye Solar Telescope (DKIST), the Vera C.  Rubin Observatory (Rubin Observatory) (formerly named the Large  Synoptic Survey Telescope), the National Ecological Observatory  Network (NEON), the Regional Class Research Vessels (RCRV), and the  Antarctic Infrastructure Modernization for Science (AIMS) projects—and  projects which were in design at the time of our review—the Large  Hadron Collider High Luminosity Upgrade (HL-LHC) and the Leadership- Class Computing Facility (LCCF). We reviewed, for example,  cooperative agreements, progress reports, risk reports and risk registers,  documentation on available scope reduction options, and other NSF,  award recipient, and external panel project documents, as applicable,  related to project cost, schedule, scope, and risks. We assessed the  reliability of project data by obtaining supporting documentation when  possible, conducting routine checks for consistency with other information  contained in the documentation provided by NSF, and clarifying any  discrepancies with NSF project officials. Through this process, we  determined that the project data were sufficiently reliable for our purpose  of describing information available on the projects’ cost and schedule  performance and current status.

To assess the extent to which NSF has implemented prior GAO  recommendations related to its oversight of major facilities, we reviewed  NSF documentation and NSF’s written responses related to actions the  agency took to implement these recommendations. We took additional  steps to assess NSF’s implementation of certain recommendations.  Specifically, for our recommendation on revising policies for estimating  the costs of major facilities to incorporate the best practices in GAO’s cost  guide, GAO staff with cost estimating expertise compared the updated  procedures documented in NSF’s policies with the best practices.  Similarly, for our prior recommendation to revise the Rubin Observatory’s  schedule to incorporate the best practices contained in GAO’s schedule  guide, GAO staff with schedule expertise compared the updated schedule  to the best practices. For both of these recommendations, we focused on  cost estimating and scheduling best practices that we found to be  minimally met or partially met in our 2018 and 2019 reports. We did not  address best practices that we assessed as substantially or fully met in  our prior reports.

In comparing NSF’s procedures and the Rubin Observatory’s schedule to  best practices in GAO’s cost and schedule guides, we used the following  ratings:  “Fully met” means there was complete evidence that satisfied the  entire best practice.  “Substantially met” means there was evidence that satisfied a large  portion of the best practice.  “Partially met” means there was evidence that satisfied about half of  the best practice.  “Minimally met” means there was evidence that satisfied a small  portion of the best practice.  “Not met” means there was no evidence that satisfied any of the  elements of the best practice.

After conducting our initial assessments of NSF’s procedures and the  Rubin Observatory’s schedule, we shared our draft analyses with NSF  officials to provide the agency with an opportunity to comment. Based on  their comments and additional information provided, we revised our draft  assessments, as appropriate, to produce the final assessments.

To identify the steps NSF has taken to make awards to and provide  guidance on oversight for mid-scale research infrastructure projects, we  reviewed documentation pertaining to NSF’s solicitation for mid-scale  projects in order to understand NSF’s selection criteria and award time  frames. We also reviewed NSF’s fiscal year 2020 budget request to  identify how NSF plans to fund such projects. In order to describe NSF’s  plans for oversight of mid-scale projects, we reviewed available guidance,  such as NSF’s Major Facilities Guide, to understand what policies NSF  already has in place to oversee the projects. We compared such  guidance with NSF’s policies for overseeing major facilities projects to  describe any differences between NSF’s guidance for mid-scale and  major facilities projects. Finally, we interviewed relevant NSF officials to  understand how NSF incorporated existing guidance for major facilities  projects into its guidance for mid-scale projects.

We conducted this performance audit from May 2019 to April 2020 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Stages in the Life Cycles of NSF’s Major Facilities Projects

Each major facilities project has a sponsoring office from within NSF’s  seven research directorates. The sponsoring office assesses the  scientific merit of a potential project, proposes projects for funding  through NSF’s MREFC account, and is responsible for overseeing the  project during the following five stages of its life cycle.

Development. Initial project ideas emerge, and a broad consensus is  built within the relevant scientific community for the potential long-term  needs, priorities, and general requirements for research infrastructure  that NSF may consider funding.

Design. Entrance into this stage occurs when the NSF Director  approves the proposed research infrastructure as a national priority  and the sponsoring directorate makes an award (either through a  cooperative agreement or contract) for developing detailed project  cost, scope, and schedule for possible construction. This stage is  divided into conceptual, preliminary, and final design phases.  According to NSF documentation, the goal of the conceptual design  phase is to create a comprehensive design that clearly articulates  project elements that NSF will consider, such as a description of  research infrastructure and technical requirements, a concept of  operations, and an initial risk analysis, among others. The preliminary  design phase further develops projects through the formulation of a  site-specific scope, an accurate budget estimate, a revised and  updated project execution plan, and other deliverables to establish a  project baseline. In the final design phase, a candidate project will  refine cost and contingency estimates, complete recruitment of key  staff needed to undertake construction of the project, and develop the  necessary documentation needed to undergo final design review. A  candidate project will exit the design stage and enter the construction  stage after a successful review by the NSF director and other key  stakeholders of its project execution plan and authorization of its not- to-exceed total project cost by the National Science Board, as  discussed below.

Construction. The construction stage begins when NSF makes  awards to external recipients for acquisition or construction of  research infrastructure. Such awards generally take the form of  cooperative agreements, although NSF occasionally uses contracts,  according to agency officials. The policies and procedures in NSF’s  Major Facilities Guide apply to research infrastructure projects  regardless of the award instrument employed. According to NSF’s  Major Facilities Guide, the transition from construction to operations  could be a single acceptance event or multiple events depending on  the nature of the project, and many projects require an integration and  testing phase, followed by a commissioning phase to bring the facility  up to the design level of operational readiness. The construction stage  ends after final delivery and acceptance of the defined scope of work  and facility performance per terms of the award instrument.

Operations. The operations stage includes the day-to-day work  necessary to operate and maintain the research infrastructure  (including refurbishment or upgrade activities) and to perform  research. Operations awards, which are separate from construction  awards, may be made to the construction award recipients or to a  different entity. Depending on the project, initial operations may begin  before completion of construction. Integration and testing activities  may continue during the operations stage, depending upon the  complexity and time needed to reach design specifications.

Divestment. Divestment can include the transfer of the research  infrastructure to another entity’s operational and financial control or  the decommissioning of the research infrastructure, including its  complete deconstruction and removal. NSF generally decides to  divest when the agency or the scientific community determines that  the facility is no longer considered an operational priority with regard  to advancing science, according to NSF’s Major Facilities Guide.

NSF funding for the development, design, operations, and divestment  stages generally comes from the sponsoring directorate. Funding for the  construction stage generally comes from the MREFC account. However,  if the sponsoring directorate funds construction, the policies and  procedures in NSF’s Major Facilities Guide apply if total project costs  meet the definition of a major multiuser research facility project under the  American Innovation and Competitiveness Act—that is, if the costs  exceed $100 million or 10 percent of the responsible directorate’s annual  budget, whichever is less.

NSF Oversight of Major Facilities Projects

NSF has established an oversight structure for major facilities projects  that includes offices from across the agency (see fig. 1). This includes the  National Science Board, a policy and advisory body that is part of NSF  and consists of the NSF Director and 24 members, drawn from industry  and universities, who represent a variety of science and engineering  disciplines. The NSF Office of the Director and the National Science  Board provide high-level, ongoing oversight of major facilities projects,  including the approval of new projects to be included in NSF’s annual  budget request.

Within NSF’s Office of Budget, Finance, and Award Management, the  Large Facilities Office (1) develops business-related oversight policies for  all life-cycle stages with a focus on the design and construction stages  and (2) provides assistance on nonscientific and nontechnical aspects of  project planning, budgeting, implementation, and management. To that  end, the office maintains the Major Facilities Guide, which contains NSF  policies for agency staff and recipients on the planning, management, and  oversight of major facilities. Prior to requesting the National Science  Board’s authorization to include a proposed project in a future NSF  budget request, the Large Facilities Office provides independent  assurance—apart from the sponsoring office and external panels—that  NSF oversight processes have been followed, project plans are  construction ready, and construction and operations budgets are justified.  In addition, it prepares a bimonthly status report for NSF leadership on all  ongoing major facilities in construction and candidate projects in design.

NSF also uses external panels of experts to review projects at several  points during their life cycles. An external panel may first review a project  proposal during the development stage. Separate panels then review the  project at the culmination of each of its design phases. In addition, an  external panel periodically reviews each project during both construction  and operations; according to NSF officials, those reviews are generally on  an annual basis. According to NSF officials and policy documents, the  agency selects panelists based on the questions that need to be  addressed and on the type of review taking place. For example, for  panels charged with reviewing all aspects of a project, NSF will generally  select panelists to represent the academic and broader national or  international research community, as well as experts in administrative  aspects of facilities and project management, according to NSF’s Major  Facilities Guide. Furthermore, the responsible directorate and the Large  Facilities Office jointly manage the external panel review process and  other NSF staff may attend as observers, according to the agency’s Major  Facilities Guide. Each panel is to provide NSF with a report summarizing  the review’s findings and any recommendations to NSF.

Components of Construction Costs and Schedules of Major Facilities Projects

Under NSF’s major facilities construction process, the recipients of design  awards develop construction cost and schedule estimates for projects  and submit them to NSF for review. In particular, after a project’s final  design review, the National Science Board authorizes a not-to-exceed  award amount and an award duration. According to NSF officials, this  finalizes the initial budget request previously submitted to Congress after  the project’s preliminary design review. The not-to-exceed award amount  that the National Science Board authorizes is the amount against which  NSF measures cost increases to implement its no cost overrun policy.

NSF’s Major Facilities Guide defines two components that together make  up the total project cost and schedule for the construction of major  facilities projects. The total project cost awarded in a project’s  construction agreement may be less than the not-to-exceed cost but not  more. These components of the total project cost and schedule are the  following:

Performance measurement baseline. During design, the cost,  scope, and schedule are refined and eventually become the project  baseline. Once the baseline has been authorized and included in a  construction award, it is known as the performance measurement  baseline. NSF documents the performance measurement baseline in  the terms and conditions of the award instrument and requires that  any changes to it be made through a formal change control process.  The performance measurement baseline does not include the  project’s budget or schedule contingency.

Contingency. This is an amount of budget or time for covering the  cost increases or delays that would result if foreseen project risks  were to occur. During development of a total project cost estimate, the  timing and impacts of such risks are uncertain. As a project  progresses, the impacts of risks that materialize may exceed the cost  or schedule in the performance measurement baseline and lead to  use of the project’s budget or schedule contingency. According to  NSF’s Standard Operating Guidance on budget contingency, it is  likely no contingency will be left over by the end of a project because  all of it will have been used during normal execution of the project to  manage known risks and uncertainties. NSF approval is needed when  use of contingency exceeds certain project-specific thresholds, which  are described in the project’s execution plan and codified in the  award.

In this report, we identify total project costs for the construction of major  facility projects which were developed during the design phase based on  the latest estimates available from NSF officials; those estimates are  subject to change before construction awards are made. For projects  under construction, we identify total project costs based on the amounts  awarded in the cooperative support agreements for construction and the  not-to-exceed amounts authorized by the National Science Board. Only at  the end of the projects—when construction is complete and the awards  have been closed out—will the final total project costs be known.

In addition to the performance measurement baseline and budget  contingency, a project’s not-to-exceed cost that the National Science  Board authorized may include the following:

Fee. NSF may provide recipients the opportunity to earn a fee  (formerly referred to by NSF as a management fee) for major facilities  projects. According to NSF’s Standard Operating Guidance on  negotiation, award, and payment of a fee, such a fee can stimulate  efficient performance.

Management reserve. NSF, not the award recipient, holds  management reserve to manage budget uncertainties, unforeseeable  events, and risks that the recipient is not able to manage, according to  NSF officials. According to agency officials and the Major Facilities  Guide, NSF does not hold a management reserve except in rare  circumstances.

NSF’s No Cost Overrun Policy for Major Facilities Projects

Since February 2008, NSF has had a policy to manage cost overruns on  major facilities projects. Under this policy, the cost estimate developed  at the preliminary design review should have adequate contingency to  cover all foreseeable risks. Any cost increases not covered by  contingency are generally to be accommodated by reductions in scope.  Figure 2 provides a breakdown of the total project cost components in  relation to the not-to-exceed award amount. NSF officials said that under  this policy, they will only request an increase to the not-to-exceed cost  that the National Science Board authorized if the recipient cannot address  the increase through use of the project’s budget contingency or  acceptable reductions to the project’s scope. Accordingly, at the  preliminary design review, projects must have a prioritized, time-phased  list of options for reducing scope during construction, known as scope  contingency, and the potential cost savings associated with those options  is to total at least 10 percent of the project’s baseline. As defined by  NSF’s Major Facilities Guide, scope contingency is scope that can be  removed without affecting the overall project’s objectives but that may still  have undesirable effects on facility performance.

NSF Experienced No Recent Cost or Schedule Increases on Ongoing Major Facilities Projects and Completed Construction of One Project

As of September 2019, NSF continued construction of three major  facilities projects with no changes to their authorized total project costs or  scheduled completion dates since our March 2019 report. In addition,  NSF approved a fourth project to enter the construction stage, completed  construction of one project, and advanced two major facilities projects in  the design stage. The four major facilities projects under construction  have a combined total cost of approximately $1.6 billion (see table 1).

Ongoing construction projects. Three projects—the Daniel K.

Inouye Solar Telescope, the Rubin Observatory and the Regional  Class Research Vessels—continued construction with no changes to  their authorized total project costs or scheduled completion dates  since our March 2019 report. Instead, NSF managed cost increases  on the projects through the use of budget contingency, as specified  under its no cost overrun policy, and managed delays through the use  of schedule contingency. For example, the Rubin Observatory utilized  $11.9 million in budget contingency and 5 months of schedule  contingency to better align testing of the camera within the project  schedule due to delays associated with the completion of the dome  enclosure and telescope mount assembly, among other delays. The  project team for the Rubin Observatory is also evaluating scope  reduction options in order to complete the project within its total  project cost and by its scheduled completion date of October 2022.

New construction project. In February 2019, the National Science  Board authorized a not-to-exceed total project cost of $410.4 million  for the AIMS project and NSF awarded an initial contract modification  for construction. We previously reported that in NSF’s fiscal year 2019  budget request, the estimated total project cost for construction of the  AIMS project was $355.0 million. By the project’s final design review  in October 2018, the AIMS team determined that it could not execute  the project with the desired scope for this amount because of  changing market conditions. NSF evaluated scope reduction options  for the project but decided to maintain the project’s scope at the  higher total project cost of $410.4 million. This change in total project  cost did not count as an increase under NSF’s no cost overrun policy  because the previous amount had not been authorized by the  National Science Board as the project’s not-to-exceed cost.

Completed construction project. In May 2019, NSF completed  construction of the National Ecological Observatory Network project  within the $35.5 million cost increase authorized by the National  Science Board and a schedule increase of 2.8 years (57 percent). In  2011, NSF made the original award for construction of this nationwide  network of ecological observation sites which was planned for  completion in July 2016 at a total project cost of $433.8 million. In  2017, NSF increased the not-to-exceed cost for the project to $469.3  million. In accordance with NSF’s no cost overrun policy, the NEON  project implemented scope reductions, such as reducing the number  of observation sites from 106 to 81 and eliminating certain scientific  instruments at the project’s observation sites. The scope reductions  resulted in an estimated cost savings of $62.4 million. According to  NSF documentation as of November 2019, NSF obligated a total of  $458.9 million from the MREFC account for the construction of NEON,  $10 million below the authorized total project cost. As of January  2020, NSF extended the construction stage award for NEON to allow  for award close-out activities, which NSF officials expected to be  complete in August 2020.

Projects in design. In addition, in 2019, NSF advanced the design of  two major facilities projects in the design stage, the Large Hadron  Collider High Luminosity Upgrade (HL-LHC) and the Leadership- Class Computing Facility (LCCF). Under NSF policy, a major facility  project’s cost, scope, and schedule are not finalized until after the final  design review, when the National Science Board authorizes a not-to- exceed cost and an award duration. The not-to-exceed cost that the  National Science Board authorized is the amount against which NSF  measures cost increases to implement its no cost overrun policy. In  September 2019, NSF convened two external panel reviews for the  final design of the two separate detector upgrades that make up the  HL-LHC program. According to NSF officials, the panels  recommended to the NSF Director that the detector upgrades proceed  to the construction stage. According to NSF documentation dated  November 2019, the HL-LHC program had an estimated total project  cost of $150 million for both upgrade projects. However, this amount  was subject to change since the projects had not yet been authorized  by the National Science Board to advance to the construction stage.  According to NSF officials, the National Science Board authorized the  total program cost at $153 million in early February 2020, setting the  not-to-exceed costs for both awards. The LCCF project entered the  conceptual design phase in March 2019. As of September 2019, the  LCCF project had not developed an initial estimated total project cost  because it had so recently entered design. Further details on the two  projects in design are located in appendix II.

NSF Has Implemented Two Prior Recommendations on Major Facilities and Has Taken Initial Steps to Address Other Recommendations

NSF has fully implemented two of the six recommendations we made in  June 2018 and March 2019—recommendations on policies for estimating  the costs of major facilities projects and revising the Rubin Observatory’s  schedule to better meet best practices. NSF has taken steps to address  but has not fully implemented the remaining four recommendations  concerning the agency’s management of major facilities, specifically our  recommendations on policies for developing schedules for major facilities projects, project management competencies of the agency’s major facilities project management expertise of award recipients for major facilities ensuring the sharing of lessons learned or best practices on major  facilities projects.

NSF Revised Its Cost Estimating Policies and the Rubin Observatory’s Schedule to Better Meet Best Practices

Cost estimating policies. In our June 2018 report, we found that  procedures documented in NSF’s policies for major facilities projects fully  or substantially met many best practices and partially or minimally met  others identified in GAO’s guide for developing project cost estimates.  Specifically, we found that NSF’s procedures fully or substantially met  seven of the 12 best practices in GAO’s cost guide and partially or  minimally met the remaining five, such as the best practice for conducting  a sensitivity analysis to understand which variables most affect the cost  estimate. The American Innovation and Competitiveness Act requires that  NSF ensure that its policies for estimating and managing costs and  schedules are consistent with the best practices in GAO’s cost guide, and  NSF requires the same of its recipients. We recommended that NSF  revise the agency’s policies for estimating the costs of major facilities  projects, and for reviewing those costs, to better incorporate best  practices. In response, NSF revised its Major Facilities Guide and certain  internal Standard Operating Guidance policies that documented  procedures for estimating costs.

In our current assessment of these revised guidance and policy  documents, we found that NSF fully met the five cost estimating best  practices in GAO’s cost guide that we previously found were minimally or  partially met. For example, in our 2018 report, we concluded that NSF’s  procedures required a sensitivity analysis but did not describe how one is  to be conducted. In our updated assessment, we found that NSF’s  procedures describe the best practice and how it should be applied to  NSF major facility cost estimates. Specifically, the procedures describe,  among other things, (1) identifying key variables—cost drivers, ground  rules, and assumptions—for inclusion in the analysis, with examples  particular to NSF major projects included as part of the procedures; (2)  evaluating the effect of these variables on the cost estimate by varying  them one at a time; and (3) developing a strategy to deal with the  variables to which the estimate is most sensitive. Table 2 provides an  overview of our original and updated assessments of NSF’s cost  estimating policies. Between our June 2018 assessment and our current assessment, NSF’s policies substantially or fully met all 12 of the best  practices in GAO’s cost guide.

Rubin Observatory schedule. In our March 2019 report, we found that  the Rubin Observatory’s schedule could not be considered reliable  because it did not substantially or fully meet all four characteristics of a  reliable schedule from GAO’s schedule guide—comprehensive,  controlled, well-constructed, and credible, as described in table 3. While  the schedule substantially met the comprehensive and controlled  characteristics, it partially met five scheduling best practices associated  with the well-constructed and credible characteristics. Specifically, we  found certain issues related to the construction of the project’s schedule,  including (1) the sequencing of activities, (2) the schedule’s critical path— a chain of dependent activities that drive a project’s earliest completion  date, and (3) the amount of float calculated in the schedule—the amount  of time by which a project activity can slip before the delay affects the  project’s estimated completion date. We recommended that NSF ensure  that the project’s schedule meets the well-constructed and credible  characteristics of a reliable schedule, as defined in GAO’s schedule  guide.

Our current assessment found that the revised schedule addressed our  recommendation. Specifically, the schedule substantially met four of the  five best practices that we previously found had been partially met within  the well-constructed and credible characteristics of a reliable schedule  and partially met the remaining best practice (ensuring reasonable total  float). Between our two assessments, the Rubin Observatory project’s  schedule substantially or fully met the four characteristics and nine of the  10 best practices in GAO’s schedule guide. We consider NSF’s actions  sufficient to address our recommendation. Table 3 provides our original  and current assessments of the Rubin Observatory project’s schedule.

NSF Has Taken Initial Steps to Address Four Recommendations Supporting Its Oversight of Major Facilities

In addition to implementing two of our recommendations, NSF has taken  initial steps to address the other four recommendations from our June  2018 and March 2019 reports, but has not fully implemented them. Once  NSF completes the steps discussed below, we will evaluate its actions to  determine whether they are sufficient to fully address our  recommendations.

Policies for developing project schedules. In our June 2018 report, we  found that NSF’s procedures for recipients substantially met one of the 10  best practices for developing project schedules—the best practice on  conducting a schedule risk analysis. In contrast, NSF’s procedures  partially or minimally met six and did not meet three of the remaining best  practices. For example, we found that NSF’s procedures did not meet the  best practice of establishing the durations of all activities because the  NSF documents we reviewed did not include policy or guidance related to  this practice, such as guidance on using realistic assumptions in  estimating durations. The American Innovation and Competitiveness Act  requires that NSF ensure that its policies for estimating and managing  costs and schedules are consistent with the best practices in GAO’s  schedule guide, and NSF requires the same of its recipients. We  recommended that NSF revise its policies for developing schedules for  major facilities projects, and for reviewing those schedules, to better  incorporate the best practices in GAO’s schedule guide.

As of November 2019, NSF had updated its internal guidance on  standardized cost analysis to include a new section related to schedule  reviews to help address this recommendation. This guidance states that  the NSF Large Facilities Office will lead analysis of the schedule for each  proposed major facilities project, which will include a technical evaluation  by the sponsoring office, and may include input from an independent cost  estimate and schedule review, or other reviews. As further steps to  implement this recommendation, NSF plans to update two other policy  and guidance documents, according to NSF officials. Specifically, NSF  plans to: develop a new section of the Major Facilities Guide on schedule  development, estimating, and analysis and post the guidance for  public comment; and develop new internal guidance to help NSF staff more fully utilize  external panels to address elements of schedule—in addition to  cost—as part of the panels’ oversight reviews.

According to NSF officials, they plan to complete these actions by the end  of fiscal year 2020. Once NSF completes these actions, we will re-assess  NSF’s procedures against the nine best practices that NSF partially or  minimally met or did not meet in the assessment we conducted for our  June 2018 report.

Project management competencies of NSF’s major facilities  oversight workforce. In our March 2019 report, we found that NSF had  not (1) assessed potential gaps in how well its key major facilities  oversight staff met project management competencies or (2) developed  human capital plans for its major facilities oversight staff to address any  gaps that may exist. Taking these steps would be consistent with leading  principles for strategic workforce planning that we and the Office of  Personnel Management have previously identified. Therefore we  recommended that NSF assess its major facilities oversight workforce to  identify any project management competency gaps, develop a plan to  address any gaps and time frames for doing so, and monitor progress in  closing them.

In September 2019, in response to our recommendation, NSF awarded a  contract for a proficiency assessment and workforce gap analysis. NSF  expects this analysis to assess the core competencies and necessary  proficiency levels of agency staff overseeing the major facilities portfolio  and promote long-term workforce development. According to contract  documentation, the contractor will take the following actions, among  others:  conduct a proficiency assessment and gap analysis based on a  review of existing workforce materials, such as relevant position  descriptions, vacancy announcements, performance plans, and other  NSF guidance documents; work with NSF staff to refine competency guidance to better meet  needs of the agency; and work with NSF to update training plans as necessary, based on the  findings in the gap analysis and a review NSF’s existing training plan.

According to contract documentation, NSF anticipates finishing the  competency assessment and workforce gap analysis by the second  quarter of calendar year 2020 and the implementation of contract tasks by  March 2021. According to NSF officials, depending on the results of the  assessment and analysis, improvements to address any identified gaps  may involve developing standards of performance for the oversight  workforce, identifying training opportunities in support or workforce  development, and clarifying minimum competency requirements.

Project management expertise of award recipients for major  facilities projects. In our March 2019 report, we found that NSF had  some procedures in place to help ensure that award recipients had  project management expertise, but that the agency had not established  criteria for the expertise needed by recipients or how they should  demonstrate it. We concluded that, as a result, NSF was at risk of making  awards to organizations that may not be well qualified to manage  construction of major facilities projects. We recommended that NSF  establish criteria for the project management expertise of award  recipients for major facilities projects and incorporate the criteria in project  requirements and external panel reviews.

As of November 2019, NSF had drafted new language for the Major  Facilities Guide and related supplemental award terms and conditions for  major facilities that would require award recipients to document how  project management competencies will be met. NSF officials told us they  had shared the draft documents with targeted recipient representatives  for review and comment in September 2019. NSF officials stated that the  supplemental terms and conditions are planned to be published in fiscal  year 2020, with an effective date of June 2020. The officials also said  that, for existing awards, the agency will work with recipients on a phased  implementation of the new guidance and terms and they will automatically  be incorporated into future awards.

Sharing of lessons learned or best practices on major facilities  projects. In our March 2019 report, we found that NSF formalized a  process for identifying and sharing lessons learned on major facilities  projects. The process, which NSF refers to as its knowledge management  program, responded to a 2015 recommendation by the National Academy  of Public Administration and to the American Innovation and  Competitiveness Act’s requirements that NSF coordinate the sharing of  best management practices and lessons learned from major facilities  projects. We recommended that NSF ensure, through a requirement or  other means, that award recipients for major facilities projects provide  information to NSF on any lessons learned or best practices.

NSF developed supplemental award terms and conditions for major  facilities to require recipients to participate in NSF’s knowledge  management program. According to NSF officials, among other things,  the requirement can be met by recipients:  sending appropriate staff to the annual major facilities workshop that  NSF hosts to provide a collaborative forum for continuous learning  and information sharing among participants; presenting lessons learned or good practices at the annual workshop; participating in a workshop planning committee; or providing lessons learned or good practices to NSF.

According to NSF officials, the draft terms and conditions will be included  in the same revision as those related to recipients’ project management  expertise, planned for publication in fiscal year 2020. As described above,  NSF officials said that for existing awards, the agency will work with  recipients on a phased implementation of the new terms and conditions,  and they will automatically be incorporated into future awards.

NSF Plans to Make Its First Awards for Mid-Scale Research Infrastructure Projects in 2020 and Is Developing Guidance to Manage Projects

NSF Plans to Award Its First Set of Mid-Scale Projects in 2020

According to NSF documentation, NSF requested $45 million for fiscal  year 2020 within the MREFC account to fund its first set of mid-scale  projects with a total project cost between $20 million and $70 million. In  response to a solicitation it issued in December 2018, NSF received  approximately 50 preliminary proposals for mid-scale projects from  research areas spanning all of NSF’s directorates, according to NSF  officials. NSF invited 14 of these applicants to submit a full proposal and  received full proposals from 11. The solicitation specified a list of  information each full proposal should contain, including a project  summary and description, a budget, and a project execution plan.

NSF is currently reviewing the full proposals and expects to award its first  portfolio of mid-scale projects in August 2020, according to NSF  documentation. NSF’s solicitation anticipated that $150 million will be  available over five years to fund its first batch of mid-scale projects.  According to NSF officials, NSF plans to award subsequent sets of mid- scale projects biennially, depending on the availability of funds for future  projects.

According to NSF’s solicitation, the agency is seeking prospective mid- scale projects that are innovative and potentially transformative, that  include a strong component of student training, and that provide unique  research capabilities relative to what currently exists in the research  community. Based on the definition of mid-scale projects in the American  Innovation and Competitiveness Act, the solicitation stated that NSF  would consider upgrades to existing major facilities projects currently in  operation as candidates for mid-scale projects. The solicitation required  full proposals to describe the full life cycle cost and schedule—including  development, design, implementation, operations, and divestment.  According to agency officials, NSF is only seeking to fund construction  and acquisition costs from the MREFC account but needs to understand  potential cost impacts on other life cycle stages.

According to NSF officials, the mid-scale program is designed to identify  potential projects with shorter implementation timelines and high levels of  readiness as compared to the multiyear, incremental refinements to cost,  scope, and schedule that occur with major facilities projects. NSF officials  also stated that, to assess the readiness of the mid-scale projects for  which full proposals were received, the agency will use an internal  proposal review process similar to the final design review process used  for major facilities projects. In addition, NSF policies state that there can  be multiple inputs to the proposal review process, such as external panels  or ad hoc reviews, which ensure that the mid-scale projects NSF awards  will reflect the needs and interests of the scientific community.

NSF Has Developed Flexible Guidance to Manage Mid-Scale Projects

To provide guidance on oversight for mid-scale projects, NSF has  included a chapter in its September 2019 update of the Major Facilities  Guide to outline minimum recipient requirements and NSF oversight  activities for mid-scale projects. In addition, NSF has created a  management plan for NSF personnel that outlines procedures for  reviewing proposals, selecting mid-scale projects, and managing the  award process. NSF last updated the plan in November 2019, and  according to NSF officials, the agency will continue to update the plan as  it leads its initial set of projects from award to execution.

According to NSF officials, oversight requirements for mid-scale projects  will be dependent upon the technical scope and complexity of each  individual project. As a result, NSF has tailored its guidance to provide the  level of oversight commensurate with each project’s technical scope, type  and mix of work, and risk profile. In addition, NSF is incorporating some  aspects of its existing guidance for major facilities projects into its  guidance for mid-scale projects. However, NSF officials anticipate that  mid-scale projects will be less complex than major facilities projects. The  following describes aspects where NSF has adapted its guidance for  major facilities projects to the lower level of complexity anticipated for  mid-scale projects.

Performance measurement baselines. Similar to major facilities  projects, NSF requires that the scope, cost, and schedule for mid-scale  projects be defined at the time of award. In addition, NSF requires budget  management, cost controls, and identification of potential risks and  mitigation strategies for mid-scale projects, and its guidance states that  budgets should be developed in accordance with GAO’s cost estimating  best practices. While NSF officials state that NSF will apply substantial  rigor in assessing the defined total project cost, mid-scale projects will not  be subject to NSF’s no-cost-overrun policy. As a result, unlike for major  facilities projects, NSF will not require all mid-scale projects to include  budget contingency and scope reduction options, both of which are  necessary for implementing the no-cost-overrun policy, although it may  choose to include contingency in the budgets for certain mid-scale  projects. For those mid-scale projects that have budget contingency, they  must follow guidance for budget contingency laid out in the Major  Facilities Guide, such as obtaining approval from NSF for using budget  contingency.

Monitoring and assessment. Like major facilities projects, NSF will  monitor the award progress of mid-scale projects through periodic reports  that provide quantifiable measurements on technical progress as well as  cost and schedule performance. Depending on the complexity of each  project, annual site visits or reviews may also be conducted. However,  recipients of mid-scale projects may use alternatives to an earned value  management system to report progress, such as reporting on milestone  events or expenditure reports. According to NSF officials, the burden of  establishing an earned value management system for some mid-scale  projects may outweigh the benefits of using such a system, depending on  the technical nature of the project.

Project execution plan. According to the Major Facilities Guide, NSF will  require a project execution plan for all mid-scale projects to demonstrate  how recipients will manage the projects. A project execution plan serves  as the stand-alone document that explains all of a project’s requirements  for execution. According to NSF officials, the project execution plan used  for major facilities projects would be excessive for mid-scale projects and  may discourage potential proposals. Thus, NSF guidance for mid-scale  projects requires only nine of the 16 sections normally required in a  project execution plan and allows the recipients to tailor the detail and  scope of each section to the specifics of each project. In addition, NSF  will not require mid-scale projects to include design and development  plans or site and environment information, which are required sections for  major facilities projects. Since it is only funding the construction of mid- scale projects and seeking to award projects with high levels of  readiness, NSF does not consider these sections to be beneficial in  assessing how a recipient would manage a mid-scale project.

We provided a draft of this report to NSF for review and comment. In its  comments, reproduced in appendix III, NSF stated that our report  provides the agency with an independent assessment of its oversight of  projects in design and construction and its stewardship of the MREFC  account. With regard to our recommendations on policies for estimating  the costs of and developing schedules for major facilities projects, NSF  stated it is proud of the progress it has made in meeting GAO best  practices for cost estimating on major facilities projects and that it  recognizes the remaining work needed to codify NSF guidance on project  schedules. NSF also provided technical comments, which we  incorporated as appropriate.

We are sending copies of this report to the appropriate congressional  committees, the Director of the National Science Foundation, and other  interested parties. In addition, the report is available at no charge on the  GAO website at https://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-6888 or neumannj@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix IV.

Appendix I: Summaries of the National Science Foundation’s Major Facilities Projects under Construction

This appendix provides individual summaries of the National Science  Foundation’s (NSF) four major facilities projects under construction: (1)  the Daniel K. Inouye Solar Telescope, (2) the Vera C. Rubin Observatory,  (3) the Regional Class Research Vessels, and (4) the Antarctic  Infrastructure Modernization for Science.

Each project’s summary is based on project documents and other  information that NSF officials provided and includes the following:

An overview of the project and its purpose.

A timeline identifying key project dates, including the date of the  original construction award, which we report as the start of  construction.

Project information, such as the project’s estimated completion date  for construction (including schedule contingency), the type and latest  amounts of the awards for construction, the responsible NSF  directorate, project partners, and expected duration of operations.

Tables summarizing the project’s current status and its cost, any cost   or schedule increases or scope reductions made under NSF’s no  cost overrun policy, and changes since our March 2019 report.

A summary of the project’s cost and schedule performance history.

A chart depicting the latest construction award’s total project cost for  construction, including the performance measurement baseline and  budget contingency.

If applicable, a chart showing the increase in the construction award’s  total project cost since the original construction award.

Information on remaining project risks and potential for cost or  schedule increases, including the amount of remaining contingency  and scope reduction options.

When completed, the National Science Foundation’s (NSF) Daniel K. Inouye Solar  Telescope (DKIST), formerly named the Advanced Technology Solar Telescope, will  be the world’s flagship facility for the study of magnetic phenomena in the solar  atmosphere. It will help answer fundamental questions in solar physics and enable  understanding of solar variability and activity, which can affect Earth through  phenomena generally described as space weather.

Construction of NSF’s DKIST project was 94 percent complete as of  September 2019. The project was in its 10th year of construction and in  the integration, testing, and commissioning phase. Since our March 2019  report, the project completed installation of all telescope optics. Testing of  the optics, originally planned for October 2019, was delayed until January  2020 to allow the project to replace a key piece of equipment that is  essential to safely perform the testing. Despite the delay, the estimated  completion of construction and beginning of full operations remained  unchanged at June 2020, including 1.5 months of schedule contingency.

Estimated construction completion  date, including schedule  contingency:   June 2020.  Construction award:   Cooperative support agreements with  the Association of Universities for  Research in Astronomy, Inc., consisting  of 42 U.S. institutional members and  five international affiliates.  Responsible NSF directorate:   Mathematical and Physical Sciences.  Project partners:   More than 20 U.S. and international  organizations. Kiepenheuer-Institut für  Sonnenphysik (Germany) and Queens  University Belfast (Northern Ireland)  are supplying additional equipment for  the project.  Expected duration of operations:   50 years.

Legend: ▲ = cost or schedule increase; ▼= scope reduction.

NSF’s DKIST project had no changes to its authorized total project cost,  June 2020 completion date or project scope since our March 2019 report,  which used data as of September 2018. From April to November 2019,  NSF approved the project’s use of about $6.2 million in budget  contingency, with the largest usage of about $4.6 million in August 2019.  Project delays requiring use of 3 months of schedule contingency— primarily because the project faced challenges with the installation and  testing of the mirror systems, as described above—accounted for $4.3  million of the $4.6 million. We previously reported that the DKIST project’s  risk of delays had the potential to increase costs for such items as labor,  utilities, real estate, and equipment. NSF officials stated that most of the  activities at risk of further delays would be achieved during the testing  planned for January 2020.

In 2013, NSF increased DKIST’s total project cost and the not-to-exceed  cost that the National Science Board authorized from $297.9 million to  $344.1 million, an increase of $46.2 million (16 percent) since 2010. NSF  also delayed the project’s estimated completion date by about 2.5 years  (31 percent), from December 2017 to June 2020. Prior to the National  Science Board’s authorization to increase the total project cost, the  recipient also reduced DKIST’s scope, resulting in estimated cost savings  of $5.9 million but generally low expected impacts for the project.  According to NSF officials, these cost and schedule increases resulted  primarily from unforeseeable legal and administrative challenges to the  construction site’s environmental permits.

Remaining Project Risks and Potential for Cost or Schedule  Increases  As of September 2019, the DKIST project had $7.8 million of budget  contingency remaining—$0.4 million more than the estimated remaining  risk exposure of about $7.4 million when weighted for the risks’  probability. The project also had 1.5 months of schedule contingency  remaining to help avoid any potential delays in completing construction.

According to the project documentation, the largest remaining risk  category is project completion and closeout risks. As of October 2019, 10  risks in this category remained, some of which had been partially realized,  according to NSF officials, with about $4.0 million in risk exposure when  weighted for probability. The remaining risks included staff retention as  the construction project nears completion, and damage to or wear of  equipment during integration and commissioning. For example,  contingency may be needed to make minor repairs to the dome enclosure  in preparation for full operations.

In accordance with NSF policy, the project maintains a list of scope  reduction options, which as of October 2019 included approximately  $56,700 in total possible project de-scopes, such as reductions in travel.  However, the ability of these remaining de-scope options to reduce costs  will continue to decrease as the project continues to spend down  remaining funds as it approaches completion.

The National Science Foundation’s (NSF) Vera C. Rubin Observatory (Rubin  Observatory), formerly named the Large Synoptic Survey Telescope (LSST), is an 8.4- meter, wide-field optical telescope. It will initially be used to image the entire visible  southern sky—every 3 days for a decade—using the world’s largest digital camera (3.2  billion pixels). Built on a mountaintop in Chile to take advantage of the location’s  pristine skies, the observatory will collect data and images that will allow for charting  billions of galaxies as well as increased knowledge about potentially hazardous  asteroids, dark matter, and dark energy. The observatory has the potential to advance  every field of astronomical study, from the inner solar system to the large-scale  structure of the universe.

As of September 2019, the Rubin Observatory was 75 percent complete  and in its sixth year of construction. NSF made the initial operations  award in October 2018, and NSF officials anticipate completion of  construction and start of full operations in October 2022, including  contingency. Since our March 2019 report, the project has experienced  delays related to both the telescope’s dome enclosure and mount  assembly, leading NSF to add the project to the Director’s Watch List.

Estimated construction completion  date, including schedule contingency:   October 2022.   Construction award:  Cooperative support agreement with the  Association of Universities for Research  in Astronomy, Inc., consisting of 42 U.S.  institutional members and five  international affiliates.  Responsible NSF directorate:  Mathematical and Physical Sciences.  Project partners:  The LSST Corporation, Department of  Energy.  Expected duration of operations:  50 years.

Since our March 2019 report, NSF’s Rubin Observatory project had no  changes to its authorized total project cost and implemented one scope  reduction option valued at $1.4 million to increase available budget  contingency. In addition, the project utilized $11.9 million in budget  contingency and 5 months of schedule contingency to better align the  testing of the camera within the project schedule. According to project  documentation, the use of schedule contingency was due to delays with  completion of the telescope mount assembly and dome enclosure that will  house the telescope and other buildings. NSF officials attributed the  delays to contractor performance and adverse weather conditions. For  example, due to high winds, the project was able to use a crane to  complete dome construction for only two days in September 2019.

Remaining Project Risks and Potential for Cost or Schedule  Increases  Project data on the remaining risks and contingencies and the findings of  two recent reviews indicate that the final cost of the Rubin Observatory  may exceed the not-to-exceed cost authorized by the National Science  Board, unless the project implements scope reduction options under  NSF’s no-cost-overrun policy. As of September 2019, the project had an  estimated remaining risk exposure of $26.4 million, which is equal to the  remaining budget contingency of $26.4 million. In addition, the project had  3.5 months of schedule contingency remaining as of September 2019 to  help avoid any potential delays in completing construction by October  2022. According to project documentation, the project’s largest remaining  risks included delays in the completion of the telescope’s dome  enclosure, the installation of the mount assembly, and delivery of the  camera from the Department of Energy (DOE). The project team is  modifying activity plans to mitigate these delays. For example, the project  plans to complete dome enclosure and telescope mount assembly  activities in parallel. As part of the Director’s Watch List, NSF plans to  closely track updates on the project, including potential execution of  scope reduction options.

In August 2019, NSF and DOE jointly convened an external committee of  experts to review the project’s construction progress. The committee  found that the project may face difficulty in completing the baseline scope  within the authorized total project cost. Specifically, the committee  expressed concerns with the rate at which schedule contingency has  been used (5 months of schedule contingency within the past 18 months),  delays in completing the dome due to contractor performance issues, and  the risks associated with maintaining an aggressive schedule composed  of parallel activities in order to minimize further delays. The review  committee recommended that NSF direct the project team to develop a  proposal for executing scope reductions in fiscal year 2020 to complete  the telescope within an acceptable level of risk at the current total project  cost, among other recommendations.

Remaining Contingency and Scope  Reduction Options   As of September 2019 with construction  75 percent complete.  Budget contingency:   $26.4 million (Equal to the probability- weighted risk exposure of $26.4 million).  Schedule contingency:   3.5 months (included in the October 2022  estimated completion date).  Estimated value of remaining scope  reduction options:   $24.8 million. as compared to a separate risk exposure analysis from July 2019 that  indicated a 50 percent confidence. The panel recommended that the  project report risk based on the analysis with the lower confidence level  and conduct more frequent risk exposure analyses based on changes  that have occurred, such as the realization or retirement of identified  risks, to better inform management decisions. According to NSF  documentation, the project team has recently acquired enhanced risk  management software for analyzing risk exposure, including the effects of  mitigating actions within the schedule.

In a July 2019 update to its scope management plan, the project team  identified 39 scope reduction items with a total value of $25.0 million.  Among them is a de-scope option for reducing the amount of final  commissioning surveys that may potentially return $4.3 million of budget  contingency and 3.5 months of schedule contingency. According to NSF  officials, NSF has yet to evaluate the impact of reducing the surveys to  the project’s capabilities or operational costs. According to the external  panel review convened by NSF and DOE, the project team identified  potential scope reductions options valued at $14 million that the project  can exercise in fiscal year 2022. However, the panel questioned the  feasibility of executing the project’s scope reduction options and  recommended that the project prioritize viable options while pursuing a  no-cost extension to complete the project without an increase to the total  project cost.

The U.S. Department of Energy (DOE), a  cosponsor of the Rubin Observatory, is  responsible for delivering the  observatory’s camera at a cost of $168  million. SLAC National Accelerator  Laboratory manages a collaboration of  DOE national laboratories and  universities to develop, fabricate, and  deliver the camera. As of September  2019, the project had the camera  integration on the telescope scheduled  for September 2021. Budget contingency  accounts for the risk of a delayed delivery  that would impact integration.   The LSST Corporation is a not-for-profit  organization representing nearly 40  institutional members and 34  international contributors. It acts as the  agent for nonfederal funding contributed  to the project and has raised more than  $50 million for certain long-lead  construction items and additional  development efforts.

The National Science Foundation’s (NSF) Regional Class Research Vessels (RCRV)  project will construct three 199-foot vessels to support the nation’s ability to conduct  fundamental scientific research in the coastal zone and continental shelf, including from  the ocean’s surface through the water column to the sea floor and subsea floor  environment. These vessels will provide enhanced capabilities beyond those of the  retiring vessels they will replace. The three vessels’ research locations will depend on  locations of the greatest science demand, but NSF planned to operate the first vessel  along the west coast, the second along the east coast, and the third along the gulf coast  of the United States.

Location: Construction site is in Louisiana.

As of September 2019, NSF’s RCRV project was 20 percent complete  and was in its third year of construction. Since our March 2019 report, the  project progressed with construction of the first vessel and began  construction of the second vessel in September 2019. NSF also awarded  funds for construction of the third vessel, which was scheduled to begin in  March 2020, and awarded a cooperative agreement for its future  operations to the Gulf-Caribbean Oceanographic Consortium. In February  2019, the RCRV project experienced a partial suspension of work due to  the status of necessary production design and modeling deliverables,  among other concerns. This resulted in 16 weeks of schedule  contingency usage. However, there was no overall increase to the  scheduled construction completion date of July 2024.

Estimated construction completion  date, including schedule contingency:  July 2024 for three vessels.  Construction award:  Cooperative support agreement with  Oregon State University, which  contracted with Gulf Island Shipyards,  LLC.    Responsible NSF directorate:  Geosciences.  Project partners:  The U.S. Navy performed initial design  for the vessels.  Expected duration of operations:  30 years.

Construction Status of the Regional Class Research Vessels, as of September 2019  Percentage complete (based on construction of three vessels) aScope changes included are reductions in response to NSF’s policy on cost overruns or as part of a  cost increase.

As of September 2019, the RCRV project had no changes to its  authorized total project cost, no changes to its estimated completion date  of July 2024 for all three vessels, and no scope reductions. The National  Science Board had authorized a not-to-exceed cost of $365.0 million for  construction of three vessels. However, the shipyard bid was ultimately  lower than expected, reducing the total project cost of building three  vessels to $354.0 million.

NSF accepted the project’s earned value management system in May  2019, following a surveillance review of the system. The review team  found that the project’s system met the intent of NSF requirements and  that its data were reliable. (In our March 2019 report, we reported that  NSF conditionally accepted the project’s earned value management  system in November 2018.)

As of September 2019 with construction  of three vessels 20 percent complete.  Budget contingency:   $44.0 million (exceeded the probability- weighted risk exposure of $24.6 million).  Schedule contingency:   6 months (included in the July 2024  estimated completion date for three  vessels).  Estimated value of remaining scope  reduction options:   $9.8 million.

Beginning in February 2019, the RCRV project utilized 16 weeks of  schedule contingency and $2.4 million of budget contingency due to a  partial suspension of work issued by the construction award recipient,  Oregon State University (OSU). OSU was concerned with Gulf Island  Shipyards’s (GIS) project management capacity and its ability to manage  subcontractors, such as engineering vendors responsible for providing  design specifications. During the work suspension, GIS developed a  corrective action plan that identified eight areas of improvement, such as  a subcontract management plan and updated schedules that better align  the development of necessary design specifications with construction  activities. OSU’s management team assessed and monitored GIS’s  progress on these areas and subsequently lifted the work suspension in  May 2019.

However, the project continues to face subcontractor management  issues. OSU has requested NSF approval for an estimated $6.1 million of  budget contingency and 4 months of schedule contingency to  compensate for the delays associated with these issues. According to  project documentation, this issue may cause the construction completion  date of each vessel to slip.

Remaining Project Risks and Potential for Cost or Schedule  Increases   According to project documentation, the project had an estimated risk  exposure of $24.6 million and $44.0 million in remaining contingency as  of September 2019. With the utilization of 4 months of schedule  contingency in 2019, the RCRV project had 6 months of contingency  remaining until construction is scheduled to end in 2024. According to  project documentation, 12 options for reducing scope were available as of  December 2019, with potential savings estimated at $9.8 million. and schedule expertise, which resulted in a decrease in the impact of the  risk.

In addition, the RCRV project is closely monitoring two risks related to  newer technologies and requirements for regional operability of each  vessel. First, the project team identified newer technologies for systems  such as communications compared to those specified during the design  phase. According to project documentation, the project may utilize  contingency to integrate such technologies into the vessels. Second, the  project may incur additional engineering, labor, and material costs  associated with certain potential design changes that NSF and the  operating institutions for the three vessels have identified. These design  changes are intended to improve quality and performance within the  different regions where the three vessels will be operating.

The National Science Foundation’s (NSF) Antarctic Infrastructure Modernization for  Science (AIMS) project will modernize the core infrastructure of McMurdo Station in  Antarctica, the largest of three stations operated by NSF’s United States Antarctic  Program and used by multiple agencies. McMurdo Station serves as a logistics hub for  remote field sites and for the Amundsen-Scott South Pole Station. The AIMS project is  expected to make environmental and safety upgrades to McMurdo Station and redevelop  it into a more compact, energy and operationally efficient core facility to support research.  The planned core facility will consolidate critical buildings, such as medical facilities and  field science support.

Estimated construction completion date, including schedule contingency: 2028. Construction award: February and April 2019 modifications to the existing Antarctic support contract with Leidos Innovations Corporation. Responsible NSF directorate: Geosciences. Project partners: Other federal agencies—such as the National Aeronautics and Space Administration, National Oceanic and Atmospheric Administration, and the Department of Energy—and international programs, such as the Scientific Committee for Antarctic Research. Expected duration of operations: 35 to 50 years.

Construction of NSF’s AIMS project was about 6 percent complete as of  September 2019. The project was in its first year of construction. In  February 2019, the National Science Board approved the project’s not-to- exceed cost of $410.4 million, and NSF awarded an initial contract  modification for construction equipment and materials to be delivered to  California by December 2019, in time for deployment to McMurdo station  through two supply vessels.  In April 2019, NSF awarded the second  contract modification for construction of the first major components of  AIMS: the Vehicle Equipment and Operation Center (VEOC) and a new  lodging facility structure and exterior shell. According to NSF, the VEOC  will facilitate maintenance and repair of both heavy and light equipment  ranging from tractors and cranes to trucks, vans, snowmobiles, and field  generators. The lodging facility will include space for 285 beds, which the  project’s final design review panel expected to be adequate to support  short- and long-term plans for McMurdo station, including construction  needs. As of September 2019, the start of initial operations for the VEOC  and lodging facility were planned for 2022 and 2023, respectively, and  completion of both facilities was planned for 2022, according to NSF  officials. Later phases of the AIMS project will include construction of  central services, emergency operations, field science support, and  industrial trades facilities.

In November 2018, the U.S. Army Corps  of Engineers completed an independent  cost estimate (ICE) report for the AIMS  project. According to NSF officials, the  ICE was critical for negotiations with the  contractor as NSF utilized data within the  ICE, such as labor rates and cost of  materials, to verify costs. Specifically, the  ICE assisted NSF in determining the  reasonableness of the contractor’s  proposed cost estimate and schedule for  the project and associated risks.  According to NSF officials, NSF and the  contractor resolved all recommendations  from the ICE report to NSF’s satisfaction  prior to setting the not-to-exceed cost.

Cost and Schedule Performance History  As of September 2019, NSF’s AIMS project had no changes to its  authorized total project cost, changes to its estimated completion date, or  scope reductions since the National Science Board authorized the  project’s not-to-exceed cost of $410.4 million, which included $67.2  million in budget contingency, in February 2019.

Remaining Contingency and Scope  Reduction Options   As of September 2019 with construction  about 6 percent complete.   $59.2 million ($7.1 million more than the  probability-weighted risk exposure of  $52.1 million).   18.4 months (included in the 2028  estimated completion date).

By the project’s final design review in October 2018, the AIMS team  determined that it could not execute the project with the desired scope for  the $355.0 million estimate—as was previously presented in NSF’s fiscal  year 2019 budget request—because of changing market conditions. In  response, NSF convened a review panel, which evaluated scope  reduction options such as relocating and reducing bed space in the  lodging facility from 285 to 100 beds, which would also entail keeping the  current lodging facility in operation instead of demolishing it to make room  for a new facility. While it accepted some of these options, such as a  reduction of warehouse space within the VEOC, the panel noted that  relocation of the lodging facility and a reduction of bed space would have  adverse effects on the project. For example, the panel found that  constructing a new 100-bed lodging facility in an alternate location would  not support the eventual construction of sky bridges. According to the  project’s Final Design Review report, these sky bridges would improve  efficiency by avoiding the need for personnel to put on Antarctic gear  before moving between buildings, reduce energy use by reducing the  need to open exterior doors, and significantly improve the quality of life for  personnel. NSF therefore decided to maintain the 285-bed plan and  finalized the total project cost at $410.4 million. and four of 11 procurements for the lodging facility to the 2021 vessel, but  NSF officials do not expect significant construction delays as a result. The  officials explained that the VEOC procurements are not required for 2020  construction and that the deferral of lodging procurements is expected to  be accommodated by re-sequencing activities on site.

Remaining Project Risks and Potential for Cost or Schedule  Increases   As of September 2019, the AIMS project had a risk exposure of $52.1  million and $59.2 million in remaining contingency, and all of the project’s  18.4 months of schedule contingency remained available. The project had  cumulatively used $7.9 million in budget contingency. Of this, $7.8 million  was used during initial award for contract modifications for initial  construction, with the remainder used for additional equipment purchases  and leases in August and September 2019.

As of September 2019, the AIMS project had $14.2 million in high- likelihood risks. The largest remaining risk, with an estimated value of  $12.5 million and a 23-day delay, was that subcontractor proposals would  exceed planned construction costs.  Another such risk was an increase in  the estimated base price of key construction materials—such as steel,  copper wire, concrete, gypsum, and specialty items—before the materials  were procured. NSF’s contractor for the project, Leidos Innovations  Corporation, was working with one of its subcontractors to ensure  material costs were accurate and consistent with market pricing.

In accordance with NSF policy, the project maintains a list of scope  reduction options, which as of April 2019 included approximately $34.0  million to $43.1 million in total possible project de-scopes. For example,  the largest scope reduction option, with an estimated value of up to $19.1  million, is to remove the new trades shop from the AIMS scope and  instead use the current facility. Another option, with an estimated value of  up to $4.0 million, is to remove the gymnasium from the emergency  operations facility and instead continue to use and maintain the existing  gymnasium.

Appendix II: Summaries of the National Science Foundation’s Plans for Future Major Facilities Projects in Design

This appendix provides individual summaries of the two National Science  Foundation (NSF) projects that were in design and planned for  construction as major facilities projects: (1) the Large Hadron Collider  High Luminosity Upgrade and (2) Leadership Class Computing Facility.  As of September 2019, no construction funds had been awarded for  these projects and all cost, schedule, scope, and design information for  these projects was subject to change.

Each project’s summary is based on project documents and other  information that NSF officials provided and includes the following:

An overview of the project and its purpose.

A timeline identifying key project dates.

Project information, such as the expected date for completion of  construction; the anticipated type of awards for construction; the  responsible NSF directorate; project partners; and expected duration  of operations.

A summary of the project’s current status.

A summary of the project’s design and construction costs, if available,  and the budget account NSF planned to use for construction of the  project.

The Large Hadron Collider (LHC) is the world’s most powerful particle accelerator. The  facility’s four detectors observe new particles that are produced when high-energy  protons are accelerated and collided, providing insight into fundamental forces of nature  and the condition of the early universe. Through the National Science Foundation’s (NSF)  Large Hadron Collider High Luminosity Upgrade (HL-LHC) program, the agency will fund  a portion of a larger international effort to upgrade the facility’s accelerator and detectors.  Specifically, NSF plans to fund the design and implementation of certain parts of the  upgrades as two separate projects for the facility’s detectors, the A Toroidal LHC  Apparatus (ATLAS) and Compact Muon Solenoid (CMS) detectors. The Department of  Energy (DOE) is also contributing to upgrades to the LHC’s accelerator and to the ATLAS  and CMS detectors.

As of September 2019, NSF’s HL-LHC program was approaching its fifth  year of design. The program has conducted several required activities to  complete the design stage. In September 2019, NSF convened an  external panel for the final design review of the program. The panel found  that both detector upgrades met the readiness criteria within NSF’s Major  Facilities Guide to proceed to construction. NSF also convened the  internal Facilities Readiness Panel in November 2019 and conducted life  cycle cost reviews for each detector upgrade in October 2019, according  to NSF officials.

Estimated construction completion  date, not including schedule  contingency:   2026.  Construction awards:   If approved, planned for 2020 as  cooperative agreements with Columbia  University (ATLAS detector) and  Cornell University (CMS detector).  Responsible NSF directorate:   Mathematical and Physical Sciences.    Project partners:   European Organization for Nuclear  Research and the Department of  Energy.  Expected duration of operations:   12 years.

According to NSF officials, NSF planned to request National Science  Board authorization in February 2020 to make construction awards. As a  prerequisite for making the awards in April 2020, NSF received the  independent cost estimates for both projects from the Army Corps of  Engineers in January 2020. According to NSF documentation, these  results align with the current total project cost reviewed during the final  design review. According to the Major Facilities Guide, NSF uses  independent cost estimates to validate recipient estimates, negotiate  awards, check for compliance with GAO best practices and Uniform  Guidance cost principles, and inform NSF’s cost analysis. According to  NSF officials, the estimated completion for both upgrade projects is 2026.

According to program documentation, NSF had obligated a total of $24.3  million for the design of its detector upgrades as of September 2019.  Funding for the design has come from NSF’s Research and Related  Activities account, rather than the Major Research Equipment and  Facilities Construction account.

Planned Contingency and Scope  Reduction Options   As of November 2019, with finalization  of the NSF cost analysis still pending.  Budget contingency:   $38.9 million as follows $20.0 million for the ATLAS detector. $18.9 million for the CMS detector.  Schedule contingency:   To be determined.  Estimated value of scope reduction  options:   $15.1 million as follows $8.4 million for the ATLAS detector. $6.7 million for the CMS detector. until authorization by the National Science Board. These figures remained  subject to change before completion of the final design phase. According  to NSF documentation, the total project cost may increase slightly based  on a detailed evaluation of both projects’ contingency budgets following  the final design review. NSF plans to fund the upgrades with separate  cooperative agreements for each detector and to monitor each agreement  in accordance with its distinct terms and conditions, total project cost, and  earned value management metrics, according to agency officials.

In August 2019, NSF initiated independent cost estimates of both projects  (ATLAS and CMS) under the HL-LHC program, as required by the  American Innovation and Competitive Act for projects in the design  phase. The U.S. Army Corps of Engineers is conducting the estimates  under an interagency agreement with NSF, with contractor support. In  addition, NSF is conducting a cost analysis that will be informed by the  final design review panels, internal assessments by the NSF’s Large  Facilities Office and other business units, and the independent cost  estimates.

DOE’s Contributions to Upgrading  the Large Hadron Collider  DOE’s High Energy Physics program  helped fund the construction of the  Large Hadron Collider and continues to  support researchers using the facility  as well as upgrades to it. According to  DOE’s fiscal year 2020 budget request,  the department planned to support the  upgrades to the ATLAS and CMS  detectors at an estimated cost range of  $149 million to $181 million for the  ATLAS detector and $125 million to  $155 million for the CMS detector. The  scope of DOE’s work on the detectors  was to focus on areas where the  expertise and infrastructure of the  department’s national labs were  needed, whereas the scope of NSF’s  work was to focus on areas led by  university researchers. In addition,  DOE approved upgrades to the  accelerator itself with a total project  cost of $242.7 million, according to  DOE’s fiscal year 2020 budget request.

NSF plans to fund the construction of the detector upgrades through its  Major Research Equipment and Facilities Construction account. While the  upgrades would involve separate cooperative agreements for each  detector, NSF considers them one program consisting of two distinct  projects, according to agency officials.

Project Risks and Potential Scope Reduction Options  Under NSF policy, a project’s cost should include enough budget  contingency to cover all foreseeable risks. Following the preliminary  design review, the amount of budget contingency included in the  construction cost for the upgrades was approximately $38.9 million, or 26  percent of the planned total project cost. At the time of this report, the  NSF cost analysis following the final design review was still pending and  therefore the estimated amount of contingency is subject to change.

NSF policy also directs a project’s design to include prioritized, time- phased options for reducing its scope during construction if needed. As of  the final design review, the project teams had identified a total of $15.1  million of potential scope reduction options for the projects, which are  subject to change throughout the design and construction of a project.  According to the projects’ scope management plans we reviewed, the  ATLAS detector has nine options to reduce scope totaling $8.4 million,  with the options ranging in value from $0.6 million to $1.7 million. The  CMS detector has 17 scope reduction options with a total value of $6.7  million. According to the project’s scope management plan, both NSF  officials and external panels reviewed and provided input to determine the  current scope reduction options.

The National Science Foundation’s (NSF) Leadership-Class Computing Facility (LCCF)  project is intended to provide advanced computational capabilities to enable  transformative research in all areas of science and engineering that would not be  possible by theory or experiment alone. According to NSF officials, future research using  LCCF might include extremely detailed simulations ranging from biological molecules to  supernovae and analyses of very large data streams such as satellite images to create  high-resolution Earth maps.

Location: Texas Advanced Computing Center, University of Texas at Austin

Project Status  As of September 2019, the LCCF project was in its first year of design;  consequently, all cost, schedule, scope, and design information for the  project was subject to change. In March 2019, the NSF Director approved  the project to enter the design stage as a candidate major facilities  project. The project represents the final phase of a two-phase deployment  of high-performance computing systems. The first phase—known as the  Frontera project at the Texas Advanced Computing Center at the  University of Texas at Austin—was completed in September 2019.  According to NSF, at that time, Frontera was the largest high- performance computing system deployed on a U.S. academic campus.  The LCCF project will support the design and construction of an upgrade  to the Frontera system as well as to the physical facility that will host it. In  project documentation, NSF has described the upgrade as providing a  substantial improvement in application performance but has not specified  the extent of improvement.

Estimated construction completion  date, not including schedule  contingency:   Fiscal Year 2025.  Construction award:   Planned for 2024.  Responsible NSF directorate:   Directorate for Computer & Information  Science & Engineering.  Project partners:   None.  Expected duration of operations:   10 years.

In July 2019, NSF awarded both an overarching cooperative agreement  for the LCCF project and a cooperative support agreement for the  conceptual design phase to the University of Texas at Austin. As of  November 2019, the project was focused on leading and participating in  activities with experts within the community for high-performance  computing. The purpose of these activities was to document the science,  technology, and facilities requirements for LCCF, as well as to shape the  design and cost of long-lead items, such as the power and cooling  infrastructure to service the facility. NSF plans to conduct the conceptual  design review in June 2020.

NSF’s Support for High- Performance Computing Systems  NSF has supported high-performance  computing capabilities for nearly 4  decades. In 2007, NSF awarded  $226.6 million for the Blue Waters high- performance computing system  through a cooperative agreement with  the University of Illinois at Urbana- Champaign. According to NSF, at the  time of its deployment in 2013, Blue  Waters was one of the most powerful  supercomputers in the world and was  one of the fastest on a university  campus. Scientists and engineers  across the country used the computing  and data power of Blue Waters to  tackle a wide range of problems,  including predicting the behavior of  complex biological systems and  simulating the evolution of the cosmos.  Because of the rapid evolution of  computer technology, by 2019, NSF no  longer considered Blue Waters to be  the leadership computing system for  fundamental science and engineering  research. Anticipating these  technological advances, in September  2018, NSF awarded about $63.0 million  to the University of Texas at Austin for  the follow-on project to Blue Waters.  Frontera was intended to provide three  to five times the computing capability  and twice the storage capacity to  support the increased computational  requirements for science and  engineering research. NSF also  anticipated that Frontera would help  inform science requirements and  reduce risks for LCCF, which is  planned to provide substantially more  computational capabilities than both  Blue Waters and Frontera. obligated $2 million from its Research and Related Activities account for  the design of LCCF. According to the project’s cooperative agreement,  NSF may provide additional funding to advance the design of LCCF— $3.5 million in fiscal year 2020 and $2.5 million in fiscal year 2022  following successful completion of the conceptual and preliminary design  reviews, respectively, subject to availability of appropriations.

As of September 2019, NSF had not yet formally identified risks for the  LCCF project because the project was early in the design stage. NSF  requires recipients to develop and follow formalized risk management  during the design and construction stages of major facility projects to  identify potential risks, assess the nature of those risks, and identify  actions that can be taken to either reduce the probability of those risks  occurring or reduce their impact to the project. NSF officials told us that  an assessment of risks associated with the LCCF project will be part of  the conceptual design review, planned for June 2020.

According to NSF officials, one anticipated challenge for the LCCF project  is the rapid pace of technological change in the field of high-performance  computing. The officials stated that forecasting the technology  marketplace in the future can be challenging as technology can change  radically because of external market forces. Conversely, the rapid pace of  change can also be an opportunity if the LCCF project can incorporate  the latest technological advances that result in the most advanced  computing capabilities. According to NSF officials, taking advantage of  such opportunities as late in the design stage as possible will be  important for the success of the project.

Appendix III: Comments from the National Science Foundation Office of the Director

Appendix IV: GAO Contact and Staff Acknowledgments

John Neumann, (202) 512-6888 or neumannj@gao.gov   In addition to the contact named above, Joseph Cook (Assistant Director),  Sean Manzano (Analyst in Charge), Louise Fickel, Yvette Gutierrez,  Patrick Harner, Douglas G. Hunker, Jason T. Lee, Serena Lo, and Anika  McMillon made key contributions to this report.

The National Science Foundation funds construction of large science and engineering infrastructure projects, like telescopes, that can take years to build and cost hundreds of millions of dollars.

Since our March 2019 report, 4 major NSF construction projects had no increases in their authorized costs or schedules. For example, the Daniel K. Inouye Solar Telescope is on track to be completed within its $344.1 million budget by June 2020.

NSF implemented 2 of our 6 prior recommendations, including revising policies for its major-project cost estimates. NSF has begun to address the remaining recommendations on its project management practices.