Agile Software Development: DHS Has Made Significant Progress in Implementing Leading Practices, but Needs to Take Additional Actions

Many of DHS's major IT acquisition programs have taken longer than expected to develop or failed to deliver the desired value. In April 2016, to help improve the department's IT acquisition and management, DHS identified Agile software development as the preferred approach for all of its IT programs and projects.

GAO was asked to examine DHS's adoption of Agile software development. The objective of this review was to assess the extent to which DHS has addressed selected leading practices for its transition to the use of Agile software development.

GAO identified leading practices for planning, implementing, and measuring organizational change that apply to DHS's transition to Agile through its review of guidance published by the Project Management Institute and GAO. GAO also reviewed work it performed to develop leading practices for Agile software development adoption. GAO analyzed DHS documentation, such as policies, guidance, plans, and working group artifacts and assessed them against the selected leading practices. GAO also reviewed the implementation of selected practices within individual IT projects. Finally, GAO interviewed DHS officials to discuss any practices that were not fully implemented.

The Department of Homeland Security (DHS) has taken steps to implement selected leading practices in its transition from waterfall, an approach that historically delivered useable software years after program initiation, to Agile software development, which is focused on incremental and rapid delivery of working software in small segments. As shown below, this quick, iterative approach is to deliver results faster and collect user feedback continuously.

DHS has fully addressed one of three leading practice areas for organization change management and partially addressed the other two. Collectively, these practices advise an organization to plan for, implement, and measure the impact when undertaking a significant change. The department has fully defined plans for transitioning to Agile development. DHS has partially addressed implementation—the department completed 134 activities but deferred roughly 34 percent of planned activities to a later date. These deferred activities are in progress or have not been started. With respect to the third practice, DHS clarified expected outcomes for the transition, such as reduced risk of large, expensive IT failures. However, these outcomes are not tied to target measures. Without these, DHS will not know if the transition is achieving its desired results.

DHS has also addressed four of the nine leading practices for adopting Agile software development. For example, the department has modified its acquisition policies to support Agile development methods. However, it needs to take additional steps to, among other things, ensure all staff are appropriately trained and establish expectations for tracking software code quality. By fully addressing leading practices, DHS can reduce the risk of continued problems in developing and acquiring current, as well as, future IT systems.

GAO is making 10 recommendations to DHS to implement selected leading practices for its transition to Agile software development. DHS agreed with GAO's recommendations and described actions taken and planned to address them.

The Department of Homeland Security (DHS) and its components invest  billions of dollars each year to acquire information technology (IT) and  other capabilities to support the department’s critical functions. However,  as we have previously reported, many of the department’s major IT  acquisition programs have taken longer than expected to develop and  implement, or have failed to deliver the desired value to mission  operations. As part of an effort to improve its IT acquisition and  management, in April 2016, the department identified Agile software  development as its preferred approach for all DHS IT programs and  projects. Such an approach—one form of incremental development—calls  for the rapid delivery of software in small, short increments.

You asked us to examine the department’s adoption of Agile software  development. Our specific objective was to assess the extent to which  DHS has addressed selected leading practices for its transition to the use  of Agile software development. To accomplish this objective, we  assessed the extent to which the department adhered to leading practices  in two specific areas: organizational change management and Agile  software development adoption.

With regard to organizational change management, we reviewed leading  practices published by the Project Management Institute and GAO on  organizational change management. Based on this review, we identified  fifteen leading practices. We then grouped these 15 practices into three  broad organizational change management areas: planning, implementing,  and measuring change.

To determine the extent to which DHS addressed leading practice areas  for organizational change management in its transition to Agile  development, we assessed DHS policies, procedures, guidance, plans,  and other working group artifacts and compared them against the leading  practices in the three areas. Our review also included analyzing DHS’s IT  Program Management Center of Excellence (ITPM COE) meeting  minutes, presentation slides, and status update charts. Further, we  interviewed officials from DHS headquarters lines of business to discuss  any practices in the three areas that were not fully addressed.  Specifically, we interviewed officials from the offices of the Chief  Procurement Officer, Chief Readiness Support Officer, Chief Financial  Officer, Chief Human Capital Officer, Chief Security Officer, the Chief  Information Officer (OCIO), Systems Engineer, and Test and Evaluation,  and the Joint Requirements Council.

With regard to leading practices for Agile software development adoption,  we reviewed work performed by GAO to develop generally accepted  leading practices. In developing these leading practices, GAO reviewed  information from a variety of sources related to Agile adoption and  compiled a draft of leading practices commonly mentioned across these  different sources. We then convened a working group of experts from the  public and private sectors and academia. This working group met three  times a year between August 2016 and August 2019 to review and  discuss these leading practices. More than 200 experts participated in the  meetings, including more than 20 officials from DHS. GAO received  comments from many of these experts both during these meetings and by  email after the meetings.

Based on this work, GAO developed a set of nine leading practices for  Agile adoption. The leading practices were described by a series of core  elements and core element expectations that, collectively, can be used to  assess the status of an agency’s implementation.

To determine the extent to which the department had addressed the  leading practices for the adoption of Agile development, we obtained and  assessed DHS policies, procedures, guidance, plans, and other  documentation such as systems engineering life cycle (SELC) technical  review completion letters, and compared them against the nine leading  practices. This included supplementary Agile documentation, such as  training materials prepared by the Homeland Security Acquisition Institute  for acquisition workforce certifications and webinars offered by the  Procurement Innovation Lab. We also interviewed department officials  responsible for the associated policies, procedures, guidance, plans, and  other documentation to discuss any practices that were not fully  implemented.

To supplement our assessment of the extent to which the department  addressed program process, and team activity and dynamics-level  leading practices, we also assessed selected projects’ implementation of  these practices. We selected only the projects supporting programs on  the Major Acquisition Oversight List because DHS expects these  programs to comply with its Agile instruction and acquisition management  policy. We then further limited the scope of projects to those within  components where GAO had not previously assessed a program using  Agile methods or was not in the process of assessing such a program.  We further refined our selection based on the following criteria: software  development life cycle methodology (iterative development only) and  project completion date (in-progress only).

We then selected a random sample of three projects, with no more than  one project selected from a component. The three case study projects we  selected were the 1) U.S. Coast Guard (USCG) Command, Control,  Communications, and Computers, Intelligence, Surveillance and  Reconnaissance (C4ISR) program New Asset Acquisition Offshore Patrol  Cutter project, with particular attention to the SeaWatch portion of this  project; 2) the U.S. Customs and Border Protection (CBP) Biometric Entry  Exit (BEE) program Air Exit project, with particular attention to the  Traveler Verification Services portion of this project; and 3) the U.S.  Immigration and Customs Enforcement (ICE) Student and Exchange  Visitor Information System (SEVIS) program 8001 project, with particular  attention to the SEVIS modernization portion of this effort.

To evaluate case studies’ implementation of selected leading practices,  we reviewed artifacts from the selected projects. In particular, we  reviewed artifacts demonstrating a project’s use of Agile including testing  metrics, evidence of Agile meetings, the existence of user stories and a  backlog, and the availability of Agile coaching and training. We then  interviewed officials responsible for program and project management  and representatives of groups responsible for software development for  the three selected case study projects to discuss gaps we identified. See  appendix I for a more detailed discussion of our objective, scope, and  methodology.

We conducted this performance audit from December 2017 to April 2020  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

DHS and its components invest billions of dollars each year to acquire IT  and other capabilities to support the department’s critical functions. The  department plans to spend approximately $2.3 billion on major IT  investments in fiscal year 2020.

However, DHS has faced long-standing challenges in acquiring and  managing IT. We have highlighted the department’s IT management  issues on our high-risk list since 2003 and have made numerous  recommendations to improve its IT management practices. For example,  in 2013, we testified that, out of 68 major IT investments that the  department had in development, 21 had one or more subsidiary projects  that were not meeting cost and/or schedule commitments due to technical  issues in the development phase, changes in agency priorities, or a lack  of understanding of user requirements, among other things.

Overview of Incremental and Agile Software Development

Many federal agencies, including DHS, are accustomed to using a  waterfall software development model. This type of model typically  consists of long, sequential phases, resulting in product delivery years  after program initiation. With many federal IT investments in a  development phase, it is important to ensure that agencies are making  the most efficient use of their financial resources through effective  management practices. However, as we have previously reported and  testified, historically federal IT projects often fail—that is, even after  exceeding their budgets by millions of dollars and delaying the schedules  by years—and the results do not meet requirements.

Recognizing the severity of challenges related to the government-wide  management of IT, in December 2014, federal IT acquisition reform  provisions (commonly referred to as FITARA) were enacted as a part of  the Carl Levin and Howard P. ”Buck” McKeon National Defense  Authorization Act for Fiscal Year 2015. One of the provisions requires  that the Office of Management and Budget (OMB) require in its annual IT  capital planning guidance that each covered agency’s chief information  officer (CIO) certify that IT investments are adequately implementing  incremental development, as defined in capital planning guidance  issued by OMB.

Agile software development—one form of incremental development— calls for the rapid delivery of software. Probably the most well-known  feature of Agile software development is iterative product development  and delivery; that is, development of software in segments that are  continuously evaluated against requirements. This method is well suited  for programs in which the final product is to include distinct features,  some of which may be discovered during the process rather than planned  at the beginning. These frequent iterations can effectively measure  progress and allow developers to respond quickly to feedback from  customers, thus reducing technical and programmatic risk. With its  emphasis on early and continuous delivery of working software, Agile can  be a valuable tool for agencies in mitigating schedule and budget risks.

Figure 1 compares requirements, design, development, and testing using  Agile software methods versus a traditional waterfall approach; illustrating  how requirements, design, development, and testing are performed  concurrently in smaller time-boxed iterations for Agile and sequentially in  waterfall development. As a result, using an Agile framework should  result in producing high-quality software with frequent reviews and  customer feedback to ensure that the highest value requirements are  being met. The figure assumes that planning for both Agile and waterfall  development has already occurred.

DHS Adopted Agile Software Development to Address IT Challenges

In February 2016, the DHS Under Secretary for Management announced  an effort to pilot the use of Agile development methodologies to improve  the department’s execution and oversight of IT acquisitions. This  resulted in five Agile pilot programs. Each pilot program was overseen by  a component integrated program team. Collectively, the first pilot  programs were also overseen and supported by a DHS integrated  program team. In April 2016, the department issued an Agile instruction,  which identified Agile software development as the preferred approach for  all DHS programs and projects that are to deliver an IT, or embedded-IT,  capability. The department also set an expectation for its component  CIOs to develop plans to increase the use of Agile development and  justify any major IT programs that did not intend to use Agile development  practices. Many DHS programs were already using Agile or similar  incremental development methods before the department identified it as  the preferred approach.

Roles and Responsibilities for Agile Programs

The DHS CIO, as the individual delegated departmentwide responsibility  for approving, managing, and overseeing all of the department’s IT  programs, sets the policies and procedures to help ensure Agile practices  meet the department’s goals and comply with acquisition management  policy. The DHS CIO is supported in this effort by the heads of other  major DHS lines of business, such as the Chief Procurement Officer.

Table 1 describes the roles and responsibilities that support Agile  development within the department.

Additionally, DHS established a headquarters-level team—the ITPM  COE—to collaborate across the department on improvements to policy,  governance, and acquisition guidance. In April 2017, the ITPM COE  assumed responsibilities for the department’s transition to Agile  development. The Office of the Chief Technology Officer (OCTO)  Strategic Technology Management (STM) division within the OCIO  facilitates the ITPM COE and serves as the official liaison between other  OCIO divisions, other partner headquarters directorate and management  offices, and operational components as needed.

GAO Previously Reported on Challenges in DHS’ Management of Agile Programs

We have reported on various programmatic and technical challenges that  were limiting DHS’ efforts on Agile programs. For example,  In 2016, we reported that the U.S. Citizenship and Immigration  Services Transformation program, which was using Agile software  development to modernize citizenship and immigration benefits  processing, needed to improve testing of its software code and ensure  its approaches to interoperability and end user testing met leading  practices. We made 12 recommendations to improve  Transformation program management, including ensuring alignment  among policy, guidance, and leading practices in areas such as Agile  software development and systems integration and testing. DHS  concurred with the recommendations and has thus far implemented  eight of them.

We reported in October 2017 that the Transportation Security  Administration Technology Infrastructure Modernization program had  not defined key roles and responsibilities, prioritized system  requirements, or implemented automated capabilities that were  essential to ensuring effective adoption of Agile. We made 14  recommendations including that DHS should prioritize requirements  and obtain leadership consensus on oversight and governance  changes. DHS concurred with the recommendations and to date has  implemented 13 of them.

In November 2018, we reported that the U.S. Secret Service OCIO  did not fully measure post-deployment user satisfaction with one  project supporting the Information Integration and Technology  Transformation investment. We made 13 recommendations to the  U.S. Secret Service including that the Secret Service establish a  process that ensures the CIO reviews all IT contracts, as appropriate;  and identify the skills needed for its IT workforce. DHS concurred with  the recommendations but has not yet implemented them.

We reported in April 2019 that the Federal Emergency Management  Agency Grants Management Modernization program had not yet fully  established plans for implementing new business processes or  established completed traceability of IT requirements. We made  eight recommendations to implement leading practices related to  reengineering processes, managing requirements, scheduling, and  implementing cybersecurity. DHS concurred with the  recommendations and has thus far implemented two of them.

According to the Project Management Institute, the practice of change  management is a comprehensive, cyclic, structured approach for  transitioning individuals, groups, and organizations from a current state to  a future state with intended business benefits. It helps organizations to  integrate and align people, processes, structures, culture, and strategy.  The Project Management Institute and GAO have both described leading  practices for effective organizational change management.

DHS Has Made Progress in Implementing Leading Practices, but Has Not Fully Addressed Others

Leading practices in organizational change management advise an  agency to (1) plan for, (2) implement, and (3) measure the impact when  undertaking a significant change, such as a transition from one software  development approach to another. Since DHS committed to its  transition to Agile software development in policy in April 2016, the  department has fully developed plans to facilitate the transition. However,  DHS has not fully implemented these plans and has experienced  challenges in measuring progress against its intended goals. In addition,  many of the plans are part of a larger effort to improve overall IT  acquisitions rather than specific to a transition to Agile development, an  approach that may delay DHS’s execution of these plans.

Leading practices for Agile software development adoption advise an  agency to focus on three organizational levels of adoption: (1) agency  environment, (2) program processes, and (3) team activities and  dynamics. DHS has partially adopted practices at all three organizational  levels. For example, the agency activities fully supported Agile methods  through actions such as developing policies and procedures that called  for the alignment of software, program goals, and agency goals.  However, the department’s culture can better support Agile methods by,  among other things, demonstrating an incentives and rewards structure to  incentivize Agile teams.

DHS Has Made Progress in Implementing Nine Leading Practices for Agile Software Development Adoption, but Has Not Fully Implemented All

Leading practices that we developed for Agile software development  adoption are organized into three areas, called organizational levels:  agency environment, program processes, and team activities and  dynamics. The organizational levels are further divided into nine leading  practices. Table 2 identifies the three organizational levels and nine  leading practices associated with these levels (three practices within each  area). A detailed assessment of DHS’s implementation of each of the  nine leading practices can be found in appendixes III, IV, and V.

We refer to the leading practices related to an agency’s processes,  culture, and acquisition strategies as agency environment practices. For  an agency to successfully transition from an agency that supports  traditional development methods, it should ensure that its activities,  culture, and acquisition policy and procedures support Agile methods.  DHS partially implemented the agency environment practice level by fully  implementing two leading practices and partially implementing the  remaining one. A more detailed assessment of DHS’s agency  environment leading practices can be found in appendix III.

Agency activities support Agile methods–fully implemented. DHS  established appropriate life cycle activities to support Agile methods.  For example, the department has outlined its policies, procedures,  and guidance in several documents to assist its components in the  acquisition and implementation of Agile software development. The  department also developed policies and procedures that called for the  alignment of software, program goals, and agency goals.

Agency culture supports Agile methods–partially implemented.  DHS established an environment that supported Agile development,  and senior stakeholders supported its development throughout the  agency. However, DHS did not take sufficient steps to ensure that  senior stakeholders serving as executive sponsors understood Agile  development, as called for by leading practices that are described in  further detail in appendix III. The Director of STM stated that Agile  sponsors were considered to be chief executive officers (e.g.  Executive Director of PARM and the Deputy Under Secretary for  Management). These parties oversaw the actions of the ITPM COE  and approved the Agile action plans in June 2017.

In addition, the department did not require training for senior  stakeholders serving as executive sponsors, as called for by leading  practices. In a written response, the Office of the Chief Human Capital  Officer said that there were no Agile training requirements for officials  at this level. By training executive-level sponsors in Agile  development, the department can mitigate the risk of setting  expectations for programs and projects that do not align with the  values and principles of Agile software development.

DHS also did not demonstrate that it established an incentives and  rewards structure to incentivize Agile teams, as called for by leading  practices. Officials from the Office of the Chief Human Capital Officer  stated that the department’s existing rewards structure allowed for  incentivizing team and individual performance even though it was not  focused specifically on Agile methods. These officials stated that they  did not believe that additional policy, guidance, or modifications to  their existing policy were necessary. The Director of STM within OCIO  stated that rewarding Agile teams was not a topic the ITPM COE was  currently considering, but that OCIO might be interested in pursuing  the topic after completing existing, higher-priority activities. By  considering modifications to policy and guidance governing the  incentives and rewards structure to promote team performance, DHS  could improve team productivity and output.

Agency acquisition policies and procedures support Agile  methods–fully implemented. DHS guidance for acquisition  strategies supported the unique needs of Agile programs. For  example, DHS offered guidance for preparing acquisition strategies  through its Procurement Innovation Lab and published Agile guidance  that discussed contracting and acquisition strategies.

Program processes involve staff being appropriately trained in Agile  methods, technical environments enabling Agile development, and project  planning controls being compatible with Agile development. DHS partially  implemented the program processes practice level by fully implementing  one leading practice and partially implementing the remaining two. A  more detailed assessment of DHS program process leading practices can  be found in appendix IV.

Staff are appropriately trained in Agile methods–partially  implemented. DHS training policy and guidance called for some of  the acquisition management program staff to be trained in Agile  methods. DHS has also taken steps to incorporate Agile concepts into  required training for members of the acquisitions workforce. In  addition, DHS offered elective training covering Agile methods and  guidance for Agile teams, including contractors, to have the  appropriate technical expertise needed to perform their role.

The department also took steps to identify the necessary  competencies for Agile teams and individuals. In April 2019, the  Strategic Workforce Planning team within OCIO published a white  paper identifying 27 competencies necessary for teams and  individuals to use and training courses associated with the  competencies. The white paper also made recommendations to help  DHS address challenges in implementing Agile methods, such as  establishing communities of practice for Agile practitioners to identify  best practices and provide workshops. According to a written  response by OCIO, the Strategic Workforce Planning team will create  an implementation and communication plan for any deliverables  associated with the white paper.

However, the department did not provide policy or guidance to ensure  that all program staff were trained in Agile methods, as called for by  leading practices described in further detail in appendix IV. Existing  Agile training requirements covered only the acquisitions workforce.  DHS did not establish training requirements for program staff outside  of the acquisitions workforce—such as a product owner or other  staff—who may be assigned to an Agile program. As a result,  individual programs must independently decide on and enforce  training requirements if they want to ensure that all staff receive the  needed training.

DHS officials stated that the department focuses on key acquisition  career fields in part because those career fields are defined in policy  and procedures. According to the Director of STM, the department  also encourages programs to independently find coaching and  training because the components are more likely to have funding. By  providing policy or guidance to ensure that all personnel staffed to an  Agile program or project receive appropriate training, the department  can better prepare program staff to plan and execute appropriately,  and increase the likelihood of achieving the expected outcomes of the  transition to Agile.

Technical environments enable Agile development–fully  implemented. DHS guidance called for technical and project tools to  be available to support Agile development. For example, DHS test  and evaluation guidance stated that automated testing should be  implemented where practical.

In addition, DHS guidance called for system designs that will support  iterative delivery. For example, DHS enterprise architecture guidance  and supplementary design considerations for acquisition programs  discussed loose coupling and different methods for establishing a  modular system.

Project planning controls are compatible with Agile  development–partially implemented. DHS guidance called for  defining and incorporating non-functional requirements and critical  features throughout development. In addition, DHS provided guidance  for establishing a sustainable development pace. For example, the  Agile instruction manual identified the benefits of monitoring the  amount of work completed by Agile teams across each iteration in  order to monitor ongoing team progress.

However, DHS was not tracking and monitoring the pace of Agile  team development as called for by DHS guidance and described  further in appendix IV. According to the Director of STM, programs  were not consistently reporting the Agile core metrics associated with  development team pace as required. The Director of STM stated that  the department was taking steps to begin tracking and monitoring the  pace of Agile teams. In addition, the Director stated that he allocated  staff to assist programs with consistently reporting the Agile core  metrics.

According to the Director of STM, the department was in the process  of updating the core metrics and intended to publish a new version of  them in the future, which would include tracking the pace.  Nevertheless, DHS did not provide assurance that the metrics  associated with development pace would be included in this revised  set of metrics or that programs would consistently report that  information in order for the department to track and monitor the pace  of Agile teams. Until the department consistently tracks and monitors  Agile programs and projects, it will not have the information needed to  help ensure the development pace is maintained.

Practices at the team activities and dynamics level include team  composition supporting Agile methods, work being prioritized to maximize  value for the customer, and repeatable processes being in place. DHS  partially implemented the team activities and dynamics practice level by  fully implementing one leading practice and partially implementing the  remaining two. A more detailed assessment of DHS team activity and  dynamics leading practices can be found in appendix V.

Team composition supports Agile methods—fully implemented.

DHS established guidance that called for self-organizing teams and  defined the role of a product owner. For example, the Agile instruction  and Agile instruction manual both explain that collaborative, self- organizing, and cross-functional teams help achieve the flexibility  needed for the iterative development that characterizes Agile  development methods. In addition, the Agile instruction manual states  that the product owner is responsible for representing stakeholders  and should be available to the development team throughout the  iteration to answer questions and clarify requirements on behalf of the  stakeholders.

Work is prioritized to maximize value for the customer—partially  implemented. DHS guidance called for Agile teams to craft user  stories to define work. The guidance also called for user stories to be  prioritized in a backlog based on value.

However, the guidance did not describe how Agile teams can  estimate the relative complexity of the user stories as called for by  leading practices and described in further detail in appendix V. The  Director of STM stated that relative estimation is a basic exercise and  that guidance on this topic can be found in a number of sources  outside of DHS. However, without providing guidance or directing  Agile teams to external sources for additional information on relative  estimation, OCIO risks that teams supporting Agile projects will not  appropriately estimate user stories relative to each other.

By providing guidance on estimating the relative complexity of user  stories, the department can help Agile teams to effectively commit to  an appropriate amount of work during a given iteration.

Repeatable processes are in place—partially implemented. DHS  guidance addressed holding daily meetings to review progress and  discuss impediments, using a demonstration for the acceptance of a  user story and conducting a retrospective to evaluate progress. In  addition, the department’s guidance called for Agile programs to  employ continuous integration and emphasized the need for  mechanisms to help ensure code quality.

However, DHS did not set expectations for automated testing and  code quality, as called for by DHS guidance and described further in  appendix V. DHS’s Agile core metrics included a series of metrics that  addressed automated testing and code quality. The core metrics  included targets but the targets were notional and, therefore, not  expectations that DHS required a program to meet. According to the  Director of STM, the initial core metrics were intended to assess the  level of DHS team achievement without imposing artificial industry- based target measures for each. The Director stated that, on receiving  the metrics for a period of time, the department would then adjust the  core metrics and begin to include target measures based on the  results achieved. According to the Director, this effort is currently  underway and an updated set of core metrics will be distributed in  early fiscal year 2020.

Moreover, the department did not track and monitor automated testing  or code quality against expectations. As discussed under project  planning controls, DHS intended to track and monitor Agile practices,  such as automated testing and code quality, through the Agile core  metrics. However, according to the Director of STM, programs and  projects were not consistently reporting these core metrics and those  that were reporting did not collect data or report on particular metrics.

By setting expectations for automated testing and code quality and  beginning to track and monitor project performance against these  expectations, DHS can increase the likelihood that Agile programs  and projects are delivered within cost, schedule, and performance  estimates.

DHS has taken many positive steps in its transition to Agile software  development. It has implemented activities and artifacts that support all  levels of adoption, from the department and component offices to Agile  programs, projects, and teams. These activities and artifacts include  providing opportunities for Agile programs and projects to streamline  acquisition and life cycle processes to allow for iterative delivery and  exhibiting senior support for the transition to Agile.

The department successfully planned for the transition to Agile software  development and completed many of its intended implementation  activities. However, because DHS did not assess the skills and resources  needed to complete deferred activities, it risks continued delays in  completing these. In addition, without identifying target measures tied to  expected outcomes, the department is limited in determining whether the  transition is achieving its desired outcomes. Moreover, until DHS can  ensure that all programs are consistently reporting on Agile core metrics,  the department will not be able to track programs’ development  techniques. Further measuring and communicating the benefits of the  transition can enable the department to know whether Agile programs are  performing better than those used prior to the transition.

DHS has demonstrated significant progress in implementing leading Agile  practices. The department can further improve its performance through  full execution of the remaining partially implemented practices. At the  agency environment level, DHS can mitigate risk and improve productivity  through executive level training and modifications to policy to incentivize  Agile teams. For program level practices, addressing training  requirements for all necessary staff and tracking and monitoring the pace  of Agile team development can help ensure teams’ success.

With respect to team-level practices, DHS has not established guidance  for estimating the relative complexity of user stories. As a result, Agile  teams are hampered in effectively committing to an appropriate amount of  work during a given period of time. Finally, because DHS has not set  expectations for performance metrics for monitoring and tracking the use  of automated testing and code quality, DHS is at a greater risk for  programs breaching their cost and schedule expectations.

We are making the following 10 recommendations to the Secretary of the  Department of Homeland Security (DHS).

The Secretary should ensure that the Director of Strategic Technology  Management (STM), in collaboration with other members of the  Information Technology Program Management Center of Excellence  (ITPM COE), identifies the skills and resources needed to complete the  work intended for the upcoming fiscal year, including the availability of  supplementary staff, such as subject matter experts. (Recommendation  1)

The Secretary should ensure that the Executive Steering Committee  overseeing the activities of the ITPM COE establishes target measures  for the department’s desired outcomes of its transition to Agile  development. (Recommendation 2)

The Secretary should ensure that the DHS Chief Information Officer (CIO)  defines a process and associated set of controls to ensure that Agile  programs and projects are reporting a set of core required performance  metrics for monitoring and measuring Agile adoption. (Recommendation  3)

The Secretary should ensure that the ITPM COE, in coordination with the  CIO, begins measuring results associated with the transition to Agile and  the success of the transition based on its impact on the department.  (Recommendation 4)

The Secretary should ensure that the CIO, in collaboration with the Chief  Procurement Officer, through the Homeland Security Acquisition Institute,  establish Agile training requirements for senior stakeholders.  (Recommendation 5)

The Secretary should ensure that the Chief Human Capital Officer, in  collaboration with the CIO, consider modifications to the current employee  recognition and performance management governance to ensure that  teamwork and team performance of Agile programs and projects are  incentivized. (Recommendation 6)

The Secretary should ensure that the CIO, in collaboration with the Chief  Procurement Officer, through the Homeland Security Acquisition Institute,  establish Agile training requirements for staff outside of the acquisition  workforce but assigned to Agile programs. (Recommendation 7)

The Secretary should ensure that the CIO, upon establishing a set of core  performance metrics, tracks and monitors the pace of Agile team  development. (Recommendation 8)

The Secretary should ensure that the CIO, in collaboration with the  Executive Director of the Office of Program Accountability and Risk  Management (PARM), update or develop new guidance on Agile  methodologies to describe how Agile teams can estimate the relative  complexity of user stories. (Recommendation 9)

The Secretary should ensure that the CIO, upon establishing a set of core  performance metrics, sets expectations for automated testing and code  quality, and tracks and monitors against those expectations.  (Recommendation 10)

DHS provided written comments on a draft of this report. In its comments  (reproduced in Appendix VI), the department agreed with our 10  recommendations and described actions that it had completed and  planned to address them.

Based on the actions DHS said it had taken, the department requested  that we close the first three recommendations as implemented. For  example, the department described steps it had taken to address our  recommendation that it identify the skills and resources needed to  complete the work intended for the upcoming fiscal year, including the  availability of supplementary staff such as subject matter experts. In  addition, the department stated that it had addressed our  recommendation to define a process and controls to ensure that Agile  programs and projects are reporting a set of core required performance  metrics for monitoring and measuring Agile adoption. We plan to follow up  with DHS to assess the sufficiency of its actions to address our  recommendations.

The department also described actions that it plans to take to address the  other seven recommendations. For example, DHS stated that it will use  the results of its Agile core metrics and Agile Software Delivery Maturity  Model to measure the success of the transition to Agile and its impact on  the department. According to the department, it expects this action to be  completed by June 30, 2021.

Further, DHS stated that it will identify Agile training requirements for staff  in Agile programs, and will use that to establish Agile training  requirements for staff outside of the acquisition workforce but assigned to  Agile programs. Specifically, DHS stated that the DHS OCIO will gather  requirements from components via its IT workforce planning integrated  project team to identify training resources available across the  department that also address the skill sets needed for Agile programs.  The department added that the DHS OCIO will utilize information from the  April 2019 white paper, titled “OCIO Agile White Paper” to inform  proposed Agile program training requirements. The department estimated  that these actions are to be completed by September 30, 2020.

DHS also provided technical comments, which we have incorporated as  appropriate.

We are sending copies of this report to the Acting Secretary of Homeland  Security and interested congressional committees. In addition, the report  will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report   are listed in appendix VII. In addition, the report is available at no charge  on the GAO website at http://www.gao.gov.

Appendix I: Objective, Scope, and Methodology

Our objective was to assess the extent to which the Department of  Homeland Security (DHS) addressed selected leading practices for its  transition to the use of Agile software development. To accomplish this  objective, we assessed the extent to which the department adhered to  leading practices in two specific areas: organizational change  management and Agile software development adoption.

With regard to organizational change management, we reviewed leading  practices published by the Project Management Institute and GAO.  Based on this review, we identified 15 leading practices. We then  grouped these 15 practices in three broad organizational change  management areas: planning, implementing, and measuring change.

To determine the extent to which DHS addressed leading practices for  organizational change management in its transition to Agile development,  we assessed DHS policies, procedures, guidance, plans, and other  working group artifacts and compared them against leading practices. In  particular, we reviewed working group charters for the DHS headquarters  Agile Acquisition Integrated Program Team and IT Program Management  Center of Excellence (ITPM COE), and any plans developed by these  working groups, including the DHS Agile Action Plans and associated  implementation plans. We then reviewed working group meeting minutes,  presentation slides, and status update charts to assess the progress of  the transition to Agile, identified artifacts prepared to support the transition  to Agile, and assessed the status of plans for the transition to Agile. We  reviewed all Agile artifacts prepared by or supporting the Agile working  groups, such as a preliminary software development maturity model, the  DHS Agile Acquisition Software Delivery Core Metrics (Agile core  metrics), and an updated test and evaluation master plan template for  Agile, among other artifacts.

We also interviewed officials from DHS headquarters line of business  representatives explicitly identified in the Agile Development and Delivery  for Information Technology instruction (Agile instruction). This included  officials from the Office of the Chief Procurement Officer, Office of the  Chief Financial Officer, Office of the Chief Information Officer (OCIO),  Office of Program Accountability and Risk Management (PARM), and the  Science and Technology Directorate, and offices of Test and Evaluation  and Systems Engineering. Within OCIO, we interviewed officials from the  Office of the Chief Technology Officer (OCTO) within the Strategic  Technology Management (STM) division, among others, as STM is the  entity tasked with facilitating the ITPM COE and serves as the official  liaison between other OCIO divisions, other partner headquarters  directorate and management offices, and operational components. We  also interviewed representatives from groups participating in ITPM COE  activities but not explicitly called out in the Agile instruction, including the  Privacy Office and Joint Requirements Council. In addition, we  interviewed representatives from other groups not represented on the  ITPM COE but potentially impacted by the transition to Agile. This  included officials from the Office of the Chief Readiness Support Officer  and Office of the Chief Human Capital Officer.

With regard to leading practices for Agile software development adoption,  we reviewed work performed by GAO to develop generally accepted  leading practices. In developing these leading practices, GAO reviewed  information from a variety of sources related to Agile adoption and  compiled a draft of leading practices commonly mentioned across these  different sources. We then convened a working group of experts from the  public and private sectors and academia. This working group met three  times a year between August 2016 and August 2019 to review and  discuss these leading practices. More than 200 experts participated in the  meetings, including more than 20 officials from DHS. GAO received  comments from some of these experts both during these meetings and by  email after the meetings.

Based on this work, GAO developed a set of nine leading practices for  Agile adoption. GAO grouped these leading practices into three  organizational levels: (1) agency environment, (2) program processes,  and (3) team activities and dynamics. The leading practices were further  described by a series of core elements and core element expectations  that, collectively, can be used to assess the status of an agency’s  implementation.

To determine the extent to which the department had implemented the  leading practices for the adoption of Agile development, we obtained and  assessed DHS policies, procedures, guidance, plans, and other  documentation and compared them against the nine leading practices. In  particular, we reviewed department acquisition policy, procedures, and  guidance, such as acquisition management directive 102-01; software  engineering life cycle policy, procedures, and guidance, such as those  published in the software engineering life cycle guidebook; requirements  policy, procedures, and guidance, such as the Joint Requirements  Integration and Management System and Requirements Engineering  User’s Guide; testing policy, procedures, and guidance, such as the Test  and Evaluation Master Plan template and Test and Evaluation  Management Guide; technical assessment and enterprise architecture  policy, procedures, and guidance; program health assessment policy,  procedures, and guidance such as the Acquisition Program Health  Assessment instruction and CIO Program Health Assessment Scoring  Guideline; and Agile-specific policy, procedures, and guidance, such as  the Agile instruction and the Agile Development and Delivery for  Information Technology Instruction Manual (Agile instruction manual),  among other policy, procedures, and guidance.

In addition to reviewing the department policy, procedures, and guidance,  we obtained and assessed supplementary Agile documentation. In  particular, we reviewed training materials prepared by the Homeland  Security Acquisition Institute for acquisition workforce certifications and  webinars offered by the Procurement Innovation Lab; ITPM COE Agile  artifacts discussed under our assessment of the implementation of  organizational change management leading practices, such as the Agile  core metrics; and Agile-specific technical review completion letters, such  as the release planning review.

We also interviewed officials from the components responsible for the  associated policy, procedures, and guidance and those specifically cited  in the Agile instruction. This included officials from the Office of the Chief  Procurement Officer, Office of the Chief Financial Officer, OCIO, PARM,  Science and Technology Directorate, offices of Test and Evaluation and  Systems Engineering, the Joint Requirements Council, Office of the Chief  Readiness Support Officer, and Office of the Chief Human Capital Officer.  As with our assessment of DHS implementation of organizational change  management practices, within OCIO, we interviewed officials from the  OCTO STM division, among others.

We assessed a core element as being “met” if the department provided  supporting documentation that demonstrated it met all of the expectations  associated with the core elements. We assessed a core element as being  “partially met” if the department provided supporting documentation that  demonstrated some, but not all, aspects of the underlying expectations.  We assessed a core element as “not met” if the officials did not provide  any supporting documentation for the core element, or if the  documentation provided did not demonstrate any aspect of the underlying  expectations. The expectations associated with each core element are  described more fully in appendixes III, IV, and V.

We assessed each leading practice and practice level as being “fully  implemented” if DHS provided evidence that it had met all of the core  elements. We assessed each leading practice and practice level as being  “not implemented” if DHS did not provide evidence that it had met or  partially met any of the core elements. We assessed each leading  practice and practice level as being “partially implemented” if DHS  provided evidence that it had not met all core elements and partially met  at least one core element.

To supplement our assessment of the department’s implementation of the  leading practices for adopting Agile development, we also assessed  selected projects’ implementation of selected program process and team  activity and dynamics leading practices. We updated the core element  test plans to include general control objectives, associated controls, and  associated test steps in order to reach a determination on the extent to  which these projects implemented a particular aspect of a leading  practice.

We identified potential case study projects based on data provided by  DHS from the Investment Evaluation, Submission, & Tracking system.  We determined that the data in the Investment Evaluation, Submission, &  Tracking system was sufficiently reliable for our use in selecting projects  for our case studies. We selected case study projects, rather than  programs, because, according to DHS officials from OCIO, programs  report software development life cycle data to the Investment Evaluation,  Submission, & Tracking system at the project level only.

We selected only the projects supporting programs on the Major  Acquisition Oversight List because these programs are expected to  comply with the Agile instruction and acquisition management policy. We  then further limited the scope of projects to those within components  where GAO has not previously assessed a program using Agile methods  or was not in the process of assessing such a program. This excluded the  U.S. Citizenship and Immigration Services, Federal Emergency  Management Agency, Transportation Security Administration, and U.S.  Secret Service.

We then further refined our selection based on the following criteria:

Software development life cycle methodology (iterative development  only)

Project completion date (in-progress only)

DHS component (selection of only one project per component)

We then selected a random sample of three projects, with no more than  one project selected from a component. The three case study projects we  selected were the U.S. Coast Guard (USCG) Command, Control,  Communications, and Computers, Intelligence, Surveillance and  Reconnaissance (C4ISR) program New Asset Acquisition Offshore Patrol  Cutter project, with particular attention to the SeaWatch portion of this  project; the U.S. Customs and Border Protection (CBP) Biometric Entry  Exit (BEE) program Air Exit project, with particular attention to the  Traveler Verification Services portion of this project; and the U.S.  Immigration and Customs Enforcement (ICE) Student and Exchange  Visitor Information System (SEVIS) program 8001 project, with particular  attention to the SEVIS modernization portion of this effort. In preliminary  interviews, we confirmed that these projects were applying Agile practices  in order to validate data reported to the Investment Evaluation,  Submission, and Tracking system.

These case studies were used to supplement our findings from our  program process and team activity and dynamics-level evaluations of the  department’s implementation of leading practices for adopting Agile  development. To evaluate case studies’ implementation of these leading  practices, we reviewed artifacts from the selected projects. In particular,  we reviewed artifacts demonstrating a project’s use of Agile including  testing metrics, evidence of Agile ceremonies, the existence of user  stories and a backlog, and the availability of Agile coaching and training.

We then interviewed officials responsible for program and project  management and representatives of groups responsible for software  development for the three selected case study projects to discuss gaps  we identified. We shared our initial assessment with DHS, USCG, CBP,  and ICE and obtained feedback and additional supporting documentation.

Regarding our analysis of project implementation of the program process  and team activity and dynamics core elements, we followed the  aforementioned process in assessing a core element as being “met”,  “partially met”, or “not met”. These assessments were used to gain insight  into the extent to which DHS policy, procedures, and guidance prepared  programs and projects for the successful adoption of Agile leading  practices. We did not evaluate the projects in order to make specific  recommendations to the individual projects.

We conducted this performance audit from December 2017 through April  2020, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

In June 2017, Department of Homeland Security (DHS) senior  stakeholders endorsed Recommendations Action Plans: Agile Acquisition  Pilots, developed by the Agile Acquisitions Working Group. These  recommendations were an effort to sustain the success of the information  technology (IT) acquisition and delivery pilot program. The action plans  were developed in response to the February 18, 2016, Acquisition  Decision Memorandum from the Under Secretary for Management, which  recognized the expressed need for both components and headquarters  directorates to continue driving organizational change and process  improvement to DHS IT acquisitions and delivery. The action plans were  intended to codify lessons learned and recommendations based on  independent interviews and retrospective meetings with those who  participated in the five acquisition pilots. These plans were organized by  priority: 12 critical, three high, and three moderate. The recommendations  were weighted against one another based on impact, level of difficulty,  and alignment with the original five goals of the Agile acquisition pilot  program charter: reduce risk, increase customer value, faster time to  market, economic value, and increased accountability and oversight.  Table 3 describes the DHS Agile action plans, including the associated  goal, primary organization(s), level of difficulty, impact, and executive  priority.

Each DHS action plan included a problem statement and  recommendation, as detailed in table 4.

Appendix III: Leading Practices for Adopting Agile Development—Agency Environment

This appendix describes in detail our evaluation of the three leading  practices for agency environment when adopting Agile development,  including a further explanation of expectations for each practice as well as  some of the findings associated with each practice. We do not present  any additional recommendations from these findings; this information is  intended to assist the Department of Homeland Security (DHS) in  implementing the recommendations described in our report.

Establish appropriate life cycle activities  Agency activities should support Agile methods by allowing for  incremental and iterative software delivery that is tailored to the cadence  of Agile software development and by incorporating technical reviews that  occur throughout the development process. These activities and  supporting policy and guidance should allow for requirements to be  changed during development and the requirements change approval  process should not impede the cadence of iterative and incremental  development. Life cycle activities should also be user-focused and call for  collaboration between the development team and users.

To manage its multi-billion dollar investments, DHS has established  policies, procedures, and guidance for IT program management. These  publications govern the complete life cycle of a system, from technology  development through integration and testing and, finally, implementation  and operations and maintenance.

DHS has outlined its policies, procedures, and guidance in several  documents to assist its components in the acquisition and implementation  of software development. Policies for managing its major acquisition  programs are primarily set forth in a directive and supporting instruction.  These policies outline an acquisition life cycle framework (ALF) that  includes a series of predetermined milestones—known as acquisition  decision events—at which the Acquisition Decision Authority reviews a  program to assess whether it is ready to proceed to the next phase of the  ALF. DHS’s Under Secretary for Management serves as the Acquisition  Decision Authority for the department’s major acquisition programs.

A separate DHS instruction and associated guidebook outline a  framework of major systems engineering activities and technical reviews,  collectively considered the systems engineering life cycle (SELC), which  should be conducted by all DHS programs, both major and non-major.  The SELC helps to ensure that appropriate systems engineering activities  are planned and implemented and that a program’s development effort is  meeting business needs. The SELC consists of major activities and a set  of related technical reviews and artifacts that fit within the acquisition life  cycle.

Figure 2 depicts the acquisition life cycle and associated technical  reviews established in DHS acquisition management policy.

DHS provided programs with flexibility in their SELC technical reviews.  Within the ALF, Agile processes are applied primarily within the obtain  phase, where design, development, testing, and implementation of a  system takes place. Prior to entering the obtain phase, a program selects  its software development approach, such as Agile. The agreed-upon  approach is then codified in an SELC Tailoring Plan, which is approved at  acquisition decision event 2A. The SELC Tailoring Plan identifies the  technical reviews and artifacts that the program is responsible for  completing based on its unique characteristics (e.g., scope, complexity,  and risk).

To assist in tailoring efforts and further guide the implementation of Agile,  DHS published an Agile instruction that includes the scope, definitions,  roles and responsibilities, and procedures for establishing an Agile  framework for developing all DHS IT acquisitions. DHS supplemented  this instruction with an Agile instruction manual and provided a template  that Agile programs can follow to tailor their activities. For example,  instead of holding a system definition review, an Agile program is  encouraged to conduct a release planning review (which encompasses  the development and release of a segment of software). This optional  approach to tailoring a technical review is depicted in figure 3.

Outside of technical reviews, DHS updated acquisition policy in February  2019 and associated guidance in May 2019 to allow programs greater  flexibility in the larger ALF. The Director of Strategic Technology  Management (STM) stated that, under the previous acquisition policy and  guidance, IT programs were using in-house expertise due to limited  funding to prepare for the 2B decision, when full program funding was  received. He noted that, by the time a contract was awarded for  development following a 2B decision, the contractor might or might not  have been using planning artifacts developed by the program and instead  might have recreated them, thereby rendering 2 to 3 years of work  useless. The Director stated that programs were unable to fully flesh out  the program architecture and other key aspects of the program because  programs did not receive funding until the 2B decision and in-house  expertise was limited. For example, if a program had not proven out its  architecture prior to a 2B decision, it could continue to refine and modify  the architecture during the course of development, thereby impacting  productivity and quality. DHS updated acquisition management policy and  guidance to modify the requirements for the acquisition decision events  and addressed a related GAO recommendation.

DHS policy and guidance also allowed for programs to modify  requirements over the course of development. The traditional process for  requirements may be modified as part of tailoring the SELC in order to  allow for increased flexibility. The DHS Requirements Engineering User’s  Guide detailed requirements engineering steps, activities, and methods  for performing those steps.

DHS developed this user guide to supplement SELC policy and guidance.  One section of this guide focused on Agile development. According to the  guide, requirements are broken down over the course of the ALF and  commitments are made at different levels of specificity. Fundamental  capability gaps are defined in the mission needs statement presented at  acquisition decision event 1. Subsequently, the analyze/select phase  would ultimately define the high-level features and functions of each  required capability, define the fundamental performance of those high- level features and functions, and establish the business case to support  approving the acquisition at an acquisition decision event 2A. Often, a  preliminary concept of operations is developed and delivered with the  mission needs statement.

The guide also states that the activities to evaluate these potential  alternatives will ultimately result in a preferred solution with defined  business practices, methods, and processes that allow the development  of business epics and associated architecture epics. Business epic is an  Agile term that defines the high-level “stories” that describe a capability,  or what the new system is required to perform. Architectural epic is an  Agile term that defines the architecture the system will be incorporated  into. In addition, the preferred solution would have defined high-level  performance requirements (stated from the operational perspective) in  terms of how well the solution must perform to be operationally effective  and suitable. Key constraints such as security, Section 508 compliance,  privacy, reliability, etc., should also be identified. These top-level  requirements will be documented in the operational requirements  document.

According to the guide, Agile teams capture the capabilities and  constraints (essentially the functional and non-functional requirements  that reflect the business epic level of performance) in an artifact called the  capabilities and constraints document. Requirements statements in this  document should follow the standard “shall statement” format for ease of  translation between the operational requirements document and the  capabilities and constraints document. The capabilities and constraints  document and its contents mature over time and, as the document  matures, business and architectural epics decompose to  features/functions or themes, and ultimately to user stories that reflect the  specific tasks that users will perform. Officials within the DHS Joint  Requirements Council noted that headquarters involvement occurred at  this level to approve the high-level operating requirements.

After headquarters oversight and approval, the program may then  decompose requirements as part of planning for and executing technical  reviews. If tailored into an Agile program, the capabilities and constraints  document should drive the development of a backlog. The backlog is a  list of all the user stories that describe what the system needs to do. The  backlog should become more refined as the program decomposes the  high-level features (a service that fulfills a user need) and functions down  to specific stories that an individual software developer will code and test  during a specific iteration.

To prevent the backlog from becoming unmanageable, DHS guidance  stated that backlogs may be established at different levels. For example,  the business and architectural epics along with the associated operating  requirements would constitute the “program backlog.” Sub-epics are  usually broken down into “high-level features” with business epics broken  down into business features and architectural epics broken down into  architectural features. Features or functions are decomposed into detailed  stories that are then allocated to a “release”. The list of user stories in a  specific release constitutes the release backlog. This process of  decomposing stories continues to the iteration backlog.

DHS guidance places an emphasis on end user needs. The  Requirements Engineering User’s Guide raised the importance of  identifying stakeholders, including system users, and capturing the needs  of those users via requirements or, in the case of Agile, user stories. The  Agile instruction manual placed an emphasis on the importance of users  to a program and articulated that the product owner represent the user  community and was expected to continually seek ongoing feedback and  elicit requirements from users. The Agile core metrics also strongly  recommended the use of a net promoter score. This score was one  mechanism for measuring customer satisfaction through asking users to  rank how likely they would be to recommend the system or application to  a friend or colleague, based on a score of 1 to 10.

Clearly align goals and objectives  Program goals should clearly reflect stakeholder needs and concerns  based on input from stakeholders and stakeholder review and approval.  Program goals should align with strategic IT objectives. Software-related  goals should be defined and clearly aligned with program goals. The  agency should collect objective measures that are well defined to track  progress towards achieving software goals so the agency knows which  features and capabilities have been achieved.

The Requirements Engineering User’s Guide described program  expectations for tracing from mission needs to operating and functional  requirements, or user stories. The guide recognized that, as a program  progressed through the ALF and SELC, it was important to trace  requirements from the top-level mission needs or capabilities and/or  business requirements down to the system/sub-system, component, or  configuration item level that enabled those requirements to be met. This  helped ensure continuity across various DHS artifacts, such as the  program’s mission need statement, concept of operations, and  operational requirements document, to vendor specifications (or  applicable equivalent artifacts). Although an Agile program will modify the  SELC to accommodate its needs, generally programs were expected to  follow the same conceptual approach to the requirements of planning,  development, and management.

The user’s guide stated that collaboration among the various  stakeholders was important and the program requirements team must  continuously work to establish partnerships and networks. To do so, the  guide stated that the program team must identify all individuals and  organizations that may be impacted by their program and ensure those  stakeholders were engaged throughout development to facilitate  understanding of their perspectives and needs. The first step was to  identify applicable stakeholders, which would include end users, program  sponsors, developers, maintainers, trainers, and other affected individuals  or organizations. The program requirements team then solicited input  from these stakeholders to understand their needs, policies, processes,  and operations to begin the requirements definition effort. It identified  some ways a team might begin the process of eliciting requirements from  the stakeholders.

After collaborating with stakeholders, the stakeholder needs must be  translated into the program requirements, or goals. The guide stated that  the program requirements team should take the inputs from the various  stakeholders and decompose, prioritize, de-conflict, and validate the  needs identified. It clarified that a “good” requirement was achievable,  testable, clear, concise, technology-independent, feasible, and able to  stand alone.

The guide grounded all of the requirements elicitation and development  process in the overall contribution to the agency mission, recognizing the  need for general strategic alignment. In particular, the guide noted that  requirements were “mission need” driven as opposed to “solution” driven.  Requirements were developed throughout the life of a program, with the  first formal requirements being the operating requirements documented in  the operational requirements document.

To ensure that DHS’s mission or strategic goals were key inputs for  decision making, DHS relied, in part, on its enterprise architecture  process. DHS policy for enterprise architecture stated that the enterprise  architecture program provided a vehicle to tie the strategic mission goals  and objectives of DHS to the business processes, information resources,  and technology investments necessary to reach key performance  outcomes. This methodology was intended to integrate IT into the mission  and strategic priorities of DHS, which provided the core foundation for all  subsequent processes. DHS capital planning and investment control  guidance reinforced this fact, stating that the Federal Acquisition  Streamlining Act of 1994 required capital investments to align with  mission and strategic goals. This included the framework within which the  department formulated, managed, and maintained its portfolio of  investments as critical assets for achieving success in the DHS mission  and alignment to the DHS IT Strategic Plan and the DHS Strategic Plan.

Cascading sponsorship for Agile software development  Senior stakeholders should support and model the use of Agile, along  with its values and principles, through explicit policy or guidance  impacting the business and should take steps to complete responsibilities  defined in agency Agile policy or guidance. Agile should also be  supported in all relevant areas of the business impacting a software  development project through the use of Agile sponsors. These sponsors  should represent the lines of business in key agency decisions on Agile.

Senior stakeholders at DHS demonstrated support for Agile through the  publication of policy and guidance that established Agile development as  the department’s preferred approach for software development. As  discussed previously, the department published Instruction 102-01-004  Agile Development and Delivery for Information Technology (Agile  instruction), which provided the scope, definitions, roles and  responsibilities, and procedures to establish an Agile framework for the  development of IT acquisitions at DHS. Specifically, the Agile instruction  established responsibilities for the CIO, the Chief Procurement Officer,  the Chief Financial Officer, the Director, Office of Test and Evaluation  within the Science & Technology Directorate, and the Executive Director  of PARM.

Each of these five stakeholders and their associated components  demonstrated their support for Agile development by taking steps to  complete their responsibilities defined in the Agile instruction. For  example, the Office of the Chief Information Officer (OCIO), the Office of  the Chief Procurement Officer, and the Director, Office of Test and  Evaluation within the Science and Technology Directorate all had  responsibilities related to providing guidance for the implementation of  Agile within their specific area of expertise. All three components had  taken steps to execute these responsibilities, such as by publishing the  Agile instruction manual, providing supplementary guidance for test and  evaluation in an Agile environment, and offering elective training on  contracting strategies for Agile services.

Representatives from offices with a role in software development also  supported Agile via membership in the IT Program Management Center  of Excellence (ITPM COE). In addition to the stakeholder organizations  identified in the Agile instruction, the ITPM COE membership included  representatives from the Joint Requirements Council and the Chief  Privacy Officer. According to the ITPM COE charter, the ITPM COE  served as a cross-functional team to identify and promote best practices,  provide tools, and coordinate assistance for programs and projects to  maximize the successful management of DHS IT investments. This  included making progress towards the 18 Agile action plans that resulted  from the Agile acquisition pilots.

The ITPM COE membership requirements called for representatives of  the member organizations to be involved in key decisions regarding Agile.  According to the Director of STM within OCTO, ITPM COE members  were selected and approved by their organization’s executives. The ITPM  COE charter stated that these representatives must be authorized to  represent or make decisions on behalf of their officers or organizations.  Officials from all ITPM COE member organizations expressed support for  the ITPM COE and confirmed that their component was appropriately  represented in decision making. This was represented, in part, by the fact  that at least one representative for each ITPM COE member group  attended at least half of the meetings. For example, at least one  representative from the Science and Technology Directorate attended  approximately 95% of the meetings.

Sponsor understanding of Agile software development  Sponsors should understand and communicate changes resulting from  Agile development. Sponsors should attend training or receive coaching  on Agile and the agency’s framework, the agency should monitor  completion of training, and sponsors should transmit learning from  training to staff. Sponsors should also commit to achieving those intended  results and sponsor performance should be tied to achieving those  intended results.

The Director of STM stated that Agile sponsors were considered to be  chief executive officers (e.g. Executive Director of PARM and the Deputy  Under Secretary for Management). They oversaw the actions of the ITPM  COE and approved the Agile action plans in June 2017.

DHS did not ensure that Agile sponsors attended training or received  coaching in Agile development. The department made training available  for Agile, including courses such as those required for acquisition  professionals. However, in a written response, the Office of the Chief  Human Capital Officer stated, and the Director of STM confirmed, that the  department did not administer mandatory training on Agile for Agile  sponsors.

The department also did not monitor the completion of sponsor training in  Agile. Although DHS employees leveraged the Federal Acquisition  Institute Training Application System to track their training and  certifications, the department was not using this system to monitor  sponsor training in Agile. According to a written response from the Office  of the Chief Human Capital Officer, the department did not keep a record  of whether sponsors completed training in Agile because the department  did not require Agile training specifically for sponsors.

DHS Agile sponsors exhibited support for achieving the intended results  from the transition to Agile. Agile sponsors committed to achieving these  results through an endorsement of the 18 Agile action plans and the  associated implementation plans.

However, DHS did not demonstrate that Agile sponsor performance was  tied to achieving the intended results of the transition to Agile. According  to a written response from the Office of the Chief Human Capital Officer,  the department’s employee performance management policy did not  specifically address Agile. This written response further stated that  addressing Agile in these policies was unnecessary because the Office of  the Chief Human Capital Officer incorporated goals derived from project  plans in individual performance plans. DHS policy and guidance for  performance management identified individual performance goals as a  component of employee performance, but the department did not provide  evidence that specific performance plans for the sponsors were linked to  such goals.

Establish an environment supportive of Agile software development  Team dynamics should be facilitated through access to common team  rooms and/or modern communication and social media methods and  headquarters infrastructure operations should allow for communal spaces  and co-location in program offices. A headquarters technical environment  should allow access to tools by programs to foster distributed  communication, and there should be a process for continuous feedback  on the Agile environment and modifications to that process (e.g.  communities of practice, routine working group sessions). Agency  governance bodies should allow programs greater autonomy and  flexibility within existing acquisition processes through the modification of  gate reviews and other touchpoints in the acquisition process for Agile  projects and increased transparency for governance bodies into project  operations when necessary.

DHS policy and guidance allowed for team dynamics to be facilitated  through access to common team rooms and modern communication  methods. In addition, department policy promoted and allowed program  offices to support team dynamics through the use of communal spaces  and co-location. Specifically, the Director of Systems and Information  Integration within the Chief Readiness Support Office confirmed that DHS  had modified policy related to infrastructure operations to allow any office  to reorganize their space, citing the USCIS Transformation program as an  example of this reorganization. The Director of Systems and Information  Integration also noted that he was not aware of any restrictions to this  practice in policy. With respect to facilitating access to modern methods  of communication, DHS offered programs the option of using a suite of  tools that included those for distributed communication.

DHS took multiple steps to establish a process for continuous feedback  related to the department’s Agile environment and process modifications.  According to the Director of STM, OCIO built support for Agile through the  Centers of Excellence, communities of interest, brown bag lunches, and  public speaking engagements. The Director added that these sessions  facilitated the discussion of Agile and could be used to compile feedback.

The Director of STM explained that, as this feedback came in, it was  either addressed immediately or put into a backlog. Efforts to further  streamline the acquisition process were tracked via Agile action plan 6.

The department’s governance bodies also increased transparency into  project operations when necessary. The Agile Development and Delivery  for Information Technology Instruction Manual (Agile instruction manual)  stated that the program or project manager should coordinate with the  various oversight bodies that govern IT development. These bodies  varied depending on the level of investment, but, for major programs,  executive steering committees were often established to oversee all  aspects of program planning and execution between major acquisition  decision events. In addition, PARM officials stated that DHS increased  the frequency of acquisition review board reviews and modified the  content presented at the reviews to allow it to be more actively involved  with projects earlier in the acquisition life cycle. Specifically, PARM  updated the Acquisition Review Board slide templates and informed us of  its intent to update acquisition management policy to require Agile  projects to hold Acquisition Review Board reviews once every six months,  as opposed to once every 12 months.

Align incentives and rewards to Agile methods  The agency should establish an incentive and reward structure promoting  team successes and the value of individuals within those teams.  Management should establish agency goals to align incentives and  rewards with Agile methods. Goals for incentives and rewards should  align with the agency’s goal(s) and focus on team success. The agency  should allocate incentives and rewards based on team success.

DHS did not establish an incentives and rewards structure that promoted  team successes and did not demonstrate that management had  established agency goal(s) to align incentives and rewards with Agile  methods. Furthermore, the department did not demonstrate that human  resources and others were actively involved in setting goals for incentives  and rewards alignment.

DHS guidance specifically discussed contract incentives for Agile  projects. For example, the Agile instruction manual suggested that  consideration be given to address the duration of the base term and  options, scalability, deliverables, and pricing with a mindset that  contractors need appropriate incentives to encourage them to perform  well. The manual also stated that contract award terms could provide a  greater incentive for contractors working on longer-term Agile projects.

Although the department made efforts to adapt incentives and rewards for  contractors supporting Agile projects, it acknowledged that it did not  update existing incentives and rewards for federal employees working on  Agile projects. Officials within the Office of the Chief Human Capital  Officer stated that existing human capital and performance plan policy  allowed for rewarding and incentivizing Agile teams as well as individuals.  These officials further noted that DHS had numerous opportunities to  recognize and reward team or individual performance, regardless of the  development methodology a program relied on. Specifically, these  officials clarified that the Office of the Chief Human Capital Officer used  project plans to set goals and included those goals in employee  performance plans. As these officials felt the existing performance plan  policy was sufficient, they did not believe additional guidance or  modifications to existing policy were necessary.

Agency acquisition policies and procedures support Agile methods

Guidance is appropriate for Agile acquisition strategies  Agency acquisition policy and guidance should support awarding  contracts for the unique needs of an Agile program. Acquisition strategies  should recognize the need for interim delivery of software, allow for close  coordination between the contracting office and program office staff, and  allow for changing requirements and contract oversight mechanisms to be  tailored to support Agile development methods.

DHS offered guidance for preparing acquisition strategies through its  Procurement Innovation Lab. Webinars offered by the Procurement  Innovation Lab on acquisition strategies for Agile programs discussed the  need for interim delivery of software, close coordination between  contractors and program office staff, contract oversight mechanisms that  were tailored to support Agile development, and changing requirements.  For example, the “Transportation Security Administration Agile Services  Procurement” webinar discussed planning, executing, and de-briefing  technical demonstrations used to select the contract recipient, paying  particular attention to the value of transparency and modifying contract  oversight mechanisms. Officials from the Office of the Chief Procurement  Officer clarified that the webinars were available as needed and were not  required training.

DHS also published Agile guidance that discussed contracting and  acquisition strategies. From an oversight perspective, according to the  Agile instruction manual, DHS executive steering committees oversee all  aspects of program planning and execution between acquisition decision  events. This authority extends to assisting programs in developing  acquisition strategies where appropriate. The manual included a section  that specifically called out Agile contracting considerations that pointed  back to Office of Management and Budget Contracting Guidance to  Support Modular Development, the TechFAR handbook, the Digital  Services playbook, and innovative contracting case studies.

Among other useful information in the Agile instruction manual were key  contracting considerations for an Agile program or project manager.  These considerations included, among other things, frequent, iterative  deliveries of software, an ability to monitor changes to maintain contract  and project scope, flexibility to accommodate refinement of requirements,  transparency and collaboration, and prior experience in the Agile  methodology. The manual also highlighted goals for the acquisition to  discuss with a contracting officer, such as rapid contracting processes to  keep pace with Agile development, contracting to accommodate  incompletely defined scope and requirements, and the ability to respond  to requirements changes without requiring extensive change orders.

According to officials within OCTO and the Office of the Chief  Procurement Officer, the department also supported Agile programs in  preparing acquisition strategies through the IT acquisition review process.  This process was established to provide a mechanism for the DHS Chief  Information Officer to review and guide agency IT expenditures. The  process was intended to analyze IT acquisitions to ensure alignment with  DHS missions, goals, policies, and guidelines. This process relied on  subject matter experts to assist in the review of IT acquisitions, including  one for Agile reviews. According to the IT Acquisition Review Essentials  Guide, Agile reviews occurred where software was being developed to  ensure development activities adhered to Agile best practices and DHS  SELC guidance. The Agile subject matter expert was expected to review  acquisition materials against an established set of criteria for both the  acquisition plan and the requirements document. For example, when  reviewing the acquisition plan for approval, the subject matter expert  should consider if the statement of need adequately addresses Agile or  iterative project-specific activities and/or deliverables.

The Director of STM stated that there is one staff member in STM who  actively participates in the IT acquisition review process and was  responsible for ensuring Agile language was correctly implemented in  contract statements of work. The Director also added that they were  willing to help teams that were having trouble providing explanations of  Agile processes in their statements of work. The Director of STM stated  that there was no policy to guide his staff member reviewing Agile  language in the statement of work, but that he asked his division to put  together a checklist review to govern this process. The Director added  that the department sent programs and projects requiring assistance with  Agile contracting to the Procurement Innovation Lab by request to  streamline the acquisition plan.

According to the Leader of the Procurement Innovation Lab Team, the  Office of the Chief Procurement Officer was primarily focused on  supporting Agile pilot programs, such as the Federal Emergency  Management Agency Grants Management Modernization program. The  team leader noted that, while the procurement office supported these  programs, it relied on the program offices to ensure accuracy. For  example, the program management office ensures that the requirements  are structured and delivered, which could be challenging for Agile  programs. The team leader mentioned that a particular focus at the  moment was defining the pricing for contract line item numbers in such a  way as to afford the flexibility needed for Agile development while still  holding contractors accountable.

Appendix IV: Leading Practices for Adopting Agile Development—Program Processes

This appendix describes in greater detail our evaluation of the three  leading practices for program processes when adopting Agile  development. It does not present new findings; rather, the information is  intended to assist the Department of Homeland Security (DHS) in  implementing the recommendations described in this report.

Program processes refer to leading practices related to the program office  and technical environment. For programs to successfully transition from  processes used for traditional development projects, programs should  ensure that  staff are appropriately trained in Agile methods by ensuring Agile teams have the appropriate technical expertise  needed to perform their roles  technical environments enable Agile development through making technical and project support tools available, and designing a system that supports iterative delivery project planning controls are compatible with Agile methods by maintaining a sustainable development pace and tracking and defining and incorporating non-functional requirements in defining and incorporating critical features in development  The department develops an environment that supports these processes.  Within DHS, program management offices are responsible for planning  and executing individual programs and implementing applicable Agile  methodologies. In addition, the DHS Office of the Chief Information  Officer (OCIO) is responsible for setting policies and procedures to  ensure that programs leverage Agile development best practices to meet  the department’s goals and are within acquisition policy. The DHS OCIO  is also responsible for providing guidance for and reviewing the adoption  and execution of Agile development.

Staff are appropriately trained in Agile methods

Train all program staff in Agile methods  The agency should provide a training program in Agile for staff and track  and monitor the training. All members supporting the team, not only the  software development team, should be trained in the specific Agile  framework they will be using.

DHS required its acquisitions workforce to take training that incorporated  Agile methods. DHS Instruction 102-01-006, Acquisition Program  Management Staffing, established certifications for key acquisition career  fields, which included training requirements. According to the Associate  Director for Training from the Homeland Security Acquisitions Institute,  the certification requirements included training that has been updated to  incorporate Agile methods. Specifically, the department updated course  content for AQN 101: DHS Fundamentals of Systems Acquisition to  include Agile development concepts, such as small team management  and Agile metrics, following the issuance of department policy governing  Agile development. This course was required training for seven of the  acquisition career fields, including program and project managers,  systems engineers, and test and evaluation managers.

DHS tracked and monitored the completion of training requirements for  the acquisitions workforce. According to DHS Directive 064-04,  Acquisition Professional Career Information, component acquisition  executives were responsible for ensuring that acquisition personnel met  the mandatory training requirements. Officials from the Homeland  Security Acquisitions Institute within the Office of the Chief Procurement  Officer stated that DHS employees leveraged the Federal Acquisition  Institute’s training application system to track their training and  certifications. According to the catalog of product services of the institute,  members of the DHS acquisition workforce were required to attach copies  of their training certificates to request certification of completion of the  required training.

Because the DHS acquisitions workforce may not cover all personnel  staffed to Agile projects, some program staff may not be subject to  training requirements that incorporate Agile methods. According to the  Director of the Homeland Security Acquisition Institute, certain Agile team  members, such as the product owner, were not necessarily classified as  part of the acquisitions workforce. For example, according to the U.S.  Immigration and Customs Enforcement (ICE) Student and Exchange  Visitor Information System (SEVIS) program staffing plan, the product  owner role was not part of the acquisitions workforce and did not require  any certifications.

To help address the Agile training needs of all staff, including those who  are not part of the acquisitions workforce, DHS also provided elective  training in Agile methods. The department offered commercial training  through the Homeland Security Acquisition Institute, such as acquisition  of Agile services and Agile requirements for creating user stories. The  DHS Agile instruction manual also identified training offered by the U.S.  Citizenship and Immigration Services Office of Information and  Technology as another resource on Agile concepts, such as user stories  and automated testing.

In addition to elective training, the Agile Development and Delivery for  Information Technology Instruction Manual (Agile instruction manual)  encouraged program managers to seek out an Agile coach to help teams  adopt Agile methods and supplement training. The instruction manual  suggested that program managers should identify an Agile coach to serve  as an embedded trainer, consultant, and team advisor. This Agile coach  could help the team adapt Agile methods to their environment and work  through challenges. An Agile coach could also help individual team  members understand the responsibilities of their role on an Agile team.

Although DHS did not provide coaches for Agile teams, the department  offered resources that could help programs select and obtain an Agile  coach. First, the department established a blanket purchase agreement  for programs to acquire Agile development support in the form of hands- on coaching services for the design and use of Agile methods. According  to Homeland Security Acquisition Institute officials, this agreement would  enable programs and projects to acquire Agile coaching. Among other  things, this agreement defined the scope of Agile coaching services and  their pricing so that programs would not need to develop these terms on  their own. Second, the Agile instruction manual included considerations to  help program managers select a qualified Agile coach. For example, the  instruction manual encouraged program managers to collaborate with  contracting officials to identify an Agile coach who had demonstrated  successful past performance on projects implementing similar technology  and Agile methodologies.

The U.S. Customs and Border Protection’s (CBP) Biometric Entry Exit  (BEE) program’s Air Exit project provided informal training for new team  members that included a discussion of Agile methods. According to the  Air Exit project manager in the Office of Information and Technology, new  team members received onboarding training that covered CBP’s  approach to Agile methods. The project did not track attendance for this  onboarding training, but an Air Exit project manager noted that team  members were incentivized to attend the training in order to learn how to  satisfy their responsibilities. The Scrum master for the Air Exit project  stated that this training was also available to the team as a refresher  course approximately every fiscal quarter.

The BEE program also relied on an Agile coach to support the Agile  team. According to the Agile coach supporting the Air Exit project, this  role included training for the Agile team on basic Agile topics and working  with the team on their use of a project management software tool.  According to a project manager within the Office of Field Operations Air  Exit project management office, the Agile coach that supported the  project was instrumental in designing the CBP Office of Information and  Technology’s Agile development program beyond the BEE program.

Ensure Agile teams have the appropriate technical expertise needed  to perform their roles  The agency should have policy or guidance in place to help programs  ensure Agile teams have the appropriate technical expertise. A program  should also consider Agile-centric skills when forming teams. In addition,  programs should define requirements for contractor proposals and  evaluate contractor proposals for Agile services (e.g., source selection).

DHS guidance provided programs with considerations for forming teams  with Agile-centric skills. The DHS Agile instruction manual stated that a  development team with experience in Agile practices can mitigate risks to  on-time delivery. This experience included Agile processes as well as  technical skills, such as automated testing. In the context of the Agile  Scrum methodology in particular, the Agile instruction manual stated that  teams needed to be cross-functional and have all of the skills required to  deliver a project from conception to delivery.

To enable teams to deliver a project from conception to delivery, the Agile  instruction manual stated that program managers should seek team  members with general skills. The manual advised that team members  should contribute to routine development activities and possess cross- functional expertise that allows the team to achieve work without  depending on individuals outside of the team. For example, in Agile  development, testers are part of the development team and should  therefore possess both testing and development skills. In addition, the  instruction manual stated that, according to industry experts, program  managers should seek some overlap in team member’s skillsets to  mitigate risks associated with a key person becoming temporarily  unavailable.

DHS guidance further provided programs with considerations for defining  requirements in solicitations for contract proposals for Agile services. For  example, DHS supplemental guidance for incorporating testing and  evaluation into contract requirements noted that contracts should specify  government test and evaluation staff, as well as contractors, on the  development team in order to access the test data they need.

The DHS IT acquisition review process also helped to ensure that  requirements were defined in solicitations for contractor proposals.  According to the Information Technology Acquisition Review Essentials  Guide, Agile subject matter experts in the department review proposed  contracts to ensure that they will enable development activities that  adhere to Agile best practices and DHS systems engineering life cycle  (SELC) guidance. For example, Agile subject matter experts should  assess whether contract requirements documents, such as the statement  of work, are prepared in terms that will enable vendors to clearly  understand the Agile requirements.

The department also provided guidance to assist programs in evaluating  contractor proposals for Agile services. The Agile instruction manual  noted that programs can consider certifications in various Agile  methodologies and recommended that programs coordinate with  contracting officials to review vendors’ past performance in implementing  Agile methods.

In addition, the department established the Procurement Innovation Lab  within the Office of the Chief Procurement Officer to help programs  address challenges in procuring Agile services, such as validating  contractor qualifications. According to a Procurement Innovation Lab  team leader, the lab shares lessons learned from Agile services contracts  via webinars, which are available to staff on an as-needed basis. Several  of these webinars highlighted the value of using technical demonstrations  to validate the qualifications of vendors.

The ICE SEVIS program provided training for all team members,  including contractors, to ensure they had the necessary Agile-centric  skills and expertise. A team process agreement for one development  module showed that the technical lead, development team, test engineer,  and Scrum master roles were filled by contractors, while other positions  such as the project manager, product owner, and test automation subject  matter expert roles were filled by government employees. According to  the ICE SEVIS program manager and Scrum master, the program  provided training for contractors that covered Agile processes as well as  technical and project management support tools. In addition, some  government employees took role-specific training. For example, the  program’s test automation subject matter expert completed training in  continuous integration and test automation.

To further ensure contractors on ICE SEVIS Agile teams had the  necessary Agile-centric skills, the ICE SEVIS program defined the Agile  methodology and necessary technical expertise for contractors in the  contract requirements. For example, the performance work statement for  one development module required contractors to use the program’s  management software tool to track user stories. The performance work  statement also required use of the program’s continuous integration and  automated testing tools. The terms and conditions for this contract also  identified the required experience for key personnel, such as proven  experience working in an Agile environment.

The ICE SEVIS program also evaluated contractor qualifications to  ensure they had the necessary technical expertise. According to the  program manager, contractor qualifications were evaluated in two stages;  first, by assessing the contractor’s proposal, and second, by conducting a  technical challenge to ensure that contractors could demonstrate the  technical skills in the proposal. According to the instructions included in  the request for contractor proposals, this technical challenge required the  contractor to leverage Agile best practices to design, develop, and  demonstrate working software that addressed user stories provided by  the program. Although the instructions stated that contractors were  required to follow Agile methods, the ICE SEVIS program manager stated  that the primary goal of the technical challenge was to assess  development skills rather than knowledge of Agile.

Technical environments enable Agile development

Agency policy or guidance should call for technical and project tools to be  available to support Agile development and for system design that will  support iterative delivery.

Make technical and project support tools available  Project management and technical support tools should be integrated into  a program’s technical environment, where appropriate. The tools within  this technical environment should be readily available to Agile teams.

DHS policy and guidance called for Agile projects’ technical environments  to support Agile methods. The department published guidance for  standing up technical environments specifically for Agile projects. For  example, the DHS Agile instruction manual identified the benefits of using  program support tools for tracking program progress, reporting on that  progress as part of program governance, and automating tests within an  Agile technical environment. The manual stated that a program or project  manager is responsible for fostering an environment that enables the  Agile team to succeed, including obtaining the appropriate tools.

To supplement this guidance, DHS offered a suite of tools that Agile  programs could access. The suite of tools was referenced in a checklist of  activities for program or project managers in the Agile instruction manual.  According to an IT specialist from the Technical Architecture and  Engineering division within the Office of the Chief Technology Officer  (OCTO), the tools available included program management tools as well  as technical tools. The specialist stated that OCTO provided programs  with access to this suite of tools to build support for and familiarity with  the tools, evaluating any requested plug-ins from programs and doing  their best to accommodate them.

The ICE SEVIS program defined the technical environment to include  technical tools for automated testing and continuous integration. The  team process agreement for one of the program’s development modules  identified technical tools that supported continuous integration and testing  within the program’s technical environment. This included Jenkins for  continuous integration as well as MUnit and Soap UI for continuous  testing. In addition, the ICE SEVIS Modernization Test and Evaluation  Master Plan discussed that tools for helping to ensure code quality, such  as an automated code analytics tool, should be used to identify test  coverage of code and cybersecurity code vulnerabilities.

The program also defined management support tools in the process  agreement. Specifically, it identified support tools for tracking and  knowledge management, such as JIRA and Confluence. The team  process agreement stated that JIRA should be the main knowledge  management tool and that all changes, discussion, and history should be  tracked in each ticket. This process agreement also stated that JIRA  should be the team’s tracking tool with Confluence used to provide  transparency.

Design a system that supports iterative delivery  The agency should adopt policy or guidance that allows project designs to  develop modular system components and the program should establish a  loosely coupled architecture that allows for modular development.

DHS guidance allowed project designs to develop modular system  components through upfront architecture planning. The DHS Technical  Review Guide advises stakeholders to discuss and approve the technical  design of the system, including its top-level architecture, as part of the  system definition review. This review should take place prior to  development work.

For Agile programs, DHS suggested that programs may elect to switch  the system definition review with a release planning review. The SELC  Tailoring Examples for Selected Types of DHS Acquisition Programs  specified that this design discussion should take place as a part of  release planning. The department referred to this design as an  “architectural runway”, a description that should enable the team to  conceptualize how the user stories will be implemented. In exiting the  release planning review, the Technical Review Guide noted that  programs should answer whether or not an architecture exists, if the  architecture enables the deployment of the release, if architecture  collaboration is explained and understood for this development process,  and if the appropriate resources are available.

In addition to transitioning to a release planning review, DHS guidance  urged Agile programs to move away from traditional artifacts associated  with a system definition review. In this shift from traditional artifacts, the  department proposed that programs document software design within a  system design document on a release-by-release basis. According to the  Requirements Engineering Users Guide, in Agile methodologies detailed  design occurs at the iteration level and, as such, the design is  documented in an iterative fashion in the system design document. The  guide further stated that the system design document allows the  development team to communicate the design to others including  customers, managers, and other developers and that industry best  practice was to represent the design through a series of “design views.”  Each software design stakeholder could have a distinct perspective on  what are the essential aspects of a software design. Together, these  views provide a comprehensive description of the design in a concise and  usable form that simplifies information access and assimilation.

DHS guidance did not discuss the system design document as a  delivered artifact until after the sprint review and demo and a release  readiness review had been discussed. At the end of each iteration, DHS  guidance stated that the system design document should represent the  design of the feature, function, and/or system as it existed at that  moment. To facilitate communication between Agile teams and to ensure  the most up-to-date description of the design is available, guidance called  for the system design document to be developed and maintained in an  electronic form using any number of programs or web tools that are  available. The Requirements Engineering Users Guide noted that the  system design document is to be considered complete when each  identified design concern is the topic of at least one design view, all  design constraints have been applied, and sufficient detail exists to be an  authoritative and primary “code-to” artifact.

The system design document should also provide traceability to the  feature, epic, and operational requirements document “shall” statements.  The SELC Tailoring Examples for Selected Types of DHS Acquisition  Programs stated that, prior to releasing software to the production  environment, a release readiness review should be conducted. As part of  this guidance, the department stated that the intent of this release  readiness review included ensuring that all elements of the release were  complete, including a system design document.

DHS guidance also discussed designing a loosely coupled architecture,  another important aspect of project design that facilitates modular  development. A member of the contractor support staff for the DHS OCIO  stated that the Enterprise Architecture Team was expected to consider  modularity and loose coupling generally through consideration of  technical complexity. According to DHS Enterprise Architecture principles,  technical complexity is to be mitigated in part by the implementation of  loose coupling. According to the principles, DHS will incorporate loose  coupling into architecture and systems design to minimize the risk  resulting from changes within one system necessitating changes within  an interoperable system.

The BEE Air Exit project design document defined the planned design for  the system and addressed design and architectural concerns that could  affect the system’s operating environment.

As part of this design consideration, the project established a loosely  coupled architecture. This loosely coupled architecture was illustrated  within the project’s system design document. This system design  document defined the Traveler Verification Services software as  consisting of two distinct components: 1) traveler verification services  core and 2) traveler verification services matcher. The functionality and  responsibility of these two components were distinguished throughout the  document. Moreover, the document detailed how the Traveler Verification  Services software would be delivered as a system of applications,  combining an integration layer, business layer, data access layer, and  data layer.

Project planning controls are compatible with Agile development

Agency policy or guidance should call for teams to maintain a sustainable  development pace and track and monitor that pace and for non-functional  requirements and critical features to be defined and incorporated in  development.

Maintain a sustainable development pace and track and monitor that  development pace  The agency should have policy or guidance that calls for Agile projects to  establish a sustainable development pace. This guidance should be  supplemented by tracking and monitoring the pace. The program should  establish a sustainable pace for Agile projects and that pace should be  tracked and monitored.

DHS guidance called for Agile projects to manage the pace of the  software development. The Agile instruction manual stated that Agile  projects should consider velocity and burndown rates to track the overall  project status and update the project plan to reflect this status. In a  separate appendix, the Agile instruction manual also identified metrics for  project and program managers and executives to consider in order to  monitor how a project was progressing, how Agile was optimizing the use  of team members and resources, and where the project stood in terms of  key Agile measures. In the list of Agile metrics, DHS highlighted  burndown rate and velocity, and offered a description and method of  calculation for each.

In addition to the Agile instruction manual, the department provided  training that spoke to development team pace. For example, the  curriculum for lesson six of course APM 350 on managing program  execution included a section covering Agile development metrics. Among  the metrics discussed were those associated with progress, including  velocity and burndown charts. Progress metrics were also covered in  other course offerings. However, DHS guidance and training materials did  not cover the concept of ensuring a sustainable pace.

In order to track and monitor the development team’s pace, the  department incorporated several related measures into the Agile core  metrics. Among others, programs executing Agile were expected to report  on the following pace-related metrics after each iteration:  story points planned to be completed, number of production deployment per quarter, and average product deployment lead time.

These measures could provide programs and the department with an  understanding of the development team’s pace and the extent to which it  was or was not sustainable.

However, the department was not tracking and monitoring development  team pace as intended. The Agile instruction required Agile programs to  submit Agile core metrics within six months of the instruction’s  publication. However, according to the Director of STM, programs were  not consistently reporting these core metrics. According to the Director of  STM, the department was still working with programs to ensure they  consistently reported the core metrics to the Investment Evaluation,  Submission, & Tracking system.

The SeaWatch project at the United States Coast Guard (USCG)  demonstrated that it was monitoring development pace on a monthly  basis. SeaWatch officials stated that they used TAIGA as a tool to  manage the overall project and to auto-calculate pace. Additionally,  SeaWatch officials stated that contractors delivered a monthly progress  report, which contained the accomplishments of each team and a  snapshot from the latest TAIGA report. For example, one monthly report  for SeaWatch included a burndown chart for the SeaWatch project’s  development backlog and the monthly output of user stories and  associated story points by development effort that could be used to  assess development pace over time.

SeaWatch officials stated that the teams used velocity to help plan for the  next iteration. Officials added that they tracked the collective velocity of all  four teams as they were all working together on the same ship build. In  the future, officials stated that this tracking of velocity could also be used  to track individual team velocities as necessary.

The project demonstrated that it was adapting in order to achieve a  sustainable pace. According to the April 2018 monthly report, the team  completed 55 user stories worth 500 story points. The following month, in  the May 2018 monthly report, the number of user stories dropped from 55  user stories to 17, worth 130 story points. According to the June 2018  monthly report, the team completed a development effort of 31 user  stories and 278 story points.

According to the SeaWatch acquisition manager, development pace  fluctuated because not all sprints were of equal difficulty. The acquisition  manager added that the number of completed story points per sprint  could also be inconsistent due to inaccurate user story estimates,  changes in staff availability from sprint to sprint, and other external factors  such as weather.

Define and incorporate non-functional requirements in development  The agency should have policy or guidance in place for incorporating  non-functional requirements for Agile projects and the program should  account for non-functional requirements, such as security and privacy, in  the program strategy and throughout development.

DHS guidance addressed the incorporation of non-functional  requirements for Agile projects. According to the Technical Review Guide,  non-functional requirements could be governed via a system definition  review. According to the guide, this review was required at the end of the  requirements definition phase to focus on the completeness of the  requirements engineering activities, including the gathering, analysis, and  documentation of functional and non-functional requirements. This review  assessed the traceability of these requirements to the operational  requirements document and concept of operations.

In the case of Agile programs, DHS suggested replacing the system  definition review with a release planning review. In place of traditional  artifacts associated with a system definition review, DHS guidance stated  that the capabilities and constraints document, backlogs, and the system  design document, which are developed iteratively throughout the release,  should document the requirements and provide traceability to the  operational requirements document. These artifacts served the function  and filled in for the functional requirements document and the system  requirements document previously required for a system definition review.

The Technical Review Guide noted that, as the capabilities and  constraints document matures, business and architectural epics should  decompose to features or themes, and, ultimately, user stories that reflect  the specific tasks that users will perform. The Technical Review Guide  cited as exit criteria that a program or project should answer whether the  capabilities and constraints document identified the specific features and  non-functional requirements to be addressed in the release.

DHS requirements engineering guidance expanded on how Agile  programs and projects could manage non-functional requirements. The  guidance explained that there were various ways that the constraints or  non-functional requirements such as security, Section 508 accessibility,  privacy, or reliability could be translated down to the iteration level. It  stated that some Agile teams may include these non-functional  requirements in the backlog, while other teams may include them as part  of acceptance criteria or in an artifact called the “definition of done”.  According to officials from the Science and Technology Directorate Office  of Systems Engineering, once defined, the day-to-day operations and  testing for non-functional requirements were the responsibility of the  operational test agent.

DHS maintained some governance over non-functional requirements.  According to the DHS acquisition management instruction, the  operational requirements document should be approved by the  Acquisition Decision Authority after validation by the Joint Requirements  Council. The operational requirements document should include both the  functional and non-functional requirements. Officials from the Office of the  Director of Test and Evaluation said that they do not usually provide  feedback on the decomposed functional or technical requirements for  software development projects, focusing only on the operating  requirements, because that is what directly impacts operations.

The CBP BEE program’s functional requirements document outlined a  series of non-functional requirements as the requirements used to define  how the system is to behave as opposed to functional requirements that  define what the system should do. The project included 10 non-functional  requirements in the functional requirements document. For example, the  biometric match service should have an overall availability of greater than  or equal to 99%, which included both scheduled and unscheduled  downtime. These ten non-functional requirements comprised five related  to availability, three related to reliability, one related to scalability, and one  related to security. All of these non-functional requirements were  scheduled for release as part of the initial operating capability.

CBP officials noted that non-functional requirements were also captured  within the operational requirements document as measures of  effectiveness. According to project officials, measures of effectiveness  and other security-related parameters translated into the key performance  parameters for the project. Officials noted that these key performance  parameters were tracked on a daily basis and that information was fed  into a monthly report. The operational requirements document stated that  the program’s suitability requirements conformed to the DHS and CBP  enterprise architectures and all DHS and CBP infrastructure policies and  guidelines. Moreover, it noted that National Institute for Standards and  Technology guidance and DHS guidance factored into the development  of security related non-functional requirements. For example, system  security controls should be compliant with National Institute of Standards  and Technology and DHS sensitive system guidelines based on its  Federal Information Processing Standard 199 rating for availability,  integrity, and confidentiality.

Define and incorporate critical features in development  The agency should have policy or guidance in place for incorporating  critical features for Agile projects. The program should ensure that its  strategy considers all mission, architectural, and critical safety  components, along with their dependencies, on a regular basis.

DHS policy and guidance addressed the incorporation of critical features  for Agile projects. As discussed in the non-functional requirements  section, programs were expected to document functional requirements  via the systems design review or, as recommended for Agile programs, a  release planning review. Artifacts associated with these reviews served to  capture the functional requirements for the program and should be  evaluated as part of the entrance and exit criteria defined in the technical  review guide. Additional guidance elaborated on the process for  decomposing requirements.

Unlike non-functional requirements, applicable exit criteria on critical  features expanded into the solution engineering review. This criteria  included questions devoted to critical features and how they tied back to  performance measures (e.g. key performance parameters). According to  the Director of STM, headquarters oversight of critical features was  limited to the higher-level requirements defined in the operational  requirements and concept of operations documents.

The ICE SEVIS program captured critical features in documents required  by department acquisition management policy and guidance. The ICE  SEVIS Modernization Concept of Operations listed specific functional  capabilities associated with mission and mission support scenarios.

The ICE SEVIS Modernization Operational Requirements document  expanded on these functional capabilities and identified the operational  and program-level requirements. These requirements were necessary to  achieve the performance goals and mission of the Student and Exchange  Visitor Program and the Department of State, the primary sponsors for  the program. In particular, the SEVIS Modernization Operational  Requirements document identified business capabilities and key  performance parameters that measured system capabilities.

The core capabilities are long-term initiatives intended to span multiple  contracts and deliver the major components necessary for SEVIS  modernization. The SEVIS Modernization Operational Requirements  document stated that these capabilities must be present for the SEVIS  modernization to be considered a success. These business capabilities  represented the core SEVIS functions needed to close outstanding  SEVIS vulnerabilities. According to the ICE SEVIS Modernization SELC  Tailoring Plan, there were 79 sub-capabilities supporting the eight core  capabilities. The sub-capabilities generally fulfilled one or more  stakeholder needs and were delivered within a release or series of  releases. The SEVIS Modernization Operational Requirements document  confirmed that the program should prioritize and sequence the  capabilities for delivery during the release planning and delivery  processes.

The program provided a road map for one development module. This  road map listed areas for development in the order they were intended to  be developed and identified the associated business capabilities. The  business capabilities identified in the road map aligned with the sub- capabilities listed in the SEVIS Modernization Operational Requirements  document. Examples of business capabilities in the road map that were  also sub-capabilities identified in the operational requirements document  included:  create nonimmigrant record (including supporting forms), align nonimmigrant eligibility information with unique nonimmigrant, update nonimmigrant biographical information, and add/update dependent information.

Appendix V: Leading Practices for Adopting Agile Development—Team Activities and Dynamics

This appendix describes in more detail our evaluation of the three leading  practices for team activities and dynamics when adopting Agile  development. It does not present new findings; rather, the information is  intended to assist the Department of Homeland Security (DHS) in  implementing the recommendations described in this report.

For teams to successfully transition from processes using traditional  software development methods to Agile methods, leading practices for  team activities and dynamics recommend that  the composition of the team supports Agile methods by defining the role of a product owner work is prioritized to maximize value for the customer through  creating user stories to define work prioritizing requirements in a backlog based on value estimating the relative complexity of user stories  repeatable processes are in place by meeting daily to review progress and discuss impediments ensuring the quality of code being developed  Within DHS, program management offices are responsible for planning  and executing individual programs and implementing applicable Agile  methodologies. According to Office of the Chief Technology Officer  (OCTO) officials, DHS contracts for Agile services, including  development, rather than performing development in-house. As a result,  Agile teams may be predominantly contractors rather than federal  employees. In addition, DHS Office of the Chief Information Officer  (OCIO) is responsible for setting the policies and procedures to ensure  that programs and, in turn, the teams that make up those programs,  leverage Agile development best practices to meet the department’s  goals and are within acquisition policy. DHS OCIO is also responsible for  providing guidance for and reviewing the adoption and execution of Agile  development.

Agency policy or guidance should require individual, self-organizing Agile  teams for each segment or iteration and define the role and  responsibilities of the product owner.

Agile teams should be self-organizing, meaning they are empowered to  collectively control how to accomplish their work and the resulting  product. An Agile team’s authority should include lower-level decision  making and team formation and highlight the importance of team stability.  The team’s composition should be cross-functional and consist of  members who possess all the skills needed to produce working software,  including, but not limited to, contract specialists, developers, and testers.

DHS provided guidance to Agile teams on self-governance. The Agile  Development and Delivery for Information Technology instruction (Agile  instruction) and the Agile Development and Delivery for Information  Technology Instruction Manual (Agile instruction manual) both explain  that collaborative, self-organizing, and cross-functional teams help  achieve the flexibility needed for the iterative development that  characterizes Agile development methods. The Agile instruction manual  notes that most Agile methodologies assume the dedicated involvement  of all stakeholder, developer, and integration staff throughout the project.

DHS guidance also discusses team formation. The Agile instruction  manual recommends that the project team include the roles of the  program or project manager, a product owner, a development team of  approximately five to nine members, testers, and an Agile coach, and any  additional expertise as needed. According to DHS guidance, a program or  project manager is responsible for establishing the project team. The  program or project manager is supported in this by the component  acquisition executive and other component management.

At DHS, U.S. Immigration and Customs Enforcement’s (ICE) Student and  Exchange Visitor Information System (SEVIS) program had self-organizing teams that defined their own processes for completing work.  ICE Agile teams, including those supporting the SEVIS program, were  expected to document their processes in a team process agreement,  where a team had the authority to define its own operational strategy and  make decisions about the product, including when to consider the product  completed according to the program’s “definition of done.” According to  the ICE Agile principles instruction, a program chooses a baseline set of  practices that are documented in a team process agreement and are  adjusted over time.

ICE SEVIS teams were self-managing and included the roles necessary  to deliver what they committed to in a sprint. ICE’s Agile playbook  suggested minimum levels of experience, knowledge, and certifications  necessary for key personnel to support Agile methodologies. For  instance, the playbook suggests that Scrum masters be certified and  have a minimum of one year of experience. To help ensure that  contractors have the requisite skills necessary, ICE SEVIS officials stated  that vendors are required to demonstrate their ability to develop a small  software application before a contract is awarded to them.

Define the role of a product owner  A product owner should understand the business and strategic values of  the agency and its alignment with the vision of the product team and  support Agile methods. A product owner’s responsibilities include  availability to the team, authority for making programmatic decisions,  general responsibilities as a member of the team, and the need to  possess subject matter expertise related to the business needs. A  product owner is an authoritative user who manages the requirements  prioritization, communicates operational concepts, and provides continual  feedback to the team.

DHS provided guidance on the role and responsibilities of a product  owner. According to the Agile instruction manual, the product owner is  responsible for representing stakeholders. To do so, the product owner  should be available to the development team throughout the iteration to  answer questions and clarify requirements on behalf of the stakeholders.  The manual stated that the product owner is also responsible for ensuring  that the product meets user needs and delivers value. This includes, for  example, prioritizing user stories in the backlog and serving as an  acceptance authority for work completed by the team.

The department also provided elective training on the role of a product  owner. For example, the U.S. Citizenship and Immigration Services Office  of Information Technology offered an elective product owner training  course. The USCIS product owner training covered concepts such as the  importance of the product owner’s availability to the team and the product  owner’s authority for making programmatic decisions.

ICE identified a product owner for SEVIS to represent two user  communities. The program identified one product owner from ICE’s  Student and Exchange Visitor Program and a second product owner from  a stakeholder organization within the Department of State. Both product  owners were identified in the ICE SEVIS staffing plan.

According to a team process agreement for one development module, a  product owner is responsible for, among other things:

Prioritizing and deciding which user stories will be implemented in  each iteration.

Making an acceptance decision for each user story based on the  story’s acceptance criteria.

Ensuring that the intended value of the functionality is delivered.

According to program officials, product owners for the ICE SEVIS  program prioritized user stories during planning sessions. The Student  and Exchange Visitor Program Agile Overview slides stated that the  team, including the product owner, attends sprint planning to review the  prioritized product backlog and to ensure a common understanding of the  product owner’s immediate priorities.

Product owners also exercised authority to validate acceptance criteria  and subsequently close user stories. The program’s “definition of done”  stated that the product owner must test and indicate acceptance of each  user story in order for a user story to be considered complete. In a written  response, ICE SEVIS officials stated that ICE SEVIS product owners  indicated a user story had met the acceptance criteria and could be  closed by changing the user story’s status to “closed” using the team’s  program management software tool.

In addition, product owners were available to the development team to  ensure timely input. According to a team process agreement for a  development module, the product owner should work closely with the  development team to communicate the details of requirements and  answer questions about user stories. In an interview, the ICE SEVIS  product owner representing the Student and Exchange Visitor Program  stated that the role was a full-time position and did not have any  competing responsibilities. To ensure availability, the ICE SEVIS product  owner representing the Student and Exchange Visitor Program stated  that there was a designated backup who had the same authority and  responsibilities as the full-time product owner.

Work is prioritized to maximize value for the customer

Agency policy or guidance should call for Agile teams to create user  stories to define the work; prioritize requirements in a backlog based on  value, including tracking and monitoring the value of work accomplished;  and estimate the relative complexity of user stories. Individual Agile  teams within the respective programs and projects should implement  these aspects of Agile development.

Create user stories to define work  A user story is to reflect a small segment of work that can be completed in  a single iteration. The agency should have policy or guidance in place for  writing user stories for Agile projects. The product owner should  determine the value of a user story in consultation with the development  team, including the acceptance criteria and defining what “done” means.  User story value should then be re-evaluated based on requirements to  ensure the greatest return on investment.

DHS provided guidance that Agile programs and projects could leverage  when writing a user story. The Agile instruction manual, Homeland  Security Acquisition Institute Agile lessons, such as “Managing Program  Execution” and the Requirements Engineering User’s Guide provided a  basic format for how to craft a user story. These resources noted that a  user story defines where a “role” wants some “goal/desire” accomplished  to result in a “benefit”.

The Requirements Engineering User’s Guide also discusses the role of  acceptance criteria and a definition of done in user story development.  The guide highlighted that acceptance criteria defines the boundaries of a  user story and confirms when a story has been completed and is working  as intended. It specifies that acceptance criteria should be included in an  Agile program or project’s capabilities and constraints document, a DHS  artifact unique to Agile development and highlighted in the systems  engineering life cycle (SELC) tailoring example for Agile. This guide  added that the definition of done identifies all of the activities/artifacts  besides working code that must be completed for a feature or sub-epic to  be ready for deployment or release including testing, documentation,  training material development, certifications, etc.

The Agile instruction manual places much of the responsibility for defining  a user story under the purview of the product owner. The Agile instruction  manual stated that the product owner is the individual tasked with  providing requirements to the development team and is responsible for  determining the features necessary for the product release. The manual  also emphasized that the product owner is only responsible for clarifying  the user story requirements that would meet his or her needs and not  responsible for clarifying how user stories should be implemented to meet  those needs.

The ICE SEVIS program developed user stories based on business  capabilities and other requirements as determined by the product owner  and the business stakeholders. The SEVIS Modernization Operational  Requirements Document describes eight business capabilities that  represent core SEVIS functions. According to ICE SEVIS officials, these  business capabilities are addressed through user stories, so there is  traceability in the backlog from user stories to epics to business  capabilities/operating requirements. The team’s process agreement for  one development module—Information Sharing—assigned responsibility  for writing user stories to the product owner. This agreement also noted  that acceptance criteria would be required for most stories.

User stories for the program were managed through a program  management software tool. An output of the backlog from the program  management software tool for one development module—Managing  Nonimmigrant Information—contained 525 user stories. These user  stories generally followed DHS and ICE guidance for capturing what a  user needs and why. Most of these user stories also included acceptance  criteria.

The program also developed a “definition of done” for all user stories in  the team process agreement. According to the definition, a user story  was “done” when the following steps were addressed:

All code to meet the story’s needs was written according to the  system’s development standards.

Unit tests were written and run successfully.

All code was checked in and the build completed successfully.

All database changes (if required) were complete and checked in (a  functional test could be run).

The software had been deployed to the system test environment and  passed system tests.

The product owner agreed that the implementation met the  acceptance criteria written in the story as appropriate.

All documentation required to support the story was completed (test  cases, interface updates, etc.)

Prioritize requirements in a backlog based on value  Agile teams should pull work from a prioritized backlog and provide  frequent deliveries of software with immediate value to the user. The  team should determine the value of the user stories, prioritize work in a  product backlog, and provide an ongoing assessment of value expected  versus value delivered. The value of the work accomplished by Agile  projects should be tracked and monitored.

DHS guidance called for prioritizing user stories in a backlog. The  department published an example of a SELC tailoring plan for Agile  development that encouraged programs and projects to prioritize user  stories in a backlog as part of each release. To ensure that programs or  projects took these steps, the Technical Review Guide exit criteria for the  release planning review asks if programs or projects will have a process  in place for prioritizing user stories prior to the development of features for  each release.

Planning sessions were one such process that programs and projects  could use to prioritize user stories in the backlog. The DHS Agile  instruction manual stated that, during sprint planning, the product owner  meets with the development team in order to identify user stories from the  backlog that should be prioritized for the upcoming sprint and that  prioritization decisions should be made based on value to the users. In  addition, the product owner should ensure that prioritization decisions  maximize mission values. The Requirements Engineering User’s Guide  also states that requirements should be prioritized based on continuous  stakeholder input so that programs can prioritize what users need the  most.

DHS guidance also discussed how to determine the value of individual  user stories. While the Director of STM said that the product owner is  responsible for interpreting the concept of value as it applies to a user  story and the relative prioritization of the backlog, Agile Requirements and  Road Mapping Guidance for DHS includes a discussion on how a  program can sequence its road map for learning, risk, and economic  value. In this section, DHS offers models to consider to assist in user  story prioritization decisions and considerations for the product owner,  such as seeking to balance between business value and cost. The  Director added that there were venues, such as Agile “chat and chews,”  where program staff could ask questions and receive informal guidance.

DHS modified acquisition procedures to allow for an ongoing assessment  of progress, and indirectly the value of work accomplished, via the  release road map. DHS guidance stated that the release road map is  submitted to the Acquisition Review Board prior to acquisition decision  event 2B, as required by the Agile instruction. The Technical Review  Guide exit criteria for the release planning review and the release  readiness review asked if the development team was following the  release road map and making adjustments that supported the successful  completion requirements defined at acquisition decision event 2B.  Thereafter, programs submitted a road map to the Acquisition Review  Board during regular program reviews.

In addition to tracking and monitoring the value of work accomplished  against a release road map, regular Acquisition Review Board program  reviews allowed for the assessment of value expected versus value  delivered. The presentation template for Acquisition Review Board  program reviews included a slide for programs to report their progress  toward planned features. For each review, programs identified a  percentage of each capability that they planned to complete by the next  review. In addition, programs reported on the percentage of each  capability that they had completed since the last review.

The U.S. Coast Guard (USCG) SeaWatch Agile teams prioritized  requirements in a backlog based on the team’s ability to complete them  within a sprint. According to the acquisition manager for the Command,  Control, Communications, and Computers, Intelligence, Surveillance and  Reconnaissance (C4ISR) program, the SeaWatch product owner for new  development determined priorities for new requirements with  stakeholders. The product owner then defined those requirements as an  epic or as a user story. The C4ISR acquisition manager stated that the  user stories were prioritized in the backlog during sprint planning primarily  based on whether the Agile team could complete the work in the  upcoming sprint rather than on the value assigned by the stakeholders.

According to SeaWatch officials, user stories that could not be completed  during the current sprint were marked as a priority item for the next sprint.

Although the SeaWatch program assessed value to the user for some  epics, this did affect how the epic or its associated user stories were  prioritized in the backlog. The C4ISR acquisition manager stated that  SeaWatch assigned a value (e.g. extra large, large, or medium) to an epic  based on the epic’s value to the user. However, the acquisition manager  noted that user stories were not typically prioritized by the value of the  associated epic. User stories were instead prioritized based on the Agile  team’s ability to complete the work within the current sprint.

The project reported on its accomplishments via a road map. In May  2018, SeaWatch reported on progress toward milestones in its road map  during an annual briefing for the Non-Major Acquisition Oversight Council.  The program reported that it had installed SeaWatch v3.0 on 65 out of 70  in-service cutters.

Estimate the relative complexity of user stories  The agency should have policy or guidance in place for relative  estimation practices for Agile projects. Teams should use relative  estimation for sizing the effort of work required to satisfy a user story by  estimating its complexity based on work of similar size and complexity.  Relative estimation enables teams to maintain a sustainable software  development pace and predict work commitments. The team should size  user stories relative to one another, assess the complexity of the work,  refine user stories and estimates over time, and use prior estimates to  inform future estimates. The product owner and team should continually  revisit the estimates as they learn more about the business priorities and  as user stories rise in the order of priority.

DHS did not provide policy or guidance for relative estimation. Although  the Agile instruction manual identified estimating user story size as an  integral part to sprint planning, it did not describe the specific techniques  or processes for estimating the relative complexity of user stories.  Instead, the Agile instruction manual discussed how programs could  successfully apply traditional earned value management and cost  estimating principles to Agile projects. DHS guidance noted that programs  had largely moved from measuring story points to feature points to help  programs quantify incremental progress  The U.S. Customs and Border Protection’s (CBP) Biometric Entry Exit  (BEE) program defined practices and guidelines for how the program  expected to estimate user stories. For example, the Traveler Verification  Services process definition document identified a formula for calculating  story point values on the basis that one story point would equate to  approximately four working hours. Moreover, the process definition  document noted that story points must be reconciled to better reflect the  level of effort and task completion at the end of a given sprint.

However, it was not evident that the BEE program had implemented its  own guidance on the estimation of story points. Although the process  definition document outlined procedures for estimating user stories, only  two of 358 user stories in the Air Exit project backlog were estimated  using story points.

Agency policy or guidance should call for Agile teams to meet daily to  review progress and discuss impediments, and to observe end-iteration  demonstrations and end-iteration retrospectives. In addition, agency  policy or guidance should call for Agile projects to employ continuous  integration and confirm mechanisms are in place to ensure the quality of  code being developed. This includes setting expectations for automated  testing and code quality and tracking and monitoring against these  expectations. Responsibility for these aspects of Agile development  should lie with the individual Agile teams.

Meet daily to review progress and discuss impediments  The agency should have policy or guidance in place for holding the daily  stand-up and teams should hold daily meetings in order to stay on track  to meet the iteration goals for Agile projects and adjust as necessary.

DHS guidance defined the general procedure for holding a daily stand-up.  The Agile instruction manual stated that teams should conduct a daily  stand-up meeting for all team members. It can be conducted in person or  via another method of communication (particularly for remote employees)  for a brief, informal meeting every work day. According to the manual, all  team members should discuss the work each has accomplished since the  last daily stand-up, the work to be accomplished by the next daily stand- up, and highlight any impediments that are preventing the team members  from completing their work. Additionally, the manual suggests that it is  necessary to conduct the daily stand-up with strict discipline, so that the  meetings stick to their allotted brief time and are consistently productive.

The Agile instruction manual also highlighted the importance of the daily  stand-up meeting to an Agile process. It called this meeting an essential  collaboration event during which all team members were expected to  participate and discuss their work. The manual suggested that holding  these meetings allowed the team to practice discipline that would assist  them in their work and foster mentoring and partnering relationships  within the team that were reinforced through the constant communication  of meeting every single day. The manual added that this activity allowed  the team to hold its members accountable and be made aware of issues  that may be mitigated through collaboration.

The Traveler Verification Services team supporting the BEE program Air  Exit project at CBP held daily stand-up meetings. According to project  officials and supporting project artifacts, a daily stand-up meeting was  held each day at 10:00 a.m. Project officials noted that the daily stand- ups included the entire 40-person team.

The agency should have policy or guidance in place for holding  demonstrations or other interactions for acceptance of user stories in  Agile projects. Teams should hold frequent demonstrations to showcase  features that have been implemented and obtain feedback for acceptance  of user stories in Agile projects.

DHS guidance defined the general procedure for holding an end-iteration  demonstration or review. In the SELC Tailoring Plan example for Agile  development, DHS recommended a sprint review and demo as one type  of technical review at the end of each iteration. The purpose of the review  was to demonstrate the working software to end users and other  stakeholders and to obtain feedback that could result in additional items  being added to the backlog. It stated that this review should also ensure  that the software design was documented for inclusion in the system  design document, a proposed DHS Agile-specific artifact. The tailoring  example noted that this review should formally end the iteration’s work  with no further development or testing occurring on any stories. The Agile  instruction manual added that this demonstration should confirm the value  of the incremental piece of software produced.

DHS guidance also encouraged the use of demonstrations. The Agile  instruction manual states that a demonstration or review could be used to  reach a consensus on whether the work associated with a user story met  expectations or not. The manual also recommended that program and  project managers ensure that the functional software developed during  each iteration was demonstrated to the stakeholder at an iteration review  meeting.

The ICE SEVIS program held end-iteration demonstrations. The ICE  SEVIS Modernization Systems Engineering Lifecycle Tailoring Plan  stated that sprint demonstrations were tailored into the program to  replace other review activities, such as the preliminary design, critical  design, and integration readiness review. The Test and Evaluation Master  Plan for SEVIS Modernization stated that standard sprint testing results  were to be reported at sprint reviews. According to program artifacts, the  sprint demonstration was to be conducted at the completion of each  sprint, every other Wednesday from 11:00 a.m. to 12:00 p.m.

The agency should have policy or guidance in place for holding a  retrospective to adapt and continuously improve on Agile projects. Teams  should hold a retrospective at the end of each iteration to identify areas  for improvement to adapt and continuously improve Agile practices.

DHS guidance defined the general procedure for holding a retrospective.  The program or project manager and team reviewed progress after each  iteration and release. This included the use of a retrospective to discuss  what went well, what didn’t go well, and to identify actions to correct  problems. Guidance noted that the team should immediately incorporate  feedback from the retrospective into future iterations.

The DHS Agile instruction manual highlighted the importance of the  retrospective. The manual stated that the end-iteration retrospective is a  key part of ensuring that teams following Agile methodologies are able to  identify problems and adapt to continuously improve for future sprints.  Additionally, the manual stated that end-iteration retrospectives are useful  in satisfying governance needs. For example, the Agile instruction manual  stated that programs could tailor standard-format SELC artifacts (as  codified in the SELC Tailoring Plan) to instead rely on assessment and  performance data addressed in end iteration retrospectives.

The Traveler Verification Services team supporting the BEE program’s Air  Exit project held end-iteration retrospectives. According to the process  definition for this team, a retrospective was to be held between the end- iteration review and the subsequent planning session for the upcoming  sprint. The process definition defined the goal of the retrospective as  obtaining an honest review of the process with a consensus on how to  adapt it. In an interview, project officials noted that the team documented  the results of retrospectives on a release-by-release basis in a project  management software tool.

The agency should have policy or guidance that defines and emphasizes  the use of automated testing and continuous integration. This guidance  should be supplemented by defining expectations for automated testing  and tracking and monitoring against these expectations. Agile teams  should adopt practices for continuous integration and automated testing  to ensure that software handoffs are repeatable and dependable.  Automated testing should be tracked and monitored based on established  expectations.

The DHS Agile instruction manual defined continuous integration as the  practice where delivery teams frequently integrate their code into a  shared master copy. It noted that these integrations are verified by an  automated build process, which performs testing to detect any integration  errors quickly and automatically. The manual stated that continuous  integration in Agile projects should be planned and recorded on a  release-by-release basis.

The Agile instruction manual also emphasized the importance of  continuous integration and automated testing. With regard to automated  testing, the manual set an expectation for program or project managers  and stakeholders to consider both automated testing tools and  infrastructure support for the Agile software build and test processes as  part of general project planning efforts. Moreover, the manual identified  continuous integration, automated acceptance testing, and automated  unit testing as key practices program or project managers can use for  continuously monitoring and reporting project health. These practices  could also help to identify opportunities for improving project team  performance.

DHS officials acknowledged that current DHS programs implemented  testing and evaluation inconsistently and that the department’s existing  guidance and policies did not effectively support modern best practices in  automated testing and continuous integration. To address these gaps,  DHS had an Agile action plan that set an expectation for updating DHS  acquisition guidance, policy, and practices for testing and evaluation to  enable modern best practices in automated testing and continuous  integration.

In lieu of more explicit guidance, DHS incorporated training as part of a  curriculum geared toward test and evaluation managers that discussed  both continuous integration and automated testing. According to the  Deputy Director of Policy and Workforce Development in the Test and  Evaluation Division of the Science and Technology Directorate, an  alternative course containing content addressing Agile and continuous  integration and automated testing was recently merged with a required  test and evaluation course, creating a new course. According to the  Deputy Director, the new course was piloted during fiscal year 2019 and  will be standard in fiscal year 2020 as the required course for level II test  and evaluation certification.

In order to track and monitor automated testing, the department  incorporated several measures into the Agile core metrics. Programs  executing Agile were expected to report on the following testing-related  metrics after each iteration:

Percentage of regression testing coverage.

DHS had not established expectations for these Agile core metrics. The  Agile core metrics included a target. For example, the department suggested a program strive for seventy percent of tests to be automated.  However, the instructions accompanying the Agile core metrics stated  that all targets were notional and not expected to be reached. According  to the Director of STM, the initial core metrics were intended to assess  the level of DHS team achievement without imposing artificial industry- based target measures for each. The Director stated that, on receiving  the metrics for a period of time, the department would then adjust the  core metrics and begin to include target measures based on the results  achieved. According to the Director, this effort was underway and an  updated set of core metrics would be distributed in early fiscal year 2020.  Moreover, the department was not tracking and monitoring automated  testing as intended.

The CBP BEE program Air Exit project stood up a technical environment  that allowed for continuous integration. This technical environment was  outlined within the process definition of the Traveler Verification Services  team that was developing software. The Traveler Verification Services  process definition identified three operating environments: the  development, test, and production environments. All development  activities during the sprint were conducted within the development  environment. Similarly, all testing activities in preparation for the release  were conducted in the test environment. The final approved software  would then be deployed to the production environment.

CBP officials noted that the BEE program primarily used Jenkins to  integrate code for both continuous builds and deployment. The Air Exit  systems design document also mentioned the role of Jenkins in  continuous integration and continuous deployment for the project.

The Traveler Verification Services team incorporated JaCoCo and  FindBugs automated tests as part of the continuous delivery process and  they were run automatically when the code was checked in. Moreover,  the project’s system design document noted that the Traveler Verification  Services team integrated JaCoCo with the Eclipse Integrated  Development Environment as a code coverage inspection tool for unit  testing. Officials also noted that Selenium was used for automating the  testing within the technical environment.

Ensure the quality of the code being developed  The agency should have policy or guidance for an Agile project on  ensuring the quality of code being developed. This guidance should be  supplemented by defining expectations for code quality and tracking and  monitoring against these expectations. Agile teams should adopt  practices for code quality, such as having a test-driven development, pair  programming, and manual code reviews to supplement automated  testing. Agile teams should incorporate refactoring into code quality  practices and understand the importance of setting aside time for  refactoring.

DHS guidance recognizes the importance of ensuring code quality as part  of the development and testing process. The SELC Guidebook set an  expectation that code review and testing should be part of the software  development environment. The guide recommended setting up servers  where developers could test code and check whether the developed  application runs successfully with that code. The guide suggested another  level of tests on application reliability to help ensure that the application  did not fail on the production server. The guide stated that the program  manager should ensure that the team takes corrective action for any  hardware and software deficiencies.

In order to find deficiencies early, DHS guidance identified coding and  testing practices that could help development teams. The Agile instruction  manual cited pair programming as one practice where two programmers  work simultaneously on a single task: one programmer observes and  reviews each line of code as it is written. DHS guidance also identified  test-driven development as a practice that could motivate developers to  write effective code. The Supplemental Guidance for Test and Evaluation  stated that this approach consists of writing test cases that define a  desired improvement, then writing the code to meet the desired  functionality, ensuring that the test passes, and refactoring the code as  necessary.

Refactoring, or re-coding, without changing the way the application  functions, is an Agile practice that DHS guidance recommends for  correcting deficiencies in the code. The Agile instruction manual stated  that refactoring aims to improve code readability and reduce the  complexity of previously delivered increments of software. It noted that refactoring is important because development teams are focused on  adding the desired functionality with each release and may proceed with  making improvements to the code. Refactoring was cited as one way to  address this accumulation of needed improvements to the code, which  are known as technical debt.

The Agile instruction manual further emphasizes the importance of setting  aside time for refactoring to address risks associated with technical debt.  The manual states that refactoring a previously developed increment of  software to improve code quality may force a change in the release  schedule. However, if the team does not make these revisions in a timely  manner, the effort required to correct them later tends to increase. The  manual states that this increasing technical debt is a risk factor to be  addressed as soon as feasible. If the technical debt is allowed to  accumulate unchecked, or if the project team loses track of the scope of  its technical debt, the project could suffer from schedule and performance  problems.

In order to track and monitor the quality of code being developed, the  department incorporated several code quality and testing measures into  the Agile core metrics. Among others, programs executing Agile were  expected to report on the following quality-related metrics after each  iteration:

Number of critical or major defects fixed.

Number of critical or major defects in the backlog.

Number of technical debt issues completed.

Number of technical debt issues in the backlog.

However, the department was not tracking and monitoring code quality as  intended. These measures could provide programs and the department  with an understanding of the development team’s ability to address  defects and technical debt. In addition to these metrics, programs are  also expected to report quarterly on the number of outages requiring a  rollback or patch after production deployment.

The ICE SEVIS program used manual testing to ensure code quality. The  definition of done for the program stated that new code should be peer  reviewed to identify risk to the existing code, assess compliance with  coding best practices, and evaluate refactoring. According to ICE SEVIS  officials, an independent specialist provides internal code reviews and  offers feedback on areas for improvement.

The ICE SEVIS program also employed automated testing to ensure  code quality. The definition of done required that unit tests cover a  minimum of 85 percent of code. Program officials stated that  vulnerabilities and bugs identified through this process were added to the  backlog and classified as technical debt.

The program refactored code to address technical debt, but did not set  aside time for refactoring each sprint. According to ICE SEVIS officials,  the development team refactored code as necessary to improve overall  quality but did not set aside time for refactoring unless they were  addressing a consistent issue. ICE SEVIS officials stated that the  development team could propose refactoring code during sprint planning  if there was a specific technical debt they had identified. However,  according to the Scrum master for the program, addressing technical debt  was additional work for the team to take on beyond the user stories they  planned to complete and this additional work incentivized the  development team to prevent the accumulation of technical debt.

Although DHS allowed Agile programs to tailor the core metrics, ICE  SEVIS submitted some of the code quality-related Agile metrics to the  department. The program included Agile metrics in June 2018  presentation slides for the Acquisition Review Board. For this initial  reporting period, the program reported no critical or major defects in the  backlog and no technical debt issues in the backlog. It also provided a  screenshot of the Agile core metrics reported to DHS via the Investment,  Evaluation, Submission, & Tracking system in February 2019. This  reporting period covered two iterations. The program reported that it fixed  four critical or major defects during the first iteration and did not have any  critical or major defects in the backlog for either iteration. The program  also reported that it completed eight technical debt issues in the first  iteration, out of 14 technical debt issues in the backlog. The program did  not report on the number of outages after deployment as part of the  Acquisition Review Board program review or as part of the metrics  submitted via the Investment Evaluation, Submission, and Tracking  system.

Appendix VI: Comments from the Department of Homeland Security

Appendix VII: GAO Contact and Staff Acknowledgments

In addition to the contact named above, the following staff made key  contributions to this report: Michael Holland (assistant director), Mathew  Bader (analyst in charge), Lamis Alabed, Jennifer Beddor, Christina  Bixby, Hannah Brookhart, Chris Businsky, Alan Daigle, Aryn Ehlow,  Nancy Glover, Gina Hoover, Anna Irvine, Hoyt Lacy, Jennifer Leotta,  Alexis Olson, Zsaroq Powe, Martin Skorczynski, Natalie Smith, and  Daniel Spence.

Many of the Department of Homeland Security’s IT acquisitions have taken longer than planned or failed to deliver desired results.

In April 2016, DHS started transitioning to Agile software development to help improve its IT acquisitions. Agile focuses on collaborative processes and workflows to quickly and frequently deliver working software.

DHS has made significant progress implementing leading practices during this transition but needs to take additional steps. For example, it needs to ensure all staff are trained in this new approach.

We recommended that DHS fully implement leading practices in its transition to Agile software development.