Technology Assessment Design Handbook

In January 2019, at the direction of Congress, GAO formed the Science, Technology Assessment, and Analytics team to expand its work on cutting-edge science and technology issues, and to provide oversight, insight, and foresight for science and technology. TAs can be used to strengthen decision-making, enhance knowledge and awareness, and provide early insights into the potential impacts of technology. TA design can enhance TA quality, credibility, and usefulness; ensure its independence; and ensure effective use of resources.

Under the Comptroller General authority, we developed this handbook using the format of the 2012 GAO methodology transfer paper, Designing Evaluations . Below is a summary of the approach we used to affirm and document TA design steps and considerations for this handbook.

Reviewed select GAO documents, including Designing Evaluations (GAO-12-208G), published GAO TAs, select GAO products that presented policy options, and other GAO reports

Reviewed select Office of Technology Assessment reports

Reviewed select Congressional Research Service reports

Reviewed select literature on TAs and related to development and analysis of policy options

Held an expert forum to gather experts’ input on TA design

Considered experiences of GAO teams that have successfully assessed and incorporated policy options into GAO products, as well as GAO teams that are currently incorporating policy options into their TA design

Collected input from GAO staff who provided key contributions to GAO TAs, regarding challenges to TA design and implementation and possible solutions

The Technology Assessment Design Handbook identifies tools and approaches GAO staff and others can consider in the design of robust and rigorous technology assessments (TAs). The handbook underscores the importance of TA design (Chapter 1), outlines the process of designing TAs (Chapter 2), and describes approaches for mitigating selected TA design and implementation challenges (Chapter 3). While the primary audience of this handbook is GAO staff, we expect that other organizations engaged or interested in TAs will find portions of this handbook useful. We anticipate modifying and refining this handbook, as needed, based on experience and public comments received. We will accept comments on this handbook at TAHandbook@gao.gov for approximately 1 year after publication.

The handbook identifies three general design phases, as appropriate, as shown in the figure below. T terative nature of TA design, the requester’s interests, resources, independence, stakeholder engagement, potential challenges, and communication. In addition, ormulating initial policy options to consider; gathering evidence, determining relevant dimensions to analyze, and analyzing the policy options; and presenting the results of the policy analysis.

Summary of Key Phases of Technology Assessment Design

We found that GAO TAs have and can use a variety of design approaches and methods. The handbook provides TA design and methodology examples, including related to objectives commonly found in GAO TAs, such as: describe a technology, assess opportunities and challenges of a technology, and assess policy considerations. One example provided is: some GAO TAs include an objective related to describing the status and feasibility of a technology, which GAO teams have done by using methodologies such as expert panels, interviews, literature and document reviews, site visits, and determining the Technology Readiness Level.

Also included in the handbook are examples of TA design and implementation challenges we found, along with possible mitigation strategies. We identified four general categories of challenges, including: (1) ensuring TA products are useful for Congress and others; (2) determining policy goals and measuring impact; (3) researching and communicating complicated issues; and (4) engaging all relevant stakeholders. An example of a potential mitigation strategy to the specific challenge of writing simply and clearly about technical subjects includes: allowing sufficient amount of time for writing, including reviewing and revising writing.

GAO provides Congress, federal agencies, and the public with objective,  reliable information to help the government save money and work more  efficiently. Science and technology (S&T) issues figure prominently in  problems that Congress confronts, and one component of the assistance  GAO provides to Congress is the production of technology assessments  (TAs). This TA Design Handbook provides GAO staff and others with  tools to consider for supporting robust and rigorous assessments. This  handbook is particularly important given the need for GAO to provide  insight and foresight on the effects of technologies and corresponding  policy implications related to a wide range of S&T issues. While other  organizations—including, previously, the Office of Technology  Assessment (OTA) and a number of TA organizations elsewhere, such as  in Europe—conduct TAs, each has different relationships with its  stakeholders and government bodies. While their TA approaches and  considerations may vary, some may still find portions of this handbook  useful. We are seeking comments on this draft of the handbook.

This handbook elaborates on GAO’s approach to TA design and outlines  the importance of TA design (Chapter 1), describes the process of  developing TA design (Chapter 2), and provides approaches to select TA  design and implementation challenges (Chapter 3). The handbook  generally follows the format of the 2012 GAO methodology transfer  paper, Designing Evaluations. Given that GAO is likely to learn from its  current expansion of TA work, GAO will review and update this draft  handbook as needed, based on experience gained through ongoing TA  activities and external feedback.

GAO has defined TA as the thorough and balanced analysis of significant  primary, secondary, indirect, and delayed interactions of a technological  innovation with society, the environment, and the economy and the  present and foreseen consequences and impacts of those interactions.  The effects of those interactions can have implications. Recognizing this,  GAO has in some of its products included policy options, which  policymakers could consider in the context of a given technology and  policy goal. In this context, policy goals serve to guide the development of  policy options by stating the overall aim of the policy options, and helping  to identify the landscape and scope of policy options. Policy options can  be defined as a set of alternatives or menu of options (including the status  quo) that policymakers, such as legislative bodies, government agencies,  and other groups, could consider taking. GAO is exploring approaches to  making policy options a more standard feature or component of TAs. In  this handbook, we include considerations related to the development of  policy options that TA teams may wish to consider at each phase of TA  design.

In the United States, the Technology Assessment Act of 1972 established  OTA, which was an analytical support agency of the Congress, but was  defunded in 1995. In 2002, Congress asked GAO to begin conducting  TAs, and in 2008, a permanent TA function was established at GAO. In  2019, the Science, Technology Assessment, and Analytics (STAA) team  was created at GAO. STAA has taken a number of steps to account for  the unique requirements of TAs and related S&T work to meet the needs  of Congress.

GAO TAs share some common design principles with GAO’s general  audit engagement process, which is centered around intentional and  purpose-driven design. While general design principles are shared  across GAO’s product lines, TAs are distinct from other GAO product  lines, such as performance audits, financial audits, and other routine non- audit products. The specialized content of TAs, their scope, and their  purpose, warrant some different considerations. Table 1 highlights some  similarities and differences between TAs and other GAO product lines,  including where TAs follow aspects of GAO’s general audit engagement  process, and where TAs may further emphasize certain steps or require  additional steps during the engagement process. Not all steps have been  included in Table 1.

We expect to continue to regularly seek input and advice from external  experts related to the TA Design Handbook initiative, as well as  throughout the conduct of GAO TAs. While the primary audience of this  handbook is GAO staff, we expect that other organizations engaged or  interested in TAs will find portions of this handbook useful. For example,  these organizations could use the handbook to gain insight into GAO’s TA  design approaches, as well as use aspects of GAO’s TA design  approaches that they deem helpful. We will accept comments on this  handbook at TAHandbook@gao.gov for approximately 1 year after  publication. The handbook seeks to affirm and document GAO’s  approach, and we expect to modify and refine this handbook, as needed,  based both on comments received and further experience in conducting  TAs that include policy options. We anticipate that the final handbook will  contain additional information and details related to TA design, such as  elaborating on specific methodologies that could be applied within this  general design framework, including those designed to identify policy  options.

Below is a summary of the approach we used to identify and document  TA design steps and considerations for this handbook. For more  information, please refer to Appendix I: Objectives, Scope, and  Methodology.

Reviewed select GAO documents, including Designing Evaluations  (GAO-12-208G), published GAO TAs, select GAO products utilizing  policy analysis approaches to present policy options, and other GAO  reports

Reviewed select Office of Technology Assessment reports

Reviewed select Congressional Research Service reports

Reviewed select literature regarding TAs and related to development  and analysis of policy options

Held an expert forum to gather experts’ input regarding TA design

Considered experiences of GAO teams that have successfully  assessed and incorporated policy options into GAO products, as well  as GAO teams that are incorporating policy options into their TA  design

Collected input from GAO staff who provided key contributions to  GAO TAs, regarding challenges to TA design and implementation and  possible solutions  We conducted our work to develop this handbook from April 2019 to  December 2019 in accordance with all sections of GAO’s Quality  Assurance Framework that are relevant to our objectives. The Framework  requires that we plan and perform the engagement to obtain sufficient  and appropriate evidence to meet our stated objectives and to discuss  any limitations in our work. We believe that the information and data  obtained, and the analysis conducted, provide a reasonable basis for any  findings and conclusions in this product.

This chapter underscores the importance of technology assessment (TA)  design, outlining reasons for performing TAs and for spending time on the  design of TAs. The information presented in this chapter is based on  review of results of a literature search, an expert forum, select GAO  reports, and experiences of GAO teams and technical specialists. For  more information, please refer to Appendix I: Objectives, Scope, and  Methodology.

1.1 Reasons to Conduct and Uses of a Technology Assessment

TAs are significant given their increasing importance to policymakers, and  the growing effects of S&T on society, economy, and other areas. While  technological changes can be positive, they can also be disruptive.  Therefore, it is critical for Congress to be able to understand and evaluate  these changes, to ensure, for example, national security and global  competitiveness. Examples of potential uses of TAs related to enhancing  knowledge and awareness to assist decision-making include:

Highlight potential short, medium, and long-term impacts of a

Elaborate on and communicate the risks and benefits associated with  a technology, including early insights into the potential impacts of  technology

Highlight the status, viability, and relative maturity of a technology

Plan and evaluate federal investments in S&T  GAO TAs are most commonly requested by congressional committees,  which may use them to, among other things, make decisions regarding  allocating or reallocating resources to address research gaps, support  updated rulemaking for a regulatory agency, or inform a legislative  agenda or the development of a national strategy.

Technologies present opportunities and challenges that may vary,  depending in part on the policy context in which they are evaluated.  Therefore, part of a TA is considering the policy context surrounding a  given technology. GAO may, where appropriate, identify and analyze  policy options as part of its TAs, which may also include: clarifying and  summarizing policy-related issues and challenges, and providing  information that can be used for decision-making. In this situation, policy  options can be defined as a set of alternatives or menu of options  (including the status quo) that policymakers, such as legislative bodies,  government agencies, and other groups, could consider taking. Policy  options can be used to articulate a range of possible actions a  policymaker could consider in the context of a given technology and  policy goal. Policy options do not state what policymakers should do in a  given circumstance with a certain technology. Policy options do not  endorse or recommend a particular course of action; they are not  recommendations or matters for congressional consideration, which GAO  makes in its audits. In addition, policy options are addressed to  policymakers more broadly, and are not addressed to a specific federal  agency or entity.

1.2 Importance of Spending Time on Design

Developing a written TA design helps TA teams agree on and  communicate a clear plan of action to the project team and the team’s  advisers, requesters, and other stakeholders. Written TA designs also  help guide and coordinate the project team’s activities and facilitate  documentation of decisions and procedures in the final report. In addition,  focusing the TA on answering specific researchable questions can assist  teams to define and select the appropriate scope, approach, and type of  product, ensuring usefulness of the product to the intended users. More  specific reasons for spending time on systematically designing a TA  include:

Enhance its quality, credibility, and usefulness

Ensure effective use of resources, including time  Data collection and quality assurance of data can be costly and time- consuming. A thorough consideration of design options can ensure that  collection and analysis of the data are relevant, sufficient, and appropriate  to answer the researchable question(s), and helps to mitigate the risk of  collecting unnecessary evidence and incurring additional costs.

This chapter highlights design phases, cross-cutting considerations, and  GAO TA design examples for sound technology assessment (TA) design.  To ensure that the information and analyses in TAs meet policymakers’  needs, it is particularly useful to outline the phases and considerations  involved in sound TA design, while remaining aware of the iterative and  nonlinear process of designing a TA. The information presented in this  chapter is based on review of results of a literature search, an expert  forum, select GAO reports, and experiences of GAO teams and technical  specialists. For more information, please refer to Appendix I: Objectives,  Scope, and Methodology.

Below are questions to consider for a sound TA design. Reflecting on  these questions may help teams make important decisions (like selecting  an appropriate design) and ensure quality TAs.

Does the design address the needs of the congressional requester?

Will the design yield a quality, independent, balanced, thorough, and  objective product?

Will the design likely yield information that will be useful to  stakeholders?

Will the design likely yield valid conclusions on the basis of sufficient  and credible evidence?

Will the design yield results in the desired time frame?

Will the design likely yield results within the constraints of the  resources available?

How will policy options be identified and assessed, if applicable?

2.2 Phases and Considerations for Technology Assessment Design

Figure 1 outlines three phases and seven considerations for TA design.  While Figure 1 presents TA design as a series of phases, actual  execution is highly iterative and nonlinear. Teams may need to be  prepared to re-visit design decisions as information is gathered or  circumstances change.

Below are some considerations for the team  to think about while designing a TA and  throughout the process of performing the  TA. This list is not exhaustive, and some of  the considerations may not be unique to  TAs. of the technology) and context of the technology (such as social,  political, legal, and economic factors) circumstances change and new information  comes to light, it may be necessary to  revisit scope and design.

The initial situational analysis may also be used to:  Inform the goal(s), purpose, and objectives (also known as  researchable questions) challenges to design and implementation of  the TA, such as: (1) possible changes in  operating environment; (2) characterizing or  quantifying anticipatory factors, uncertainty,  and future condition(s); and (3) lack of or  limitations with data. See Chapter 3 for  more specific examples.

Communication strategy: Consider potential  users of the product(s) and how information  regarding the TA will be communicated.  How results are communicated can affect  how they are used, so it is important for TA  teams to discuss communication options. statement. TA teams will need to think about whether the initial policy  options are appropriate to the size and scope of the TA, as well as  whether they are in line with the policy goal and the overall TA purpose  and objectives. In keeping with the iterative nature of TA design and  execution, any initial policy option list will be revisited, modified, or  refined, as needed, as the work progresses and more information is  gained. TA teams may also need to plan to include policy analysis and  exploration of the ramifications of each policy option during subsequent  design and implementation phases.

During this phase, TA teams continue to build on the situational analysis  work and gather more background information. In addition, TA teams:

Confirm and validate the scope from phase 1

Reach agreement with stakeholders on the initial design

May perform an “environmental scan” to further highlight limitations,  assumptions, divergent points of view, potential bias, and other  factors that may help the team select a design  Other specific activities that take place during this phase include:   Identify and select appropriate design, methodologies, and analytical  approaches (refer to the next section of this chapter for example TA  design approaches and App. III for examples of TA methods)

Examples of data collection and analytical  techniques used in GAO TAs to date include:  interviews, literature review, expert forums,  site visits, technology readiness assessments,  surveys, conceptual models, small group  discussion, content analysis such as Delphi,  among others. OTA reported using similar  methodologies for its TAs (OTA, Policy  Analysis at OTA: A Staff Assessment, 1983).

Identify and select appropriate data sources, or the need to gather  data  Identify, select, and possibly develop appropriate dimensions of  analysis, if applicable

Clarify the possible initial policy options that will be considered and  describe how they may be analyzed, if applicable  Identify and consult with external experts to inform design and  implementation, and assist with external review, as appropriate  If policy options are being considered, it is important to determine the  relevant dimensions along which to analyze the options. The dimensions  will be highly context- specific, vary from TA to TA, and depend on the  scope and policy goal statement of the TA.

During this phase, the design and project plan are being implemented,  potentially while aspects of phase 2 are still underway. It is important to  consider changes in the operating context—such as changes in the  operating environment, understanding of the issues, and access to  information—and review and make changes to the design and project  plan accordingly.

We reviewed select GAO products that used  policy analysis to present policy options. We  found that these products used a variety of  data collection and analytical approaches,  such as: interviews, literature review, survey,  expert forum, site visits, case studies, analysis  of secondary data, content analysis, among  others.

If an initial policy options list was developed earlier in design, it may be  necessary to revisit the list as work progresses. During this phase, TA  teams may gather additional information regarding the policy options,  further analyze policy options, and present the results of the analysis.  Policy options are to be presented in a balanced way, including  presentation of opportunities and considerations, and not resulting in a  single overall ranking of policy options.

2.2.1 GAO Technology Assessment Design Examples

We found that GAO TAs used a variety of design approaches and  methodologies to answer various categories of design objectives  (researchable questions). GAO TAs generally include one or more of the  following categories of design objectives, which are not mutually  exclusive: (1) describe status of and challenges to development of a  technology; (2) assess opportunities and challenges arising from the use  of a technology; and (3) identify and assess cost-effectiveness, other  policy considerations, or options related to the use of a technology.  Provided below are example questions, design approaches, and GAO  TAs, for each of these categories of objectives. GAO TA examples were  used given our familiarity with GAO products, though numerous non-GAO  TA design examples exist. This is not intended to be a comprehensive list  of design examples. For more examples of methodologies, please refer to  App. III.

Describing the status and challenges to the development of a  technology. Table 2 provides example questions, design approaches,  and GAO TAs, for design objectives related to describing the status and  challenges to the development of a technology. Questions may address,  for example, what the current state of the technology is, and may involve  identifying and describing the status of the technology, which GAO TAs  have done using a variety of methods.

Assessing opportunities and challenges that may result from the  use of a technology. Table 3 provides example questions, design  approaches, and GAO TAs, for design objectives related to assessing  opportunities and challenges that may result from the use of a  technology. Questions may address, for example, what are the expected  or realized benefits of the technology, and may involve gathering and  assessing evidence on the results from using the technology, which GAO  TAs have done using a variety of methods.

Assessing cost-effectiveness, policy considerations, or policy  options related to the use of a technology. Table 4 provides example  questions, design approaches, and GAO TAs, for design objectives  related to assessing cost-effectiveness, policy considerations, or policy  options related to the use of a technology. Questions may address, for  example, what are the economic trade-offs of a technology, and may  involve gathering and analyzing evidence related to cost, which GAO TAs  have done using a variety of methods.

This chapter describes select challenges regarding technology  assessment (TA) design and implementation, as well as possible  strategies to mitigate those challenges. The information in this chapter is  based on review of results of a literature search, an expert forum, select  GAO reports, and experiences of GAO teams and technical specialists.  The tables provided below are not intended to be a comprehensive list of  challenges or strategies. For more information, please refer to Appendix I:  Objectives, Scope, and Methodology.

3.1 Ensuring Technology Assessment Products are Useful for Congress and Others

To be useful, TA assessment products must be readable and timely,  among other things, which may present a challenge for numerous  reasons. Table 5 provides examples of potential mitigation strategies to  address these challenges.

3.2 Determining Policy Goals and Measuring Impact

Another challenge in TA design arises from determining policy goals and  policy options, and estimating their potential impacts. Many of the effects  of policy decisions may be distant, and policy outcomes may be uncertain  at the time of the TA. Table 6 provides examples of potential mitigation  strategies to address these challenges.

3.3 Researching and Communicating Complicated Issues

TAs are complex and interdisciplinary, and emerging technologies are  inherently difficult to assess. Table 7 provides examples of potential  mitigation strategies to address these challenges.

An additional challenge in conducting TAs is engaging all relevant internal  and external stakeholders, ensuring none are overlooked. Table 8  provides examples of potential mitigation strategies to address this  challenge.

Appendix I: Objectives, Scope, and Methodology

This handbook identifies key steps and considerations in designing  technology assessments (TAs). Below is a summary of methodologies  used for all chapters of the handbook.

Designing Evaluations (GAO-12-208G)  select GAO products utilizing policy analysis approaches to identify  and assess policy options  We reviewed and analyzed 14 GAO TAs, including their designs and  considerations, using a data collection instrument that contained fields  regarding each report’s purpose, methodologies, and key considerations  for each methodology used (such as strengths and weaknesses). The  data collection instrument also contained fields regarding whether policy  considerations were presented or if specific policy options were identified  and assessed in each TA report, what methodologies were used to  identify and assess policy options, and key considerations associated  with the methodologies used.

We also reviewed GAO reports from non-TA product lines that utilized  policy analysis approaches to assess policy options. An initial pool of 56  GAO reports was generated based on a keyword search of GAO’s reports  database. Of the 56 GAO reports, 12 were selected for review based on  the following criteria: (1) the reports were publicly released after January  1, 2013 and (2) the reports included identification and assessment of  policy options (not solely a presentation of agency actions related to  policy options or general policy considerations). Testimonies and  correspondence were excluded. We analyzed each of these selected  GAO reports according to a data collection instrument that contained the  following fields regarding policy options in the report: purpose,  methodologies, and key considerations for each methodology used (such  as strengths and weaknesses). A list of GAO documents reviewed is  provided below.

GAO Documents Reviewed for Preparing this Handbook

Retirement Security: Some Parental and Spousal Caregivers Face  Financial Risks. GAO-19-382. Washington, D.C.: May 1, 2019.

GAO Science Technology Assessment, and Analytics Team: Initial Plan  and Considerations Moving Forward. Washington, D.C.: April 10, 2019.

Retirement Savings: Additional Data and Analysis Could Provide Insight  into Early Withdrawals. GAO-19-179. Washington, D.C.: March 28, 2019.

Critical Infrastructure Protection: Protecting the Electric Grid from  Geomagnetic Disturbances. GAO-19-98. Washington, D.C.: December  19, 2018.

Postal Retiree Health Benefits: Unsustainable Finances Need to Be  Addressed. GAO-18-602. Washington, D.C.: August 31, 2018.

Data Collection Seminar Participant Manual. Washington, D.C.: March  2018.

Artificial Intelligence: Emerging Opportunities, Challenges and  Implications. GAO-18-142SP. Washington, D.C.: March 28, 2018.

Chemical Innovation: Technologies to Make Processes and Products  More Sustainable. GAO-18-307. Washington, D.C.: February 8, 2018.

Federal Regulations: Key Considerations for Agency Design and  Enforcement Decisions. GAO-18-22. Washington, D.C.: October 19,  2017.

Medical Devices: Capabilities and Challenges of Technologies to Enable  Rapid Diagnoses of Infectious Diseases. GAO-17-347. Washington, D.C.:  August 14, 2017.

U.S. Postal Service: Key Considerations for Potential Changes to USPS’s  Monopolies. GAO-17-543. Washington, D.C.: June 22, 2017.

Internet of Things: Status and Implications of an Increasingly Connected  World. GAO-17-75. Washington, D.C.: May 15, 2017.

Flood Insurance: Comprehensive Reform Could Improve Solvency and  Enhance Resilience. GAO-17-425. Washington, D.C.: April 27, 2017.

Flood Insurance: Review of FEMA Study and Report on Community- Based Options. GAO-16-766. Washington, D.C.: August 24, 2016.

Medicaid: Key Policy and Data Considerations for Designing a Per Capita  Cap on Federal Funding. GAO-16-726. Washington, D.C.: August 10,  2016.

Municipal Freshwater Scarcity: Using Technology to Improve Distribution  System Efficiency and Tap Nontraditional Water Sources. GAO-16-474.  Washington, D.C.: April 29, 2016.

GAO Memorandum: Quality Assurance Framework Requirements for  Technology Assessments. Washington, D.C.: April 6, 2016.

Biosurveillance: Ongoing Challenges and Future Considerations for DHS  Biosurveillance Efforts. GAO-16-413T. Washington, D.C.: February 11,  2016.

Social Security’s Future: Answers to Key Questions. GAO-16-75SP.  Washington, D.C.: October 2015.

Water in the Energy Sector: Reducing Freshwater Use in Hydraulic  Fracturing and Thermoelectric Power Plant Cooling. GAO-15-545.  Washington, D.C.: August 7, 2015.

Nuclear Reactors: Status and Challenges in Development and  Deployment of New Commercial Concepts. GAO-15-652. Washington,  D.C.: July 28, 2015.

Veterans’ Disability Benefits: Improvements Needed to Better Ensure VA  Unemployability Decisions Are Well Supported. GAO-15-735T.  Washington, D.C.: July 15, 2015.

Debt Limit: Market Response to Recent Impasses Underscores Need to  Consider Alternative Approaches. GAO-15-476. Washington, D.C.: July 9,  2015.

Temporary Assistance for Needy Families: Potential Options to Improve  Performance and Oversight. GAO-13-431. Washington, D.C.: May 15,  2013.

Private Pensions: Timely Action Needed to Address Impending  Multiemployer Plan Insolvencies. GAO-13-240. Washington, D.C.: March  28, 2013.

Designing Evaluations: 2012 Revision. GAO-12-208G. Washington, D.C.:  January 2012.

Neutron Detectors: Alternatives to Using Helium-3. GAO-11-753.  Washington, D.C.: September 3, 2011.

Climate Engineering: Technical Status, Future Directions, and Potential  Responses. GAO-11-71. Washington, D.C.: July 28, 2011.

Technology Assessment: Explosives Detection Technologies to Protect  Passenger Rail. GAO-10-898. Washington, D.C.: July 28, 2010.

Technology Assessment: Protecting Structures and Improving  Communications during Wildland Fires. GAO-05-380. Washington, D.C.:  April 26, 2005.

Technology Assessment: Cybersecurity for Critical Infrastructure  Protection. GAO-04-321. Washington, D.C.: May 28, 2004.

Technology Assessment: Using Biometrics for Border Security.  GAO-03-174. Washington, D.C: November 15, 2002.

Review of Experiences of GAO Teams and Technical Specialists

We spoke with and gathered input from GAO teams that are in the  process of or have successfully assessed and incorporated policy options  into GAO products. In addition, to augment our understanding of TA  design and implementation challenges, we collected input from GAO staff  who had provided key contributions to GAO TAs. Specifically, we asked  for their thoughts regarding: (1) the strengths and limitations of TA  methodologies and (2) challenges they faced, and strategies to address  those challenges.

Review of Select Office of Technology Assessment Reports

A GAO librarian performed a search for relevant Office of Technology  Assessment (OTA) reports, using keyword searches. From this initial list  of OTA reports, we selected 17 reports to review that were frameworks,  guides, models, or other compilations. We also reviewed the  methodologies of the OTA reports selected for review. A list of OTA  reports reviewed is included below.

Office of Technology Assessment Reports Reviewed for Preparing this Handbook

Office of Technology Assessment. Insider’s Guide to OTA. Washington,  D.C.: January 1995.

Office of Technology Assessment. Policy Analysis at OTA: A Staff  Assessment. Washington, D.C.: May 1993.

Office of Technology Assessment. Research Assistants Handbook.  Washington, D.C.: June 1992.

Office of Technology Assessment. Strengths and Weaknesses of OTA  Policy Analysis. Washington, D.C.: 1992.

Office of Technology Assessment. The OTA Orange Book: Policies and  Procedures of the Office of Technology Assessment: Communication with  Congress and the Public. Washington, D.C.: February 1986.

Office of Technology Assessment. What OTA Is, What OTA Does, How  OTA Works. Washington, D.C.: March 1983.

Office of Technology Assessment. Draft: An OTA Handbook. Washington,  D.C.: June 7, 1982.

Office of Technology Assessment. Draft: A Management Overview  Methodology for Technology Assessment. Washington, D.C.: February 2,  1981.*  Office of Technology Assessment. Draft: Technology Assessment in  Industry: A Counterproductive Myth. Washington, D.C.: January 30,  1981.*  Office of Technology Assessment. Draft: Technology Assessment  Methodology and Management Practices. Washington, D.C.: January 12,  1981.*  Office of Technology Assessment. Draft: Technology Assessment in the  Private Sector. Washington, D.C.: January 9, 1981.*  Office of Technology Assessment. Draft: A Process for Technology  Assessment Based on Decision Analysis. Washington, D.C.: January  1981.*  Office of Technology Assessment. Draft: Technology as Social  Organization. Washington, D.C.: January 1981.*  Office of Technology Assessment. A Summary of the Doctoral  Dissertation: A Decision Theoretic Model of Congressional Technology  Assessment. Washington, D.C.: January 1981.*  Office of Technology Assessment. Report on Task Force Findings and  Recommendations: Prepared by the OTA Task Force on TA Methodology  and Management. Washington, D.C.: August 13, 1980.

Office of Technology Assessment. Phase I Survey Results: Draft Papers  Prepared for the Task Force on TA Methodology and Management.  Washington, D.C.: April 10, 1980.

Review of Select Congressional Research Service Reports

We identified a pool of 29 Congressional Research Service (CRS) reports  to consider reviewing that were technology assessments or included an  analysis of policy options, based on a keyword search of CRS’s website.  We also interviewed CRS officials. Of the initial 29 CRS reports we  identified, we selected six CRS reports to review, based on the following  criteria: (1) published within the past 15 years (2004-2019) and (2) if a  review of technology (technology assessment) and/or policy options was  included. Reports were excluded based on the following criteria: (1) for  technology assessment related reports—if they represented a summary  of a technology assessment that was included in our review or (2) for  policy options related reports—the report did not indicate how CRS  arrived at the policy options (no methodology to review or analyze). A list  of CRS reports reviewed is included below.

Congressional Research Service Reports Reviewed for Preparing this Handbook

Congressional Research Service. Advanced Nuclear Reactors:  Technology Overview and Current Issues. Washington, D.C.: April 18,  2019.

Congressional Research Service. Drug Shortages: Causes, FDA  Authority, and Policy Options. Washington, D.C.: December 27, 2018.

Congressional Research Service. Policy Options for Multiemployer  Defined Benefit Pension Plans. Washington, D.C.: September 12, 2018.

Congressional Research Service. Shale Energy Technology Assessment:  Current and Emerging Water Practices. Washington, D.C.: July 14, 2014.

Congressional Research Service. Carbon Capture: A Technology  Assessment. Washington, D.C.: November 5, 2013.

Congressional Research Service. Energy Storage for Power Grids and  Electric Transportation: A Technology Assessment. Washington, D.C.:  March 27, 2012.

A GAO librarian performed a literature search based on keyword  searches for two areas—TA and policy options. For TA literature, the  team selected 29 documents to review that were frameworks, guides,  models, or other compilations, based on a review of the literature titles  and abstracts. In general, we excluded specialized types of TAs, such as  health-related TAs, as we focused on TA design more broadly. For policy  options literature, the team selected 14 documents to review that were  frameworks, guides, models, or other compilations and focused on policy  options related to science and technology. We also asked experts we  consulted to suggest literature for our review; these suggestions  confirmed the literature list noted below. A list of literature reviewed is  included below.

Literature Reviewed for Preparing this Handbook

Grunwald, Armin. Technology Assessment in Practice and Theory.  London and New York: Routledge, 2019.

Armstrong, Joe E., and Willis W. Harman. Strategies For Conducting  Technology Assessments. London and New York: Routledge, 2019.

Noh, Heeyong, Ju-Hwan Seo, Hyoung Sun Yoo, and Sungjoo Lee. “How  to Improve a Technology Evaluation Model: A Data-driven Approach.”  Technovation, vol. 72/73 (2018): p. 1-12.

Larsson, A., T. Fasth, M. Wärnhjelm, L. Ekenberg, and M. Danielson.  “Policy Analysis on the Fly With an Online Multicriteria Cardinal Ranking  Tool.” Journal of Multi-Criteria Decision Analysis, vol. 25 (2018): p. 55-66.

Nooren, P., N. van Gorp, N. van Eijk, and R. O. Fathaigh. “Should We  Regulate Digital Platforms? A New Framework for Evaluating Policy  Options.” Policy and Internet, vol. 10, no. 3 (2018): p. 264-301.

Smith, A., K. Collins, and D. Mavris. “Survey of Technology Forecasting  Techniques for Complex Systems.” Paper presented at 58th  AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials  Conference, Grapevine, TX (2017).

Ibrahim, O., and A. Larsson. “A Systems Tool for Structuring Public Policy  Problems and Design of Policy Options.” Int. J. Electronic Governance,  vol. 9 , nos. 1/2 (2017): p. 4-26.

Christopher, A. Simon. Public Policy Preferences and Outcomes. 3rd ed.  New York: Routledge, 2017.

Weimer, David L., and R. Aidan Vining. Policy Analysis Concepts and  Practice. 6th ed. London and New York: Routledge, 2017.

Mulder, K. “Technology Assessment.” In Foresight in Organizations:  Methods and Tools, edited by Van Der Duin, Patrick, 109-124, 2016.

Coates, Joseph F. “A 21st Century Agenda for Technology Assessment.”  Technological Forecasting and Social Change, vol. 113 part A (2016): p.  107-109.

Coates, Joseph F. “Next Stages in Technology Assessment: Topics and  Tools.” Technological Forecasting and Social Change, vol. 113 (2016): p.  112-114.

Mazurkiewicz, A., B. Belina, B. Poteralska, T. Giesko, and W. Karsznia.  “Universal Methodology for the Innovative Technologies Assessment.”  Proceedings of the European Conference on Innovation and  Entrepreneurship (2015): p. 458-467.

Sadowski, J. “Office of Technology Assessment: History, Implementation,  and Participatory Critique.” Technology in Society, vol. 42 (2015): p. 9-20.

Larsson, A., O. Ibrahim. “Modeling for Policy Formulation: Causal  Mapping, Scenario Generation, and Decision Evaluation.” In Electronic  Participation: 7th IFIP 8.5 International Conference, 135-146, Springer,  2015.

Moseley, C., H. Kleinert, K. Sheppard-Jones, and S. Hall. “Using  Research Evidence to Inform Public Policy Decisions.” Intellectual and  Developmental Disabilities, vol. 51 (2013): p. 412-422.

Calof, J., R. Miller, and M. Jackson. “Towards Impactful Foresight:  Viewpoints from Foresight Consultants and Academics.” Foresight, vol.  14 (2012): p. 82-97.

Parliaments and Civil Society in Technology Assessment, Collaborative  Project on Mobilization and Mutual Learning Actions in European  Parliamentary Technology Assessment. The Netherlands: Rathenau  Instituut, 2012.

Blair, P. D. “Scientific Advice for Policy in the United States: Lessons from  the National Academies and the Former Congressional Office of  Technology Assessment.” In The Politics of Scientific Advice: Institutional  Design for Quality Assurance, ed. Lentsch, Justus, 297-333, 2011.

Paracchini, M.L., C. Pacini, M.L.M. Jones, and M. Pérez-Soba. “An  Aggregation Framework to Link Indicators Associated With Multifunctional  Land Use to the Stakeholder Evaluation of Policy Options.” Ecological  Indicators, vol. 11 (2011): p 71-80.

Roper, A. T., S. W. Cunningham, A. L. Porter, T. W. Mason, F. A. Rossini,  and J. Banks. Forecasting and Management of Technology, 2nd ed. New  Jersey: Wiley, 2011.

Lepori, B., E. Reale, and R. Tijssen. “Designing Indicators for Policy  Decisions: Challenges, Tensions and Good Practices: Introduction to a  Special Issue.” Research Evaluation, vol. 20, no. 1 (2011): p. 3-5.

Russel, A. W., F. M. Vanclay, and H. J. Aslin H.J. “Technology  Assessment in Social Context: The Case for a New Framework for  Assessing and Shaping Technological Developments.” Impact  Assessment and Project Appraisal, vol. 28, no. 2 (2010): p. 109-116.

Shiroyama, H., G. Yoshizawa, G., M. Matsuo, and T. Suzuki. “Institutional  Options and Operational Issues in Technology Assessment: Lessons  from Experiences in the United States and Europe.” Paper presented at  Atlanta Conference on Science and Innovation Policy, Atlanta, 2009.

Tran, T.A., and T. Daim T. “A Taxonomic Review of Methods and Tools  Applied in Technology Assessment.” Technological Forecasting and  Social Change, vol. 75 (2008): p. 1396-1405.

Brun, G., and G. Hirsch Hadorn. “Ranking Policy Options for Sustainable  Development.” Poiesis Prax, vol. 5 (2008): p. 15-31.

Tran, T.A. “Review of Methods and Tools applied in Technology  Assessment Literature.” Paper presented at Portland International  Conference on Management of Engineering and Technology, Portland  Oregon, 2007.

Burgess, J., A. Stirling, J. Clark, G. Davies, M. Eames, K. Staley, and S.  Williamson. “Deliberative Mapping: A Novel Analytic-Deliberative  Methodology to Support Contested Science-Policy Decisions.” Public  Understanding of Science, vol. 16 (2007): p. 299-322.

Decker, M., and M. Ladikas. Bridges Between Science, Society and  Policy: Technology Assessment — Methods and Impacts. Berlin:  Springer-Verlag, 2004.

Guston, D. H., and D. Sarewitz. “Real-time Technology Assessment.”  Technology in Society, vol. 24 (2002): p. 93-109.

Rip, A. “Technology Assessment.” In International Encyclopedia of the  Social & Behavioral Science, vol. 23, edited by Smelster, N. J. and B. P.  Baltes, 15512-15515. Amsterdam: Elsevier, 2001.

Van Den Ende, J., K. Mulder, M. Knot, E. Moors, and P. Vergragt.  “Traditional and Modern Technology Assessment: Toward a Toolkit.”  Technological Forecasting and Social Change, vol. 58 (1998): p. 5-21.

Wood, F. B. “Lessons in Technology Assessment: Methodology and  Management at OTA.” Technological Forecasting and Social Change, vol.  54 (1997): p. 145-162.

Janes, M. C. “A Review of the Development of Technology Assessment.”  International Journal of Technology Management, vol. 11, no. 5-6 (1996):  p. 507-522.

Hastbacka, M. A., and C. G. Greenwald. “Technology Assessment - Are  You Doing it Right?” Arthur D. Little – PRISM, no. 4 (1994).

Rivera, W. M., D. J. Gustafson, and S. L. Corning. “Policy Options in  Developing Agricultural Extension Systems: A Framework for Analysis.”  International Journal of Lifelong Education, vol. 10, no. 1 (1991): p. 61-74.

Lee, A. M., and P. L. Bereano. “Developing Technology Assessment  Methodology: Some Insights and Experiences.” Technological  Forecasting and Social Change, vol. 19 (1981): p. 15-31.

Porter, A. L., F. A. Rossini, S. R. Carpenter, and A. T. Roper. A  Guidebook for Technology Assessment and Impact Analysis, vol. 4. New  York and Oxford: North Holland, 1980.

Pulver, G.C. “A Theoretical Framework for the Analysis of Community  Economic Development Policy Options.” In Nonmetropolitan Industrial  Growth and Community Change, edited by Summers, G. and A. Selvik,  105-117. Massachusetts and Toronto: Lexington Books, 1979.

Ascher, W. “Problems of Forecasting and Technology Assessment.”  Technological Forecasting and Social Change, vol. 13, no. 2 (1979): p.  149-156.

Majone, G. “Technology Assessment and Policy Analysis.” Policy  Sciences, vol. 8, no. 2 (1977): p. 173-175.

Berg, M., K. Chen, and G. Zissis. “A Value-Oriented Policy Generation  Methodology for Technology Assessment.” Technological Forecasting  and Social Change, vol. 4, no. 4 (1976): p. 401-420.

Lasswell, Harold D. A Pre-View of Policy Sciences. Policy Sciences Book  Series. New York: Elsevier, 1971.

We held a forum to gather experts’ opinions regarding TA design. An  initial list of experts was prepared based on a review of GAO TA reports,  literature, and referral by other experts. Experts were selected based on  their knowledge and expertise in the subject, including: (1) prior  participation on a National Academy of Sciences panel or other similar  meeting; (2) leadership position in one or more organizations or sectors  relevant to technology research and development implementation or  policy; and (3) relevant publications or sponsorship of reports. Care was  also taken to ensure a balance of sectors, backgrounds, and specific  areas of expertise (e.g., science, technology, policy, information  technology, and law). We also asked the experts to suggest literature for  our review; these suggestions confirmed the literature list noted above. A  list of external experts consulted is included below.

External Experts Consulted for the Handbook

Dr. Jeffrey M. Alexander, Senior Manager, Innovation Policy, RTI  International  Dr. Robert D. Atkinson, President, Information Technology and Innovation  Foundation  Mr. David Bancroft, Executive Director, International Association for  Impact Assessment  Mr. Duane Blackburn, S&T Policy Analyst, Office of the CTO, MITRE  Dr. Peter D. Blair, Executive Director, Division of Engineering and  Physical Sciences, National Academies of Sciences, Engineering, and  Medicine  Ms. Marjory Blumenthal, Acting Associate Director, Acquisition and  Technology Policy Center; Senior Policy Researcher, RAND Corporation  Mr. Chris J. Brantley, Managing Director, Institute of Electrical and  Electronics Engineers, Inc., USA  Dr. Jonathan P. Caulkins, H. Guyford Stever University Professor of  Operations Research and Public Policy, Carnegie Mellon University  Mr. Dan Chenok, Executive Director, Center for The Business of  Government, IBM  Dr. Gerald Epstein, Distinguished Research Fellow, Center for the Study  of Weapons of Mass Destruction, National Defense University  Dr. Robert M. Friedman, Vice President for Policy and University  Relations, J. Craig Venter Institute  Mr. Zach Graves, Head of Policy, Lincoln Network  Ms. Allison C. Lerner, Inspector General, National Science Foundation  Mr. Mike Molnar, Director of Office of Advanced Manufacturing, National  Institute of Standards and Technology  Dr. Michael H. Moloney, CEO, American Institute of Physics  Dr. Ali Nouri, President, Federation of American Scientists  Dr. Jon M. Peha, Professor, Engineering and Public Policy; Courtesy  Professor, Electrical and Computer Engineering, Carnegie Mellon  University  Dr. Stephanie S. Shipp, Deputy Director and Professor, University of  Virginia, Biocomplexity Institute and Initiative, Social and Decision  Analytics Division  Dr. Daniel Sarewitz, Co-Director, Consortium for Science, Policy &  Outcomes Professor of Science and Society, School for the Future of  Innovation in Society, Arizona State University  Ms. Rosemarie Truman, Founder and CEO, Center for Advancing  Innovation  Dr. Chris Tyler, Director of Research and Policy, Department of Science,  Technology, Engineering and Public Policy (STEaPP), University College  London (UCL)

Appendix II: Summary of Steps for GAO’s General Engagement Process

As part of GAO’s Quality Assurance Framework, GAO’s general design  and project plan templates contain five phases that are followed in  sequential order, with modifications or changes as needed. GAO  technology assessments (TAs) use these templates, as applicable.  Throughout the phases, the status of the work, including decisions, is  communicated to stakeholders and congressional committees that  requested the work. Provided below is a summary of the activities GAO  staff undertake during each of the phases, and is based on a review of  GAO documentation related to engagement phases.

Engagement characteristics such as risk level or internal  stakeholders are determined at a high-level Engagement  Acceptance Meeting.

Engagement teams obtain a copy of and review the congressional  request letter(s), as applicable.

Staff are assigned to the engagement and set up the electronic  engagement documentation set folders.

Staff enter standard information regarding the engagement in  GAO’s Engagement Management System (EMS), which is used  to monitor the status of the engagement throughout the  engagement process and regularly updated.

Engagement teams hold an initiation meeting with engagement  stakeholders to discuss potential research questions, design  options, and stakeholder involvement.

Engagement teams clarify engagement objectives and approach  through discussions with the congressional requesters, as  applicable.

Engagement teams obtain background information. For example,  to gather information about the topic and any work already  performed, teams may conduct a literature review, search prior  and ongoing GAO work related to the topic, or consult with  external stakeholders, outside experts, and agency officials,  including the Congressional Research Service, Congressional  Budget Office, and Inspectors General of federal agencies.

Engagement teams formally notify agencies of the engagement  through a notification letter, and hold an entrance conference, as  applicable.

Engagement teams prepare a design matrix, project plan, risk  assessment tool, data reliability assessment, and all participants  on engagements, including stakeholders, affirm their  independence. The design matrix is a tool that describes:  researchable questions; criteria; information required and sources;  scope and methodology; and limitations. The project plan  identifies key activities and tasks, dates for completing them, and  staff assigned.

Engagement teams secure approval to move forward with  engagement approach at a high-level Engagement Review  Meeting.

Phase III: Evidence Gathering, Finalizing Design, and Analysis

Engagement teams finalize design: teams work with internal  stakeholders to confirm soundness and reach agreement on  proposed initial design. If engagement teams and stakeholders  conclude that additional work is needed or the design faces  significant implementation challenges, design is reviewed and  modified, as needed.

Engagement teams collect and analyze evidence: teams may  collect and analyze evidence using a variety of methodologies  including document review, interviews, surveys, focus groups, and  various forms of data analysis. For example, engagement teams  may meet with agency officials and outside experts, as applicable,  to gather evidence.

Engagement teams assess evidence and agree on conclusions:  teams assess whether the evidence collected is sufficient and  appropriate to support findings and conclusions reached for each  objective. Once sufficient evidence is collected and analyzed, the  team discusses how the evidence supports potential findings and  shares these findings with stakeholders, generally in the form of a  formal message agreement meeting.

Engagement teams update congressional requesters, as  applicable, on the engagement status and potential findings.

Engagement teams draft product: after drafting the product, teams  send draft to internal stakeholders for review. Teams also send  draft to relevant external parties, including relevant agencies, to  confirm facts and obtain their views.

Teams identify sources of all information in the draft and an  independent analyst (not on the team) verifies the sources through  a process called indexing and referencing.

Engagement teams perform exit conferences with agencies, as  applicable, to discuss findings and potential recommendations.  Agencies and external parties are given the opportunity to  comment on the draft, as applicable.

Engagement teams communicate findings and potential  recommendations, as well as timeframes for issuing the product,  to congressional requesters, as applicable.

The draft product is copy-edited, prepared for issuance, and  publicly released on GAO’s website, as applicable.

Engagement teams conduct follow-up, track the results, and  prepare reports on the status of recommendations and financial  and non-financial benefits, as applicable, using GAO’s results  tracking system.

Appendix III: Example Methods for Technology Assessment

This appendix provides examples of methods and analytical approaches  that GAO technology assessment (TA) teams can use to examine  different types of evidence. Also included in this appendix are  considerations of the strengths, limitations, and synergies among  evidence types and methods, which can be useful to consider throughout  design to ensure that evidence is sufficient and appropriate to answer the  researchable questions. Examples from GAO TAs were used given our  familiarity with GAO products, though numerous other (non-GAO)  examples of TA methods exist. This appendix included a review of GAO  reports and select literature, and is not intended to be comprehensive.  This is a simplified presentation of methods, and there is variation in the  levels of structure of the example methods.

This appendix is divided into several sections, including by evidentiary  types: Testimonial, Documentary, and Physical. For each of these types  of evidence, example methods are presented with low and high levels of  structure, and include examples of considerations (such as general  benefits and limitations) that analysts may consider. In general, more  highly structured approaches generate increased consistency and  comparability of results that allows for stronger quantification. Less  structured approaches tend to provide more flexibility and context, and  richer illustrative evidence.

Examples of Methodologies for Testimonial Evidence

Testimonial evidence is elicited from respondents to understand their  experience, opinions, knowledge, and behavior, and it can be obtained  through a variety of methods, including inquiries, interviews, focus  groups, expert forums, or questionnaires. Testimonial evidence can be  gathered from individuals who may be responding personally based on  their own experience in an official capacity to represent agencies or other  entities, or groups, who may share individual level responses, or may  present a single group response. Group testimony enables interactions  that can be used to explore similarities and differences among  participants, to identify tensions or consensus in a group, or to explore  ideas for subsequent research and collaboration. It is important to  evaluate the objectivity, credibility, and reliability of testimonial evidence.  Analysts may use a combination of approaches to gather testimonial  evidence, depending on the relevant population(s) of respondents,  intended analytical approach(es), likely respondent burden, and resource  considerations. Table 9 provides more examples.

Examples of Methodologies for Documentary Evidence

Documentary evidence is existing information, such as letters, contracts,  accounting records, invoices, spreadsheets, database extracts,  electronically stored information, and management information on  performance. It is important to evaluate the objectivity, credibility, and  reliability of documentary evidence. Analysts may use a combination of  approaches to gather documentary evidence, depending on the relevant  sources and types of documents, intended analytical approach(es), and  resource considerations. Table 10 provides more examples.

Examples of Methodologies for Physical Evidence

Physical evidence is obtained by direct inspection or observation of  people, property, or events. The appropriateness of physical evidence  depends on when, where, and how the inspection or observation was  made and whether it was recorded in a manner that fairly represents the  facts observed. Common considerations for physical evidence include the  reliability of site selection, intended analytical approaches, and resource  considerations. Table 11 provides more examples.

GAO may also rely on agency and other secondary data. Considerations  for those secondary data are dependent on the type, source, and  collection method, and could include all of the considerations above. Use  of secondary data is usually more efficient than collecting new data on a  topic, and administrative records (a form of documentary evidence) are  generally not as prone to self-reporting biases that may be present in  testimonial evidence. However, when secondary data are used, more  work may be required to assess whether data are reliable and appropriate  for a given purpose. For example, analysts will gather all appropriate  documentation, including record layout, data element dictionaries, user’s  guides, and data maintenance procedures. Depending on the database,  procedures and analysis can be very complex—and it would be important  to note assumptions, limitations, and caveats pertaining to the data, which  may affect the conclusions that can be drawn based on the analyses.

Examples of analytical approaches found in the literature to analyze data  include:  Interpretive structural modeling: shows a graphical relationship among  all elements to aid in structuring a complex issue area, and may be  helpful in delineating scope.

Trend extrapolation: is a family of techniques to project time-series  data using specific rules, and may be helpful in forecasting  technology.

Scenarios: is a composite description of possible future states  incorporating a number of characteristics, and may be helpful in policy  analysis.

Scanning methods, such as checklists: is listing factors to consider in  a particular area of inquiry, and may be helpful in identifying potential  impacts.

Tracing methods, such as relevance trees: includes identifying  sequential chains of cause and effect or other relationships, and may  be helpful in identifying potential impacts.

Cross-effect matrices: are two-dimensional matrix representations to  show the interaction between two sets of elements, and may be  helpful in analyzing consequences of policy options.

Simulation models: are a simplified representation of a real system  that is used to explain dynamic relationships of the system, and may  be helpful in identifying impacts and forecasting technology.

Benefit-cost analysis: is a systematic quantitative method of  assessing the desirability of government projects or policies when it is  important to take a long view of future effects and a broad view of  possible side effects.

Decision analysis: is an aid to compare alternatives by weighing the  probabilities of occurrences and the magnitudes of their impacts, and  may be helpful in determining impacts and assessing policy options.

Scaling: is an aid that may include developing a matrix that identifies  potential impact related to an activity and stakeholder group, and  qualitatively or quantitatively assesses the potential impact, and may  be helpful analyzing potential impacts, including of policy options.

Appendix IV: GAO Contact and Staff Acknowledgments

In addition to the contacts named above, key contributors to this report  were R. Scott Fletcher (Assistant Director), Diantha Garms (Analyst-in- charge), Nora Adkins, Colleen Candrl, Virginia Chanley, Robert Cramer,  David Dornisch, John De Ferrari, Dennis Mayo, Anika McMillon, SaraAnn  Moessbauer, Amanda Postiglione, Steven Putansu, Oliver Richard, Meg  Tulloch, Ronald Schwenn, Ben Shouse, Amber Sinclair, Ardith Spence,  Andrew Stavisky, David C. Trimble, and Edith Yuh.

The Government Accountability Office, the audit, evaluation, and investigative  arm of Congress, exists to support Congress in meeting its constitutional  responsibilities and to help improve the performance and accountability of the  federal government for the American people. GAO examines the use of public  funds; evaluates federal programs and policies; and provides analyses,  recommendations, and other assistance to help Congress make informed  oversight, policy, and funding decisions. GAO’s commitment to good government  is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony Order by Phone

The fastest and easiest way to obtain copies of GAO documents at no cost is  through our website. Each weekday afternoon, GAO posts on its website newly  released reports, testimony, and correspondence. You can also subscribe to  GAO’s email updates to receive notification of newly posted products.

The price of each GAO publication reflects GAO’s actual cost of production and  distribution and depends on the number of pages in the publication and whether  the publication is printed in color or black and white. Pricing and ordering  information is posted on GAO’s website, https://www.gao.gov/ordering.htm.

Orders may be paid for using American Express, Discover Card, MasterCard,  Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.  Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts.  Visit GAO on the web at https://www.gao.gov.

To Report Fraud, Waste, and Abuse in Federal Programs

James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707  U.S. Government Accountability Office, 441 G Street NW, Room 7814,  Washington, DC 20548  Please Print on Recycled Paper.

The Technology Assessment Design Handbook offers both GAO’s own staff and other interested users tools and approaches to think about when designing technology assessments. The handbook helps users analyze the impact of technology and make complex issues more easily understood and useful to policymakers.

what to think about when designing assessments

what to think about when including policy options in assessments

potential challenges and some ways to address them

We expect to update the handbook based on public comments and additional experiences. Send comments to TAHandbook@gao.gov.