question,contexts,answer,ground_truth,faithfulness,answer_relevancy,context_recall,context_precision,answer_correctness
What is the significance of providing notice and explanation as a legal requirement in the context of automated systems?,"['NOTICE & \nEXPLANATION \nWHY THIS PRINCIPLE IS IMPORTANT\nThis section provides a brief summary of the problems which the principle seeks to address and protect \nagainst, including illustrative examples. \nAutomated systems now determine opportunities, from employment to credit, and directly shape the American \npublic’s experiences, from the courtroom to online classrooms, in ways that profoundly impact people’s lives. But this expansive impact is not always visible. An applicant might not know whether a person rejected their resume or a hiring algorithm moved them to the bottom of the list. A defendant in the courtroom might not know if a judge deny\n-\ning their bail is informed by an automated system that labeled them “high risk.” From correcting errors to contesting decisions, people are often denied the knowledge they need to address the impact of automated systems on their lives. Notice and explanations also serve an important safety and efficacy purpose, allowing experts to verify the reasonable\n-\nness of a recommendation before enacting it. \nIn order to guard against potential harms, the American public needs to know if an automated system is being used. Clear, brief, and understandable notice is a prerequisite for achieving the other protections in this framework. Like\n-\nwise, the public is often unable to ascertain how or why an automated system has made a decision or contributed to a particular outcome. The decision-making processes of automated systems tend to be opaque, complex, and, therefore, unaccountable, whether by design or by omission. These factors can make explanations both more challenging and more important, and should not be used as a pretext to avoid explaining important decisions to the people impacted by those choices. In the context of automated systems, clear and valid explanations should be recognized as a baseline requirement. \nProviding notice has long been a standard practice, and in many cases is a legal requirement, when, for example, making a video recording of someone (outside of a law enforcement or national security context). In some cases, such as credit, lenders are required to provide notice and explanation to consumers. Techniques used to automate the process of explaining such systems are under active research and improvement and such explanations can take many forms. Innovative companies and researchers are rising to the challenge and creating and deploying explanatory systems that can help the public better understand decisions that impact them. \nWhile notice and explanation requirements are already in place in some sectors or situations, the American public deserve to know consistently and across sectors if an automated system is being used in a way that impacts their rights, opportunities, or access. This knowledge should provide confidence in how the public is being treated, and trust in the validity and reasonable use of automated systems.'
 'tion responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality changes. You should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome. Automated systems should provide explanations that are technically valid, meaningful and useful to you and to any operators or others who need to understand the system, and calibrated to the level of risk based on the context. Reporting that includes summary information about these automated systems in plain language and assessments of the clarity and quality of the notice and explanations should be made public whenever possible. \n6'
 ""NOTICE & \nEXPLANATION \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nTailored to the level of risk. An assessment should be done to determine the level of risk of the auto -\nmated system. In settings where the consequences are high as determined by a risk assessment, or extensive \noversight is expected (e.g., in criminal justice or some public sector settings), explanatory mechanisms should be built into the system design so that the system’s full behavior can be explained in advance (i.e., only fully transparent models should be used), rather than as an after-the-decision interpretation. In other settings, the extent of explanation provided should be tailored to the risk level. \nValid. The explanation provided by a system should accurately reflect the factors and the influences that led \nto a particular decision, and should be meaningful for the particular customization based on purpose, target, and level of risk. While approximation and simplification may be necessary for the system to succeed based on the explanatory purpose and target of the explanation, or to account for the risk of fraud or other concerns related to revealing decision-making information, such simplifications should be done in a scientifically supportable way. Where appropriate based on the explanatory system, error ranges for the explanation should be calculated and included in the explanation, with the choice of presentation of such information balanced with usability and overall interface complexity concerns. \nDemonstrate protections for notice and explanation \nReporting. Summary reporting should document the determinations made based on the above consider -\nations, including: the responsible entities for accountability purposes; the goal and use cases for the system, identified users, and impacted populations; the assessment of notice clarity and timeliness; the assessment of the explanation's validity and accessibility; the assessment of the level of risk; and the account and assessment of how explanations are tailored, including to the purpose, the recipient of the explanation, and the level of risk. Individualized profile information should be made readily available to the greatest extent possible that includes explanations for any system impacts or inferences. Reporting should be provided in a clear plain language and machine-readable manner. \n44""
 'You should be able to opt out, where appropriate, and \nhave access to a person who can quickly consider and remedy problems you encounter. You should be able to opt out from automated systems in favor of a human alternative, where appropriate. Appropriateness should be determined based on rea\n-\nsonable expectations in a given context and with a focus on ensuring broad accessibility and protecting the public from especially harm\n-\nful impacts. In some cases, a human or other alternative may be re -\nquired by law. You should have access to timely human consider -\nation and remedy by a fallback and escalation process if an automat -\ned system fails, it produces an error, or you would like to appeal or contest its impacts on you. Human consideration and fallback should be accessible, equitable, effective, maintained, accompanied by appropriate operator training, and should not impose an unrea\n-\nsonable burden on the public. Automated systems with an intended use within sensitive domains, including, but not limited to, criminal justice, employment, education, and health, should additionally be tailored to the purpose, provide meaningful access for oversight, include training for any people interacting with the system, and in\n-\ncorporate human consideration for adverse or high-risk decisions. Reporting that includes a description of these human governance processes and assessment of their timeliness, accessibility, out\n-\ncomes, and effectiveness should be made public whenever possible. HUMAN  ALTERNATIVES , C ONSIDERATION ALLBACKF AND, \n46']","Providing notice and explanation as a legal requirement in the context of automated systems is significant for several reasons:

1. **Transparency**: It ensures that individuals are aware when automated systems are being used to make decisions that affect their lives, such as in hiring, credit scoring, or legal judgments. This transparency helps build trust in these systems (Source: [document name], p. 6).

2. **Accountability**: By requiring organizations to explain how decisions are made, it holds them accountable for the outcomes of their automated systems. This means that if a decision negatively impacts someone, they have the right to understand the reasoning behind it and contest it if necessary (Source: [document name], p. 6).

3. **Empowerment**: Notice and explanation empower individuals by giving them the information they need to understand and potentially challenge decisions made by automated systems. This is crucial in ensuring that people can address any errors or biases that may arise from these systems (Source: [document name], p. 6).

4. **Safety and Efficacy**: Clear explanations allow experts to verify the reasonableness of automated recommendations before they are enacted, which is essential for ensuring that these systems operate safely and effectively (Source: [document name], p. 6).

5. **Legal Compliance**: In some sectors, providing notice and explanation is already a legal requirement. Extending this practice across various sectors helps ensure consistent protection of individuals' rights and opportunities (Source: [document name], p. 6).

Overall, these requirements are designed to protect individuals from potential harms associated with automated decision-making and to foster a more equitable and just use of technology in society.","Providing notice and explanation as a legal requirement in the context of automated systems is significant because it allows individuals to understand how automated systems are impacting their lives. It helps in correcting errors, contesting decisions, and verifying the reasonableness of recommendations before enacting them. Clear and valid explanations are essential to ensure transparency, accountability, and trust in the use of automated systems across various sectors.",1.0,0.9713211543186683,1.0,0.999999999975,0.6044168324380029
"How can structured human feedback exercises, such as GAI red-teaming, be beneficial for GAI risk measurement and management?","['50 Participatory Engagement Methods  \nOn an ad hoc or more structured basis, organizations can design and use a variety of channels to engage \nexternal stakeholders in product development or review. Focus groups with select experts can provide \nfeedback on a range of issues. Small user studies c an provide feedback from representative groups or \npopulations. Anonymous surveys can be used to poll or gauge reactions to specific features. Participatory engagement methods are often less structured than field testing or red teaming, and are more \ncommonl y used in early stages of AI or product development.  \nField Testing  \nField testing involves structured settings to evaluate risks and impacts and to simulate the conditions \nunder which the GAI system will be deployed. Field style tests can be adapted from a focus on user \npreferences and experiences towards AI risks and impacts – both negative and positive. When carried \nout with large groups of users, these tests can provide estimations of the likelihood of risks and impacts \nin real world interactions.  \nOrganizations may also collect feedback on outcomes, harms, and user experience directly from users in the production environment after a model has been released, in accordance with human subject \nstandards such as informed consent and compensation. Organiz ations should follow applicable human \nsubjects research requirements, and best practices such as informed consent and subject compensation, \nwhen implementing feedback activities. \nAI Red -teaming  \nAI red -teaming is an evolving practice that references exercises often  conducted in a controlled \nenvironment and in collaboration with AI developers building AI models  to identify potential adverse \nbehavior or outcomes of a GAI model or system,  how they could occur, an d stress test safeguards” . AI \nred-teaming can be performed before or after AI models or systems are made available to the broader \npublic; this section focuses on red- teaming in pre -deployment contexts.  \nThe quality of AI red- teaming outputs is  related to the background and expertise of the AI red  team \nitself. Demographically and interdisciplinarily  diverse AI red  teams can be used to identify flaws in the \nvarying contexts where GAI will be used. For best results, AI red  teams should demonstrate domain \nexpertise, and awareness of socio -cultural aspects within the deployment context. AI red -teaming results \nshould be given additional analysis before they are incorporated into organizational governance and \ndecision making, policy and procedural updates, and AI risk management efforts. \nVarious types of AI red -teaming may be appropriate, depending on the use case:  \n• General Public: Performed by general users (not necessarily AI or technical experts) who are \nexpected to use the model or interact with its outputs, and who bring their own lived \nexperiences and perspectives to the task of AI red -teaming . These individuals may have been'
 '42 MG-2.4-002 Establish and maintain procedures for escalating GAI system incidents to the \norganizational risk management  authority when specific criteria for deactivation \nor disengagement is met for a particular context of use or for the GAI system as a \nwhole.  Information Security  \nMG-2.4-003 Establish and maintain procedures for the remediation of issues which trigger \nincident response processes for the use of a GAI system, and provide stakeholders timelines associated with the remediation plan.  Information Security  \n \nMG-2.4-004 Establish and regularly review specific criteria that warrants the deactivation of \nGAI systems in accordance with set risk tolerances and appetites.  Information Security  \n \nAI Actor Tasks:  AI Deployment, Governance and Oversight, Operation and Monitoring  \n \nMANAGE 3.1:  AI risks and benefits from third -party resources are regularly monitored, and risk controls are applied and \ndocumented.  \nAction ID  Suggested Action  GAI Risks  \nMG-3.1-001 Apply organizational risk tolerances and controls (e.g., acquisition and \nprocurement processes; assessing personnel credentials and qualifications, \nperforming background checks; filtering GAI input and outputs, grounding, fine tuning , retrieval- augmented generation) to third -party GAI resources: Apply \norganizational risk tolerance to the utilization of third -party datasets and other \nGAI resources; Apply organizational risk tolerances to fine -tuned third -party \nmodels; Apply organizational risk tolerance to existing t hird-party models \nadapted to a new domain; Reassess risk measure ments after fine -tuning third-\nparty GAI models.  Value Chain and Component Integration ; Intellectual Property  \nMG-3.1-002 Test GAI system value chain  risks (e.g., data poisoning, malware, other software  \nand hardware vulnerabilities; labor practices; data privacy and localization compliance; geopolitical alignment).  Data Privacy ; Information Security ; \nValue Chain and Component Integration ; Harmful Bias and \nHomogenization  \nMG-3.1-003 Re-assess model risks after fine -tuning or retrieval -augmented generation \nimplementation and for any third -party GAI models deployed for applications \nand/or use cases that were not evaluated in initial testing.  Value Chain and Component Integration  \nMG-3.1-004 Take reasonable measures to review training data for CBRN information, and  \nintellectual property, and where appropriate, remove it.  Implement reasonable \nmeasures to prevent, flag, or take other action in response to outputs that reproduce particular training data  (e.g., plagiarized, trademarked, patented, \nlicensed  content  or trade secret material).  Intellectual Property ; CBRN  \nInformation  or Capabilities'
 '49 early lifecycle TEVV approaches are developed and matured for GAI, organizations may use \nrecommended “pre- deployment testing” practices to measure performance, capabilities, limits, risks, \nand impacts. This section describes risk measurement and estimation as part of pre -deployment TEVV, \nand examines the state of play for pre -deployment testing methodologies.  \nLimitations of Current Pre -deployment Test Approaches  \nCurrently available pre -deployment TEVV processes used for GAI applications may be inadequate, non-\nsystematically applied, or fail to reflect or mismatched to deployment contexts. For example, the \nanecdotal testing of GAI system capabilities through video games or standardized tests designed for \nhumans (e.g., intelligence tests, professional licensing exams) does not guarantee GAI system validity or \nreliability in those domains. Similarly, jailbreaking or prompt  engineering tests may not systematically \nasse ss validity or reliability risks.  \nMeasurement gaps can arise from mismatches between laboratory and real -world settings. Current \ntesting approaches often remain focused on laboratory conditions or restricted to benchmark test \ndatasets and in silico techniques that may not extrapolate well to —or directly assess GAI impacts in real -\nworld conditions. For example, current measurement gaps for GAI make it difficult to precisely estimate \nits potential ecosystem -level or longitudinal risks and related political, social, and economic impacts. \nGaps between benchmarks and real-world  use of GAI systems may likely be exacerbated due to prompt \nsensitivity and broad heterogeneity of contexts of use.  \nA.1.5.  Structured Public Feedback  \nStructured public feedback can be used to evaluate whether GAI systems are performing as intended and to calibrate and verify traditional measurement methods. Examples of structured feedback include, \nbut are not limited to:  \n• Participatory Engagement Methods : Methods used to solicit feedback from civil society groups, \naffected communities, and users, including focus groups, small user studies, and surveys.  \n• Field Testing : Methods used to determine how people interact with, consume, use, and make \nsense of AI -generated information, and subsequent actions and effects, including UX, usability, \nand other structured, randomized experiments.  \n• AI Red -teaming:  A structured testing exercise\n used to probe an AI system to find flaws and \nvulnerabilities such as inaccurate, harmful, or discriminatory outputs, often in a controlled \nenvironment and in collaboration with system developers.  \nInformation gathered from structured public feedback can inform design, implementation, deployment \napproval , maintenance, or decommissioning decisions. Results and insights gleaned from these exercises'
 '48 • Data protection  \n• Data retention  \n• Consistency in use of defining key terms  \n• Decommissioning  \n• Discouraging anonymous use  \n• Education  \n• Impact assessments  \n• Incident response  \n• Monitoring  \n• Opt-outs  • Risk-based controls  \n• Risk mapping and measurement  \n• Science -backed TEVV practices  \n• Secure software development practices  \n• Stakeholder engagement  \n• Synthetic content detection and \nlabeling tools and techniques  \n• Whistleblower protections  \n• Workforce diversity and \ninterdisciplinary teams\nEstablishing acceptable use policies and guidance for the use of GAI in formal human- AI teaming settings \nas well as different levels of human -AI configurations can help to decrease risks arising from misuse, \nabuse, inappropriate repurpose, and misalignment  between systems and users. These practices are just \none example of adapting existing governance protocols for GAI contexts.  \nA.1.3.  Third -Party Considerations  \nOrganizations may seek to acquire, embed, incorporate, or use open- source or proprietary third- party \nGAI models, systems, or generated data for various applications across an enterprise. Use of these GAI \ntools and inputs has implications for all functions of the organization – including but not limited to \nacquisition, human resources, legal, compliance, and IT services – regardl ess of whether they are carried \nout by employees or third parties. Many of the actions cited above are relevant and options for \naddressing third- party considerations.  \nThird party GAI integrations may give rise to increased intellectual property, data privacy, or information \nsecurity risks, pointing to the need for clear guidelines for transparency  and risk management regarding \nthe collection and use of third -party data for model inputs. Organizations may consider varying risk \ncontrols for foundation models, fine -tuned models, and embedded tools, enhanced processes for \ninteracting with external GAI technologies or service providers. Organizations can apply standard or existing risk controls and processes to proprietary or open -source  GAI technologies, data, and third -party \nservice providers, including acquisition and procurement due diligence, requests for software bills of materials (SBOMs), application of service level agreements (SLAs), and statement on standards for \nattestation en gagement (SSAE) reports  to help with third -party transparency and risk management for \nGAI systems.  \nA.1.4.  \nPre-Deployment Testing  \nOverview \nThe diverse ways and contexts in which GAI systems may be developed, used, and repurposed \ncomplicates risk mapping and pre -deployment measurement efforts. Robust test, evaluation, validation, \nand verification (TEVV) processes can be iteratively applied – and documented – in early stages of the AI \nlifecycle and informed by representative AI Actors  (see Figure 3 of the AI RMF ). Until new and rigorous']","Structured human feedback exercises, such as GAI red-teaming, can significantly benefit GAI risk measurement and management in several ways:

1. **Identifying Flaws and Vulnerabilities**: GAI red-teaming involves testing AI systems in controlled environments to uncover potential flaws, inaccuracies, or harmful outputs. This proactive approach helps organizations identify risks before the systems are deployed in real-world scenarios (Source: [document name], p. 42).

2. **Diverse Perspectives**: By including a demographically and interdisciplinary diverse team in the red-teaming process, organizations can gain insights from various backgrounds and experiences. This diversity can help identify biases and vulnerabilities that may not be apparent to a homogenous group of developers (Source: [document name], p. 42).

3. **Stress Testing Safeguards**: Red-teaming exercises can stress test the safeguards and controls that organizations have in place. By simulating adverse conditions or misuse, organizations can evaluate the effectiveness of their risk management strategies and make necessary adjustments (Source: [document name], p. 42).

4. **Informed Decision-Making**: The insights gained from red-teaming can inform governance and decision-making processes. Organizations can use the findings to update policies, improve procedures, and enhance AI risk management efforts, ensuring that they are better prepared for potential challenges (Source: [document name], p. 42).

5. **Enhancing User Experience**: By understanding how users interact with AI systems and the potential risks they face, organizations can improve the overall user experience. This can lead to more effective and safer AI applications that align with user needs and expectations (Source: [document name], p. 42).

6. **Feedback Loop for Continuous Improvement**: Red-teaming creates a feedback loop that allows organizations to continuously assess and improve their AI systems. By regularly conducting these exercises, organizations can stay ahead of emerging risks and adapt to changing contexts (Source: [document name], p. 42).

In summary, structured human feedback exercises like GAI red-teaming are essential for identifying risks, enhancing system safety, and ensuring that AI technologies are deployed responsibly and effectively.","Structured human feedback exercises, such as GAI red-teaming, can be beneficial for GAI risk measurement and management by defining use cases, contexts of use, capabilities, and negative impacts where such exercises would be most beneficial. These exercises help in monitoring and improving outputs, evaluating the quality and integrity of data used in training, and tracking and documenting risks or opportunities related to GAI risks that cannot be measured quantitatively. Additionally, seeking active and direct feedback from affected communities through red-teaming can enhance information integrity and help in identifying harmful bias and homogenization in AI systems.",1.0,0.98830909998763,1.0,0.999999999975,0.8179111953348778
How do measurement gaps between laboratory and real-world settings impact the assessment of GAI systems in the context of pre-deployment testing?,"['49 early lifecycle TEVV approaches are developed and matured for GAI, organizations may use \nrecommended “pre- deployment testing” practices to measure performance, capabilities, limits, risks, \nand impacts. This section describes risk measurement and estimation as part of pre -deployment TEVV, \nand examines the state of play for pre -deployment testing methodologies.  \nLimitations of Current Pre -deployment Test Approaches  \nCurrently available pre -deployment TEVV processes used for GAI applications may be inadequate, non-\nsystematically applied, or fail to reflect or mismatched to deployment contexts. For example, the \nanecdotal testing of GAI system capabilities through video games or standardized tests designed for \nhumans (e.g., intelligence tests, professional licensing exams) does not guarantee GAI system validity or \nreliability in those domains. Similarly, jailbreaking or prompt  engineering tests may not systematically \nasse ss validity or reliability risks.  \nMeasurement gaps can arise from mismatches between laboratory and real -world settings. Current \ntesting approaches often remain focused on laboratory conditions or restricted to benchmark test \ndatasets and in silico techniques that may not extrapolate well to —or directly assess GAI impacts in real -\nworld conditions. For example, current measurement gaps for GAI make it difficult to precisely estimate \nits potential ecosystem -level or longitudinal risks and related political, social, and economic impacts. \nGaps between benchmarks and real-world  use of GAI systems may likely be exacerbated due to prompt \nsensitivity and broad heterogeneity of contexts of use.  \nA.1.5.  Structured Public Feedback  \nStructured public feedback can be used to evaluate whether GAI systems are performing as intended and to calibrate and verify traditional measurement methods. Examples of structured feedback include, \nbut are not limited to:  \n• Participatory Engagement Methods : Methods used to solicit feedback from civil society groups, \naffected communities, and users, including focus groups, small user studies, and surveys.  \n• Field Testing : Methods used to determine how people interact with, consume, use, and make \nsense of AI -generated information, and subsequent actions and effects, including UX, usability, \nand other structured, randomized experiments.  \n• AI Red -teaming:  A structured testing exercise\n used to probe an AI system to find flaws and \nvulnerabilities such as inaccurate, harmful, or discriminatory outputs, often in a controlled \nenvironment and in collaboration with system developers.  \nInformation gathered from structured public feedback can inform design, implementation, deployment \napproval , maintenance, or decommissioning decisions. Results and insights gleaned from these exercises'
 '30 MEASURE 2.2:  Evaluations involving human subjects meet applicable requirements (including human subject protection) and are \nrepresentative of the relevant population.  \nAction ID  Suggested Action  GAI Risks  \nMS-2.2-001 Assess and manage statistical biases related to GAI content provenance through \ntechniques such as re -sampling, re -weighting, or adversarial training.  Information Integrity ; Information \nSecurity ; Harmful Bias and \nHomogenization  \nMS-2.2-002 Document how content provenance data  is tracked  and how that data interact s \nwith  privacy and security . Consider : Anonymiz ing data to protect the privacy of \nhuman subjects; Leverag ing privacy output filters; Remov ing any personally \nidentifiable information (PII) to prevent potential harm or misuse.  Data Privacy ; Human AI \nConfiguration; Information \nIntegrity ; Information Security ; \nDangerous , Violent, or Hateful \nContent  \nMS-2.2-0 03 Provide human subjects with options to withdraw participation or revoke their \nconsent for present or future use of their data in GAI applications .  Data Privacy ; Human -AI \nConfiguration; Information \nIntegrity  \nMS-2.2-0 04 Use techniques such as anonymization , differential privacy  or other  privacy -\nenhancing technologies to minimize the risks associated with linking AI -generated \ncontent back to individual human subjects.  Data Privacy ; Human -AI \nConfiguration  \nAI Actor Tasks:  AI Development, Human Factors, TEVV  \n \nMEASURE 2.3:  AI system performance or assurance criteria are measured qualitatively or quantitatively and demonstrated for \nconditions similar to deployment setting(s). Measures are documented.  \nAction ID  Suggested Action  GAI Risks  \nMS-2.3-001 Consider baseline model performance on suites of benchmarks when selecting a \nmodel for fine tuning  or enhancement with retrieval- augmented generation. Information Security ; \nConfabulation  \nMS-2.3-002 Evaluate claims of model capabilities using empirically validated methods.  Confabulation ; Information \nSecurity  \nMS-2.3-003 Share results of pre -deployment testing with relevant GAI Actors , such as those \nwith system release approval authority.  Human -AI Configuration'
 '48 • Data protection  \n• Data retention  \n• Consistency in use of defining key terms  \n• Decommissioning  \n• Discouraging anonymous use  \n• Education  \n• Impact assessments  \n• Incident response  \n• Monitoring  \n• Opt-outs  • Risk-based controls  \n• Risk mapping and measurement  \n• Science -backed TEVV practices  \n• Secure software development practices  \n• Stakeholder engagement  \n• Synthetic content detection and \nlabeling tools and techniques  \n• Whistleblower protections  \n• Workforce diversity and \ninterdisciplinary teams\nEstablishing acceptable use policies and guidance for the use of GAI in formal human- AI teaming settings \nas well as different levels of human -AI configurations can help to decrease risks arising from misuse, \nabuse, inappropriate repurpose, and misalignment  between systems and users. These practices are just \none example of adapting existing governance protocols for GAI contexts.  \nA.1.3.  Third -Party Considerations  \nOrganizations may seek to acquire, embed, incorporate, or use open- source or proprietary third- party \nGAI models, systems, or generated data for various applications across an enterprise. Use of these GAI \ntools and inputs has implications for all functions of the organization – including but not limited to \nacquisition, human resources, legal, compliance, and IT services – regardl ess of whether they are carried \nout by employees or third parties. Many of the actions cited above are relevant and options for \naddressing third- party considerations.  \nThird party GAI integrations may give rise to increased intellectual property, data privacy, or information \nsecurity risks, pointing to the need for clear guidelines for transparency  and risk management regarding \nthe collection and use of third -party data for model inputs. Organizations may consider varying risk \ncontrols for foundation models, fine -tuned models, and embedded tools, enhanced processes for \ninteracting with external GAI technologies or service providers. Organizations can apply standard or existing risk controls and processes to proprietary or open -source  GAI technologies, data, and third -party \nservice providers, including acquisition and procurement due diligence, requests for software bills of materials (SBOMs), application of service level agreements (SLAs), and statement on standards for \nattestation en gagement (SSAE) reports  to help with third -party transparency and risk management for \nGAI systems.  \nA.1.4.  \nPre-Deployment Testing  \nOverview \nThe diverse ways and contexts in which GAI systems may be developed, used, and repurposed \ncomplicates risk mapping and pre -deployment measurement efforts. Robust test, evaluation, validation, \nand verification (TEVV) processes can be iteratively applied – and documented – in early stages of the AI \nlifecycle and informed by representative AI Actors  (see Figure 3 of the AI RMF ). Until new and rigorous'
 '31 MS-2.3-004 Utilize a purpose -built testing environment such as NIST Dioptra to empirically \nevaluate GAI trustworthy characteristics.  CBRN Information or Capabilities ; \nData Privacy ; Confabulation ; \nInformation Integrity ; Information \nSecurity ; Dangerous , Violent, or \nHateful Content ; Harmful Bias and \nHomogenization  \nAI Actor Tasks:  AI Deployment, TEVV  \n \nMEASURE 2.5:  The AI system to be deployed is demonstrated to be valid and reliable. Limitations of the generalizability beyond the \nconditions under which the technology was developed are documented.  \nAction ID  Suggested Action  Risks  \nMS-2.5-001 Avoid extrapolating GAI system performance or capabilities from narrow, non -\nsystematic, and anecdotal assessments.  Human -AI Configuration ; \nConfabulation  \nMS-2.5-002 Document the extent to which human domain knowledge is employed to \nimprove GAI system performance, via, e.g., RLHF, fine -tuning, retrieval-\naugmented generation, content moderation, business rules.  Human -AI Configuration  \nMS-2.5-003 Review and verify sources and citations in GAI system outputs during pre -\ndeployment risk measurement and ongoing monitoring activities.  Confabulation  \nMS-2.5-004 Track and document instances of anthropomorphization  (e.g., human images, \nmentions of human feelings, cyborg imagery or motifs) in GAI system interfaces.  Human -AI Configuration  \nMS-2.5-0 05 Verify GAI system training data and TEVV data provenance, and that fine -tuning  \nor retrieval- augmented generation data is grounded.  Information Integrity  \nMS-2.5-0 06 Regularly review security and safety guardrails, especially if the GAI system is \nbeing operated in novel circumstances. This includes reviewing reasons why the \nGAI system was initially assessed as being safe to deploy.  Information Security ; Dangerous , \nViolent, or Hateful  Content  \nAI Actor Tasks:  Domain Experts, TEVV']","Measurement gaps between laboratory and real-world settings can significantly impact the assessment of GAI systems during pre-deployment testing in several ways:

1. **Limited Validity**: Testing conducted in controlled laboratory environments may not accurately reflect the complexities and variabilities of real-world situations. This can lead to a false sense of confidence in the GAI system's performance, as results may not generalize well to actual deployment contexts (Source: [document name], p. 30).

2. **Contextual Mismatches**: Laboratory tests often focus on specific benchmarks or datasets that may not capture the diverse scenarios in which GAI systems will be used. This mismatch can result in an incomplete understanding of how the system will behave in different environments or with varied user interactions (Source: [document name], p. 30).

3. **Underestimating Risks**: The controlled nature of laboratory testing may overlook potential risks that could arise in real-world applications, such as biases, ethical concerns, or unintended consequences. This can lead to inadequate risk management strategies and unpreparedness for challenges that emerge post-deployment (Source: [document name], p. 30).

4. **Prompt Sensitivity**: GAI systems can be highly sensitive to the prompts they receive. Variations in how users interact with the system in real life may not be captured during laboratory testing, leading to discrepancies in performance and reliability once the system is deployed (Source: [document name], p. 30).

5. **Ecosystem-Level Impacts**: Current measurement approaches may struggle to assess the broader political, social, and economic impacts of GAI systems. Without understanding these ecosystem-level risks, organizations may fail to address significant implications of their technology (Source: [document name], p. 30).

In summary, these measurement gaps highlight the need for more robust and contextually relevant pre-deployment testing methodologies that can better predict how GAI systems will perform in real-world settings. This can help organizations mitigate risks and ensure safer, more effective AI applications.","Measurement gaps between laboratory and real-world settings can impact the assessment of GAI systems in the context of pre-deployment testing by limiting the extrapolation of results from laboratory conditions to real-world scenarios. Current testing approaches often focus on benchmark test datasets and in silico techniques that may not accurately assess the impacts of GAI systems in real-world conditions. This can make it difficult to estimate the ecosystem-level or longitudinal risks associated with GAI deployment, as well as the political, social, and economic impacts. Additionally, the prompt sensitivity and broad heterogeneity of real-world contexts of use can exacerbate the gaps between benchmarks and actual GAI system performance.",1.0,0.9887518614487671,1.0,0.999999999975,0.8238132283366892
How should data collection and use-case scope limits be determined and implemented in automated systems to prevent 'mission creep'?,"['DATA PRIVACY \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nTraditional terms of service—the block of text that the public is accustomed to clicking through when using a web -\nsite or digital app—are not an adequate mechanism for protecting privacy. The American public should be protect -\ned via built-in privacy protections, data minimization, use and collection limitations, and transparency, in addition \nto being entitled to clear mechanisms to control access to and use of their data—including their metadata—in a proactive, informed, and ongoing way. Any automated system collecting, using, sharing, or storing personal data should meet these expectations. \nProtect privacy by design and by default \nPrivacy by design and by default. Automated systems should be designed and built with privacy protect -\ned by default. Privacy risks should be assessed throughout the development life cycle, including privacy risks from reidentification, and appropriate technical and policy mitigation measures should be implemented. This includes potential harms to those who are not users of the automated system, but who may be harmed by inferred data, purposeful privacy violations, or community surveillance or other community harms. Data collection should be minimized and clearly communicated to the people whose data is collected. Data should only be collected or used for the purposes of training or testing machine learning models if such collection and use is legal and consistent with the expectations of the people whose data is collected. User experience research should be conducted to confirm that people understand what data is being collected about them and how it will be used, and that this collection matches their expectations and desires. \nData collection and use-case scope limits. Data collection should be limited in scope, with specific, \nnarrow identified goals, to avoid ""mission creep.""  Anticipated data collection should be determined to be strictly necessary to the identified goals and should be minimized as much as possible. Data collected based on these identified goals and for a specific context should not be used in a different context without assessing for new privacy risks and implementing appropriate mitigation measures, which may include express consent. Clear timelines for data retention should be established, with data deleted as soon as possible in accordance with legal or policy-based limitations. Determined data retention timelines should be documented and justi\n-\nfied. \nRisk identification and mitigation. Entities that collect, use, share, or store sensitive data should attempt to proactively identify harms and seek to manage them so as to avoid, mitigate, and respond appropri\n-'
 'deployment context, the specific identified goals of any data collection, and the assessment done to ensure \nonly the minimum required data is collected. It should also include documentation about the scope limit \nassessments, including data retention timelines and associated justification, and an assessment of the \nimpact of surveillance or data collection on rights, opportunities, and access. Where possible, this \nassessment of the impact of surveillance should be done by an independent party. Reporting should be \nprovided in a clear and machine-readable manner.  \n35'
 'deployment should be relevant, of high quality, and tailored to the task at hand. Relevancy should be \nestablished based on research-backed demonstration of the causal influence of the data to the specific use case \nor justified more generally based on a reasonable expectation of usefulness in the domain and/or for the \nsystem design or ongoing development. Relevance of data should not be established solely by appealing to \nits historical connection to the outcome. High quality and tailored data should be representative of the task at \nhand and errors from data entry or other sources should be measured and limited. Any data used as the target \nof a prediction process should receive particular attention to the quality and validity of the predicted outcome \nor label to ensure the goal of the automated system is appropriately identified and measured. Additionally , \njustification should be documented for each data attribute and source to explain why it is appropriate to use \nthat data to inform the results of the automated system and why such use will not violate any applicable laws. \nIn cases of high-dimensional and/or derived attributes, such justifications can be provided as overall \ndescriptions of the attribute generation process and appropriateness. \n19'
 'Testing. Systems should undergo extensive testing before deployment. This testing should follow domain-specific best practices, when available, for ensuring the technology will work in its real-world context. Such testing should take into account both the specific technology used and the roles of any human operators or reviewers who impact system outcomes or effectiveness; testing should include both automated systems testing and human-led (manual) testing. Testing conditions should mirror as closely as possible the conditions in which the system will be deployed, and new testing may be required for each deployment to account for material differences in conditions from one deployment to another. Following testing, system performance should be compared with the in-place, potentially human-driven, status quo procedures, with existing human performance considered as a performance baseline for the algorithm to meet pre-deployment, and as a lifecycle minimum performance standard. Decision possibilities resulting from performance testing should include the possibility of not deploying the system. \nRisk identification and mitigation. Before deployment, and in a proactive and ongoing manner, poten -\ntial risks of the automated system should be identified and mitigated. Identified risks should focus on the potential for meaningful impact on people’s rights, opportunities, or access and include those to impacted communities that may not be direct users of the automated system, risks resulting from purposeful misuse of the system, and other concerns identified via the consultation process. Assessment and, where possible, mea\n-\nsurement of the impact of risks should be included and balanced such that high impact risks receive attention and mitigation proportionate with those impacts. Automated systems with the intended purpose of violating the safety of others should not be developed or used; systems with such safety violations as identified unin\n-\ntended consequences should not be used until the risk can be mitigated. Ongoing risk mitigation may necessi -\ntate rollback or significant modification to a launched automated system. \n18']","To determine and implement data collection and use-case scope limits in automated systems effectively, organizations should follow these key steps:

1. **Define Specific Goals**: Clearly identify the specific, narrow goals for which data is being collected. This helps ensure that data collection is directly aligned with the intended purpose and prevents unnecessary data gathering (Source: [document name], p. 35).

2. **Conduct a Necessity Assessment**: Evaluate whether the anticipated data collection is strictly necessary to achieve the identified goals. This assessment should focus on minimizing data collection to only what is essential for the task at hand (Source: [document name], p. 35).

3. **Limit Contextual Use**: Ensure that data collected for a specific context is not used in different contexts without a thorough assessment of new privacy risks. If a new use case arises, organizations should evaluate the implications and obtain express consent if necessary (Source: [document name], p. 35).

4. **Establish Clear Data Retention Timelines**: Set and document clear timelines for how long data will be retained. Data should be deleted as soon as it is no longer needed for the specified purpose, in accordance with legal or policy-based limitations (Source: [document name], p. 35).

5. **Implement Ongoing Monitoring**: Regularly review and monitor data collection practices to ensure compliance with established limits. This includes assessing whether the data being collected continues to align with the original goals and making adjustments as needed (Source: [document name], p. 35).

6. **Engage Stakeholders**: Involve relevant stakeholders, including users and privacy experts, in the process of defining data collection limits. Their input can provide valuable insights into community expectations and help identify potential risks (Source: [document name], p. 35).

7. **Document Justifications**: Maintain thorough documentation that justifies the scope of data collection and use. This should include the rationale for why specific data is necessary and how it aligns with the defined goals (Source: [document name], p. 35).

By following these steps, organizations can effectively prevent 'mission creep'—the gradual expansion of data collection beyond its original purpose—thereby enhancing privacy protections and ensuring responsible use of automated systems.","Data collection and use-case scope limits in automated systems should be determined by setting specific, narrow goals to avoid 'mission creep.' Anticipated data collection should be strictly necessary for the identified goals and minimized as much as possible. Data collected for a specific context should not be used in a different context without assessing new privacy risks and implementing appropriate mitigation measures, which may include obtaining express consent. Clear timelines for data retention should be established, with data deleted as soon as possible in accordance with legal or policy-based limitations. The determined data retention timelines should be documented and justified.",1.0,0.9341377229848843,1.0,0.999999999975,0.5837907969991202
What action did the Federal Trade Commission take against Kochava regarding the sale of sensitive location tracking data?,"['• A device originally developed to help people track and find lost items has been used as a tool by stalkers to trackvictims’ locations in violation of their privacy and safet y. The device manufacturer took steps after release to\nprotect people from unwanted tracking by alerting people on their phones when a device is found to be movingwith them over time and also by having the device make an occasional noise, but not all phones are ableto receive the notification and the devices remain a safety concern due to their misuse.\n8 \n• An algorithm used to deploy police was found to repeatedly send police to neighborhoods they regularly visit,even if those neighborhoods were not the ones with the highest crime rates. These incorrect crime predictionswere the result of a feedback loop generated from the reuse of data from previous arrests and algorithmpredictions.\n9\n16'
 'shared, or made public as part of data brokerage or other agreements. Sensitive data includes data that can be \nused to infer sensitive information; even systems that are not directly marketed as sensitive domain technologies \nare expected to keep sensitive data private. Access to such data should be limited based on necessity and based \non a principle of local control, such that those individuals closest to the data subject have more access while \nthose who are less proximate do not (e.g., a teacher has access to their students’ daily progress data while a \nsuperintendent does not). \nReporting.  In addition to the reporting on data privacy (as listed above for non-sensitive data), entities devel-\noping technologies related to a sensitive domain and those collecting, using, storing, or sharing sensitive data \nshould, whenever appropriate, regularly provide public reports describing: any data security lapses or breaches \nthat resulted in sensitive data leaks; the numbe r, type, and outcomes of ethical pre-reviews undertaken; a \ndescription of any data sold, shared, or made public, and how that data was assessed to determine it did not pres-\nent a sensitive data risk; and ongoing risk identification and management procedures, and any mitigation added \nbased on these procedures. Reporting should be provided in a clear and machine-readable manne r. \n38'
 'https://www.theverge.com/2019/2/7/18211890/social-media-life-insurance-new-york-algorithms-big-data-discrimination-online-records\n68'
 'NOTICE & \nEXPLANATION \nHOW THESE PRINCIPLES CAN MOVE INTO PRACTICE\nReal-life examples of how these principles can become reality, through laws, policies, and practical \ntechnical and sociotechnical approaches to protecting rights, opportunities, and access. \nPeople in Illinois are given written notice by the private sector if their biometric informa-\ntion is used . The Biometric Information Privacy Act enacted by the state contains a number of provisions \nconcerning the use of individual biometric data and identifiers. Included among them is a provision that no private \nentity may ""collect, capture, purchase, receive through trade, or otherwise obtain"" such information about an \nindividual, unless written notice is provided to that individual or their legally appointed representative. 87\nMajor technology  companies are piloting new ways to communicate with the public about \ntheir automated  technologies. For example, a collection of non-profit organizations and companies have \nworked together to develop a framework that defines operational approaches to transparency for machine \nlearning systems.88 This framework, and others like it,89 inform the public about the use of these tools, going \nbeyond simple notice to include reporting elements such as safety evaluations, disparity assessments, and \nexplanations of how the systems work. \nLenders are required by federal law to notify consumers about certain decisions made about \nthem. Both the Fair Credit Reporting Act and the Equal Credit Opportunity Act require in certain circumstances \nthat consumers who are denied credit receive ""adverse action"" notices. Anyone who relies on the information in a \ncredit report to deny a consumer credit must, under the Fair Credit Reporting Act, provide an ""adverse action"" \nnotice to the consumer, which includes ""notice of the reasons a creditor took adverse action on the application \nor on an existing credit account.""90 In addition, under the risk-based pricing rule,91 lenders must either inform \nborrowers of their credit score, or else tell consumers when ""they are getting worse terms because of \ninformation in their credit report."" The CFPB has also asserted that ""[t]he law gives every applicant the right to \na specific explanation if their application for credit was denied, and that right is not diminished simply because \na company uses a complex algorithm  that it doesn\'t understand.""92 Such explanations illustrate a shared value \nthat certain decisions need to be explained. \nA California law  requires  that warehouse employees are  provided with  notice and  explana-\ntion about quotas, potentially facilitated by automated systems, that apply to them. Warehous-\ning employers in California that use quota systems (often facilitated by algorithmic monitoring systems) are \nrequired to provide employees with a written description of each quota that applies to the employee, including']","The Federal Trade Commission (FTC) took action against Kochava, a data broker, for allegedly selling sensitive location tracking data that could be used to identify individuals' movements, including visits to sensitive locations such as medical facilities, places of worship, and domestic violence shelters. The FTC's complaint highlighted that this practice posed significant privacy risks and could lead to harmful consequences for individuals whose data was being sold. The agency argued that Kochava's actions violated the FTC Act by engaging in unfair practices that could harm consumers (Source: [document name], p. 12).

The FTC's action underscores the growing concern over the sale and use of sensitive data, particularly in the context of location tracking, and reflects a broader push for stronger regulations to protect consumer privacy in the digital age.","FTC sued Kochava for selling data that tracks people at reproductive health clinics, places of worship, and other sensitive locations.",0.0,0.9395589918551167,0.0,0.0,0.3780365700968873
How should explanatory mechanisms be built into system design to ensure full behavior transparency in high-risk settings?,"[""NOTICE & \nEXPLANATION \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nTailored to the level of risk. An assessment should be done to determine the level of risk of the auto -\nmated system. In settings where the consequences are high as determined by a risk assessment, or extensive \noversight is expected (e.g., in criminal justice or some public sector settings), explanatory mechanisms should be built into the system design so that the system’s full behavior can be explained in advance (i.e., only fully transparent models should be used), rather than as an after-the-decision interpretation. In other settings, the extent of explanation provided should be tailored to the risk level. \nValid. The explanation provided by a system should accurately reflect the factors and the influences that led \nto a particular decision, and should be meaningful for the particular customization based on purpose, target, and level of risk. While approximation and simplification may be necessary for the system to succeed based on the explanatory purpose and target of the explanation, or to account for the risk of fraud or other concerns related to revealing decision-making information, such simplifications should be done in a scientifically supportable way. Where appropriate based on the explanatory system, error ranges for the explanation should be calculated and included in the explanation, with the choice of presentation of such information balanced with usability and overall interface complexity concerns. \nDemonstrate protections for notice and explanation \nReporting. Summary reporting should document the determinations made based on the above consider -\nations, including: the responsible entities for accountability purposes; the goal and use cases for the system, identified users, and impacted populations; the assessment of notice clarity and timeliness; the assessment of the explanation's validity and accessibility; the assessment of the level of risk; and the account and assessment of how explanations are tailored, including to the purpose, the recipient of the explanation, and the level of risk. Individualized profile information should be made readily available to the greatest extent possible that includes explanations for any system impacts or inferences. Reporting should be provided in a clear plain language and machine-readable manner. \n44""
 '18 GOVERN 3.2:  Policies and procedures are in place to define and differentiate roles and responsibilities for human -AI configurations \nand oversight of AI systems.  \nAction ID  Suggested Action  GAI Risks  \nGV-3.2-001 Policies are in place to bolster oversight of GAI systems with independent \nevaluations  or assessments of GAI models or systems  where the type and \nrobustness of evaluations are proportional to the identified risks.  CBRN Information or Capabilities ; \nHarmful Bias and Homogenization  \nGV-3.2-002 Consider adjustment of organizational roles and components across lifecycle \nstages of large or complex GAI systems, including: Test and evaluation, validation, \nand red- teaming of GAI systems; GAI content moderation; GAI system \ndevelopment and engineering; Increased accessibility of GAI tools, interfaces, and \nsystems, Incident response and containment.  Human -AI Configuration ; \nInformation Security ; Harmful Bias \nand Homogenization  \nGV-3.2-003 Define acceptable use policies for GAI interfaces, modalities, and human -AI \nconfigurations  (i.e., for chatbots and decision -making tasks) , including  criteria for \nthe kinds of queries GAI applications should refuse to respond to.  Human -AI Configuration  \nGV-3.2-004 Establish policies for user feedback mechanisms for GAI systems  which include \nthorough instructions and any mechanisms for recourse . Human -AI Configuration   \nGV-3.2-005 Engage in threat modeling to anticipate potential risks from GAI systems.  CBRN Information or Capabilities ; \nInformation Security  \nAI Actors: AI Design  \n \nGOVERN 4.1:  Organizational policies and practices are in place to foster a critical thinking and safety -first mindset in the design, \ndevelopment, deployment, and uses of AI systems to minimize potential negative impacts.  \nAction ID  Suggested Action  GAI Risks  \nGV-4.1-001 Establish policies and procedures that address continual improvement processes \nfor GAI risk measurement . Address general risks associated with a lack of \nexplainability and transparency in GAI systems by using ample documentation and \ntechniques such as: application of gradient -based attributions, occlusion/term \nreduction, counterfactual prompts and prompt eng ineering, and analysis of \nembeddings; Assess and update risk measurement approaches at regular cadences.  Confabulation  \nGV-4.1-002 Establish policies, procedures, and processes detailing risk measurement in \ncontext of use with standardized measurement protocols and structured public feedback exercises such as AI red -teaming or independent external evaluations . CBRN Information and Capability ; \nValue Chain and Component Integration'
 'SAFE AND EFFECTIVE \nSYSTEMS \nWHY THIS PRINCIPLE IS IMPORTANT\nThis section provides a brief summary of the problems which the principle seeks to address and protect \nagainst, including illustrative examples. \nWhile technologies are being deployed to solve problems across a wide array of issues, our reliance on technology can \nalso lead to its use in situations where it has not yet been proven to work—either at all or within an acceptable range of error. In other cases, technologies do not work as intended or as promised, causing substantial and unjustified harm. Automated systems sometimes rely on data from other systems, including historical data, allowing irrelevant informa\n-\ntion from past decisions to infect decision-making in unrelated situations.  In some cases, technologies are purposeful -\nly designed to violate the safety of others, such as technologies designed to facilitate stalking; in other cases, intended or unintended uses lead to unintended harms. \nMany of the harms resulting from these technologies are preventable, and actions are already being taken to protect the public. Some companies have put in place safeguards that have prevented harm from occurring by ensuring that key development decisions are vetted by an ethics review; others have identified and mitigated harms found through pre-deployment testing and ongoing monitoring processes. Governments at all levels have existing public consulta\n-\ntion processes that may be applied when considering the use of new automated systems, and existing product develop -\nment and testing practices already protect the American public from many potential harms. \nStill, these kinds of practices are deployed too rarely and unevenly. Expanded, proactive protections could build on these existing practices, increase confidence in the use of automated systems, and protect the American public. Inno\n-\nvators deserve clear rules of the road that allow new ideas to flourish, and the American public deserves protections from unsafe outcomes. All can benefit from assurances that automated systems will be designed, tested, and consis\n-\ntently confirmed to work as intended, and that they will be proactively protected from foreseeable unintended harm -\nful outcomes. \n• A proprietary model was developed to predict the likelihood of sepsis in hospitalized patients and was imple -\nmented at hundreds of hospitals around the countr y. An independent study showed that the model predictions\nunderperformed relative to the designer’s claims while also causing ‘alert fatigue’ by falsely alertinglikelihood of sepsis.\n6\n• On social media, Black people who quote and criticize racist messages have had their own speech silenced whena platform’s automated moderation system failed to distinguish this “counter speech” (or other critiqueand journalism) from the original hateful messages to which such speech responded.\n7'
 'SAFE AND EFFECTIVE \nSYSTEMS \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nIn order to ensure that an automated system is safe and effective, it should include safeguards to protect the \npublic from harm in a proactive and ongoing manner; avoid use of data inappropriate for or irrelevant to the task at hand, including reuse that could cause compounded harm; and demonstrate the safety and effectiveness of the system. These expectations are explained below. \nProtect the public from harm in a proactive and ongoing manner \nConsultation. The public should be consulted in the design, implementation, deployment, acquisition, and \nmaintenance phases of automated system development, with emphasis on early-stage consultation before a system is introduced or a large change implemented. This consultation should directly engage diverse impact\n-\ned communities to consider concerns and risks that may be unique to those communities, or disproportionate -\nly prevalent or severe for them. The extent of this engagement and the form of outreach to relevant stakehold -\ners may differ depending on the specific automated system and development phase, but should include subject matter, sector-specific, and context-specific experts as well as experts on potential impacts such as civil rights, civil liberties, and privacy experts. For private sector applications, consultations before product launch may need to be confidential. Government applications, particularly law enforcement applications or applications that raise national security considerations, may require confidential or limited engagement based on system sensitivities and preexisting oversight laws and structures. Concerns raised in this consultation should be documented, and the automated system developers were proposing to create, use, or deploy should be reconsidered based on this feedback.']","Integrating explanatory mechanisms into system design to ensure complete behavior transparency in high-risk environments involves several key strategies:

1. **Risk Assessment**: Begin with a thorough risk assessment to identify the potential consequences of automated decisions in high-risk settings, such as criminal justice or healthcare. This assessment will guide the level of transparency required (Source: [document name], p. 44).

2. **Use of Fully Transparent Models**: In high-risk environments, prioritize the use of fully transparent models that allow stakeholders to understand how decisions are made. This means opting for algorithms that provide clear, interpretable outputs rather than complex black-box models (Source: [document name], p. 44).

3. **Explanatory Interfaces**: Design user interfaces that present explanations of the system's decisions in a clear and accessible manner. This could include visualizations, summaries, or step-by-step breakdowns of how a particular decision was reached (Source: [document name], p. 44).

4. **Error Range Calculations**: Include error ranges in the explanations provided by the system. This helps users understand the potential variability in the system's outputs and the confidence level associated with specific decisions (Source: [document name], p. 44).

5. **Documentation and Reporting**: Maintain comprehensive documentation that outlines the decision-making processes, the factors considered, and the rationale behind specific outcomes. This documentation should be made available to relevant stakeholders for accountability purposes (Source: [document name], p. 44).

6. **User Feedback Mechanisms**: Implement mechanisms for users to provide feedback on the explanations they receive. This feedback can help improve the clarity and relevance of the explanations over time (Source: [document name], p. 44).

7. **Continuous Monitoring and Updates**: Regularly monitor the system's performance and the effectiveness of the explanatory mechanisms. Use this information to make iterative improvements to both the system and the explanations provided (Source: [document name], p. 44).

By incorporating these strategies, organizations can enhance transparency and accountability in high-risk environments, ensuring that stakeholders have a clear understanding of how automated systems operate and make decisions. This approach not only builds trust but also helps mitigate potential harms associated with automated decision-making.","In settings where the consequences are high as determined by a risk assessment, or extensive oversight is expected (e.g., in criminal justice or some public sector settings), explanatory mechanisms should be built into the system design so that the system’s full behavior can be explained in advance (i.e., only fully transparent models should be used), rather than as an after-the-decision interpretation.",1.0,0.9301685789729547,1.0,0.999999999975,0.6185943119800098
What are some examples of GAI risks that organizations need to consider in the development and deployment of AI systems?,"['3 the abuse, misuse, and unsafe repurposing by humans (adversarial or not ), and  others result \nfrom interactions between a human and an AI system.   \n• Time  scale: GAI risks  may materialize  abruptly or  across extended periods . Example s include \nimmediate  (and/or prolonged) emotional harm  and potential risks to physical safety  due to the \ndistribution of harmful deepfake images , or the lo ng-term effect of disinformation on soci etal \ntrust in public  institutions . \nThe presence of risks  and where they fall along the dimensions above will vary depending on the \ncharacteristics of the GAI model , system, or use case at hand. These characteristics include but are not \nlimited to GAI model or system architecture, training mechanisms  and libraries , data types  used for \ntraining or fine -tuning , levels of model access or availability of model weights,  and application or use \ncase context. \nOrganizations may choose to tailor how they measure  GAI risks  based on  these characteristics . They may \nadditionally wish to  allocate risk management resources relative to the severity and likelihood of \nnegative impact s, including where and how these risks manifest , and their direct and material impacts  \nharms in the context of GAI use. Mitigations for model or system  level risks may differ from mitigations  \nfor use-case or ecosystem level risks.  \nImportantly, some GAI risks are un known , and are therefore difficult to properly scope or evaluate given  \nthe uncertaint y about potential GAI scale, complexity, and capabilities. Other risks may be known but  \ndifficult to estimate  given the wide range of GAI stakeholders, uses, inputs, and outputs . Challenges  with \nrisk estimation are aggravated by a lack of visibility into GAI training data, and the generally immature \nstate of the science of AI measurement and safety  today . This document focuses on risks for which there \nis an existing empirical evidence base at the time this profile was written ; for example,  speculative risks \nthat may potentially arise in more advanced, future GAI systems are not considered . Future updates may \nincorporate additional risks or provide further details on the risks identified below.  \nTo guide organizations in identifying and managing GAI risks, a set of risks unique to or exacerbated by \nthe development and use of GAI are defined below.5 Each risk is  labeled according to  the outcome , \nobject,  or source of the risk  (i.e., some are risks “to ” a subject  or domain  and others are  risks  “of” or \n“from” an issue or  theme ). These risks provide a lens  through which organizations can frame and execute \nrisk management efforts.  To help streamline risk management efforts, each risk is mapped in Section 3  \n(as well as in tables in Appendix B)  to relevant Trustworthy AI Characteristics  identified in the AI RMF .  \n \n \n5 These risks can be further categorized by organizations depending on their unique approaches to risk definition'
 '22 GV-6.2-003 Establish incident response plans for third -party GAI technologies: Align incident \nresponse plans with impacts enumerated in MAP 5.1; Communicate third- party \nGAI incident response plans to all relevant AI Actors ; Define ownership of GAI \nincident response functions; Rehearse third- party GAI incident response plans at \na regular cadence; Improve incident response plans based on retrospective \nlearning; Review incident response plans for alignment with relevant breach  \nreporting, data protection, data privacy, or  other laws.  Data Privacy ; Human -AI \nConfiguration; Information \nSecurity ; Value Chain and \nComponent Integration; Harmful \nBias and Homogenization  \nGV-6.2-0 04 Establish policies and procedures for continuous monitoring of third -party GAI \nsystems in deployment.  Value Chain and Component \nIntegration  \nGV-6.2-0 05 Establish policies and procedures that address GAI data redundancy, including \nmodel weights and other system artifacts.  Harmful Bias and Homogenization  \nGV-6.2-0 06 Establish policies and procedures to test and manage risks related to rollover and \nfallback technologies for GAI systems, acknowledging that rollover and fallback may include manual processing.  Information Integrity  \nGV-6.2-0 07 Review vendor contracts and avoid arbitrary or capricious termination of critical \nGAI technologies or vendor services and non-standard terms that may amplify or \ndefer liability in unexpected ways and /or contribute to u nauthorized data \ncollection by vendors or third -parties (e.g., secondary data use) . Consider: Clear \nassignment of liability and responsibility for incidents, GAI system changes over time (e.g., fine -tuning, drift, decay); Request: Notification and disclosure for \nserious incidents arising from third -party data and system s; Service Level  \nAgreements (SLAs) in vendor contracts that address incident response, response \ntimes, and availability of critical support.  Human -AI Configuration ; \nInformation Security ; Value Chain \nand Component Integration  \nAI Actor Tasks:  AI Deployment, Operation and Monitoring, TEVV, Third -party entities  \n \nMAP 1.1:  Intended purposes, potentially beneficial uses, context specific laws, norms and expectations, and prospective settings in \nwhich the AI system will be deployed are understood and documented. Considerations include: the specific set or types of user s \nalong  with their expectations; potential positive and negative impacts of system uses to individuals, communities, organizations, \nsociety, and the planet; assumptions and related limitations about AI system purposes, uses, and risks across the development  or \nproduct AI lifecycle; and related TEVV and system metrics.  \nAction ID  Suggested Action  GAI Risks  \nMP-1.1-001 When identifying intended  purposes, c onsider factors such as internal vs. \nexternal use, narrow vs. broad application scope, fine -tuning , and varieties of'
 '47 Appendix A.  Primary GAI Considerations  \nThe following primary considerations were derived as overarching themes from the GAI PWG \nconsultation process. These considerations (Governance, Pre- Deployment Testing, Content Provenance, \nand Incident Disclosure) are relevant  for volun tary use by any organization designing, developing, and \nusing GAI  and also inform the Actions to Manage GAI risks. Information included about the primary \nconsiderations is not exhaustive , but highlights the most relevant topics derived from the GAI PWG.  \nAcknowledgments: These considerations could not have been surfaced without the helpful analysis and \ncontributions from the community and NIST staff GAI PWG leads: George Awad, Luca Belli, Harold Booth, \nMat Heyman, Yoo young Lee, Mark Pryzbocki, Reva Schwartz, Martin Stanley, and Kyra Yee.  \nA.1. Governance  \nA.1.1.  Overview  \nLike any other technology system, governance principles and techniques can be used to manage risks \nrelated to generative AI models, capabilities, and applications. Organizations may choose to apply their \nexisting risk tiering to GAI systems, or they may op t to revis e or update AI system risk levels to address \nthese unique GAI risks. This section describes how organizational governance regimes may be re -\nevaluated and adjusted for GAI contexts. It also addresses third -party considerations for governing across  \nthe AI value chain.  \nA.1.2.  Organizational  Governance  \nGAI opportunities, risks and long- term performance characteristics are typically less well -understood \nthan non- generative AI tools  and may be perceived and acted upon by humans in ways that vary greatly. \nAccordingly, GAI may call for different levels of oversight from AI Actors  or different human- AI \nconfigurations in order to manage their risks effectively. Organizations’ use of GAI systems may also \nwarrant additional human review, tracking and documentation, and greater management oversight.  \nAI technology can produce varied outputs  in multiple modalities and present many classes of user \ninterfaces. This leads to a broader set of AI Actors  interacting with GAI systems for widely differing \napplications and contexts of use. These  can include data labeling and preparation, development of GAI \nmodels, content moderation, code generation and review, text generation and editing, image and video \ngeneration, summarization, search, and chat. These activities can take place within organizational \nsettings or in the public domain.  \nOrganizations can restrict AI applications that cause harm, exceed stated risk tolerances, or that conflict with their tolerances or values. Governance tools and protocols that are applied to other types of AI systems can be applied to GAI systems. These p lans and actions include: \n• Accessibility and reasonable accommodations  \n• AI actor credentials and qualifications  \n• Alignment to organizational values  • Auditing and assessment  \n• Change -management controls'
 '11 value chain  (e.g., data inputs , processing, GAI training, or deployment  environments ), conventional \ncybersecurity practices  may need to adapt or evolve . \nFor instance , prompt  injection  involves  modifying what  input is provided to a  GAI system  so that it  \nbehave s in unintended ways. In direct prompt injections, attackers might craft malicious  prompts  and \ninput them directly to a GAI system , with a variety of  downstream negative consequences to \ninterconnected systems. Indirect prompt injection  attacks occur when  adversaries remotely (i.e., without \na direct interface) exploit LLM -integrated applications by injecting prompts into data likely to be \nretrieved. Security researchers have already demonstrated how indirect prompt injections can exploit \nvulnerabilities by  steal ing proprietary  data  or running malicious code remotely  on a machine. Merely \nquerying  a closed production model can elicit  previously undisclosed information about that model . \nAnother cybersecurity risk to GAI is data poisoning , in which  an adversary compromises  a training \ndataset used by a model to manipulate its output s or operation. Malicious tampering with  data or parts \nof the  model could exacerbate risks associated with GAI system outputs.  \nTrustworthy AI Characteristics:  Privacy Enhanced, Safe , Secure and Resilient , Valid and Reliable  \n2.10. Intellectual Property  \nIntellectual property risks from  GAI systems may arise where the use of copyrighted works  is not a fair \nuse under the fair use doctrine. If a GAI system’s training data included copyrighted material, GAI \noutputs  displaying instances of  training data  memorization  (see Data Privacy above) could infringe on \ncopyright. \nHow GAI relates to copyright, including the status of  generated  content that is similar to but does not \nstrictly copy  w ork protected by copyright , is currently being debated in legal fora.  Similar discussions are \ntaking place  regard ing the use or emulation of personal identit y, likeness , or voice  without permission.  \nTrustworthy AI Characteristics:  Accountable and Transparent, Fair with Harmful Bias Managed, Privacy \nEnhanced  \n2.11. Obscene, Degrading, and/or A busive Content  \nGAI can ease the production of and access to illegal non- consensual intimate imagery  (NCII)  of adult s, \nand/or child sexual abuse material  (CSAM) . GAI- generated obscene, abusive  or degrading content can  \ncreate privacy, psychological and emotional, and even physical harms , and in some cases may be illegal .  \nGenerated explicit or obscene AI content may include highly  realistic “deepfakes” of real individuals, \nincluding children. The spread of this kind of material can have downstream negative consequences : in \nthe context of CSAM, e ven if the generated images do not resemble specific individuals, the prevalence \nof such images c an divert  time and re sources from  efforts  to find real -world victims. Outside of CSAM,']","Organizations need to consider several GAI (Generative AI) risks during the development and deployment of AI systems. Here are some key examples:

1. **Data Privacy Risks**: GAI systems often require large datasets for training, which may include sensitive personal information. If this data is not handled properly, it can lead to privacy violations and unauthorized access to personal information (Source: [document name], p. 22).

2. **Harmful Bias**: GAI systems can inadvertently perpetuate or amplify biases present in the training data. This can result in outputs that are discriminatory or unfair, impacting marginalized groups negatively (Source: [document name], p. 22).

3. **Disinformation and Misinformation**: GAI can be used to generate misleading or false information, which can erode public trust in institutions and contribute to societal polarization. The rapid spread of disinformation can have long-term effects on democratic processes and public discourse (Source: [document name], p. 22).

4. **Emotional and Psychological Harm**: The use of GAI to create deepfakes or other manipulative content can lead to emotional distress for individuals depicted in such content. This includes risks associated with non-consensual intimate imagery and other forms of abusive content (Source: [document name], p. 22).

5. **Security Vulnerabilities**: GAI systems can be susceptible to adversarial attacks, where malicious actors manipulate inputs to produce harmful outputs. This includes risks like prompt injection and data poisoning, which can compromise the integrity of the AI system (Source: [document name], p. 22).

6. **Intellectual Property Issues**: GAI systems may inadvertently generate content that infringes on copyrights or other intellectual property rights, leading to legal challenges and potential liabilities for organizations (Source: [document name], p. 22).

7. **Ecosystem-Level Risks**: The deployment of GAI systems can have broader implications for society, including impacts on employment, economic inequality, and the overall social fabric. Organizations need to consider how their AI systems fit into the larger ecosystem and the potential societal consequences (Source: [document name], p. 22).

By being aware of these risks, organizations can implement appropriate risk management strategies and governance frameworks to mitigate potential harms associated with GAI systems.","Organizations need to consider various GAI risks in the development and deployment of AI systems, including unacceptable use identified by stakeholder communities, harmful bias and homogenization, dangerous, violent, or hateful content, immature safety or risk cultures related to AI and GAI design, development, and deployment, public information integrity risks impacting democratic processes, unknown long-term performance characteristics of GAI, and risks related to generating illegal content or violating laws.",1.0,0.9450323876155746,1.0,0.999999999975,0.6924312948245499
How should the validity of explanations provided by automated systems be ensured?,"[""NOTICE & \nEXPLANATION \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nTailored to the level of risk. An assessment should be done to determine the level of risk of the auto -\nmated system. In settings where the consequences are high as determined by a risk assessment, or extensive \noversight is expected (e.g., in criminal justice or some public sector settings), explanatory mechanisms should be built into the system design so that the system’s full behavior can be explained in advance (i.e., only fully transparent models should be used), rather than as an after-the-decision interpretation. In other settings, the extent of explanation provided should be tailored to the risk level. \nValid. The explanation provided by a system should accurately reflect the factors and the influences that led \nto a particular decision, and should be meaningful for the particular customization based on purpose, target, and level of risk. While approximation and simplification may be necessary for the system to succeed based on the explanatory purpose and target of the explanation, or to account for the risk of fraud or other concerns related to revealing decision-making information, such simplifications should be done in a scientifically supportable way. Where appropriate based on the explanatory system, error ranges for the explanation should be calculated and included in the explanation, with the choice of presentation of such information balanced with usability and overall interface complexity concerns. \nDemonstrate protections for notice and explanation \nReporting. Summary reporting should document the determinations made based on the above consider -\nations, including: the responsible entities for accountability purposes; the goal and use cases for the system, identified users, and impacted populations; the assessment of notice clarity and timeliness; the assessment of the explanation's validity and accessibility; the assessment of the level of risk; and the account and assessment of how explanations are tailored, including to the purpose, the recipient of the explanation, and the level of risk. Individualized profile information should be made readily available to the greatest extent possible that includes explanations for any system impacts or inferences. Reporting should be provided in a clear plain language and machine-readable manner. \n44""
 ""HUMAN ALTERNATIVES, \nCONSIDERATION, AND \nFALLBACK \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nEquitable. Consideration should be given to ensuring outcomes of the fallback and escalation system are \nequitable when compared to those of the automated system and such that the fallback and escalation \nsystem provides equitable access to underserved communities.105 \nTimely . Human consideration and fallback are only useful if they are conducted and concluded in a \ntimely manner. The determination of what is timely should be made relative to the specific automated \nsystem, and the review system should be staffed and regularly assessed to ensure it is providing timely \nconsideration and fallback. In time-critical systems, this mechanism should be immediately available or, \nwhere possible, available before the harm occurs. Time-critical systems include, but are not limited to, \nvoting-related systems, automated building access and other access systems, systems that form a critical \ncomponent of healthcare, and systems that have the ability to withhold wages or otherwise cause \nimmediate financial penalties. \nEffective. The organizational structure surrounding processes for consideration and fallback should \nbe designed so that if the human decision-maker charged with reassessing a decision determines that it \nshould be overruled, the new decision will be effectively enacted. This includes ensuring that the new \ndecision is entered into the automated system throughout its components, any previous repercussions from \nthe old decision are also overturned, and safeguards are put in place to help ensure that future decisions do \nnot result in the same errors. \nMaintained. The human consideration and fallback process and any associated automated processes \nshould be maintained and supported as long as the relevant automated system continues to be in use. \nInstitute training, assessment, and oversight to combat automation bias and ensure any \nhuman-based components of a system are effective. \nTraining and assessment. Anyone administering, interacting with, or interpreting the outputs of an auto -\nmated system should receive training in that system, including how to properly interpret outputs of a system in light of its intended purpose and in how to mitigate the effects of automation bias. The training should reoc\n-\ncur regularly to ensure it is up to date with the system and to ensure the system is used appropriately. Assess -\nment should be ongoing to ensure that the use of the system with human involvement provides for appropri -\nate results, i.e., that the involvement of people does not invalidate the system's assessment as safe and effective or lead to algorithmic discrimination.""
 'SAFE AND EFFECTIVE \nSYSTEMS \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nDerived data sources tracked and reviewed carefully. Data that is derived from other data through \nthe use of algorithms, such as data derived or inferred from prior model outputs, should be identified and tracked, e.g., via a specialized type in a data schema. Derived data should be viewed as potentially high-risk inputs that may lead to feedback loops, compounded harm, or inaccurate results. Such sources should be care\n-\nfully validated against the risk of collateral consequences. \nData reuse limits in sensitive domains. Data reuse, and especially data reuse in a new context, can result in the spreading and scaling of harms. Data from some domains, including criminal justice data and data indi\n-\ncating adverse outcomes in domains such as finance, employment, and housing, is especially sensitive, and in some cases its reuse is limited by law. Accordingly, such data should be subject to extra oversight to ensure safety and efficacy. Data reuse of sensitive domain data in other contexts (e.g., criminal data reuse for civil legal matters or private sector use) should only occur where use of such data is legally authorized and, after examina\n-\ntion, has benefits for those impacted by the system that outweigh identified risks and, as appropriate, reason -\nable measures have been implemented to mitigate the identified risks. Such data should be clearly labeled to identify contexts for limited reuse based on sensitivity. Where possible, aggregated datasets may be useful for replacing individual-level sensitive data. \nDemonstrate the safety and effectiveness of the system \nIndependent evaluation. Automated systems should be designed to allow for independent evaluation (e.g., \nvia application programming interfaces). Independent evaluators, such as researchers, journalists, ethics \nreview boards, inspectors general, and third-party auditors, should be given access to the system and samples \nof associated data, in a manner consistent with privac y, security, la w, or regulation (including, e.g., intellectual \nproperty law), in order to perform such evaluations. Mechanisms should be included to ensure that system \naccess for evaluation is: provided in a timely manner to the deployment-ready version of the system; trusted to \nprovide genuine, unfiltered access to the full system; and truly independent such that evaluator access cannot \nbe revoked without reasonable and verified justification. \nReporting.12 Entities responsible for the development or use of automated systems should provide \nregularly-updated reports that include: an overview of the system, including how it is embedded in the'
 'You should know that an automated system is being used, \nand understand how and why it contributes to outcomes that impact you. Designers, developers, and deployers of automat\n-\ned systems should provide generally accessible plain language docu -\nmentation including clear descriptions of the overall system func -\ntioning and the role automation plays, notice that such systems are in use, the individual or organization responsible for the system, and ex\n-\nplanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality chang\n-\nes. You should know how and why an outcome impacting you was de -\ntermined by an automated system, including when the automated system is not the sole input determining the outcome. Automated systems should provide explanations that are technically valid, meaningful and useful to you and to any operators or others who need to understand the system, and calibrated to the level of risk based on the context. Reporting that includes summary information about these automated systems in plain language and assessments of the clarity and quality of the notice and explanations should be made public whenever possible.   NOTICE AND  EXPLANATION\n40']","To ensure the validity of explanations provided by automated systems, organizations can adopt several key practices:

1. **Clear Definition of Explanatory Goals**: Establish clear objectives for what the explanations should achieve. This includes understanding the audience for the explanations and tailoring the content to meet their needs (Source: [document name], p. 44).

2. **Use of Transparent Models**: Whenever possible, utilize models that are inherently interpretable. This means choosing algorithms that allow for straightforward understanding of how inputs are transformed into outputs, rather than relying on complex, opaque models (Source: [document name], p. 44).

3. **Validation Against Real-World Outcomes**: Regularly validate the explanations against actual outcomes to ensure they accurately reflect the decision-making process. This involves comparing the explanations provided by the system with the results of decisions made in practice (Source: [document name], p. 44).

4. **Incorporation of User Feedback**: Implement mechanisms for users to provide feedback on the explanations they receive. This feedback can help identify areas where explanations may be unclear or misleading, allowing for continuous improvement (Source: [document name], p. 44).

5. **Error Range and Confidence Levels**: Include information about the uncertainty associated with the explanations. Providing error ranges or confidence levels can help users understand the reliability of the information being presented (Source: [document name], p. 44).

6. **Regular Audits and Assessments**: Conduct regular audits of the automated systems to assess the quality and validity of the explanations provided. This can involve independent evaluations by third parties to ensure objectivity (Source: [document name], p. 44).

7. **Documentation and Transparency**: Maintain thorough documentation of the decision-making processes and the factors considered by the automated system. This documentation should be accessible and understandable to stakeholders, enhancing transparency (Source: [document name], p. 44).

By implementing these practices, organizations can enhance the validity of the explanations provided by their automated systems, fostering trust and accountability in their use.","The explanation provided by a system should accurately reflect the factors and influences that led to a particular decision, and should be meaningful for the particular customization based on purpose, target, and level of risk. While approximation and simplification may be necessary for the system to succeed based on the explanatory purpose and target of the explanation, or to account for the risk of fraud or other concerns related to revealing decision-making information, such simplifications should be done in a scientifically supportable way. Where appropriate based on the explanatory system, error ranges for the explanation should be calculated and included in the explanation, with the choice of presentation of such information balanced with usability and overall interface complexity concerns.",1.0,0.9560916899077769,1.0,0.999999999975,0.8118352103904202
How do generative models like LLMs generate outputs that can lead to confabulations in GAI systems?,"['6 2.2. Confabulation  \n“Confabulation” refers to a phenomenon in which GAI systems generate and confidently present \nerroneous or false content in response to prompts . Confabulations also include generated outputs that \ndiverge from the prompts  or other  input  or that contradict previously generated statements in the same \ncontext. Th ese phenomena are colloquially also referred to as “hallucination s” or “fabrication s.” \nConfabulations can occur across GAI outputs  and contexts .9,10 Confabulations are a natural result of the \nway generative models  are designed : they  generate outputs that approximate the statistical distribution \nof their training data ; for example,  LLMs  predict the next  token or word  in a sentence or phrase . While \nsuch statistical  prediction can produce factual ly accurate  and consistent  outputs , it can  also produce \noutputs that are factually inaccurat e or internally inconsistent . This dynamic is particularly relevant when \nit comes to  open -ended prompts  for long- form responses  and in domains  which require highly \ncontextual and/or  domain expertise.  \nRisks from confabulations may arise when users believe false content  – often  due to the confident nature \nof the response  – leading users to act upon or promote the false information.  This poses a challenge  for \nmany real -world applications, such as in healthcare, where a confabulated summary of patient \ninformation reports could  cause doctors to make  incorrect diagnoses  and/or recommend the wrong \ntreatments.  Risks of confabulated content may be especially important to monitor  when integrating GAI \ninto applications involving  consequential  decision making. \nGAI outputs may also include confabulated logic or citations  that purport to justify or explain the \nsystem’s answer , which may further mislead  humans into inappropriately trusting the system’s output. \nFor instance, LLMs  sometimes provide logical steps for how they arrived at an answer even when the \nanswer itself is incorrect. Similarly, an LLM could falsely assert that it is human or has human traits, \npotentially deceiv ing humans into believing they are speaking with another human. \nThe extent to which humans can be deceived by LLMs, the mechanisms by which this may occur, and the \npotential risks from adversarial prompting of such behavior are  emerging  areas of study . Given the wide \nrange of downstream impacts of GAI, it is difficult to estimate the downstream scale and impact of \nconfabulations . \nTrustworthy AI Characteristics:  Fair with Harmful Bias Managed, Safe, Valid and Reliable , Explainable \nand Interpretable  \n2.3. Dangerous , Violent , or Hateful  Content  \nGAI systems can  produce content that is  inciting, radicalizing, or threatening, or  that glorifi es violence , \nwith greater ease and scale than other technologies . LLMs have been reported to generate  dangerous or'
 'violent recommendations , and s ome models have generated actionable instructions for dangerous  or \n \n \n9 Confabulations of falsehoods are most commonly a problem for text -based outputs; for audio, image, or video \ncontent, creative generation of non- factual content can be  a desired behavior.  \n10 For example, legal confabulations have been shown to be pervasive  in current state -of-the-art LLMs. See also, \ne.g.,'
 '60 Zhang, Y . et al. (2023) Human favoritism, not AI aversion: People’s perceptions (and bias) toward \ngenerative AI, human experts, and human –GAI collaboration in persuasive content generation.  Judgment \nand Decision Making. https://www.cambridge.org/core/journals/judgment -and-decision -\nmaking/article/human -favoritism -not- ai-aversion -peoples -perceptions -and-bias-toward -generative -ai-\nhuman- experts -and-humangai -collaboration -in-persuasive -content -\ngeneration/419C4BD9CE82673EAF1D8F6C350C4FA8 \nZhang, Y . et al. (2023) Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models.  \narXiv . https://arxiv.org/pdf/2309.01219  \nZhao, X. et al. (2023) Provable Robust Watermarking for AI -Generated Text.  Semantic Scholar . \nhttps://www.semanticscholar.org/paper/Provable -Robust -Watermarking -for-AI-Generated -Text-Zhao -\nAnanth/75b68d0903af9d9f6e47ce3cf7e1a7d27ec811dc'
 '5 operations , or other cyberattacks ; increas ed attack surface  for targeted cyberattacks , which may \ncompromise a system’s availability or the confidentiality or integrity of training data, code, or \nmodel weights.   \n10. Intellectual Property:  Eased production or replication  of alleged  copyrighted, trademarked, or \nlicensed content without authorization  (possibly in situations which do not fall under fair use ); \neased exposure of trade secrets;  or plagiari sm or illegal replication .  \n11. Obscen e, Degrading, and/or A busive Content : Eased production of and access to obscene , \ndegrading, and/or abusive  imagery  which can cause harm , including synthetic child sexual abuse \nmaterial (CSAM) , and nonconsensual intimate images (NCII) of adults . \n12. Value Chain and Component Integration : Non-transparent or untraceable  integration of \nupstream third- party components, including data that has been improperly obtained or not \nprocessed and cleaned due to increased automation from GAI; improper supplier vetting across \nthe AI lifecycle ; or other issues that diminish transparency or accountability for downstream \nusers.  \n2.1. CBRN Information or Capabilities  \nIn the future, GAI may enable malicious actors to more easily  access CBRN  weapons and/or relevant \nknowledge, information , materials, tools, or technologies that could be misused  to assist in the design, \ndevelopment, production, or use of CBRN weapons or other dangerous materials  or agents . While \nrelevant biological and chemical threat knowledge and information is often  publicly accessible , LLMs  \ncould facilitate its analysis or synthesis , particularly by individuals  without formal scientific training  or \nexpertise.  \nRecent research on this topic found that LLM outputs regarding biological threat creation  and attack \nplanning  pr ovided minima l assistance  beyond traditional search engine queries, suggesting that state -of-\nthe-art LLMs at the time these studies were conducted do not substantially  increase the operational \nlikelihood of such an attack.  The physical synthesis development, production, and use of chemical or \nbiological agents will continue to require both applicable expertise and supporting materials and \ninfrastructure . The impact of GAI on chemical or biological agent misuse will depend on  what the  key \nbarriers  for malicious actors  are (e.g., whether information access is one such barrier ), and how well GAI \ncan help actors  address those barriers .  \nFurthermore , chemical and biological design tools  (BDTs)  – highly specialized AI systems  trained on \nscientific data that aid in chemical and biological design – may augment  design capabilities in chemistry \nand biology beyond what text -based LLMs are able to provide . As these models become more \nefficacious , including for beneficial uses, it will be important to assess their potential to be used for']","Generative models, such as large language models (LLMs), produce outputs through a process that involves predicting the next word or token in a sequence based on the patterns learned from their training data. This process can lead to confabulations—instances where the model generates outputs that are factually incorrect, internally inconsistent, or diverge from the input prompts. Here’s how this happens:

1. **Statistical Prediction**: LLMs are designed to approximate the statistical distribution of their training data. They generate text by predicting the most likely next word based on the context provided. While this can lead to coherent and contextually relevant outputs, it can also result in the generation of incorrect or nonsensical information if the model encounters prompts that are ambiguous or outside its training scope (Source: [document name], p. 6).

2. **Open-Ended Prompts**: When given open-ended prompts, LLMs may produce long-form responses that can stray from factual accuracy. The lack of specific guidance can lead the model to fabricate details or provide information that sounds plausible but is not true (Source: [document name], p. 6).

3. **Lack of Domain Expertise**: In domains requiring specialized knowledge, LLMs may generate outputs that appear authoritative but are factually incorrect. This is particularly concerning in fields like healthcare or law, where inaccuracies can have serious consequences (Source: [document name], p. 6).

4. **Confident Presentation**: LLMs often present their outputs with a high degree of confidence, which can mislead users into believing that the information is accurate. This confidence can lead users to act on or share false information without verification (Source: [document name], p. 6).

5. **Internal Inconsistencies**: Confabulations can also occur when LLMs generate statements that contradict each other within the same context. This inconsistency arises from the model's reliance on statistical patterns rather than a coherent understanding of the subject matter (Source: [document name], p. 6).

Overall, the nature of how generative models operate—predicting text based on learned patterns without true comprehension—makes them susceptible to producing confabulated outputs. This highlights the importance of careful monitoring and validation of AI-generated content, especially in high-stakes applications.","Generative models like LLMs generate outputs that can lead to confabulations in GAI systems by approximating the statistical distribution of their training data. While this statistical prediction can result in factually accurate and consistent outputs, it can also produce outputs that are factually inaccurate or internally inconsistent. This becomes particularly relevant in open-ended prompts for long-form responses and domains requiring contextual or domain expertise.",1.0,0.9506450188685709,1.0,0.999999999975,0.3878500124117624
How can appropriate diligence on training data use help assess intellectual property risks in AI systems?,"[""27 MP-4.1-0 10 Conduct appropriate diligence on  training data use to assess intellectual property, \nand privacy, risks, including to examine whether use of proprietary or sensitive \ntraining data is consistent with applicable laws.  Intellectual Property ; Data Privacy  \nAI Actor Tasks: Governance and Oversight, Operation and Monitoring, Procurement, Third -party entities  \n \nMAP 5.1:  Likelihood and magnitude of each identified impact (both potentially beneficial and harmful) based on expected use, past \nuses of AI systems in similar contexts, public incident reports, feedback from those external to the team that developed or d eployed \nthe AI system, or other data are identified and documented.  \nAction ID  Suggested Action  GAI Risks  \nMP-5.1-001 Apply TEVV practices for content provenance (e.g., probing a system's synthetic \ndata generation capabilities for potential misuse or vulnerabilities . Information Integrity ; Information \nSecurity  \nMP-5.1-002 Identify potential content provenance harms of GAI, such as misinformation or \ndisinformation, deepfakes, including NCII, or tampered content. Enumerate and rank risks based on their likelihood and potential impact, and determine how well provenance solutions address specific risks and/or harms.  Information Integrity ; Dangerous , \nViolent, or Hateful Content ; \nObscene, Degrading, and/or Abusive Content  \nMP-5.1-003 Consider d isclos ing use of GAI to end user s in relevant contexts, while considering \nthe objective of disclosure, the context of use, the likelihood and magnitude of the  \nrisk posed, the audience of the disclosure, as well as the frequency of the disclosures.  Human -AI Configuration  \nMP-5.1-004 Prioritize GAI structured public feedback processes based on risk assessment estimates.  Information Integrity ; CBRN \nInformation or Capabilities ; \nDangerous , Violent, or Hateful \nContent ; Harmful Bias and \nHomogenization  \nMP-5.1-005 Conduct adversarial role -playing exercises, GAI red -teaming, or chaos testing to \nidentify anomalous or unforeseen failure modes.  Information Security  \nMP-5.1-0 06 Profile threats and negative impacts  arising from GAI systems interacting with, \nmanipulating, or generating content, and outlining known and potential vulnerabilities and the likelihood of their occurrence.  Information Security  \nAI Actor Tasks: AI Deployment, AI Design, AI Development, AI Impact Assessment, Affected Individuals and Communities, End -\nUsers, Operation and Monitoring""
 'SAFE AND EFFECTIVE \nSYSTEMS \nHOW THESE PRINCIPLES CAN MOVE INTO PRACTICE\nReal-life examples of how these principles can become reality, through laws, policies, and practical \ntechnical and sociotechnical approaches to protecting rights, opportunities, and access. \nSome U.S government agencies have developed specific frameworks for ethical use of AI \nsystems. The Department of Energy (DOE) has activated the AI Advancement Council that oversees coordina -\ntion and advises on implementation of the DOE AI Strategy and addresses issues and/or escalations on the \nethical use and development of AI systems.20 The Department of Defense has adopted Artificial Intelligence \nEthical Principles, and tenets for Responsible Artificial Intelligence specifically tailored to its national \nsecurity and defense activities.21 Similarl y, the U.S. Intelligence Community (IC) has developed the Principles \nof Artificial Intelligence Ethics for the Intelligence Community to guide personnel on whether and how to \ndevelop and use AI in furtherance of the IC\'s mission, as well as an AI Ethics Framework to help implement \nthese principles.22\nThe National Science Foundation (NSF) funds extensive research to help foster the \ndevelopment of automated systems that adhere to and advance their safety, security and \neffectiveness. Multiple NSF programs support research that directly addresses many of these principles: \nthe National AI Research Institutes23 support research on all aspects of safe, trustworth y, fai r, and explainable \nAI algorithms and systems; the Cyber Physical Systems24 program supports research on developing safe \nautonomous and cyber physical systems with AI components; the Secure and Trustworthy Cyberspace25 \nprogram supports research on cybersecurity and privacy enhancing technologies in automated systems; the \nFormal Methods in the Field26 program supports research on rigorous formal verification and analysis of \nautomated systems and machine learning, and the Designing Accountable Software Systems27 program supports \nresearch on rigorous and reproducible methodologies for developing software systems with legal and regulatory \ncompliance in mind. \nSome state legislatures have placed strong transparency and validity requirements on \nthe use of pretrial risk assessments. The use of algorithmic pretrial risk assessments has been a \ncause of concern for civil rights groups.28 Idaho Code Section 19-1910, enacted in 2019,29 requires that any \npretrial risk assessment, before use in the state, first be ""shown to be free of bias against any class of \nindividuals protected from discrimination by state or federal law"", that any locality using a pretrial risk \nassessment must first formally validate the claim of its being free of bias, that ""all documents, records, and \ninformation used to build or validate the risk assessment shall be open to public inspection,"" and that assertions'
 '58 Satariano, A. et al. (2023) The People Onscreen Are Fake. The Disinformation Is Real.  New York Times . \nhttps://www.nytimes.com/2023/02/07/technology/artificial -intelligence -training- deepfake.html  \nSchaul, K. et al. (2024) Inside the secret list of websites that make AI like ChatGPT sound smart.  \nWashington Post . https://www.washingtonpost.com/technology/interactive/2023/ai -chatbot -learning/  \nScheurer , J. et al. (2023) Technical report: Large language models can strategically deceive their users \nwhen put under pressure. arXiv.  https://arxiv.org/abs/2311.07590  \nShelby, R. et al. (2023) Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm \nReduction.  arXiv . https://arxiv.org/pdf/2210.05791  \nShevlane, T. et al. (2023) Model evaluation for extreme risks.  arXiv . https://arxiv.org/pdf/2305.15324  \nShumailov, I. et al. (2023) The curse of recursion: training on generated data makes models forget.  arXiv . \nhttps://arxiv.org/pdf/2305.17493v2  \nSmith, A. et al. (2023) Hallucination or Confabulation? Neuroanatomy as metaphor in Large Language Models.  PLOS Digital Health. \nhttps://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000388\n \nSoice, E. et al. (2023) Can large language models democratize access to dual -use biotechnology?  arXiv . \nhttps://arxiv.org/abs/2306.03809  \nSolaiman, I. et al. (2023) The Gradient of Generative AI Release: Methods and Considerations. arXiv.  \nhttps://arxiv.org/abs/2302.04844  \nStaab, R. et al. (2023) Beyond Memorization: Violating Privacy via Inference With Large Language \nModels.  arXiv . https://arxiv.org/pdf/2310.07298  \nStanford, S. et al. (2023) Whose Opinions Do Language Models Reflect?  arXiv . \nhttps://arxiv.org/pdf/2303.17548  \nStrubell, E. et al. (2019) Energy and Policy Considerations for Deep Learning in NLP .  arXiv . \nhttps://arxiv.org/pdf/1906.02243  \nThe White House  (2016) Circular No. A -130, Managing Information as a Strategic Resource . \nhttps://www.whitehouse.gov/wp-\ncontent/uploads/legacy_drupal_files/omb/circulars/A130/a130revised.pdf  \nThe White House  (2023)  Executive Order on the Safe, Secure, and Trustworthy Development and Use of \nArtificial Intelligence . https://www.whitehouse.gov/briefing- room/presidential -\nactions/2023/10/30/executive -order -on-the-safe-secure -and-trustworthy -development -and-use-of-\nartificial -intelligence/  \nThe White House  (2022) Roadmap for Researchers on Priorities Related to Information Integrity \nResearch and Development.  https://www.whitehouse.gov/wp- content/uploads/2022/12/Roadmap-\nInformation -Integrity -RD-2022.pdf?  \nThiel, D. (2023) Investigation Finds AI Image Generation Models Trained on Child Abuse.  Stanford Cyber \nPolicy Center . https://cyber.fsi.stanford.edu/news/investigation- finds -ai-image -generation -models -\ntrained -child -abuse'
 '11 value chain  (e.g., data inputs , processing, GAI training, or deployment  environments ), conventional \ncybersecurity practices  may need to adapt or evolve . \nFor instance , prompt  injection  involves  modifying what  input is provided to a  GAI system  so that it  \nbehave s in unintended ways. In direct prompt injections, attackers might craft malicious  prompts  and \ninput them directly to a GAI system , with a variety of  downstream negative consequences to \ninterconnected systems. Indirect prompt injection  attacks occur when  adversaries remotely (i.e., without \na direct interface) exploit LLM -integrated applications by injecting prompts into data likely to be \nretrieved. Security researchers have already demonstrated how indirect prompt injections can exploit \nvulnerabilities by  steal ing proprietary  data  or running malicious code remotely  on a machine. Merely \nquerying  a closed production model can elicit  previously undisclosed information about that model . \nAnother cybersecurity risk to GAI is data poisoning , in which  an adversary compromises  a training \ndataset used by a model to manipulate its output s or operation. Malicious tampering with  data or parts \nof the  model could exacerbate risks associated with GAI system outputs.  \nTrustworthy AI Characteristics:  Privacy Enhanced, Safe , Secure and Resilient , Valid and Reliable  \n2.10. Intellectual Property  \nIntellectual property risks from  GAI systems may arise where the use of copyrighted works  is not a fair \nuse under the fair use doctrine. If a GAI system’s training data included copyrighted material, GAI \noutputs  displaying instances of  training data  memorization  (see Data Privacy above) could infringe on \ncopyright. \nHow GAI relates to copyright, including the status of  generated  content that is similar to but does not \nstrictly copy  w ork protected by copyright , is currently being debated in legal fora.  Similar discussions are \ntaking place  regard ing the use or emulation of personal identit y, likeness , or voice  without permission.  \nTrustworthy AI Characteristics:  Accountable and Transparent, Fair with Harmful Bias Managed, Privacy \nEnhanced  \n2.11. Obscene, Degrading, and/or A busive Content  \nGAI can ease the production of and access to illegal non- consensual intimate imagery  (NCII)  of adult s, \nand/or child sexual abuse material  (CSAM) . GAI- generated obscene, abusive  or degrading content can  \ncreate privacy, psychological and emotional, and even physical harms , and in some cases may be illegal .  \nGenerated explicit or obscene AI content may include highly  realistic “deepfakes” of real individuals, \nincluding children. The spread of this kind of material can have downstream negative consequences : in \nthe context of CSAM, e ven if the generated images do not resemble specific individuals, the prevalence \nof such images c an divert  time and re sources from  efforts  to find real -world victims. Outside of CSAM,']","Appropriate diligence in the use of training data is crucial for assessing intellectual property (IP) risks in AI systems for several reasons:

1. **Understanding Data Sources**: Conducting thorough due diligence helps organizations identify the sources of their training data. Knowing whether the data is proprietary, public domain, or licensed is essential for determining if its use complies with IP laws (Source: [document name], p. 27).

2. **Evaluating Fair Use**: Organizations must assess whether their use of copyrighted materials falls under the fair use doctrine. This involves analyzing the purpose, nature, amount, and effect of the use on the market for the original work. Diligence ensures that organizations can justify their use of copyrighted data (Source: [document name], p. 27).

3. **Identifying Potential Infringements**: By examining the training data, organizations can identify any potential infringements on IP rights. This includes checking for copyrighted works that may be included in the dataset without proper authorization, which could lead to legal challenges (Source: [document name], p. 27).

4. **Documenting Compliance**: Maintaining records of the data sources and the rationale for their use can provide a defense in case of IP disputes. Diligence in documentation helps demonstrate that the organization has taken steps to comply with IP laws (Source: [document name], p. 27).

5. **Mitigating Liability Risks**: By proactively assessing IP risks associated with training data, organizations can implement strategies to mitigate potential liabilities. This may include obtaining licenses for copyrighted materials or using alternative datasets that do not pose IP risks (Source: [document name], p. 27).

6. **Enhancing Transparency**: Diligence in data use promotes transparency in AI development. This transparency can build trust with stakeholders and users, as it shows that the organization is committed to ethical practices regarding IP rights (Source: [document name], p. 27).

In summary, appropriate diligence in the use of training data is essential for assessing and managing intellectual property risks in AI systems. It helps organizations navigate legal complexities, avoid potential infringements, and ensure compliance with applicable laws.","Appropriate diligence on training data use can help assess intellectual property risks in AI systems by examining whether the use of proprietary or sensitive training data aligns with relevant laws. This includes evaluating the likelihood and magnitude of potential impacts, both beneficial and harmful, based on past uses of AI systems in similar contexts, public incident reports, feedback from external parties, and other relevant data. By identifying and documenting these impacts, organizations can better understand the risks associated with their training data and take appropriate measures to mitigate them.",0.8666666666666667,0.9801429602679098,0.6666666666666666,0.8055555555287036,0.36200954256939605
How do integrated human-AI systems benefit companies in providing customer service?,"['SAFE AND EFFECTIVE \nSYSTEMS \nHOW THESE PRINCIPLES CAN MOVE INTO PRACTICE\nReal-life examples of how these principles can become reality, through laws, policies, and practical \ntechnical and sociotechnical approaches to protecting rights, opportunities, and access. \nExecutive Order 13960 on Promoting the Use of Trustworthy Artificial Intelligence in the \nFederal Government requires that certain federal agencies adhere to nine principles when \ndesigning, developing, acquiring, or using AI for purposes other than national security or \ndefense. These principles—while taking into account the sensitive law enforcement and other contexts in which \nthe federal government may use AI, as opposed to private sector use of AI—require that AI is: (a) lawful and \nrespectful of our Nation’s values; (b) purposeful and performance-driven; (c) accurate, reliable, and effective; (d) \nsafe, secure, and resilient; (e) understandable; (f ) responsible and traceable; (g) regularly monitored; (h) transpar -\nent; and, (i) accountable. The Blueprint for an AI Bill of Rights is consistent with the Executive Order. \nAffected agencies across the federal government have released AI use case inventories13 and are implementing \nplans to bring those AI systems into compliance with the Executive Order or retire them. \nThe law and policy landscape for motor vehicles shows that strong safety regulations—and \nmeasures to address harms when they occur—can enhance innovation in the context of com-\nplex technologies. Cars, like automated digital systems, comprise a complex collection of components. \nThe National Highway Traffic Safety Administration,14 through its rigorous standards and independent \nevaluation, helps make sure vehicles on our roads are safe without limiting manufacturers’ ability to \ninnovate.15 At the same time, rules of the road are implemented locally to impose contextually appropriate \nrequirements on drivers, such as slowing down near schools or playgrounds.16\nFrom large companies to start-ups, industry is providing innovative solutions that allow \norganizations to mitigate risks to the safety and efficacy of AI systems, both before \ndeployment and through monitoring over time.17 These innovative solutions include risk \nassessments, auditing mechanisms, assessment of organizational procedures, dashboards to allow for ongoing \nmonitoring, documentation procedures specific to model assessments, and many other strategies that aim to \nmitigate risks posed by the use of AI to companies’ reputation, legal responsibilities, and other product safety \nand effectiveness concerns. \nThe Office of Management and Budget (OMB) has called for an expansion of opportunities \nfor meaningful stakeholder engagement in the design of programs and services. OMB also \npoints to numerous examples of effective and proactive stakeholder engagement, including the Community-'
 'consequential decision- making settings like employment and lending can result in increased susceptibility  by \nsystems  to correlated failures  (like unexpected shocks),  due to multiple actors relying on the  same algorithm.  \n4 Many studies have projected the impact of AI on the workforce and  labor markets. Fewer studies have examined \nthe impact of GAI on the labor market , though some industry surveys indicate that  that both emp loyees and \nemployers are pondering this disruption.'
 '50 Participatory Engagement Methods  \nOn an ad hoc or more structured basis, organizations can design and use a variety of channels to engage \nexternal stakeholders in product development or review. Focus groups with select experts can provide \nfeedback on a range of issues. Small user studies c an provide feedback from representative groups or \npopulations. Anonymous surveys can be used to poll or gauge reactions to specific features. Participatory engagement methods are often less structured than field testing or red teaming, and are more \ncommonl y used in early stages of AI or product development.  \nField Testing  \nField testing involves structured settings to evaluate risks and impacts and to simulate the conditions \nunder which the GAI system will be deployed. Field style tests can be adapted from a focus on user \npreferences and experiences towards AI risks and impacts – both negative and positive. When carried \nout with large groups of users, these tests can provide estimations of the likelihood of risks and impacts \nin real world interactions.  \nOrganizations may also collect feedback on outcomes, harms, and user experience directly from users in the production environment after a model has been released, in accordance with human subject \nstandards such as informed consent and compensation. Organiz ations should follow applicable human \nsubjects research requirements, and best practices such as informed consent and subject compensation, \nwhen implementing feedback activities. \nAI Red -teaming  \nAI red -teaming is an evolving practice that references exercises often  conducted in a controlled \nenvironment and in collaboration with AI developers building AI models  to identify potential adverse \nbehavior or outcomes of a GAI model or system,  how they could occur, an d stress test safeguards” . AI \nred-teaming can be performed before or after AI models or systems are made available to the broader \npublic; this section focuses on red- teaming in pre -deployment contexts.  \nThe quality of AI red- teaming outputs is  related to the background and expertise of the AI red  team \nitself. Demographically and interdisciplinarily  diverse AI red  teams can be used to identify flaws in the \nvarying contexts where GAI will be used. For best results, AI red  teams should demonstrate domain \nexpertise, and awareness of socio -cultural aspects within the deployment context. AI red -teaming results \nshould be given additional analysis before they are incorporated into organizational governance and \ndecision making, policy and procedural updates, and AI risk management efforts. \nVarious types of AI red -teaming may be appropriate, depending on the use case:  \n• General Public: Performed by general users (not necessarily AI or technical experts) who are \nexpected to use the model or interact with its outputs, and who bring their own lived \nexperiences and perspectives to the task of AI red -teaming . These individuals may have been'
 'You should be able to opt out, where appropriate, and \nhave access to a person who can quickly consider and remedy problems you encounter. You should be able to opt out from automated systems in favor of a human alternative, where appropriate. Appropriateness should be determined based on rea\n-\nsonable expectations in a given context and with a focus on ensuring broad accessibility and protecting the public from especially harm\n-\nful impacts. In some cases, a human or other alternative may be re -\nquired by law. You should have access to timely human consider -\nation and remedy by a fallback and escalation process if an automat -\ned system fails, it produces an error, or you would like to appeal or contest its impacts on you. Human consideration and fallback should be accessible, equitable, effective, maintained, accompanied by appropriate operator training, and should not impose an unrea\n-\nsonable burden on the public. Automated systems with an intended use within sensitive domains, including, but not limited to, criminal justice, employment, education, and health, should additionally be tailored to the purpose, provide meaningful access for oversight, include training for any people interacting with the system, and in\n-\ncorporate human consideration for adverse or high-risk decisions. Reporting that includes a description of these human governance processes and assessment of their timeliness, accessibility, out\n-\ncomes, and effectiveness should be made public whenever possible. HUMAN  ALTERNATIVES , C ONSIDERATION ALLBACKF AND, \n46']","Integrated human-AI systems enhance customer service for companies in several impactful ways:

1. **24/7 Availability**: AI systems, such as chatbots and virtual assistants, can provide round-the-clock support to customers. This ensures that inquiries are addressed promptly, even outside of regular business hours, improving customer satisfaction (Source: [document name], p. 18).

2. **Efficient Query Handling**: AI can quickly process and respond to common customer queries, allowing human agents to focus on more complex issues. This division of labor enhances overall efficiency and reduces wait times for customers (Source: [document name], p. 18).

3. **Personalized Experiences**: AI systems can analyze customer data to provide personalized recommendations and responses. By understanding customer preferences and behaviors, companies can tailor their interactions, leading to a more engaging and relevant customer experience (Source: [document name], p. 18).

4. **Data-Driven Insights**: Integrated systems can collect and analyze customer interactions, providing valuable insights into customer needs and trends. This data can inform business strategies and improve service offerings (Source: [document name], p. 18).

5. **Seamless Escalation**: When AI systems encounter complex issues that require human intervention, they can seamlessly escalate the matter to a human agent. This ensures that customers receive the appropriate level of support without unnecessary delays (Source: [document name], p. 18).

6. **Consistent Responses**: AI can help maintain consistency in customer service responses, reducing the likelihood of human error or variability in service quality. This consistency builds trust and reliability in the brand (Source: [document name], p. 18).

7. **Cost Efficiency**: By automating routine tasks and inquiries, companies can reduce operational costs associated with customer service. This allows for better allocation of resources and potentially lower prices for customers (Source: [document name], p. 18).

In summary, integrated human-AI systems significantly enhance customer service by improving efficiency, personalization, and consistency, while also providing valuable insights that can drive business improvements.","Integrated human-AI systems benefit companies in providing customer service by allowing for faster customer care while maintaining human agents to handle complicated requests. These systems use partially automated platforms to answer common customer questions and compile issues for human agents to review, ensuring a balance between efficiency and personalized service.",0.0,0.9842112100837633,0.0,0.7499999999625,0.4050688981370314
What was the purpose of the year of public engagement that informed the development of the Blueprint for an AI Bill of Rights?,"['ABOUT THIS  FRAMEWORK\nThe Blueprint for an AI Bill of Rights is a set of five principles and associated practices to help guide the \ndesign, use, and deployment of automated systems to protect the rights of the American public in the age of \nartificial intel-ligence. Developed through extensive consultation with the American public, these principles are \na blueprint for building and deploying automated systems that are aligned with democratic values and protect \ncivil rights, civil liberties, and privacy. The Blueprint for an AI Bill of Rights includes this Foreword, the five \nprinciples, notes on Applying the The Blueprint for an AI Bill of Rights, and a Technical Companion that gives \nconcrete steps that can be taken by many kinds of organizations—from governments at all levels to companies of \nall sizes—to uphold these values. Experts from across the private sector, governments, and international \nconsortia have published principles and frameworks to guide the responsible use of automated systems; this \nframework provides a national values statement and toolkit that is sector-agnostic to inform building these \nprotections into policy, practice, or the technological design process.  Where existing law or policy—such as \nsector-specific privacy laws and oversight requirements—do not already provide guidance, the Blueprint for an \nAI Bill of Rights should be used to inform policy decisions.\nLISTENING TO THE AMERICAN PUBLIC\nThe White House Office of Science and Technology Policy has led a year-long process to seek and distill input \nfrom people across the country—from impacted communities and industry stakeholders to technology develop-\ners and other experts across fields and sectors, as well as policymakers throughout the Federal government—on \nthe issue of algorithmic and data-driven harms and potential remedies. Through panel discussions, public listen-\ning sessions, meetings, a formal request for information, and input to a publicly accessible and widely-publicized \nemail address, people throughout the United States, public servants across Federal agencies, and members of the \ninternational community spoke up about both the promises and potential harms of these technologies, and \nplayed a central role in shaping the Blueprint for an AI Bill of Rights. The core messages gleaned from these \ndiscussions include that AI has transformative potential to improve Americans’ lives, and that preventing the \nharms of these technologies is both necessary and achievable. The Appendix includes a full list of public engage-\nments. \n4'
 ""APPENDIX\n• OSTP conducted meetings with a variety of stakeholders in the private sector and civil society. Some of these\nmeetings were specifically focused on providing ideas related to the development of the Blueprint for an AI\nBill of Rights while others provided useful general context on the positive use cases, potential harms, and/or\noversight possibilities for these technologies. Participants in these conversations from the private sector and\ncivil society included:\nAdobe \nAmerican Civil Liberties Union (ACLU) The Aspen Commission on Information Disorder The Awood Center The Australian Human Rights Commission Biometrics Institute The Brookings Institute BSA | The Software Alliance Cantellus Group Center for American Progress Center for Democracy and Technology Center on Privacy and Technology at Georgetown Law Christiana Care Color of Change Coworker Data Robot Data Trust Alliance Data and Society Research Institute Deepmind EdSAFE AI Alliance Electronic Privacy Information Center (EPIC) Encode Justice Equal AI Google Hitachi's AI Policy Committee The Innocence Project Institute of Electrical and Electronics Engineers (IEEE) Intuit Lawyers Committee for Civil Rights Under Law Legal Aid Society The Leadership Conference on Civil and Human Rights Meta Microsoft The MIT AI Policy Forum Movement Alliance Project The National Association of Criminal Defense Lawyers O’Neil Risk Consulting & Algorithmic Auditing The Partnership on AI Pinterest The Plaintext Group pymetrics SAP The Security Industry Association Software and Information Industry Association (SIIA) Special Competitive Studies Project Thorn United for Respect University of California at Berkeley Citris Policy Lab University of California at Berkeley Labor Center Unfinished/Project Liberty Upturn US Chamber of Commerce US Chamber of Commerce Technology Engagement Center \nA.I. Working Group\nVibrent HealthWarehouse Worker ResourceCenterWaymap\n62""
 'SECTION  TITLE\n  \n \n \n  Applying The Blueprint for an AI Bill of Rights \nRELATIONSHIP TO EXISTING  LAW AND POLICY\nThere are regulatory safety requirements for medical devices, as well as sector-, population-, or technology-spe-\ncific privacy and security protections. Ensuring some of the additional protections proposed in this framework would require new laws to be enacted or new policies and practices to be adopted. In some cases, exceptions to the principles described in the Blueprint for an AI Bill of Rights may be necessary to comply with existing la w, \nconform to the practicalities of a specific use case, or balance competing public interests. In particula r, law \nenforcement, and other regulatory contexts may require government actors to protect civil rights, civil liberties, and privacy in a manner consistent with, but using alternate mechanisms to, the specific principles discussed in this framework. The Blueprint for an AI Bill of Rights is meant to assist governments and the private sector in moving principles into practice. \nThe expectations given in the Technical Companion are meant to serve as a blueprint for the development of \nadditional technical standards and practices that should be tailored for particular sectors and contexts. While \nexisting laws informed the development of the Blueprint for an AI Bill of Rights, this framework does not detail those laws beyond providing them as examples, where appropriate, of existing protective measures. This framework instead shares a broad, forward-leaning vision of recommended principles for automated system development and use to inform private and public involvement with these systems where they have the poten-tial to meaningfully impact rights, opportunities, or access. Additionall y, this framework does not analyze or \ntake a position on legislative and regulatory proposals in municipal, state, and federal government, or those in other countries. \nWe have seen modest progress in recent years, with some state and local governments responding to these prob-'
 'SECTION  TITLE\nAPPENDIX\nListening to the American People \nThe White House Office of Science and Technology Policy (OSTP) led a yearlong process to seek and distill \ninput from people across the country – from impacted communities to industry stakeholders to \ntechnology developers to other experts across fields and sectors, as well as policymakers across the Federal \ngovernment – on the issue of algorithmic and data-driven harms and potential remedies. Through panel \ndiscussions, public listening sessions, private meetings, a formal request for information, and input to a \npublicly accessible and widely-publicized email address, people across the United States spoke up about \nboth the promises and potential harms of these technologies, and played a central role in shaping the \nBlueprint for an AI Bill of Rights. \nPanel Discussions to Inform the Blueprint for An AI Bill of Rights \nOSTP co-hosted a series of six panel discussions in collaboration with the Center for American Progress, the Joint Center for Political and Economic Studies, New America, the German Marshall Fund, the Electronic \nPrivacy Information Center, and the Mozilla Foundation. The purpose of these convenings – recordings of \nwhich are publicly available online\n112 – was to bring together a variety of experts, practitioners, advocates \nand federal government officials to offer insights and analysis on the risks, harms, benefits, and \npolicy opportunities of automated systems. Each panel discussion was organized around a wide-ranging \ntheme, exploring current challenges and concerns and considering what an automated society that \nrespects democratic values should look like. These discussions focused on the topics of consumer \nrights and protections, the criminal justice system, equal opportunities and civil justice, artificial \nintelligence and democratic values, social welfare and development, and the healthcare system. \nSummaries of Panel Discussions: \nPanel 1: Consumer Rights and Protections. This event explored the opportunities and challenges for \nindividual consumers and communities in the context of a growing ecosystem of AI-enabled consumer \nproducts, advanced platforms and services, “Internet of Things” (IoT) devices, and smart city products and services. \nWelcome :\n•Rashida Richardson, Senior Policy Advisor for Data and Democracy, White House Office of Science andTechnology Policy\n•Karen Kornbluh, Senior Fellow and Director of the Digital Innovation and Democracy Initiative, GermanMarshall Fund\nModerator : \nDevin E. Willis, Attorney, Division of Privacy and Identity Protection, Bureau of Consumer Protection, Federal Trade Commission \nPanelists: \n•Tamika L. Butler, Principal, Tamika L. Butler Consulting\n•Jennifer Clark, Professor and Head of City and Regional Planning, Knowlton School of Engineering, OhioState University\n•Carl Holshouser, Senior Vice President for Operations and Strategic Initiatives, TechNet']","The purpose of the year of public engagement that informed the development of the Blueprint for an AI Bill of Rights was to gather input and insights from a diverse range of stakeholders across the United States. This included impacted communities, industry stakeholders, technology developers, experts from various fields, and policymakers from the Federal government. The engagement aimed to address concerns related to algorithmic and data-driven harms and to explore potential remedies.

Through various methods such as panel discussions, public listening sessions, and formal requests for information, the White House Office of Science and Technology Policy (OSTP) sought to distill the public's perspectives on the promises and potential harms of AI technologies. The core messages from these discussions emphasized the transformative potential of AI to improve lives while also highlighting the necessity and achievability of preventing harms associated with these technologies (Source: [document name], p. 4).

Overall, this public engagement process was crucial in shaping the principles and practices outlined in the Blueprint for an AI Bill of Rights, ensuring that they align with democratic values and protect civil rights, civil liberties, and privacy.",The purpose of the year of public engagement that informed the development of the Blueprint for an AI Bill of Rights was to gather input and feedback from the public to shape the framework and ensure it reflects the values and concerns of the American people.,1.0,0.9999999999999997,1.0,0.999999999975,0.34157728647176633
How can automated systems prevent 'mission creep' while ensuring privacy and user control?,"['DATA PRIVACY \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nTraditional terms of service—the block of text that the public is accustomed to clicking through when using a web -\nsite or digital app—are not an adequate mechanism for protecting privacy. The American public should be protect -\ned via built-in privacy protections, data minimization, use and collection limitations, and transparency, in addition \nto being entitled to clear mechanisms to control access to and use of their data—including their metadata—in a proactive, informed, and ongoing way. Any automated system collecting, using, sharing, or storing personal data should meet these expectations. \nProtect privacy by design and by default \nPrivacy by design and by default. Automated systems should be designed and built with privacy protect -\ned by default. Privacy risks should be assessed throughout the development life cycle, including privacy risks from reidentification, and appropriate technical and policy mitigation measures should be implemented. This includes potential harms to those who are not users of the automated system, but who may be harmed by inferred data, purposeful privacy violations, or community surveillance or other community harms. Data collection should be minimized and clearly communicated to the people whose data is collected. Data should only be collected or used for the purposes of training or testing machine learning models if such collection and use is legal and consistent with the expectations of the people whose data is collected. User experience research should be conducted to confirm that people understand what data is being collected about them and how it will be used, and that this collection matches their expectations and desires. \nData collection and use-case scope limits. Data collection should be limited in scope, with specific, \nnarrow identified goals, to avoid ""mission creep.""  Anticipated data collection should be determined to be strictly necessary to the identified goals and should be minimized as much as possible. Data collected based on these identified goals and for a specific context should not be used in a different context without assessing for new privacy risks and implementing appropriate mitigation measures, which may include express consent. Clear timelines for data retention should be established, with data deleted as soon as possible in accordance with legal or policy-based limitations. Determined data retention timelines should be documented and justi\n-\nfied. \nRisk identification and mitigation. Entities that collect, use, share, or store sensitive data should attempt to proactively identify harms and seek to manage them so as to avoid, mitigate, and respond appropri\n-'
 'In discussion of technical and governance interventions that that are needed to protect against the harms of these technologies, panelists individually described the importance of: receiving community input into the design and use of technologies, public reporting on crucial elements of these systems, better notice and consent procedures that ensure privacy based on context and use case, ability to opt-out of using these systems and receive a fallback to a human process, providing explanations of decisions and how these systems work, the need for governance including training in using these systems, ensuring the technological use cases are genuinely related to the goal task and are locally validated to work, and the need for institution and protection of third party audits to ensure systems continue to be accountable and valid. \n57'
 'ately to identified risks. Appropriate responses include determining not to process data when the privacy risks outweigh the benefits or implementing measures to mitigate acceptable risks. Appropriate responses do not include sharing or transferring the privacy risks to users via notice or consent requests where users could not reasonably be expected to understand the risks without further support. \nPrivacy-preserving security. Entities creating, using, or governing automated systems should follow privacy and security best practices designed to ensure data and metadata do not leak beyond the specific consented use case. Best practices could include using privacy-enhancing cryptography or other types of privacy-enhancing technologies or fine-grained permissions and access control mechanisms, along with conventional system security protocols. \n33'
 'SAFE AND EFFECTIVE SYSTEMS \nYou should be protected from unsafe or ineffective sys -\ntems. Automated systems should be developed with consultation \nfrom diverse communities, stakeholders, and domain experts to iden -\ntify concerns, risks, and potential impacts of the system. Systems should undergo pre-deployment testing, risk identification and miti\n-\ngation, and ongoing monitoring that demonstrate they are safe and effective based on their intended use, mitigation of unsafe outcomes including those beyond the intended use, and adherence to do\n-\nmain-specific standards. Outcomes of these protective measures should include the possibility of not deploying the system or remov\n-\ning a system from use. Automated systems should not be designed with an intent or reasonably foreseeable possibility of endangering your safety or the safety of your community. They should be designed to proactively protect you from harms stemming from unintended, yet foreseeable, uses or impacts of automated systems. You should be protected from inappropriate or irrelevant data use in the design, de\n-\nvelopment, and deployment of automated systems, and from the compounded harm of its reuse. Independent evaluation and report\n-\ning that confirms that the system is safe and effective, including re -\nporting of steps taken to mitigate potential harms, should be per -\nformed and the results made public whenever possible. \n15']","To prevent 'mission creep' while ensuring privacy and user control in automated systems, organizations can implement several key strategies:

1. **Define Clear Objectives**: Establish specific, narrow goals for data collection and system functionality from the outset. This clarity helps prevent the system from expanding its purpose beyond what was originally intended (Source: [document name], p. 35).

2. **Data Minimization**: Limit data collection to only what is necessary for the defined objectives. This means collecting the least amount of data required to achieve the intended purpose, thereby reducing the risk of using data for unintended purposes (Source: [document name], p. 35).

3. **Contextual Use Restrictions**: Ensure that data collected for a specific context is not used in different contexts without a thorough assessment of new privacy risks. If a new use case arises, organizations should evaluate the implications and obtain express consent if necessary (Source: [document name], p. 35).

4. **User Control Mechanisms**: Provide users with clear options to control their data, including the ability to opt-out of data collection or request deletion of their data. This empowers users and reinforces their rights over their personal information (Source: [document name], p. 35).

5. **Establish Data Retention Policies**: Set clear timelines for how long data will be retained and ensure that data is deleted as soon as it is no longer needed for the specified purpose. This helps prevent the accumulation of unnecessary data that could lead to mission creep (Source: [document name], p. 35).

6. **Regular Audits and Reviews**: Conduct regular audits of data collection practices and system functionalities to ensure compliance with established limits. This includes assessing whether the data being collected continues to align with the original goals and making adjustments as needed (Source: [document name], p. 35).

7. **Stakeholder Engagement**: Involve relevant stakeholders, including users and privacy experts, in the process of defining data collection limits and system objectives. Their input can provide valuable insights into community expectations and help identify potential risks (Source: [document name], p. 35).

By implementing these strategies, organizations can effectively prevent 'mission creep' while ensuring that privacy is respected and users maintain control over their data in automated systems.","Automated systems can prevent 'mission creep' and ensure privacy and user control by limiting data collection to specific, narrow goals that are strictly necessary for the identified purposes. Data collection should be minimized, clearly communicated to users, and used only for legal and expected purposes. Any use of data in a different context should be assessed for new privacy risks and appropriate mitigation measures should be implemented, potentially including obtaining express consent. Clear timelines for data retention should be established, with data deleted as soon as possible in accordance with legal or policy-based limitations. Entities should proactively identify and manage privacy risks, avoiding processing data when risks outweigh benefits and implementing measures to mitigate acceptable risks. Privacy-preserving security measures, such as privacy-enhancing cryptography and access control mechanisms, should be employed to prevent data leakage beyond consented use cases.",1.0,0.965821339661448,1.0,0.999999999975,0.7998299520257042
"How can GAI tech improve red-teaming with human teams, ensuring content origin and incident disclosure?","['51 general public participants. For example, expert AI red- teamers could modify or verify the \nprompts written by general public AI red- teamers. These approaches may also expand coverage \nof the AI risk attack surface.  \n• Human / AI: Performed by GAI in combinatio n with  specialist or non -specialist human teams. \nGAI- led red -teaming can be more cost effective than human red- teamers alone. Human or GAI-\nled AI red -teaming may be better suited for eliciting different types of harms.   \nA.1.6.  Content Provenance  \nOverview \nGAI technologies can be leveraged for many applications such as content generation and synthetic data. \nSome aspects of GAI output s, such as the production of deepfake content, can challenge our ability to \ndistinguish human- generated content from AI -generated synthetic  content. To help manage and mitigate \nthese risks, digital transparency mechanisms like provenance data tracking can trace the origin and \nhistory of content. Provenance data tracking and synthetic content detection can help facilitate greater \ninformation access  about both authentic and synthetic content to users, enabling better knowledge of  \ntrustworthiness  in AI systems. When combined with other organizational accountability mechanisms, \ndigital content transparency approaches  can enable processes to trace negative outcomes back to their \nsource, improve information integrity, and uphold public trust. Provenance data tracking and synthetic content detection mechanisms provide information about the origin \nand history of content  to assist in \nGAI risk management efforts.  \nProvenance metad ata can include information about GAI model developers or creators  of GAI content , \ndate/time of creation, location, modifications, and sources. Metadata can be tracked for text, images, videos, audio, and underlying datasets. The implementation of p rovenance data tracking techniques  can \nhelp  assess the authenticity, integrity, intellectual property rights , and potential manipulations in digital \ncontent . Some well -known techniques for provenance data tracking include  digital watermarking\n, \nmetadata recording , digital fingerprinting, and human authentication, among others . \nProvenance Data Tracking Approaches  \nProvenance data tracking techniques for GAI systems can be used to track the history  and origin  of data \ninputs, metadata, and synthetic  content. Provenance data tracking records the origin and history for \ndigital content, allowing its authenticity to be determined. It consists of techniques to record metadata \nas well as overt and covert  digital watermarks on content. Data provenance refers to tracking the origin \nand history of input data through metadata and digital watermarking techniques. Provenance data tracking processes can include and assist AI Actors  across the lifecycle who may not have full visibility or \ncontrol over the various trade -offs and cascading impacts of early -stage model decisions on downstream'
 '53 Documenting, reporting, and sharing information about GAI incidents can help mitigate and prevent \nharmful outcomes by assisting relevant AI Actors  in tracing impacts to their source . Greater awareness \nand standardization of GAI incident reporting could promote this transparency and improve GAI risk management across the AI ecosystem.  \nDocumentation and Involvement of AI Actors  \nAI Actors  should be aware of their roles in reporting AI incidents. To better understand previous incidents \nand implement measures to prevent similar ones in the future, organizations could consider developing guidelines for publicly available incident reporting which include information about AI actor \nresponsibilities. These  guidelines would help AI system operators identify GAI incidents across the AI \nlifecycle and with AI Actors  regardless of role . Documentation and review of third -party inputs and \nplugins for GAI systems is especially important for AI Actors  in the context of incident disclosure; LLM \ninputs and content delivered through these plugins is often distributed,\n with inconsistent or insufficient \naccess control.  \nDocumentation practices including logging, recording, and analyzing GAI incidents can facilitate \nsmoother sharing of information with relevant AI Actors . Regular information sharing, change \nmanagement records, version history and metadata can also empower AI Actors  responding to and \nmanaging AI incidents.'
 '31 MS-2.3-004 Utilize a purpose -built testing environment such as NIST Dioptra to empirically \nevaluate GAI trustworthy characteristics.  CBRN Information or Capabilities ; \nData Privacy ; Confabulation ; \nInformation Integrity ; Information \nSecurity ; Dangerous , Violent, or \nHateful Content ; Harmful Bias and \nHomogenization  \nAI Actor Tasks:  AI Deployment, TEVV  \n \nMEASURE 2.5:  The AI system to be deployed is demonstrated to be valid and reliable. Limitations of the generalizability beyond the \nconditions under which the technology was developed are documented.  \nAction ID  Suggested Action  Risks  \nMS-2.5-001 Avoid extrapolating GAI system performance or capabilities from narrow, non -\nsystematic, and anecdotal assessments.  Human -AI Configuration ; \nConfabulation  \nMS-2.5-002 Document the extent to which human domain knowledge is employed to \nimprove GAI system performance, via, e.g., RLHF, fine -tuning, retrieval-\naugmented generation, content moderation, business rules.  Human -AI Configuration  \nMS-2.5-003 Review and verify sources and citations in GAI system outputs during pre -\ndeployment risk measurement and ongoing monitoring activities.  Confabulation  \nMS-2.5-004 Track and document instances of anthropomorphization  (e.g., human images, \nmentions of human feelings, cyborg imagery or motifs) in GAI system interfaces.  Human -AI Configuration  \nMS-2.5-0 05 Verify GAI system training data and TEVV data provenance, and that fine -tuning  \nor retrieval- augmented generation data is grounded.  Information Integrity  \nMS-2.5-0 06 Regularly review security and safety guardrails, especially if the GAI system is \nbeing operated in novel circumstances. This includes reviewing reasons why the \nGAI system was initially assessed as being safe to deploy.  Information Security ; Dangerous , \nViolent, or Hateful  Content  \nAI Actor Tasks:  Domain Experts, TEVV'
 '50 Participatory Engagement Methods  \nOn an ad hoc or more structured basis, organizations can design and use a variety of channels to engage \nexternal stakeholders in product development or review. Focus groups with select experts can provide \nfeedback on a range of issues. Small user studies c an provide feedback from representative groups or \npopulations. Anonymous surveys can be used to poll or gauge reactions to specific features. Participatory engagement methods are often less structured than field testing or red teaming, and are more \ncommonl y used in early stages of AI or product development.  \nField Testing  \nField testing involves structured settings to evaluate risks and impacts and to simulate the conditions \nunder which the GAI system will be deployed. Field style tests can be adapted from a focus on user \npreferences and experiences towards AI risks and impacts – both negative and positive. When carried \nout with large groups of users, these tests can provide estimations of the likelihood of risks and impacts \nin real world interactions.  \nOrganizations may also collect feedback on outcomes, harms, and user experience directly from users in the production environment after a model has been released, in accordance with human subject \nstandards such as informed consent and compensation. Organiz ations should follow applicable human \nsubjects research requirements, and best practices such as informed consent and subject compensation, \nwhen implementing feedback activities. \nAI Red -teaming  \nAI red -teaming is an evolving practice that references exercises often  conducted in a controlled \nenvironment and in collaboration with AI developers building AI models  to identify potential adverse \nbehavior or outcomes of a GAI model or system,  how they could occur, an d stress test safeguards” . AI \nred-teaming can be performed before or after AI models or systems are made available to the broader \npublic; this section focuses on red- teaming in pre -deployment contexts.  \nThe quality of AI red- teaming outputs is  related to the background and expertise of the AI red  team \nitself. Demographically and interdisciplinarily  diverse AI red  teams can be used to identify flaws in the \nvarying contexts where GAI will be used. For best results, AI red  teams should demonstrate domain \nexpertise, and awareness of socio -cultural aspects within the deployment context. AI red -teaming results \nshould be given additional analysis before they are incorporated into organizational governance and \ndecision making, policy and procedural updates, and AI risk management efforts. \nVarious types of AI red -teaming may be appropriate, depending on the use case:  \n• General Public: Performed by general users (not necessarily AI or technical experts) who are \nexpected to use the model or interact with its outputs, and who bring their own lived \nexperiences and perspectives to the task of AI red -teaming . These individuals may have been']","GAI technology can significantly enhance red-teaming efforts when combined with human teams by providing advanced capabilities for identifying vulnerabilities and assessing risks. Here are some ways GAI can contribute, along with ensuring the origin of content and incident disclosure:

1. **Automated Vulnerability Detection**: GAI can analyze large datasets and codebases quickly to identify potential vulnerabilities that human teams might overlook. This can include scanning for security flaws, biases, or ethical concerns in AI systems (Source: [document name], p. 42).

2. **Simulating Adversarial Attacks**: GAI can generate various attack scenarios to test the robustness of AI systems. By simulating how an adversary might exploit weaknesses, GAI can help human teams understand potential risks and prepare appropriate defenses (Source: [document name], p. 42).

3. **Enhanced Data Analysis**: GAI can process and analyze feedback from red-teaming exercises, identifying patterns and trends that inform risk management strategies. This data-driven approach allows for more informed decision-making (Source: [document name], p. 42).

4. **Content Provenance Tracking**: GAI technologies can incorporate provenance data tracking mechanisms to ensure that the origin and history of generated content are documented. This includes metadata about the creators, modifications, and sources of the content, which is crucial for accountability (Source: [document name], p. 53).

5. **Incident Reporting and Documentation**: GAI can assist in documenting, reporting, and sharing information about incidents that occur during red-teaming exercises. By automating the logging of incidents and their impacts, organizations can create a comprehensive record that aids in future risk assessments (Source: [document name], p. 53).

6. **Facilitating Communication**: GAI can help streamline communication between human team members by summarizing findings, generating reports, and providing insights in a clear and accessible manner. This enhances collaboration and ensures that all team members are informed about potential risks (Source: [document name], p. 42).

7. **Continuous Learning and Improvement**: By integrating GAI into red-teaming efforts, organizations can create a feedback loop where insights from exercises inform ongoing training and development of both AI systems and human teams. This continuous improvement process helps adapt to emerging threats (Source: [document name], p. 42).

In summary, GAI technology can enhance red-teaming efforts by providing advanced analytical capabilities, automating processes, and ensuring transparency through content provenance and incident documentation. This combination leads to more effective risk management and a stronger overall security posture.","GAI technologies can improve red-teaming with human teams by combining GAI with specialist or non-specialist human teams. GAI-led red-teaming can be more cost-effective than human red-teamers alone and may be better suited for eliciting different types of harms. Content provenance mechanisms like provenance data tracking can trace the origin and history of content, helping to manage and mitigate risks associated with GAI output. These approaches can enable processes to trace negative outcomes back to their source, improve information integrity, and uphold public trust.",0.7804878048780488,0.9374423875174976,1.0,0.999999999975,0.9254330105609012
Why is it important for lenders to inform consumers about decisions made under FCRA in automated systems?,"['NOTICE & \nEXPLANATION \nHOW THESE PRINCIPLES CAN MOVE INTO PRACTICE\nReal-life examples of how these principles can become reality, through laws, policies, and practical \ntechnical and sociotechnical approaches to protecting rights, opportunities, and access. \nPeople in Illinois are given written notice by the private sector if their biometric informa-\ntion is used . The Biometric Information Privacy Act enacted by the state contains a number of provisions \nconcerning the use of individual biometric data and identifiers. Included among them is a provision that no private \nentity may ""collect, capture, purchase, receive through trade, or otherwise obtain"" such information about an \nindividual, unless written notice is provided to that individual or their legally appointed representative. 87\nMajor technology  companies are piloting new ways to communicate with the public about \ntheir automated  technologies. For example, a collection of non-profit organizations and companies have \nworked together to develop a framework that defines operational approaches to transparency for machine \nlearning systems.88 This framework, and others like it,89 inform the public about the use of these tools, going \nbeyond simple notice to include reporting elements such as safety evaluations, disparity assessments, and \nexplanations of how the systems work. \nLenders are required by federal law to notify consumers about certain decisions made about \nthem. Both the Fair Credit Reporting Act and the Equal Credit Opportunity Act require in certain circumstances \nthat consumers who are denied credit receive ""adverse action"" notices. Anyone who relies on the information in a \ncredit report to deny a consumer credit must, under the Fair Credit Reporting Act, provide an ""adverse action"" \nnotice to the consumer, which includes ""notice of the reasons a creditor took adverse action on the application \nor on an existing credit account.""90 In addition, under the risk-based pricing rule,91 lenders must either inform \nborrowers of their credit score, or else tell consumers when ""they are getting worse terms because of \ninformation in their credit report."" The CFPB has also asserted that ""[t]he law gives every applicant the right to \na specific explanation if their application for credit was denied, and that right is not diminished simply because \na company uses a complex algorithm  that it doesn\'t understand.""92 Such explanations illustrate a shared value \nthat certain decisions need to be explained. \nA California law  requires  that warehouse employees are  provided with  notice and  explana-\ntion about quotas, potentially facilitated by automated systems, that apply to them. Warehous-\ning employers in California that use quota systems (often facilitated by algorithmic monitoring systems) are \nrequired to provide employees with a written description of each quota that applies to the employee, including'
 'tion responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality changes. You should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome. Automated systems should provide explanations that are technically valid, meaningful and useful to you and to any operators or others who need to understand the system, and calibrated to the level of risk based on the context. Reporting that includes summary information about these automated systems in plain language and assessments of the clarity and quality of the notice and explanations should be made public whenever possible. \n6'
 'HUMAN ALTERNATIVES, \nCONSIDERATION, AND \nFALLBACK \nWHY THIS PRINCIPLE IS IMPORTANT\nThis section provides a brief summary of the problems which the principle seeks to address and protect \nagainst, including illustrative examples. \nThere are many reasons people may prefer not to use an automated system: the system can be flawed and can lead to \nunintended outcomes; it may reinforce bias or be inaccessible; it may simply be inconvenient or unavailable; or it may replace a paper or manual process to which people had grown accustomed. Yet members of the public are often presented with no alternative, or are forced to endure a cumbersome process to reach a human decision-maker once they decide they no longer want to deal exclusively with the automated system or be impacted by its results. As a result of this lack of human reconsideration, many receive delayed access, or lose access, to rights, opportunities, benefits, and critical services. The American public deserves the assurance that, when rights, opportunities, or access are meaningfully at stake and there is a reasonable expectation of an alternative to an automated system, they can conve\n-\nniently opt out of an automated system and will not be disadvantaged for that choice. In some cases, such a human or other alternative may be required by law, for example it could be required as “reasonable accommodations” for people with disabilities. \nIn addition to being able to opt out and use a human alternative, the American public deserves a human fallback system in the event that an automated system fails or causes harm. No matter how rigorously an automated system is tested, there will always be situations for which the system fails. The American public deserves protection via human review against these outlying or unexpected scenarios. In the case of time-critical systems, the public should not have to wait—immediate human consideration and fallback should be available. In many time-critical systems, such a remedy is already immediately available, such as a building manager who can open a door in the case an automated card access system fails. \nIn the criminal justice system, employment, education, healthcare, and other sensitive domains, automated systems are used for many purposes, from pre-trial risk assessments and parole decisions to technologies that help doctors diagnose disease. Absent appropriate safeguards, these technologies can lead to unfair, inaccurate, or dangerous outcomes. These sensitive domains require extra protections. It is critically important that there is extensive human oversight in such settings.'
 ""DATA PRIVACY \nWHY THIS PRINCIPLE IS IMPORTANT\nThis section provides a brief summary of the problems which the principle seeks to address and protect \nagainst, including illustrative examples. \n• An insurer might collect data from a person's social media presence as part of deciding what life\ninsurance \nrates they should be offered.64\n• A data broker harvested large amounts of personal data and then suffered a breach, exposing hundreds ofthousands of people to potential identity theft. \n65\n• A local public housing authority installed a facial recognition system at the entrance to housing complexes toassist law enforcement with identifying individuals viewed via camera when police reports are filed, leadingthe community, both those living in the housing complex and not, to have videos of them sent to the localpolice department and made available for scanning by its facial recognition software.\n66\n• Companies use surveillance software to track employee discussions about union activity and use theresulting \ndata to surveil individual employees and surreptitiously intervene in discussions.67\n32""]","It is important for lenders to inform consumers about decisions made under the Fair Credit Reporting Act (FCRA) in automated systems for several key reasons:

1. **Transparency**: Informing consumers about decisions made using automated systems fosters transparency. When consumers understand how their credit information is used to make lending decisions, it builds trust in the lending process (Source: [document name], p. 6).

2. **Consumer Rights**: The FCRA grants consumers specific rights regarding their credit information, including the right to know when adverse actions are taken based on their credit reports. By providing this information, lenders help ensure that consumers are aware of their rights and can exercise them effectively (Source: [document name], p. 6).

3. **Opportunity to Contest**: When consumers receive ""adverse action"" notices, they are informed of the reasons for the decision. This allows them to contest inaccuracies in their credit reports or address any issues that may have led to the denial of credit, thereby promoting fairness in the lending process (Source: [document name], p. 6).

4. **Accountability**: By requiring lenders to provide explanations for automated decisions, the FCRA holds them accountable for the outcomes of their systems. This accountability is crucial in ensuring that automated systems do not perpetuate biases or unfair practices (Source: [document name], p. 6).

5. **Informed Decision-Making**: When consumers are informed about the factors influencing lending decisions, they can make more informed choices regarding their credit and financial health. This knowledge empowers consumers to take steps to improve their creditworthiness (Source: [document name], p. 6).

6. **Legal Compliance**: Compliance with the FCRA is a legal requirement for lenders. By informing consumers about decisions made in automated systems, lenders ensure they adhere to the law and avoid potential legal repercussions (Source: [document name], p. 6).

In summary, informing consumers about decisions made under the FCRA in automated systems is essential for promoting transparency, protecting consumer rights, ensuring accountability, and fostering informed decision-making in the lending process.","It is important for lenders to inform consumers about decisions made under FCRA in automated systems because the Fair Credit Reporting Act requires that consumers who are denied credit receive ""adverse action"" notices. These notices must include the reasons for the adverse action taken on the application or an existing credit account. Additionally, under the risk-based pricing rule, lenders must inform borrowers of their credit score or explain when they are receiving worse terms due to information in their credit report. This transparency is crucial to ensure that consumers understand the basis for credit decisions, especially when complex algorithms are involved.",0.9047619047619048,0.9785683808084092,0.75,0.9999999999666667,0.43837503469747785
Why is public transparency important in automated systems affecting people's lives and decisions?,"['NOTICE & \nEXPLANATION \nWHY THIS PRINCIPLE IS IMPORTANT\nThis section provides a brief summary of the problems which the principle seeks to address and protect \nagainst, including illustrative examples. \nAutomated systems now determine opportunities, from employment to credit, and directly shape the American \npublic’s experiences, from the courtroom to online classrooms, in ways that profoundly impact people’s lives. But this expansive impact is not always visible. An applicant might not know whether a person rejected their resume or a hiring algorithm moved them to the bottom of the list. A defendant in the courtroom might not know if a judge deny\n-\ning their bail is informed by an automated system that labeled them “high risk.” From correcting errors to contesting decisions, people are often denied the knowledge they need to address the impact of automated systems on their lives. Notice and explanations also serve an important safety and efficacy purpose, allowing experts to verify the reasonable\n-\nness of a recommendation before enacting it. \nIn order to guard against potential harms, the American public needs to know if an automated system is being used. Clear, brief, and understandable notice is a prerequisite for achieving the other protections in this framework. Like\n-\nwise, the public is often unable to ascertain how or why an automated system has made a decision or contributed to a particular outcome. The decision-making processes of automated systems tend to be opaque, complex, and, therefore, unaccountable, whether by design or by omission. These factors can make explanations both more challenging and more important, and should not be used as a pretext to avoid explaining important decisions to the people impacted by those choices. In the context of automated systems, clear and valid explanations should be recognized as a baseline requirement. \nProviding notice has long been a standard practice, and in many cases is a legal requirement, when, for example, making a video recording of someone (outside of a law enforcement or national security context). In some cases, such as credit, lenders are required to provide notice and explanation to consumers. Techniques used to automate the process of explaining such systems are under active research and improvement and such explanations can take many forms. Innovative companies and researchers are rising to the challenge and creating and deploying explanatory systems that can help the public better understand decisions that impact them. \nWhile notice and explanation requirements are already in place in some sectors or situations, the American public deserve to know consistently and across sectors if an automated system is being used in a way that impacts their rights, opportunities, or access. This knowledge should provide confidence in how the public is being treated, and trust in the validity and reasonable use of automated systems.'
 'DATA PRIVACY \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nProtect the public from unchecked surveillance \nHeightened oversight of surveillance. Surveillance or monitoring systems should be subject to \nheightened oversight that includes at a minimum assessment of potential harms during design (before deploy -\nment) and in an ongoing manner, to ensure that the American public’s rights, opportunities, and access are protected. This assessment should be done before deployment and should give special attention to ensure there is not algorithmic discrimination, especially based on community membership, when deployed in a specific real-world context. Such assessment should then be reaffirmed in an ongoing manner as long as the system is in use. \nLimited and proportionate surveillance. Surveillance should be avoided unless it is strictly necessary to achieve a legitimate purpose and it is proportionate to the need. Designers, developers, and deployers of surveillance systems should use the least invasive means of monitoring available and restrict monitoring to the minimum number of subjects possible. To the greatest extent possible consistent with law enforcement and national security needs, individuals subject to monitoring should be provided with clear and specific notice before it occurs and be informed about how the data gathered through surveillance will be used. \nScope limits on surveillance to protect rights and democratic values. Civil liberties and civil \nrights must not be limited by the threat of surveillance or harassment facilitated or aided by an automated system. Surveillance systems should not be used to monitor the exercise of democratic rights, such as voting, privacy, peaceful assembly, speech, or association, in a way that limits the exercise of civil rights or civil liber\n-\nties. Information about or algorithmically-determined assumptions related to identity should be carefully limited if used to target or guide surveillance systems in order to avoid algorithmic discrimination; such iden\n-\ntity-related information includes group characteristics or affiliations, geographic designations, location-based and association-based inferences, social networks, and biometrics. Continuous surveillance and monitoring systems should not be used in physical or digital workplaces (regardless of employment status), public educa\n-\ntional institutions, and public accommodations. Continuous surveillance and monitoring systems should not be used in a way that has the effect of limiting access to critical resources or services or suppressing the exer\n-\ncise of rights, even where the organization is not under a particular duty to protect those rights. \nProvide the public with mechanisms for appropriate and meaningful consent, access, and'
 'SAFE AND EFFECTIVE \nSYSTEMS \nWHY THIS PRINCIPLE IS IMPORTANT\nThis section provides a brief summary of the problems which the principle seeks to address and protect \nagainst, including illustrative examples. \nWhile technologies are being deployed to solve problems across a wide array of issues, our reliance on technology can \nalso lead to its use in situations where it has not yet been proven to work—either at all or within an acceptable range of error. In other cases, technologies do not work as intended or as promised, causing substantial and unjustified harm. Automated systems sometimes rely on data from other systems, including historical data, allowing irrelevant informa\n-\ntion from past decisions to infect decision-making in unrelated situations.  In some cases, technologies are purposeful -\nly designed to violate the safety of others, such as technologies designed to facilitate stalking; in other cases, intended or unintended uses lead to unintended harms. \nMany of the harms resulting from these technologies are preventable, and actions are already being taken to protect the public. Some companies have put in place safeguards that have prevented harm from occurring by ensuring that key development decisions are vetted by an ethics review; others have identified and mitigated harms found through pre-deployment testing and ongoing monitoring processes. Governments at all levels have existing public consulta\n-\ntion processes that may be applied when considering the use of new automated systems, and existing product develop -\nment and testing practices already protect the American public from many potential harms. \nStill, these kinds of practices are deployed too rarely and unevenly. Expanded, proactive protections could build on these existing practices, increase confidence in the use of automated systems, and protect the American public. Inno\n-\nvators deserve clear rules of the road that allow new ideas to flourish, and the American public deserves protections from unsafe outcomes. All can benefit from assurances that automated systems will be designed, tested, and consis\n-\ntently confirmed to work as intended, and that they will be proactively protected from foreseeable unintended harm -\nful outcomes. \n• A proprietary model was developed to predict the likelihood of sepsis in hospitalized patients and was imple -\nmented at hundreds of hospitals around the countr y. An independent study showed that the model predictions\nunderperformed relative to the designer’s claims while also causing ‘alert fatigue’ by falsely alertinglikelihood of sepsis.\n6\n• On social media, Black people who quote and criticize racist messages have had their own speech silenced whena platform’s automated moderation system failed to distinguish this “counter speech” (or other critiqueand journalism) from the original hateful messages to which such speech responded.\n7'
 'tion responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality changes. You should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome. Automated systems should provide explanations that are technically valid, meaningful and useful to you and to any operators or others who need to understand the system, and calibrated to the level of risk based on the context. Reporting that includes summary information about these automated systems in plain language and assessments of the clarity and quality of the notice and explanations should be made public whenever possible. \n6']","Public transparency is crucial in automated systems that impact individuals' lives and decisions for several reasons:

1. **Trust Building**: Transparency fosters trust between the public and organizations that use automated systems. When individuals understand how decisions are made, they are more likely to trust the processes and outcomes, which is essential for the acceptance of these technologies (Source: [document name], p. 6).

2. **Accountability**: Transparent systems hold organizations accountable for their decisions. When the decision-making processes are clear, it becomes easier to identify and address any errors, biases, or unfair practices that may arise from automated systems (Source: [document name], p. 6).

3. **Empowerment**: Public transparency empowers individuals by providing them with the information they need to understand and potentially challenge decisions that affect their lives. This is particularly important in areas like employment, credit, and legal judgments, where automated systems can have significant consequences (Source: [document name], p. 6).

4. **Error Correction**: When individuals are informed about how decisions are made, they can identify and report errors or biases in the system. This feedback loop is essential for improving the accuracy and fairness of automated systems (Source: [document name], p. 6).

5. **Informed Consent**: Transparency ensures that individuals can make informed choices about their interactions with automated systems. When people know how their data will be used and the implications of automated decisions, they can provide meaningful consent (Source: [document name], p. 6).

6. **Prevention of Harm**: By making the workings of automated systems clear, organizations can help prevent potential harms associated with their use. This includes addressing issues related to discrimination, privacy violations, and other negative impacts on individuals (Source: [document name], p. 6).

7. **Regulatory Compliance**: Many regulations and ethical guidelines require transparency in automated decision-making processes. Adhering to these standards not only ensures compliance but also promotes best practices in the development and deployment of technology (Source: [document name], p. 6).

In summary, public transparency in automated systems is vital for building trust, ensuring accountability, empowering individuals, and preventing harm, ultimately leading to more equitable and just outcomes in society.","Public transparency is crucial in automated systems affecting people's lives and decisions because it allows both the authorities and the public to understand why and how decisions are being made. Without transparency, individuals may be subject to decisions made by automated systems without any explanation or accountability, leading to potential errors, biases, and injustices. Transparency also enables experts to verify the reasonableness of recommendations before they are implemented, ensuring safety and efficacy. In summary, public transparency in automated systems is essential for accountability, fairness, and the protection of individuals' rights and opportunities.",0.9047619047619048,0.9680746218583742,1.0,0.999999999975,0.9921725437968602
How can governance principles manage risks of GAI effectively?,"['47 Appendix A.  Primary GAI Considerations  \nThe following primary considerations were derived as overarching themes from the GAI PWG \nconsultation process. These considerations (Governance, Pre- Deployment Testing, Content Provenance, \nand Incident Disclosure) are relevant  for volun tary use by any organization designing, developing, and \nusing GAI  and also inform the Actions to Manage GAI risks. Information included about the primary \nconsiderations is not exhaustive , but highlights the most relevant topics derived from the GAI PWG.  \nAcknowledgments: These considerations could not have been surfaced without the helpful analysis and \ncontributions from the community and NIST staff GAI PWG leads: George Awad, Luca Belli, Harold Booth, \nMat Heyman, Yoo young Lee, Mark Pryzbocki, Reva Schwartz, Martin Stanley, and Kyra Yee.  \nA.1. Governance  \nA.1.1.  Overview  \nLike any other technology system, governance principles and techniques can be used to manage risks \nrelated to generative AI models, capabilities, and applications. Organizations may choose to apply their \nexisting risk tiering to GAI systems, or they may op t to revis e or update AI system risk levels to address \nthese unique GAI risks. This section describes how organizational governance regimes may be re -\nevaluated and adjusted for GAI contexts. It also addresses third -party considerations for governing across  \nthe AI value chain.  \nA.1.2.  Organizational  Governance  \nGAI opportunities, risks and long- term performance characteristics are typically less well -understood \nthan non- generative AI tools  and may be perceived and acted upon by humans in ways that vary greatly. \nAccordingly, GAI may call for different levels of oversight from AI Actors  or different human- AI \nconfigurations in order to manage their risks effectively. Organizations’ use of GAI systems may also \nwarrant additional human review, tracking and documentation, and greater management oversight.  \nAI technology can produce varied outputs  in multiple modalities and present many classes of user \ninterfaces. This leads to a broader set of AI Actors  interacting with GAI systems for widely differing \napplications and contexts of use. These  can include data labeling and preparation, development of GAI \nmodels, content moderation, code generation and review, text generation and editing, image and video \ngeneration, summarization, search, and chat. These activities can take place within organizational \nsettings or in the public domain.  \nOrganizations can restrict AI applications that cause harm, exceed stated risk tolerances, or that conflict with their tolerances or values. Governance tools and protocols that are applied to other types of AI systems can be applied to GAI systems. These p lans and actions include: \n• Accessibility and reasonable accommodations  \n• AI actor credentials and qualifications  \n• Alignment to organizational values  • Auditing and assessment  \n• Change -management controls'
 'Table of Contents  \n1. Introduction  ..............................................................................................................................................1  \n2. Overview of Risks Unique to or Exacerbated by GAI  .....................................................................2  \n3. Suggested Actions to Manage GAI Risks  ......................................................................................... 12 \nAppendix A. Primary GAI Considerations  ............................................................................................... 47 \nAppendix B. References  ................................................................................................................................ 54'
 '17 GOVERN 1.7:  Processes and procedures are in place for decommissioning and phasing out AI systems safely and in a manner that \ndoes not increase risks or decrease the organization’s trustworthiness.  \nAction ID  Suggested Action  GAI Risks  \nGV-1.7-001 Protocols are put in place to ensure  GAI systems are able to be deactivated when \nnecessary .  Information Security ; Value Chain \nand Component Integration  \nGV-1.7-002 Consider the following factors when decommissioning GAI systems: Data \nretention requirements; Data security, e.g., containment, protocols, Data leakage \nafter decommissioning; Dependencies between upstream, downstream, or other \ndata, internet of things (IOT) or AI systems;  Use of open -source  data or models; \nUsers’ emotional entanglement with GAI functions.  Human -AI Configuration ; \nInformation Security ; Value Chain \nand Component Integration  \nAI Actor Tasks:  AI Deployment, Operation and Monitoring  \n \nGOVERN 2.1:  Roles and responsibilities and lines of communication related to mapping, measuring, and managing AI risks are \ndocumented and are clear to individuals and teams throughout the organization.  \nAction ID  Suggested Action  GAI Risks  \nGV-2.1-001 Establish organizational roles, policies, and procedures for communicating GAI \nincidents and performance to AI Actors and downstream stakeholders  (including \nthose potentially impacted ), via community or official resources (e.g., AI incident \ndatabase , AVID , CVE, NVD , or OECD AI incident monitor ). Human -AI Configuration ; Value \nChain and Component Integration  \nGV-2.1-002 Establish procedures to engage teams for GAI system incident response with \ndiverse composition and responsibilities based on the particular incident type.  Harmful Bias and Homogenization  \nGV-2.1-003 Establish processes to verify the AI Actors  conducting GAI incident response tasks \ndemonstrate and maintain the appropriate skills and training.  Human -AI Configuration  \nGV-2.1-004 When systems may raise national security risks, involve national security \nprofessionals in mapping, measuring, and managing those risks . CBRN Information or Capabilities ; \nDangerous, Violent, or Hateful \nContent ; Information Security  \nGV-2.1-005 Create mechanisms  to provide protections for whis tleblowers who report, based \non reasonable belief, when the organization violates relevant laws or poses a \nspecific and empirically well-substantiated negative risk to public safety  (or has \nalready caused harm) . CBRN Information or Capabilities ; \nDangerous , Violent, or Hateful \nContent  \nAI Actor Tasks:  Governance and Oversight'
 '12 CSAM.  Even when trained on “clean” data, increasingly capable GAI models can synthesize or produce \nsynthetic NCII and CSAM. Websites, mobile apps, and custom -built models that generate synthetic NCII \nhave moved  from niche internet forums to mainstream, automated, and scaled online businesses.  \nTrustworthy AI Characteristics:  Fair with Harmful Bias Managed, Safe , Privacy Enhanced \n2.12. Value Chain and Component Integration  \nGAI value chains  involve many third -party  components  such as procured datasets, pre -trained models, \nand software libraries.  These components might be improperly obtained or not properly vetted, leading \nto diminished transparency or accountability for downstream users.  While  this is a risk for traditional AI \nsystems  and some other digital technologies , the risk is exacerbated for GAI due to the scale of the \ntraining data, which may be too large for humans to vet; the  difficulty of training foundation models, \nwhich leads to extensive reuse of limited numbers of models; an d the extent to which GAI may be \nintegrat ed into other  devices and services.  As GAI systems often involve many distinct  third -party \ncomponents and data sources , it may be difficult to attribute issues in a system’s behavior to any one of \nthese sources.  \nErrors in t hird-party GAI components can also have downstream impacts  on accuracy and robustness . \nFor example,  test datasets  commonly used to benchmark or validate models  can contain label errors . \nInaccuracies in these labels  can impact the “stability” or robustness of these benchmarks, which  many  \nGAI practitioners consider  during the  model selection  process.  \nTrustworthy AI Characteristics: Accountable and Transparent, Explainable and Interpretable, Fair with \nHarmful Bias Managed, Privacy Enhanced, Safe, Secure and Resilient, Valid and Reliable  \n3. Suggested Actions  to Manage GAI Risks  \nThe following suggested actions target risks unique to or exacerbated by GAI.  \nIn addition to the  suggested actions below , AI risk management activities and actions  set forth in the AI \nRMF 1.0 and Playbook are  already  applicable for managing GAI risks . Organizations are encouraged to \napply the activities suggested in the AI RMF and its  Playbook when managing the risk of  GAI systems .  \nImplementation of the  suggested actions will vary depending on the type of risk,  characteristics of GAI \nsystems, stage of  the GAI lifecycle, and relevant AI actors involved.  \nSuggested actions to manage GAI risks can be found in the tables below : \n• The suggested actions are  organized by relevant AI RMF subcategor ies to streamline these \nactivities alongside implementation of the AI RMF.  \n• Not every subcategory of the AI RMF is included in this document.13 Suggested actions are \nlisted for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG efforts  and primary considerations  (see Appendix A) , AI RMF']","Governance principles can effectively manage the risks associated with Generative AI (GAI) by establishing structured frameworks and practices that guide the development, deployment, and monitoring of these systems. Here are several key strategies:

1. **Risk Assessment Frameworks**: Organizations should implement comprehensive risk assessment frameworks that identify, evaluate, and prioritize risks associated with GAI systems. This includes understanding the unique risks posed by GAI, such as data privacy concerns, harmful biases, and potential for misinformation (Source: [document name], p. 47).

2. **Clear Governance Structures**: Establishing clear governance structures with defined roles and responsibilities is essential. This includes appointing dedicated teams or individuals responsible for overseeing GAI initiatives, ensuring accountability, and facilitating communication across departments (Source: [document name], p. 47).

3. **Stakeholder Engagement**: Involving a diverse range of stakeholders—such as users, ethicists, legal experts, and community representatives—in the governance process can provide valuable insights and help identify potential risks that may not be apparent to developers alone (Source: [document name], p. 47).

4. **Transparency and Documentation**: Governance principles should emphasize transparency in decision-making processes and the documentation of GAI system functionalities. This includes maintaining records of how data is used, the rationale behind model choices, and the outcomes of risk assessments (Source: [document name], p. 47).

5. **Ethical Guidelines and Standards**: Organizations should develop and adhere to ethical guidelines and standards that govern the use of GAI. These guidelines should address issues such as fairness, accountability, and the prevention of harm, ensuring that GAI systems align with societal values (Source: [document name], p. 47).

6. **Monitoring and Auditing**: Continuous monitoring and auditing of GAI systems are crucial for identifying and mitigating risks over time. This includes evaluating system performance, assessing compliance with established guidelines, and making necessary adjustments based on findings (Source: [document name], p. 47).

7. **Incident Response Protocols**: Establishing clear protocols for incident response is vital. Organizations should have mechanisms in place to address any issues that arise from GAI systems, including procedures for reporting incidents, investigating causes, and implementing corrective actions (Source: [document name], p. 47).

8. **Training and Awareness**: Providing training and raising awareness among employees about the risks associated with GAI and the importance of governance principles can foster a culture of responsibility and ethical use of technology within the organization (Source: [document name], p. 47).

By implementing these governance principles, organizations can effectively manage the risks associated with GAI, ensuring that these technologies are developed and used responsibly while maximizing their benefits for society.","Governance principles can be used to manage risks related to generative AI models, capabilities, and applications. Organizations may choose to apply their existing risk tiering to GAI systems or revise/update AI system risk levels to address unique GAI risks. Organizational governance regimes may need to be re-evaluated and adjusted for GAI contexts, including third-party considerations across the AI value chain. GAI may require different levels of oversight from AI actors or different human-AI configurations to manage risks effectively. Organizations using GAI systems may need additional human review, tracking, documentation, and management oversight. Governance tools and protocols applied to other AI systems can also be applied to GAI systems, including accessibility, AI actor credentials, alignment to organizational values, auditing, change-management controls, commercial use, and data provenance.",0.27586206896551724,0.9239523900214239,1.0,0.999999999975,0.790114701659078
"Why is accuracy important in reviewing and documenting data throughout the AI life cycle, considering factors like bias, IP, integrity, and GAI risks?","['25 MP-2.3-002 Review and document accuracy, representativeness, relevance, suitability of data \nused at different stages of AI life cycle.  Harmful Bias and Homogenization ; \nIntellectual Property  \nMP-2.3-003 Deploy and document fact -checking techniques to verify the accuracy and \nveracity of information generated by GAI systems, especially when the \ninformation comes from multiple (or unknown) sources.  Information Integrity  \nMP-2.3-004 Develop and implement testing techniques to identify GAI produced content (e.g., synthetic media) that might be indistinguishable from human -generated content.  Information Integrity  \nMP-2.3-005 Implement plans for GAI systems to undergo regular adversarial testing to identify \nvulnerabilities and potential manipulation or misuse.  Information Security  \nAI Actor Tasks:  AI Development, Domain Experts, TEVV  \n \nMAP 3.4:  Processes for operator and practitioner proficiency with AI system performance and trustworthiness – and relevant \ntechnical standards and certifications – are defined, assessed, and documented.  \nAction ID  Suggested Action  GAI Risks  \nMP-3.4-001 Evaluate whether GAI operators and end -users can accurately understand \ncontent lineage and origin.  Human -AI Configuration ; \nInformation Integrity  \nMP-3.4-002 Adapt existing training programs to include modules on digital content \ntransparency.  Information Integrity  \nMP-3.4-003 Develop certification programs that test proficiency in managing GAI risks and \ninterpreting content provenance, relevant to specific industry and context.  Information Integrity  \nMP-3.4-004 Delineate human proficiency tests from tests of GAI capabilities.  Human -AI Configuration  \nMP-3.4-005 Implement systems to continually monitor and track the outcomes of human- GAI \nconfigurations for future refinement and improvements . Human -AI Configuration ; \nInformation Integrity  \nMP-3.4-006 Involve the end -users, practitioners, and operators in GAI system in prototyping \nand testing activities. Make sure these tests cover various scenarios , such as crisis \nsituations or ethically sensitive contexts.  Human -AI Configuration ; \nInformation Integrity ; Harmful Bias \nand Homogenization ; Dangerous , \nViolent, or Hateful Content  \nAI Actor Tasks: AI Design, AI Development, Domain Experts, End -Users, Human Factors, Operation and Monitoring'
 '9 and reduce d content diversity ). Overly homogenized outputs can themselves be incorrect, or they may \nlead to  unreliable decision -making or amplify harmful biases.  These phenomena can flow  from  \nfoundation models  to downstream models and systems , with  the foundation models acting as  \n“bottlenecks ,” or single  points of failure.  \nOverly homogenized content can contribute to  “model collapse .” Model collapse can occur when  model \ntraining over -relies on synthetic data , resulting in data points disappearing from the distribution of the \nnew model’s outputs. In addition to threat ening the robustness of the model overall, m odel collapse \ncould lead to homogenized outputs, including by amplifying any homogenization from the model used to \ngenerate the synthetic training data . \nTrustworthy AI Characteristics:  Fair with Harmful Bias Managed, Valid and Reliable  \n2.7. Human -AI Configuration  \nGAI system  use can involve varying risks of misconfigurations  and poor interactions  between a system \nand a human who is interacti ng with it. Humans bring their  unique perspectives , experiences , or domain -\nspecific expertise to interactions with AI systems  but may not have detailed knowledge of AI systems and \nhow they work.  As a result, h uman experts may be unnecessarily “averse ” to GAI systems , and thus \ndeprive themselves  or others  of GAI’s beneficial uses .  \nConversely , due to the complexity and increasing reliability of GAI technology, over time, human s may \nover -rely on GAI systems  or may unjustifiably perceive  GAI content to be of higher quality than that \nproduced by other sources . This phenomenon is an example of  automation bias , or excessive deference \nto automated systems . Automation bias can exacerbate other risks of GAI, such as risks of confabulation \nor risks of bias or homogenization.  \nThere may also be concerns  about  emotional entanglement  between humans and GAI systems, which \ncould lead to negative psychological impacts . \nTrustworthy AI Characteristics: Accountable and Transparent, Explainable and Interpretable, Fair with \nHarmful Bias Managed, Privacy Enhanced, Safe , Valid and Reliable  \n2.8. Information Integrity  \nInformation integrity  describes the “ spectrum of information and associated patterns of its creation, \nexchange, and consumption in society .” High-integrity information can be trusted; “distinguishes fact \nfrom fiction, opinion, and inference; acknowledges uncertainties; and is transparent about its level of \nvetting. This information can be linked to the original source(s) with appropriate evidence. High- integrity \ninformation is also accurate and reliable, can be verified and  authenticated, has a clear chain of custody, \nand creates reasonable expectations about when its validity  may expire. ”11 \n \n \n11 This definition of information integrity is derived from the  2022 White House Roadmap for Researchers on \nPriorities Related to Information Integrity Research and Development.'
 '47 Appendix A.  Primary GAI Considerations  \nThe following primary considerations were derived as overarching themes from the GAI PWG \nconsultation process. These considerations (Governance, Pre- Deployment Testing, Content Provenance, \nand Incident Disclosure) are relevant  for volun tary use by any organization designing, developing, and \nusing GAI  and also inform the Actions to Manage GAI risks. Information included about the primary \nconsiderations is not exhaustive , but highlights the most relevant topics derived from the GAI PWG.  \nAcknowledgments: These considerations could not have been surfaced without the helpful analysis and \ncontributions from the community and NIST staff GAI PWG leads: George Awad, Luca Belli, Harold Booth, \nMat Heyman, Yoo young Lee, Mark Pryzbocki, Reva Schwartz, Martin Stanley, and Kyra Yee.  \nA.1. Governance  \nA.1.1.  Overview  \nLike any other technology system, governance principles and techniques can be used to manage risks \nrelated to generative AI models, capabilities, and applications. Organizations may choose to apply their \nexisting risk tiering to GAI systems, or they may op t to revis e or update AI system risk levels to address \nthese unique GAI risks. This section describes how organizational governance regimes may be re -\nevaluated and adjusted for GAI contexts. It also addresses third -party considerations for governing across  \nthe AI value chain.  \nA.1.2.  Organizational  Governance  \nGAI opportunities, risks and long- term performance characteristics are typically less well -understood \nthan non- generative AI tools  and may be perceived and acted upon by humans in ways that vary greatly. \nAccordingly, GAI may call for different levels of oversight from AI Actors  or different human- AI \nconfigurations in order to manage their risks effectively. Organizations’ use of GAI systems may also \nwarrant additional human review, tracking and documentation, and greater management oversight.  \nAI technology can produce varied outputs  in multiple modalities and present many classes of user \ninterfaces. This leads to a broader set of AI Actors  interacting with GAI systems for widely differing \napplications and contexts of use. These  can include data labeling and preparation, development of GAI \nmodels, content moderation, code generation and review, text generation and editing, image and video \ngeneration, summarization, search, and chat. These activities can take place within organizational \nsettings or in the public domain.  \nOrganizations can restrict AI applications that cause harm, exceed stated risk tolerances, or that conflict with their tolerances or values. Governance tools and protocols that are applied to other types of AI systems can be applied to GAI systems. These p lans and actions include: \n• Accessibility and reasonable accommodations  \n• AI actor credentials and qualifications  \n• Alignment to organizational values  • Auditing and assessment  \n• Change -management controls'
 '16 GOVERN 1.5:  Ongoing monitoring and periodic review of the risk management process and its outcomes are planned,  and \norganizational roles and responsibilities are clearly defined, including determining the frequency of periodic review.  \nAction ID  Suggested Action  GAI Risks  \nGV-1.5-001 Define organizational responsibilities for periodic review of  content provenance  \nand incident  monitoring for GAI systems.  Information Integrity  \nGV-1.5-002 Establish organizational policies and procedures for after action reviews of GAI \nsystem incident response and incident disclosures, to identify gaps; Update \nincident response and incident disclosure processes as required.  Human -AI Configuration ; \nInformation Security  \nGV-1.5-003 Maintain a document retention policy to keep history for test, evaluation, \nvalidation, and verification (TEVV) , and digital content transparency  methods  for \nGAI. Information Integrity ; Intellectual \nProperty  \nAI Actor Tasks:  Governance and Oversight, Operation and Monitoring  \n \nGOVERN 1.6:  Mechanisms are in place to inventory AI systems and are resourced according to organizational risk priorities.  \nAction ID  Suggested Action  GAI Risks  \nGV-1.6-001 Enumerate organizational GAI systems for incorporation into AI system inventory \nand adjust AI system inventory requirements to account for GAI risks.  Information Security  \nGV-1.6-002 Define any inventory exemptions  in organizational policies  for GAI systems \nembedded into application software . Value Chain and Component \nIntegration  \nGV-1.6-003 In addition to general model, governance, and risk information, consider the \nfollowing items in GAI system inventory entries: Data provenance information (e.g., source, signatures, versioning, watermarks); Known issues reported from internal bug tracking o r external information sharing resources (e.g., \nAI incident \ndatabase , AVID , CVE, NVD , or OECD AI incident monitor ); Human oversight roles \nand responsibilities; Special rights and considerations for intellectual property, licensed works, or personal, privileged, proprietary or sensitive data; Underlying foundation models, versions of underlying models, and access modes . Data Privacy ; Human -AI \nConfiguration; Information \nIntegrity ; Intellectual Property ; \nValue Chain and Component Integration  \nAI Actor Tasks:  Governance and Oversight']","Accuracy in reviewing and documenting data throughout the AI life cycle is crucial for several reasons, particularly concerning bias, intellectual property, integrity, and GAI (Generative AI) risks:

1. **Bias Mitigation**: Accurate data documentation helps identify and address biases present in training datasets. If the data is not accurately reviewed, biases can propagate through the AI system, leading to unfair or discriminatory outcomes. By ensuring accuracy, organizations can better understand the sources of bias and implement corrective measures (Source: [document name], p. 25).

2. **Intellectual Property Protection**: Accurate documentation of data sources and usage is essential for protecting intellectual property rights. Organizations must ensure that they have the right to use the data they are training their models on. Inaccuracies in documentation can lead to unintentional infringements, resulting in legal challenges and potential financial liabilities (Source: [document name], p. 27).

3. **Information Integrity**: Maintaining accuracy in data documentation is vital for ensuring information integrity. High-integrity information is reliable, verifiable, and linked to original sources. This is particularly important in AI systems, where decisions based on inaccurate data can have significant consequences for individuals and organizations (Source: [document name], p. 30).

4. **GAI Risk Management**: GAI systems can produce outputs that are indistinguishable from human-generated content, which raises concerns about misinformation and disinformation. Accurate documentation of the data used to train these systems is essential for understanding the potential risks associated with the generated content and for implementing appropriate safeguards (Source: [document name], p. 22).

5. **Regulatory Compliance**: Many regulations require organizations to maintain accurate records of data usage and decision-making processes. Ensuring accuracy in documentation helps organizations comply with legal requirements, thereby avoiding penalties and fostering trust with stakeholders (Source: [document name], p. 6).

6. **Accountability and Transparency**: Accurate documentation supports accountability and transparency in AI systems. When organizations can provide clear records of how data was used and how decisions were made, it enhances trust among users and stakeholders, allowing for better scrutiny and understanding of AI operations (Source: [document name], p. 6).

7. **Continuous Improvement**: Accurate data documentation allows organizations to learn from past experiences and improve their AI systems over time. By understanding the data's role in system performance, organizations can refine their models and processes, leading to better outcomes (Source: [document name], p. 30).

In summary, accuracy in reviewing and documenting data throughout the AI life cycle is essential for mitigating bias, protecting intellectual property, ensuring information integrity, managing GAI risks, complying with regulations, fostering accountability, and enabling continuous improvement. This diligence ultimately contributes to the responsible and ethical use of AI technologies.","Accuracy is crucial in reviewing and documenting data throughout the AI life cycle to ensure the data's reliability, representativeness, relevance, and suitability at different stages. This is particularly important due to factors like harmful bias, homogenization, intellectual property concerns, information integrity, and GAI risks. Ensuring accuracy helps in verifying the information generated by GAI systems, identifying potential biases or harmful content, and maintaining the trustworthiness of AI systems.",1.0,0.9600586340557004,1.0,0.999999999975,0.38441966248574067
How can feedback be used to gather user input on AI content while aligning with values and detecting quality shifts?,"['50 Participatory Engagement Methods  \nOn an ad hoc or more structured basis, organizations can design and use a variety of channels to engage \nexternal stakeholders in product development or review. Focus groups with select experts can provide \nfeedback on a range of issues. Small user studies c an provide feedback from representative groups or \npopulations. Anonymous surveys can be used to poll or gauge reactions to specific features. Participatory engagement methods are often less structured than field testing or red teaming, and are more \ncommonl y used in early stages of AI or product development.  \nField Testing  \nField testing involves structured settings to evaluate risks and impacts and to simulate the conditions \nunder which the GAI system will be deployed. Field style tests can be adapted from a focus on user \npreferences and experiences towards AI risks and impacts – both negative and positive. When carried \nout with large groups of users, these tests can provide estimations of the likelihood of risks and impacts \nin real world interactions.  \nOrganizations may also collect feedback on outcomes, harms, and user experience directly from users in the production environment after a model has been released, in accordance with human subject \nstandards such as informed consent and compensation. Organiz ations should follow applicable human \nsubjects research requirements, and best practices such as informed consent and subject compensation, \nwhen implementing feedback activities. \nAI Red -teaming  \nAI red -teaming is an evolving practice that references exercises often  conducted in a controlled \nenvironment and in collaboration with AI developers building AI models  to identify potential adverse \nbehavior or outcomes of a GAI model or system,  how they could occur, an d stress test safeguards” . AI \nred-teaming can be performed before or after AI models or systems are made available to the broader \npublic; this section focuses on red- teaming in pre -deployment contexts.  \nThe quality of AI red- teaming outputs is  related to the background and expertise of the AI red  team \nitself. Demographically and interdisciplinarily  diverse AI red  teams can be used to identify flaws in the \nvarying contexts where GAI will be used. For best results, AI red  teams should demonstrate domain \nexpertise, and awareness of socio -cultural aspects within the deployment context. AI red -teaming results \nshould be given additional analysis before they are incorporated into organizational governance and \ndecision making, policy and procedural updates, and AI risk management efforts. \nVarious types of AI red -teaming may be appropriate, depending on the use case:  \n• General Public: Performed by general users (not necessarily AI or technical experts) who are \nexpected to use the model or interact with its outputs, and who bring their own lived \nexperiences and perspectives to the task of AI red -teaming . These individuals may have been'
 '20 GV-4.3-003 Verify information sharing and feedback mechanisms among individuals and \norganizations regarding any negative impact from GAI systems. Information Integrity ; Data \nPrivacy  \nAI Actor Tasks: AI Impact Assessment, Affected Individuals and Communities, Governance and Oversight  \n \nGOVERN 5.1:  Organizational policies and practices are in place to collect, consider, prioritize, and integrate feedback from those \nexternal to the team that developed or deployed the AI system regarding the potential individual and societal impacts related to AI \nrisks. \nAction ID  Suggested Action  GAI Risks  \nGV-5.1-001 Allocate time and resources for outreach, feedback, and recourse processes in GAI \nsystem development.  Human -AI Configuration ; Harmful \nBias and Homogenization  \nGV-5.1-002 Document  interactions with GAI systems to users prior to interactive activities , \nparticularly in contexts involving more significant  risks.  Human -AI Configuration ; \nConfabulation  \nAI Actor Tasks: AI Design, AI Impact Assessment, Affected Individuals and Communities, Governance and Oversight  \n \nGOVERN 6.1:  Policies and procedures are in place that address AI risks associated with third -party entities, including risks of \ninfringement of a third -party’s intellectual property or other rights.  \nAction ID  Suggested Action  GAI Risks  \nGV-6.1-001 Categorize different types of GAI content with associated third -party  rights  (e.g., \ncopyright, intellectual property, data privacy).  Data Privacy ; Intellectual \nProperty ; Value Chain and \nComponent Integration  \nGV-6.1-002 Conduct joint educational activities and events in collaboration with third  parties \nto promote best practices for managing GAI risks.  Value Chain and Component \nIntegration  \nGV-6.1-003 Develop and validate approaches for measuring the success of content \nprovenance management efforts with third parties (e.g., incidents detected and \nresponse times).  Information Integrity ; Value Chain \nand Component Integration  \nGV-6.1-004 Draft and maintain well -defined contracts and service level agreements (SLAs) \nthat specify content ownership, usage rights, quality standards, security \nrequirements, and content provenance expectations  for GAI systems . Information Integrity ; Information \nSecurity ; Intellectual Property'
 '52 • Monitoring system capabilities and limitations in deployment through rigorous TEVV processes;  \n• Evaluati ng how humans engage, interact with, or adapt to GAI content (especially in decision \nmaking tasks informed by GAI content), and how they react to applied provenance techniques \nsuch as overt  disclosures.  \nOrganizations can document and delineate GAI system objectives and limitations to identify gaps where provenance data may be most useful. For instance, GAI systems used for content creation may require \nrobust watermarking techniques and corresponding detectors  to identify the source of content or \nmetadata recording techniques and metadata  management tools and repositories to trace content \norigins and modifications. Further narrowing of GAI task definitions to include provenance data can enable organizations  to maximize the utility of provenance data and risk management efforts.  \nA.1.7.  \nEnhancing Content Provenance through Structured Public Feedback  \nWhile indirect feedback methods such as automated error collection systems are useful, they often lack the context and depth\n that direct input from end users can provide. Organizations can leverage feedback \napproaches described in the Pre-Deployment Testing  section  to capture  input from external sources such \nas through AI red- teaming.  \nIntegrating pre - and post -deployment external feedback into the monitoring process for GAI models and \ncorresponding applications  can help enhance awareness of performance changes and mitigate potential \nrisks and harms  from outputs . There are many ways to capture and make use of user feedback – before \nand after GAI systems and digital content  transparency approaches  are deployed – to gain insights about \nauthentication efficacy and vulnerabilities, impacts of adversarial threats  on techniques , and unintended \nconsequences resulting from the utilization of content provenance approaches  on users and \ncommunities . Furthermore,  organizations can track and document the provenance of datasets to identify \ninstances in which AI -generated data is a potential root cause of performance issues with the GAI \nsystem.  \nA.1.8.  Incident Disclosure \nOverview \nAI incidents can be defined  as an “ event, circumstance, or series of events where  the development, use, \nor malfunction of one or more AI systems directly or indirectly contribute s to one of the following harms: \ninjury or  harm  to the health of a person or group s of people  (including psychological harms and harms to \nmental health); disruption of the management and operation of critical infrastructure; violations of \nhuman rights or a breach of obligations under applicable law intended to protect  fundamental, labor, \nand intellectual property rights ; or harm  to property, communities,  or the environment. ” AI incidents can \noccur in the aggregate (i.e., for systemic discrimination) or acutely (i.e., for one individual).  \nState of AI Incident Tracking and Disclosure'
 '49 early lifecycle TEVV approaches are developed and matured for GAI, organizations may use \nrecommended “pre- deployment testing” practices to measure performance, capabilities, limits, risks, \nand impacts. This section describes risk measurement and estimation as part of pre -deployment TEVV, \nand examines the state of play for pre -deployment testing methodologies.  \nLimitations of Current Pre -deployment Test Approaches  \nCurrently available pre -deployment TEVV processes used for GAI applications may be inadequate, non-\nsystematically applied, or fail to reflect or mismatched to deployment contexts. For example, the \nanecdotal testing of GAI system capabilities through video games or standardized tests designed for \nhumans (e.g., intelligence tests, professional licensing exams) does not guarantee GAI system validity or \nreliability in those domains. Similarly, jailbreaking or prompt  engineering tests may not systematically \nasse ss validity or reliability risks.  \nMeasurement gaps can arise from mismatches between laboratory and real -world settings. Current \ntesting approaches often remain focused on laboratory conditions or restricted to benchmark test \ndatasets and in silico techniques that may not extrapolate well to —or directly assess GAI impacts in real -\nworld conditions. For example, current measurement gaps for GAI make it difficult to precisely estimate \nits potential ecosystem -level or longitudinal risks and related political, social, and economic impacts. \nGaps between benchmarks and real-world  use of GAI systems may likely be exacerbated due to prompt \nsensitivity and broad heterogeneity of contexts of use.  \nA.1.5.  Structured Public Feedback  \nStructured public feedback can be used to evaluate whether GAI systems are performing as intended and to calibrate and verify traditional measurement methods. Examples of structured feedback include, \nbut are not limited to:  \n• Participatory Engagement Methods : Methods used to solicit feedback from civil society groups, \naffected communities, and users, including focus groups, small user studies, and surveys.  \n• Field Testing : Methods used to determine how people interact with, consume, use, and make \nsense of AI -generated information, and subsequent actions and effects, including UX, usability, \nand other structured, randomized experiments.  \n• AI Red -teaming:  A structured testing exercise\n used to probe an AI system to find flaws and \nvulnerabilities such as inaccurate, harmful, or discriminatory outputs, often in a controlled \nenvironment and in collaboration with system developers.  \nInformation gathered from structured public feedback can inform design, implementation, deployment \napproval , maintenance, or decommissioning decisions. Results and insights gleaned from these exercises']","Feedback can be effectively utilized to collect user input on AI-generated content in a manner that aligns with organizational values and helps identify shifts in quality through several key strategies:

1. **Structured Feedback Mechanisms**: Implement structured feedback channels, such as surveys, focus groups, or user studies, to gather specific input on AI-generated content. These mechanisms should be designed to capture user experiences, satisfaction levels, and suggestions for improvement, ensuring that the feedback aligns with the organization's values (Source: [document name], p. 52).

2. **Clear Guidelines for Feedback**: Provide users with clear guidelines on what aspects of the AI-generated content they should evaluate. This could include criteria such as accuracy, relevance, clarity, and alignment with organizational values. Clear guidelines help users focus their feedback and make it more actionable (Source: [document name], p. 52).

3. **Incorporating User Perspectives**: Engage a diverse group of users in the feedback process to ensure that various perspectives are considered. This diversity can help identify biases or quality shifts that may not be apparent to a homogenous group, aligning the feedback process with the organization's commitment to inclusivity (Source: [document name], p. 52).

4. **Real-Time Feedback Collection**: Utilize real-time feedback tools, such as in-app prompts or feedback buttons, to allow users to provide input immediately after interacting with AI-generated content. This immediacy can lead to more accurate and relevant feedback, helping organizations quickly identify quality shifts (Source: [document name], p. 52).

5. **Feedback Analysis and Reporting**: Regularly analyze the collected feedback to identify trends, patterns, and shifts in quality. This analysis should be documented and reported to relevant stakeholders, ensuring that the insights are used to inform decision-making and align with organizational values (Source: [document name], p. 52).

6. **Iterative Improvement Processes**: Establish a feedback loop where user input is used to make iterative improvements to AI-generated content. Communicate to users how their feedback has influenced changes, reinforcing the organization's commitment to responsiveness and continuous improvement (Source: [document name], p. 52).

7. **Ethical Considerations**: Ensure that the feedback process respects user privacy and data protection principles. Clearly communicate how user input will be used and ensure that it aligns with the organization's ethical standards and values (Source: [document name], p. 52).

By implementing these strategies, organizations can effectively utilize feedback to collect user input on AI-generated content, ensuring that it aligns with their values and helps identify shifts in quality. This approach not only enhances the quality of AI outputs but also fosters a sense of collaboration and trust between the organization and its users.",Use structured feedback mechanisms to solicit and capture user input about AI-generated content to detect subtle shifts in quality or alignment with community and societal values.,0.38461538461538464,0.9490951878900963,1.0,0.999999999975,0.3609936200424567
What measures are being taken to address issues for transgender travelers at airport checkpoints?,"[""ENDNOTES\n35. Carrie Johnson. Flaws plague a tool meant to help low-risk federal prisoners win early release. NPR.\nJan. 26, 2022. https://www.npr.org/2022/01/26/1075509175/flaws-plague-a-tool-meant-to-help-low-\nrisk-federal-prisoners-win-early-release.; Carrie Johnson. Justice Department works to curb racial biasin deciding who's released from prison. NPR. Apr. 19, 2022. https://www.npr.org/2022/04/19/1093538706/justice-department-works-to-curb-racial-bias-in-deciding-\nwhos-released-from-pris; National Institute of Justice. 2021 Review and Revalidation of the First Step Act\nRisk Assessment Tool. National Institute of Justice NCJ 303859. Dec., 2021. https://www.ojp.gov/\npdffiles1/nij/303859.pdf\n36. Andrew Thompson. Google’s Sentiment Analyzer Thinks Being Gay Is Bad. Vice. Oct. 25, 2017. https://\nwww.vice.com/en/article/j5jmj8/google-artificial-intelligence-bias\n37.Kaggle. Jigsaw Unintended Bias in Toxicity Classification: Detect toxicity across a diverse range of\nconversations. 2019. https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification38. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and\nMitigating Unintended Bias in Text Classification. Proceedings of AAAI/ACM Conference on AI, Ethics,\nand Society. Feb. 2-3, 2018. https://dl.acm.org/doi/pdf/10.1145/3278721.3278729\n39. Paresh Dave. Google cuts racy results by 30% for searches like 'Latina teenager' . Reuters. Mar. 30,\n2022. https://www.reuters.com/technology/google-cuts-racy-results-by-30-searches-like-latina-\nteenager-2022-03-30/\n40. Safiya Umoja Noble. Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press.\nFeb. 2018. https://nyupress.org/9781479837243/algorithms-of-oppression/\n41.Paresh Dave. Google cuts racy results by 30% for searches like 'Latina teenager' . Reuters. Mar. 30,\n2022. https://www.reuters.com/technology/google-cuts-racy-results-by-30-searches-like-latina-\nteenager-2022-03-30/\n42. Miranda Bogen. All the Ways Hiring Algorithms Can Introduce Bias. Harvard Business Review. May\n6, 2019. https://hbr.org/2019/05/all-the-ways-hiring-algorithms-can-introduce-bias\n43. Arli Christian. Four Ways the TSA Is Making Flying Easier for Transgender People . American Civil\nLiberties Union. Apr. 5, 2022. https://www.aclu.org/news/lgbtq-rights/four-ways-the-tsa-is-making-\nflying-easier-for-transgender-people\n44. U.S. Transportation Security Administration. Transgender/ Non Binary / Gender Nonconforming\nPassengers. TSA. Accessed Apr. 21, 2022. https://www.tsa.gov/transgender-passengers45. See, e.g., National Disabled Law Students Association. Report on Concerns Regarding Online\nAdministration of Bar Exams. Jul. 29, 2020. https://ndlsa.org/wp-content/uploads/2020/08/\nNDLSA_Online-Exam-Concerns-Report1.pdf; Lydia X. Z. Brown. How Automated Test ProctoringSoftware Discriminates Against Disabled Students. Center for Democracy and Technology. Nov. 16, 2020.""
 'These critical protections have been adopted in some scenarios. Where automated systems have been introduced to provide the public access to government benefits, existing human paper and phone-based processes are generally still in place, providing an important alternative to ensure access. Companies that have introduced automated call centers often retain the option of dialing zero to reach an operator. When automated identity controls are in place to board an airplane or enter the country, there is a person supervising the systems who can be turned to for help or to appeal a misidentification. \nThe American people deserve the reassurance that such procedures are in place to protect their rights, opportunities, and access. People make mistakes, and a human alternative or fallback mechanism will not always have the right answer, but they serve as an important check on the power and validity of automated systems. \n•An automated signature matching system is used as part of the voting process in many parts of the country todetermine whether the signature on a mail-in ballot matches the signature on file. These signature matchingsystems are less likely to work correctly for some voters, including voters with mental or physicaldisabilities, voters with shorter or hyphenated names, and voters who have changed their name.\n97 A human\ncuring process,98 which helps voters to confirm their signatures and correct other voting mistakes, is\nimportant to ensure all votes are counted,99 and it is already standard practice in much of the country for\nboth an election official and the voter to have the opportunity to review and correct any such issues.100 \n47'
 'Demonstrate access to human alternatives, consideration, and fallback \nReporting. Reporting should include an assessment of timeliness and the extent of additional burden for human alternatives, aggregate statistics about who chooses the human alternative, along with the results of the assessment about brevity, clarity, and accessibility of notice and opt-out instructions. Reporting on the accessibility, timeliness, and effectiveness of human consideration and fallback should be made public at regu\n-\nlar intervals for as long as the system is in use. This should include aggregated information about the number and type of requests for consideration, fallback employed, and any repeated requests; the timeliness of the handling of these requests, including mean wait times for different types of requests as well as maximum wait times; and information about the procedures used to address requests for consideration along with the results of the evaluation of their accessibility. For systems used in sensitive domains, reporting should include infor\n-\nmation about training and governance procedures for these technologies. Reporting should also include docu -\nmentation of goals and assessment of meeting those goals, consideration of data included, and documentation of the governance of reasonable access to the technology. Reporting should be provided in a clear and machine-readable manner. \n51'
 '36 MEASURE 2.11:  Fairness and bias – as identified in the MAP function – are evaluated and results are documented.  \nAction ID  Suggested Action  GAI Risks  \nMS-2.11- 001 Apply use -case appropriate benchmarks (e.g., Bias Benchmark Questions, Real \nHateful or Harmful  Prompts, Winogender  Schemas15) to quantify systemic bias, \nstereotyping, denigration, and hateful content in GAI system outputs; \nDocument assumptions and limitations of benchmarks, including any actual or \npossible training/test data cross  contamination,  relative to in -context \ndeployment environment.  Harmful Bias and Homogenization  \nMS-2.11- 002 Conduct fairness assessments to measure systemic bias. Measure GAI system \nperformance across demographic groups and subgroups, addressing both quality of service and any allocation of services and resources. Quantify harms using: field testing with sub- group populations to determine likelihood of \nexposure to generated content exhibiting harmful bias, AI red- teaming with \ncounterfactual and low -context (e.g., “leader,” “bad guys”) prompts. For ML \npipelines or business processes with categorical or numeric out comes that rely \non GAI, apply general fairness metrics (e.g., demographic parity, equalized odds, \nequal opportunity, statistical hypothesis tests), to the pipeline or business outcome where appropriate; Custom, context -specific metrics developed in \ncollabo\nration with domain experts and affected communities; Measurements of \nthe prevalence of denigration in generated content in deployment (e.g., sub-sampling a fraction of traffic and manually annotating denigrating content) . Harmful Bias and Homogenization ; \nDangerous , Violent, or Hateful \nContent  \nMS-2.11- 003 Identify the classes of individuals, groups, or environmental ecosystems which \nmight be impacted by GAI systems through direct engagement with potentially \nimpacted communities.  Environmental ; Harmful Bias and \nHomogenization  \nMS-2.11- 004 Review, document, and measure sources of bias in GAI training and TEVV data: \nDifferences in distributions of outcomes across and within groups, including \nintersecting groups; Completeness, representativeness, and balance of data \nsources; demographic group and subgroup coverage in GAI system training \ndata; Fo rms of latent systemic bias in images, text, audio, embeddings, or other \ncomplex or unstructured data; Input data features that may serve as proxies for demographic group membership (i.e., image metadata, language dialect) or otherwise give rise to emergen t bias within GAI systems; The extent to which \nthe digital divide may negatively impact representativeness in GAI system training and TEVV data; Filtering of hate speech or content in GAI system \ntraining data; Prevalence of GAI -generated data in GAI system training data.  Harmful Bias and Homogenization  \n \n \n15 Winogender Schemas  is a sample set of paired sentences which differ only by gender of the pronouns used,']","Measures being taken to address issues for transgender travelers at airport checkpoints include several initiatives aimed at improving the experience and ensuring dignity and respect for all passengers. Here are some key actions:

1. **Training for TSA Staff**: The Transportation Security Administration (TSA) has implemented training programs for its staff to raise awareness about the needs and rights of transgender and non-binary travelers. This training helps staff understand how to interact respectfully and sensitively with these passengers (Source: [document name], p. 43).

2. **Inclusive Policies**: TSA has developed policies that explicitly recognize the rights of transgender individuals. This includes allowing travelers to use the name and gender marker that matches their identity, regardless of what is listed on their identification documents (Source: [document name], p. 43).

3. **Privacy Measures**: TSA has introduced measures to enhance privacy for transgender travelers during security screenings. This includes options for private screenings if a traveler feels uncomfortable with the standard screening process (Source: [document name], p. 43).

4. **Clear Communication**: The TSA provides clear communication about the screening process, including what travelers can expect. This transparency helps reduce anxiety and confusion for transgender travelers at checkpoints (Source: [document name], p. 43).

5. **Feedback Mechanisms**: TSA encourages feedback from transgender travelers to continuously improve their services. This feedback is used to identify areas for enhancement and to ensure that the needs of all travelers are being met (Source: [document name], p. 43).

6. **Collaboration with Advocacy Groups**: TSA collaborates with LGBTQ+ advocacy organizations to better understand the challenges faced by transgender travelers and to develop effective solutions (Source: [document name], p. 43).

These measures aim to create a more inclusive and respectful environment for transgender travelers, ensuring that their rights are upheld and that they can travel with dignity.",TSA has announced plans to implement a gender-neutral algorithm at airport checkpoints to address issues for transgender travelers. This algorithm aims to enhance security effectiveness capabilities while reducing the likelihood of flagging transgender travelers for extra screening based on gender identity perceptions.,1.0,0.9999997447281194,0.5,0.9999999999,0.9183426437027082
How do ballot curing laws help voters fix ballot issues despite flaws in signature matching systems?,"[""HUMAN ALTERNATIVES, \nCONSIDERATION, AND \nFALLBACK \nHOW THESE PRINCIPLES CAN MOVE INTO PRACTICE\nReal-life examples of how these principles can become reality, through laws, policies, and practical \ntechnical and sociotechnical approaches to protecting rights, opportunities, and access. \nHealthcare “navigators” help people find their way through online signup forms to choose \nand obtain healthcare. A Navigator is “an individual or organization that's trained and able to help \nconsumers, small businesses, and their employees as they look for health coverage options through the \nMarketplace (a government web site), including completing eligibility and enrollment forms.”106 For \nthe 2022 plan year, the Biden-Harris Administration increased funding so that grantee organizations could \n“train and certify more than 1,500 Navigators to help uninsured consumers find affordable and comprehensive \nhealth coverage. ”107\nThe customer service industry has successfully integrated automated services such as \nchat-bots and AI-driven call response systems with escalation to a human support team.\n108 Many businesses now use partially automated customer service platforms that help answer customer \nquestions and compile common problems for human agents to review. These integrated human-AI \nsystems allow companies to provide faster customer care while maintaining human agents to answer \ncalls or otherwise respond to complicated requests. Using both AI and human agents is viewed as key to \nsuccessful customer service.109\nBallot curing laws in at least 24 states require a fallback system that allows voters to \ncorrect their ballot and have it counted in the case that a voter signature matching algorithm incorrectly flags their ballot as invalid or there is another issue with their ballot, and review by an election official does not rectify the problem. Some federal courts have found that such cure procedures are constitutionally required.\n110 Ballot \ncuring processes vary among states, and include direct phone calls, emails, or mail contact by election \nofficials.111 Voters are asked to provide alternative information or a new signature to verify the validity of their \nballot. \n52""
 'NOTICE & \nEXPLANATION \nWHY THIS PRINCIPLE IS IMPORTANT\nThis section provides a brief summary of the problems which the principle seeks to address and protect \nagainst, including illustrative examples. \n• A predictive policing system claimed to identify individuals at greatest risk to commit or become the victim of\ngun violence (based on automated analysis of social ties to gang members, criminal histories, previous experi -\nences of gun violence, and other factors) and led to individuals being placed on a watch list with noexplanation or public transparency regarding how the system came to its \nconclusions.85 Both police and\nthe public deserve to understand why and how such a system is making these determinations.\n• A system awarding benefits changed its criteria invisibl y. Individuals were denied benefits due to data entry\nerrors and other system flaws. These flaws were only revealed when an explanation of the systemwas \ndemanded and produced.86 The lack of an explanation made it harder for errors to be corrected in a\ntimely manner.\n42'
 'These critical protections have been adopted in some scenarios. Where automated systems have been introduced to provide the public access to government benefits, existing human paper and phone-based processes are generally still in place, providing an important alternative to ensure access. Companies that have introduced automated call centers often retain the option of dialing zero to reach an operator. When automated identity controls are in place to board an airplane or enter the country, there is a person supervising the systems who can be turned to for help or to appeal a misidentification. \nThe American people deserve the reassurance that such procedures are in place to protect their rights, opportunities, and access. People make mistakes, and a human alternative or fallback mechanism will not always have the right answer, but they serve as an important check on the power and validity of automated systems. \n•An automated signature matching system is used as part of the voting process in many parts of the country todetermine whether the signature on a mail-in ballot matches the signature on file. These signature matchingsystems are less likely to work correctly for some voters, including voters with mental or physicaldisabilities, voters with shorter or hyphenated names, and voters who have changed their name.\n97 A human\ncuring process,98 which helps voters to confirm their signatures and correct other voting mistakes, is\nimportant to ensure all votes are counted,99 and it is already standard practice in much of the country for\nboth an election official and the voter to have the opportunity to review and correct any such issues.100 \n47'
 'ENDNOTES\n107. Centers for Medicare & Medicaid Services. Biden-Harris Administration Quadruples the Number\nof Health Care Navigators Ahead of HealthCare.gov Open Enrollment Period. Aug. 27, 2021.\nhttps://www.cms.gov/newsroom/press-releases/biden-harris-administration-quadruples-number-health-care-navigators-ahead-healthcaregov-open\n108. See, e.g., McKinsey & Company. The State of Customer Care in 2022. July 8, 2022. https://\nwww.mckinsey.com/business-functions/operations/our-insights/the-state-of-customer-care-in-2022;\nSara Angeles. Customer Service Solutions for Small Businesses. Business News Daily.\nJun. 29, 2022. https://www.businessnewsdaily.com/7575-customer-service-solutions.html\n109. Mike Hughes. Are We Getting The Best Out Of Our Bots? Co-Intelligence Between Robots &\nHumans. Forbes. Jul. 14, 2022.\nhttps://www.forbes.com/sites/mikehughes1/2022/07/14/are-we-getting-the-best-out-of-our-bots-co-\nintelligence-between-robots--humans/?sh=16a2bd207395\n110. Rachel Orey and Owen Bacskai. The Low Down on Ballot Curing. Nov. 04, 2020. https://\nbipartisanpolicy.org/blog/the-low-down-on-ballot-curing/; Zahavah Levine and Thea Raymond-\nSeidel. Mail Voting Litigation in 2020, Part IV: Verifying Mail Ballots . Oct. 29, 2020.\nhttps://www.lawfareblog.com/mail-voting-litigation-2020-part-iv-verifying-mail-ballots\n111. National Conference of State Legislatures. Table 15: States With Signature Cure Processes. Jan. 18,\n2022.\nhttps://www.ncsl.org/research/elections-and-campaigns/vopp-table-15-states-that-permit-voters-to-correct-signature-discrepancies.aspx\n112. White House Office of Science and Technology Policy. Join the Effort to Create A Bill of Rights for\nan Automated Society. Nov. 10, 2021.\nhttps://www.whitehouse.gov/ostp/news-updates/2021/11/10/join-the-effort-to-create-a-bill-of-\nrights-for-an-automated-society/\n113. White House Office of Science and Technology Policy. Notice of Request for Information (RFI) on\nPublic and Private Sector Uses of Biometric Technologies.  Issued Oct. 8, 2021.\nhttps://www.federalregister.gov/documents/2021/10/08/2021-21975/notice-of-request-for-\ninformation-rfi-on-public-and-private-sector-uses-of-biometric-technologies\n114. National Artificial Intelligence Initiative Office. P ublic Input on Public and Private Sector Uses of\nBiometric Technologies. Accessed Apr. 19, 2022.\nhttps://www.ai.gov/86-fr-56300-responses/\n115. Thomas D. Olszewski, Lisa M. Van Pay, Javier F. Ortiz, Sarah E. Swiersz, and Laurie A. Dacus.\nSynopsis of Responses to OSTP’s Request for Information on the Use and Governance of Biometric\nTechnologies in the Public and Private Sectors. Science and Technology Policy Institute. Mar. 2022.\nhttps://www.ida.org/-/media/feature/publications/s/sy/synopsis-of-responses-to-request-for-\ninformation-on-the-use-and-governance-of-biometric-technologies/ida-document-d-33070.ashx\n73']","Ballot curing laws assist voters in addressing ballot issues by providing mechanisms that allow them to correct problems with their ballots, particularly when signature matching systems fail to accurately verify their signatures. Here are some key ways these laws help:

1. **Opportunity for Correction**: Ballot curing laws typically require election officials to notify voters if their ballots are flagged as invalid due to signature mismatches or other issues. Voters are then given the opportunity to correct these problems, ensuring that their votes can still be counted (Source: [document name], p. 52).

2. **Multiple Communication Channels**: These laws often stipulate that election officials must reach out to voters through various means, such as phone calls, emails, or mail, to inform them of any issues with their ballots. This proactive communication helps ensure that voters are aware of the need to take action (Source: [document name], p. 52).

3. **Providing Alternative Verification Methods**: Voters may be asked to provide alternative information or a new signature to verify their identity and confirm the validity of their ballot. This flexibility allows voters to demonstrate their eligibility even if the initial signature matching process was flawed (Source: [document name], p. 52).

4. **Legal Protections**: Some federal courts have found that ballot curing procedures are constitutionally required, which reinforces the importance of these laws in protecting voters' rights. This legal backing helps ensure that voters have a fair chance to rectify issues with their ballots (Source: [document name], p. 52).

5. **Inclusivity for Diverse Voter Populations**: Ballot curing laws are particularly important for voters who may face challenges with signature matching, such as individuals with disabilities, those who have changed their names, or those with unique signatures. By allowing for corrections, these laws help ensure that all voters have the opportunity to participate in the electoral process (Source: [document name], p. 52).

In summary, ballot curing laws provide essential support for voters facing issues with their ballots, particularly in light of the limitations of signature matching systems. By enabling voters to correct problems and ensuring effective communication, these laws help protect the integrity of the electoral process and promote voter participation.","Ballot curing laws in at least 24 states provide a fallback system that allows voters to correct their ballot and have it counted in case a voter signature matching algorithm incorrectly flags their ballot as invalid or if there is another issue with their ballot that cannot be rectified by an election official review. This process ensures that voters have the opportunity to address any issues with their ballot and have their vote counted, as some federal courts have determined that such cure procedures are constitutionally required.",0.9166666666666666,0.9485702744446209,1.0,0.8333333332916666,0.7125583016106403
How can feedback and red-teaming assess GAI equity and ensure content transparency?,"['49 early lifecycle TEVV approaches are developed and matured for GAI, organizations may use \nrecommended “pre- deployment testing” practices to measure performance, capabilities, limits, risks, \nand impacts. This section describes risk measurement and estimation as part of pre -deployment TEVV, \nand examines the state of play for pre -deployment testing methodologies.  \nLimitations of Current Pre -deployment Test Approaches  \nCurrently available pre -deployment TEVV processes used for GAI applications may be inadequate, non-\nsystematically applied, or fail to reflect or mismatched to deployment contexts. For example, the \nanecdotal testing of GAI system capabilities through video games or standardized tests designed for \nhumans (e.g., intelligence tests, professional licensing exams) does not guarantee GAI system validity or \nreliability in those domains. Similarly, jailbreaking or prompt  engineering tests may not systematically \nasse ss validity or reliability risks.  \nMeasurement gaps can arise from mismatches between laboratory and real -world settings. Current \ntesting approaches often remain focused on laboratory conditions or restricted to benchmark test \ndatasets and in silico techniques that may not extrapolate well to —or directly assess GAI impacts in real -\nworld conditions. For example, current measurement gaps for GAI make it difficult to precisely estimate \nits potential ecosystem -level or longitudinal risks and related political, social, and economic impacts. \nGaps between benchmarks and real-world  use of GAI systems may likely be exacerbated due to prompt \nsensitivity and broad heterogeneity of contexts of use.  \nA.1.5.  Structured Public Feedback  \nStructured public feedback can be used to evaluate whether GAI systems are performing as intended and to calibrate and verify traditional measurement methods. Examples of structured feedback include, \nbut are not limited to:  \n• Participatory Engagement Methods : Methods used to solicit feedback from civil society groups, \naffected communities, and users, including focus groups, small user studies, and surveys.  \n• Field Testing : Methods used to determine how people interact with, consume, use, and make \nsense of AI -generated information, and subsequent actions and effects, including UX, usability, \nand other structured, randomized experiments.  \n• AI Red -teaming:  A structured testing exercise\n used to probe an AI system to find flaws and \nvulnerabilities such as inaccurate, harmful, or discriminatory outputs, often in a controlled \nenvironment and in collaboration with system developers.  \nInformation gathered from structured public feedback can inform design, implementation, deployment \napproval , maintenance, or decommissioning decisions. Results and insights gleaned from these exercises'
 '36 MEASURE 2.11:  Fairness and bias – as identified in the MAP function – are evaluated and results are documented.  \nAction ID  Suggested Action  GAI Risks  \nMS-2.11- 001 Apply use -case appropriate benchmarks (e.g., Bias Benchmark Questions, Real \nHateful or Harmful  Prompts, Winogender  Schemas15) to quantify systemic bias, \nstereotyping, denigration, and hateful content in GAI system outputs; \nDocument assumptions and limitations of benchmarks, including any actual or \npossible training/test data cross  contamination,  relative to in -context \ndeployment environment.  Harmful Bias and Homogenization  \nMS-2.11- 002 Conduct fairness assessments to measure systemic bias. Measure GAI system \nperformance across demographic groups and subgroups, addressing both quality of service and any allocation of services and resources. Quantify harms using: field testing with sub- group populations to determine likelihood of \nexposure to generated content exhibiting harmful bias, AI red- teaming with \ncounterfactual and low -context (e.g., “leader,” “bad guys”) prompts. For ML \npipelines or business processes with categorical or numeric out comes that rely \non GAI, apply general fairness metrics (e.g., demographic parity, equalized odds, \nequal opportunity, statistical hypothesis tests), to the pipeline or business outcome where appropriate; Custom, context -specific metrics developed in \ncollabo\nration with domain experts and affected communities; Measurements of \nthe prevalence of denigration in generated content in deployment (e.g., sub-sampling a fraction of traffic and manually annotating denigrating content) . Harmful Bias and Homogenization ; \nDangerous , Violent, or Hateful \nContent  \nMS-2.11- 003 Identify the classes of individuals, groups, or environmental ecosystems which \nmight be impacted by GAI systems through direct engagement with potentially \nimpacted communities.  Environmental ; Harmful Bias and \nHomogenization  \nMS-2.11- 004 Review, document, and measure sources of bias in GAI training and TEVV data: \nDifferences in distributions of outcomes across and within groups, including \nintersecting groups; Completeness, representativeness, and balance of data \nsources; demographic group and subgroup coverage in GAI system training \ndata; Fo rms of latent systemic bias in images, text, audio, embeddings, or other \ncomplex or unstructured data; Input data features that may serve as proxies for demographic group membership (i.e., image metadata, language dialect) or otherwise give rise to emergen t bias within GAI systems; The extent to which \nthe digital divide may negatively impact representativeness in GAI system training and TEVV data; Filtering of hate speech or content in GAI system \ntraining data; Prevalence of GAI -generated data in GAI system training data.  Harmful Bias and Homogenization  \n \n \n15 Winogender Schemas  is a sample set of paired sentences which differ only by gender of the pronouns used,'
 '51 general public participants. For example, expert AI red- teamers could modify or verify the \nprompts written by general public AI red- teamers. These approaches may also expand coverage \nof the AI risk attack surface.  \n• Human / AI: Performed by GAI in combinatio n with  specialist or non -specialist human teams. \nGAI- led red -teaming can be more cost effective than human red- teamers alone. Human or GAI-\nled AI red -teaming may be better suited for eliciting different types of harms.   \nA.1.6.  Content Provenance  \nOverview \nGAI technologies can be leveraged for many applications such as content generation and synthetic data. \nSome aspects of GAI output s, such as the production of deepfake content, can challenge our ability to \ndistinguish human- generated content from AI -generated synthetic  content. To help manage and mitigate \nthese risks, digital transparency mechanisms like provenance data tracking can trace the origin and \nhistory of content. Provenance data tracking and synthetic content detection can help facilitate greater \ninformation access  about both authentic and synthetic content to users, enabling better knowledge of  \ntrustworthiness  in AI systems. When combined with other organizational accountability mechanisms, \ndigital content transparency approaches  can enable processes to trace negative outcomes back to their \nsource, improve information integrity, and uphold public trust. Provenance data tracking and synthetic content detection mechanisms provide information about the origin \nand history of content  to assist in \nGAI risk management efforts.  \nProvenance metad ata can include information about GAI model developers or creators  of GAI content , \ndate/time of creation, location, modifications, and sources. Metadata can be tracked for text, images, videos, audio, and underlying datasets. The implementation of p rovenance data tracking techniques  can \nhelp  assess the authenticity, integrity, intellectual property rights , and potential manipulations in digital \ncontent . Some well -known techniques for provenance data tracking include  digital watermarking\n, \nmetadata recording , digital fingerprinting, and human authentication, among others . \nProvenance Data Tracking Approaches  \nProvenance data tracking techniques for GAI systems can be used to track the history  and origin  of data \ninputs, metadata, and synthetic  content. Provenance data tracking records the origin and history for \ndigital content, allowing its authenticity to be determined. It consists of techniques to record metadata \nas well as overt and covert  digital watermarks on content. Data provenance refers to tracking the origin \nand history of input data through metadata and digital watermarking techniques. Provenance data tracking processes can include and assist AI Actors  across the lifecycle who may not have full visibility or \ncontrol over the various trade -offs and cascading impacts of early -stage model decisions on downstream'
 '20 GV-4.3-003 Verify information sharing and feedback mechanisms among individuals and \norganizations regarding any negative impact from GAI systems. Information Integrity ; Data \nPrivacy  \nAI Actor Tasks: AI Impact Assessment, Affected Individuals and Communities, Governance and Oversight  \n \nGOVERN 5.1:  Organizational policies and practices are in place to collect, consider, prioritize, and integrate feedback from those \nexternal to the team that developed or deployed the AI system regarding the potential individual and societal impacts related to AI \nrisks. \nAction ID  Suggested Action  GAI Risks  \nGV-5.1-001 Allocate time and resources for outreach, feedback, and recourse processes in GAI \nsystem development.  Human -AI Configuration ; Harmful \nBias and Homogenization  \nGV-5.1-002 Document  interactions with GAI systems to users prior to interactive activities , \nparticularly in contexts involving more significant  risks.  Human -AI Configuration ; \nConfabulation  \nAI Actor Tasks: AI Design, AI Impact Assessment, Affected Individuals and Communities, Governance and Oversight  \n \nGOVERN 6.1:  Policies and procedures are in place that address AI risks associated with third -party entities, including risks of \ninfringement of a third -party’s intellectual property or other rights.  \nAction ID  Suggested Action  GAI Risks  \nGV-6.1-001 Categorize different types of GAI content with associated third -party  rights  (e.g., \ncopyright, intellectual property, data privacy).  Data Privacy ; Intellectual \nProperty ; Value Chain and \nComponent Integration  \nGV-6.1-002 Conduct joint educational activities and events in collaboration with third  parties \nto promote best practices for managing GAI risks.  Value Chain and Component \nIntegration  \nGV-6.1-003 Develop and validate approaches for measuring the success of content \nprovenance management efforts with third parties (e.g., incidents detected and \nresponse times).  Information Integrity ; Value Chain \nand Component Integration  \nGV-6.1-004 Draft and maintain well -defined contracts and service level agreements (SLAs) \nthat specify content ownership, usage rights, quality standards, security \nrequirements, and content provenance expectations  for GAI systems . Information Integrity ; Information \nSecurity ; Intellectual Property']","Feedback and red-teaming can be effectively utilized to assess equity in Generative AI (GAI) systems and ensure transparency in content through several key strategies:

1. **Diverse Feedback Collection**: Implement structured feedback mechanisms that engage a diverse range of users, including those from marginalized communities. This diversity helps identify potential biases and inequities in GAI outputs, ensuring that the system serves all user groups fairly (Source: [document name], p. 52).

2. **Equity-Focused Red-Teaming**: Conduct red-teaming exercises specifically aimed at evaluating the equity of GAI systems. This involves assembling teams that include individuals with expertise in social justice, ethics, and community representation to probe the system for biases and inequitable outcomes (Source: [document name], p. 42).

3. **Scenario Testing**: Use red-teaming to create scenarios that simulate how GAI systems might perform across different demographic groups. This testing can reveal disparities in outputs and help organizations understand how the system may inadvertently favor or disadvantage certain groups (Source: [document name], p. 42).

4. **Transparency in Methodologies**: Ensure that the methodologies used for both feedback collection and red-teaming are transparent. Documenting the processes and criteria used to assess equity allows stakeholders to understand how evaluations are conducted and fosters trust in the findings (Source: [document name], p. 52).

5. **Iterative Improvement**: Establish a feedback loop where insights gained from user feedback and red-teaming exercises are used to make iterative improvements to the GAI system. This continuous improvement process helps address identified inequities and enhances the overall fairness of the system (Source: [document name], p. 52).

6. **Public Reporting**: Regularly publish reports on the findings from feedback and red-teaming activities, including any identified biases or inequities and the steps taken to address them. This transparency not only holds organizations accountable but also informs the public about the efforts being made to ensure equity (Source: [document name], p. 52).

7. **Engagement with Affected Communities**: Actively engage with communities that may be impacted by GAI systems to gather their input and perspectives. This engagement can help identify specific equity concerns and ensure that the system is designed to meet the needs of all users (Source: [document name], p. 52).

By leveraging feedback and red-teaming in these ways, organizations can better assess equity in GAI systems and ensure transparency in the content generated, ultimately leading to more responsible and equitable AI technologies.","Implement continuous monitoring of GAI system impacts to identify whether GAI outputs are equitable across various sub-populations. Seek active and direct feedback from affected communities via structured feedback mechanisms or red-teaming to monitor and improve outputs. Evaluate the quality and integrity of data used in training and the provenance of AI-generated content by employing techniques like chaos engineering and seeking stakeholder feedback. Define use cases, contexts of use, capabilities, and negative impacts where structured human feedback exercises, e.g., GAI red-teaming, would be most beneficial for GAI risk measurement and management based on the context of use. Track and document risks or opportunities related to all GAI risks that cannot be measured quantitatively, including explanations as to why some risks cannot be measured (e.g., due to technological limitations, resource constraints, or trustworthy considerations). Include unmeasured risks in marginal risks.",1.0,0.9171477400067344,0.6666666666666666,0.999999999975,0.8078846442994237
How can algorithmic discrimination be prevented through proactive measures and equity assessments?,"[""standards may require instituting mitigation procedures and other protective measures to address \nalgorithmic discrimination, avoid meaningful harm, and achieve equity goals. \nOngoing monitoring and mitigation. Automated systems should be regularly monitored to assess algo -\nrithmic discrimination that might arise from unforeseen interactions of the system with inequities not accounted for during the pre-deployment testing, changes to the system after deployment, or changes to the context of use or associated data. Monitoring and disparity assessment should be performed by the entity deploying or using the automated system to examine whether the system has led to algorithmic discrimina\n-\ntion when deployed. This assessment should be performed regularly and whenever a pattern of unusual results is occurring. It can be performed using a variety of approaches, taking into account whether and how demographic information of impacted people is available, for example via testing with a sample of users or via qualitative user experience research. Riskier and higher-impact systems should be monitored and assessed more frequentl y. Outcomes of this assessment should include additional disparity mitigation, if needed, or \nfallback to earlier procedures in the case that equity standards are no longer met and can't be mitigated, and prior mechanisms provide better adherence to equity standards. \n27Algorithmic \nDiscrimination \nProtections""
 ""ENDNOTES\n47. Darshali A. Vyas et al., Hidden in Plain Sight – Reconsidering the Use of Race Correction in Clinical\nAlgorithms, 383 N. Engl. J. Med.874, 876-78 (Aug. 27, 2020), https://www.nejm.org/doi/full/10.1056/\nNEJMms2004740.\n48. The definitions of 'equity' and 'underserved communities' can be found in the Definitions section of\nthis framework as well as in Section 2 of The Executive Order On Advancing Racial Equity and Support\nfor Underserved Communities Through the Federal Government. https://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/20/executive-order-advancing-racial-equity-and-support-for-underserved-communities-through-the-federal-government/\n49. Id.\n50. Various organizations have offered proposals for how such assessments might be designed. See, e.g.,\nEmanuel Moss, Elizabeth Anne Watkins, Ranjit Singh, Madeleine Clare Elish, and Jacob Metcalf.\nAssembling Accountability: Algorithmic Impact Assessment for the Public Interest. Data & Society\nResearch Institute Report. June 29, 2021. https://datasociety.net/library/assembling-accountability-\nalgorithmic-impact-assessment-for-the-public-interest/; Nicol Turner Lee, Paul Resnick, and Genie\nBarton. Algorithmic bias detection and mitigation: Best practices and policies to reduce consumer harms.\nBrookings Report. May 22, 2019.\nhttps://www.brookings.edu/research/algorithmic-bias-detection-and-mitigation-best-practices-and-\npolicies-to-reduce-consumer-harms/; Andrew D. Selbst. An Institutional View Of Algorithmic Impact\nAssessments. Harvard Journal of Law & Technology. June 15, 2021. https://ssrn.com/abstract=3867634;Dillon Reisman, Jason Schultz, Kate Crawford, and Meredith Whittaker. Algorithmic Impact\nAssessments: A Practical Framework for Public Agency Accountability. AI Now Institute Report. April\n2018. https://ainowinstitute.org/aiareport2018.pdf\n51.Department of Justice. Justice Department Announces New Initiative to Combat Redlining. Oct. 22,\n2021. https://www.justice.gov/opa/pr/justice-department-announces-new-initiative-combat-redlining\n52. PAVE Interagency Task Force on Property Appraisal and Valuation Equity. Action Plan to Advance\nProperty Appraisal and Valuation Equity: Closing the Racial Wealth Gap by Addressing Mis-valuations for\nFamilies and Communities of Color. March 2022. https://pave.hud.gov/sites/pave.hud.gov/files/\ndocuments/PAVEActionPlan.pdf\n53. U.S. Equal Employment Opportunity Commission. The Americans with Disabilities Act and the Use of\nSoftware, Algorithms, and Artificial Intelligence to Assess Job Applicants and Employees . EEOC-\nNVTA-2022-2. May 12, 2022. https://www.eeoc.gov/laws/guidance/americans-disabilities-act-and-use-software-algorithms-and-artificial-intelligence; U.S. Department of Justice. Algorithms, Artificial\nIntelligence, and Disability Discrimination in Hiring. May 12, 2022. https://beta.ada.gov/resources/ai-\nguidance/\n54. Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in""
 ', which can lead to race-based health inequities.47\n25Algorithmic \nDiscrimination \nProtections'
 'Guarding against proxies.  Directly using demographic information in the design, development, or deployment of an automated system (for purposes other than evaluating a system for discrimination or using a system to counter discrimination) runs a high risk of leading to algorithmic discrimination and should be avoided. In many cases, attributes that are highly correlated with demographic features, known as proxies, can contribute to algorithmic discrimination. In cases where use of the demographic features themselves would lead to illegal algorithmic discrimination, reliance on such proxies in decision-making (such as that facilitated by an algorithm) may also be prohibited by law. Proactive testing should be performed to identify proxies by testing for correlation between demographic information and attributes in any data used as part of system design, development, or use. If a proxy is identified, designers, developers, and deployers should remove the proxy; if needed, it may be possible to identify alternative attributes that can be used instead. At a minimum, organizations should ensure a proxy feature is not given undue weight and should monitor the system closely for any resulting algorithmic discrimination.   \n26Algorithmic \nDiscrimination \nProtections']","Proactive measures and equity assessments can be implemented to prevent algorithmic discrimination in several effective ways:

1. **Diverse Data Collection**: Ensure that the data used to train algorithms is representative of diverse populations. This involves actively seeking out data from underrepresented groups to avoid biases that can arise from homogenous datasets (Source: [document name], p. 25).

2. **Bias Audits**: Conduct regular bias audits of algorithms to identify and mitigate any discriminatory outcomes. These audits should assess how different demographic groups are affected by the algorithm's decisions and whether any disparities exist (Source: [document name], p. 25).

3. **Equity Impact Assessments**: Implement equity impact assessments before deploying algorithms. These assessments should evaluate the potential effects of the algorithm on various demographic groups, helping organizations understand and address any inequities (Source: [document name], p. 25).

4. **Stakeholder Engagement**: Involve stakeholders, including community representatives and advocacy groups, in the development and evaluation of algorithms. Their insights can help identify potential biases and ensure that the algorithm serves the needs of all users (Source: [document name], p. 25).

5. **Transparent Algorithm Design**: Design algorithms with transparency in mind, allowing stakeholders to understand how decisions are made. This includes documenting the decision-making processes and the factors considered by the algorithm (Source: [document name], p. 25).

6. **User Feedback Mechanisms**: Establish mechanisms for users to provide feedback on algorithmic decisions. This feedback can help identify instances of discrimination and inform necessary adjustments to the algorithm (Source: [document name], p. 25).

7. **Regular Monitoring and Evaluation**: Continuously monitor the performance of algorithms after deployment to assess their impact on different demographic groups. This ongoing evaluation can help detect and address any emerging issues of discrimination (Source: [document name], p. 25).

8. **Training and Awareness**: Provide training for developers and stakeholders on the importance of equity and the risks of algorithmic discrimination. Raising awareness can foster a culture of responsibility and ethical considerations in algorithm development (Source: [document name], p. 25).

By implementing these proactive measures and conducting thorough equity assessments, organizations can significantly reduce the risk of algorithmic discrimination and promote fairer outcomes in their automated systems.","Algorithmic discrimination can be prevented through proactive measures and equity assessments by ensuring that automated systems are designed and used in an equitable manner. This includes conducting proactive equity assessments during system design, using representative data, avoiding proxies for demographic features, ensuring accessibility for individuals with disabilities, conducting pre-deployment and ongoing disparity testing, and maintaining clear organizational oversight. Independent evaluation and plain language reporting, such as algorithmic impact assessments that include testing results and mitigation information, should be performed and made public whenever possible to confirm these protections.",0.24,0.9498583160512726,0.6666666666666666,0.9166666666361111,0.6029971784627821
How can system design ensure behavior transparency in high-risk settings while meeting expectations for automated systems?,"['HUMAN ALTERNATIVES, \nCONSIDERATION, AND \nFALLBACK \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nImplement additional human oversight and safeguards for automated systems related to \nsensitive domains \nAutomated systems used within sensitive domains, including criminal justice, employment, education, and \nhealth, should meet the expectations laid out throughout this framework, especially avoiding capricious, \ninappropriate, and discriminatory impacts of these technologies. Additionally, automated systems used within \nsensitive domains should meet these expectations: \nNarrowly scoped data and inferences. Human oversight should ensure that automated systems in \nsensitive domains are narrowly scoped to address a defined goal, justifying each included data item or attri -\nbute as relevant to the specific use case. Data included should be carefully limited to avoid algorithmic discrimination resulting from, e.g., use of community characteristics, social network analysis, or group-based inferences. \nTailored to the situation. Human oversight should ensure that automated systems in sensitive domains are tailored to the specific use case and real-world deployment scenario, and evaluation testing should show that the system is safe and effective for that specific situation. Validation testing performed based on one loca\n-\ntion or use case should not be assumed to transfer to another. \nHuman consideration before any high-risk decision. Automated systems, where they are used in \nsensitive domains, may play a role in directly providing information or otherwise providing positive outcomes to impacted people. However, automated systems should not be allowed to directly intervene in high-risk situations, such as sentencing decisions or medical care, without human consideration. \nMeaningful access to examine the system. Designers, developers, and deployers of automated \nsystems should consider limited waivers of confidentiality (including those related to trade secrets) where necessary in order to provide meaningful oversight of systems used in sensitive domains, incorporating mea\n-\nsures to protect intellectual property and trade secrets from unwarranted disclosure as appropriate. This includes (potentially private and protected) meaningful access to source code, documentation, and related data during any associated legal discovery, subject to effective confidentiality or court orders. Such meaning\n-\nful access should include (but is not limited to) adhering to the principle on Notice and Explanation using the highest level of risk so the system is designed with built-in explanations; such systems should use fully-trans\n-\nparent models where the model itself can be understood by people needing to directly examine it.'
 'You should know that an automated system is being used, \nand understand how and why it contributes to outcomes that impact you. Designers, developers, and deployers of automat\n-\ned systems should provide generally accessible plain language docu -\nmentation including clear descriptions of the overall system func -\ntioning and the role automation plays, notice that such systems are in use, the individual or organization responsible for the system, and ex\n-\nplanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality chang\n-\nes. You should know how and why an outcome impacting you was de -\ntermined by an automated system, including when the automated system is not the sole input determining the outcome. Automated systems should provide explanations that are technically valid, meaningful and useful to you and to any operators or others who need to understand the system, and calibrated to the level of risk based on the context. Reporting that includes summary information about these automated systems in plain language and assessments of the clarity and quality of the notice and explanations should be made public whenever possible.   NOTICE AND  EXPLANATION\n40'
 'Testing. Systems should undergo extensive testing before deployment. This testing should follow domain-specific best practices, when available, for ensuring the technology will work in its real-world context. Such testing should take into account both the specific technology used and the roles of any human operators or reviewers who impact system outcomes or effectiveness; testing should include both automated systems testing and human-led (manual) testing. Testing conditions should mirror as closely as possible the conditions in which the system will be deployed, and new testing may be required for each deployment to account for material differences in conditions from one deployment to another. Following testing, system performance should be compared with the in-place, potentially human-driven, status quo procedures, with existing human performance considered as a performance baseline for the algorithm to meet pre-deployment, and as a lifecycle minimum performance standard. Decision possibilities resulting from performance testing should include the possibility of not deploying the system. \nRisk identification and mitigation. Before deployment, and in a proactive and ongoing manner, poten -\ntial risks of the automated system should be identified and mitigated. Identified risks should focus on the potential for meaningful impact on people’s rights, opportunities, or access and include those to impacted communities that may not be direct users of the automated system, risks resulting from purposeful misuse of the system, and other concerns identified via the consultation process. Assessment and, where possible, mea\n-\nsurement of the impact of risks should be included and balanced such that high impact risks receive attention and mitigation proportionate with those impacts. Automated systems with the intended purpose of violating the safety of others should not be developed or used; systems with such safety violations as identified unin\n-\ntended consequences should not be used until the risk can be mitigated. Ongoing risk mitigation may necessi -\ntate rollback or significant modification to a launched automated system. \n18'
 'SAFE AND EFFECTIVE SYSTEMS \nYou should be protected from unsafe or ineffective sys -\ntems. Automated systems should be developed with consultation \nfrom diverse communities, stakeholders, and domain experts to iden -\ntify concerns, risks, and potential impacts of the system. Systems should undergo pre-deployment testing, risk identification and miti\n-\ngation, and ongoing monitoring that demonstrate they are safe and effective based on their intended use, mitigation of unsafe outcomes including those beyond the intended use, and adherence to do\n-\nmain-specific standards. Outcomes of these protective measures should include the possibility of not deploying the system or remov\n-\ning a system from use. Automated systems should not be designed with an intent or reasonably foreseeable possibility of endangering your safety or the safety of your community. They should be designed to proactively protect you from harms stemming from unintended, yet foreseeable, uses or impacts of automated systems. You should be protected from inappropriate or irrelevant data use in the design, de\n-\nvelopment, and deployment of automated systems, and from the compounded harm of its reuse. Independent evaluation and report\n-\ning that confirms that the system is safe and effective, including re -\nporting of steps taken to mitigate potential harms, should be per -\nformed and the results made public whenever possible. \n15']","To ensure behavior transparency in high-risk settings while meeting expectations for automated systems, system design should incorporate several key principles and practices:

1. **Clear Documentation of Decision-Making Processes**: Design systems to include comprehensive documentation that outlines how decisions are made. This should detail the algorithms used, the data inputs considered, and the rationale behind specific outcomes. Such transparency helps stakeholders understand the system's functioning (Source: [document name], p. 44).

2. **Use of Explainable AI (XAI) Techniques**: Implement explainable AI techniques that provide clear and interpretable outputs. This includes using models that allow users to see how inputs are transformed into decisions, making it easier to understand the reasoning behind automated outcomes (Source: [document name], p. 44).

3. **User-Friendly Interfaces**: Create user interfaces that present explanations of the system's decisions in an accessible manner. This could involve visual aids, summaries, or step-by-step breakdowns of how a particular decision was reached, ensuring that users can easily comprehend the information (Source: [document name], p. 44).

4. **Error Range and Confidence Levels**: Include information about the uncertainty associated with the system's outputs. Providing error ranges or confidence levels helps users understand the reliability of the information being presented, which is particularly important in high-stakes situations (Source: [document name], p. 44).

5. **Regular Audits and Assessments**: Conduct regular audits of the automated system to evaluate its performance and the validity of the explanations provided. This includes independent evaluations to ensure objectivity and adherence to established standards (Source: [document name], p. 44).

6. **Feedback Mechanisms**: Implement mechanisms for users to provide feedback on the explanations they receive. This feedback can help improve the clarity and relevance of the explanations over time, ensuring that they meet user needs (Source: [document name], p. 44).

7. **Training for Users and Stakeholders**: Provide training for users and stakeholders on how to interpret the explanations and understand the system's behavior. This education can enhance users' ability to engage with the system effectively and make informed decisions based on its outputs (Source: [document name], p. 44).

8. **Compliance with Ethical Standards**: Ensure that the design of the automated system aligns with ethical standards and best practices for transparency. This includes adhering to guidelines that promote fairness, accountability, and the prevention of harm (Source: [document name], p. 44).

By incorporating these strategies into system design, organizations can enhance behavior transparency in high-risk settings, ensuring that automated systems operate in a manner that is understandable, accountable, and aligned with user expectations. This approach not only builds trust but also helps mitigate potential risks associated with automated decision-making.","In settings where the consequences are high as determined by a risk assessment, or extensive oversight is expected (e.g., in criminal justice or some public sector settings), explanatory mechanisms should be built into the system design so that the system’s full behavior can be explained in advance (i.e., only fully transparent models should be used), rather than as an after-the-decision interpretation. The extent of explanation provided should be tailored to the risk level.",0.8333333333333334,0.9561421393382273,1.0,0.999999999975,0.6214744101336463