README.md

metadata

base_model: sentence-transformers/all-mpnet-base-v2
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
  - dot_accuracy@1
  - dot_accuracy@3
  - dot_accuracy@5
  - dot_accuracy@10
  - dot_precision@1
  - dot_precision@3
  - dot_precision@5
  - dot_precision@10
  - dot_recall@1
  - dot_recall@3
  - dot_recall@5
  - dot_recall@10
  - dot_ndcg@10
  - dot_mrr@10
  - dot_map@100
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:714
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
widget:
  - source_sentence: >-
      What does the term 'rights, opportunities, or access' encompass in this
      framework?
    sentences:
      - >-
        10 

        GAI systems can ease the unintentional production or dissemination of
        false, inaccurate, or misleading 

        content (misinformation) at scale, particularly if the content stems
        from confabulations.  

        GAI systems can also ease the deliberate production or dissemination of
        false or misleading information 

        (disinformation) at scale, where an actor has the explicit intent to
        deceive or cause harm to others. Even 

        very subtle changes to text or images can manipulate human and machine
        perception. 

        Similarly, GAI systems could enable a higher degree of sophistication
        for malicious actors to produce 

        disinformation that is targeted towards specific demographics. Current
        and emerging multimodal models 

        make it possible to generate both text-based disinformation and highly
        realistic “deepfakes” – that is, 

        synthetic audiovisual content and photorealistic images.12 Additional
        disinformation threats could be 

        enabled by future GAI models trained on new data modalities.
      - >-
        74. See, e.g., Heather Morrison. Virtual Testing Puts Disabled Students
        at a Disadvantage. Government

        Technology. May 24, 2022.

        https://www.govtech.com/education/k-12/virtual-testing-puts-disabled-students-at-a-disadvantage;

        Lydia X. Z. Brown, Ridhi Shetty, Matt Scherer, and Andrew Crawford.
        Ableism And Disability

        Discrimination In New Surveillance Technologies: How new surveillance
        technologies in education,

        policing, health care, and the workplace disproportionately harm
        disabled people. Center for Democracy

        and Technology Report. May 24, 2022.

        https://cdt.org/insights/ableism-and-disability-discrimination-in-new-surveillance-technologies-how­

        new-surveillance-technologies-in-education-policing-health-care-and-the-workplace­

        disproportionately-harm-disabled-people/

        69
      - >-
        persons, Asian Americans and Pacific Islanders and other persons of
        color; members of religious minorities; 

        women, girls, and non-binary people; lesbian, gay, bisexual,
        transgender, queer, and intersex (LGBTQI+) 

        persons; older adults; persons with disabilities; persons who live in
        rural areas; and persons otherwise adversely 

        affected by persistent poverty or inequality. 

        RIGHTS, OPPORTUNITIES, OR ACCESS: “Rights, opportunities, or access” is
        used to indicate the scoping 

        of this framework. It describes the set of: civil rights, civil
        liberties, and privacy, including freedom of speech, 

        voting, and protections from discrimination, excessive punishment,
        unlawful surveillance, and violations of 

        privacy and other freedoms in both public and private sector contexts;
        equal opportunities, including equitable 

        access to education, housing, credit, employment, and other programs;
        or, access to critical resources or
  - source_sentence: >-
      What are some broad negative risks associated with GAI design,
      development, and deployment?
    sentences:
      - >-
        actually occurring, or large-scale risks could occur); and broad GAI
        negative risks, 

        including: Immature safety or risk cultures related to AI and GAI
        design, 

        development and deployment, public information integrity risks,
        including impacts 

        on democratic processes, unknown long-term performance characteristics
        of GAI. 

        Information Integrity; Dangerous, 

        Violent, or Hateful Content; CBRN 

        Information or Capabilities 

        GV-1.3-007 Devise a plan to halt development or deployment of a GAI
        system that poses 

        unacceptable negative risk. 

        CBRN Information and Capability; 

        Information Security; Information 

        Integrity 

        AI Actor Tasks: Governance and Oversight 
         
        GOVERN 1.4: The risk management process and its outcomes are established
        through transparent policies, procedures, and other 

        controls based on organizational risk priorities. 

        Action ID 

        Suggested Action 

        GAI Risks 

        GV-1.4-001 

        Establish policies and mechanisms to prevent GAI systems from generating
      - >-
        39 

        MS-3.3-004 

        Provide input for training materials about the capabilities and
        limitations of GAI 

        systems related to digital content transparency for AI Actors, other 

        professionals, and the public about the societal impacts of AI and the
        role of 

        diverse and inclusive content generation. 

        Human-AI Configuration; 

        Information Integrity; Harmful Bias 

        and Homogenization 

        MS-3.3-005 

        Record and integrate structured feedback about content provenance from 

        operators, users, and potentially impacted communities through the use
        of 

        methods such as user research studies, focus groups, or community
        forums. 

        Actively seek feedback on generated content quality and potential
        biases. 

        Assess the general awareness among end users and impacted communities 

        about the availability of these feedback channels. 

        Human-AI Configuration; 

        Information Integrity; Harmful Bias 

        and Homogenization 

        AI Actor Tasks: AI Deployment, Affected Individuals and Communities,
        End-Users, Operation and Monitoring, TEVV
      - >-
        NOTICE & 

        EXPLANATION 

        WHY THIS PRINCIPLE IS IMPORTANT

        This section provides a brief summary of the problems which the
        principle seeks to address and protect 

        against, including illustrative examples. 

        Automated systems now determine opportunities, from employment to
        credit, and directly shape the American 

        public’s experiences, from the courtroom to online classrooms, in ways
        that profoundly impact people’s lives. But this 

        expansive impact is not always visible. An applicant might not know
        whether a person rejected their resume or a 

        hiring algorithm moved them to the bottom of the list. A defendant in
        the courtroom might not know if a judge deny­

        ing their bail is informed by an automated system that labeled them
        “high risk.” From correcting errors to contesting 

        decisions, people are often denied the knowledge they need to address
        the impact of automated systems on their lives.
  - source_sentence: >-
      Who should conduct the assessment of the impact of surveillance on rights
      and opportunities?
    sentences:
      - >-
        APPENDIX

        •

        Julia Simon-Mishel, Supervising Attorney, Philadelphia Legal Assistance

        •

        Dr. Zachary Mahafza, Research & Data Analyst, Southern Poverty Law
        Center

        •

        J. Khadijah Abdurahman, Tech Impact Network Research Fellow, AI Now
        Institute, UCLA C2I1, and

        UWA Law School

        Panelists separately described the increasing scope of technology use in
        providing for social welfare, including 

        in fraud detection, digital ID systems, and other methods focused on
        improving efficiency and reducing cost. 

        However, various panelists individually cautioned that these systems may
        reduce burden for government 

        agencies by increasing the burden and agency of people using and
        interacting with these technologies. 

        Additionally, these systems can produce feedback loops and compounded
        harm, collecting data from 

        communities and using it to reinforce inequality. Various panelists
        suggested that these harms could be
      - >-
        assessments, including data retention timelines and associated
        justification, and an assessment of the 

        impact of surveillance or data collection on rights, opportunities, and
        access. Where possible, this 

        assessment of the impact of surveillance should be done by an
        independent party. Reporting should be 

        provided in a clear and machine-readable manner.  

        35
      - >-
        access to education, housing, credit, employment, and other programs;
        or, access to critical resources or 

        services, such as healthcare, financial services, safety, social
        services, non-deceptive information about goods 

        and services, and government benefits. 

        10
  - source_sentence: How can voting-related systems impact privacy and security?
    sentences:
      - >-
        as custody and divorce information, and home, work, or school
        environmental data); or have the reasonable potential 

        to be used in ways that are likely to expose individuals to meaningful
        harm, such as a loss of privacy or financial harm 

        due to identity theft. Data and metadata generated by or about those who
        are not yet legal adults is also sensitive, even 

        if not related to a sensitive domain. Such data includes, but is not
        limited to, numerical, text, image, audio, or video 

        data. “Sensitive domains” are those in which activities being conducted
        can cause material harms, including signifi­

        cant adverse effects on human rights such as autonomy and dignity, as
        well as civil liberties and civil rights. Domains 

        that have historically been singled out as deserving of enhanced data
        protections or where such enhanced protections 

        are reasonably expected by the public include, but are not limited to,
        health, family planning and care, employment,
      - >-
        agreed upon the importance of advisory boards and compensated community
        input early in the design process 

        (before the technology is built and instituted). Various panelists also
        emphasized the importance of regulation 

        that includes limits to the type and cost of such technologies. 

        56
      - >-
        Surveillance and criminal justice system algorithms such as risk
        assessments, predictive  
            policing, automated license plate readers, real-time facial recognition systems (especially  
            those used in public places or during protected activities like peaceful protests), social media  
            monitoring, and ankle monitoring devices; 
        Voting-related systems such as signature matching tools; 

        Systems with a potential privacy impact such as smart home systems and
        associated data,  
            systems that use or collect health-related data, systems that use or collect education-related  
            data, criminal justice system data, ad-targeting systems, and systems that perform big data  
            analytics in order to build profiles or infer personal information about individuals; and 
        Any system that has the meaningful potential to lead to algorithmic
        discrimination. 

        • Equal opportunities, including but not limited to:
  - source_sentence: What impact do automated systems have on underserved communities?
    sentences:
      - >-
        generation, summarization, search, and chat. These activities can take
        place within organizational 

        settings or in the public domain. 

        Organizations can restrict AI applications that cause harm, exceed
        stated risk tolerances, or that conflict 

        with their tolerances or values. Governance tools and protocols that are
        applied to other types of AI 

        systems can be applied to GAI systems. These plans and actions include: 

        • Accessibility and reasonable 

        accommodations 

        • AI actor credentials and qualifications  

        • Alignment to organizational values 

        • Auditing and assessment 

        • Change-management controls 

        • Commercial use 

        • Data provenance
      - >-
        automated systems make on underserved communities and to institute
        proactive protections that support these 

        communities. 

        •

        An automated system using nontraditional factors such as educational
        attainment and employment history as

        part of its loan underwriting and pricing model was found to be much
        more likely to charge an applicant who

        attended a Historically Black College or University (HBCU) higher loan
        prices for refinancing a student loan

        than an applicant who did not attend an HBCU. This was found to be true
        even when controlling for

        other credit-related factors.32

        •

        A hiring tool that learned the features of a company's employees
        (predominantly men) rejected women appli­

        cants for spurious and discriminatory reasons; resumes with the word
        “women’s,” such as “women’s

        chess club captain,” were penalized in the candidate ranking.33

        •

        A predictive model marketed as being able to predict whether students
        are likely to drop out of school was
      - >-
        on a principle of local control, such that those individuals closest to
        the data subject have more access while 

        those who are less proximate do not (e.g., a teacher has access to their
        students’ daily progress data while a 

        superintendent does not). 

        Reporting. In addition to the reporting on data privacy (as listed above
        for non-sensitive data), entities devel-

        oping technologies related to a sensitive domain and those collecting,
        using, storing, or sharing sensitive data 

        should, whenever appropriate, regularly provide public reports
        describing: any data security lapses or breaches 

        that resulted in sensitive data leaks; the number, type, and outcomes of
        ethical pre-reviews undertaken; a 

        description of any data sold, shared, or made public, and how that data
        was assessed to determine it did not pres-

        ent a sensitive data risk; and ongoing risk identification and
        management procedures, and any mitigation added
model-index:
  - name: SentenceTransformer based on sentence-transformers/all-mpnet-base-v2
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: Unknown
          type: unknown
        metrics:
          - type: cosine_accuracy@1
            value: 0.8881578947368421
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.9868421052631579
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9868421052631579
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 1
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.8881578947368421
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.32894736842105265
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19736842105263155
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09999999999999999
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.8881578947368421
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.9868421052631579
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.9868421052631579
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 1
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9499393562918366
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.9331140350877194
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.9331140350877194
            name: Cosine Map@100
          - type: dot_accuracy@1
            value: 0.8881578947368421
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.9868421052631579
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.9868421052631579
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 1
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.8881578947368421
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.32894736842105265
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.19736842105263155
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.09999999999999999
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.8881578947368421
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.9868421052631579
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.9868421052631579
            name: Dot Recall@5
          - type: dot_recall@10
            value: 1
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.9499393562918366
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.9331140350877194
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.9331140350877194
            name: Dot Map@100
          - type: cosine_accuracy@1
            value: 0.8828125
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.96875
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9921875
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 1
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.8828125
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.32291666666666663
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19843750000000004
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.10000000000000002
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.8828125
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.96875
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.9921875
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 1
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9458381646710927
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.9279296875
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.9279296875
            name: Cosine Map@100
          - type: dot_accuracy@1
            value: 0.8828125
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.96875
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 0.9921875
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 1
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.8828125
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.32291666666666663
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.19843750000000004
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.10000000000000002
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.8828125
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.96875
            name: Dot Recall@3
          - type: dot_recall@5
            value: 0.9921875
            name: Dot Recall@5
          - type: dot_recall@10
            value: 1
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.9458381646710927
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.9279296875
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.9279296875
            name: Dot Map@100

SentenceTransformer based on sentence-transformers/all-mpnet-base-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-mpnet-base-v2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-mpnet-base-v2
• Maximum Sequence Length: 384 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("jet-taekyo/mpnet_finetuned_semantic")
# Run inference
sentences = [
    'What impact do automated systems have on underserved communities?',
    "automated systems make on underserved communities and to institute proactive protections that support these \ncommunities. \n•\nAn automated system using nontraditional factors such as educational attainment and employment history as\npart of its loan underwriting and pricing model was found to be much more likely to charge an applicant who\nattended a Historically Black College or University (HBCU) higher loan prices for refinancing a student loan\nthan an applicant who did not attend an HBCU. This was found to be true even when controlling for\nother credit-related factors.32\n•\nA hiring tool that learned the features of a company's employees (predominantly men) rejected women appli\xad\ncants for spurious and discriminatory reasons; resumes with the word “women’s,” such as “women’s\nchess club captain,” were penalized in the candidate ranking.33\n•\nA predictive model marketed as being able to predict whether students are likely to drop out of school was",
    'on a principle of local control, such that those individuals closest to the data subject have more access while \nthose who are less proximate do not (e.g., a teacher has access to their students’ daily progress data while a \nsuperintendent does not). \nReporting. In addition to the reporting on data privacy (as listed above for non-sensitive data), entities devel-\noping technologies related to a sensitive domain and those collecting, using, storing, or sharing sensitive data \nshould, whenever appropriate, regularly provide public reports describing: any data security lapses or breaches \nthat resulted in sensitive data leaks; the number, type, and outcomes of ethical pre-reviews undertaken; a \ndescription of any data sold, shared, or made public, and how that data was assessed to determine it did not pres-\nent a sensitive data risk; and ongoing risk identification and management procedures, and any mitigation added',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8882
cosine_accuracy@3	0.9868
cosine_accuracy@5	0.9868
cosine_accuracy@10	1.0
cosine_precision@1	0.8882
cosine_precision@3	0.3289
cosine_precision@5	0.1974
cosine_precision@10	0.1
cosine_recall@1	0.8882
cosine_recall@3	0.9868
cosine_recall@5	0.9868
cosine_recall@10	1.0
cosine_ndcg@10	0.9499
cosine_mrr@10	0.9331
cosine_map@100	0.9331
dot_accuracy@1	0.8882
dot_accuracy@3	0.9868
dot_accuracy@5	0.9868
dot_accuracy@10	1.0
dot_precision@1	0.8882
dot_precision@3	0.3289
dot_precision@5	0.1974
dot_precision@10	0.1
dot_recall@1	0.8882
dot_recall@3	0.9868
dot_recall@5	0.9868
dot_recall@10	1.0
dot_ndcg@10	0.9499
dot_mrr@10	0.9331
dot_map@100	0.9331

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8828
cosine_accuracy@3	0.9688
cosine_accuracy@5	0.9922
cosine_accuracy@10	1.0
cosine_precision@1	0.8828
cosine_precision@3	0.3229
cosine_precision@5	0.1984
cosine_precision@10	0.1
cosine_recall@1	0.8828
cosine_recall@3	0.9688
cosine_recall@5	0.9922
cosine_recall@10	1.0
cosine_ndcg@10	0.9458
cosine_mrr@10	0.9279
cosine_map@100	0.9279
dot_accuracy@1	0.8828
dot_accuracy@3	0.9688
dot_accuracy@5	0.9922
dot_accuracy@10	1.0
dot_precision@1	0.8828
dot_precision@3	0.3229
dot_precision@5	0.1984
dot_precision@10	0.1
dot_recall@1	0.8828
dot_recall@3	0.9688
dot_recall@5	0.9922
dot_recall@10	1.0
dot_ndcg@10	0.9458
dot_mrr@10	0.9279
dot_map@100	0.9279

Training Details

Training Dataset

Unnamed Dataset

• Size: 714 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 714 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 7 tokens mean: 17.7 tokens max: 36 tokens	min: 2 tokens mean: 176.29 tokens max: 384 tokens

• Samples:

  sentence_0	sentence_1
  What are the key characteristics of high-integrity information?	This information can be linked to the original source(s) with appropriate evidence. High-integrity information is also accurate and reliable, can be verified and authenticated, has a clear chain of custody, and creates reasonable expectations about when its validity may expire.”11 11 This definition of information integrity is derived from the 2022 White House Roadmap for Researchers on Priorities Related to Information Integrity Research and Development.
  How can the validity of information be verified and authenticated?	This information can be linked to the original source(s) with appropriate evidence. High-integrity information is also accurate and reliable, can be verified and authenticated, has a clear chain of custody, and creates reasonable expectations about when its validity may expire.”11 11 This definition of information integrity is derived from the 2022 White House Roadmap for Researchers on Priorities Related to Information Integrity Research and Development.
  What should trigger the use of a human alternative in the attainment process?	In many scenarios, there is a reasonable expectation of human involvement in attaining rights, opportunities, or access. When automated systems make up part of the attainment process, alternative timely human-driven processes should be provided. The use of a human alternative should be triggered by an opt-out process. Timely and not burdensome human alternative. Opting out should be timely and not unreasonably burdensome in both the process of requesting to opt-out and the human-driven alternative provided. Provide timely human consideration and remedy by a fallback and escalation system in the event that an automated system fails, produces error, or you would like to appeal or con­ test its impacts on you Proportionate. The availability of human consideration and fallback, along with associated training and safeguards against human bias, should be proportionate to the potential of the automated system to meaning­ fully impact rights, opportunities, or access. Automated systems that have greater control over outcomes, provide input to high-stakes decisions, relate to sensitive domains, or otherwise have a greater potential to meaningfully impact rights, opportunities, or access should have greater availability (e.g., staffing) and over­ sight of human consideration and fallback mechanisms. Accessible. Mechanisms for human consideration and fallback, whether in-person, on paper, by phone, or otherwise provided, should be easy to find and use. These mechanisms should be tested to ensure that users who have trouble with the automated system are able to use human consideration and fallback, with the under­ standing that it may be these users who are most likely to need the human assistance. Similarly, it should be tested to ensure that users with disabilities are able to find and use human consideration and fallback and also request reasonable accommodations or modifications. Convenient. Mechanisms for human consideration and fallback should not be unreasonably burdensome as compared to the automated system’s equivalent. 49

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 20
• per_device_eval_batch_size: 20
• num_train_epochs: 5
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 20
• per_device_eval_batch_size: 20
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	cosine_map@100
1.0	36	0.9395
1.3889	50	0.9320
2.0	72	0.9298
2.7778	100	0.9348
3.0	108	0.9304
4.0	144	0.9342
4.1667	150	0.9342
5.0	180	0.9331
1.0	31	0.9163
1.6129	50	0.9279

Framework Versions

• Python: 3.11.9
• Sentence Transformers: 3.1.0
• Transformers: 4.44.2
• PyTorch: 2.4.1+cu121
• Accelerate: 0.34.2
• Datasets: 3.0.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}