|
--- |
|
base_model: BAAI/bge-base-en-v1.5 |
|
library_name: setfit |
|
metrics: |
|
- accuracy |
|
pipeline_tag: text-classification |
|
tags: |
|
- setfit |
|
- sentence-transformers |
|
- text-classification |
|
- generated_from_setfit_trainer |
|
widget: |
|
- text: "Reasoning:\n\n1. **Context Grounding**: The provided answer does not directly\ |
|
\ reference where to access specific training resources. It rather gives a mixture\ |
|
\ of unrelated information such as using a password manager, secure file sharing,\ |
|
\ personal development budget, etc. These points, while valid, do not directly\ |
|
\ pertain to accessing training resources specifically.\n \n2. **Relevance**:\ |
|
\ The response deviates from speaking directly about accessing training resources\ |
|
\ and instead covers a broader range of topics, which includes using company systems\ |
|
\ for other purposes, security protocols, and personal development budget discussions.\ |
|
\ These areas do not directly answer the question.\n\n3. **Conciseness**: The\ |
|
\ answer lacks conciseness as it includes quite a bit of unrelated information,\ |
|
\ which makes it unnecessarily lengthy and potentially confusing.\n\n4. **Specificity**:\ |
|
\ The answer fails to be specific about how to access training resources. If the\ |
|
\ document does contain the relevant information, it hasn't been effectively pulled\ |
|
\ out or emphasized in the answer.\n\n5. **No-Response Principle**: If a specific\ |
|
\ question cannot be answered based on the given document, it should be clearly\ |
|
\ stated. Here, it seems the document may not contain a straightforward guide\ |
|
\ to accessing training resources, but this isn't clarified.\n\nFinal Result:\ |
|
\ **Bad**" |
|
- text: '**Reasoning:** |
|
|
|
|
|
1. **Context Grounding:** The answer is well-supported by Document 1, which specifies |
|
that questions about travel reimbursement should be directed to finance@ORGANIZATION_2. |
|
|
|
2. **Relevance:** The answer is directly relevant to the question, which asks |
|
whom to contact about travel reimbursement. |
|
|
|
3. **Conciseness:** The answer is concise and directly addresses the question |
|
without including any unnecessary information. |
|
|
|
4. **Response Appropriateness:** The answer correctly utilizes the detail provided |
|
in the document and does not attempt to provide unnecessary information. |
|
|
|
5. **Specificity:** The answer provides the specific email address for the relevant |
|
point of contact. |
|
|
|
6. **Balanced Detail:** The answer includes the required contact information and |
|
is not too general. |
|
|
|
|
|
**Final Result: Good.**' |
|
- text: 'The answer provided fulfills the criteria as outlined: |
|
|
|
|
|
1. **Context Grounding**: The answer accurately reflects the contents of the documents, |
|
particularly Document 1, which specifies the steps team leads should take when |
|
they consider the possibility that someone''s time at the ORGANIZATION is up. |
|
|
|
|
|
2. **Relevance**: The answer directly addresses the question by explaining why |
|
it is important for team leads to think about the possibility of someone leaving, |
|
including the potential benefits of addressing issues before they become unmanageable. |
|
|
|
|
|
3. **Conciseness**: The answer is somewhat verbose but manages to stay mostly |
|
on-topic and addresses the question throughout, ensuring it doesn''t veer into |
|
unrelated topics. |
|
|
|
|
|
4. **Specificity**: While a bit generalized in parts, the answer includes specifics |
|
such as underperformance, lack of growth, and disagreement with company direction, |
|
which are supported by the details in the documents. |
|
|
|
|
|
Therefore, despite minor verbosity, the answer is accurate, relevant, and detailed |
|
enough to be considered good. |
|
|
|
|
|
**Final Result: Good**' |
|
- text: 'Reasoning: |
|
|
|
1. Context Grounding: The response draws from the documents providing relevant |
|
sources such as the organization''s website, job ads, and newsletter link. |
|
|
|
2. Relevance: The answer is directly related to the question about understanding |
|
the organization''s products, challenges, and future. |
|
|
|
3. Conciseness: The answer is clear and to the point. |
|
|
|
4. Does not attempt to respond when the document lacks information: It addresses |
|
the question appropriately with the available information. |
|
|
|
5. Specificity: The answer is specific and provides concrete steps to follow. |
|
|
|
6. Relevant tips: The answer includes actionable steps like visiting the website, |
|
viewing job ads, and signing up for a newsletter, which are relevant. |
|
|
|
|
|
The answer precisely matches all the criteria set for evaluation. |
|
|
|
|
|
Final Result: Good' |
|
- text: 'Reasoning: |
|
|
|
1. Context Grounding: The answer is well-grounded in the document, explaining |
|
the roles of ORGANIZATION_2, Thomas Barnes, and Charlotte Herrera correctly based |
|
on the provided information. |
|
|
|
2. Relevance: The answer directly addresses the question, detailing the extent |
|
of ORGANIZATION_2''s participation in the farewell process. |
|
|
|
3. Conciseness: While mostly concise, the answer could be slightly more succinct. |
|
Repetition of certain elements could be minimized. |
|
|
|
4. Appropriateness of Response: The answer does not attempt to respond beyond |
|
the document. |
|
|
|
5. Specificity: The answer is specific, describing the involvement of specific |
|
people and their roles. |
|
|
|
6. Tips and Generality: The answer includes relevant insights into the roles, |
|
though it does not go into overly general territory. |
|
|
|
|
|
Final Result: Good' |
|
inference: true |
|
model-index: |
|
- name: SetFit with BAAI/bge-base-en-v1.5 |
|
results: |
|
- task: |
|
type: text-classification |
|
name: Text Classification |
|
dataset: |
|
name: Unknown |
|
type: unknown |
|
split: test |
|
metrics: |
|
- type: accuracy |
|
value: 0.6716417910447762 |
|
name: Accuracy |
|
--- |
|
|
|
# SetFit with BAAI/bge-base-en-v1.5 |
|
|
|
This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification. |
|
|
|
The model has been trained using an efficient few-shot learning technique that involves: |
|
|
|
1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning. |
|
2. Training a classification head with features from the fine-tuned Sentence Transformer. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
- **Model Type:** SetFit |
|
- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) |
|
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance |
|
- **Maximum Sequence Length:** 512 tokens |
|
- **Number of Classes:** 2 classes |
|
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) --> |
|
<!-- - **Language:** Unknown --> |
|
<!-- - **License:** Unknown --> |
|
|
|
### Model Sources |
|
|
|
- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit) |
|
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055) |
|
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit) |
|
|
|
### Model Labels |
|
| Label | Examples | |
|
|:------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
|
| 0 | <ul><li>"Reasoning: \n\n1. **Context Grounding**: The answer pulls loosely from the documents provided but does not ground the response specifically to any of the documents' key points about budgeting or investment decisions.\n2. **Relevance**: While the query is about saving money in general, the answer ambiguously addresses saving money by suggesting wise spending and seeking guidance, which are indirectly related to what's mentioned in the documents.\n3. **Conciseness**: The answer contains unnecessary details like mentioning health and priorities, which are somewhat related based on Document 3 but are not directly in line with the exact scope of the question.\n4. **Too General**: The answer is broad and generic. The document contains specific guidelines (such as asking if the investment would be made with personal money and consulting with ORGANIZATION_2 for high-stakes cases), which the response fails to integrate effectively.\n5. **Detail Inclusion**: The answer needed to incorporate specific strategies mentioned in the documents, like ensuring if an expense aligns with personal and organizational benefits or seeking guidance on higher-budget items.\n\nFinal result: **Bad**"</li><li>'**Reasoning:**\n\n1. **Context Grounding**: The answer does mention several pet peeves directly from the provided documents, though it incorrectly repeats the names and has some formatting errors, it aligns generally with the content.\n2. **Relevance**: The answer focuses on the specific question about pet peeves, summarizing relevant points from the documents.\n3. **Conciseness**: The answer is cluttered with repeated names and formatting errors that make it unnecessarily lengthy and harder to read.\n4. **Avoiding unfounded responses**: The answer is founded on the document but suffers from poor editing issues.\n5. **Specificity**: The answer includes specific pet peeves, but it also includes unnecessary repetition.\n6. **Tip Inclusion**: The query did not ask for tips, so this criterion is not strictly applicable.\n\n**Final result**: **Bad**\n\nThe repeated names and formatting errors indicate lack of editing and care in the response, making it difficult to read and comprehend fully. It does not meet the required standard for a good, concise, and clear response.'</li><li>"Reasoning:\n\n1. **Context Grounding**: The answer only indirectly relates to the question. The provided information gives various ways to interact with the company's systems or tools rather than specifically how to access the training resources.\n2. **Relevance**: The answer deviates from the main question. It mentions systems for personal documents, expense claims, password management, secure sharing, feedback processes, and learning budgets, none of which directly answer how to access training resources.\n3. **Conciseness**: While the answer is detailed, it includes unnecessary information that isn't directly related to accessing training resources, making it not concise.\n4. **Specificity**: The answer lacks details on accessing training materials or platforms where these resources are housed.\n5. **Generalization**: It provides general guidance on several aspects but fails to narrow down specifically on training resources or offer relevant tips directly tied to them.\n\nFinal result: **Bad**"</li></ul> | |
|
| 1 | <ul><li>'**Reasoning:**\n\n1. **Context Grounding**: The answer references various elements from the documents provided, such as giving feedback at the time of the event, focusing on the situation and not the person, and showing appreciation. It generally aligns with the tips described in the documents.\n \n2. **Relevance**: The answer directly addresses the question of how feedback should be given, touching on many of the specific points mentioned in the documents.\n\n3. **Conciseness**: The response is overly wordy and includes some repetition (e.g., focusing on the situation not the person and the intention behind giving feedback). It can be more concise while still covering the necessary points.\n\n4. **Document Dependency**: The response accurately reflects the content of the provided documents. However, it brings in names like "Christopher Estes," "emichelle James Johnson MD," which may be elements of the document\'s placeholders rather than relevant names in the context of feedback tips. This adds unnecessary confusion to the answer.\n\n5. **Specificity**: The answer appropriately includes specific advice, though it overdoes some parts (like the role and intentions in giving feedback).\n\n6. **Accuracy and Clarity**: There are minor grammatical errors and some parts like "emichelle James Johnson MDamples" which are not clear and probably are typos or formatting errors.\n\nGiven the evaluation above:\n\n**Final Result: Bad**'</li><li>'Reasoning:\n1. **Context Grounding:** The answer refers to sharing information from high-level meetings which is supported by Document 4. However, the inclusion of "Robin Nichols" appears to be an error or typo, which diminishes the context grounding slightly.\n2. **Relevance:** The answer is relevant to the question asked; it covers the need for transparency, team alignment, addressing concerns, and fostering an open environment, which is in line with the information in Document 4.\n3. **Conciseness:** The answer is moderately concise but slightly verbose. It can be trimmed down while still maintaining the key points.\n4. **Directness:** The answer attempts to respond to the question based on the content provided in Document 4.\n5. **Specificity:** The answer includes specific points about decisions made and the rationale being shared, aligning the team and fostering a sense of purpose.\n6. **Tips:** The answer implicitly includes tips such as ensuring transparency and addressing team concerns.\n\nFinal Result: Good'</li><li>'**Reasoning:**\n\n1. **Context Grounding:** The given answer closely follows the instructions provided in Document 1 regarding car travel expenses. The essential details, such as keeping track of kilometers and submitting an Excel document or email to specified email addresses, are accurately mentioned.\n \n2. **Relevance:** The answer is pertinent to the question and provides specific steps on how to report car travel expenses, which is exactly what was asked.\n\n3. **Conciseness:** The response is generally concise, though it includes a minor unnecessary detail about requesting a parking card for a specific date which doesn’t directly pertain to the monthly reporting of car travel expenses.\n\n4. **Specificity:** The answer effectively covers the key points mentioned in the document for reporting travel expenses, including the specific reimbursement rate.\n\n5. **General Tips:** The tip about requesting a parking card is accurate but not directly relevant to the primary question of how to report car travel expenses.\n\nGiven the accurate detailing and mostly relevant content, while removing the unnecessary detail about the parking card date would improve the response, the answer is still sufficiently useful.\n\n**Final Result:** Good'</li></ul> | |
|
|
|
## Evaluation |
|
|
|
### Metrics |
|
| Label | Accuracy | |
|
|:--------|:---------| |
|
| **all** | 0.6716 | |
|
|
|
## Uses |
|
|
|
### Direct Use for Inference |
|
|
|
First install the SetFit library: |
|
|
|
```bash |
|
pip install setfit |
|
``` |
|
|
|
Then you can load this model and run inference. |
|
|
|
```python |
|
from setfit import SetFitModel |
|
|
|
# Download from the 🤗 Hub |
|
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_cot-instructions_only_reasoning_1726750606.384621") |
|
# Run inference |
|
preds = model("Reasoning: |
|
1. Context Grounding: The response draws from the documents providing relevant sources such as the organization's website, job ads, and newsletter link. |
|
2. Relevance: The answer is directly related to the question about understanding the organization's products, challenges, and future. |
|
3. Conciseness: The answer is clear and to the point. |
|
4. Does not attempt to respond when the document lacks information: It addresses the question appropriately with the available information. |
|
5. Specificity: The answer is specific and provides concrete steps to follow. |
|
6. Relevant tips: The answer includes actionable steps like visiting the website, viewing job ads, and signing up for a newsletter, which are relevant. |
|
|
|
The answer precisely matches all the criteria set for evaluation. |
|
|
|
Final Result: Good") |
|
``` |
|
|
|
<!-- |
|
### Downstream Use |
|
|
|
*List how someone could finetune this model on their own dataset.* |
|
--> |
|
|
|
<!-- |
|
### Out-of-Scope Use |
|
|
|
*List how the model may foreseeably be misused and address what users ought not to do with the model.* |
|
--> |
|
|
|
<!-- |
|
## Bias, Risks and Limitations |
|
|
|
*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* |
|
--> |
|
|
|
<!-- |
|
### Recommendations |
|
|
|
*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* |
|
--> |
|
|
|
## Training Details |
|
|
|
### Training Set Metrics |
|
| Training set | Min | Median | Max | |
|
|:-------------|:----|:---------|:----| |
|
| Word count | 87 | 141.3077 | 245 | |
|
|
|
| Label | Training Sample Count | |
|
|:------|:----------------------| |
|
| 0 | 32 | |
|
| 1 | 33 | |
|
|
|
### Training Hyperparameters |
|
- batch_size: (16, 16) |
|
- num_epochs: (5, 5) |
|
- max_steps: -1 |
|
- sampling_strategy: oversampling |
|
- num_iterations: 20 |
|
- body_learning_rate: (2e-05, 2e-05) |
|
- head_learning_rate: 2e-05 |
|
- loss: CosineSimilarityLoss |
|
- distance_metric: cosine_distance |
|
- margin: 0.25 |
|
- end_to_end: False |
|
- use_amp: False |
|
- warmup_proportion: 0.1 |
|
- l2_weight: 0.01 |
|
- seed: 42 |
|
- eval_max_steps: -1 |
|
- load_best_model_at_end: False |
|
|
|
### Training Results |
|
| Epoch | Step | Training Loss | Validation Loss | |
|
|:------:|:----:|:-------------:|:---------------:| |
|
| 0.0061 | 1 | 0.2339 | - | |
|
| 0.3067 | 50 | 0.2693 | - | |
|
| 0.6135 | 100 | 0.2364 | - | |
|
| 0.9202 | 150 | 0.0942 | - | |
|
| 1.2270 | 200 | 0.0031 | - | |
|
| 1.5337 | 250 | 0.0019 | - | |
|
| 1.8405 | 300 | 0.0016 | - | |
|
| 2.1472 | 350 | 0.0016 | - | |
|
| 2.4540 | 400 | 0.0015 | - | |
|
| 2.7607 | 450 | 0.0013 | - | |
|
| 3.0675 | 500 | 0.0013 | - | |
|
| 3.3742 | 550 | 0.0012 | - | |
|
| 3.6810 | 600 | 0.0012 | - | |
|
| 3.9877 | 650 | 0.0012 | - | |
|
| 4.2945 | 700 | 0.0012 | - | |
|
| 4.6012 | 750 | 0.0011 | - | |
|
| 4.9080 | 800 | 0.0011 | - | |
|
|
|
### Framework Versions |
|
- Python: 3.10.14 |
|
- SetFit: 1.1.0 |
|
- Sentence Transformers: 3.1.0 |
|
- Transformers: 4.44.0 |
|
- PyTorch: 2.4.1+cu121 |
|
- Datasets: 2.19.2 |
|
- Tokenizers: 0.19.1 |
|
|
|
## Citation |
|
|
|
### BibTeX |
|
```bibtex |
|
@article{https://doi.org/10.48550/arxiv.2209.11055, |
|
doi = {10.48550/ARXIV.2209.11055}, |
|
url = {https://arxiv.org/abs/2209.11055}, |
|
author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren}, |
|
keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences}, |
|
title = {Efficient Few-Shot Learning Without Prompts}, |
|
publisher = {arXiv}, |
|
year = {2022}, |
|
copyright = {Creative Commons Attribution 4.0 International} |
|
} |
|
``` |
|
|
|
<!-- |
|
## Glossary |
|
|
|
*Clearly define terms in order to be accessible across audiences.* |
|
--> |
|
|
|
<!-- |
|
## Model Card Authors |
|
|
|
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* |
|
--> |
|
|
|
<!-- |
|
## Model Card Contact |
|
|
|
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* |
|
--> |
