Netta1994/setfit_baai_squad_gpt-4o_improved-cot-instructions_chat_few_shot_generated_only_reaso

SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: BAAI/bge-base-en-v1.5
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 512 tokens
• Number of Classes: 2 classes

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
1	'Reasoning:\nThe provided document confirms that Silius Italicus' epic in which Virgil was referenced in almost every line is titled "Punica." The answer accurately reflects this information, demonstrating good context grounding and relevance to the question. The response is also concise and directly addresses the question without unnecessary details.\n\nEvaluation: Good' 'Reasoning:\nThe given answer correctly identifies Father John Francis O'Hara as the individual who became vice-president of Notre Dame in 1933. This is well-supported by the document, which clearly states "Holy Cross Father John Francis O'Hara was elected vice-president in 1933." The answer directly addresses the specific question without deviation or inclusion of unnecessary details.\n\nEvaluation: Good' 'Reasoning:\nThe answer provided, "3,000 police were said to have protected the torch in France," is well-supported by the document, which mentions that an estimated 3,000 French police protected the Olympic torch relay. The answer is directly related to the question asked and does not include any additional, unnecessary information, making it concise and to the point.\nEvaluation: Good'
0	"Reasoning:\nThe given answer states that Father James Edward O'Hara became vice-president of Notre Dame in 1934. However, the document clearly indicates that Holy Cross Father John Francis O'Hara was elected vice-president in 1933, not 1934. The answer provided does not correctly reflect the information in the document and therefore does not accurately address the question asked.\n\nEvaluation: Bad" 'Reasoning:\nThe answer states that Beyoncé grossed $119.5 million during her second world tour in 2009. The document supports this, noting that her I Am... World Tour in 2009 consisted of 108 shows and grossed $119.5 million. The answer is directly relevant to the question, and it provides the correct information concisely without adding unnecessary details.\n\nEvaluation: Good' 'Reasoning:\nThe answer states that "Manhattan contains the highest population of Asian-Americans among all the boroughs of New York," which is incorrect according to the provided document. The document clearly specifies that the New York City borough of Queens is home to the state's largest Asian American population. Thus, the answer does not match the information provided and fails to address the question accurately.\n\nEvaluation: Bad'

Evaluation

Metrics

Label	Accuracy
all	0.9180

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_squad_gpt-4o_improved-cot-instructions_chat_few_shot_generated_only_reaso")
# Run inference
preds = model("Reasoning:
The provided answer correctly identifies Mick LaSalle as the writer for the San Francisco Chronicle who awarded \"Spectre\" with a perfect score. This is directly supported by the document, which states, \"Other positive reviews from Mick LaSalle from the San Francisco Chronicle, gave it a perfect 100 score...\"

Evaluation: Good")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	43	62.3158	91

Label	Training Sample Count
0	27
1	30

Training Hyperparameters

• batch_size: (16, 16)
• num_epochs: (5, 5)
• max_steps: -1
• sampling_strategy: oversampling
• num_iterations: 20
• body_learning_rate: (2e-05, 2e-05)
• head_learning_rate: 2e-05
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• l2_weight: 0.01
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0070	1	0.1842	-
0.3497	50	0.2439	-
0.6993	100	0.0606	-
1.0490	150	0.0055	-
1.3986	200	0.0028	-
1.7483	250	0.0022	-
2.0979	300	0.0018	-
2.4476	350	0.0017	-
2.7972	400	0.0015	-
3.1469	450	0.0014	-
3.4965	500	0.0014	-
3.8462	550	0.0014	-
4.1958	600	0.0013	-
4.5455	650	0.0012	-
4.8951	700	0.0013	-

Framework Versions

• Python: 3.10.14
• SetFit: 1.1.0
• Sentence Transformers: 3.1.0
• Transformers: 4.44.0
• PyTorch: 2.4.1+cu121
• Datasets: 2.19.2
• Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}