Hugging Face's logo

cferreiragonz
/
bge-base-fastdds-questions-30-epochs

Sentence Similarity
sentence-transformers
Safetensors
English
bert
feature-extraction
Generated from Trainer
dataset_size:3853
loss:MatryoshkaLoss
loss:MultipleNegativesRankingLoss
Eval Results
text-embeddings-inference
Inference Endpoints
Model card Files Community
Edit model card

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

Fine tuning poc1-30e

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: BAAI/bge-base-en-v1.5
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 768 tokens
  • Similarity Function: Cosine Similarity
  • Language: en
  • License: apache-2.0

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("cferreiragonz/bge-base-fastdds-questions-30-epochs")
# Run inference
sentences = [
    'After calling the "DataReader::read()" or "DataReader::take()"\noperations, accessing the data on the returned sequences is quite\neasy. The sequences API provides a **length()** operation returning\nthe number of elements in the collections. The application code just\nneeds to check this value and use the **[]** operator to access the\ncorresponding elements. Elements on the DDS data sequence should only\nbe accessed when the corresponding element on the SampleInfo sequence\nindicate that valid data is present. When using Data Sharing, it is\nalso important to check that the sample is valid (i.e, not replaced,\nrefer to DataReader and DataWriter history coupling for further\ninformation in this regard).',
    'What is the primary method described in the text for accessing data on returned sequences after calling "DataReader::read()" or "DataReader::take()" operations?',
    'What are the steps to install Fast DDS library, Python bindings, and Gen generation tool from sources in a Linux environment?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Metric Value
cosine_accuracy@1 0.3427
cosine_accuracy@3 0.5291
cosine_accuracy@5 0.5758
cosine_accuracy@10 0.6643
cosine_precision@1 0.3427
cosine_precision@3 0.1764
cosine_precision@5 0.1152
cosine_precision@10 0.0664
cosine_recall@1 0.3427
cosine_recall@3 0.5291
cosine_recall@5 0.5758
cosine_recall@10 0.6643
cosine_ndcg@10 0.4999
cosine_mrr@10 0.4478
cosine_map@100 0.4573

Information Retrieval

Metric Value
cosine_accuracy@1 0.3333
cosine_accuracy@3 0.5315
cosine_accuracy@5 0.5804
cosine_accuracy@10 0.655
cosine_precision@1 0.3333
cosine_precision@3 0.1772
cosine_precision@5 0.1161
cosine_precision@10 0.0655
cosine_recall@1 0.3333
cosine_recall@3 0.5315
cosine_recall@5 0.5804
cosine_recall@10 0.655
cosine_ndcg@10 0.4931
cosine_mrr@10 0.4415
cosine_map@100 0.4521

Information Retrieval

Metric Value
cosine_accuracy@1 0.331
cosine_accuracy@3 0.5385
cosine_accuracy@5 0.5734
cosine_accuracy@10 0.662
cosine_precision@1 0.331
cosine_precision@3 0.1795
cosine_precision@5 0.1147
cosine_precision@10 0.0662
cosine_recall@1 0.331
cosine_recall@3 0.5385
cosine_recall@5 0.5734
cosine_recall@10 0.662
cosine_ndcg@10 0.4946
cosine_mrr@10 0.4415
cosine_map@100 0.4518

Information Retrieval

Metric Value
cosine_accuracy@1 0.3287
cosine_accuracy@3 0.5291
cosine_accuracy@5 0.5781
cosine_accuracy@10 0.6643
cosine_precision@1 0.3287
cosine_precision@3 0.1764
cosine_precision@5 0.1156
cosine_precision@10 0.0664
cosine_recall@1 0.3287
cosine_recall@3 0.5291
cosine_recall@5 0.5781
cosine_recall@10 0.6643
cosine_ndcg@10 0.4917
cosine_mrr@10 0.4371
cosine_map@100 0.4465

Information Retrieval

Metric Value
cosine_accuracy@1 0.317
cosine_accuracy@3 0.5058
cosine_accuracy@5 0.5734
cosine_accuracy@10 0.655
cosine_precision@1 0.317
cosine_precision@3 0.1686
cosine_precision@5 0.1147
cosine_precision@10 0.0655
cosine_recall@1 0.317
cosine_recall@3 0.5058
cosine_recall@5 0.5734
cosine_recall@10 0.655
cosine_ndcg@10 0.4805
cosine_mrr@10 0.4252
cosine_map@100 0.4342

Training Details

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: epoch
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • gradient_accumulation_steps: 16
  • learning_rate: 2e-05
  • num_train_epochs: 30
  • lr_scheduler_type: cosine
  • warmup_ratio: 0.1
  • fp16: True
  • tf32: False
  • load_best_model_at_end: True
  • optim: adamw_torch_fused
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: epoch
  • prediction_loss_only: True
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 16
  • eval_accumulation_steps: None
  • learning_rate: 2e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 30
  • max_steps: -1
  • lr_scheduler_type: cosine
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: True
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: False
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch_fused
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss dim_128_cosine_map@100 dim_256_cosine_map@100 dim_512_cosine_map@100 dim_64_cosine_map@100 dim_768_cosine_map@100
0.6639 10 5.6138 - - - - -
0.9959 15 - 0.3594 0.3735 0.3723 0.3161 0.3807
1.3278 20 4.9173 - - - - -
1.9917 30 3.7581 0.3874 0.4014 0.4026 0.3729 0.4032
2.6556 40 3.0018 - - - - -
2.9876 45 - 0.4031 0.4200 0.4212 0.3858 0.4223
3.3195 50 2.5035 - - - - -
3.9834 60 1.9031 0.4187 0.4303 0.4178 0.3958 0.4291
4.6473 70 1.474 - - - - -
4.9793 75 - 0.4293 0.4332 0.4318 0.4172 0.4401
5.3112 80 1.2801 - - - - -
5.9751 90 0.9577 0.4397 0.4382 0.4444 0.4275 0.4518
6.6390 100 0.7539 - - - - -
6.9710 105 - 0.4434 0.4414 0.4496 0.4262 0.4466
7.3029 110 0.694 - - - - -
7.9668 120 0.5147 0.4423 0.4488 0.4507 0.4358 0.4495
8.6307 130 0.4589 - - - - -
8.9627 135 - 0.4488 0.4575 0.4544 0.4407 0.4493
9.2946 140 0.3843 - - - - -
9.9585 150 0.3506 0.4521 0.4465 0.4559 0.4420 0.4485
10.6224 160 0.2723 - - - - -
10.9544 165 - 0.4497 0.4435 0.4499 0.4304 0.4453
11.2863 170 0.2555 - - - - -
11.9502 180 0.2077 0.4448 0.4472 0.4468 0.4287 0.4453
12.6141 190 0.1894 - - - - -
12.9461 195 - 0.4516 0.4463 0.4566 0.4336 0.452
13.2780 200 0.1725 - - - - -
13.9419 210 0.1395 0.4528 0.4520 0.4561 0.4333 0.4534
14.6058 220 0.155 - - - - -
14.9378 225 - 0.4461 0.4491 0.4527 0.4369 0.4517
15.2697 230 0.132 - - - - -
15.9336 240 0.1148 - - - - -
16.0 241 - 0.4482 0.4537 0.4540 0.4303 0.4538
16.5975 250 0.1061 - - - - -
16.9959 256 - 0.4464 0.4538 0.4551 0.4294 0.4577
17.2614 260 0.0961 - - - - -
17.9253 270 0.087 - - - - -
17.9917 271 - 0.4485 0.4483 0.4495 0.4326 0.4568
18.5892 280 0.1009 - - - - -
18.9876 286 - 0.4483 0.4517 0.4545 0.4396 0.4565
19.2531 290 0.0854 - - - - -
19.9170 300 0.073 - - - - -
19.9834 301 - 0.4473 0.4502 0.4521 0.4349 0.4548
20.5809 310 0.0726 - - - - -
20.9793 316 - 0.4466 0.4525 0.4538 0.4341 0.4583
21.2448 320 0.0747 - - - - -
21.9087 330 0.0621 - - - - -
21.9751 331 - 0.4441 0.4537 0.4534 0.4388 0.4564
22.5726 340 0.0682 - - - - -
22.9710 346 - 0.4454 0.4529 0.4544 0.4385 0.4589
23.2365 350 0.0612 - - - - -
23.9004 360 0.0546 - - - - -
23.9668 361 - 0.4464 0.4494 0.4551 0.4381 0.4567
24.5643 370 0.0599 - - - - -
24.9627 376 - 0.4465 0.4506 0.4553 0.4363 0.4567
25.2282 380 0.0591 - - - - -
25.8921 390 0.0562 - - - - -
25.9585 391 - 0.4454 0.4515 0.4532 0.4343 0.4575
26.5560 400 0.0623 - - - - -
26.9544 406 - 0.4452 0.4531 0.4544 0.4342 0.4573
27.2199 410 0.061 - - - - -
27.8838 420 0.053 - - - - -
27.9502 421 - 0.4454 0.4514 0.4533 0.4330 0.4573
28.5477 430 0.0564 - - - - -
28.9461 436 - 0.4465 0.4516 0.4533 0.4338 0.4562
29.2116 440 0.056 - - - - -
29.8755 450 0.0586 0.4465 0.4518 0.4521 0.4342 0.4573
  • The bold row denotes the saved checkpoint.

Framework Versions

  • Python: 3.10.13
  • Sentence Transformers: 3.0.1
  • Transformers: 4.41.2
  • PyTorch: 2.1.2
  • Accelerate: 0.30.1
  • Datasets: 2.19.1
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss 

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss 

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
Downloads last month
0
Safetensors
Model size
109M params
Tensor type
F32
·
Inference Examples
Sentence Similarity
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for cferreiragonz/bge-base-fastdds-questions-30-epochs

Base model

BAAI/bge-base-en-v1.5
Finetuned
(253)
this model

Evaluation results

View on Papers With Code