srikarvar's picture
Add new SentenceTransformer model.
747cfc9 verified
metadata
base_model: intfloat/multilingual-e5-small
datasets: []
language: []
library_name: sentence-transformers
metrics:
  - cosine_accuracy
  - dot_accuracy
  - manhattan_accuracy
  - euclidean_accuracy
  - max_accuracy
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:1204
  - loss:TripletLoss
widget:
  - source_sentence: How do I publish articles?
    sentences:
      - How do I publish an article?
      - Steps to meditate
      - How I publish my article on Yahoo?
  - source_sentence: Who is the author of '1984'?
    sentences:
      - North America's largest lake by area
      - Writer of the novel '1984'
      - Who is the author of 'Pride and Prejudice'?
  - source_sentence: What are adverbs? What are some kind of adverbs?
    sentences:
      - How can I get rid of flying cockroaches?
      - What are some examples of adverbs?
      - What's the difference between adverbial phrase and adverb phrase?
  - source_sentence: Do you believe in astrology? Is it true?
    sentences:
      - Are horoscopes legitimate? Do they ever come true?
      - Today is my birthday. Why does no one wish me a happy birthday?
      - Do you believe in horoscope?
  - source_sentence: >-
      After marriage, why do women have to change their surnames to their
      husband’s? Why can't they keep their maiden ones?
    sentences:
      - Steps to start a blog
      - After marriage, why do women have to change their surname?
      - >-
        Is it possible for an Indian woman not to change her surname after
        marriage?
model-index:
  - name: SentenceTransformer based on intfloat/multilingual-e5-small
    results:
      - task:
          type: triplet
          name: Triplet
        dataset:
          name: triplet validation
          type: triplet-validation
        metrics:
          - type: cosine_accuracy
            value: 0.9917355371900827
            name: Cosine Accuracy
          - type: dot_accuracy
            value: 0.008264462809917356
            name: Dot Accuracy
          - type: manhattan_accuracy
            value: 0.9917355371900827
            name: Manhattan Accuracy
          - type: euclidean_accuracy
            value: 0.9917355371900827
            name: Euclidean Accuracy
          - type: max_accuracy
            value: 0.9917355371900827
            name: Max Accuracy

SentenceTransformer based on intfloat/multilingual-e5-small

This is a sentence-transformers model finetuned from intfloat/multilingual-e5-small. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: intfloat/multilingual-e5-small
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 384 tokens
  • Similarity Function: Cosine Similarity

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("srikarvar/e-small-triplet-balanced")
# Run inference
sentences = [
    "After marriage, why do women have to change their surnames to their husband’s? Why can't they keep their maiden ones?",
    'After marriage, why do women have to change their surname?',
    'Is it possible for an Indian woman not to change her surname after marriage?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

Metric Value
cosine_accuracy 0.9917
dot_accuracy 0.0083
manhattan_accuracy 0.9917
euclidean_accuracy 0.9917
max_accuracy 0.9917

Training Details

Training Dataset

Unnamed Dataset

  • Size: 1,204 training samples
  • Columns: anchor, positive, and negative
  • Approximate statistics based on the first 1000 samples:
    anchor positive negative
    type string string string
    details
    • min: 6 tokens
    • mean: 12.25 tokens
    • max: 43 tokens
    • min: 4 tokens
    • mean: 11.44 tokens
    • max: 50 tokens
    • min: 4 tokens
    • mean: 12.68 tokens
    • max: 59 tokens
  • Samples:
    anchor positive negative
    What are the ingredients of a pizza? ingredients of pizza? What are the ingredients of a burger?
    How does photosynthesis work? Explain the process of photosynthesis How does respiration work?
    How do I reset my password? Steps to reset password How do I change my username?
  • Loss: TripletLoss with these parameters:
    {
        "distance_metric": "TripletDistanceMetric.EUCLIDEAN",
        "triplet_margin": 5
    }
    

Evaluation Dataset

Unnamed Dataset

  • Size: 121 evaluation samples
  • Columns: anchor, positive, and negative
  • Approximate statistics based on the first 1000 samples:
    anchor positive negative
    type string string string
    details
    • min: 7 tokens
    • mean: 12.83 tokens
    • max: 38 tokens
    • min: 6 tokens
    • mean: 11.77 tokens
    • max: 38 tokens
    • min: 6 tokens
    • mean: 13.2 tokens
    • max: 48 tokens
  • Samples:
    anchor positive negative
    What is the best way to learn a new language? How can I effectively learn a new language? What is the fastest way to travel?
    Can people actively control their emotions? Does our mind control our emotions? How can I control my positive emotions for the people whom I love but they don't care about me?
    Which can be the best laptop under 30000? which laptop will be best under Rs 30,000? What is the best phone to buy under 30000 in India?
  • Loss: TripletLoss with these parameters:
    {
        "distance_metric": "TripletDistanceMetric.EUCLIDEAN",
        "triplet_margin": 5
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: epoch
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 32
  • gradient_accumulation_steps: 2
  • learning_rate: 3e-05
  • weight_decay: 0.01
  • num_train_epochs: 8
  • lr_scheduler_type: reduce_lr_on_plateau
  • warmup_ratio: 0.1
  • load_best_model_at_end: True
  • optim: adamw_torch_fused

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: epoch
  • prediction_loss_only: True
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 32
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 2
  • eval_accumulation_steps: None
  • learning_rate: 3e-05
  • weight_decay: 0.01
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 8
  • max_steps: -1
  • lr_scheduler_type: reduce_lr_on_plateau
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch_fused
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss loss triplet-validation_max_accuracy
0.5263 10 4.8459 - -
1.0 19 - 4.4155 -
1.0526 20 4.7205 - -
1.5789 30 4.5948 - -
2.0 38 - 4.2163 -
2.1053 40 4.5125 - -
2.6316 50 4.4761 - -
3.0 57 - 4.1338 -
3.1579 60 4.452 - -
3.6842 70 4.4082 - -
4.0 76 - 4.0659 -
4.2105 80 4.3978 - -
4.7368 90 4.3495 - -
5.0 95 - 4.0202 -
5.2632 100 4.287 - -
5.7895 110 4.2805 - -
6.0 114 - 3.9441 -
6.3158 120 4.2631 - -
6.8421 130 4.213 - -
7.0 133 - 3.8866 -
7.3684 140 4.1921 - -
7.8947 150 4.1854 - -
8.0 152 - 3.8757 0.9917
  • The bold row denotes the saved checkpoint.

Framework Versions

  • Python: 3.10.12
  • Sentence Transformers: 3.0.1
  • Transformers: 4.41.2
  • PyTorch: 2.1.2+cu121
  • Accelerate: 0.32.1
  • Datasets: 2.19.1
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification}, 
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}