README.md

metadata

language: []
library_name: sentence-transformers
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - dataset_size:10K<n<100K
  - loss:CoSENTLoss
base_model: sentence-transformers/all-MiniLM-L6-v2
metrics:
  - pearson_cosine
  - spearman_cosine
  - pearson_manhattan
  - spearman_manhattan
  - pearson_euclidean
  - spearman_euclidean
  - pearson_dot
  - spearman_dot
  - pearson_max
  - spearman_max
widget:
  - source_sentence: >-
      Driving or commuting to work feels draining, even if it's a short
      distance.
    sentences:
      - >-
        Symptoms during a manic episode include decreased need for sleep, more
        talkative than usual, flight of ideas, distractibility
      - >-
        I feel like I have lost a part of myself since the traumatic event, and
        I struggle to connect with others on a deeper level.
      - >-
        Diagnosis requires at least one hypomanic episode and one major
        depressive episode.
  - source_sentence: >-
      I felt like my thoughts were disconnected and chaotic during a manic
      episode.
    sentences:
      - >-
        Diagnosis requires one or more manic episodes, which may be preceded or
        followed by hypomanic or major depressive episodes.
      - >-
        I feel like I have lost a part of myself since the traumatic event, and
        I struggle to connect with others on a deeper level.
      - >-
        Depressed mood for most of the day, for more days than not, as indicated
        by subjective account or observation, for at least 2 years.
  - source_sentence: >-
      My insomnia has caused me to experience frequent headaches and muscle
      soreness.
    sentences:
      - Insomnia or hypersomnia nearly every day.
      - >-
        I have difficulty standing in long lines at the grocery store or the
        bank due to the fear of feeling trapped or overwhelmed.
      - >-
        Diagnosis requires at least one hypomanic episode and one major
        depressive episode.
  - source_sentence: >-
      The phobic object or situation almost always provokes immediate fear or
      anxiety.
    sentences:
      - The agoraphobic situations almost always provoke fear or anxiety.
      - >-
        I have difficulty standing in long lines at the grocery store or the
        bank due to the fear of feeling trapped or overwhelmed.
      - >-
        Exclusion of schizoaffective disorder and depressive or bipolar disorder
        with psychotic features, based on the absence of concurrent depressive
        or manic episodes during the active-phase symptoms, or these mood
        episodes being present for a minority of the total duration of the
        active and residual phases.
  - source_sentence: >-
      I engage in risky behaviors like reckless driving or reckless sexual
      encounters.
    sentences:
      - >-
        Symptoms during a manic episode include inflated self-esteem or
        grandiosity,increased goal-directed activity, or excessive involvement
        in risky activities.
      - >-
        Marked decrease in functioning in areas like work, interpersonal
        relations, or self-care since the onset of the disturbance.
      - >-
        During the specified period, symptoms from Criterion A are present at
        least half the time with no symptom-free interval lasting longer than 2
        months.
pipeline_tag: sentence-similarity
model-index:
  - name: SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2
    results:
      - task:
          type: semantic-similarity
          name: Semantic Similarity
        dataset:
          name: FT label
          type: FT_label
        metrics:
          - type: pearson_cosine
            value: 0.4627701543833943
            name: Pearson Cosine
          - type: spearman_cosine
            value: 0.4076356119364853
            name: Spearman Cosine
          - type: pearson_manhattan
            value: 0.48164714740150605
            name: Pearson Manhattan
          - type: spearman_manhattan
            value: 0.406731043246377
            name: Spearman Manhattan
          - type: pearson_euclidean
            value: 0.4840582172096936
            name: Pearson Euclidean
          - type: spearman_euclidean
            value: 0.407636256115058
            name: Spearman Euclidean
          - type: pearson_dot
            value: 0.46277015122653486
            name: Pearson Dot
          - type: spearman_dot
            value: 0.4076359510487126
            name: Spearman Dot
          - type: pearson_max
            value: 0.4840582172096936
            name: Pearson Max
          - type: spearman_max
            value: 0.407636256115058
            name: Spearman Max

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Hgkang00/FT-label-consent-20")
# Run inference
sentences = [
    'I engage in risky behaviors like reckless driving or reckless sexual encounters.',
    'Symptoms during a manic episode include inflated self-esteem or grandiosity,increased goal-directed activity, or excessive involvement in risky activities.',
    'Marked decrease in functioning in areas like work, interpersonal relations, or self-care since the onset of the disturbance.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Semantic Similarity

• Dataset: FT_label
• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.4628
spearman_cosine	0.4076
pearson_manhattan	0.4816
spearman_manhattan	0.4067
pearson_euclidean	0.4841
spearman_euclidean	0.4076
pearson_dot	0.4628
spearman_dot	0.4076
pearson_max	0.4841
spearman_max	0.4076

Training Details

Training Dataset

Unnamed Dataset

• Size: 33,800 training samples
• Columns: sentence1, sentence2, and score
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	score
  type	string	string	float
  details	min: 11 tokens mean: 31.63 tokens max: 63 tokens	min: 14 tokens mean: 25.22 tokens max: 41 tokens	min: -1.0 mean: -0.87 max: 1.0

• Samples:

  sentence1	sentence2	score
  Presence of one or more of the following intrusion symptoms associated with the traumatic event: recurrent distressing memories, dreams, flashbacks, psychological distress, or physiological reactions to cues of the traumatic event.	I avoid making phone calls, even to close friends or family, because I'm afraid of saying something wrong or sounding awkward.	0.0
  The phobic object or situation almost always provokes immediate fear or anxiety.	I find it hard to stick to a consistent eating schedule, sometimes going days without feeling the need to eat at all.	-1.0
  The fear or anxiety is out of proportion to the actual danger posed by the specific object or situation and to the sociocultural context.	I have difficulty going to places where I feel there are no immediate exits, such as cinemas or auditoriums, as the fear of being stuck or unable to escape escalates my anxiety.	-1.0

• Loss: CoSENTLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 4,225 evaluation samples
• Columns: sentence1, sentence2, and score
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	score
  type	string	string	float
  details	min: 11 tokens mean: 31.24 tokens max: 63 tokens	min: 15 tokens mean: 24.86 tokens max: 41 tokens	min: -1.0 mean: -0.87 max: 1.0

• Samples:

  sentence1	sentence2	score
  Excessive anxiety and worry occurring more days than not for at least 6 months, about a number of events or activities such as work or school performance.	Simple activities like going for a walk or doing household chores feel like daunting tasks due to my low energy levels.	-1.0
  The individual fears acting in a way or showing anxiety symptoms that will be negatively evaluated, leading to humiliation, embarrassment, rejection, or offense to others.	I often find myself mindlessly snacking throughout the day due to changes in my appetite.	-1.0
  Persistent avoidance of stimuli associated with the trauma, evidenced by avoiding distressing memories, thoughts, or feelings, or external reminders of the event.	Simple activities like going for a walk or doing household chores feel like daunting tasks due to my low energy levels.	-1.0

• Loss: CoSENTLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• num_train_epochs: 20
• warmup_ratio: 0.1

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 20
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	loss	FT_label_spearman_cosine
1.0	265	-	6.9529	0.3450
2.0	530	7.5663	7.1002	0.4103
3.0	795	-	7.4786	0.4155
4.0	1060	5.5492	8.6710	0.4115
5.0	1325	-	10.3786	0.4056
6.0	1590	4.3991	10.4239	0.3987
7.0	1855	-	11.8681	0.4238
8.0	2120	3.5916	13.0752	0.4030
9.0	2385	-	12.8567	0.4240
10.0	2650	3.1139	12.4373	0.4270
11.0	2915	-	13.6725	0.4212
12.0	3180	2.6658	15.0521	0.4134
13.0	3445	-	15.4305	0.4114
14.0	3710	2.2024	15.5511	0.4060
15.0	3975	-	14.9427	0.4165
16.0	4240	1.8955	14.8399	0.4162
17.0	4505	-	15.0070	0.4170
18.0	4770	1.712	15.4417	0.4105
19.0	5035	-	15.6241	0.4086
20.0	5300	1.5088	15.6818	0.4076

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.0.0
• Transformers: 4.41.1
• PyTorch: 2.3.0+cu121
• Accelerate: 0.30.1
• Datasets: 2.19.1
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CoSENTLoss

@online{kexuefm-8847,
    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
    author={Su Jianlin},
    year={2022},
    month={Jan},
    url={https://kexue.fm/archives/8847},
}