--- language: - id library_name: sentence-transformers tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:10000 - loss:SoftmaxLoss base_model: indobenchmark/indobert-base-p2 datasets: - afaji/indonli metrics: - pearson_cosine - spearman_cosine - pearson_manhattan - spearman_manhattan - pearson_euclidean - spearman_euclidean - pearson_dot - spearman_dot - pearson_max - spearman_max widget: - source_sentence: Dengan meniupnya, perawat bisa segera mengerti bahwa ia dipanggil dan akan segera datang menolong. sentences: - 38% pemilih tidak mendukung meninggalkan Uni Eropa. - Perawat mengerti bahwa ia dipanggil dan akan segera datang menolong. - Dari fakta-fakta tersebut dapat diindikasikan pembakaran gereja dilakukan secara sengaja. - source_sentence: Kebudayaan jawa lainnya adalah Sintren, Sintren adalan kesenian tradisional masyarakat Jawa, khususnya Pekalongan. sentences: - Sintren merupakan kesenian tradisional masyarakat Jawa yang ada sejak zaman kerajaan. - Klinik ini melarang pasiennya menghisap ganja. - Perubahan dunia saat itu dipengaruhi oleh Krisis Suez. - source_sentence: Saat ini, sudah empat wanita yang mengaku dilecehkan. Yang terakhir ialah aktris Rose McGowan, dengan tuntutan pemerkosaan. sentences: - Di Maroko Tenggara tidak pernah ada fosil vertebrata. - Tidak ada yang dilecehkan. - Ganja tidak boleh diberikan kepada pasien penyakit apapun. - source_sentence: Peperangan di tanah berubah dari lini depan statis Perang Dunia I menjadi peningkatan mobilitas dan persenjataan gabungan. sentences: - Peperangan di tanah awalnya berbentuk lini depan statis Perang Dunia I. - Ia berdarah keturunan India. - Kesultanan Yogyakarta berasal dari Kerajaan Mataram. - source_sentence: Bahan dasar Dalgona Coffee hanya tiga jenis yaitu bubuk kopi, gula, dan air. Banyak resep beredar dengan komposisi dua sendok bubuk kopi, dua sendok gula, dan dua sendok air panas. sentences: - Semua orang di dunia menyukai air putih. - Jutting berada di Pengadilan Tinggi Hongkong 5 tahun kemudian. - Resep komposisi Dalgona Coffee adalah 2 sendok bubuk kopi. pipeline_tag: sentence-similarity model-index: - name: SentenceTransformer based on indobenchmark/indobert-base-p2 results: - task: type: semantic-similarity name: Semantic Similarity dataset: name: sts dev type: sts-dev metrics: - type: pearson_cosine value: -0.4766226820019628 name: Pearson Cosine - type: spearman_cosine value: -0.4665046363205431 name: Spearman Cosine - type: pearson_manhattan value: -0.46278474137062864 name: Pearson Manhattan - type: spearman_manhattan value: -0.46103038796182516 name: Spearman Manhattan - type: pearson_euclidean value: -0.4732431317820645 name: Pearson Euclidean - type: spearman_euclidean value: -0.4673139200425683 name: Spearman Euclidean - type: pearson_dot value: -0.4679129419420587 name: Pearson Dot - type: spearman_dot value: -0.4577457216480116 name: Spearman Dot - type: pearson_max value: -0.46278474137062864 name: Pearson Max - type: spearman_max value: -0.4577457216480116 name: Spearman Max - task: type: semantic-similarity name: Semantic Similarity dataset: name: sts test type: sts-test metrics: - type: pearson_cosine value: -0.20358655624514646 name: Pearson Cosine - type: spearman_cosine value: -0.20098073423584242 name: Spearman Cosine - type: pearson_manhattan value: -0.16857445418120778 name: Pearson Manhattan - type: spearman_manhattan value: -0.18417229002858432 name: Spearman Manhattan - type: pearson_euclidean value: -0.17954736289799147 name: Pearson Euclidean - type: spearman_euclidean value: -0.1907831094006202 name: Spearman Euclidean - type: pearson_dot value: -0.2158654981443921 name: Pearson Dot - type: spearman_dot value: -0.2141585054513143 name: Spearman Dot - type: pearson_max value: -0.16857445418120778 name: Pearson Max - type: spearman_max value: -0.18417229002858432 name: Spearman Max --- # SentenceTransformer based on indobenchmark/indobert-base-p2 This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [indobenchmark/indobert-base-p2](https://huggingface.co/indobenchmark/indobert-base-p2) on the [afaji/indonli](https://huggingface.co/datasets/afaji/indonli) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [indobenchmark/indobert-base-p2](https://huggingface.co/indobenchmark/indobert-base-p2) - **Maximum Sequence Length:** 512 tokens - **Output Dimensionality:** 768 tokens - **Similarity Function:** Cosine Similarity - **Training Dataset:** - [afaji/indonli](https://huggingface.co/datasets/afaji/indonli) - **Language:** id ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("cassador/indobert-base-p2-nli-v1") # Run inference sentences = [ 'Bahan dasar Dalgona Coffee hanya tiga jenis yaitu bubuk kopi, gula, dan air. Banyak resep beredar dengan komposisi dua sendok bubuk kopi, dua sendok gula, dan dua sendok air panas.', 'Resep komposisi Dalgona Coffee adalah 2 sendok bubuk kopi.', 'Jutting berada di Pengadilan Tinggi Hongkong 5 tahun kemudian.', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Semantic Similarity * Dataset: `sts-dev` * Evaluated with [EmbeddingSimilarityEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator) | Metric | Value | |:--------------------|:------------| | pearson_cosine | -0.4766 | | **spearman_cosine** | **-0.4665** | | pearson_manhattan | -0.4628 | | spearman_manhattan | -0.461 | | pearson_euclidean | -0.4732 | | spearman_euclidean | -0.4673 | | pearson_dot | -0.4679 | | spearman_dot | -0.4577 | | pearson_max | -0.4628 | | spearman_max | -0.4577 | #### Semantic Similarity * Dataset: `sts-test` * Evaluated with [EmbeddingSimilarityEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator) | Metric | Value | |:--------------------|:-----------| | pearson_cosine | -0.2036 | | **spearman_cosine** | **-0.201** | | pearson_manhattan | -0.1686 | | spearman_manhattan | -0.1842 | | pearson_euclidean | -0.1795 | | spearman_euclidean | -0.1908 | | pearson_dot | -0.2159 | | spearman_dot | -0.2142 | | pearson_max | -0.1686 | | spearman_max | -0.1842 | ## Training Details ### Training Dataset #### afaji/indonli * Dataset: [afaji/indonli](https://huggingface.co/datasets/afaji/indonli) * Size: 10,000 training samples * Columns: premise, hypothesis, and label * Approximate statistics based on the first 1000 samples: | | premise | hypothesis | label | |:--------|:------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:-------------------------------------------------------------------| | type | string | string | int | | details | | | | * Samples: | premise | hypothesis | label | |:-----------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------|:---------------| | Presiden Joko Widodo (Jokowi) menyampaikan prediksi bahwa wabah virus Corona (COVID-19) di Indonesia akan selesai akhir tahun ini. | Prediksi akhir wabah tidak disampaikan Jokowi. | 2 | | Meski biasanya hanya digunakan di fasilitas kesehatan, saat ini masker dan sarung tangan sekali pakai banyak dipakai di tingkat rumah tangga. | Masker sekali pakai banyak dipakai di tingkat rumah tangga. | 0 | | Data dari Nielsen Music mencatat, "Joanne" telah terjual 201 ribu kopi di akhir minggu ini, seperti dilansir aceshowbiz.com. | Nielsen Music mencatat pada akhir minggu ini. | 1 | * Loss: [SoftmaxLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#softmaxloss) ### Evaluation Dataset #### afaji/indonli * Dataset: [afaji/indonli](https://huggingface.co/datasets/afaji/indonli) * Size: 1,000 evaluation samples * Columns: premise, hypothesis, and label * Approximate statistics based on the first 1000 samples: | | premise | hypothesis | label | |:--------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:-------------------------------------------------------------------| | type | string | string | int | | details | | | | * Samples: | premise | hypothesis | label | |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------|:---------------| | Manuskrip tersebut berisi tiga catatan yang menceritakan bagaimana peristiwa jatuhnya meteorit serta laporan kematian akibat kejadian tersebut seperti dilansir dari Science Alert, Sabtu (25/4/2020). | Manuskrip tersebut tidak mencatat laporan kematian. | 2 | | Dilansir dari Business Insider, menurut observasi dari Mauna Loa Observatory di Hawaii pada karbon dioksida (CO2) di level mencapai 410 ppm tidak langsung memberikan efek pada pernapasan, karena tubuh manusia juga masih membutuhkan CO2 dalam kadar tertentu. | Tidak ada observasi yang pernah dilansir oleh Business Insider. | 2 | | Perekonomian Jakarta terutama ditunjang oleh sektor perdagangan, jasa, properti, industri kreatif, dan keuangan. | Sektor jasa memberi pengaruh lebih besar daripada industri kreatif dalam perekonomian Jakarta. | 1 | * Loss: [SoftmaxLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#softmaxloss) ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: epoch - `learning_rate`: 2e-05 - `num_train_epochs`: 4 - `warmup_ratio`: 0.1 - `fp16`: True #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: epoch - `prediction_loss_only`: True - `per_device_train_batch_size`: 8 - `per_device_eval_batch_size`: 8 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 4 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: False - `fp16`: True - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: False - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: False - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | loss | sts-dev_spearman_cosine | sts-test_spearman_cosine | |:------:|:----:|:-------------:|:------:|:-----------------------:|:------------------------:| | 0 | 0 | - | - | -0.0893 | - | | 0.08 | 100 | 1.0851 | - | - | - | | 0.16 | 200 | 1.0163 | - | - | - | | 0.24 | 300 | 0.9524 | - | - | - | | 0.32 | 400 | 0.9257 | - | - | - | | 0.4 | 500 | 0.9397 | - | - | - | | 0.48 | 600 | 0.9125 | - | - | - | | 0.56 | 700 | 0.913 | - | - | - | | 0.64 | 800 | 0.8792 | - | - | - | | 0.72 | 900 | 0.932 | - | - | - | | 0.8 | 1000 | 0.9112 | - | - | - | | 0.88 | 1100 | 0.8809 | - | - | - | | 0.96 | 1200 | 0.8567 | - | - | - | | 1.0 | 1250 | - | 0.8585 | -0.4868 | - | | 1.04 | 1300 | 0.8482 | - | - | - | | 1.12 | 1400 | 0.7235 | - | - | - | | 1.2 | 1500 | 0.714 | - | - | - | | 1.28 | 1600 | 0.7053 | - | - | - | | 1.3600 | 1700 | 0.7205 | - | - | - | | 1.44 | 1800 | 0.7203 | - | - | - | | 1.52 | 1900 | 0.6957 | - | - | - | | 1.6 | 2000 | 0.7271 | - | - | - | | 1.6800 | 2100 | 0.7302 | - | - | - | | 1.76 | 2200 | 0.7054 | - | - | - | | 1.8400 | 2300 | 0.7134 | - | - | - | | 1.92 | 2400 | 0.6919 | - | - | - | | 2.0 | 2500 | 0.7416 | 0.8465 | -0.4085 | - | | 2.08 | 2600 | 0.4955 | - | - | - | | 2.16 | 2700 | 0.4484 | - | - | - | | 2.24 | 2800 | 0.4413 | - | - | - | | 2.32 | 2900 | 0.4567 | - | - | - | | 2.4 | 3000 | 0.4889 | - | - | - | | 2.48 | 3100 | 0.4284 | - | - | - | | 2.56 | 3200 | 0.5041 | - | - | - | | 2.64 | 3300 | 0.4755 | - | - | - | | 2.7200 | 3400 | 0.4726 | - | - | - | | 2.8 | 3500 | 0.4656 | - | - | - | | 2.88 | 3600 | 0.4389 | - | - | - | | 2.96 | 3700 | 0.4789 | - | - | - | | 3.0 | 3750 | - | 1.0011 | -0.4586 | - | | 3.04 | 3800 | 0.3492 | - | - | - | | 3.12 | 3900 | 0.2477 | - | - | - | | 3.2 | 4000 | 0.2556 | - | - | - | | 3.2800 | 4100 | 0.2531 | - | - | - | | 3.36 | 4200 | 0.2767 | - | - | - | | 3.44 | 4300 | 0.2665 | - | - | - | | 3.52 | 4400 | 0.2493 | - | - | - | | 3.6 | 4500 | 0.2757 | - | - | - | | 3.68 | 4600 | 0.2662 | - | - | - | | 3.76 | 4700 | 0.2666 | - | - | - | | 3.84 | 4800 | 0.2748 | - | - | - | | 3.92 | 4900 | 0.246 | - | - | - | | 4.0 | 5000 | 0.2411 | 1.2455 | -0.4665 | -0.2010 | ### Framework Versions - Python: 3.10.12 - Sentence Transformers: 3.0.1 - Transformers: 4.41.2 - PyTorch: 2.3.0+cu121 - Accelerate: 0.31.0 - Datasets: 2.20.0 - Tokenizers: 0.19.1 ## Citation ### BibTeX #### Sentence Transformers and SoftmaxLoss ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ```