First model version for roscoe-512-roberta-base

README.md

@@ -1,3 +1,32 @@
 ---
 license: cc-by-nc-4.0
 ---
+## roscoe-512-roberta-base
+
+Sentence embedding model for reasoning steps.
+
+To obtain reasoning step embeddings, we finetune SimCSE (Gao et al., 2021), a
+supervised sentence similarity model extending the RoBERTa word embedding model (Liu et al., 2019) on
+multi-step reasoning datasets we listed in §5 (see details in Golovneva et al., 2022). SimCSE is a contrastive learning model
+that is trained on triplets of reference reasoning steps, positive and hard-negative hypothesis reasoning steps
+to minimize the cross-entropy objective with in-batch negatives. For contrastive learning, we use the context
+and reference reasoning steps as a positive sample, and context and perturbed reference steps as
+hard-negative pairs. With finetuned model we embed each individual step, as well as a reasoning chain as a
+whole. We use the pretrained checkpoint of supervised SimCSE model sup-simcse-roberta-base to initialize
+our model, and further train it for five epochs on our synthetic train data.
+
+To train the model, we construct dataset by generating perturbations — i.e.,
+deterministic modifications — on half of the reference reasoning steps in the following sets: Entailment-Bank
+(deductive reasoning), ProofWriter (logical reasoning); three arithmetic reasoning datasets MATH, ASDIV and AQUA; EQASC 
+(explanations for commonsense question answering), and StrategyQA (question answering with implicit reasoning strategies).
+
+References:
+
+1. Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings.
+arXiv preprint arXiv:2104.08821, 2021.
+2. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis,
+Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv
+preprint arXiv:1907.11692, 2019.
+3. Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 
+ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning. arXiv preprint, 2022.
+4. 

config.json

@@ -0,0 +1,26 @@
+{
+  "_name_or_path": "result/cont-pos-pert-neg-simcse-roberta-base-cosine-0830",
+  "architectures": [
+    "RobertaForCL"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.2.1",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 50265
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49e43d4bc587a735991673aab6935269a25dbf09892bafa6f66eca87de4c7a5d
+size 498672439

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "sep_token": "</s>", "pad_token": "<pad>", "cls_token": "<s>", "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": false}}

tokenizer_config.json

@@ -0,0 +1 @@
+{"unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "bos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "add_prefix_space": false, "errors": "replace", "sep_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "cls_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "model_max_length": 512, "name_or_path": "princeton-nlp/sup-simcse-roberta-base", "special_tokens_map_file": "/private/home/olggol/.cache/huggingface/transformers/90ffa7c13d92d368876a3cde38912cf1fbe882d3b2ad0fc6b1ab5d11fa3f7753.a11ebb04664c067c8fe5ef8f8068b0f721263414a26058692f7b2e4ba2a1b342"}