BartekSadlej commited on
Commit
239ee0c
1 Parent(s): 0bcdf6d

Upload model

Browse files
Files changed (5) hide show
  1. README.md +199 -0
  2. config.json +56 -0
  3. config.py +31 -0
  4. model.py +264 -0
  5. model.safetensors +3 -0
README.md ADDED
@@ -0,0 +1,199 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ tags: []
4
+ ---
5
+
6
+ # Model Card for Model ID
7
+
8
+ <!-- Provide a quick summary of what the model is/does. -->
9
+
10
+
11
+
12
+ ## Model Details
13
+
14
+ ### Model Description
15
+
16
+ <!-- Provide a longer summary of what this model is. -->
17
+
18
+ This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
19
+
20
+ - **Developed by:** [More Information Needed]
21
+ - **Funded by [optional]:** [More Information Needed]
22
+ - **Shared by [optional]:** [More Information Needed]
23
+ - **Model type:** [More Information Needed]
24
+ - **Language(s) (NLP):** [More Information Needed]
25
+ - **License:** [More Information Needed]
26
+ - **Finetuned from model [optional]:** [More Information Needed]
27
+
28
+ ### Model Sources [optional]
29
+
30
+ <!-- Provide the basic links for the model. -->
31
+
32
+ - **Repository:** [More Information Needed]
33
+ - **Paper [optional]:** [More Information Needed]
34
+ - **Demo [optional]:** [More Information Needed]
35
+
36
+ ## Uses
37
+
38
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
39
+
40
+ ### Direct Use
41
+
42
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
43
+
44
+ [More Information Needed]
45
+
46
+ ### Downstream Use [optional]
47
+
48
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
49
+
50
+ [More Information Needed]
51
+
52
+ ### Out-of-Scope Use
53
+
54
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
55
+
56
+ [More Information Needed]
57
+
58
+ ## Bias, Risks, and Limitations
59
+
60
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
61
+
62
+ [More Information Needed]
63
+
64
+ ### Recommendations
65
+
66
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
67
+
68
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
69
+
70
+ ## How to Get Started with the Model
71
+
72
+ Use the code below to get started with the model.
73
+
74
+ [More Information Needed]
75
+
76
+ ## Training Details
77
+
78
+ ### Training Data
79
+
80
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
81
+
82
+ [More Information Needed]
83
+
84
+ ### Training Procedure
85
+
86
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
87
+
88
+ #### Preprocessing [optional]
89
+
90
+ [More Information Needed]
91
+
92
+
93
+ #### Training Hyperparameters
94
+
95
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
96
+
97
+ #### Speeds, Sizes, Times [optional]
98
+
99
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
100
+
101
+ [More Information Needed]
102
+
103
+ ## Evaluation
104
+
105
+ <!-- This section describes the evaluation protocols and provides the results. -->
106
+
107
+ ### Testing Data, Factors & Metrics
108
+
109
+ #### Testing Data
110
+
111
+ <!-- This should link to a Dataset Card if possible. -->
112
+
113
+ [More Information Needed]
114
+
115
+ #### Factors
116
+
117
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
118
+
119
+ [More Information Needed]
120
+
121
+ #### Metrics
122
+
123
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
124
+
125
+ [More Information Needed]
126
+
127
+ ### Results
128
+
129
+ [More Information Needed]
130
+
131
+ #### Summary
132
+
133
+
134
+
135
+ ## Model Examination [optional]
136
+
137
+ <!-- Relevant interpretability work for the model goes here -->
138
+
139
+ [More Information Needed]
140
+
141
+ ## Environmental Impact
142
+
143
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
144
+
145
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
146
+
147
+ - **Hardware Type:** [More Information Needed]
148
+ - **Hours used:** [More Information Needed]
149
+ - **Cloud Provider:** [More Information Needed]
150
+ - **Compute Region:** [More Information Needed]
151
+ - **Carbon Emitted:** [More Information Needed]
152
+
153
+ ## Technical Specifications [optional]
154
+
155
+ ### Model Architecture and Objective
156
+
157
+ [More Information Needed]
158
+
159
+ ### Compute Infrastructure
160
+
161
+ [More Information Needed]
162
+
163
+ #### Hardware
164
+
165
+ [More Information Needed]
166
+
167
+ #### Software
168
+
169
+ [More Information Needed]
170
+
171
+ ## Citation [optional]
172
+
173
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
174
+
175
+ **BibTeX:**
176
+
177
+ [More Information Needed]
178
+
179
+ **APA:**
180
+
181
+ [More Information Needed]
182
+
183
+ ## Glossary [optional]
184
+
185
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
186
+
187
+ [More Information Needed]
188
+
189
+ ## More Information [optional]
190
+
191
+ [More Information Needed]
192
+
193
+ ## Model Card Authors [optional]
194
+
195
+ [More Information Needed]
196
+
197
+ ## Model Card Contact
198
+
199
+ [More Information Needed]
config.json ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "ILKTModel"
4
+ ],
5
+ "auto_map": {
6
+ "AutoConfig": "config.ILKTConfig",
7
+ "AutoModel": "model.ILKTModel"
8
+ },
9
+ "backbone_config": {
10
+ "pretrained_model_name_or_path": "google-bert/bert-base-multilingual-cased",
11
+ "trust_remote_code": true
12
+ },
13
+ "cls_head_config": {
14
+ "dropout": 0.0,
15
+ "n_dense": 0,
16
+ "pool_type": "cls",
17
+ "use_batch_norm": true,
18
+ "use_layer_norm": false
19
+ },
20
+ "cls_heads": [
21
+ [
22
+ 3,
23
+ "allegro--klej-cdsc-e"
24
+ ],
25
+ [
26
+ 2,
27
+ "allegro--klej-psc"
28
+ ],
29
+ [
30
+ 2,
31
+ "allegro--klej-dyk"
32
+ ],
33
+ [
34
+ 5,
35
+ "PL-MTEB--scifield"
36
+ ]
37
+ ],
38
+ "embedding_head_config": {
39
+ "dropout": 0.0,
40
+ "n_dense": 1,
41
+ "normalize_embeddings": false,
42
+ "pool_type": "cls",
43
+ "use_batch_norm": false,
44
+ "use_layer_norm": false
45
+ },
46
+ "hidden_size": 768,
47
+ "mlm_head_config": {
48
+ "dropout": 0.0,
49
+ "n_dense": 0,
50
+ "use_batch_norm": true,
51
+ "use_layer_norm": false
52
+ },
53
+ "model_type": "ILKT",
54
+ "torch_dtype": "float32",
55
+ "transformers_version": "4.41.1"
56
+ }
config.py ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from typing import Any, Dict, List, Tuple
2
+
3
+ from transformers import PretrainedConfig
4
+
5
+
6
+ class ILKTConfig(PretrainedConfig):
7
+
8
+ model_type = "ILKT"
9
+
10
+ def __init__(
11
+ self,
12
+ backbone_config: Dict[str, Any] = {},
13
+ embedding_head_config: Dict[str, Any] = {},
14
+ mlm_head_config: Dict[str, Any] = {},
15
+ cls_head_config: Dict[str, Any] = {},
16
+ cls_heads: List[Tuple[int, str]] = [],
17
+ max_length: int = 512,
18
+ **kwargs
19
+ ):
20
+ self.backbone_config = backbone_config
21
+ self.embedding_head_config = embedding_head_config
22
+ self.mlm_head_config = mlm_head_config
23
+ self.cls_head_config = cls_head_config
24
+ self.cls_heads = cls_heads
25
+ self.max_length = False
26
+ self.output_hidden_states = False
27
+
28
+ # TODO:
29
+ # make config a proper HF config, save max length ets, don't know how it works exactly in hf ecosystem
30
+
31
+ super().__init__(**kwargs)
model.py ADDED
@@ -0,0 +1,264 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from typing import Any, Dict, Optional
2
+
3
+ import torch
4
+ import torch.nn as nn
5
+ from transformers import AutoConfig, AutoModel, PreTrainedModel
6
+ from transformers.modeling_outputs import (
7
+ BaseModelOutputWithPooling,
8
+ MaskedLMOutput,
9
+ BaseModelOutput,
10
+ SequenceClassifierOutput,
11
+ )
12
+ from enum import Enum
13
+
14
+ from .config import ILKTConfig
15
+
16
+ def cls_pooling(last_hidden_state, attention_mask):
17
+ return last_hidden_state[:, 0, :]
18
+
19
+
20
+ def create_head_blocks(
21
+ hidden_size: int,
22
+ n_dense: int,
23
+ use_batch_norm: bool,
24
+ use_layer_norm: bool,
25
+ dropout: float,
26
+ **kwargs,
27
+ ) -> nn.Module:
28
+ blocks = []
29
+ for _ in range(n_dense):
30
+ blocks.append(nn.Linear(hidden_size, hidden_size))
31
+ if use_batch_norm:
32
+ blocks.append(nn.BatchNorm1d(hidden_size))
33
+ elif use_layer_norm:
34
+ blocks.append(nn.LayerNorm(hidden_size))
35
+ blocks.append(nn.ReLU())
36
+ if dropout > 0:
37
+ blocks.append(nn.Dropout(dropout))
38
+ return nn.Sequential(*blocks)
39
+
40
+
41
+ class SentenceEmbeddingHead(nn.Module):
42
+ def __init__(
43
+ self, backbone_hidden_size: int, embedding_head_config: Dict[str, Any]
44
+ ):
45
+ super().__init__()
46
+ self.config = embedding_head_config
47
+
48
+ self.head = nn.Sequential(
49
+ *[
50
+ create_head_blocks(backbone_hidden_size, **embedding_head_config),
51
+ ]
52
+ )
53
+
54
+ def forward(
55
+ self, backbone_output: BaseModelOutput, attention_mask: torch.Tensor, **kwargs
56
+ ) -> BaseModelOutputWithPooling:
57
+ if self.config["pool_type"] == "cls":
58
+ embeddings = cls_pooling(backbone_output.last_hidden_state, attention_mask)
59
+ else:
60
+ raise NotImplementedError(
61
+ f"Pooling type {self.config['pool_type']} not implemented"
62
+ )
63
+ if self.config["normalize_embeddings"]:
64
+ embeddings = nn.functional.normalize(embeddings, p=2, dim=-1)
65
+ return BaseModelOutputWithPooling(
66
+ last_hidden_state=backbone_output.last_hidden_state,
67
+ pooler_output=embeddings, # type: ignore
68
+ )
69
+
70
+
71
+ class MLMHead(nn.Module):
72
+ def __init__(
73
+ self,
74
+ backbone_hidden_size: int,
75
+ vocab_size: int,
76
+ mlm_head_config: Dict[str, Any],
77
+ ):
78
+ super().__init__()
79
+ self.config = mlm_head_config
80
+
81
+ self.head = nn.Sequential(
82
+ *[
83
+ create_head_blocks(backbone_hidden_size, **mlm_head_config),
84
+ nn.Linear(backbone_hidden_size, vocab_size),
85
+ ]
86
+ )
87
+
88
+ def forward(
89
+ self,
90
+ backbone_output: BaseModelOutput,
91
+ attention_mask: torch.Tensor,
92
+ labels: Optional[torch.Tensor] = None,
93
+ **kwargs,
94
+ ) -> MaskedLMOutput:
95
+ prediction_scores = self.head(backbone_output.last_hidden_state)
96
+
97
+ loss = None
98
+ if labels is not None:
99
+ loss_fct = nn.CrossEntropyLoss()
100
+ loss = loss_fct(
101
+ prediction_scores.view(-1, prediction_scores.size(-1)),
102
+ labels.view(-1),
103
+ )
104
+ return MaskedLMOutput(loss=loss, logits=prediction_scores)
105
+
106
+
107
+ class CLSHead(nn.Module):
108
+ def __init__(
109
+ self,
110
+ backbone_hidden_size: int,
111
+ n_classes: int,
112
+ cls_head_config: Dict[str, Any],
113
+ ):
114
+ super().__init__()
115
+ self.config = cls_head_config
116
+
117
+ self.head = nn.Sequential(
118
+ *[
119
+ create_head_blocks(backbone_hidden_size, **cls_head_config),
120
+ nn.Linear(backbone_hidden_size, n_classes),
121
+ ]
122
+ )
123
+
124
+ def forward(
125
+ self,
126
+ backbone_output: BaseModelOutput,
127
+ attention_mask: torch.Tensor,
128
+ labels: Optional[torch.Tensor] = None,
129
+ **kwargs,
130
+ ) -> SequenceClassifierOutput:
131
+ if self.config["pool_type"] == "cls":
132
+ embeddings = cls_pooling(backbone_output.last_hidden_state, attention_mask)
133
+ else:
134
+ raise NotImplementedError(
135
+ f"Pooling type {self.config['pool_type']} not implemented"
136
+ )
137
+
138
+ prediction_scores = self.head(embeddings)
139
+
140
+ loss = None
141
+ if labels is not None:
142
+ loss_fct = nn.CrossEntropyLoss()
143
+ loss = loss_fct(
144
+ prediction_scores.view(-1, prediction_scores.size(-1)),
145
+ labels.view(-1),
146
+ )
147
+ return SequenceClassifierOutput(loss=loss, logits=prediction_scores)
148
+
149
+
150
+ class ForwardRouting(Enum):
151
+ GET_SENTENCE_EMBEDDING = "get_sentence_embedding"
152
+ GET_MLM_OUTPUT = "get_mlm_output"
153
+ GET_CLS_OUTPUT = "get_cls_output"
154
+
155
+
156
+ class ILKTModel(PreTrainedModel):
157
+ config_class = ILKTConfig
158
+
159
+ def __init__(self, config: ILKTConfig):
160
+ super().__init__(config)
161
+
162
+ backbone_config = AutoConfig.from_pretrained(**config.backbone_config)
163
+ pretrained_model_name_or_path = config.backbone_config[
164
+ "pretrained_model_name_or_path"
165
+ ]
166
+ self.backbone = AutoModel.from_pretrained(
167
+ pretrained_model_name_or_path, config=backbone_config
168
+ )
169
+
170
+ backbone_hidden_size = backbone_config.hidden_size
171
+ self.config.hidden_size = backbone_hidden_size
172
+ backbone_vocab_size = backbone_config.vocab_size
173
+ self.embedding_head = SentenceEmbeddingHead(
174
+ backbone_hidden_size, config.embedding_head_config
175
+ )
176
+ self.mlm_head = MLMHead(
177
+ backbone_hidden_size, backbone_vocab_size, config.mlm_head_config
178
+ )
179
+
180
+ self.cls_heads = nn.ModuleDict(
181
+ dict(
182
+ [
183
+ (
184
+ name,
185
+ CLSHead(
186
+ backbone_hidden_size, n_classes, config.cls_head_config
187
+ ),
188
+ )
189
+ for n_classes, name in config.cls_heads
190
+ ]
191
+ )
192
+ )
193
+
194
+ def forward(
195
+ self,
196
+ input_ids: torch.Tensor,
197
+ attention_mask: torch.Tensor,
198
+ token_type_ids: Optional[torch.Tensor] = None,
199
+ forward_routing: ForwardRouting = ForwardRouting.GET_SENTENCE_EMBEDDING,
200
+ **kwargs,
201
+ ):
202
+ if forward_routing == ForwardRouting.GET_SENTENCE_EMBEDDING:
203
+ return self.get_sentence_embedding(
204
+ input_ids, attention_mask, token_type_ids=token_type_ids
205
+ )
206
+ elif forward_routing == ForwardRouting.GET_MLM_OUTPUT:
207
+ return self.get_mlm_output(
208
+ input_ids, attention_mask, token_type_ids=token_type_ids, **kwargs
209
+ )
210
+ elif forward_routing == ForwardRouting.GET_CLS_OUTPUT:
211
+ return self.get_cls_output(
212
+ input_ids, attention_mask, token_type_ids=token_type_ids, **kwargs
213
+ )
214
+ else:
215
+ raise ValueError(f"Unknown forward routing {forward_routing}")
216
+
217
+ def get_sentence_embedding(
218
+ self, input_ids: torch.Tensor, attention_mask: torch.Tensor, **kwargs
219
+ ):
220
+ backbone_output: BaseModelOutput = self.backbone(
221
+ input_ids=input_ids, attention_mask=attention_mask, **kwargs
222
+ )
223
+
224
+ embedding_output = self.embedding_head(
225
+ backbone_output, attention_mask, **kwargs
226
+ )
227
+
228
+ return embedding_output
229
+
230
+ def get_mlm_output(
231
+ self,
232
+ input_ids: torch.Tensor,
233
+ attention_mask: torch.Tensor,
234
+ labels: Optional[torch.Tensor] = None,
235
+ **kwargs,
236
+ ):
237
+ backbone_output: BaseModelOutput = self.backbone(
238
+ input_ids=input_ids, attention_mask=attention_mask, **kwargs
239
+ )
240
+
241
+ mlm_output = self.mlm_head(backbone_output, attention_mask, labels, **kwargs)
242
+
243
+ return mlm_output
244
+
245
+ def get_cls_output(
246
+ self,
247
+ input_ids: torch.Tensor,
248
+ attention_mask: torch.Tensor,
249
+ head_name: str,
250
+ labels: Optional[torch.Tensor] = None,
251
+ **kwargs,
252
+ ):
253
+ backbone_output: BaseModelOutput = self.backbone(
254
+ input_ids=input_ids, attention_mask=attention_mask, **kwargs
255
+ )
256
+
257
+ if head_name not in self.cls_heads:
258
+ raise ValueError(f"Head {head_name} not found in model")
259
+
260
+ cls_output = self.cls_heads[head_name](
261
+ backbone_output, attention_mask, labels, **kwargs
262
+ )
263
+
264
+ return cls_output
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1ce902999fe6a38211fd58ffa4485837507a00c973df723c300ec023e2b15857
3
+ size 1081565060