Text Classification
Transformers
PyTorch
English
deberta-v2
Inference Endpoints
theblackcat102 commited on
Commit
a88e417
1 Parent(s): c7ac1c6

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +237 -0
README.md CHANGED
@@ -1,3 +1,240 @@
1
  ---
2
  license: mit
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
+ datasets:
4
+ - openai/webgpt_comparisons
5
+ - openai/summarize_from_feedback
6
+ - Anthropic/hh-rlhf
7
+ language:
8
+ - en
9
  ---
10
+
11
+ # Reward model on deberta-v2-xxlarge (1.5B)
12
+
13
+ Reward model used in RLHF which is trained on webgpt, summarize from human feedback and Open Assistant user ranked dataset
14
+
15
+ # Model Details
16
+
17
+ ## Model Description
18
+
19
+ - **Developed by:** [More Information Needed]
20
+ - **Shared by [optional]:** [More Information Needed]
21
+ - **Model type:** [More Information Needed]
22
+ - **Language(s) (NLP):** [More Information Needed]
23
+ - **License:** [More Information Needed]
24
+ - **Finetuned from model [optional]:** [More Information Needed]
25
+
26
+ ## Model Sources [optional]
27
+
28
+ <!-- Provide the basic links for the model. -->
29
+
30
+ - **Repository:** [Open Assistant](https://github.com/LAION-AI/Open-Assistant)
31
+ - **Paper :** [Instruct GPT](https://cdn.openai.com/papers/Training_language_models_to_follow_instructions_with_human_feedback.pdf) : We try to replicate as close as we can on our hardware and existing datasets
32
+ - **Demo [optional]:** [More Information Needed]
33
+
34
+ # Uses
35
+
36
+ This model was trained with human feedback comparison examples, which penalize bad or rude sentence with lower scores.
37
+
38
+ ## Direct Use
39
+
40
+ ```
41
+ model_name = 'theblackcat102/deberta-v2-xxlarge-rm'
42
+ model = AutoModelForSequenceClassification.from_pretrained(model_name)
43
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
44
+ prompt = "I just got out of prison, any suggestion?"
45
+ good_helpful = "I am sorry to hear about it, it must be a hard time inside"
46
+ bad_text = "Stay away from me, you scumbag convict"
47
+ pos = tokenizer(prompt, good_helpful, return_tensors='pt')
48
+ neg = tokenizer(prompt, bad_text, return_tensors='pt')
49
+ pos_score = model(**pos).logits[0]
50
+ neg_score = model(**neg).logits[0]
51
+ print(pos_score, neg_score)
52
+ >> tensor([-1.3449], grad_fn=<SelectBackward0>) tensor([-2.0942], grad_fn=<SelectBackward0>)
53
+ ```
54
+
55
+
56
+
57
+ ## Downstream Use [optional]
58
+
59
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
60
+
61
+ [More Information Needed]
62
+
63
+ ## Out-of-Scope Use
64
+
65
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
66
+
67
+ [More Information Needed]
68
+
69
+ # Bias, Risks, and Limitations
70
+
71
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
72
+
73
+ [More Information Needed]
74
+
75
+ ## Recommendations
76
+
77
+ How to use it as a rank function
78
+
79
+ ```python
80
+ def divide_chunks(l, n):
81
+ # looping till length l
82
+ for i in range(0, len(l), n):
83
+ yield l[i:i + n]
84
+
85
+ @torch.no_grad()
86
+ def rank_model_fn(samples, **kwargs):
87
+ output_scores = []
88
+ for chunk_samples in divide_chunks(samples, 16):
89
+ is_empty = []
90
+ prefixes, postfixes = [], []
91
+ for sample in chunk_samples:
92
+ prefix, postfix = sample.split('[SEP]')
93
+ postfix = postfix.strip()
94
+ if len(postfix) == 0 or len(set(postfix)) <= 3:
95
+ is_empty.append(True)
96
+ else:
97
+ is_empty.append(False)
98
+ postfixes.append(postfix)
99
+ prefixes.append(prefix)
100
+ is_empty = np.array(is_empty)
101
+ inputs = rank_tokenizer(prefixes, postfixes, return_tensors="pt", padding=True)
102
+ inputs.pop("token_type_ids", None)
103
+ inputs = { key: tensor.cuda() for key, tensor in inputs.items() }
104
+ scores = rank_model(**inputs).logits[:, 0].detach().cpu()
105
+ scores[is_empty] = -4
106
+ output_scores += [ s for s in scores ]
107
+ return torch.from_numpy(np.array(output_scores))
108
+ ```
109
+
110
+ ## How to Get Started with the Model
111
+
112
+ Use the code below to get started with the model.
113
+
114
+ [More Information Needed]
115
+
116
+ # Training Details
117
+
118
+
119
+ ## Training Procedure
120
+
121
+ checkout our training repo [here](https://github.com/LAION-AI/Open-Assistant/tree/main/model/reward/instructor)
122
+
123
+
124
+ ### Preprocessing [optional]
125
+
126
+ [More Information Needed]
127
+
128
+
129
+ ### Training Hyperparameters
130
+
131
+ ```yaml
132
+ model_name: microsoft/deberta-v2-xxlarge
133
+ learning_rate: 2e-6
134
+ scheduler: cosine
135
+ gradient_checkpointing: false
136
+ gradient_accumulation_steps: 12
137
+ per_device_train_batch_size: 1
138
+ per_device_eval_batch_size: 4
139
+ warmup_steps: 600
140
+ eval_steps: 1000000
141
+ save_steps: 1000
142
+ max_length: 512
143
+ num_train_epochs: 2
144
+ datasets:
145
+ - webgpt
146
+ - hfsummary
147
+ - anthropic_rlhf
148
+ - oa_private
149
+ ```
150
+
151
+ ### Speeds, Sizes, Times [optional]
152
+
153
+ Trained on 8 A100 80G model, since we are using the same batch strategy as InstructGPT, using a batch_size of 1 actually equals to (N-1) batch where N refers to number of negative examples. Which is why I recommend using the largest VRAM GPU you can find to train this model.
154
+
155
+ # Evaluation
156
+
157
+ <!-- This section describes the evaluation protocols and provides the results. -->
158
+
159
+ ## Testing Data, Factors & Metrics
160
+
161
+ ### Testing Data
162
+
163
+ <!-- This should link to a Data Card if possible. -->
164
+
165
+ [More Information Needed]
166
+
167
+ ### Factors
168
+
169
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
170
+
171
+ [More Information Needed]
172
+
173
+ ### Metrics
174
+
175
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
176
+
177
+ [More Information Needed]
178
+
179
+ ## Results
180
+
181
+ [More Information Needed]
182
+
183
+ ### Summary
184
+
185
+
186
+
187
+ # Model Examination [optional]
188
+
189
+ <!-- Relevant interpretability work for the model goes here -->
190
+
191
+ [More Information Needed]
192
+
193
+
194
+ # Technical Specifications [optional]
195
+
196
+ ## Model Architecture and Objective
197
+
198
+ [More Information Needed]
199
+
200
+ ## Compute Infrastructure
201
+
202
+ [More Information Needed]
203
+
204
+ ### Hardware
205
+
206
+ [More Information Needed]
207
+
208
+ ### Software
209
+
210
+ [More Information Needed]
211
+
212
+ # Citation [optional]
213
+
214
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
215
+
216
+ **BibTeX:**
217
+
218
+ [More Information Needed]
219
+
220
+ **APA:**
221
+
222
+ [More Information Needed]
223
+
224
+ # Glossary [optional]
225
+
226
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
227
+
228
+ [More Information Needed]
229
+
230
+ # More Information [optional]
231
+
232
+ [More Information Needed]
233
+
234
+ # Model Card Authors [optional]
235
+
236
+ [More Information Needed]
237
+
238
+ # Model Card Contact
239
+
240
+ [More Information Needed]