Hugging Face's logo

lukasdrg
/
clinical_longformer_same_tokens_exp_gpu_vtres

Fill-Mask
Transformers
Safetensors
longformer
Generated from Trainer
Inference Endpoints
Model card Files Community
lukasdrg commited on
Commit
bc82c36
•
1 Parent(s): 522e5e1

End of training

Browse files
Files changed (2) hide show
  1. README.md +94 -0
  2. model.safetensors +1 -1
README.md ADDED
@@ -0,0 +1,94 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: allenai/longformer-base-4096
4
+ tags:
5
+ - generated_from_trainer
6
+ model-index:
7
+ - name: clinical_longformer_same_tokens_exp_gpu_vtres
8
+ results: []
9
+ ---
10
+
11
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
12
+ should probably proofread and complete it, then remove this comment. -->
13
+
14
+ # clinical_longformer_same_tokens_exp_gpu_vtres
15
+
16
+ This model is a fine-tuned version of [allenai/longformer-base-4096](https://huggingface.co/allenai/longformer-base-4096) on the None dataset.
17
+ It achieves the following results on the evaluation set:
18
+ - Loss: 0.9360
19
+
20
+ ## Model description
21
+
22
+ More information needed
23
+
24
+ ## Intended uses & limitations
25
+
26
+ More information needed
27
+
28
+ ## Training and evaluation data
29
+
30
+ More information needed
31
+
32
+ ## Training procedure
33
+
34
+ ### Training hyperparameters
35
+
36
+ The following hyperparameters were used during training:
37
+ - learning_rate: 2e-05
38
+ - train_batch_size: 4
39
+ - eval_batch_size: 32
40
+ - seed: 42
41
+ - gradient_accumulation_steps: 128
42
+ - total_train_batch_size: 512
43
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
44
+ - lr_scheduler_type: linear
45
+ - lr_scheduler_warmup_steps: 1500
46
+ - num_epochs: 10
47
+
48
+ ### Training results
49
+
50
+ | Training Loss | Epoch | Step | Validation Loss |
51
+ |:-------------:|:-----:|:----:|:---------------:|
52
+ | 3.4475 | 0.28 | 129 | 2.4612 |
53
+ | 3.2401 | 0.57 | 258 | 2.2069 |
54
+ | 3.0136 | 0.85 | 387 | 2.0084 |
55
+ | 2.5518 | 1.13 | 516 | 1.8632 |
56
+ | 2.9398 | 1.42 | 645 | 1.7412 |
57
+ | 2.0755 | 1.7 | 774 | 1.6260 |
58
+ | 2.231 | 1.98 | 903 | 1.5247 |
59
+ | 1.9556 | 2.27 | 1032 | 1.4482 |
60
+ | 2.0415 | 2.55 | 1161 | 1.3771 |
61
+ | 2.111 | 2.83 | 1290 | 1.3097 |
62
+ | 1.6116 | 3.12 | 1419 | 1.2572 |
63
+ | 1.6044 | 3.4 | 1548 | 1.2108 |
64
+ | 1.6614 | 3.68 | 1677 | 1.1802 |
65
+ | 1.5013 | 3.97 | 1806 | 1.1371 |
66
+ | 1.4666 | 4.25 | 1935 | 1.1079 |
67
+ | 1.4608 | 4.53 | 2064 | 1.0894 |
68
+ | 1.5144 | 4.82 | 2193 | 1.0672 |
69
+ | 1.3958 | 5.1 | 2322 | 1.0390 |
70
+ | 1.3949 | 5.38 | 2451 | 1.0283 |
71
+ | 1.539 | 5.67 | 2580 | 1.0249 |
72
+ | 1.5819 | 5.95 | 2709 | 1.0085 |
73
+ | 1.2588 | 6.23 | 2838 | 1.0033 |
74
+ | 1.2704 | 6.52 | 2967 | 0.9875 |
75
+ | 1.2457 | 6.8 | 3096 | 0.9768 |
76
+ | 1.2066 | 7.08 | 3225 | 0.9707 |
77
+ | 1.226 | 7.37 | 3354 | 0.9657 |
78
+ | 1.3292 | 7.65 | 3483 | 0.9590 |
79
+ | 1.1629 | 7.93 | 3612 | 0.9468 |
80
+ | 1.2711 | 8.22 | 3741 | 0.9437 |
81
+ | 1.3092 | 8.5 | 3870 | 0.9445 |
82
+ | 1.35 | 8.78 | 3999 | 0.9424 |
83
+ | 1.3402 | 9.07 | 4128 | 0.9360 |
84
+ | 1.1433 | 9.35 | 4257 | 0.9349 |
85
+ | 1.3059 | 9.63 | 4386 | 0.9296 |
86
+ | 1.1391 | 9.92 | 4515 | 0.9360 |
87
+
88
+
89
+ ### Framework versions
90
+
91
+ - Transformers 4.35.2
92
+ - Pytorch 2.1.1+cu121
93
+ - Datasets 2.15.0
94
+ - Tokenizers 0.15.0
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:751a213d6f9a49e1f03c7d1a138359fae9906f39d0eb18440d56707f98986b14
3
  size 594879308
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0c00d20716115a43bae66f30c497dd3b33b6cd8033f84f394f7bcb29023a31d9
3
  size 594879308