Hugging Face's logo

fxmarty
/
20220713-h13m33s02_example_conll2003

Token Classification
TensorBoard
distilbert
Model card Files Metrics Community
fxmarty commited on
Commit
d4ce69e
•
1 Parent(s): 5adaaef

add experience

Browse files
Files changed (5) hide show
  1. README.md +59 -0
  2. runs.json +196 -0
  3. tensorboard/1657719189.6855562/events.out.tfevents.1657719189.ip-10-0-65-14.ec2.internal.1.1 +3 -0
  4. tensorboard/1657719189.6870131/events.out.tfevents.1657719189.ip-10-0-65-14.ec2.internal.1.2 +3 -0
  5. tensorboard/events.out.tfevents.1657719189.ip-10-0-65-14.ec2.internal.1.0 +3 -0
README.md ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: token-classification
3
+ datasets:
4
+ - conll2003
5
+ metrics:
6
+ - precision
7
+ - recall
8
+ - f1
9
+ - accuracy
10
+ tags:
11
+ - distilbert
12
+ ---
13
+
14
+ **task**: `token-classification`
15
+ **Backend:** `sagemaker-training`
16
+ **Backend args:** `{'instance_type': 'ml.g4dn.2xlarge', 'supported_instructions': None}`
17
+ **Number of evaluation samples:** `All dataset`
18
+
19
+ Fixed parameters:
20
+ * **model_name_or_path**: `elastic/distilbert-base-uncased-finetuned-conll03-english`
21
+ * **dataset**:
22
+ * **path**: `conll2003`
23
+ * **eval_split**: `validation`
24
+ * **data_keys**: `{'primary': 'tokens'}`
25
+ * **ref_keys**: `['ner_tags']`
26
+ * **calibration_split**: `train`
27
+ * **quantization_approach**: `static`
28
+ * **operators_to_quantize**: `['Add', 'MatMul']`
29
+ * **per_channel**: `False`
30
+ * **calibration**:
31
+ * **method**: `minmax`
32
+ * **num_calibration_samples**: `100`
33
+ * **framework**: `onnxruntime`
34
+ * **framework_args**:
35
+ * **opset**: `11`
36
+ * **optimization_level**: `1`
37
+ * **aware_training**: `False`
38
+
39
+ Benchmarked parameters:
40
+ * **node_exclusion**: `[]`, `['layernorm', 'gelu', 'residual', 'gather', 'softmax']`
41
+
42
+ # Evaluation
43
+ ## Non-time metrics
44
+ | node_exclusion | | precision (original) | precision (optimized) | | recall (original) | recall (optimized) | | f1 (original) | f1 (optimized) | | accuracy (original) | accuracy (optimized) |
45
+ | :------------------------------------------------------: | :-: | :------------------: | :-------------------: | :-: | :---------------: | :----------------: | :-: | :-----------: | :------------: | :-: | :-----------------: | :------------------: |
46
+ | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` | \| | 0.936 | 0.904 | \| | 0.944 | 0.921 | \| | 0.940 | 0.912 | \| | 0.988 | 0.984 |
47
+ | `[]` | \| | 0.936 | 0.065 | \| | 0.944 | 0.243 | \| | 0.940 | 0.103 | \| | 0.988 | 0.357 |
48
+
49
+ ## Time metrics
50
+ Time benchmarks were run for 15 seconds per config.
51
+
52
+
53
+ Below, time metrics for batch size = 4, input length = 64.
54
+
55
+ | node_exclusion | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) |
56
+ | :------------------------------------------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
57
+ | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` | \| | 103.46 | 53.77 | \| | 9.67 | 18.60 |
58
+ | `[]` | \| | 90.62 | 65.86 | \| | 11.07 | 15.20 |
59
+
runs.json ADDED
@@ -0,0 +1,196 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "model_name_or_path": "elastic/distilbert-base-uncased-finetuned-conll03-english",
4
+ "task": "token-classification",
5
+ "task_args": null,
6
+ "dataset": {
7
+ "path": "conll2003",
8
+ "eval_split": "validation",
9
+ "data_keys": {
10
+ "primary": "tokens",
11
+ "secondary": null
12
+ },
13
+ "ref_keys": [
14
+ "ner_tags"
15
+ ],
16
+ "name": null,
17
+ "calibration_split": "train"
18
+ },
19
+ "quantization_approach": "static",
20
+ "operators_to_quantize": [
21
+ "Add",
22
+ "MatMul"
23
+ ],
24
+ "node_exclusion": [
25
+ "layernorm",
26
+ "gelu",
27
+ "residual",
28
+ "gather",
29
+ "softmax"
30
+ ],
31
+ "aware_training": false,
32
+ "per_channel": false,
33
+ "calibration": {
34
+ "method": "minmax",
35
+ "num_calibration_samples": 100,
36
+ "calibration_histogram_percentile": null,
37
+ "calibration_moving_average": null,
38
+ "calibration_moving_average_constant": null
39
+ },
40
+ "framework": "onnxruntime",
41
+ "framework_args": {
42
+ "opset": 11,
43
+ "optimization_level": 1
44
+ },
45
+ "hardware": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nByte Order: Little Endian\nAddress sizes: 46 bits physical, 48 bits virtual\nCPU(s): 8\nOn-line CPU(s) list: 0-7\nThread(s) per core: 2\nCore(s) per socket: 4\nSocket(s): 1\nNUMA node(s): 1\nVendor ID: GenuineIntel\nCPU family: 6\nModel: 85\nModel name: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz\nStepping: 7\nCPU MHz: 3102.164\nBogoMIPS: 4999.99\nHypervisor vendor: KVM\nVirtualization type: full\nL1d cache: 128 KiB\nL1i cache: 128 KiB\nL2 cache: 4 MiB\nL3 cache: 35.8 MiB\nNUMA node0 CPU(s): 0-7\nVulnerability Itlb multihit: KVM: Vulnerable\nVulnerability L1tf: Mitigation; PTE Inversion\nVulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown: Mitigation; PTI\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke avx512_vnni\n",
46
+ "versions": {
47
+ "transformers": "4.20.1",
48
+ "optimum": "1.2.3.dev0",
49
+ "optimum_hash": "6911cc6576c9fd373dfdba597e7a6b3ec37c2063"
50
+ },
51
+ "evaluation": {
52
+ "time": [
53
+ {
54
+ "batch_size": 4,
55
+ "input_length": 64,
56
+ "baseline": {
57
+ "nb_forwards": 145,
58
+ "throughput": 9.67,
59
+ "latency_mean": 103.4629717724138,
60
+ "latency_std": 14.893062293273225,
61
+ "latency_50": 91.745478,
62
+ "latency_90": 119.3452324,
63
+ "latency_95": 119.4975372,
64
+ "latency_99": 119.96275956000001,
65
+ "latency_999": 121.099907936
66
+ },
67
+ "optimized": {
68
+ "nb_forwards": 279,
69
+ "throughput": 18.6,
70
+ "latency_mean": 53.769141777777776,
71
+ "latency_std": 0.23148430308679216,
72
+ "latency_50": 53.715052,
73
+ "latency_90": 54.059919,
74
+ "latency_95": 54.2278771,
75
+ "latency_99": 54.520317659999996,
76
+ "latency_999": 54.651304412
77
+ }
78
+ }
79
+ ],
80
+ "others": {
81
+ "baseline": {
82
+ "precision": 0.9358012339503085,
83
+ "recall": 0.9444631437226523,
84
+ "f1": 0.9401122372057961,
85
+ "accuracy": 0.9882013940267124
86
+ },
87
+ "optimized": {
88
+ "precision": 0.9038969616908851,
89
+ "recall": 0.9212386401884888,
90
+ "f1": 0.912485414235706,
91
+ "accuracy": 0.9842295860753086
92
+ }
93
+ }
94
+ },
95
+ "max_eval_samples": null,
96
+ "time_benchmark_args": {
97
+ "duration": 15,
98
+ "warmup_runs": 5
99
+ },
100
+ "model_type": "distilbert"
101
+ },
102
+ {
103
+ "model_name_or_path": "elastic/distilbert-base-uncased-finetuned-conll03-english",
104
+ "task": "token-classification",
105
+ "task_args": null,
106
+ "dataset": {
107
+ "path": "conll2003",
108
+ "eval_split": "validation",
109
+ "data_keys": {
110
+ "primary": "tokens",
111
+ "secondary": null
112
+ },
113
+ "ref_keys": [
114
+ "ner_tags"
115
+ ],
116
+ "name": null,
117
+ "calibration_split": "train"
118
+ },
119
+ "quantization_approach": "static",
120
+ "operators_to_quantize": [
121
+ "Add",
122
+ "MatMul"
123
+ ],
124
+ "node_exclusion": [],
125
+ "aware_training": false,
126
+ "per_channel": false,
127
+ "calibration": {
128
+ "method": "minmax",
129
+ "num_calibration_samples": 100,
130
+ "calibration_histogram_percentile": null,
131
+ "calibration_moving_average": null,
132
+ "calibration_moving_average_constant": null
133
+ },
134
+ "framework": "onnxruntime",
135
+ "framework_args": {
136
+ "opset": 11,
137
+ "optimization_level": 1
138
+ },
139
+ "hardware": "Architecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nByte Order: Little Endian\nAddress sizes: 46 bits physical, 48 bits virtual\nCPU(s): 8\nOn-line CPU(s) list: 0-7\nThread(s) per core: 2\nCore(s) per socket: 4\nSocket(s): 1\nNUMA node(s): 1\nVendor ID: GenuineIntel\nCPU family: 6\nModel: 85\nModel name: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz\nStepping: 7\nCPU MHz: 3161.331\nBogoMIPS: 4999.99\nHypervisor vendor: KVM\nVirtualization type: full\nL1d cache: 128 KiB\nL1i cache: 128 KiB\nL2 cache: 4 MiB\nL3 cache: 35.8 MiB\nNUMA node0 CPU(s): 0-7\nVulnerability Itlb multihit: KVM: Vulnerable\nVulnerability L1tf: Mitigation; PTE Inversion\nVulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Meltdown: Mitigation; PTI\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke avx512_vnni\n",
140
+ "versions": {
141
+ "transformers": "4.20.1",
142
+ "optimum": "1.2.3.dev0",
143
+ "optimum_hash": "6911cc6576c9fd373dfdba597e7a6b3ec37c2063"
144
+ },
145
+ "evaluation": {
146
+ "time": [
147
+ {
148
+ "batch_size": 4,
149
+ "input_length": 64,
150
+ "baseline": {
151
+ "nb_forwards": 166,
152
+ "throughput": 11.07,
153
+ "latency_mean": 90.62444262650602,
154
+ "latency_std": 1.8014393305543155,
155
+ "latency_50": 90.6704845,
156
+ "latency_90": 92.920863,
157
+ "latency_95": 93.45488975,
158
+ "latency_99": 94.330746,
159
+ "latency_999": 94.85841647
160
+ },
161
+ "optimized": {
162
+ "nb_forwards": 228,
163
+ "throughput": 15.2,
164
+ "latency_mean": 65.86275819736842,
165
+ "latency_std": 0.8604240016957982,
166
+ "latency_50": 65.8016715,
167
+ "latency_90": 66.9237761,
168
+ "latency_95": 67.35874704999999,
169
+ "latency_99": 68.89251761,
170
+ "latency_999": 69.285897298
171
+ }
172
+ }
173
+ ],
174
+ "others": {
175
+ "baseline": {
176
+ "precision": 0.9358012339503085,
177
+ "recall": 0.9444631437226523,
178
+ "f1": 0.9401122372057961,
179
+ "accuracy": 0.9882013940267124
180
+ },
181
+ "optimized": {
182
+ "precision": 0.06543578604398588,
183
+ "recall": 0.24335240659710536,
184
+ "f1": 0.10313837375178317,
185
+ "accuracy": 0.35697597445582335
186
+ }
187
+ }
188
+ },
189
+ "max_eval_samples": null,
190
+ "time_benchmark_args": {
191
+ "duration": 15,
192
+ "warmup_runs": 5
193
+ },
194
+ "model_type": "distilbert"
195
+ }
196
+ ]
tensorboard/1657719189.6855562/events.out.tfevents.1657719189.ip-10-0-65-14.ec2.internal.1.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1fe26dd11e87590c6e668ab0debb5e370e85a13fc8fae1f864006995336931dc
3
+ size 696
tensorboard/1657719189.6870131/events.out.tfevents.1657719189.ip-10-0-65-14.ec2.internal.1.2 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:31c7c2e1ed8100e4bdc29f963ca4e3065dd335d69c6facc3523b0e3e3b38a286
3
+ size 644
tensorboard/events.out.tfevents.1657719189.ip-10-0-65-14.ec2.internal.1.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8f2e2338abee092fcf1f6a3f3056f8d1c53f33f84377222db70aa717156a27ca
3
+ size 40