Hugging Face's logo

sanchit-gandhi
/
parler-tts-mini-v0.1-expresso-concatenated-combined

Text2Text Generation
Transformers
Safetensors
parler_tts
Inference Endpoints
Model card Files Community
parler-tts-mini-v0.1-expresso-concatenated-combined
File size: 63,850 Bytes 
8ed4a53
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
 
2024-05-13 20:46:52,823 INFO    StreamThr :1244775 [internal.py:wandb_internal():85] W&B internal server running at pid: 1244775, started at: 2024-05-13 20:46:52.823043
2024-05-13 20:46:52,825 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status
2024-05-13 20:46:52,826 INFO    WriterThread:1244775 [datastore.py:open_for_write():87] open: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/run-m0g0ap7d.wandb
2024-05-13 20:46:52,828 DEBUG   SenderThread:1244775 [sender.py:send():378] send: header
2024-05-13 20:46:52,829 DEBUG   SenderThread:1244775 [sender.py:send():378] send: run
2024-05-13 20:46:53,223 INFO    SenderThread:1244775 [dir_watcher.py:__init__():211] watching files in: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files
2024-05-13 20:46:53,223 INFO    SenderThread:1244775 [sender.py:_start_run_threads():1123] run started: m0g0ap7d with start time 1715626012.822173
2024-05-13 20:46:53,229 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: check_version
2024-05-13 20:46:53,229 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: check_version
2024-05-13 20:46:53,293 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: run_start
2024-05-13 20:46:53,323 DEBUG   HandlerThread:1244775 [system_info.py:__init__():26] System info init
2024-05-13 20:46:53,323 DEBUG   HandlerThread:1244775 [system_info.py:__init__():41] System info init done
2024-05-13 20:46:53,323 INFO    HandlerThread:1244775 [system_monitor.py:start():194] Starting system monitor
2024-05-13 20:46:53,323 INFO    SystemMonitor:1244775 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-05-13 20:46:53,324 INFO    HandlerThread:1244775 [system_monitor.py:probe():214] Collecting system info
2024-05-13 20:46:53,324 INFO    SystemMonitor:1244775 [interfaces.py:start():188] Started cpu monitoring
2024-05-13 20:46:53,325 INFO    SystemMonitor:1244775 [interfaces.py:start():188] Started disk monitoring
2024-05-13 20:46:53,325 INFO    SystemMonitor:1244775 [interfaces.py:start():188] Started gpu monitoring
2024-05-13 20:46:53,327 INFO    SystemMonitor:1244775 [interfaces.py:start():188] Started memory monitoring
2024-05-13 20:46:53,329 INFO    SystemMonitor:1244775 [interfaces.py:start():188] Started network monitoring
2024-05-13 20:46:53,365 DEBUG   HandlerThread:1244775 [system_info.py:probe():150] Probing system
2024-05-13 20:46:53,366 DEBUG   HandlerThread:1244775 [system_info.py:_probe_git():135] Probing git
2024-05-13 20:46:53,372 DEBUG   HandlerThread:1244775 [system_info.py:_probe_git():143] Probing git done
2024-05-13 20:46:53,372 DEBUG   HandlerThread:1244775 [system_info.py:probe():198] Probing system done
2024-05-13 20:46:53,372 DEBUG   HandlerThread:1244775 [system_monitor.py:probe():223] {'os': 'Linux-5.4.0-166-generic-x86_64-with-glibc2.31', 'python': '3.11.9', 'heartbeatAt': '2024-05-13T18:46:53.365083', 'startedAt': '2024-05-13T18:46:52.816759', 'docker': None, 'cuda': None, 'args': ('finetuning_concatenated_config.json',), 'state': 'running', 'program': '/raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/run_parler_tts_training.py', 'codePathLocal': 'run_parler_tts_training.py', 'codePath': 'run_parler_tts_training.py', 'git': {'remote': 'https://huggingface.co/sanchit-gandhi/parler-tts-mini-v0.1-expresso-concatenated-combined', 'commit': '50ba4323d7b8bb052629aa1b88283b9df081a821'}, 'email': '[email protected]', 'root': '/raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined', 'host': 'hf-dgx-01', 'username': 'sanchit', 'executable': '/home/sanchit/miniconda3/envs/venv/bin/python', 'cpu_count': 64, 'cpu_count_logical': 128, 'cpu_freq': {'current': 2257.736234375, 'min': 1500.0, 'max': 2250.0}, 'cpu_freq_per_core': [{'current': 1795.281, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.292, 'min': 1500.0, 'max': 2250.0}, {'current': 1795.78, 'min': 1500.0, 'max': 2250.0}, {'current': 1792.55, 'min': 1500.0, 'max': 2250.0}, {'current': 1742.094, 'min': 1500.0, 'max': 2250.0}, {'current': 3026.54, 'min': 1500.0, 'max': 2250.0}, {'current': 1786.214, 'min': 1500.0, 'max': 2250.0}, {'current': 1742.547, 'min': 1500.0, 'max': 2250.0}, {'current': 1728.916, 'min': 1500.0, 'max': 2250.0}, {'current': 1734.023, 'min': 1500.0, 'max': 2250.0}, {'current': 3195.219, 'min': 1500.0, 'max': 2250.0}, {'current': 1733.722, 'min': 1500.0, 'max': 2250.0}, {'current': 3341.911, 'min': 1500.0, 'max': 2250.0}, {'current': 3325.608, 'min': 1500.0, 'max': 2250.0}, {'current': 3233.258, 'min': 1500.0, 'max': 2250.0}, {'current': 1669.847, 'min': 1500.0, 'max': 2250.0}, {'current': 1791.948, 'min': 1500.0, 'max': 2250.0}, {'current': 1796.794, 'min': 1500.0, 'max': 2250.0}, {'current': 1791.49, 'min': 1500.0, 'max': 2250.0}, {'current': 1793.945, 'min': 1500.0, 'max': 2250.0}, {'current': 3342.943, 'min': 1500.0, 'max': 2250.0}, {'current': 1669.791, 'min': 1500.0, 'max': 2250.0}, {'current': 1669.593, 'min': 1500.0, 'max': 2250.0}, {'current': 1694.312, 'min': 1500.0, 'max': 2250.0}, {'current': 1873.727, 'min': 1500.0, 'max': 2250.0}, {'current': 1724.813, 'min': 1500.0, 'max': 2250.0}, {'current': 2354.471, 'min': 1500.0, 'max': 2250.0}, {'current': 1718.662, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.588, 'min': 1500.0, 'max': 2250.0}, {'current': 1665.577, 'min': 1500.0, 'max': 2250.0}, {'current': 1616.671, 'min': 1500.0, 'max': 2250.0}, {'current': 2080.81, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.666, 'min': 1500.0, 'max': 2250.0}, {'current': 1652.559, 'min': 1500.0, 'max': 2250.0}, {'current': 3323.654, 'min': 1500.0, 'max': 2250.0}, {'current': 1671.311, 'min': 1500.0, 'max': 2250.0}, {'current': 1726.286, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.365, 'min': 1500.0, 'max': 2250.0}, {'current': 3320.57, 'min': 1500.0, 'max': 2250.0}, {'current': 1669.941, 'min': 1500.0, 'max': 2250.0}, {'current': 1791.021, 'min': 1500.0, 'max': 2250.0}, {'current': 1796.246, 'min': 1500.0, 'max': 2250.0}, {'current': 1793.946, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.848, 'min': 1500.0, 'max': 2250.0}, {'current': 3339.327, 'min': 1500.0, 'max': 2250.0}, {'current': 3344.315, 'min': 1500.0, 'max': 2250.0}, {'current': 3338.901, 'min': 1500.0, 'max': 2250.0}, {'current': 1668.541, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.526, 'min': 1500.0, 'max': 2250.0}, {'current': 1792.886, 'min': 1500.0, 'max': 2250.0}, {'current': 1796.844, 'min': 1500.0, 'max': 2250.0}, {'current': 1793.81, 'min': 1500.0, 'max': 2250.0}, {'current': 1724.861, 'min': 1500.0, 'max': 2250.0}, {'current': 2294.458, 'min': 1500.0, 'max': 2250.0}, {'current': 1720.835, 'min': 1500.0, 'max': 2250.0}, {'current': 1720.155, 'min': 1500.0, 'max': 2250.0}, {'current': 1668.96, 'min': 1500.0, 'max': 2250.0}, {'current': 1976.5, 'min': 1500.0, 'max': 2250.0}, {'current': 2241.578, 'min': 1500.0, 'max': 2250.0}, {'current': 1671.964, 'min': 1500.0, 'max': 2250.0}, {'current': 3319.623, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.777, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.389, 'min': 1500.0, 'max': 2250.0}, {'current': 1669.629, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.19, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.138, 'min': 1500.0, 'max': 2250.0}, {'current': 1796.317, 'min': 1500.0, 'max': 2250.0}, {'current': 1792.821, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.716, 'min': 1500.0, 'max': 2250.0}, {'current': 1793.624, 'min': 1500.0, 'max': 2250.0}, {'current': 1796.346, 'min': 1500.0, 'max': 2250.0}, {'current': 1793.897, 'min': 1500.0, 'max': 2250.0}, {'current': 1735.424, 'min': 1500.0, 'max': 2250.0}, {'current': 1738.64, 'min': 1500.0, 'max': 2250.0}, {'current': 1979.998, 'min': 1500.0, 'max': 2250.0}, {'current': 1737.286, 'min': 1500.0, 'max': 2250.0}, {'current': 3313.748, 'min': 1500.0, 'max': 2250.0}, {'current': 3337.223, 'min': 1500.0, 'max': 2250.0}, {'current': 1671.416, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.005, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.276, 'min': 1500.0, 'max': 2250.0}, {'current': 1738.22, 'min': 1500.0, 'max': 2250.0}, {'current': 1742.737, 'min': 1500.0, 'max': 2250.0}, {'current': 1770.535, 'min': 1500.0, 'max': 2250.0}, {'current': 3320.252, 'min': 1500.0, 'max': 2250.0}, {'current': 1671.037, 'min': 1500.0, 'max': 2250.0}, {'current': 1669.549, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.948, 'min': 1500.0, 'max': 2250.0}, {'current': 2843.391, 'min': 1500.0, 'max': 2250.0}, {'current': 2348.589, 'min': 1500.0, 'max': 2250.0}, {'current': 3287.915, 'min': 1500.0, 'max': 2250.0}, {'current': 2340.192, 'min': 1500.0, 'max': 2250.0}, {'current': 2426.358, 'min': 1500.0, 'max': 2250.0}, {'current': 2415.833, 'min': 1500.0, 'max': 2250.0}, {'current': 2419.416, 'min': 1500.0, 'max': 2250.0}, {'current': 2277.433, 'min': 1500.0, 'max': 2250.0}, {'current': 2365.562, 'min': 1500.0, 'max': 2250.0}, {'current': 2400.6, 'min': 1500.0, 'max': 2250.0}, {'current': 2075.143, 'min': 1500.0, 'max': 2250.0}, {'current': 2382.295, 'min': 1500.0, 'max': 2250.0}, {'current': 3066.339, 'min': 1500.0, 'max': 2250.0}, {'current': 2466.631, 'min': 1500.0, 'max': 2250.0}, {'current': 3100.81, 'min': 1500.0, 'max': 2250.0}, {'current': 2421.93, 'min': 1500.0, 'max': 2250.0}, {'current': 3233.829, 'min': 1500.0, 'max': 2250.0}, {'current': 2234.583, 'min': 1500.0, 'max': 2250.0}, {'current': 2452.089, 'min': 1500.0, 'max': 2250.0}, {'current': 2975.985, 'min': 1500.0, 'max': 2250.0}, {'current': 3301.512, 'min': 1500.0, 'max': 2250.0}, {'current': 3336.905, 'min': 1500.0, 'max': 2250.0}, {'current': 2984.87, 'min': 1500.0, 'max': 2250.0}, {'current': 2384.306, 'min': 1500.0, 'max': 2250.0}, {'current': 2965.197, 'min': 1500.0, 'max': 2250.0}, {'current': 1929.067, 'min': 1500.0, 'max': 2250.0}, {'current': 1986.731, 'min': 1500.0, 'max': 2250.0}, {'current': 1999.412, 'min': 1500.0, 'max': 2250.0}, {'current': 2477.541, 'min': 1500.0, 'max': 2250.0}, {'current': 3111.851, 'min': 1500.0, 'max': 2250.0}, {'current': 2009.907, 'min': 1500.0, 'max': 2250.0}, {'current': 1993.784, 'min': 1500.0, 'max': 2250.0}, {'current': 2144.459, 'min': 1500.0, 'max': 2250.0}, {'current': 3337.426, 'min': 1500.0, 'max': 2250.0}, {'current': 3320.114, 'min': 1500.0, 'max': 2250.0}, {'current': 2169.719, 'min': 1500.0, 'max': 2250.0}, {'current': 3308.644, 'min': 1500.0, 'max': 2250.0}, {'current': 2111.633, 'min': 1500.0, 'max': 2250.0}, {'current': 2123.71, 'min': 1500.0, 'max': 2250.0}, {'current': 2153.49, 'min': 1500.0, 'max': 2250.0}], 'disk': {'/': {'total': 1757.8785285949707, 'used': 1663.5005989074707}}, 'gpu': 'NVIDIA A100-SXM4-80GB', 'gpu_count': 5, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA DGX Display', 'memory_total': 4294967296}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}], 'memory': {'total': 503.5396919250488}}
2024-05-13 20:46:53,372 INFO    HandlerThread:1244775 [system_monitor.py:probe():224] Finished collecting system info
2024-05-13 20:46:53,372 INFO    HandlerThread:1244775 [system_monitor.py:probe():227] Publishing system info
2024-05-13 20:46:53,372 DEBUG   HandlerThread:1244775 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-05-13 20:46:53,387 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:46:53,400 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:46:54,224 INFO    Thread-12 :1244775 [dir_watcher.py:_on_file_created():271] file/dir created: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/conda-environment.yaml
2024-05-13 20:46:55,418 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:46:55,429 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:46:55,741 DEBUG   HandlerThread:1244775 [system_info.py:_save_conda():222] Saving conda packages done
2024-05-13 20:46:55,742 INFO    HandlerThread:1244775 [system_monitor.py:probe():229] Finished publishing system info
2024-05-13 20:46:55,750 DEBUG   SenderThread:1244775 [sender.py:send():378] send: files
2024-05-13 20:46:55,750 INFO    SenderThread:1244775 [sender.py:_save_file():1389] saving file wandb-metadata.json with policy now
2024-05-13 20:46:55,863 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: python_packages
2024-05-13 20:46:55,863 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status
2024-05-13 20:46:55,863 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: python_packages
2024-05-13 20:46:55,865 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status
2024-05-13 20:46:56,093 DEBUG   SenderThread:1244775 [sender.py:send():378] send: telemetry
2024-05-13 20:46:56,093 DEBUG   SenderThread:1244775 [sender.py:send():378] send: config
2024-05-13 20:46:56,224 INFO    Thread-12 :1244775 [dir_watcher.py:_on_file_modified():288] file/dir modified: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/conda-environment.yaml
2024-05-13 20:46:56,224 INFO    Thread-12 :1244775 [dir_watcher.py:_on_file_created():271] file/dir created: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/wandb-metadata.json
2024-05-13 20:46:56,224 INFO    Thread-12 :1244775 [dir_watcher.py:_on_file_created():271] file/dir created: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/requirements.txt
2024-05-13 20:46:56,224 INFO    Thread-12 :1244775 [dir_watcher.py:_on_file_created():271] file/dir created: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/output.log
2024-05-13 20:46:56,261 INFO    wandb-upload_0:1244775 [upload_job.py:push():130] Uploaded file /tmp/tmpewnm1an9wandb/l0duo91p-wandb-metadata.json
2024-05-13 20:46:58,225 INFO    Thread-12 :1244775 [dir_watcher.py:_on_file_modified():288] file/dir modified: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/output.log
2024-05-13 20:46:58,331 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:46:58,343 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:46:58,386 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:47:00,362 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:00,375 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:03,387 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:47:03,431 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:03,443 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:05,466 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:05,478 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:08,388 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:47:08,679 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:08,692 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:10,713 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:10,724 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:10,863 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status
2024-05-13 20:47:10,863 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status
2024-05-13 20:47:12,746 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:12,757 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:14,094 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:47:15,627 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:15,637 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:17,658 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:17,668 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:19,096 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:47:20,779 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:20,799 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:22,817 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:22,830 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:24,099 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:47:24,858 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:24,870 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:25,233 INFO    Thread-12 :1244775 [dir_watcher.py:_on_file_modified():288] file/dir modified: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/config.yaml
2024-05-13 20:47:25,863 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status
2024-05-13 20:47:25,863 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status
2024-05-13 20:47:27,736 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:27,747 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:30,092 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:47:31,414 ERROR   gpu       :1244775 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero
2024-05-13 20:47:31,434 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:31,462 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:34,494 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:34,518 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:35,093 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:47:36,569 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:36,599 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:38,635 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:38,658 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:40,093 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:47:40,863 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status
2024-05-13 20:47:40,864 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status
2024-05-13 20:47:41,560 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:41,584 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:43,631 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:43,652 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:46,084 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:47:46,570 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:46,604 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:48,647 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:48,659 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:51,084 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:47:51,664 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:51,686 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:53,329 DEBUG   SystemMonitor:1244775 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-05-13 20:47:53,333 DEBUG   SenderThread:1244775 [sender.py:send():378] send: stats
2024-05-13 20:47:53,709 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:53,724 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:55,863 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status
2024-05-13 20:47:55,864 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status
2024-05-13 20:47:56,593 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:56,607 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:57,080 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:47:58,627 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:47:58,641 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:01,653 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:01,664 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:02,081 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:48:03,684 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:03,696 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:06,665 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:06,679 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:07,082 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:48:10,344 ERROR   gpu       :1244775 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero
2024-05-13 20:48:10,366 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:10,380 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:10,863 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status
2024-05-13 20:48:10,864 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status
2024-05-13 20:48:13,048 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:48:13,506 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:13,529 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:15,558 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:15,586 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:18,050 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:48:18,552 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:18,572 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:20,626 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:20,644 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:23,050 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:48:23,336 DEBUG   SenderThread:1244775 [sender.py:send():378] send: stats
2024-05-13 20:48:23,683 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:23,707 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:25,750 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:25,769 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:25,863 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status
2024-05-13 20:48:25,864 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status
2024-05-13 20:48:28,681 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:28,701 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:29,005 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:48:30,725 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:30,747 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:33,782 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:33,801 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:34,006 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:48:35,835 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:35,858 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:38,877 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:38,904 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:39,007 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:48:40,863 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status
2024-05-13 20:48:40,864 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status
2024-05-13 20:48:40,932 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:40,946 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:42,969 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:42,980 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:44,036 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:48:45,801 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:45,831 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:49,037 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:48:49,512 ERROR   gpu       :1244775 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero
2024-05-13 20:48:49,550 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:49,567 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:52,479 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:52,494 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:53,338 DEBUG   SenderThread:1244775 [sender.py:send():378] send: stats
2024-05-13 20:48:54,339 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:48:54,518 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:54,530 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:55,864 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status
2024-05-13 20:48:55,864 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status
2024-05-13 20:48:57,576 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:57,589 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:59,615 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:48:59,628 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:00,007 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:49:02,716 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:02,730 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:04,748 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:04,763 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:05,008 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:49:07,719 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:07,741 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:09,766 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:09,776 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:10,009 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:49:10,864 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status
2024-05-13 20:49:10,864 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status
2024-05-13 20:49:12,861 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:12,875 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:14,916 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:14,934 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:15,085 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:49:17,938 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:17,957 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:20,002 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:20,012 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:20,086 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:49:22,621 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:22,641 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:23,339 DEBUG   SenderThread:1244775 [sender.py:send():378] send: stats
2024-05-13 20:49:24,675 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:24,686 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:25,341 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:49:25,864 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status
2024-05-13 20:49:25,865 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status
2024-05-13 20:49:28,370 ERROR   gpu       :1244775 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero
2024-05-13 20:49:28,612 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:28,633 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:30,656 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:30,678 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:31,038 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:49:33,121 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:33,150 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:35,177 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:35,217 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:36,039 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:49:37,932 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:37,967 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:40,002 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:40,032 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:40,864 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status
2024-05-13 20:49:40,864 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status
2024-05-13 20:49:42,037 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:49:42,295 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:42,620 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:44,641 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:44,688 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:47,037 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:49:47,242 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:47,286 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:49,336 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:49,362 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:51,898 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:51,927 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:52,038 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:49:53,343 DEBUG   SenderThread:1244775 [sender.py:send():378] send: stats
2024-05-13 20:49:54,378 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:54,396 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:55,864 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status
2024-05-13 20:49:55,864 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status
2024-05-13 20:49:56,417 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:56,465 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:58,028 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:49:58,997 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:49:59,011 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:01,057 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:01,097 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:03,029 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:50:05,200 ERROR   gpu       :1244775 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero
2024-05-13 20:50:05,253 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:05,272 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:07,311 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:07,330 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:08,029 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:50:09,887 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:09,918 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:10,864 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status
2024-05-13 20:50:10,864 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status
2024-05-13 20:50:11,934 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:11,942 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:13,047 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:50:14,680 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:14,697 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:16,719 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:16,738 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:18,047 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:50:19,232 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:19,254 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:21,272 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:21,282 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:23,048 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:50:23,346 DEBUG   SenderThread:1244775 [sender.py:send():378] send: stats
2024-05-13 20:50:23,802 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:23,817 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:25,839 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:25,850 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:25,864 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: stop_status
2024-05-13 20:50:25,865 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: stop_status
2024-05-13 20:50:28,061 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:50:28,256 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:28,265 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:30,284 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:30,296 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:32,293 INFO    Thread-12 :1244775 [dir_watcher.py:_on_file_modified():288] file/dir modified: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/output.log
2024-05-13 20:50:32,787 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:32,796 ERROR   gpu       :1244775 [interfaces.py:monitor():142] Failed to sample metric: Not Supported
2024-05-13 20:50:33,152 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:50:33,158 DEBUG   SenderThread:1244775 [sender.py:send():378] send: exit
2024-05-13 20:50:33,158 INFO    SenderThread:1244775 [sender.py:send_exit():585] handling exit code: 1
2024-05-13 20:50:33,159 INFO    SenderThread:1244775 [sender.py:send_exit():587] handling runtime: 219
2024-05-13 20:50:33,159 INFO    SenderThread:1244775 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-13 20:50:33,159 INFO    SenderThread:1244775 [sender.py:send_exit():593] send defer
2024-05-13 20:50:33,159 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer
2024-05-13 20:50:33,159 INFO    HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 0
2024-05-13 20:50:33,159 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: defer
2024-05-13 20:50:33,160 INFO    SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 0
2024-05-13 20:50:33,160 INFO    SenderThread:1244775 [sender.py:transition_state():613] send defer: 1
2024-05-13 20:50:33,160 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer
2024-05-13 20:50:33,160 INFO    HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 1
2024-05-13 20:50:33,160 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: defer
2024-05-13 20:50:33,160 INFO    SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 1
2024-05-13 20:50:33,160 INFO    SenderThread:1244775 [sender.py:transition_state():613] send defer: 2
2024-05-13 20:50:33,160 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer
2024-05-13 20:50:33,160 INFO    HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 2
2024-05-13 20:50:33,160 INFO    HandlerThread:1244775 [system_monitor.py:finish():203] Stopping system monitor
2024-05-13 20:50:33,161 DEBUG   SystemMonitor:1244775 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-05-13 20:50:33,161 DEBUG   SystemMonitor:1244775 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-05-13 20:50:33,161 INFO    HandlerThread:1244775 [interfaces.py:finish():200] Joined cpu monitor
2024-05-13 20:50:33,164 INFO    HandlerThread:1244775 [interfaces.py:finish():200] Joined disk monitor
2024-05-13 20:50:33,293 INFO    Thread-12 :1244775 [dir_watcher.py:_on_file_created():271] file/dir created: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/wandb-summary.json
2024-05-13 20:50:34,293 INFO    Thread-12 :1244775 [dir_watcher.py:_on_file_modified():288] file/dir modified: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/output.log
2024-05-13 20:50:34,797 ERROR   gpu       :1244775 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero
2024-05-13 20:50:34,797 INFO    HandlerThread:1244775 [interfaces.py:finish():200] Joined gpu monitor
2024-05-13 20:50:34,797 INFO    HandlerThread:1244775 [interfaces.py:finish():200] Joined memory monitor
2024-05-13 20:50:34,797 INFO    HandlerThread:1244775 [interfaces.py:finish():200] Joined network monitor
2024-05-13 20:50:34,798 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: poll_exit
2024-05-13 20:50:34,799 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: defer
2024-05-13 20:50:34,799 INFO    SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 2
2024-05-13 20:50:34,799 INFO    SenderThread:1244775 [sender.py:transition_state():613] send defer: 3
2024-05-13 20:50:34,800 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer
2024-05-13 20:50:34,800 DEBUG   SenderThread:1244775 [sender.py:send():378] send: stats
2024-05-13 20:50:34,800 INFO    HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 3
2024-05-13 20:50:34,800 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: poll_exit
2024-05-13 20:50:34,801 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: defer
2024-05-13 20:50:34,801 INFO    SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 3
2024-05-13 20:50:34,801 INFO    SenderThread:1244775 [sender.py:transition_state():613] send defer: 4
2024-05-13 20:50:34,801 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer
2024-05-13 20:50:34,801 INFO    HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 4
2024-05-13 20:50:34,801 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: defer
2024-05-13 20:50:34,801 INFO    SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 4
2024-05-13 20:50:34,801 INFO    SenderThread:1244775 [sender.py:transition_state():613] send defer: 5
2024-05-13 20:50:34,802 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer
2024-05-13 20:50:34,802 INFO    HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 5
2024-05-13 20:50:34,802 DEBUG   SenderThread:1244775 [sender.py:send():378] send: summary
2024-05-13 20:50:34,802 INFO    SenderThread:1244775 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-13 20:50:34,802 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: defer
2024-05-13 20:50:34,802 INFO    SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 5
2024-05-13 20:50:34,802 INFO    SenderThread:1244775 [sender.py:transition_state():613] send defer: 6
2024-05-13 20:50:34,802 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer
2024-05-13 20:50:34,802 INFO    HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 6
2024-05-13 20:50:34,803 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: defer
2024-05-13 20:50:34,803 INFO    SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 6
2024-05-13 20:50:34,803 INFO    SenderThread:1244775 [sender.py:transition_state():613] send defer: 7
2024-05-13 20:50:34,803 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: status_report
2024-05-13 20:50:34,803 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer
2024-05-13 20:50:34,803 INFO    HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 7
2024-05-13 20:50:34,803 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: defer
2024-05-13 20:50:34,803 INFO    SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 7
2024-05-13 20:50:35,159 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: poll_exit
2024-05-13 20:50:35,294 INFO    Thread-12 :1244775 [dir_watcher.py:_on_file_modified():288] file/dir modified: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/wandb-summary.json
2024-05-13 20:50:38,152 INFO    SenderThread:1244775 [sender.py:transition_state():613] send defer: 8
2024-05-13 20:50:38,152 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: poll_exit
2024-05-13 20:50:38,152 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer
2024-05-13 20:50:38,153 INFO    HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 8
2024-05-13 20:50:38,153 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: defer
2024-05-13 20:50:38,153 INFO    SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 8
2024-05-13 20:50:38,153 INFO    SenderThread:1244775 [job_builder.py:build():432] Attempting to build job artifact
2024-05-13 20:50:38,153 INFO    SenderThread:1244775 [job_builder.py:_get_source_type():565] is repo sourced job
2024-05-13 20:50:38,160 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: poll_exit
2024-05-13 20:50:38,179 INFO    SenderThread:1244775 [job_builder.py:build():541] adding wandb-job metadata file
2024-05-13 20:50:38,181 INFO    SenderThread:1244775 [sender.py:transition_state():613] send defer: 9
2024-05-13 20:50:38,181 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: poll_exit
2024-05-13 20:50:38,181 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer
2024-05-13 20:50:38,182 INFO    HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 9
2024-05-13 20:50:38,182 DEBUG   SenderThread:1244775 [sender.py:send():378] send: artifact
2024-05-13 20:50:38,294 INFO    Thread-12 :1244775 [dir_watcher.py:_on_file_modified():288] file/dir modified: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/output.log
2024-05-13 20:50:39,160 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: poll_exit
2024-05-13 20:50:39,209 INFO    wandb-upload_0:1244775 [upload_job.py:push():88] Uploaded file /home/sanchit/.local/share/wandb/artifacts/staging/tmp34vs1_ku
2024-05-13 20:50:39,238 INFO    wandb-upload_1:1244775 [upload_job.py:push():88] Uploaded file /tmp/tmp_ne7l6g3/wandb-job.json
2024-05-13 20:50:40,085 INFO    SenderThread:1244775 [sender.py:send_artifact():1467] sent artifact job-https___huggingface.co_sanchit-gandhi_parler-tts-mini-v0.1-expresso-concatenated-combined_run_parler_tts_training.py - {'id': 'QXJ0aWZhY3Q6ODM0NzI5NzMx', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjE3NDIzMTI3NQ==', 'latestArtifact': None}}
2024-05-13 20:50:40,086 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: defer
2024-05-13 20:50:40,086 INFO    SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 9
2024-05-13 20:50:40,086 INFO    SenderThread:1244775 [dir_watcher.py:finish():358] shutting down directory watcher
2024-05-13 20:50:40,295 INFO    SenderThread:1244775 [dir_watcher.py:finish():388] scan: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files
2024-05-13 20:50:40,295 INFO    SenderThread:1244775 [dir_watcher.py:finish():402] scan save: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/conda-environment.yaml conda-environment.yaml
2024-05-13 20:50:40,295 INFO    SenderThread:1244775 [dir_watcher.py:finish():402] scan save: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/wandb-summary.json wandb-summary.json
2024-05-13 20:50:40,295 INFO    SenderThread:1244775 [dir_watcher.py:finish():402] scan save: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/output.log output.log
2024-05-13 20:50:40,297 INFO    SenderThread:1244775 [dir_watcher.py:finish():402] scan save: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/requirements.txt requirements.txt
2024-05-13 20:50:40,300 INFO    SenderThread:1244775 [dir_watcher.py:finish():402] scan save: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/config.yaml config.yaml
2024-05-13 20:50:40,301 INFO    SenderThread:1244775 [dir_watcher.py:finish():402] scan save: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/wandb-metadata.json wandb-metadata.json
2024-05-13 20:50:40,301 INFO    SenderThread:1244775 [sender.py:transition_state():613] send defer: 10
2024-05-13 20:50:40,301 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: poll_exit
2024-05-13 20:50:40,301 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer
2024-05-13 20:50:40,302 INFO    HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 10
2024-05-13 20:50:40,306 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: defer
2024-05-13 20:50:40,306 INFO    SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 10
2024-05-13 20:50:40,306 INFO    SenderThread:1244775 [file_pusher.py:finish():169] shutting down file pusher
2024-05-13 20:50:40,632 INFO    wandb-upload_0:1244775 [upload_job.py:push():130] Uploaded file /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/conda-environment.yaml
2024-05-13 20:50:40,639 INFO    wandb-upload_1:1244775 [upload_job.py:push():130] Uploaded file /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/output.log
2024-05-13 20:50:40,758 INFO    wandb-upload_3:1244775 [upload_job.py:push():130] Uploaded file /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/requirements.txt
2024-05-13 20:50:40,773 INFO    wandb-upload_2:1244775 [upload_job.py:push():130] Uploaded file /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/wandb-summary.json
2024-05-13 20:50:40,918 INFO    wandb-upload_4:1244775 [upload_job.py:push():130] Uploaded file /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/files/config.yaml
2024-05-13 20:50:41,118 INFO    Thread-11 (_thread_body):1244775 [sender.py:transition_state():613] send defer: 11
2024-05-13 20:50:41,119 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer
2024-05-13 20:50:41,119 INFO    HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 11
2024-05-13 20:50:41,119 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: defer
2024-05-13 20:50:41,119 INFO    SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 11
2024-05-13 20:50:41,120 INFO    SenderThread:1244775 [file_pusher.py:join():175] waiting for file pusher
2024-05-13 20:50:41,120 INFO    SenderThread:1244775 [sender.py:transition_state():613] send defer: 12
2024-05-13 20:50:41,120 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer
2024-05-13 20:50:41,120 INFO    HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 12
2024-05-13 20:50:41,120 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: defer
2024-05-13 20:50:41,120 INFO    SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 12
2024-05-13 20:50:41,120 INFO    SenderThread:1244775 [file_stream.py:finish():601] file stream finish called
2024-05-13 20:50:41,161 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: poll_exit
2024-05-13 20:50:41,510 INFO    SenderThread:1244775 [file_stream.py:finish():605] file stream finish is done
2024-05-13 20:50:41,510 INFO    SenderThread:1244775 [sender.py:transition_state():613] send defer: 13
2024-05-13 20:50:41,510 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: poll_exit
2024-05-13 20:50:41,510 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer
2024-05-13 20:50:41,510 INFO    HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 13
2024-05-13 20:50:41,510 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: defer
2024-05-13 20:50:41,510 INFO    SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 13
2024-05-13 20:50:41,510 INFO    SenderThread:1244775 [sender.py:transition_state():613] send defer: 14
2024-05-13 20:50:41,511 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: defer
2024-05-13 20:50:41,511 INFO    HandlerThread:1244775 [handler.py:handle_request_defer():184] handle defer: 14
2024-05-13 20:50:41,511 DEBUG   SenderThread:1244775 [sender.py:send():378] send: final
2024-05-13 20:50:41,511 DEBUG   SenderThread:1244775 [sender.py:send():378] send: footer
2024-05-13 20:50:41,511 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: defer
2024-05-13 20:50:41,511 INFO    SenderThread:1244775 [sender.py:send_request_defer():609] handle sender defer: 14
2024-05-13 20:50:41,512 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: poll_exit
2024-05-13 20:50:41,512 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: poll_exit
2024-05-13 20:50:41,512 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: poll_exit
2024-05-13 20:50:41,512 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: poll_exit
2024-05-13 20:50:41,513 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: server_info
2024-05-13 20:50:41,513 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: get_summary
2024-05-13 20:50:41,513 DEBUG   SenderThread:1244775 [sender.py:send_request():405] send_request: server_info
2024-05-13 20:50:41,515 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: sampled_history
2024-05-13 20:50:41,515 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: internal_messages
2024-05-13 20:50:41,651 INFO    MainThread:1244775 [wandb_run.py:_footer_history_summary_info():3994] rendering history
2024-05-13 20:50:41,651 INFO    MainThread:1244775 [wandb_run.py:_footer_history_summary_info():4026] rendering summary
2024-05-13 20:50:41,651 INFO    MainThread:1244775 [wandb_run.py:_footer_sync_info():3953] logging synced files
2024-05-13 20:50:41,651 DEBUG   HandlerThread:1244775 [handler.py:handle_request():158] handle_request: shutdown
2024-05-13 20:50:41,651 INFO    HandlerThread:1244775 [handler.py:finish():882] shutting down handler
2024-05-13 20:50:42,513 INFO    WriterThread:1244775 [datastore.py:close():296] close: /raid/sanchit/parler-tts-mini-v0.1-expresso-concatenated-combined/wandb/run-20240513_204652-m0g0ap7d/run-m0g0ap7d.wandb
2024-05-13 20:50:42,651 INFO    SenderThread:1244775 [sender.py:finish():1545] shutting down sender
2024-05-13 20:50:42,651 INFO    SenderThread:1244775 [file_pusher.py:finish():169] shutting down file pusher
2024-05-13 20:50:42,651 INFO    SenderThread:1244775 [file_pusher.py:join():175] waiting for file pusher