kertob commited on
Commit
937842a
1 Parent(s): b7faf67

Training in progress, step 60

Browse files
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:1f477507523a7bd4c4f3bb8819ad03c8bd75d2f57c022361abf95504b1acda1d
3
  size 261131840
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6c38dac264184caebde634b60dea7d9ad3b2942a76173ca94775c96d2173b2b2
3
  size 261131840
runs/Mar08_13-48-53_4997a2709b0f/events.out.tfevents.1709905734.4997a2709b0f.35964.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:7da9c532834034f413485a66bdd714ed61658be4e80b3531a59b99df80e58a71
3
- size 6767
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7bec81605b3e0e401e5393389c1106505a374ad704126f36e03f6523a57f7dd8
3
+ size 6974
wandb/debug-internal.log CHANGED
@@ -279,3 +279,46 @@
279
  2024-03-08 13:56:52,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
280
  2024-03-08 13:56:55,868 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
281
  2024-03-08 13:57:00,869 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
279
  2024-03-08 13:56:52,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
280
  2024-03-08 13:56:55,868 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
281
  2024-03-08 13:57:00,869 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
282
+ 2024-03-08 13:57:05,870 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
283
+ 2024-03-08 13:57:07,180 DEBUG SenderThread:37167 [sender.py:send():378] send: stats
284
+ 2024-03-08 13:57:07,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
285
+ 2024-03-08 13:57:07,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
286
+ 2024-03-08 13:57:07,694 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
287
+ 2024-03-08 13:57:10,899 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
288
+ 2024-03-08 13:57:15,901 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
289
+ 2024-03-08 13:57:20,902 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
290
+ 2024-03-08 13:57:22,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
291
+ 2024-03-08 13:57:22,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
292
+ 2024-03-08 13:57:22,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
293
+ 2024-03-08 13:57:26,888 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
294
+ 2024-03-08 13:57:31,888 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
295
+ 2024-03-08 13:57:36,890 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
296
+ 2024-03-08 13:57:37,181 DEBUG SenderThread:37167 [sender.py:send():378] send: stats
297
+ 2024-03-08 13:57:37,653 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
298
+ 2024-03-08 13:57:37,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
299
+ 2024-03-08 13:57:37,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
300
+ 2024-03-08 13:57:42,876 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
301
+ 2024-03-08 13:57:47,877 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
302
+ 2024-03-08 13:57:52,653 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
303
+ 2024-03-08 13:57:52,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
304
+ 2024-03-08 13:57:52,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
305
+ 2024-03-08 13:57:53,876 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
306
+ 2024-03-08 13:57:58,877 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
307
+ 2024-03-08 13:58:03,879 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
308
+ 2024-03-08 13:58:07,182 DEBUG SenderThread:37167 [sender.py:send():378] send: stats
309
+ 2024-03-08 13:58:07,653 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
310
+ 2024-03-08 13:58:07,654 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
311
+ 2024-03-08 13:58:07,695 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
312
+ 2024-03-08 13:58:09,855 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
313
+ 2024-03-08 13:58:13,676 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: partial_history
314
+ 2024-03-08 13:58:13,678 DEBUG SenderThread:37167 [sender.py:send():378] send: history
315
+ 2024-03-08 13:58:13,678 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: summary_record
316
+ 2024-03-08 13:58:13,679 INFO SenderThread:37167 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
317
+ 2024-03-08 13:58:14,144 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/wandb-summary.json
318
+ 2024-03-08 13:58:15,718 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
319
+ 2024-03-08 13:58:16,144 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/output.log
320
+ 2024-03-08 13:58:20,719 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
321
+ 2024-03-08 13:58:22,653 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
322
+ 2024-03-08 13:58:22,654 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
323
+ 2024-03-08 13:58:22,654 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
324
+ 2024-03-08 13:58:26,249 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240308_134901-hlg2f18u/files/output.log CHANGED
@@ -3,3 +3,4 @@ Checkpoint destination directory ./checkpoint-10 already exists and is non-empty
3
  Checkpoint destination directory ./checkpoint-20 already exists and is non-empty. Saving will proceed but saved results may be invalid.
4
  Checkpoint destination directory ./checkpoint-30 already exists and is non-empty. Saving will proceed but saved results may be invalid.
5
  Checkpoint destination directory ./checkpoint-40 already exists and is non-empty. Saving will proceed but saved results may be invalid.
 
 
3
  Checkpoint destination directory ./checkpoint-20 already exists and is non-empty. Saving will proceed but saved results may be invalid.
4
  Checkpoint destination directory ./checkpoint-30 already exists and is non-empty. Saving will proceed but saved results may be invalid.
5
  Checkpoint destination directory ./checkpoint-40 already exists and is non-empty. Saving will proceed but saved results may be invalid.
6
+ Checkpoint destination directory ./checkpoint-50 already exists and is non-empty. Saving will proceed but saved results may be invalid.
wandb/run-20240308_134901-hlg2f18u/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 0.5368, "train/grad_norm": 1.1711567640304565, "train/learning_rate": 8.783564079088477e-05, "train/epoch": 4.0, "train/global_step": 50, "_timestamp": 1709906197.3872814, "_runtime": 455.96698451042175, "_step": 4}
 
1
+ {"train/loss": 0.3697, "train/grad_norm": 0.572875440120697, "train/learning_rate": 8.2070410981557e-05, "train/epoch": 4.8, "train/global_step": 60, "_timestamp": 1709906293.6760013, "_runtime": 552.2557044029236, "_step": 5}
wandb/run-20240308_134901-hlg2f18u/logs/debug-internal.log CHANGED
@@ -279,3 +279,46 @@
279
  2024-03-08 13:56:52,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
280
  2024-03-08 13:56:55,868 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
281
  2024-03-08 13:57:00,869 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
279
  2024-03-08 13:56:52,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
280
  2024-03-08 13:56:55,868 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
281
  2024-03-08 13:57:00,869 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
282
+ 2024-03-08 13:57:05,870 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
283
+ 2024-03-08 13:57:07,180 DEBUG SenderThread:37167 [sender.py:send():378] send: stats
284
+ 2024-03-08 13:57:07,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
285
+ 2024-03-08 13:57:07,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
286
+ 2024-03-08 13:57:07,694 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
287
+ 2024-03-08 13:57:10,899 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
288
+ 2024-03-08 13:57:15,901 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
289
+ 2024-03-08 13:57:20,902 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
290
+ 2024-03-08 13:57:22,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
291
+ 2024-03-08 13:57:22,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
292
+ 2024-03-08 13:57:22,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
293
+ 2024-03-08 13:57:26,888 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
294
+ 2024-03-08 13:57:31,888 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
295
+ 2024-03-08 13:57:36,890 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
296
+ 2024-03-08 13:57:37,181 DEBUG SenderThread:37167 [sender.py:send():378] send: stats
297
+ 2024-03-08 13:57:37,653 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
298
+ 2024-03-08 13:57:37,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
299
+ 2024-03-08 13:57:37,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
300
+ 2024-03-08 13:57:42,876 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
301
+ 2024-03-08 13:57:47,877 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
302
+ 2024-03-08 13:57:52,653 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
303
+ 2024-03-08 13:57:52,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
304
+ 2024-03-08 13:57:52,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
305
+ 2024-03-08 13:57:53,876 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
306
+ 2024-03-08 13:57:58,877 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
307
+ 2024-03-08 13:58:03,879 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
308
+ 2024-03-08 13:58:07,182 DEBUG SenderThread:37167 [sender.py:send():378] send: stats
309
+ 2024-03-08 13:58:07,653 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
310
+ 2024-03-08 13:58:07,654 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
311
+ 2024-03-08 13:58:07,695 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
312
+ 2024-03-08 13:58:09,855 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
313
+ 2024-03-08 13:58:13,676 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: partial_history
314
+ 2024-03-08 13:58:13,678 DEBUG SenderThread:37167 [sender.py:send():378] send: history
315
+ 2024-03-08 13:58:13,678 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: summary_record
316
+ 2024-03-08 13:58:13,679 INFO SenderThread:37167 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
317
+ 2024-03-08 13:58:14,144 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/wandb-summary.json
318
+ 2024-03-08 13:58:15,718 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
319
+ 2024-03-08 13:58:16,144 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/output.log
320
+ 2024-03-08 13:58:20,719 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
321
+ 2024-03-08 13:58:22,653 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
322
+ 2024-03-08 13:58:22,654 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
323
+ 2024-03-08 13:58:22,654 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
324
+ 2024-03-08 13:58:26,249 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240308_134901-hlg2f18u/run-hlg2f18u.wandb CHANGED
Binary files a/wandb/run-20240308_134901-hlg2f18u/run-hlg2f18u.wandb and b/wandb/run-20240308_134901-hlg2f18u/run-hlg2f18u.wandb differ