kertob commited on
Commit
b7faf67
1 Parent(s): 83e9ded

Training in progress, step 50

Browse files
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:154c96d9dcba521d7788dc8d810e0ef73cb69ca8097d6968402bbed02d575e25
3
  size 261131840
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1f477507523a7bd4c4f3bb8819ad03c8bd75d2f57c022361abf95504b1acda1d
3
  size 261131840
runs/Mar08_13-48-53_4997a2709b0f/events.out.tfevents.1709905734.4997a2709b0f.35964.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:625810ba3c41c66e2b3fa4c474e6050f50649597e5af558685758a5256bd4ec0
3
- size 6560
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7da9c532834034f413485a66bdd714ed61658be4e80b3531a59b99df80e58a71
3
+ size 6767
wandb/debug-internal.log CHANGED
@@ -233,3 +233,49 @@
233
  2024-03-08 13:55:22,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
234
  2024-03-08 13:55:22,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
235
  2024-03-08 13:55:22,923 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
233
  2024-03-08 13:55:22,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
234
  2024-03-08 13:55:22,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
235
  2024-03-08 13:55:22,923 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
236
+ 2024-03-08 13:55:27,924 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
237
+ 2024-03-08 13:55:32,925 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
238
+ 2024-03-08 13:55:37,178 DEBUG SenderThread:37167 [sender.py:send():378] send: stats
239
+ 2024-03-08 13:55:37,651 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
240
+ 2024-03-08 13:55:37,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
241
+ 2024-03-08 13:55:37,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
242
+ 2024-03-08 13:55:38,904 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
243
+ 2024-03-08 13:55:43,905 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
244
+ 2024-03-08 13:55:48,906 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
245
+ 2024-03-08 13:55:52,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
246
+ 2024-03-08 13:55:52,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
247
+ 2024-03-08 13:55:52,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
248
+ 2024-03-08 13:55:54,850 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
249
+ 2024-03-08 13:55:59,850 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
250
+ 2024-03-08 13:56:04,851 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
251
+ 2024-03-08 13:56:07,179 DEBUG SenderThread:37167 [sender.py:send():378] send: stats
252
+ 2024-03-08 13:56:07,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
253
+ 2024-03-08 13:56:07,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
254
+ 2024-03-08 13:56:07,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
255
+ 2024-03-08 13:56:09,877 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
256
+ 2024-03-08 13:56:14,878 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
257
+ 2024-03-08 13:56:19,880 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
258
+ 2024-03-08 13:56:22,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
259
+ 2024-03-08 13:56:22,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
260
+ 2024-03-08 13:56:22,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
261
+ 2024-03-08 13:56:24,927 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
262
+ 2024-03-08 13:56:29,928 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
263
+ 2024-03-08 13:56:34,929 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
264
+ 2024-03-08 13:56:37,180 DEBUG SenderThread:37167 [sender.py:send():378] send: stats
265
+ 2024-03-08 13:56:37,387 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: partial_history
266
+ 2024-03-08 13:56:37,389 DEBUG SenderThread:37167 [sender.py:send():378] send: history
267
+ 2024-03-08 13:56:37,389 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: summary_record
268
+ 2024-03-08 13:56:37,390 INFO SenderThread:37167 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
269
+ 2024-03-08 13:56:37,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
270
+ 2024-03-08 13:56:37,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
271
+ 2024-03-08 13:56:37,653 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
272
+ 2024-03-08 13:56:38,107 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/wandb-summary.json
273
+ 2024-03-08 13:56:39,940 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
274
+ 2024-03-08 13:56:40,108 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/output.log
275
+ 2024-03-08 13:56:44,942 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
276
+ 2024-03-08 13:56:49,943 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
277
+ 2024-03-08 13:56:52,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
278
+ 2024-03-08 13:56:52,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
279
+ 2024-03-08 13:56:52,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
280
+ 2024-03-08 13:56:55,868 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
281
+ 2024-03-08 13:57:00,869 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240308_134901-hlg2f18u/files/output.log CHANGED
@@ -2,3 +2,4 @@
2
  Checkpoint destination directory ./checkpoint-10 already exists and is non-empty. Saving will proceed but saved results may be invalid.
3
  Checkpoint destination directory ./checkpoint-20 already exists and is non-empty. Saving will proceed but saved results may be invalid.
4
  Checkpoint destination directory ./checkpoint-30 already exists and is non-empty. Saving will proceed but saved results may be invalid.
 
 
2
  Checkpoint destination directory ./checkpoint-10 already exists and is non-empty. Saving will proceed but saved results may be invalid.
3
  Checkpoint destination directory ./checkpoint-20 already exists and is non-empty. Saving will proceed but saved results may be invalid.
4
  Checkpoint destination directory ./checkpoint-30 already exists and is non-empty. Saving will proceed but saved results may be invalid.
5
+ Checkpoint destination directory ./checkpoint-40 already exists and is non-empty. Saving will proceed but saved results may be invalid.
wandb/run-20240308_134901-hlg2f18u/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 0.6586, "train/grad_norm": 0.8443416953086853, "train/learning_rate": 9.261084118279847e-05, "train/epoch": 3.2, "train/global_step": 40, "_timestamp": 1709906104.3603015, "_runtime": 362.94000458717346, "_step": 3}
 
1
+ {"train/loss": 0.5368, "train/grad_norm": 1.1711567640304565, "train/learning_rate": 8.783564079088477e-05, "train/epoch": 4.0, "train/global_step": 50, "_timestamp": 1709906197.3872814, "_runtime": 455.96698451042175, "_step": 4}
wandb/run-20240308_134901-hlg2f18u/logs/debug-internal.log CHANGED
@@ -233,3 +233,49 @@
233
  2024-03-08 13:55:22,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
234
  2024-03-08 13:55:22,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
235
  2024-03-08 13:55:22,923 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
233
  2024-03-08 13:55:22,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
234
  2024-03-08 13:55:22,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
235
  2024-03-08 13:55:22,923 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
236
+ 2024-03-08 13:55:27,924 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
237
+ 2024-03-08 13:55:32,925 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
238
+ 2024-03-08 13:55:37,178 DEBUG SenderThread:37167 [sender.py:send():378] send: stats
239
+ 2024-03-08 13:55:37,651 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
240
+ 2024-03-08 13:55:37,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
241
+ 2024-03-08 13:55:37,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
242
+ 2024-03-08 13:55:38,904 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
243
+ 2024-03-08 13:55:43,905 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
244
+ 2024-03-08 13:55:48,906 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
245
+ 2024-03-08 13:55:52,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
246
+ 2024-03-08 13:55:52,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
247
+ 2024-03-08 13:55:52,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
248
+ 2024-03-08 13:55:54,850 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
249
+ 2024-03-08 13:55:59,850 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
250
+ 2024-03-08 13:56:04,851 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
251
+ 2024-03-08 13:56:07,179 DEBUG SenderThread:37167 [sender.py:send():378] send: stats
252
+ 2024-03-08 13:56:07,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
253
+ 2024-03-08 13:56:07,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
254
+ 2024-03-08 13:56:07,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
255
+ 2024-03-08 13:56:09,877 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
256
+ 2024-03-08 13:56:14,878 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
257
+ 2024-03-08 13:56:19,880 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
258
+ 2024-03-08 13:56:22,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
259
+ 2024-03-08 13:56:22,652 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
260
+ 2024-03-08 13:56:22,692 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
261
+ 2024-03-08 13:56:24,927 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
262
+ 2024-03-08 13:56:29,928 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
263
+ 2024-03-08 13:56:34,929 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
264
+ 2024-03-08 13:56:37,180 DEBUG SenderThread:37167 [sender.py:send():378] send: stats
265
+ 2024-03-08 13:56:37,387 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: partial_history
266
+ 2024-03-08 13:56:37,389 DEBUG SenderThread:37167 [sender.py:send():378] send: history
267
+ 2024-03-08 13:56:37,389 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: summary_record
268
+ 2024-03-08 13:56:37,390 INFO SenderThread:37167 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
269
+ 2024-03-08 13:56:37,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
270
+ 2024-03-08 13:56:37,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
271
+ 2024-03-08 13:56:37,653 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
272
+ 2024-03-08 13:56:38,107 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/wandb-summary.json
273
+ 2024-03-08 13:56:39,940 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
274
+ 2024-03-08 13:56:40,108 INFO Thread-12 :37167 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_134901-hlg2f18u/files/output.log
275
+ 2024-03-08 13:56:44,942 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
276
+ 2024-03-08 13:56:49,943 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
277
+ 2024-03-08 13:56:52,652 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: stop_status
278
+ 2024-03-08 13:56:52,653 DEBUG SenderThread:37167 [sender.py:send_request():405] send_request: stop_status
279
+ 2024-03-08 13:56:52,693 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: internal_messages
280
+ 2024-03-08 13:56:55,868 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
281
+ 2024-03-08 13:57:00,869 DEBUG HandlerThread:37167 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240308_134901-hlg2f18u/run-hlg2f18u.wandb CHANGED
Binary files a/wandb/run-20240308_134901-hlg2f18u/run-hlg2f18u.wandb and b/wandb/run-20240308_134901-hlg2f18u/run-hlg2f18u.wandb differ