kertob commited on
Commit
03289ab
1 Parent(s): bdedb3d

Training in progress, step 20

Browse files
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:395c8dfa32f16e0fb0e9c83f70fc7bc9bf4d1d036050d6bacd8b762611f22c75
3
  size 261131840
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1ac30dac5d662aba0e9c2c433f9d802bdab0b896f0320f850174b591f34d306a
3
  size 261131840
runs/Mar08_13-01-52_4997a2709b0f/events.out.tfevents.1709902913.4997a2709b0f.24536.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:222dc23fe211fea7a258f3c809e77a991c4de351761b46a87f28d7dc2a3292c5
3
- size 5938
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:53bc2e3988fc85a013ae1436a5c28d700c2e2005b968ce245182f91766c75a4b
3
+ size 6145
wandb/debug-internal.log CHANGED
@@ -109,3 +109,52 @@
109
  2024-03-08 13:04:07,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
110
  2024-03-08 13:04:10,687 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
111
  2024-03-08 13:04:15,688 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
109
  2024-03-08 13:04:07,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
110
  2024-03-08 13:04:10,687 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
111
  2024-03-08 13:04:15,688 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
112
+ 2024-03-08 13:04:20,689 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
113
+ 2024-03-08 13:04:22,430 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: stop_status
114
+ 2024-03-08 13:04:22,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
115
+ 2024-03-08 13:04:22,471 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: internal_messages
116
+ 2024-03-08 13:04:25,695 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
117
+ 2024-03-08 13:04:30,696 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
118
+ 2024-03-08 13:04:35,697 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
119
+ 2024-03-08 13:04:36,587 DEBUG SenderThread:25394 [sender.py:send():378] send: stats
120
+ 2024-03-08 13:04:37,430 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: stop_status
121
+ 2024-03-08 13:04:37,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
122
+ 2024-03-08 13:04:37,471 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: internal_messages
123
+ 2024-03-08 13:04:40,733 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
124
+ 2024-03-08 13:04:45,735 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
125
+ 2024-03-08 13:04:50,735 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
126
+ 2024-03-08 13:04:52,431 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: stop_status
127
+ 2024-03-08 13:04:52,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
128
+ 2024-03-08 13:04:52,471 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: internal_messages
129
+ 2024-03-08 13:04:56,638 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
130
+ 2024-03-08 13:05:01,639 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
131
+ 2024-03-08 13:05:06,588 DEBUG SenderThread:25394 [sender.py:send():378] send: stats
132
+ 2024-03-08 13:05:07,431 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: stop_status
133
+ 2024-03-08 13:05:07,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
134
+ 2024-03-08 13:05:07,471 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: internal_messages
135
+ 2024-03-08 13:05:07,720 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
136
+ 2024-03-08 13:05:12,721 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
137
+ 2024-03-08 13:05:17,722 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
138
+ 2024-03-08 13:05:22,431 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: stop_status
139
+ 2024-03-08 13:05:22,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
140
+ 2024-03-08 13:05:22,472 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: internal_messages
141
+ 2024-03-08 13:05:23,716 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
142
+ 2024-03-08 13:05:28,717 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
143
+ 2024-03-08 13:05:33,717 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
144
+ 2024-03-08 13:05:36,589 DEBUG SenderThread:25394 [sender.py:send():378] send: stats
145
+ 2024-03-08 13:05:37,431 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: stop_status
146
+ 2024-03-08 13:05:37,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
147
+ 2024-03-08 13:05:37,472 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: internal_messages
148
+ 2024-03-08 13:05:38,252 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: partial_history
149
+ 2024-03-08 13:05:38,253 DEBUG SenderThread:25394 [sender.py:send():378] send: history
150
+ 2024-03-08 13:05:38,253 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: summary_record
151
+ 2024-03-08 13:05:38,254 INFO SenderThread:25394 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
152
+ 2024-03-08 13:05:38,411 INFO Thread-12 :25394 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_130200-qomx2zsg/files/wandb-summary.json
153
+ 2024-03-08 13:05:39,294 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
154
+ 2024-03-08 13:05:40,412 INFO Thread-12 :25394 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_130200-qomx2zsg/files/output.log
155
+ 2024-03-08 13:05:44,295 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
156
+ 2024-03-08 13:05:49,296 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
157
+ 2024-03-08 13:05:52,431 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: stop_status
158
+ 2024-03-08 13:05:52,432 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: internal_messages
159
+ 2024-03-08 13:05:52,432 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
160
+ 2024-03-08 13:05:54,647 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240308_130200-qomx2zsg/files/output.log CHANGED
@@ -1 +1,2 @@
1
 
 
 
1
 
2
+ Checkpoint destination directory ./checkpoint-10 already exists and is non-empty. Saving will proceed but saved results may be invalid.
wandb/run-20240308_130200-qomx2zsg/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 2.2808, "train/grad_norm": 1.2515443563461304, "train/learning_rate": 7.805935326811912e-05, "train/epoch": 0.8, "train/global_step": 10, "_timestamp": 1709903027.4653056, "_runtime": 106.67865467071533, "_step": 0}
 
1
+ {"train/loss": 0.3516, "train/grad_norm": 0.2534314692020416, "train/learning_rate": 2.65795779650105e-05, "train/epoch": 1.6, "train/global_step": 20, "_timestamp": 1709903138.2519133, "_runtime": 217.4652624130249, "_step": 1}
wandb/run-20240308_130200-qomx2zsg/logs/debug-internal.log CHANGED
@@ -109,3 +109,52 @@
109
  2024-03-08 13:04:07,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
110
  2024-03-08 13:04:10,687 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
111
  2024-03-08 13:04:15,688 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
109
  2024-03-08 13:04:07,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
110
  2024-03-08 13:04:10,687 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
111
  2024-03-08 13:04:15,688 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
112
+ 2024-03-08 13:04:20,689 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
113
+ 2024-03-08 13:04:22,430 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: stop_status
114
+ 2024-03-08 13:04:22,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
115
+ 2024-03-08 13:04:22,471 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: internal_messages
116
+ 2024-03-08 13:04:25,695 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
117
+ 2024-03-08 13:04:30,696 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
118
+ 2024-03-08 13:04:35,697 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
119
+ 2024-03-08 13:04:36,587 DEBUG SenderThread:25394 [sender.py:send():378] send: stats
120
+ 2024-03-08 13:04:37,430 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: stop_status
121
+ 2024-03-08 13:04:37,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
122
+ 2024-03-08 13:04:37,471 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: internal_messages
123
+ 2024-03-08 13:04:40,733 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
124
+ 2024-03-08 13:04:45,735 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
125
+ 2024-03-08 13:04:50,735 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
126
+ 2024-03-08 13:04:52,431 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: stop_status
127
+ 2024-03-08 13:04:52,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
128
+ 2024-03-08 13:04:52,471 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: internal_messages
129
+ 2024-03-08 13:04:56,638 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
130
+ 2024-03-08 13:05:01,639 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
131
+ 2024-03-08 13:05:06,588 DEBUG SenderThread:25394 [sender.py:send():378] send: stats
132
+ 2024-03-08 13:05:07,431 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: stop_status
133
+ 2024-03-08 13:05:07,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
134
+ 2024-03-08 13:05:07,471 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: internal_messages
135
+ 2024-03-08 13:05:07,720 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
136
+ 2024-03-08 13:05:12,721 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
137
+ 2024-03-08 13:05:17,722 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
138
+ 2024-03-08 13:05:22,431 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: stop_status
139
+ 2024-03-08 13:05:22,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
140
+ 2024-03-08 13:05:22,472 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: internal_messages
141
+ 2024-03-08 13:05:23,716 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
142
+ 2024-03-08 13:05:28,717 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
143
+ 2024-03-08 13:05:33,717 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
144
+ 2024-03-08 13:05:36,589 DEBUG SenderThread:25394 [sender.py:send():378] send: stats
145
+ 2024-03-08 13:05:37,431 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: stop_status
146
+ 2024-03-08 13:05:37,431 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
147
+ 2024-03-08 13:05:37,472 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: internal_messages
148
+ 2024-03-08 13:05:38,252 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: partial_history
149
+ 2024-03-08 13:05:38,253 DEBUG SenderThread:25394 [sender.py:send():378] send: history
150
+ 2024-03-08 13:05:38,253 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: summary_record
151
+ 2024-03-08 13:05:38,254 INFO SenderThread:25394 [sender.py:_save_file():1365] saving file wandb-summary.json with policy end
152
+ 2024-03-08 13:05:38,411 INFO Thread-12 :25394 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_130200-qomx2zsg/files/wandb-summary.json
153
+ 2024-03-08 13:05:39,294 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
154
+ 2024-03-08 13:05:40,412 INFO Thread-12 :25394 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240308_130200-qomx2zsg/files/output.log
155
+ 2024-03-08 13:05:44,295 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
156
+ 2024-03-08 13:05:49,296 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
157
+ 2024-03-08 13:05:52,431 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: stop_status
158
+ 2024-03-08 13:05:52,432 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: internal_messages
159
+ 2024-03-08 13:05:52,432 DEBUG SenderThread:25394 [sender.py:send_request():405] send_request: stop_status
160
+ 2024-03-08 13:05:54,647 DEBUG HandlerThread:25394 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240308_130200-qomx2zsg/run-qomx2zsg.wandb CHANGED
Binary files a/wandb/run-20240308_130200-qomx2zsg/run-qomx2zsg.wandb and b/wandb/run-20240308_130200-qomx2zsg/run-qomx2zsg.wandb differ