README.md

---
license: apache-2.0
---
---
license: apache-2.0
---
# LLM360 Research Suite: K2 Loss Spike 2
We encountered two major loss spikes while training K2. 
* The [first loss spike](https://huggingface.co/LLM360/K2-Spike-1/) occured after X checkpoints and lasted over ~34 checkpoints. We restarted training at checkpoint X and training returned to normal.
* The second loss spike occured after restarting training to fix the first loss spike at checkpoint X and lasted from ~8 checkpoints.

We are releasing these checkpoints so others can study this interesting phenomena in large model training.
<img src="k2_spike_1.png" alt="k2 spike 1"/>

# Purpose
Loss spikes are still a relatively unknown phenomena. By making these spikes and associated training details available, we hope others use these artifacts to further the worlds knowledge on this topic.

## All Checkpoints
| Checkpoints      |  |
| ----------- | ----------- |
| [Checkpoint 186](https://huggingface.co/LLM360/K2-Spike-2/tree/spike_ckpt_186)     | [Checkpoint 194](https://huggingface.co/LLM360/K2-Spike-2/tree/spike_ckpt_194)       |
| [Checkpoint 188](https://huggingface.co/LLM360/K2-Spike-2/tree/spike_ckpt_188)   | [Checkpoint 196](https://huggingface.co/LLM360/K2-Spike-2/tree/spike_ckpt_196)        |
| [Checkpoint 190](https://huggingface.co/LLM360/K2-Spike-2/tree/spike_ckpt_190)   | [Checkpoint 198](https://huggingface.co/LLM360/K2-Spike-2/tree/spike_ckpt_198)        |
| [Checkpoint 192](https://huggingface.co/LLM360/K2-Spike-2/tree/spike_ckpt_192)   | [Checkpoint 200](https://huggingface.co/LLM360/K2-Spike-2/tree/spike_ckpt_200)        |


[to find all branches: git branch -a]

## Loss Spike's on the LLM360 Evaluation Suite

something here

## About the LLM360 Research Suite
The LLM360 Research Suite is a comprehensive set of large language model (LLM) artifacts from Amber, CrystalCoder, and K2 for academic and industry researchers to explore LLM training dynamics. Additional resources can be found at llm360.ai.