---
datasets:
- chargoddard/Open-Platypus-Chat
language:
- en
tags:
- llama
---
Experimental ReLoRA-trained model using the OpenPlatypus dataset. Ran for one epoch, with three lora restarts.

Not recommended for use yet. Mostly tossing this up for testing.

Base model was [llama2-22b-blocktriangular](https://huggingface.co/chargoddard/llama2-22b-blocktriangular).

Relevant training parameters:
```
adapter: qlora
load_in_4bit: true
lora_r: 32
lora_alpha: 16
lora_dropout: 0.001
lora_target_linear: true
relora_steps: 150
relora_warmup_steps: 10
gradient_accumulation_steps: 2
micro_batch_size: 3
```

Uses the same prompt format as [Ypotryll-22b](https://huggingface.co/chargoddard/ypotryll-22b-epoch2-qlora).
Prefix messages with `" ***System:"`, `" ***Query:"`, or `" ***Response:"`, paying attention to whitespace.


[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_chargoddard__platypus-2-22b-relora)

| Metric                | Value                     |
|-----------------------|---------------------------|
| Avg.                  | 52.21   |
| ARC (25-shot)         | 57.68          |
| HellaSwag (10-shot)   | 82.44    |
| MMLU (5-shot)         | 55.33         |
| TruthfulQA (0-shot)   | 43.61   |
| Winogrande (5-shot)   | 77.35   |
| GSM8K (5-shot)        | 6.6        |
| DROP (3-shot)         | 42.46         |