|
--- |
|
tags: |
|
- merge |
|
- gguf |
|
- llama3 |
|
- iMat |
|
--- |
|
<img src="https://i.imgur.com/P68dXux.png" width="400"/> |
|
|
|
# Llama-3-70B-Instruct-Storywriter-iMat-GGUF |
|
|
|
|
|
<b>Special request.</b> Quantized from fp32 with love. If you can't fit IQ quants in your VRAM, try using the K quants in this repo instead. |
|
* Weighted quantizations created using this [process](https://huggingface.co/jukofyork/WizardLM-2-8x22B-imatrix) |
|
* Calculated in 88 chunks with n_ctx=512 using groups_merged.txt |
|
|
|
For a brief rundown of iMatrix quant performance please see this [PR](https://github.com/ggerganov/llama.cpp/pull/5747) |
|
|
|
<i>All quants are verified working prior to uploading to repo for your safety and convenience. </i> |
|
|
|
|
|
<b>Tip:</b> Pick a file size under your GPU's VRAM while still allowing some room for context for best speed. You may need to pad this further depending on if you are running image gen or TTS as well. |
|
|
|
BFloat16 model card can be found [here](https://huggingface.co/tdrussell/Llama-3-70B-Instruct-Storywriter) |
