|
--- |
|
inference: false |
|
tags: |
|
- gguf |
|
- quantized |
|
- roleplay |
|
- multimodal |
|
- vision |
|
- llava |
|
- sillytavern |
|
- merge |
|
- mistral |
|
- conversational |
|
license: other |
|
--- |
|
|
|
> [!TIP] |
|
> **Support:** <br> |
|
> My upload speeds have been cooked and unstable lately. <br> |
|
> Realistically I'd need to move to get a better provider. <br> |
|
> If you **want** and you are able to... <br> |
|
> [**You can support my various endeavors here (Ko-fi).**](https://ko-fi.com/Lewdiculous) <br> |
|
> I apologize for disrupting your experience. |
|
|
|
|
|
# #Roleplay #Multimodal #Vision #Based #Unhinged #Unaligned |
|
|
|
In this repository you can find **GGUF-IQ-Imatrix** quants for [ChaoticNeutrals/Nyanade_Stunna-Maid-7B-v0.2](https://huggingface.co/ChaoticNeutrals/Nyanade_Stunna-Maid-7B-v0.2) and if needed you can get some basic SillyTavern presets [here](https://huggingface.co/Lewdiculous/Model-Requests/tree/main/data/presets/lewdicu-3.0.2-mistral-0.2), if you have issues with repetitiveness or lack or variety in responses I recommend changing the **Temperature** to 1.15, **MinP** to 0.075, **RepPen** to 1.15 and **RepPenRange** to 1024. |
|
|
|
> [!TIP] |
|
> **Vision:** <br> |
|
> This is a **#multimodal** model that also has optional **#vision** capabilities. <br> Expand the relevant sections bellow and read the full card information if you also want to make use that functionality. |
|
> |
|
> **Quant options:** <br> |
|
> Reading bellow you can also find quant option recommendations for some common GPU VRAM capacities. |
|
|
|
**"Unhinged RP with the spice of the previous 0.420 remixes, 32k context and vision capabilities."** |
|
|
|
 |
|
|
|
|
|
# General recommendations for quant options: |
|
|
|
<details><summary> |
|
⇲ Click here to expand/hide general common recommendations. |
|
</summary> |
|
|
|
*Assuming a context size of 8192 for simplicity and 1GB of Operating System VRAM overhead with some safety margin to avoid overflowing buffers...* <br> <br> |
|
**For 11-12GB VRAM:** <br> A GPU with **11-12GB** of VRAM capacity can comfortably use the **Q6_K-imat** quant option and run it at good speeds. <br> This is the same with or without using #vision capabilities. <br> <br> |
|
**For 8GB VRAM:** <br> If not using #vision, for GPUs with **8GB** of VRAM capacity the **Q5_K_M-imat** quant option will fit comfortably and should run at good speeds. <br> If **you are** also using #vision from this model opt for the **Q4_K_M-imat** quant option to avoid filling the buffers and potential slowdowns. <br><br> |
|
**For 6GB VRAM:** <br> If not using #vision, for GPUs with **6GB** of VRAM capacity the **IQ3_M-imat** quant option should fit comfortably to run at good speeds. <br> If **you are** also using #vision from this model opt for the **IQ3_XXS-imat** quant option. <br><br> |
|
|
|
</details><br> |
|
|
|
|
|
# Quantization process information: |
|
|
|
<details><summary> |
|
⇲ Click here to expand/hide more information about this topic. |
|
</summary> |
|
|
|
```python |
|
quantization_options = [ |
|
"IQ3_M", "IQ3_XXS", |
|
"Q4_K_M", "Q4_K_S", "IQ4_XS", "IQ4_NL", |
|
"Q5_K_M", "Q5_K_S", |
|
"Q6_K", |
|
"Q8_0" |
|
] |
|
``` |
|
|
|
**Steps performed:** |
|
|
|
``` |
|
Base⇢ GGUF(F16)⇢ Imatrix-Data(F16)⇢ GGUF(Imatrix-Quants) |
|
``` |
|
The latest of **llama.cpp** available at the time was used, with [imatrix-with-rp-ex.txt](https://huggingface.co/Lewdiculous/Model-Requests/blob/main/data/imatrix/imatrix-with-rp-ex.txt) as calibration data. |
|
|
|
</details><br> |
|
|
|
# What does "Imatrix" mean? |
|
|
|
<details><summary> |
|
⇲ Click here to expand/hide more information about this topic. |
|
</summary> |
|
|
|
It stands for **Importance Matrix**, a technique used to improve the quality of quantized models. |
|
The **Imatrix** is calculated based on calibration data, and it helps determine the importance of different model activations during the quantization process. |
|
The idea is to preserve the most important information during quantization, which can help reduce the loss of model performance, especially when the calibration data is diverse. |
|
[[1]](https://github.com/ggerganov/llama.cpp/discussions/5006) [[2]](https://github.com/ggerganov/llama.cpp/discussions/5263#discussioncomment-8395384) |
|
|
|
> [!NOTE] |
|
> For imatrix data generation, kalomaze's `groups_merged.txt` with additional roleplay chats was used, you can find it [here](https://huggingface.co/Lewdiculous/Model-Requests/blob/main/data/imatrix/imatrix-with-rp-ex.txt) for reference. This was just to add a bit more diversity to the data with the intended use case in mind. |
|
|
|
</details><br> |
|
|
|
# Vision/multimodal capabilities: |
|
|
|
<details><summary> |
|
⇲ Click here to expand/hide how this would work in practice in a roleplay chat. |
|
</summary> |
|
|
|
 |
|
|
|
</details><br> |
|
|
|
<details><summary> |
|
⇲ Click here to expand/hide how your SillyTavern Image Captions extension settings should look. |
|
</summary> |
|
|
|
 |
|
|
|
</details><br> |
|
|
|
# Required for vision functionality: |
|
|
|
> [!WARNING] |
|
> To use the multimodal capabilities of this model, such as **vision**, you also need to load the specified **mmproj** file, you can get it [here](https://huggingface.co/cjpais/llava-1.6-mistral-7b-gguf/blob/main/mmproj-model-f16.gguf) or as uploaded in the **mmproj** folder in the repository. |
|
|
|
1: Make sure you are using the latest version of [KoboldCpp](https://github.com/LostRuins/koboldcpp). |
|
|
|
2: Load the **mmproj file** by using the corresponding section in the interface: |
|
|
|
 |
|
|
|
2.1: For **CLI** users, you can load the **mmproj file** by adding the respective flag to your usual command: |
|
|
|
``` |
|
--mmproj your-mmproj-file.gguf |
|
``` |
