Hugging Face's logo

Anthonyg5005
/
rishiraj-meow-10.7B-exl2

Text Generation
English
ExLlamaV2
SOLAR
Mistral
Quantized Model
exl2
conversational
Model card Files Community
rishiraj-meow-10.7B-exl2 / README.md
Anthonyg5005's picture
Anthonyg5005
Update README.md
e5961f1 verified
preview code
|
raw
history blame
2.84 kB
metadata
datasets:
  - c-s-ale/alpaca-gpt4-data
  - Open-Orca/OpenOrca
  - Intel/orca_dpo_pairs
  - allenai/ultrafeedback_binarized_cleaned
  - HuggingFaceH4/no_robots
license: cc-by-nc-4.0
language:
  - en
library_name: ExLlamaV2
pipeline_tag: text-generation
tags:
  - Mistral
  - SOLAR
  - Quantized Model
  - exl2
base_model:
  - rishiraj/meow

exl2 quants for meow

This repository includes the quantized models for the meow model by Rishiraj Acharya. meow is a fine-tune of SOLAR-10.7B-Instruct-v1.0 with the no_robots dataset.

Current models

exl2 BPW Model Branch Model Size Minimum VRAM (4096 Context, fp16 cache)
2-Bit main 3.28 GB 6GB GPU
4-Bit 4bit 5.61 GB 8GB GPU
5-Bit 5bit 6.92 GB 10GB GPU, 8GB with swap
6-Bit 6bit 8.23 GB 10GB GPU
8-Bit 8bit 10.84 GB 12GB GPU

Note

Using a 12GB Nvidia GeForce RTX 3060 I got on average around 20 tokens per second on the 8-bit quant in full 4096 context.

Where to use

There are a couple places you can use an exl2 model, here are a few:

How to download:

oobabooga's downloader

use something like download-model.py to download with python requests.
Install requirements:

pip install requests tqdm

Example for downloading 5bpw:

python download-model.py Anthonyg5005/rishiraj-meow-10.7B-exl2:5bit

huggingface-cli

You may also use huggingface-cli
To install it, install python hf-hub

pip install huggingface-hub

Example for 5bpw:

huggingface-cli download Anthonyg5005/rishiraj-meow-10.7B-exl2 --local-dir rishiraj-meow-10.7B-exl2-5bpw --revision 5bit

Git LFS (not recommended)

I would recommend the http downloaders over using git, they can resume downloads if failed and are much easier to work with.
Make sure to have git and git LFS installed.
Example for 5bpw download with git:

Have LFS file skip disabled

# windows
set GIT_LFS_SKIP_SMUDGE=0
# linux
export GIT_LFS_SKIP_SMUDGE=0

Clone repo branch

git clone https://huggingface.co/Anthonyg5005/rishiraj-meow-10.7B-exl2 -b 5bit