InternLM Math Plus
Collection
4 items
β’
Updated
Llama.cpp imatrix quantization of internlm/internlm2-math-plus-20b
Original Model: internlm/internlm2-math-plus-20b
Original dtype: BF16
(bfloat16
)
Quantized by: llama.cpp b3008
IMatrix dataset: here
Status: β
Available
Link: here
Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
---|---|---|---|---|---|
internlm2-math-plus-20b.Q8_0.gguf | Q8_0 | 21.11GB | β Available | βͺ Static | π¦ No |
internlm2-math-plus-20b.Q6_K.gguf | Q6_K | 16.30GB | β Available | βͺ Static | π¦ No |
internlm2-math-plus-20b.Q4_K.gguf | Q4_K | 11.98GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.Q3_K.gguf | Q3_K | 9.72GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.Q2_K.gguf | Q2_K | 7.55GB | β Available | π’ IMatrix | π¦ No |
Filename | Quant type | File Size | Status | Uses IMatrix | Is Split |
---|---|---|---|---|---|
internlm2-math-plus-20b.FP16.gguf | F16 | 39.73GB | β Available | βͺ Static | π¦ No |
internlm2-math-plus-20b.BF16.gguf | BF16 | 39.73GB | β Available | βͺ Static | π¦ No |
internlm2-math-plus-20b.Q5_K.gguf | Q5_K | 14.08GB | β Available | βͺ Static | π¦ No |
internlm2-math-plus-20b.Q5_K_S.gguf | Q5_K_S | 13.73GB | β Available | βͺ Static | π¦ No |
internlm2-math-plus-20b.Q4_K_S.gguf | Q4_K_S | 11.40GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.Q3_K_L.gguf | Q3_K_L | 10.55GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.Q3_K_S.gguf | Q3_K_S | 8.76GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.Q2_K_S.gguf | Q2_K_S | 7.01GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.IQ4_NL.gguf | IQ4_NL | 11.36GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.IQ4_XS.gguf | IQ4_XS | 10.77GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.IQ3_M.gguf | IQ3_M | 9.12GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.IQ3_S.gguf | IQ3_S | 8.80GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.IQ3_XS.gguf | IQ3_XS | 8.36GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.IQ3_XXS.gguf | IQ3_XXS | 7.81GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.IQ2_M.gguf | IQ2_M | 6.97GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.IQ2_S.gguf | IQ2_S | 6.47GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.IQ2_XS.gguf | IQ2_XS | 6.10GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.IQ2_XXS.gguf | IQ2_XXS | 5.54GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.IQ1_M.gguf | IQ1_M | 4.92GB | β Available | π’ IMatrix | π¦ No |
internlm2-math-plus-20b.IQ1_S.gguf | IQ1_S | 4.54GB | β Available | π’ IMatrix | π¦ No |
If you do not have hugginface-cli installed:
pip install -U "huggingface_hub[cli]"
Download the specific file you want:
huggingface-cli download legraphista/internlm2-math-plus-20b-IMat-GGUF --include "internlm2-math-plus-20b.Q8_0.gguf" --local-dir ./
If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:
huggingface-cli download legraphista/internlm2-math-plus-20b-IMat-GGUF --include "internlm2-math-plus-20b.Q8_0/*" --local-dir internlm2-math-plus-20b.Q8_0
# see FAQ for merging GGUF's
<s><|im_start|>user
Can you provide ways to eat combinations of bananas and dragonfruits?<|im_end|>
<|im_start|>assistant
Sure! Here are some ways to eat bananas and dragonfruits together:
1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey.
2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey.<|im_end|>
<|im_start|>user
What about solving an 2x + 3 = 7 equation?<|im_end|>
<s><|im_start|>system
You are a helpful AI.<|im_end|>
<|im_start|>user
Can you provide ways to eat combinations of bananas and dragonfruits?<|im_end|>
<|im_start|>assistant
Sure! Here are some ways to eat bananas and dragonfruits together:
1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey.
2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey.<|im_end|>
<|im_start|>user
What about solving an 2x + 3 = 7 equation?<|im_end|>
llama.cpp/main -m internlm2-math-plus-20b.Q8_0.gguf --color -i -p "prompt here (according to the chat template)"
According to this investigation, it appears that lower quantizations are the only ones that benefit from the imatrix input (as per hellaswag results).
gguf-split
availablegguf-split
, navigate to https://github.com/ggerganov/llama.cpp/releasesgguf-split
internlm2-math-plus-20b.Q8_0
)gguf-split --merge internlm2-math-plus-20b.Q8_0/internlm2-math-plus-20b.Q8_0-00001-of-XXXXX.gguf internlm2-math-plus-20b.Q8_0.gguf
gguf-split
to the first chunk of the split.Got a suggestion? Ping me @legraphista!
Base model
internlm/internlm2-math-plus-20b