Edit model card

Provided files

Name Quant method Size
colossal-llama-2-7b-base.Q2_K.gguf Q2_K 2.8 GB
colossal-llama-2-7b-base.Q3_K.gguf Q3_K 3.2 GB
colossal-llama-2-7b-base.Q3_K_L.gguf Q3_K_L 3.5 GB
colossal-llama-2-7b-base.Q3_K_S.gguf Q3_K_S 2.9 GB
colossal-llama-2-7b-base.Q4_0.gguf Q4_0 3.8 GB
colossal-llama-2-7b-base.Q4_1.gguf Q4_1 4.2 GB
colossal-llama-2-7b-base.Q4_K.gguf Q4_K 4.0 GB
colossal-llama-2-7b-base.Q4_K_S.gguf Q4_K_S 3.8 GB
colossal-llama-2-7b-base.Q5_0.gguf Q5_0 4.5 GB
colossal-llama-2-7b-base.Q5_1.gguf Q5_1 4.9 GB
colossal-llama-2-7b-base.Q5_K.gguf Q5_K 4.7 GB
colossal-llama-2-7b-base.Q5_K_S.gguf Q5_K_S 4.5 GB
colossal-llama-2-7b-base.Q6_K.gguf Q6_K 5.4 GB
colossal-llama-2-7b-base.Q8_0.gguf Q8_0 7.0 GB
colossal-llama-2-7b-base.gguf full 13 GB

Usage:

docker run --rm -it -p 8000:8000 -v /path/to/models:/models -e MODEL=/models/gguf-model-name.gguf shaowenchen/llama-cpp-python:0.2.6

Provided images

Name Quant method Compressed Size
shaowenchen/colossal-llama-2-7b-base-gguf:Q2_K Q2_K 3.24 GB
shaowenchen/colossal-llama-2-7b-base-gguf:Q3_K Q3_K 3.68 GB
shaowenchen/colossal-llama-2-7b-base-gguf:Q3_K_L Q3_K_L 3.98 GB
shaowenchen/colossal-llama-2-7b-base-gguf:Q3_K_S Q3_K_S 3.38 GB
shaowenchen/colossal-llama-2-7b-base-gguf:Q4_0 Q4_0 4.05 GB
shaowenchen/colossal-llama-2-7b-base-gguf:Q4_1 Q4_1 4.47 GB
shaowenchen/colossal-llama-2-7b-base-gguf:Q4_K Q4_K 4.39 GB
shaowenchen/colossal-llama-2-7b-base-gguf:Q4_K_S Q4_K_S 4.18 GB
shaowenchen/colossal-llama-2-7b-base-gguf:Q5_0 Q5_0 4.99 GB
shaowenchen/colossal-llama-2-7b-base-gguf:Q5_1 Q5_1 5.35 GB
shaowenchen/colossal-llama-2-7b-base-gguf:Q5_K Q5_K 5.12 GB
shaowenchen/colossal-llama-2-7b-base-gguf:Q5_K_S Q5_K_S 5 GB
shaowenchen/colossal-llama-2-7b-base-gguf:Q6_K Q6_K 5.82 GB
shaowenchen/colossal-llama-2-7b-base-gguf:Q8_0 Q8_0 7.18 GB
shaowenchen/colossal-llama-2-7b-base-gguf:full full 10.49 GB

Usage:

docker run --rm -p 8000:8000 shaowenchen/colossal-llama-2-7b-base-gguf:Q2_K

and you can view http://localhost:8000/docs to see the swagger UI.

Downloads last month
446
GGUF
Model size
7.04B params
Architecture
llama

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Inference Examples
Inference API (serverless) has been turned off for this model.

Space using shaowenchen/colossal-llama-2-7b-base-gguf 1