README.md

---
pipeline_tag: text-generation
inference: false
license: apache-2.0
library_name: transformers
tags:
- language
- granite-3.0
- llama-cpp
- gguf-my-repo
base_model: ibm-granite/granite-3.0-8b-instruct

---

# eformat/granite-3.0-8b-instruct-Q4_K_M-GGUF

Not all tools (vllm, llama.cpp) seem to support the new model config params it seems (25/10/2024).

```json
# config.json
"model_type": "granite"
"architectures": [
  "GraniteForCausalLM"
]
```

This gguf conversion done using old ones

```json
# config.json
"model_type": "llama"
"architectures": [
  "LlamaForCausalLM"
]
```

This gguf loads OK - tested using:

```bash
# llama.cpp
./llama-server --verbose --gpu-layers 99999 --parallel 2 --ctx-size 4096 -m ~/instructlab/models/granite-3.0-8b-instruct-Q4_K_M.gguf
```

```bash
# vllm
vllm serve ~/instructlab/models/granite-3.0-8b-instruct-Q4_K_M.gguf
```