Trying to quantize granite-3B
#1
by
Interpause
- opened
FYI if you decide to quantize granite-3B. It is missing from the model config, but the paper mentioned it was trained with RoPE, although it never mentioned what scale. From the ppl scores, seem it was indeed trained with RoPE after all.
The 3B and 8B variants are just Llama models, architecturally. Only 20B and up are GPTBigCode. ExLlama should automatically detect it based on the architecture string in config.json.