|
--- |
|
license: other |
|
license_name: gemma-terms-of-use |
|
license_link: https://ai.google.dev/gemma/terms |
|
tags: |
|
- gemma |
|
- gguf |
|
--- |
|
|
|
# Gemma 7B Instruct GGUF |
|
|
|
Contains Q4 & Q8 quantized GGUFs for [google/gemma](https://huggingface.co/collections/google/gemma-release-65d5efbccdbb8c4202ec078b) |
|
|
|
## Perf |
|
|
|
| Variant | Device | Perf | |
|
| - | - | - | |
|
| Q4 | RTX 2070S | 22 tok/s | |
|
| | M1 Pro 10-core GPU | 28 tok/s | |
|
| Q8 | RTX 2070S | 7 tok/s (could only offload 23/29 layers to GPU) | |
|
| | M1 Pro 10-core GPU | 17 tok/s | |
