metadata

license: other
license_name: gemma-terms-of-use
license_link: https://ai.google.dev/gemma/terms
tags:
  - gemma
  - gguf

Gemma 7B Instruct GGUF

Contains Q4 & Q8 quantized GGUFs for google/gemma

Perf

Variant	Device	Perf
Q4	RTX 2070S	22 tok/s
	M1 Pro 10-core GPU	28 tok/s
Q8	RTX 2070S	7 tok/s (could only offload 23/29 layers to GPU)
	M1 Pro 10-core GPU	17 tok/s