Hugging Face's logo

cmeraki
/
OpenHathi-7B-Hi-v0.1-Base-gptq

Text Generation
Transformers
Safetensors
English
llama
text-generation-inference
4-bit precision
Model card Files Community
5
OpenHathi-7B-Hi-v0.1-Base-gptq / README.md
apurvagup's picture
apurvagup
Update README.md
8defed1
preview code
|
raw
history blame
2.39 kB
metadata
language:
  - en
license: llama2
model_name: OpenHathi-7B-Hi-v0.1-Base-gptq
base_model: meta-llama/Llama-2-7b-chat-hf
inference: false
model_creator: SarvamAI
model_type: llama
pipeline_tag: text-generation
prompt_template: >
  [INST] <<SYS>>

  You are a helpful, respectful and honest assistant. Always answer as helpfully
  as possible, while being safe.  Your answers should not include any harmful,
  unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure
  that your responses are socially unbiased and positive in nature. If a
  question does not make any sense, or is not factually coherent, explain why
  instead of answering something not correct. If you don't know the answer to a
  question, please don't share false information.

  <</SYS>>

  {prompt}[/INST]
quantized_by: cmeraki

OpenHathi Base GPTQ

Description

This repo contains GPTQ model files for Sarvam's OpenHathi.

Files are made using AutoGPTQ with following config. 

quantization_config : {"bits": 4,
  "group_size": 128,
  "damp_percent": 0.1,
  "desc_act": true,

}

We use a custom dataset which has both Hindi and English wiki articles. We truncate to max_length=1024 and model may not perform well beyond that context size.

Prompt template

This is a base model not tuned for any instructions. Feel free to use any format. Alpaca/Vicuna works fine.

Oobagooba

Standard oobagooba works with exllama2 / autogptq loader

Using in code 

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer

model_dir = 'cmeraki/OpenHathi-7B-Hi-v0.1-Base-gptq'

model = AutoGPTQForCausalLM.from_quantized(model_dir, device="cuda:0")
tokenizer = AutoTokenizer.from_pretrained(model_dir, fast=True)
tokens = tokenizer("do aur do", return_tensors="pt").to(model.device)

print(tokenizer.decode(model.generate(**tokens, max_length=1024)[0]))