|
--- |
|
language: |
|
- en |
|
license: llama2 |
|
model_name: OpenHathi-7B-Hi-v0.1-Base-gptq |
|
base_model: meta-llama/Llama-2-7b-chat-hf |
|
inference: false |
|
model_creator: SarvamAI |
|
model_type: llama |
|
pipeline_tag: text-generation |
|
prompt_template: '[INST] <<SYS>> |
|
|
|
You are a helpful, respectful and honest assistant. Always answer as helpfully as |
|
possible, while being safe. Your answers should not include any harmful, unethical, |
|
racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses |
|
are socially unbiased and positive in nature. If a question does not make any sense, |
|
or is not factually coherent, explain why instead of answering something not correct. |
|
If you don''t know the answer to a question, please don''t share false information. |
|
|
|
<</SYS>> |
|
|
|
{prompt}[/INST] |
|
|
|
' |
|
quantized_by: cmeraki |
|
--- |
|
|
|
# OpenHathi Base GPTQ |
|
- Model creator: [Sarvam AI](https://huggingface.co/sarvamai) |
|
- Original model: [sarvamai/OpenHathi-7B-Hi-v0.1-Base](https://huggingface.co/sarvamai/OpenHathi-7B-Hi-v0.1-Base/) |
|
|
|
<!-- description start --> |
|
## Description |
|
|
|
This repo contains GPTQ model files for [Sarvam's OpenHathi](https://huggingface.co/sarvamai/OpenHathi-7B-Hi-v0.1-Base/). |
|
|
|
Files are made using AutoGPTQ with following config. |
|
``` |
|
quantization_config : {"bits": 4, |
|
"group_size": 128, |
|
"damp_percent": 0.1, |
|
"desc_act": true, |
|
|
|
} |
|
``` |
|
|
|
We use a custom [dataset](cmeraki/wiki_en_hi) which has both Hindi and English wiki articles. We truncate to max_length=1024 and model may not perform well beyond that context size. |
|
|
|
<!-- description end --> |
|
|
|
<!-- prompt-template start --> |
|
## Prompt template |
|
|
|
This is a base model not tuned for any instructions. Feel free to use any format. Alpaca/Vicuna works fine. |
|
|
|
<!-- prompt-template end --> |
|
|
|
## Oobagooba |
|
Standard oobagooba works with exllama2 / autogptq loader |
|
|
|
## Using in code |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline |
|
|
|
model_name_or_path = "cmeraki/OpenHathi-7B-Hi-v0.1-Base-gptq" |
|
model = AutoModelForCausalLM.from_pretrained(model_name_or_path, |
|
device_map="auto", |
|
trust_remote_code=False, |
|
revision="main") |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True) |
|
|
|
prompt = "do aur do" |
|
prompt_template=f'''[INST] <<SYS>> |
|
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information. |
|
<</SYS>> |
|
{prompt}[/INST] |
|
|
|
''' |
|
|
|
print("\n\n*** Generate:") |
|
|
|
input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda() |
|
output = model.generate(inputs=input_ids, temperature=0.7, do_sample=True, top_p=0.95, top_k=40, max_new_tokens=512) |
|
print(tokenizer.decode(output[0])) |
|
|
|
# Inference can also be done using transformers' pipeline |
|
|
|
print("*** Pipeline:") |
|
pipe = pipeline( |
|
"text-generation", |
|
model=model, |
|
tokenizer=tokenizer, |
|
max_new_tokens=512, |
|
do_sample=True, |
|
temperature=0.7, |
|
top_p=0.95, |
|
top_k=40, |
|
repetition_penalty=1.1 |
|
) |
|
|
|
print(pipe(prompt_template)[0]['generated_text']) |
|
``` |
|
<!-- README_GPTQ.md-use-from-python end --> |
|
|
|
<!-- README_GPTQ.md-compatibility start --> |
|
|
