cmeraki apurvagup commited on
Commit
1303a43
1 Parent(s): e1bf3a9

Update README.md (#4)

Browse files

- Update README.md (8defed105212809bebac6ddbfa05797a4ff5196b)


Co-authored-by: Apurva Kumar Gupta <[email protected]>

Files changed (1) hide show
  1. README.md +10 -40
README.md CHANGED
@@ -61,46 +61,16 @@ Standard oobagooba works with exllama2 / autogptq loader
61
  ## Using in code
62
 
63
  ```python
64
- from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
65
-
66
- model_name_or_path = "cmeraki/OpenHathi-7B-Hi-v0.1-Base-gptq"
67
- model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
68
- device_map="auto",
69
- trust_remote_code=False,
70
- revision="main")
71
-
72
- tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
73
-
74
- prompt = "do aur do"
75
- prompt_template=f'''[INST] <<SYS>>
76
- You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
77
- <</SYS>>
78
- {prompt}[/INST]
79
-
80
- '''
81
-
82
- print("\n\n*** Generate:")
83
-
84
- input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
85
- output = model.generate(inputs=input_ids, temperature=0.7, do_sample=True, top_p=0.95, top_k=40, max_new_tokens=512)
86
- print(tokenizer.decode(output[0]))
87
-
88
- # Inference can also be done using transformers' pipeline
89
-
90
- print("*** Pipeline:")
91
- pipe = pipeline(
92
- "text-generation",
93
- model=model,
94
- tokenizer=tokenizer,
95
- max_new_tokens=512,
96
- do_sample=True,
97
- temperature=0.7,
98
- top_p=0.95,
99
- top_k=40,
100
- repetition_penalty=1.1
101
- )
102
-
103
- print(pipe(prompt_template)[0]['generated_text'])
104
  ```
105
  <!-- README_GPTQ.md-use-from-python end -->
106
 
 
61
  ## Using in code
62
 
63
  ```python
64
+ from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
65
+ from transformers import AutoTokenizer
66
+
67
+ model_dir = 'cmeraki/OpenHathi-7B-Hi-v0.1-Base-gptq'
68
+
69
+ model = AutoGPTQForCausalLM.from_quantized(model_dir, device="cuda:0")
70
+ tokenizer = AutoTokenizer.from_pretrained(model_dir, fast=True)
71
+ tokens = tokenizer("do aur do", return_tensors="pt").to(model.device)
72
+
73
+ print(tokenizer.decode(model.generate(**tokens, max_length=1024)[0]))
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
74
  ```
75
  <!-- README_GPTQ.md-use-from-python end -->
76