Update README.md

README.md

@@ -59,7 +59,44 @@ pip install "deepspeed>=0.14.2"
 
 ### Inference
 
-The Arctic github page has several code snippets and examples around running inference:
+Due to the model size we recommend using a single 8xH100 instance from your
+favorite cloud provider such as: AWS [p5.48xlarge](https://aws.amazon.com/ec2/instance-types/p5/), 
+Azure [ND96isr_H100_v5](https://learn.microsoft.com/en-us/azure/virtual-machines/nd-h100-v5-series), etc.
+
+In this example we are using FP8 quantization provided by DeepSpeed in the backend, we can also use FP6 
+quantization by specifying `q_bits=6` in the `ArcticQuantizationConfig` config. The `"150GiB"` setting 
+for max_memory is required until we can get DeepSpeed's FP quantization supported natively as a [HFQuantizer](https://huggingface.co/docs/transformers/main/en/hf_quantizer#build-a-new-hfquantizer-class) which we 
+are actively working on.
+
+```python
+import os
+# enable hf_transfer for faster ckpt download
+os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from transformers.models.arctic.configuration_arctic import ArcticQuantizationConfig
+
+tokenizer = AutoTokenizer.from_pretrained("Snowflake/snowflake-arctic-instruct")
+
+quant_config = ArcticQuantizationConfig(q_bits=8)
+
+model = AutoModelForCausalLM.from_pretrained(
+    "Snowflake/snowflake-arctic-instruct",
+    low_cpu_mem_usage=True,
+    device_map="auto",
+    ds_quantization_config=quant_config,
+    max_memory={i: "150GiB" for i in range(8)},
+    torch_dtype=torch.bfloat16)
+
+messages = [{"role": "user", "content": "What is 1 + 1 "}]
+input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to("cuda")
+
+outputs = model.generate(input_ids=input_ids, max_new_tokens=20)
+print(tokenizer.decode(outputs[0]))
+```
+
+The Arctic github page has additional code snippets and examples around running inference:
 
 * Example with pure-HF: https://github.com/Snowflake-Labs/snowflake-arctic/blob/main/inference
 * Tutorial using vLLM: https://github.com/Snowflake-Labs/snowflake-arctic/tree/main/inference/vllm