|
--- |
|
license: apache-2.0 |
|
base_model: |
|
- rhymes-ai/Aria-sequential_mlp |
|
- rhymes-ai/Aria |
|
pipeline_tag: image-text-to-text |
|
--- |
|
# Aria-sequential_mlp-FP8-dynamic |
|
Warning: There is no inference code for transformers/vLLM yet! |
|
|
|
FP8-Dynamic quantization from [Aria-sequential_mlp](https://huggingface.co/rhymes-ai/Aria-sequential_mlp) made with [LLM Compressor](https://github.com/vllm-project/llm-compressor). |
|
|
|
Generated with the following code: |
|
|
|
```python |
|
from transformers import AutoProcessor, AutoModelForCausalLM |
|
from llmcompressor.modifiers.quantization import QuantizationModifier |
|
from llmcompressor.transformers import SparseAutoModelForCausalLM, oneshot |
|
|
|
model_name = "rhymes-ai/Aria-sequential_mlp" |
|
|
|
model = SparseAutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype="auto", trust_remote_code=True) |
|
processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True) |
|
|
|
recipe = QuantizationModifier( |
|
targets="Linear", |
|
scheme="FP8_DYNAMIC", |
|
ignore=["re:.*lm_head", "re:multi_modal_projector.*", "re:vision_tower.*"], |
|
) |
|
|
|
folder = model_name.split("/")[1] + "-FP8-Dynamic" |
|
oneshot(model=model, recipe=recipe, output_dir=folder) |
|
processor.save_pretrained(folder) |
|
``` |
