mobiuslabsgmbh/Mixtral-8x7B-Instruct-v0.1-hf-4bit_g64-HQQ

Mixtral-8x7B-Instruct-v0.1-hf-4bit_g64-HQQ

This is a version of the Mixtral-8x7B-Instruct-v0.1 model (https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) quantized to 4-bit via Half-Quadratic Quantization (HQQ).

Basic Usage

To run the model, install the HQQ library from https://github.com/mobiusml/hqq and use it as follows:

model_id = 'mobiuslabsgmbh/Mixtral-8x7B-Instruct-v0.1-hf-4bit_g64-HQQ'

#Load the model
from hqq.engine.hf import HQQModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
model     = HQQModelForCausalLM.from_quantized(model_id)

#Optional
from hqq.core.quantize import *
HQQLinear.set_backend(HQQBackend.PYTORCH_COMPILE) 

#Text Generation
prompt = "<s> [INST] How do I build a car? [/INST] "

inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
outputs = model.generate(**(inputs.to('cuda')), max_new_tokens=1000)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Output:

1. Gather the necessary materials and tools. To build a car, you will need a variety of materials, including a frame, engine, transmission, wheels, tires, brakes, and other components. You will also need a variety of tools, such as wrenches, sockets, pliers, and a welder.

2. Design your car. Before you start building, you will need to design your car. This includes deciding on the type of car you want to build, the size and shape of the car, and the specific components you will use. You can use computer-aided design (CAD) software to help with this process.

3. Build the frame. The frame is the skeleton of the car, and it is responsible for supporting the weight of the car and its occupants. To build the frame, you will need to cut and weld steel tubing or other materials together to create the desired shape.

4. Install the engine and transmission. The engine and transmission are the heart and lungs of the car, and they are responsible for providing the power to move the car. To install the engine and transmission, you will need to mount them in the frame and connect them to the drive shaft and wheels.

5. Install the wheels and tires. The wheels and tires are what allow the car to move, and they are an essential part of the car. To install the wheels and tires, you will need to mount them on the hubs and tighten the lug nuts.

6. Install the brakes. The brakes are what allow the car to stop, and they are an essential safety feature. To install the brakes, you will need to connect the brake lines to the master cylinder and bleed the brakes to remove any air.

7. Install the electrical system. The electrical system is what provides power to the lights, radio, and other accessories in the car. To install the electrical system, you will need to run wires and connect them to the battery and other components.

8. Install the interior. The interior of the car includes the seats, dashboard, and other features that make the car comfortable and functional. To install the interior, you will need to install the seats, dashboard, and other components.

9. Test the car. Before you start driving the car, you will need to test it to make sure everything is working properly. This includes checking the engine, transmission, brakes, electrical system, and other components.

10. Register and insure the car. Once the car is built and tested, you will need to register it with the appropriate government agency and purchase insurance to protect yourself and others.

Building a car is a complex and challenging project, but it can also be a rewarding and fulfilling experience. If you are interested in building a car, I would recommend doing some research and learning as much as you can about the process before you get started. Good luck!

---