databricks

/

dbrx-instruct

like 1.1k

Follow

Databricks 1,092

Hello @elysiia , we generally recommend fine-tuning the base model. There are many excellent finetuning libraries out there -- e.g. axolotl by @winglian , the huggingface trainer itself, and others. We have an open source library as well that include some finetuning examples (e.g. see https://github.com/databricks/DBRX?tab=readme-ov-file#finetune, which has configs for both full parameter finetuning and LoRa-based finetuning of DBRX). These examples will include details on how to format the dataset.

As @hanlintang said, fine-tuning the base model is the best option. You can also use Hugging Face's AutoTrain Feature to easily fine-tune large language models (LLMs) on your own Question Answering Dataset.

I also recommend reviewing this article (https://exnrt.com/blog/ai/finetune-llms-with-huggingface/) to understand all the theoretical and technical guidance related to fine-tuning.

Choosing the Right DBRX Model for Fine-Tuning

Here's how to decide between the base and instruction-tuned DBRX model for your task:

Opt for the base model (databricks/dbrx-base) if:

• Your task requires learning general semantic representations from scratch specific to your domain. This is because the base model is trained on a broader dataset and can learn these general representations more effectively.

Choose the instruction-tuned model (databricks/dbrx-instruct) if:

• Your task heavily relies on following instructions within the context to generate responses. The instruction-tuned model is pre-trained on a dataset of instruction following tasks, giving it a head start in this area.

Prompt Template for Training Data

Here's a template you can use to format your training data for fine-tuning the DBRX model:

data_example = [
    {
        "instruction": "Summarize the following article",
        "context": "This is a long article about...",
        "question": "What is the main idea of the article?",
        "response": "The main idea of the article is..."
    },
    # Add more examples...
]

Each data point in your list should be a dictionary with the following keys:

• instruction: The instruction you want the model to follow.
• context: The context to which the instruction applies.
• question: The question you want the model to answer based on the instruction and context.
• response: The desired answer to the question.

By adding more examples with various instructions, contexts, questions, and responses, you can enrich your training data and improve the fine-tuned model's performance on your specific task.

Morning folks,
I'm trying to figure out the PEFT target modules for LoRA, the link from https://huggingface.co/hanlintang is broken, ideas?

Sorry @ChristianPalaArtificialy let me fix my link.

See https://github.com/databricks/DBRX?tab=readme-ov-file#finetune for the details.

Can you share the gpu memory requirements for finetuning the dbrx with 32k context?

@shaohuay
according to the configs you need 64x80GB GPUs for the full parameter finetune and ~16x80GB GPUs for lora finetuning

@hanlintang what is the smallest number of GPU HBM needed to finetune (full parameter) DBRX. I wonder if I could finetune it in 8 bits.
Some context, I'm using llm-foundery and try to leverage the composer library on a 8XH100 GPU cluster.

Thanks