HPAI-BSC/Llama3-Aloe-8B-Alpha

Aloe: A New Family of Healthcare LLMs

Aloe is a new family of healthcare LLMs that is highly competitive with all previous open models of its range and reaches state-of-the-art results at its size by using model merging and advanced prompting strategies. Aloe scores high in metrics measuring ethics and factuality, thanks to a combined red teaming and alignment effort. Complete training details, model merging configurations, and all training data (including synthetically generated data) will be shared. Additionally, the prompting repository used in this work to produce state-of-the-art results during inference will also be shared. Aloe comes with a healthcare-specific risk assessment to contribute to the safe use and deployment of such systems.

Model Details

• Developed by: HPAI
• Model type: Causal decoder-only transformer language model
• Language(s) (NLP): English (mainly)
• License: This model is based on Meta Llama 3 8B and is governed by the Meta Llama 3 License. All our modifications are available with a CC BY-NC 4.0 license.
• Finetuned from model : meta-llama/Meta-Llama-3-8B

• Repository: https://github.com/HPAI-BSC/prompt_engine (more coming soon)
• Paper: https://arxiv.org/abs/2405.01886 (more coming soon)

Model Performance

Aloe has been tested on the most popular healthcare QA datasets, with and without medprompting inference technique. Results show competitive performance, even against bigger models.

Results using advanced prompting methods (aka Medprompt) are achieved through a repo made public with this work.

Uses

Direct Use

We encourage the use of Aloe for research purposes, as a stepping stone to build better foundational models for healthcare.

Out-of-Scope Use

These models are not to be used for clinical practice, medical diagnosis, or any other form of direct or indirect healthcare advice. Models are prone to error and can produce toxic content. The use of Aloe models for activities harmful for individuals, such as spam, fraud, or impersonation, is prohibited.

Bias, Risks, and Limitations

We consider three risk cases:

• Healthcare professional impersonation, a fraudulent behaviour which currently generates billions of dollars in profit. A model such as Aloe could be used to increase the efficacy of such deceiving activities, making them more widespread. The main preventive actions are public literacy on the unreliability of digitised information and the importance of medical registration, and legislation enforcing AI-generated content disclaimers.
• Medical decision-making without professional supervision. While this is already an issue in modern societies (eg self-medication) a model such as Aloe, capable of producing high-quality conversational data, can facilitate self-delusion, particularly in the presence of sycophancy. By producing tailored responses, it can also be used to generate actionable answers. Public literacy on the dangers of self-diagnosis is one of the main defences, together with the introduction of disclaimers and warnings on the models' outputs.
• Access to information on dangerous substances or procedures. While the literature on sensitive content can already be found on different sources (eg libraries, internet, dark web), LLMs can centralize such access, making it nearly impossible to control the flow of such information. Model alignment can help in that regard, but so far the effects remain insufficient, as jailbreaking methods still overcome it.

Table below shows the performance of Aloe at several AI safety tasks:

Recommendations

We avoid the use of all personal data in our training. Model safety cannot be guaranteed. Aloe can produce toxic content under the appropriate prompts. For these reasons, minors should not be left alone to interact with Aloe without supervision.

How to Get Started with the Model

Use the code below to get started with the model. You can run conversational inference using the Transformers pipeline abstraction, or by leveraging the Auto classes with the generate() function. Let's see examples of both.

Transformers pipeline

import transformers
import torch

model_id = "HPAI-BSC/Llama3-Aloe-8B-Alpha"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are an expert medical assistant named Aloe, developed by the High Performance Artificial Intelligence Group at Barcelona Supercomputing Center(BSC). You are to be a helpful, respectful, and honest assistant."},
    {"role": "user", "content": "Hello."},
]

prompt = pipeline.tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
)

terminators = [
    pipeline.tokenizer.eos_token_id,
    pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

outputs = pipeline(
    prompt,
    max_new_tokens=256,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
)
print(outputs[0]["generated_text"][len(prompt):])

Transformers AutoModelForCausalLM

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "HPAI-BSC/Llama3-Aloe-8B-Alpha"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are an expert medical assistant named Aloe, developed by the High Performance Artificial Intelligence Group at Barcelona Supercomputing Center(BSC). You are to be a helpful, respectful, and honest assistant."},
    {"role": "user", "content": "Hello"},
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
)
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))

Training Details

Supervised fine-tuning on top of Llama 3 8B using medical and general domain datasets, model merging using DARE-TIES process, two-stage DPO process for human preference alignment. More details coming soon.

Training Data

• Medical domain datasets, including synthetic data generated using Mixtral-8x7B and Genstruct
  • HPAI-BSC/pubmedqa-cot
  • HPAI-BSC/medqa-cot
  • HPAI-BSC/medmcqa-cot
• LDJnr/Capybara
• hkust-nlp/deita-10k-v0
• jondurbin/airoboros-3.2
• argilla/dpo-mix-7k
• nvidia/HelpSteer
• Custom preference data with adversarial prompts generated from Anthropic Harmless, Chen et al., and original prompts

Evaluation

Testing Data, Factors & Metrics

Testing Data

• MedQA (USMLE)
• MedMCQA
• PubMedQA
• MMLU-Medical
• MedQA-4-Option
• CareQA

Metrics

• Accuracy: suite the evaluation of multiple-choice question-answering tasks.

Results

Summary

To compare Aloe with the most competitive open models (both general purpose and healthcare-specific) we use popular healthcare datasets (PubMedQA, MedMCQA, MedQA and MMLU for six medical tasks only), together with the new and highly reliable CareQA. We produce the standard MultiMedQA score for reference, by computing the weighted average accuracy on all scores except CareQA. Additionally, we calculate the arithmetic mean across all datasets. The Medical MMLU is calculated by averaging the six medical subtasks: Anatomy, Clinical knowledge, College Biology, College medicine, Medical genetics, and Professional medicine.

Benchmark results indicate the training conducted on Aloe has boosted its performance above Llama3-8B-Instruct. Llama3-Aloe-8B-Alpha outperforms larger models like Meditron 70B, and is close to larger base models, like Yi-34. For the former, this gain is consistent even when using SC-CoT, using their best-reported variant. All these results make Llama3-Aloe-8B-Alpha the best healthcare LLM of its size.

With the help of prompting techniques the performance of Llama3-Aloe-8B-Alpha is significantly improved. Medprompting in particular provides a 7% increase in reported accuracy, after which Llama3-Aloe-8B-Alpha only lags behind the ten times bigger Llama-3-70B-Instruct. This improvement is mostly consistent across medical fields. Llama3-Aloe-8B-Alpha with medprompting beats the performance of Meditron 70B with their self reported 20 shot SC-CoT in MMLU med and is slightly worse in the other benchmarks.

Environmental Impact

• Hardware Type: 4xH100
• Hours used: 7,000
• Hardware Provider: Barcelona Supercomputing Center
• Compute Region: Spain
• Carbon Emitted: 439.25kg

Model Card Authors

Ashwin Kumar Gururajan

Model Card Contact

mailto:[email protected]

Citations

If you use this repository in a published work, please cite the following papers as source:

@misc{gururajan2024aloe,
      title={Aloe: A Family of Fine-tuned Open Healthcare LLMs}, 
      author={Ashwin Kumar Gururajan and Enrique Lopez-Cuena and Jordi Bayarri-Planas and Adrian Tormos and Daniel Hinjos and Pablo Bernabeu-Perez and Anna Arias-Duart and Pablo Agustin Martin-Torres and Lucia Urcelay-Ganzabal and Marta Gonzalez-Mallo and Sergio Alvarez-Napagao and Eduard Ayguadé-Parra and Ulises Cortés Dario Garcia-Gasulla},
      year={2024},
      eprint={2405.01886},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}