matthewchung74/MedMistral-7B-do-not-use

Model Card for MedMistral-7B

Model Details

Model Description

MedMistral-7B is a Mistral fine tune 180993 samples from the medmcqa dataset.

• Developed by: segmed.ai
• Model type: QLoRA Fine tune Mistral 7B
• License: Apache 2.0
• Finetuned from model : Mistral-7B-v0.1

Model Sources [optional]

• Repository: https://github.com/segmed/med_mistral

Uses

This model is to demonstrate medical domain knowledge, but is not intended for medical advise.

Direct Use

Prompts used in training followed this format:

You are a helpful medical assistant. Your task is to answer the following question one of the options and explain why.
### Question:  Turban epiglottitis is a clinical finding in - [0] Tubercular laryngitis [1] Tubercular pharyngitis [2] Polypoid degeneration of vocal cord [3] Subglottic hemangioma
    
### Answer: 0
### Explanation: Ans. is 4a' i.e.. Tubercular laryngitis Laryngeal examination in TB lary ngitiso Hyperaemia of the vocal cord in its whole extent or confined to posterior part with impairment of adduction is the first sign.o Swelling in the interarytenoid region giving a mammilated appearance,o Ulceration of vocal cord giving mouse-nibbled appearance.o Superficial ragged ulceration on the arytenoids and interarytenoid region,o Granulation tissue in interarvtenoid region or vocal process of arytenoid,o Pseudoedema of the epiglottis 'Turban epiglottis",o Swelling of ventricular bands and aryepiglottic folds,o Marked pallor of surrounding mucosa.

And for inference, you would need to remove the answer.

You are a helpful medical assistant. Your task is to answer the following question one of the options and explain why.
### Question:  Turban epiglottitis is a clinical finding in - [0] Tubercular laryngitis [1] Tubercular pharyngitis [2] Polypoid degeneration of vocal cord [3] Subglottic hemangioma
    
### Answer: 

Downstream Use

This model could be further fine tuned on your specific medical dataset.

Bias, Risks, and Limitations

This model is not intended for medical use.

How to Get Started with the Model

Download the model and call generate_tokens using the prompt format defined above.

def generate_tokens(m, prompt, max_new_tokens=32):
    model_input = tokenizer(prompt, return_tensors="pt").to("cuda")
    m.eval()
    with torch.no_grad():
        return tokenizer.decode(m.generate(**model_input, max_new_tokens=max_new_tokens, do_sample=True, top_k=0, num_return_sequences=1, temperature=0.1, eos_token_id=tokenizer.eos_token_id)[0].cuda())

These are the hyper-parameters which yielded the best results when experimenting.

Training Details

Training Data

180k samples from the medmcqa were used for training. 914 samples were reserved for test and eval.

A small number of samples over 512 tokens were removed to simplify training and to keep the max tokens size small.

Training Procedure

Training

Training Hyperparameters

• Training regime: This used the standard qLoRA peft parameters as defined by

peft_config = LoraConfig(
        r=16,
        lora_alpha=16,
        lora_dropout=0.05,
        bias="none",
        task_type="CAUSAL_LM",
        target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
        "lm_head",
    ]
    )

Speeds, Sizes, Times [optional]

[More Information Needed]

Evaluation

Evaluation was performed on the holdout of 914 samples from the medmcqa dataset. Given the multiple choice nature of the data, the F1 was computed and explanations thrown away.

F1 medmcqa : 66%

Evaluation was also performed on the pubmed_qa labeled dataset. Since Yes/No answers were provided, an F1 was calculated. I waant to point out this evaluation is on a completely different dataset with different prompt format.

F1 pubmed_qa: