--- license: apache-2.0 tags: - merge - mergekit - segmed/MedMistral-7B-v0.1 - Guilherme34/Samantha-v2 datasets: - medmcqa - cognitivecomputations/samantha-data base_model: - segmed/MedMistral-7B-v0.1 - Guilherme34/Samantha-v2 model-index: - name: Dr_Samantha_7b_mistral results: - task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics: - type: acc_norm value: 60.41 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Dr_Samantha_7b_mistral name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics: - type: acc_norm value: 83.65 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Dr_Samantha_7b_mistral name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics: - type: acc value: 63.14 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Dr_Samantha_7b_mistral name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics: - type: mc2 value: 41.37 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Dr_Samantha_7b_mistral name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics: - type: acc value: 75.45 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Dr_Samantha_7b_mistral name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics: - type: acc value: 31.46 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Dr_Samantha_7b_mistral name: Open LLM Leaderboard --- # Dr_Samantha_7b_mistral

SynthIQ

Dr. Samantha represents a blend of AI in healthcare, offering a balance between technical medical knowledge and the softer skills of communication and empathy, crucial for patient interaction and care. This model is a merge of the following models made with mergekit(https://github.com/cg123/mergekit): * [segmed/MedMistral-7B-v0.1](https://huggingface.co/segmed/MedMistral-7B-v0.1) * [Guilherme34/Samantha-v2](https://huggingface.co/Guilherme34/Samantha-v2) Has capabilities of a medical knowledge-focused model (trained on USMLE databases and doctor-patient interactions) with the philosophical, psychological, and relational understanding of the Samantha-7b model. As both a medical consultant and personal counselor, Dr.Samantha could effectively support both physical and mental wellbeing - important for whole-person care. ## 🧩 Configuration ```yaml slices: - sources: - model: segmed/MedMistral-7B-v0.1 layer_range: [0, 32] - model: Guilherme34/Samantha-v2 layer_range: [0, 32] merge_method: slerp base_model: OpenPipe/mistral-ft-optimized-1218 parameters: t: - filter: self_attn value: [0, 0.5, 0.3, 0.7, 1] - filter: mlp value: [1, 0.5, 0.7, 0.3, 0] - value: 0.5 dtype: bfloat16 ``` ## OpenLLM Evaluation Details about that can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_sethuiyer__Dr_Samantha_7b_mistral). Overall, with regards to the subjects related to medical domain, the model's performance is as follows: | Subject | Accuracy | |-----------------------|------------| | Clinical Knowledge | 70.57% | | Medical Genetics | 71.00% | | Human Aging | 69.06% | | Human Sexuality | 75.57% | | College Medicine | 63.01% | | Anatomy | 58.52% | | College Biology | 72.92% | | College Medicine | 63.01% | | High School Biology | 75.48% | | Professional Medicine | 65.44% | | Nutrition | 76.79% | | High School Psychology | 83.12% | | Professional Psychology | 65.35% | | Virology | 53.61% | | Average | **68.82%** | Dr. Samantha performs reasonably well on various medical-related subjects, averaging 68.82% overall in medical sciences, biology, and psychology, however it's important to note that medical diagnosis and treatment decisions often require a much higher level of accuracy, reliability, and context awareness. ## 💻 Usage ```python !pip install -qU transformers accelerate from transformers import AutoTokenizer import transformers import torch model = "sethuiyer/Dr_Samantha_7b_mistral" ask_samantha = ''' Symptoms: Dizziness, headache and nausea. What is the differnetial diagnosis? ''' messages = [{"role": "system", "content": '''You are Doctor Samantha, a virtual AI doctor known for your friendly and approachable demeanor, combined with a deep expertise in the medical field. You're here to provide professional, empathetic, and knowledgeable advice on health-related inquiries. You'll also provide differential diagnosis. If you're unsure about any information, Don't share false information.'''}, {"role": "user", "content": f"{ask_samantha}"}] tokenizer = AutoTokenizer.from_pretrained(model) prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) pipeline = transformers.pipeline( "text-generation", model=model, torch_dtype=torch.float16, device_map="auto", ) outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95) print(outputs[0]["generated_text"]) ``` ```text Dizziness, headache and nausea can be caused by a variety of conditions, including: Vertigo: A sensation of spinning or dizziness that can be caused by problems with the inner ear or brain. Migraine: A type of headache that can cause throbbing pain, sensitivity to light and sound, and nausea. Concussion: A type of traumatic brain injury that can cause dizziness, headache, and nausea. Dehydration: A lack of fluids in the body can cause dizziness, headache, and nausea. Low blood sugar: A drop in blood sugar levels can cause dizziness, headache, and nausea. It's important to consult with a healthcare professional for a proper diagnosis and treatment plan. ``` ## GGUF Files GGUF files are available at [s3nh/sethuiyer-Dr_Samantha_7b_mistral-GGUF](https://huggingface.co/s3nh/sethuiyer-Dr_Samantha_7b_mistral-GGUF), thanks to [s3nh](https://huggingface.co/s3nh) # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_sethuiyer__Dr_Samantha_7b_mistral) | Metric |Value| |---------------------------------|----:| |Avg. |59.25| |AI2 Reasoning Challenge (25-Shot)|60.41| |HellaSwag (10-Shot) |83.65| |MMLU (5-Shot) |63.14| |TruthfulQA (0-shot) |41.37| |Winogrande (5-shot) |75.45| |GSM8k (5-shot) |31.46|