|
|
|
--- |
|
license: other |
|
language: |
|
- en |
|
base_model: |
|
- mistralai/Mistral-Large-Instruct-2407 |
|
- NeverSleep/Lumimaid-v0.2-123B |
|
--- |
|
|
|
# SmartMaid-123b |
|
|
|
This **experimental model** is a hybrid creation combining aspects of [Mistral-Large-Instruct-2407](https://huggingface.co/mistralai/Mistral-Large-Instruct-2407) and [Lumimaid-v0.2-123B](https://huggingface.co/NeverSleep/Lumimaid-v0.2-123B) using LoRA (Low-Rank Adaptation) on the `mlp.down_proj` module. |
|
|
|
## Model Details |
|
|
|
- **Base Model**: Mistral-Large-Instruct-2407 |
|
- **Influence Model**: Lumimaid-v0.2-123B |
|
- **Method**: LoRA extraction from Lumimaid and targetted application to Mistral-Large |
|
- **LoRA Configuration**: |
|
- Rank: 32 |
|
- Alpha: 64 |
|
- Target Module: `mlp.down_proj` |
|
|
|
## Process |
|
|
|
1. A LoRA was extracted from Lumimaid-v0.2-123B |
|
2. This LoRA was then applied to a fresh instance of Mistral-Large-Instruct-2407 targetting only the mlp.down_proj modules. |
|
3. The resulting model was merged to create this standalone version. |
|
|
|
## Purpose |
|
|
|
The aim of this model is to incorporate the enhanced prose qualities of Lumimaid-v0.2-123B while retaining the core intelligence and capabilities of Mistral-Large. |
|
By applying the LoRA to the `mlp.down_proj` module, we sought to influence the model's language generation style without significantly altering its underlying knowledge and reasoning abilities. |
|
|
|
## Prompt Template |
|
``` |
|
<s>[INST] {input} [/INST] {output}</s> |
|
``` |
|
|
