Gunulhona/Gemma-Ko-Merge

merge

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the breadcrumbs_ties merge method using lemon07r/Gemma-2-Ataraxy-9B as a base.

Models Merged

The following models were included in the merge:

• wzhouad/gemma-2-9b-it-WPO-HB
• rtzr/ko-gemma-2-9b-it + ghost613/gemma9_on_korean_summary_events
• rtzr/ko-gemma-2-9b-it

Configuration

The following YAML configuration was used to produce this model:

slices:
  - sources:
      - model: lemon07r/Gemma-2-Ataraxy-9B
        layer_range: [0, 42]
        parameters:
            weight: 1
            density: 0.7
            gamma: 0.03
      - model: wzhouad/gemma-2-9b-it-WPO-HB
        layer_range: [0, 42]
        parameters:
            weight: 1
            density: 0.42
            gamma: 0.03
      - model: rtzr/ko-gemma-2-9b-it
        layer_range: [0, 42]
        parameters:
            weight: 1
            density: 0.42
            gamma: 0.03
      - model: rtzr/ko-gemma-2-9b-it+ghost613/gemma9_on_korean_summary_events # lora model loading
        layer_range: [0, 42]
        parameters:
            weight: 1
            density: 0.42
            gamma: 0.03
merge_method: breadcrumbs_ties
base_model: lemon07r/Gemma-2-Ataraxy-9B
dtype: bfloat16

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	25.94
IFEval (0-Shot)	64.16
BBH (3-Shot)	38.79
MATH Lvl 5 (4-Shot)	0.15
GPQA (0-shot)	11.41
MuSR (0-shot)	9.12
MMLU-PRO (5-shot)	31.99