metadata
language:
- en
license: apache-2.0
library_name: transformers
tags:
- mergekit
- merge
- lazymergekit
base_model:
- Qwen/Qwen2.5-32B-Instruct
license_name: tongyi-qianwen
license_link: https://huggingface.co/Qwen/Qwen2-72B-Instruct/blob/main/LICENSE
pipeline_tag: text-generation
model-index:
- name: BigQwen2.5-Echo-47B-Instruct
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: IFEval (0-Shot)
type: HuggingFaceH4/ifeval
args:
num_few_shot: 0
metrics:
- type: inst_level_strict_acc and prompt_level_strict_acc
value: 73.57
name: strict accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=mlabonne/BigQwen2.5-Echo-47B-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: BBH (3-Shot)
type: BBH
args:
num_few_shot: 3
metrics:
- type: acc_norm
value: 44.52
name: normalized accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=mlabonne/BigQwen2.5-Echo-47B-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MATH Lvl 5 (4-Shot)
type: hendrycks/competition_math
args:
num_few_shot: 4
metrics:
- type: exact_match
value: 3.47
name: exact match
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=mlabonne/BigQwen2.5-Echo-47B-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GPQA (0-shot)
type: Idavidrein/gpqa
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 8.61
name: acc_norm
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=mlabonne/BigQwen2.5-Echo-47B-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MuSR (0-shot)
type: TAUR-Lab/MuSR
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 10.19
name: acc_norm
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=mlabonne/BigQwen2.5-Echo-47B-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU-PRO (5-shot)
type: TIGER-Lab/MMLU-Pro
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 41.49
name: accuracy
source:
url: >-
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=mlabonne/BigQwen2.5-Echo-47B-Instruct
name: Open LLM Leaderboard
BigQwen2.5-Echo-47B-Instruct
BigQwen2.5-Echo-47B-Instruct is a Qwen/Qwen2-32B-Instruct self-merge made with MergeKit.
🔉 Echo Merge
I've tried a more gradual approach with a distributed repetition pattern. Instead of replicating blocks of 8 or more layers, I'm replicating individual layers in these blocks:
- First 8 layers: No replication
- Next 8 layers: Replicate 2 layers (first one, middle one)
- Next 8 layers: Replicate 4 layers (1st, 3rd, 5th, 7th)
- Next 8 layers: Replicate 8 layers (all of them)
- Next 8 layers: Replicate 4 layers (1st, 3rd, 5th, 7th)
- Next 8 layers: Replicate 2 layers (first one, middle one)
- First 8 layers: No replication
I used this string to visualize it, where 0 are original layers and 1 duplicated ones (the order doesn't matter):
00000000 1000010000 100100100100 1010101010101010 1010101010101010 100100100100 1000010000 00000000
The main idea is that the input/output difference of middle layers is quite small, so replicating a middle layer has a small impact on the output. The additional layers are designed to increase the model's capacity without breaking the information flow, which often creates "insane" self-merges.
🏆 Evaluation
| Metric | BigQwen2.5-Echo-47B-Instruct | BigQwen2.5-52B-Instruct | Qwen2.5-32B-Instruct |
|---|---|---|---|
| Avg. | 30.31 | 37.42 | 36.17 |
| IFEval (0-Shot) | 73.57 | 79.29 | 83.46 |
| BBH (3-Shot) | 44.52 | 59.81 | 56.49 |
| MATH Lvl 5 (4-Shot) | 3.47 | 17.82 | 0 |
| GPQA (0-shot) | 8.61 | 6.94 | 11.74 |
| MuSR (0-shot) | 10.19 | 10.45 | 13.5 |
| MMLU-PRO (5-shot) | 41.49 | 50.22 | 51.85 |
🧩 Configuration
The following YAML configuration was used to produce this model:
slices:
# First 8 layers: No replication
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [0, 8]
# Next 8 layers: Replicate 2 layers
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [8, 9]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [8, 9]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [9, 13]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [13, 14]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [13, 14]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [14, 16]
# Next 8 layers: Replicate 4 layers
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [16, 18]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [17, 19]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [18, 20]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [19, 21]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [20, 22]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [21, 23]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [22, 24]
# Next 8 layers: Replicate all 8 layers
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [24, 25]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [24, 26]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [25, 27]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [26, 28]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [27, 29]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [28, 30]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [29, 31]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [30, 32]
# Middle 8 layers: Replicate all 8 layers
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [32, 33]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [32, 34]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [33, 35]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [34, 36]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [35, 37]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [36, 38]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [37, 39]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [38, 40]
# Next 8 layers: Replicate 4 layers
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [40, 42]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [41, 43]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [42, 44]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [43, 45]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [44, 46]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [45, 47]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [46, 48]
# Next 8 layers: Replicate 2 layers
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [48, 49]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [48, 49]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [49, 53]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [53, 54]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [53, 54]
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [54, 56]
# Last 8 layers: No replication
- sources:
- model: Qwen/Qwen2.5-32B-Instruct
layer_range: [56, 64]
merge_method: passthrough
dtype: bfloat16
💻 Usage
!pip install -qU transformers accelerate
from transformers import AutoTokenizer
import transformers
import torch
model = "mlabonne/BigQwen2.5-Echo-47B-Instruct"
messages = [{"role": "user", "content": "What is a large language model?"}]
tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
"text-generation",
model=model,
torch_dtype=torch.float16,
device_map="auto",
)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])