Qwen

/

Qwen2-57B-A14B-Instruct

like 76

Follow

Qwen 1,715

Something tells me A14B stands for active weight of the model.

That's true. A14B means out of 57B parameters, 14B are activated each time.

• The general introduction of the architecture is at https://qwenlm.github.io/blog/qwen-moe/#architecture.
• The hyperparameters of the architecture of this model are at https://huggingface.co/Qwen/Qwen2-57B-A14B-Instruct/blob/main/config.json.
• The implemenation is at https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen2_moe/modeling_qwen2_moe.py.