Snowflake
/

snowflake-arctic-instruct

Text Generation

Mixture of Experts

Model card Files Files and versions Community

rajhans-snowflake commited on May 14

Commit

fd069fd

•

1 Parent(s): 314d82d

Update config.json

Files changed (1) hide show

config.json +1 -1

config.json CHANGED Viewed

@@ -22,7 +22,7 @@
   "moe_eval_capacity_factor": 1,
   "moe_layer_frequency": 1,
   "moe_min_capacity": 0,
-  "moe_token_dropping": false,
   "moe_train_capacity_factor": 1,
   "num_attention_heads": 56,
   "num_experts_per_tok": 2,

   "moe_eval_capacity_factor": 1,
   "moe_layer_frequency": 1,
   "moe_min_capacity": 0,
+  "moe_token_dropping": true,
   "moe_train_capacity_factor": 1,
   "num_attention_heads": 56,
   "num_experts_per_tok": 2,