max_window_layers is 70?
The config.json is just copied from other place, and even did not modify?
"max_window_layers": 70,
"num_hidden_layers": 64,
The total layer is 64, will it take effect to set the max_window_layers to 70?
Hi, use_sliding_window, sliding_window, and max_window_layers control how silding window attention (SWA) is applied. If use_sliding_window, and layer index is not greater than max_window_layers, SWA will be used with a context length of sliding_window. However, we don't recommend enabling SWA for Qwen2.5 (or for Qwen2) as they were not trained with SWA in mind, and we have no idea how SWA will perform. You can still enable SWA and max_window_layers=70 is valid, which means all layers will use SWA, but you should adjust the value based on your own tests.
Hi,
use_sliding_window,sliding_window, andmax_window_layerscontrol how silding window attention (SWA) is applied. If use_sliding_window, and layer index is not greater than max_window_layers, SWA will be used with a context length of sliding_window. However, we don't recommend enabling SWA for Qwen2.5 (or for Qwen2) as they were not trained with SWA in mind, and we have no idea how SWA will perform. You can still enable SWA and max_window_layers=70 is valid, which means all layers will use SWA, but you should adjust the value based on your own tests.
got it.