zwt123home123
/

llava-1.5-7b-attncut-11

zwt123home123 commited on 18 days ago

Commit

e2e3643

•

1 Parent(s): af7dbc6

Update modeling_llama_attncut_window.py

Files changed (1) hide show

modeling_llama_attncut_window.py CHANGED Viewed

@@ -821,7 +821,7 @@ class LlamaDecoderLayer(nn.Module):
         offset=35
         #import pdb; pdb.set_trace()
-        if attention_mask.shape[2]!=1:
             attention_mask[:,:,offset:offset+576,offset:offset+576]=self.attention_mask_local
         # import pdb; pdb.set_trace()

         offset=35
         #import pdb; pdb.set_trace()
+        if attention_mask.shape[2]>576:
             attention_mask[:,:,offset:offset+576,offset:offset+576]=self.attention_mask_local
         # import pdb; pdb.set_trace()