Does it support Flash Attention-2?
#12
by
deshwalmahesh
- opened
More importantly, if it does, is it bug free unlike Phi-2 as that one is still having lots of issues in Flash-Attention-2 in either loading time or results are bad with Flash-Attn-2
Yes, it does support and was tested with Flash-Attention-2.
Thanks a lot
@caiom
. Out of curiosity, will it still work good if I load it with attn_implementation = None
My code is breaking with dropout_layer_norm: an issue already opened in the discussions, with solution
Please re-download the latest revision and dropout_layer_norm will not be a problem anymore.
Oh amazing! Thanks. My Flash-Attn was breaking when I tried the hack given in that thread. Anyways, still curious:
If it still works good when loaded with
attn_implementation = None
nguyenbh
changed discussion status to
closed