mosaicml

/

mpt-7b

like 1.16k

Follow

Mosaic ML, Inc. 354

Hi @abhinavkulkarni , is the ask here if attention_mask is being passed as a kwarg to the forward of MPTForCausalLM?

Hey @sam-mosaic ,

Thanks for the reply. You can see here, output_attentions options is not specified yet in modeling_mpt.py: https://huggingface.co/mosaicml/mpt-7b/blob/main/modeling_mpt.py#L140

It would be nice if this if block were filled up instead of rasing NotImplementedError. I think it should be trivial given MPT uses traditional transformer, so collecting attention outputs from every hidden layer in the forward function and then returning it in a tuple.

You can see these line numbers from modeling_opt.py for a reference:

https://github.com/huggingface/transformers/blob/main/src/transformers/models/opt/modeling_opt.py#L245
https://github.com/huggingface/transformers/blob/main/src/transformers/models/opt/modeling_opt.py#L368
https://github.com/huggingface/transformers/blob/main/src/transformers/models/opt/modeling_opt.py#L725

Thanks for the great work!

Thanks @abhinavkulkarni , I get it now. IIUC, output_attentions outputs the attention matrix from the attention module?

We do not use the torch code path much, we usually train with Triton Flash or CUDA Flash. However, neither of those attention implementations can support outputting the attention matrix. So, if we supported this flag it would only be for torch. Does AutoGPTQ mainly focus on lower-resource inference and fine-tuning?

Hey @sam-mosaic ,

So, it seems the recent changes have solved most of the issues, except for line 110 of modeling_mpt.py which needs to be changed from:

return (attn_bias, None)

to

return (attn_bias, attention_mask).

I made changes in my local copy of modeling_mpt.py in site-packages and was able to GPTQ quantize this model using AutoGPTQ repo.

To improve efficiency, in line 109 of modeling_mpt.py, we integrate attention_mask into attn_bias if it exists.
If the requested attn_impl does not support an attn bias, then we use attention_mask (eg attn_impl: flash does not support attn bias and therefore the output of the _attn_bias fn is (None, attention_mask); see line 88)

This does not control if output_attentions are available.