[Bug] Import error when running on environment without flash_attn

#2
by Isotr0py - opened

I'm running this model with below code on an environment with T4 GPU which doesn't support building flash-attention:

import torch
from transformers import AutoModel, AutoTokenizer
from huggingface_hub import snapshot_download

path = 'OpenGVLab/InternVL2-1B'
model = AutoModel.from_pretrained(
    path,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True).eval().cuda()

tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)

And it raises an import error:

ImportError: This modeling file requires the following packages that were not found in your environment: flash_attn. Run `pip install flash_attn

According to the modeling_intern_vit.py, flash_attn should be an alternative attention implementation. Is it intended to design or just a bug?

OpenGVLab org

Hello, thank you for your feedback.

Could you please let me know which version of transformers you are using? When I used transformers version 4.37.2, it was able to run normally in an environment without flash_attn. I suspect that there might have been some changes in the newer version of transformers that caused this bug.

I used transformers==4.43.2 before. I also tested with transformers==4.37.2 on colab and the issue still exists:

!pip install transformers==4.37.2 timm

import torch
from transformers import AutoModel, AutoTokenizer

path = 'OpenGVLab/InternVL2-1B'
model = AutoModel.from_pretrained(
    path,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True).eval().cuda()

tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)

Outputs:

Requirement already satisfied: transformers==4.37.2 in /usr/local/lib/python3.10/dist-packages (4.37.2)
Requirement already satisfied: timm in /usr/local/lib/python3.10/dist-packages (1.0.8)
...
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-48060db57bdf> in <cell line: 7>()
      5 
      6 path = 'OpenGVLab/InternVL2-1B'
----> 7 model = AutoModel.from_pretrained(
      8     path,
      9     torch_dtype=torch.bfloat16,

4 frames
/usr/local/lib/python3.10/dist-packages/transformers/dynamic_module_utils.py in check_imports(filename)
    178 
    179     if len(missing_packages) > 0:
--> 180         raise ImportError(
    181             "This modeling file requires the following packages that were not found in your environment: "
    182             f"{', '.join(missing_packages)}. Run `pip install {' '.join(missing_packages)}`"

ImportError: This modeling file requires the following packages that were not found in your environment: flash_attn. Run `pip install flash_attn`

Thanks for your great work!
Are there any pre-trained checkpoint link rather than safetensors?

This has to do with how transformers.dynamic_module_utils looks for and tests availability of imports. It doesn't support nested try/catch.

I proposed a fix in https://huggingface.co/OpenGVLab/InternVL2-1B/discussions/4.

OpenGVLab org

Thank you for your feedback. This bug was caused by transformers.dynamic_module_utils not supporting nested try/catch. I have fixed it.

czczup changed discussion status to closed

Sign up or log in to comment