ImportError: Using `bitsandbytes` 8-bit quantization requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes

#29
by WQW - opened

Error:

  • as following
  • pip install accelerate bitsandbytes doesn't work
ImportError: Using `bitsandbytes` 8-bit quantization requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes

Env info

Python            3.12.1
accelerate               0.27.2
bitsandbytes             0.42.0
torch                    2.2.1
tqdm                     4.65.0
transformers             4.38.2
trl                      0.7.11

Someone says we need to downgrade transformers library to version 4.30, on order to fix this error

However, GemmerTokenizer need to upgrade transformers to version 4.38 ...

Hi @WQW
Thanks for the issue, if you are using a jupyter notebook / google colab make sure to restart the kernel and re-install everything again. Also make sure to have access to a GPU device

Hi @WQW
Thanks for the issue, if you are using a jupyter notebook / google colab make sure to restart the kernel and re-install everything again. Also make sure to have access to a GPU device

I am running code on linux server terminal locally

Can you retry on a fresh new env? Also can you confirm you have access to a GPU device?

I also encountered the same issue

I am having the same issue. I am using Google Colab with an A100 GPU and have tried several versions of bitsandbytes, all >=4.1.3 since only this version supports the quantization method. I had to install version 0.4.2 on Google Colab, and that one worked.

@WQW @HONGMAOLIN I think if you are experimenting this locally. Accelerate requires GPU (Mandatory). without GPU accelerate won't work. You may try that on google colab free T4.

Hi @WQW
Thanks for the issue, if you are using a jupyter notebook / google colab make sure to restart the kernel and re-install everything again. Also make sure to have access to a GPU device

this worked for me! Thank you! (PS: Restarting kernel on Google CoLab!)

Happening for me on Windows(no GPU) with python 3.11 and:
accelerate 0.28.0
bitsandbytes 0.43.0
transformers 4.38.2

Hi @zeebrog ,
In order to use bnb you need to have access to a GPU

same issue on a mac m1

@thepandapi as far as I know Mac doesn't have a dedicated GPU. Please make sure if you are using the library locally you have a dedicated GPU.

I am having the same issue

packages installed with CUDA 11.8 -
torch - 2.3.0+cu118
llama-index - 0.10.37
llama-index-llms-huggingface - 0.2.0
transformers - 4.39.0
accelerate - 0.27.0
bitsandbytes - 0.43.1

@ybelkada Thank you mate! It worked after restarting the jupyter notebook

Hi everyone, there are news about this issue? I have the same issue In a windows machine with a dedicated GPU, a rtx 3060 of 20gbs.

Sign up or log in to comment