TheBloke

/

Wizard-Vicuna-13B-Uncensored-GPTQ

like 319

This happens when you don't have GPTQ-for-LLaMa installed in text-generation-webui/repositories. Please do the following:

cd text-generation-webui # wherever you have it installed
mkdir -p repositories
cd repositories
git clone https://github.com/oobabooga/GPTQ-for-LLaMa -b cuda GPTQ-for-LLaMa
cd GPTQ-for-LLaMa
python setup_cuda install

This will install GPTQ-for-LLaMa and now you can do GPTQ inference.

Alternatively, here is a Google Colab Notebook already set up which you could use as a base: https://colab.research.google.com/github/eucdee/AI/blob/main/4bit_TextGen_Gdrive.ipynb

Thanks, it works now. Bless you.

Hey, I've been trying to use the colab link as a base. I've ticket everything, but I'm not getting the link. Am I missing something?

Edit: The last line when trying to load the model says "server.py: error: unrecognized arguments: --cai-chat"

That mode has been deprecated, use --chat instead.

I used the pre setup google collab link, so I don't know how to use --chat instead. I'm new to this, so let me know if I just edit this myself or need to look for a different link

I don't really use google collab, but look under section 3. Launch and find

if chat:
params.add('--cai-chat')

...and replace --cai-chat with --chat

I'll try that later and edit if it worked, thank you :)

Yeah that should work - --cai-chat was the old argument that changed a month or two ago. That Google Notebook hasn't been updated in a while

This time I got an error at the end "NotADirectoryError: [Errno 20] Not a directory: 'characters'" don't know what to do about it. Is there an alternative to run this? My pc probably won't be able to run this locally, that's why I opted for collab instead.

I have updated the Notebook so it uses AutoGPTQ, and the issue with characters is gone.

Please try: https://colab.research.google.com/drive/1ptEmUZkIVOv9h9N-ieLILmzzKoZZmA0s?usp=sharing

https://colab.research.google.com/drive/1ptEmUZkIVOv9h9N-ieLILmzzKoZZmA0s?usp=sharing . This link works but it has a lot of error inside, whenever i upload a character it shows error and even if I created a character it shows the same error. Could you run this and see what's the issue here?

I have updated the Notebook so it uses AutoGPTQ, and the issue with characters is gone.

Please try: https://colab.research.google.com/drive/1ptEmUZkIVOv9h9N-ieLILmzzKoZZmA0s?usp=sharing

Now I'm getting this "server.py: error: unrecognized arguments: --autogptq"