GOT CUDA out of memory

#5
by weiminw - opened

when I run the 4bit model in vllm. I got following error. I am using rtx4090 (24G GPU memory).

Screenshot from 2023-12-01 12-28-40.png
Screenshot from 2023-12-01 12-28-09.png

01-ai org

Please make sure no other processors occupying the GPU memory.
24G memory is enough to run 34B-4bits version

tianjun changed discussion status to closed

@weiminw You might want to manully set ctx len to 4096

Sign up or log in to comment