multi-gpu?
#1
by
bdambrosio
- opened
does exllamav2 support multi-gpu for this? (fingers crossed...)
main or experimental?
thanks!
It supports multple GPUs, yes, but not tensor parallelism across GPUs.
Thank you thank you thank you!
Just tested 5.0bit. works magnificently across 2 4090's w -gs 18,20
for those who follow - might lower that 18, it actually allocates almost 23.8GB on gpu 0, but seems to run fine.
update - yup, 16,20 does a less worrisome split, 18.6/15.6
bdambrosio
changed discussion status to
closed