CPU support?

#4
by liadlevy - opened

Is there a CPU support for this model?

Yes, you can run on CPU with the standard model.generate() instructions as long as you have enough RAM to hold the model weights. This is ~30GB if using FP32 or ~15GB if using BF16.

Yes, you can run on CPU with the standard model.generate() instructions as long as you have enough RAM to hold the model weights. This is ~30GB if using FP32 or ~15GB if using BF16.

Thanks, Also I’m having trouble to load the model using oobabooga webui. Any instructions how can I use this model with the WebUI framework?

We leave it to the developers of oobabooga to add support for our model architecture.

jfrankle changed discussion status to closed

Sign up or log in to comment