System crashing on collab

#1
by tunesh - opened

While I am loading your models in the collab or in fact my trained model using peft the kernel is crashing. Do I need to do something else? Please let me know.

This is happening due to the less amount RAM available on collab.

I tried to load the fine-tuned model in my VM (30GB RAM and GPU T4) but still my system crashed due to OOM. Is there any other tested way to load the fine-tuned model binaries with ludwig?

@tunesh Yes, you need to load the base model with 4 bit quantization a T4 GPU before you wrap it with your trained PEFT weights.

arnavgrg changed discussion status to closed

Sign up or log in to comment