Enabling PEFT for jais-13b
#13
by
yassineafr
- opened
When trying to fine-tune jais-13b using qlora, I encourted this error :
This error says that the "hidden_states" is leaf variable(Leaf Variable: A tensor that is not the result of an operation and has requires_grad=True.) therefore it doesn't accept in-place operation like in this error:
hidden_states *= torch.tensor(float(self.embeddings_scale), dtype=hidden_states.dtype, device=hidden_states.device )