what's the dtype (precision) used during training?
#33
by
keunwoochoi
- opened
hi, thanks for the nice model.
when we get the model through transformers, the default data format is float32.
https://huggingface.co/nomic-ai/nomic-embed-text-v1.5/blob/main/config.json#L49
but the technical report says the model was done on bfloat16.
i'd like to confirm what is the correct dtype. asking because it is critical as the size doubles with float32.
any insight would be great. thanks!
zpn
changed discussion status to
closed
To clarify, we use torch.autocast with bfloat16 precision during training
I've also run inference of the model in bfloat16 and found pretty similar results but YMMV
@keunwoochoi