Learning Rate during pretraining

#58
by shuyuej - opened

May I know the initial learning rate and scheduler settings during the pre-training process?

Thank you very much, and have a nice day!

Best regards,

Shuyue
July 18th, 2024

May I also know the max_grad_norm during the pre-training process?

Thank you very much, and have a nice day!

Best regards,

Shuyue
July 18th, 2024

Sign up or log in to comment