Are the new update of weights having error?
#17
by
wyxwangmed
- opened
I test the newest model weight. But there is something wrong. The loss downed very quick. I want to ask that is there someone has the same issue as me.
@wyxwangmed Would you mind elaborating a bit more on how you tested this?
@wyxwangmed Would you mind elaborating a bit more on how you tested this?
@MeisterDeLaV
https://github.com/InternLM/xtuner/blob/main/xtuner/model/modules/dispatch/llama.py I used xtuner to fine-tuned Yi-34b-200k, and xtuner will use this code to use flash_attn, I comment this section of code to close the flash_attn, the loss is right. So I think is not the reason of weights. But I don't know why I need to comment this section of code.
wyxwangmed
changed discussion status to
closed