invisible models in openllm leaderboard

#554
by leejunhyeok - opened

Hi, I have noticed that some of our models is not visible now.
I changed model name(from MoMo-70B-* to MoMo-72B-*), and it seems something went wrong in the leaderboard.

image.png

In the image above, listed models below are checked as deleted(so they are invisible in image)

these are targets that are public and should be seen(not flagged, in my knowledge)

Could you please check the reason for this issue?
Thank you!

Open LLM Leaderboard org

Hi!
Yes, if you rename a model, it's the same as if you deleted from the leaderboard's point of view, as it queries the old name which no longer exists.
However, you can use @Weyaxi 's cool model renamer here to adress the problem: it will open PRs on all files which need renaming, and we will just have to merge them.

That is a cool utility! I will wait until admins merge PR.

leejunhyeok changed discussion status to closed
Open LLM Leaderboard org

@leejunhyeok could you please ping @SaylorTwift or myself on all the PRs to merge so we make sure we don't miss one? :)

@leejunhyeok could you please ping @SaylorTwift or myself on all the PRs to merge so we make sure we don't miss one? :)

I can automate pinging you and @SaylorTwift if you like.

Open LLM Leaderboard org

I'd rather not ^^
Atm, we ask people wanting a rename to open a discussion like this, and confirms this is a needed change first, then tags us on the PRs.
Otherwise anybody could open a renaming PR about any model - tagging us automatically would create a risk for us to accidentally merge an unwanted renaming.

while trying to make & ping PRs, I met a wierd issue.

image.png
in renaming v1.8.4, I cannot find related PR in https://huggingface.co/datasets/open-llm-leaderboard/results/discussions?status=closed&type=pull_request
(differs with 1.8.6, 1.4)

So I cannot ping you for this model's PR. can you check manually? @clefourrier

model link(current): https://huggingface.co/moreh/MoMo-72B-lora-1.8.4-DPO
result: https://huggingface.co/datasets/open-llm-leaderboard/details_moreh__MoMo-70B-lora-1.8.4-DPO

many thanks

leejunhyeok changed discussion status to open

Hi @leejunhyeok ,

Can you try entering moreh/MoMo-70B-lora-1.8.4-DPO in the from_model part, replacing LoRa with lora.

@Weyaxi thank you for your suggestion.
however this issue remains the same. PR is not created properly.

image.png
I noticed that https://huggingface.co/datasets/open-llm-leaderboard/details_moreh__MoMo-70B-lora-1.8.4-DPO does not have README.md file.
Can this be the problem?

I noticed that https://huggingface.co/datasets/open-llm-leaderboard/details_moreh__MoMo-70B-lora-1.8.4-DPO does not have README.md file.
Can this be the problem?

Yes, I believe that's the problem. This is the first time I have seen a details repo without a README file. Maybe the upload process has been interrupted etc. (I don't know) The code is designed to edit the README part, so when the README is not found, it gives an error.

Open LLM Leaderboard org

I see, can you link to the request file of this model ? I will check the logs to see why the readme wasn't uploaded

hi.
https://huggingface.co/datasets/open-llm-leaderboard/results/tree/main/moreh/MoMo-70B-lora-1.8.4-DPO
this is the request file of MoMo-70B-lora-1.8.4-DPO(current: MoMo-72B-lora-1.8.4-DPO)

Open LLM Leaderboard org

@SaylorTwift , just checked, there was a connection error on the hub at upload time - it's unrelated to the lb.
@leejunhyeok I'll have to edit your files manually haha! I'll do it tomorrow :)

Open LLM Leaderboard org

Should be good!

clefourrier changed discussion status to closed

Sign up or log in to comment