A course-based letter from a named professor or a research letter from a young no-name researcher? I can get a letter from a course instructor that I got a highest possible grade. He is a very well known professor at a top 5 institute. Of course, he can say little about my research.

I can also get a letter from my research collaborator who just got his PhD from a not-so-famous school. We worked on three papers so he can definitely comment my research in details. He is an assistant professor in a research-focused school rank at 100.

Which one is better for a research (PhD) program?

The program I am applying to requires two letters. The first letter is from a big name who I've done research with.
Assuming you cannot submit both, I would go for the second one. You are applying to a research program and have research experience: the committee are going to be interested in that. If the committee is interested in your grades, they can already see them in your transcript.