Should I use recommendation letter from young professor I worked with or well-known professor I met in class? I'm choosing reference writers for my math PhD application this year, and need to decide between the 2 following professors: 


Professor X is a young professor with whom I took measure theory and
did a research project in my freshman year. The project went pretty
well - we obtained some results earlier than he expected, and
published a paper. He moved to another university and no longer
worked with me, but we still keep in touch.
Professor Y is a fairly well-known professor whom I met at a 1-month
summer workshop. I didn't interact much with her outside of the
classroom, but she seemed to like me: after the workshop, she said she
was really impressed by me and voluntarily asked if I want a
recommendation letter from her.


Neither professor works in the field that I plan to study in grad school.

I often hear that "good at research" letters are way better than "do well in class" ones. However, in my sophomore year, I applied to some REU programs with professor X's letter (and another not so strong "do well in class" letter), and was rejected by all of them. Of course the recommendation is just one factor, but I think my course work was not bad - probably better than many of my junior friends who were admitted. My Putnam score was pretty good too, although I don't think it mattered. The following year I reapplied with both professor X's and Y's letters, and got first round offers from most of the programs.

Do you think this indicates that professor Y's letter substantially strengthened my application, or actually REU programs looked at some other factors I wasn't aware of? Do grad schools and REU programs look at recommendations the same way? Which letter would you recommend me to use? I can't use both because I have 2 other letters already, and many grad schools say they may not read more than 3 letters.

Thank you very much.

Edit:


About the 2 other reference writers that I've already chosen: both did research with me, one of them also taught me several courses.
The other letter I used in my sophomore year (along with professor X's) was probably not that bad. The reference writer said he was pretty impressed that my course grade was 95% or something, while everyone else got less than 60%. It's just that I didn't do research with him.
Your REU evidence that X wrote you a weak letter is not very strong.  I run an REU, and we accept less than 3% of our applicants.  Our target participants are either legendary at small colleges (best in a decade), or excellent at top universities (top 25% this year at Harvard).  Plenty of excellent people get turned away for essentially random reasons.  Maybe one of your letters was submitted late.  Maybe you forgot to list a course on your application that was considered essential that year.  Maybe I had a bad burrito for lunch and hated your essay when I read it.  Maybe there were just lots of really good applicants that year.  And yes, maybe professor X (or professor Z, the other letter-writer) said or implied something negative about you.

In addition, Professor Y offering to write on your behalf after a one month course says more about her than it does about you.  This does not guarantee that she will write a strong letter, and if she does she may write strong letters for a lot of people, which might be known by the people reading her letters.

Here is my advice.  Go to Professor Y and tell her that you did research with Professor X and published a paper together.  Then, ask her whether you should ask for X's recommendation or hers.  If she really feels that you are spectacular, she will insist on writing the letter herself.