How important is the prestige of recommendation letter writers? If it makes a difference, I am an undergrad. 

I am wondering how important prestige in the following situation:

I can choose between two advisors, A and B, to work with. I think that both would be great advisors, but A has more publications and seems to be more well known than B. However, B's work is much more closely related to what I am interested in, though A is somewhat familiar with this area as well. All else equal, I would like to work with B because their research interests match mine more closely. However, I will need a letter of recommendation, so I would like to get the best letter possible and I think that A would be able to provide a better letter because they have more publications and are more well known.

How should I choose between A and B?
I am assuming the two choices are at the same institution. If they are not, choose the higher-ranked institution (if there is a clear winner), because moving forward, name DOES matter).  That said:
You have two currently incompatible, yet not mutually-exclusive motivations:
a) work with someone who is well known in the field
b) work with someone who has closely aligned research interests

and, you are concerned about the letter of recommendation, to come later.

the correct choice is B.  You should work with the faculty member whose research interests closely match yours.  Any other decision is to enter into a failing marriage.  It is important that you can be excited about what you are doing, and your excitement will in turn excite your mentor, which will pay dividends.

THAT SAID, names do matter.  If the faculty that matches your interests is junior, you will have to plan ahead to ensure that in addition to his/her glowing recommendation, you also get a letter from the more senior colleague. This isn't impossible, or even difficult.  If you are starting grad school, just be open and honest about it.  Engage him/her. Go to their lab meetings. Later, when you do your prelim or comp exams, you can invite that person to be on your committee.  You'll get both of best worlds: an advisor that shares your passion and a letter from a big wig. 

You have to love what you're doing.