How ambitious should one be in picking a journal to submit to? I've finished a mathematics paper with two co-authors, and I need to decide on a journal to submit it to.

More so than my other papers, I don't have much of a sense for how "good" it is. It is a substantial generalization of one of the main results proved in a very top journal, although not the most significant of these results, and not the one that got the most attention.

To my mind, (and after discussing the results with a few colleagues), the paper does not pass the "gut test" to be published in this same top journal. But it is not out of the question that it would be accepted. I would (very roughly) guess a 15% chance of acceptance, a 50% chance of positive reviews that are not quite positive enough, and a 35% chance of vaguely negative reviews that say "This is a nice paper, but it certainly isn't good enough to be published here."

Would it be advisable to submit to this top journal? Or would a little bit more modesty be wise? (And of course I will ask my co-authors, but I wouldn't be surprised if they defer to my judgment.) Thanks.
The answer should certainly depend on many different elements, each of which should be judged with respect to your particular situation.

First, the time issue. You should have a look at the AMS survey on journal backlogs, which also indicates the median time to get an answer (for accepted papers only, but we can assume it gives an idea of what happens to all not-rejected-upfront papers). This survey is published regularly in the Notices, and you should be able to find it on line. Then, ask yourself if waiting that time is worth a likely negative answer. Some journals have a pre-refereeing procedure, and reject a fair amount of papers very quickly. This mitigates the risk (either you get rejected quickly, or you have a much-higher-than-average probability to get your paper accepted).

Second, the impression you give to editors. I guess that an editor could remember slightly abusive submissions, and that they would not help you future cases with her; however I have no data or evidence to back up this guess.

Third, your own feelings. Receiving bad reviews is not something to underestimate: it could diminish your enthusiasm toward your own work, alter your willingness to pursue in this direction. This is probably only a small parameter, but if you have had a few difficult cases recently, you might want to take a break from rejections. Added in edit after a few more years of experience: you should also think your possible chain of resubmission. What could happen is that after waiting two years and getting rejected by a top journal, you are a bit down and then submit much lower. Aiming at an intermediate level first might give you a better outcome. This applies even more at resubmission: how much do you try high? How high? This of course depends on the feedback from referees and editors, but some long-term planning might help.

Fourth, you should ask yourself if the journal you consider to submit to is an appropriate venue if your paper do get published there. I would not fear too much people telling that your paper is clearly below the journal's level, as we often judge papers by journal's name anyway. But it might happen that a more specialized and less prestigious journal could reach the intended audience better than the most prestigious, general journal. This is not a very common situation, though.

Last, but far from least, what you expect to get from getting published in the top journal? Is it so much of a career boost compared to very-good-but-not-top journals?

Note also that some editors have the habit of suggesting another venue to papers that have good but not good enough reviews in a very selective journal. It is not easy to know this beforehand, though.