How much travel is too much travel? I am a first-year postdoc and I am currently structuring my summer plans for travel/conferences/etc.

May already includes a 2 week international trip of a one-week workshop and two seminar talks. June includes 2 one-week conferences domestically. July includes a 2 week international jumbo conference. August includes at least a one-week international research visit. If I did all the conferences pencilled into my calendar, I would be gone for 3 weeks in September.

To top it all off, there's a few more invites that just occurred for a week long summer school in June and a colloquium invitation in August. Both international. 

I have been told that "you do not say no until tenure," but this seems too much to handle. At some point I have to sit down and do research, keep up with collaborations, and recover from travel. To be fair, I love travel, I am single, and I do not have a child or pet, so I have no obligations for being home; however, I am fearful of burning out. I am in mathematics, so there's no need for a lab and could potentially do work on the road but I am much more efficient when with my collaborators at home or at their home institution (which is not where these things are).

So the chain of questions here: 


At what point does one have too much travel? Should I just pack my bags and try to learn how to research on the road? 
When can one say no? 
Do people care about what conferences you have been to and if they are on the CV or just about speaking? Is there a point where more invited talks hits a point of diminishing returns?
Different people set their thresholds differently, but it all comes down to balance and cost/benefit analysis.

I don't buy into the "don't say no until tenure" argument, because a) until tenure is a pretty big chunk of your life, especially when you add in postdoc, and b) the patterns and expectations you establish before tenure will likely remain with you afterwards.  You need to make a decision which is sustainable for you as your career is now and reassess every couple of years to make sure it is still working.

Myself, I travel 1-2 times per month.  Some academics I know travel nearly once a week; others travel only once or twice a year.  In my observation, this does not really correlate with career stage, but more with desire for public recognition vs. the tradeoffs the person is comfortable with in their life.

So, how to make that judgement for yourself?
Personally, I find Latour's model of science as "credibility investment" a useful analysis tool.  Under this model, you can view scientific life as manipulation of three currencies:


Credibility is the main and most unusual currency of academia, which is generated by your publications, your position and appointments, your visibility in the community, service, etc, and which is invested in order to obtain funding.
Results are the data, theorems, etc. produced by your work, which are the raw material required for generation of credibility, and which generally requires money to produce.
Funding is money or other forms of support, which can be obtained through investment of credibility and is itself invested in order to produce results.


Success as an academic requires management of the flow of these three quantities: your relative levels in any may rise and fall with time, but if you go broke in any of the three areas, your career is in deep trouble.  They are hard to quantify, but with self-reflection and comparison to peers you can often have a sense of whether you are "doing well" or "doing poorly" in each area.

Bringing it back to travel, then: most travel can be viewed as part of obtaining credibility (e.g., conferences, invited talks) or obtaining funding (e.g., visits to program managers).  Their cost, as you have identified, is primarily opportunity to produce results and to write your results up in papers (yes, they cost money too, but your time is usually more valuable).  For each individual conference, you can ask how much this conference is likely to serve those goals, relative to the time that you will lose as a result.  If you become good at working on the road (I love airplanes as internet-free time), then the effective cost may decrease, changing your relative weightings, but the principles remain the same.