I've been experiencing high RPM spikes recently. Something like this:When I debugged, I've found reasons to believe the reason is the msnbot suddenly makes a massive crawl and then stops. I assume I'm not the only site that has a problem to suddenly handle 5x the normal RPM, so why does msnbot do this? Is there any valid explanation or technical reason to perform such a HIT & RUN?
Is there a reason msnbot crawls in spikes?
robots.txt;bing;web crawlers