1. A method of scheduling documents for crawling, performed on a computer system having one or more processors and memory storing one or more programs for execution by the one or more processors, the method comprising: storing sitemap information for a plurality of websites, wherein the information includes a predicted update period for at least a plurality of documents identified by the sitemap information; analyzing the stored sitemap information to identify a respective website having sitemap information that is at least potentially out of date; updating the stored sitemap information for the identified respective website by downloading updated sitemap information for the identified respective website; and scheduling documents for crawling in accordance with the updated stored sitemap information for the identified respective website.