1. A method of indexing documents in websites, the method comprising: on a server system having one or more processors and memory storing programs to be executed by the one or more processors: storing, for each website of a multiplicity of websites, a corresponding current crawl rate limit; and providing a crawl rate control mechanism to a respective owner of a respective website, wherein the crawl rate control mechanism enables selection of a new crawl rate limit corresponding to the respective website by the respective owner.