1. A method of crawling at least one website comprising at least one universal resource locator (URL), said method comprising: maintaining a lookup structure comprising all of the URLs known to be on a website; calculating a hub score for each webpage of said website to be recrawled, wherein said hub score measures how likely the to be recrawled webpage includes links to fresh content published on said website, wherein said fresh content is differentiated from other content based on how long ago said fresh content was authored; sorting all said to be recrawled pages by their hub scores; and crawling said to be recrawled pages in an order from highest hub scores to lowest hub scores to crawl said fresh content before crawling relatively older published content.