public boolean shouldVisit ( Page referringPage , WebURL url ) { if ( myController . getConfig ( ) . isRespectNoFollow ( ) ) { return ! ( ( referringPage != null && referringPage . getContentType ( ) != null && referringPage . getContentType ( ) . contains ( "html" ) && ( ( HtmlParseData ) referringPage . getParseData ( ) ) . getMetaTagValue ( "robots" ) . contains ( "nofollow" ) ) || url . getAttribute ( "rel" ) . contains ( "nofollow" ) ) ; } return true ; }
Classes that extends WebCrawler should overwrite this function to tell the crawler whether the given url should be crawled or not . The following default implementation indicates that all urls should be included in the crawl except those with a nofollow flag .