public void addSeenUrl ( String url , int docId ) throws UnsupportedEncodingException { String canonicalUrl = URLCanonicalizer . getCanonicalURL ( url ) ; if ( canonicalUrl == null ) { logger . error ( "Invalid Url: {} (can't cannonicalize it!)" , url ) ; } else { try { docIdServer . addUrlAndDocId ( canonicalUrl , docId ) ; } catch ( RuntimeException e ) { if ( config . isHaltOnError ( ) ) { throw e ; } else { logger . error ( "Could not add seen url: {}" , e . getMessage ( ) ) ; } } } }
This function can called to assign a specific document id to a url . This feature is useful when you have had a previous crawl and have stored the Urls and their associated document ids and want to have a new crawl which is aware of the previously seen Urls and won t re - crawl them .