public boolean isRobotPermitted ( String url , String userAgent ) throws IOException , RobotsUnavailableException { RobotRules rules = getRulesForUrl ( url , userAgent ) ; return ! rules . blocksPathForUA ( new LaxURI ( url , false ) . getPath ( ) , userAgent ) ; }
Returns true if a robot with the given user - agent is allowed to access the given url .