I have a couple of doubts / questions / ideas related to robots.txt:Can we deny website for all bots except for chosen ones in order totell other bots not to crawl site:User-agent: *Disallow: /User-Agent: GooglebotUser-Agent: bingbotUser-Agent: SlurpAllow: /person/Allow: /products/Can we deny whole site and then just specify list of pages we wantto allow for indexing like in example above? I do not want to giveaway URLs that I want to exclude from crawling, as this informationcan be used against me. Note that we do not have Allow: /, so willbot be able to access home page? In example above, will Allow beapplied for all three user agents on the list or Allow needs to bepasted under each user agent?User-Agent: GooglebotAllow: /person/Allow: /products/User-Agent: bingbotAllow: /person/Allow: /products/User-Agent: SlurpAllow: /person/Allow: /products/
Optimizing and securing robots.txt
robots.txt