I want to allow a certain html file and my site's index file to be indexed by search engines. Everything else should be disallowed. My home directory does not actually contain an index file, I am using .htaccess to redirect to /cgi-bin/index.cgi. I am currently using this:User-agent: * Allow: /cgi-bin/index.cgiAllow: /contact.html  Disallow: / However, google webmaster tools is saying:Googlebot is blocked from http://mydomain.com/Is there a way of allowing indexing of the root while blocking all other files i.e., mydomain.com/*
Problem with my robots.txt
robots.txt
As suggested by Pekka you may want to try to place the Allow directives after the Disallow directives.But given the differences in interpretations between Google, Bing and others, you may want to use a robots meta tag instead. This will be safer and more granular.In your disallowed pages:<meta name=robots content=noindex />In your allowed pages:<meta name=robots content=index />(to be placed in your <head> tag)See http://googlewebmastercentral.blogspot.com/2007/03/using-robots-meta-tag.html