I would like for User-agents to index my relative pages only without accessing any directory on my server. As initial thought, i had this version in mind: User-agent: *Disallow: */*Sitemap: http://www.mydomain.com/sitemap.xmlMy Questions: Is it correct to block all directories like that - Disallow: */*? Would still search engines be able to see and index my sitemap if i disallowed all directories? What are the best practices for securing the robots.txt file?For Reference:Here is a good tutorial for robots.txt#Add this if you want to stop Alexa from indexing your site.User-agent: ia_archiverDisallow: /#Add this to stop duggmirror User-agent: duggmirrorDisallow: /#Add this to allow specific agents User-agent: GooglebotDisallow: #Add this to allow all agents while blocking specific directoriesUser-agent:  *Disallow: /cgi-bin/Disallow: /*?*
How to secure robots.txt file?
google;seo;sitemap;user agent;robots.txt
That's going to block your entire website from being crawled.NoThere is no such thing as securing your robots.txt. If you don't want to keep visitors out of your directory root you need to prevent that using more secure means. Putting a blank index.html file will easily do the trick. If you're running Apache you can also do it easily using htaccess.