What is the best initial or general setup for the robots.txt to allow search engines to go through the site, but maybe restrict a few folders?Is there a general setup that should always be used?
How do you configure robots.txt to allow crawling of the site except for a few directories?
seo;robots.txt
Google Webmaster tools has a Section called Crawler accessThis section allows you very easily to create your robots.txtFor example to allow everything except blog a folder called test your robot.txt would look something likeUser-agent: *Disallow: /TestAllow: /