Can anyone tell me what does this mean in Blogger's robots.txt file? Do I need to edit anything in it? Should I remove /search from the Disallow: line?User-agent: Mediapartners-GoogleDisallow: User-agent: *Disallow: /searchAllow: /Sitemap: http://css3wdesign.blogspot.com/sitemap.xml
Why is Disallow: /search in Blogger's robots.txt?
robots.txt;blogger;blogspot
In addition to closetnoc's answer...Should I remove /search from the Disallow: line?No. It is a good idea to block bots from crawling your search results (which I assume is what this is referring to).You don't normally want your search result pages from appearing in Google's search results pages! And Google doesn't want this either. Google wants to index your actual pages, and return these in the SERPs. Allowing bots to crawl your search results (which could potentially be infinite) could also use up a lot of unnecessary bandwidth.However, Mediapartners-Google (Google's AdSense bot) is permitted to crawl your /search results. I believe this is necessary if you wish to serve adverts from your search results pages.Do I need to edit anything in it?Not unless you want/need to block some bots from crawling certain areas of your site. Note that some bots will completely ignore your robots.txt file anyway.