Looking on the robot file of our ,soon to be, website. I want to know what prevent the site to be crawled. Is it this line ? If not, what will it disallow ?Disallow: *?s=
Robot.txt disallow *?s=
web crawlers;robots.txt
Disallow: *?s=Bots following the original robots.txt specification would not be allowed to crawl URLs like these:http://example.com/*?s=http://example.com/*?s=foohttp://example.com/*?s=/So they interpret *, ? and = literally (i.e., these characters have to appear at the beginning of the URL path).But many bots use (their own) extensions to the robots.txt specification, where some characters are reserved, i.e., they get a specific meaning.Google, for example, uses * for pattern matching:To block any sequence of characters, use an asterisk (*).That means the Googlebot is not allowed to crawl URLs like these:http://example.com/?s=http://example.com/?s=foohttp://example.com/foo?s=http://example.com/foo?s=barhttp://example.com/foo/foo/foo?s=barOther bots may have other interpretations.