I am running a Mediawiki at the domain someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively scanned by search engines with queries like:/index.php/Special:Whatlinkshere/some_topicor as well/index.php?title=Special:Whatlinkshere&target=some_topicwhere some_topic is an article of the wiki.These requests seem to be of very low benefit but they consume a good deal of bandwidth and in addition I fear those automatically generated pages are not so good for my site's quality reputation in the serchengines evaluation.As the requests a mostly done by 'good' engines such as Google or Bing I am quite sure they would obey robots.txt. So I added the following robots.txt to the folder of the base url someurl.com (I added the whole robots.txt, even though only line 1 and 6 are relevant for the queries named above):User-agent: *Disallow: User:Disallow: Discussion:Disallow: MediaWiki:Disallow: Special:Disallow: /login.phpDisallow: /profile.phpDisallow: /author/Disallow: /category/Disallow: /tag/This robots.txt is active for about two days now and has been crawled already, but still there are many requests to URLs like the above which I thought blocked.So I have the following questions now:1) is the above logic correct and capable of denying access (to well behaving bots). Especially I wonder whether Disallow: Spezial: correctly works as a wildcard to deny all request having Special: in URL or in parameter. I also wonder if the : in Special: might be a problem.2) If so why then there is no effect yet? I consider I just have to grant more time to see the effect?3) Will denying in the robots.txt lead to the de-indexing of this sites from the searchengines results? If not how can I get this huge amount of automatically generated URLs de-indexed?
Using robots.txt to deny access to MediaWiki special pages using substring matching
web crawlers;robots.txt;indexing;mediawiki
robots.txt disallow rules are all starts with rules, not substring rules.MediaWiki suggests using this in robots.txt for a case like yours:User-agent: *Disallow: /index.php?Disallow: /index.php/HelpDisallow: /index.php/MediaWikiDisallow: /index.php/Special:Disallow: /index.php/TemplateDisallow: /skins/Google says that it supports more advanced syntax along with some other major search engines:Google, Bing, Yahoo, and Ask support a limited form of wildcards for path values. These are:* designates 0 or more instances of any valid character$ designates the end of the URLFor those user agents you could use rules like:Disallow: *HelpDisallow: *MediaWikiDisallow: *Special:Disallow: *TemplateOther crawlers will end up just ignoring those rules because none of your URLs will start with any of the rules.