We have a global brand website project for which we are only working the LATAM portion.  There is a website installation process here that allows to have one website installation with several ccTLDs, in order to reduce costs.Because of this the robots.txt in www.domain.com/robots.txt is the same file in www.domain.com.ar/robots.txt.We would like to implement custom robots.txt files for each LATAM country locale (AR, CO, CL, etc..). One solution we are thinking about is having a redirect placed at www.domain.com.ar/robots.txt to 301 to www.domain.com.ar/directory/robots.txt.This way we could have custom robots.txt files for each country locale. Does this make sense?Is it possible to redirect a robots.txt file to another robots.txt file?Any other suggestions?Thanks in advance for any input you might have.
Serve a different robots.txt file for every site hosted in the same directory
google;redirects;web crawlers;robots.txt
I wouldn't count on all spiders being able to follow a redirect to get to a robots.txt file.  See: Does Google respect a redirect header for robots.txt to a different file name?Assuming you are hosted on an Apache server, you could use mod_rewrite from your .htaccess file to to serve the correct file for the correct domain:RewriteEngine OnRewriteCond %{HTTP_HOST} ^www\.example\.([a-z\.]+)$RewriteRule ^robots.txt /%1/robots.txt [L]In that case your robots.txt file for your .cl domain would be in /cl/robots.txt and your .com.au robots.txt file would be in /com.au/robots.txt