I'm in a situation where I'm looking to set a different DNS record for a certain file on a domain, (specifically www.domain.com/robots.txt), from the root (www.domain.com).(The situation is that I've set the web root to point to a non-live record so the website appears to be non-existent but still has mx records to allow email - but I've now realised  I actually need robots.txt to be live and accessible for crawlers such as archive.org so that they respect my site's wishes to not show past snapshots of the site.)Is it possible to do this it in DNS (like some sort of 'TXT' record?), or is there no way to differentiate the DNS of anything trailing .com, from what's already been set for www.domain.com?The issue is that I really need the web root to solidly appear like a non-existent site, as discussed in my earlier above-linked question. It can't be a 'fake' non-existent page, but an actual one.Perhaps those two things are simply incompatible?
Can you set custom DNS for a file/directory trailing the TLD of a domain (e.g. domain.com/robots.txt)?
domains;web hosting;dns;webserver
With DNS A records, you cannot set records for a specific file.  DNS is all or none for the domain.If you must serve a robots.txt file, then you have to have DNS resolve to a valid domain and have a web server that will answer that request.Here is what I recommend:DNSSetup a CNAME to point www.domain.com to domain.com.Setup an A record to point domain.com to your server's IP address.Web ServerOn your server (assuming Apache), you can use .htaccess to limit what files are being served.Options +FollowSymLinksRewriteEngine OnRewriteCond %{REQUEST_URI} !/robots.txt [NC]RewriteRule ^(.*) - [L,R=404]This will return a 404 (or any code you want) for all pages but /robots.txt.This way you can host robots.txt but not any other pages. There is also the 410 Gone HTTP response code but I am not sure how bots handle it.10.4.11 410 GoneThe requested resource is no longer available at the server and no  forwarding address is known. This condition is expected to be  considered permanent. Clients with link editing capabilities SHOULD  delete references to the Request-URI after user approval. If the  server does not know, or has no facility to determine, whether or not  the condition is permanent, the status code 404 (Not Found) SHOULD be  used instead. This response is cacheable unless indicated otherwise.The 410 response is primarily intended to assist the task of web  maintenance by notifying the recipient that the resource is  intentionally unavailable and that the server owners desire that  remote links to that resource be removed. Such an event is common for  limited-time, promotional services and for resources belonging to  individuals no longer working at the server's site. It is not  necessary to mark all permanently unavailable resources as gone or  to keep the mark for any length of time -- that is left to the  discretion of the server owner.