Is there a way to tell search engines, that a page they crawl should be included in the search results now, but have to be deleted at a certain time in the future?I have a website where hundreds of publications happen each day and I want them to be crawled and be searchable, but I am legally required to remove the information after a while (individual date for each page).After that given date, the page will not be visible at my website anymore (HTTP response 410 gone), but the page will linger in e.g. the google cache for a while, which could cause legal issues for me. Obviously, it's not viable to issue hundreds of content removal requests to google by hand. On the other hand, the individual pages do not get modified for some months until they have to be discarded, so google bot won't check in often.For what I understand, the HTTP Expires header is a label for minimum freshnes and not for maximum lifetime, correct? I am sending last-modified-at and etag headers, but they don't help here. Is there any way to say cache, but only until 2011-08-15?
HTTP: How to be deleted from search engines at a certain point in time in the future?
http;search engines
For google there is a meta tag called unavailable_after, which does exactly what I was looking for: It tells google to remove a certain page at a specific time in the future.It is the only way to achieve what I was hoping to accomplish: Getting the pages removed automatically, at the right time, not relying on the crawler to come back and notice the 410 Gone response, which can take a weeks after the content has been removed.Example: <META NAME=GOOGLEBOT CONTENT=unavailable_after: 25-Aug-2007 15:00:00 EST />Or with HTTP header, for PDFs etc.:X-Robots-Tag: unavailable_after: 23 Jul 2007 15:00:00 PSTSources: http://googleblog.blogspot.com/2007/07/robots-exclusion-protocol-now-with-even.html and http://www.google.com/support/webmasters/bin/answer.py?answer=79812I could not find out if Bing, Yahoo & Co have adopted this Google specific tag.