I have been using the Perlfect Search Script http://www.perlfect.com/freescripts/search/ for several years on my sites. For reasons unknown it stopped working properly a few months ago. When I run the indexing script I get the following error:Loading http://emetnews.org/robots.txt...Error: Couldn't get 'http://emetnews.org/robots.txt': response code 403Not using any robots.txt.Error: Couldn't get 'http://emetnews.org/': response code 403The robots.txt file is easily accessible by Google and even visitors to the site. It has the permissions set to 644. I have not changed anything prior to the script no longer working. I can't get in contact with the script developer (they haven't updated the script or the site for a few years now). My Web host doesn't work with outside scripts.Anyone have any idea what's wrong? I like the layout of the script. It looks very professional when in use (and its free).Result of running curl --user-agent libwww-perl/6.08 http://emetnews.org/robots.txt:<!DOCTYPE HTML PUBLIC -//IETF//DTD HTML 2.0//EN><html><head><title>403 Forbidden</title></head><body><h1>Forbidden</h1><p>You don't have permission to access /robots.txton this server.</p><p>Additionally, a 500 Internal Server Errorerror was encountered while trying to use an ErrorDocument to handle the request.</p></body></html>curl http://emetnews.org/robots.txt:User-agent: Mediapartners-GoogleDisallow:Sitemap: http://emetnews.org/sitemap.xml# User-agent: Browsershots# Disallow:User-agent: NinjaBotAllow: /User-agent: *Disallow: /_lee/Disallow: /blosxom/flavours/Disallow: /blosxom/plugins/Disallow: /contact/Disallow: /cgi-bin/Disallow: /feedback/Disallow: /img/Disallow: /includes/# Disallow: /javascript/Disallow: /lastrss/# Disallow: /media/Disallow: /mp3s/Disallow: /print/Disallow: /r/Disallow: /sendPage/# Disallow: /style/Disallow: /talkback/Disallow: /trip/# block any URL that includes a ?Disallow: /*?# Disallowing the robot from Alexa from listing files in the Internet ArchiveUser-agent: ia_archiverDisallow: /Thanks for your help. I did have this in my .htaccess file:# Blocks access from libwww-perl user-agents and URLS which include the command =http:RewriteCond %{HTTP_USER_AGENT} libwww [NC,OR]RewriteCond %{QUERY_STRING} ^(.*)=http [NC]RewriteRule ^(.*)$ - [F,L]I commented it out and I now get the text of the robots.txt file when I runcurl --user-agent libwww-perl/6.08 http://emetnews.org/robots.txtNow, however, when I run the indexer, instead of it saying can't get the robots.txt file, it now says it ignores the file. ???
Problem accessing robots.txt from search script
perl;search