I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via myschool.edu/myusername/myfolder/myfile.html returns a 403 error.And yet Google somehow managed to grab the contents of those private files and store them in its cache! How is this possible? [I've since removed those files, so I'm just curious how Google managed to do this.]
How did Google manage to crawl my 403 pages?
web crawlers;security;googlebot