Ensuring our travel/accommodation directory is FRESH!
Total No. of sites temporarily removed from directory due to offline status: 12
No. of sites temporarily removed TODAY from directory due to offline status: 2
No. of URLs updated in total: 105 (3.75% of total listings)
No. of URLs updated TODAY: 1 (0.04% of total listings)
No. of websites spidered today: 168 (6% of total listings) - view list
Curating an online website directory doesn't just mean adding websites to it. Websites go offline, get repurposed, get 301'd to a different domain. They are also being updated all the time. There is a LOT of "churn" online, and our directory has to reflect these changes regularly. Most directories fail at these tasks...you will see a LOT of "link rot" in the vast majority of directories online - you browse a category, you click on twenty links, and five or more are "dead" links, or 301'd to different domains owned by different companies.
How do we keep our directory fresh?
There are four things we need to be aware of:
websites that go temporarily offline
301 redirects
websites that go permanently offline
website content is being created and updated all the time
Let's look at all four, and see live data from our directory of examples where this is happening right now.
Listings that are considered "temporarily" offline
These are listings that our spider has detected have been offline for up-to the last 20 days. Currently we check the site every 3 day(s) to check its status. It has to have "7 strikes" against it (roughly 20 days in total) before it is removed from our directory. If by that time it is still offline, then we consider it "permanently offline" as around 20 days is a LONG time for a website to be offline. Having said that, if a website comes back online within the (roughly) 20 day period, its counter is reset back to zero to be as lenient as possible to websites (as there are many reasons why sites go offline that are temporary in nature).
Websites found by our spider to have been recently discovered offline
moorlandviewcottage.co.uk
offline today
Found in England
Last spidered: 1/19/2022
No. of times spidered in a row and found offline:
friedastearoom.co.uk
mhlestateagents.co.uk
1 day(s) offline
armagh.gavabiz.co.uk
Found in Northern Ireland
plazaresortandspa.com
Found in Rest of the World
visitdesertsafaridubai.com
smartgo.co.uk
acestudios.bm
Found in Travel-related Websites
marrakech4you.com
copper-canyon-tour.com
blueviu.com
kenyatraveltips.com
How common are these? VERY common! Especially when redirecting from http:// to https://. However, sometimes a 301 redirects to a brand new domain name. Most of the time it's a company rebranding, but sometimes it's a completely different company that has bought the domain name (that's usually just been made available), then they do a 301 redirect to their own website. This is something we have to look out for, because the website we review can end up redirecting to another website that has nothing to do with the website we reviewed! This involves both an automated process to record the 301 redirect, and ultimately a human-review process to ensure the new domain name belongs to the same company as the previous domain name, and that the nature of the website is the same.
Even if it's just a regular http:// to https:// redirect, it's good housekeeping to update our directory to reflect that change.
301 redirects discovered in the last 14 days
https://www.clenthillsvets.co.uk/ 301'd to >>
https://www.clenthillsvets.co.uk/en-gb
[internal change]
Change discovered on 17 January 2022
https://www.fantasysports.co.uk/2016/12/16/arsenal-travel-to-manchester-city-for-gameweek-17/ 301'd to >>
https://fantasysports.co.uk/2016/12/16/arsenal-travel-to-manchester-city-for-gameweek-17/
https://www.travelcounsellors.co.uk/offers 301'd to >>
https://www.travelcounsellors.co.uk/error-page?aspxerrorpath=/offers
Change discovered on 5 January 2022
View full list of 301 redirects >>
Websites that are considered permanently offline
Sometimes it happens - a website is offline too long, and it is considered "dead". Therefore, we have to remove it from our directory. We have a page dedicated to websites we have removed from our directory for this reason. Note, that these are NOT websites we have rejected initially (that would be a MUCH bigger list, and it's not one we need to show here!).
Newly created and updated website content
We spider the listings in our directory constantly throughout the day, 24/7. Our spider is looking for new content: new articles, videos, social posts, new pages, new products - maybe an updated phone number or address. Maybe there's new accreditations or associations added to the site. If we find them, we update the listing.