I've been looking at most important sites and none of them have a sitemap.xml defined. My list includes:AmazonEbayYahooMicrosoft... and othersBut none of them include a sitemap.xml file. Why is this? It's not that important?Note: I've been using the Google Bot user agent to check it. The only site that returned a 200 Ok was google.com
Why important/big sites don't include a sitemap?
sitemap;xml sitemap
Just checking sitemap.xml is not enough.  There are several reasons that that check might fail even though the site has a sitemap. Did you check sitemap.xml.gz?  Google supports gzip of sitemaps.  Large sites with large sitemaps are likely to take advantage of this feature.You can specify the name of the sitemap file in robots.txt.  It doesn't have to be sitemap.xml.   A robots.txt can have a line like this that says where to get the sitemap. Sitemap: http://example.com/mysitemapfile.xmlIt is also possible to create a sitemap file at a secret URL.  You can tell Google about this URL through webmaster tools.  That way Google will get your sitemap, but it won't be easily accessible to the public.Looking in robots.txt, many of the sites you list do have sitemaps.AmazonSitemap: http://www.amazon.com/sitemap-manual-index.xmlSitemap: http://www.amazon.com/sitemap_vendor_videos_us.xmlSitemap: http://www.amazon.com/sitemap_vod_index.xmlSitemap: http://www.amazon.com/sitemaps.f3053414d236e84.SitemapIndex_0.xml.gzSitemap: http://www.amazon.com/sitemaps.1946f6b8171de60.SitemapIndex_0.xml.gzSitemap: http://www.amazon.com/sitemaps.bbb7d657c7e29fa.SitemapIndex_0.xml.gzSitemap: http://www.amazon.com/sitemaps.11aafed315ee654.SitemapIndex_0.xml.gzSitemap: http://www.amazon.com/sitemaps.c21f969b5f03d33.SitemapIndex_0.xml.gzEbaySitemap: http://www.ebay.com/lst/PDP_US_main_index.xmlSitemap: http://www.ebay.com/lst/PDP_US_incremental_index.xmlSitemap: http://www.ebay.com/lst/VIP_US_index.xmlSitemap: http://www.ebay.com/lst/SRP_US_index.xmlMicrosoft...Sitemap: http://www.microsoft.com/en-us/dynamics/sitemap.xmlSitemap: http://www.microsoft.com/en-us/kinectforwindows/sitemap.xmlSitemap: http://www.microsoft.com/en-us/sqlazure/sitemap.xmlSitemap: http://www.microsoft.com/en-us/cloud/sitemap.xmlSitemap: http://www.microsoft.com/en-us/server-cloud/sitemap.xml...