I have a website and I don't want this website to be visible on Google. So I deployed a robots.txt file. But the homepage of the website is still visible on Google. Here is the full code of my robots.txt copy-pasted below-User-agent: *Disallow: /cgi-bin/Disallow: /private/Disallow: /tmp/Disallow: /_/Disallow: /cgi-bin/Disallow: /index.htmlDisallow: /imagesDisallow: /about.html
What's wrong with my robots.txt file to block Google?
robots.txt
If you don't want to allowed to crawl your website on Google search result, then use this robots.txtUser-agent: *Disallow: /These will block all directories.But if many website links to your website, then Google will start displaying this snippet in future.Now if you don't want to index that website competely, then use noindex meta-tags/HTTP headers. That meta tags simply no index your all pages, but it is allowed to crawl. So if your main concern is about to, not visible in search result, then I highly recommended to use noindex tags.But don't use it both, because when the site is blocked by robots.txt then Google will not going to see your meta-tags/http headers.