If I want my website to be crawled, do I need an empty disallow?Is there any difference betweenUser-agent: *Disallow:Disallow: /folder/and User-agent: *Disallow: /folder/I have seen a robots.txt where the first option is used but I dont understand why.
To allow crawling of all but a specific folder, do I need to include an empty disallow directive in robots.txt?
robots.txt