Let's say we have a website www.example.com with the following pages:www.example.com/page1www.example.com/unknownwww.example.com/unknownIs there a Wget command that would produce the following output:page1page2page3How would this command change if the website structure was:www.example.com/xxxx/page1www.example.com/xxxx/page2www.example.com/xxxx/page3Essentially I have a server ip address and would like to be able to list all the pages held on the server, to see if there are any I am interested in to download.For instance I can do wget -r www.example.com/page1 and I want to know if there are any other directories (page2, page3, etc) that I might be interested in.I have researched the --spider option and others, but with no joy.
Wget command that returns [a list of] all webpages on a webserver
directory;wget
You can tell wget to recursively download an entire website, but it does so by following the links on each page. If it doesn't know page1, page2 and page3 are there, it will never retrieve them.Put bluntly, HTTP does not work that way -- fortunately.