Most of the sitemap generators I've found all scan my internal files and base it on that. However using apache and htaccess I have many aliases for things for example:/brand/model/photos/ is empty directory wise, but on the web it relocates it to a zenphoto gallery using htaccess.http://www.xml-sitemaps.com/ does What I need, but I'd like to have more control over it and I need it to be dynamic and run weekly or daily. Allow for specific change frequencies based on the file/directory. I also want to be able to make a pretty sitemap html file.Counting the photo gallery I have around 300 links, without the photo gallery more like 100.I have MySQL, PHP 5.3 installed.Update:I am using Debian Linux, so I'll need a crawler that can be run with cron or automated through linux.
What are some client-side sitemap generators?
sitemap
The best solution is to just generate dynamic sitemaps server-side. You have access to all of the application code and rewrite rules, so it doesn't really make sense to use a spidering tool to crawl your site. zenphoto has a plugin for generating sitemaps, as do most other CMSes. Just use a sitemap index to link to those sitemaps.But if you still want to use a webcrawler, there are a few listed here.Some people at Google have also developed a pretty ingenious sitemap tool called the Google Sitemap Generator. Instead of spidering the site (which is just the same thing that Google/Bing/Yahoo already do), it monitors server traffic to see how users access your pages and creates sitemaps that way.