Scraping Answer Sites for ML Project?
tocomment: That's a cool project.  Let me know if you need any help.  I'm not sure what I could do but it sounds interesting.Here a couple ideas that might help (not that I condone violating a site's TOS).Have a random time interval between each page download so someone looking at the logs doesn't see a regular pattern.Pick 4 or 5 common user agent strings to alternate randomly between.Go through free proxies, or package up your program and have 5 or more friends run it on different sections of the data you want to download.