I'm programming a small web scraper in python which I want to speed up by parallelizing things. The scraper is crawling URLs whereby a single URL can represent an 'item' or an 'index'. An index in this case is a list of URLs of other items and indexes. The parsing for an item URL differs from the parsing of an index URL.So for every URL the following happens:fetch source from URLcheck if URL is index or itemparse source (depending on index or item)I'm now thinking of two different solutions for parallelizing the process:a)  use a queue of URLs, a worker process takes one URL and does 1 - 3 sequentialb) use a queue of tasks ( e.g. fetch source), a worker process does only 1, 2 or 3I think a) is more logical and maybe better readable in the implementation. Also I don't know if one solution is faster then the other.What would be the best way to realize my project?
Relating to design and runtime - is it better to parallelize smaller sub-tasks or bundled tasks?
design patterns;architecture;python;patterns and practices;architectural patterns