textbooks, references for crawl+ingestion architecture
babyshake: You might want to flesh out your description a bit more, since it's somewhat generic as is.Maybe Stackoverflow.com would be a good venue for this type of question.