I'm planning out a system for a data-processing pipeline, which:ingests various data sets from 3rd party sourcesstandardizes the data into a common formattransforms the data in a variety of ways, depending on the data setposts the transformed data to a DBMy requirements are:I need each process to be independently scalableI need to be able to fork processes (eg. an incoming data set gets transformed in multiple different ways, or sent to multiple places)I need to keep a history of past data, as some data transformations depend on the data history being in a certain state.I'd like to have each transform done by dumb worker processes, which don't need to know anything about the specific data set, or the current state of the data in in the pipeline. This way, workers can be reused by multiple pipelines. And I can keep all of my state management outside of my workers, in a centralized mediator process.I've been looking at Amazon Simple Workflow Service (SWF) for inspiration on how to proceed. The pattern I'm seeing is:Worker processes, that intake data via events, process the data, and send events to a mediatorA central mediator, that gets events from all the workers, makes decisions about the events, and sends out events with instructions on how to proceed to the worker processes.I have some hesitation about using SWF itself, but I think I may be able to adapt pieces of the pattern for my own use. I'd like to do more research on this type of architecture, but I haven't found much information on it. I'm thinking it would help if I knew what to call it.How would you call this pattern? Workflow? Pipeline? Traffic Controller? Event Bus? Some mixture of those?
What is this pattern? Data processing pipeline with SQS + Traffic Controller, similar to AWS SWF
design patterns;architecture;workflows;stream processing