Here my scenario: I have a client application that needs a dedicated powerful computation system that every seconds process some data in input (few kilobyte) and return a few megabytes of processed data that the client displays in a certain way. The algorithm to process the data has a sort of precision parameter so we can, on a normal computer, process one second of data in one second with low resolution. Although if we want high resolution, it could take minutes. So that's why we need to distribute the load and we are looking into Apache Storm right now (even though Cuda or other systems are under evaluation so suggestions are appreciated but I'd like to get help for Storm here).I've been looking into Storm in general and some tests in both Aws and Azure but what I can't understand is how to have the source and the destination to be the same sprout and so our client application. So for example I could send the data to a sprout, the sprout partitions the data to many instances of a bolt, the processed data get saved in memory to Redis, the client accesses (or call an app that accesses) the Redis db. It seems to me the last 2 steps could be cut out and send back the processed/output data on the same channel the client sent the input data (meaning the sprout..)Ps: once data is sent back to the client, we don't need it anymore on the server/storm system and no point on be available to be requested in the future.
How can I configure storm to receive data from a client, process the data in bolts and then return the processed data to the same client as a stream?
latency;apache storm