<story><title>Do We Need Distributed Stream Processing?</title><url>https://lsds.doc.ic.ac.uk/blog/do-we-need-distributed-stream-processing</url></story><parent_chain></parent_chain><comment><author>sriku</author><text>On another note - these kinds of &amp;quot;let&amp;#x27;s go back and check what we can do with a single node today&amp;quot; approaches are valuable to recalibrate our thinking once in a while. I didn&amp;#x27;t interpret this post as a case against distributed processing - there are many things going for it - but just to avoid falling into hype.&lt;p&gt;A couple of great posts by Frank McSherry along the same lines a few years ago, doing page rank for a 128bil edge graph on a single laptop (in the second post below) -&lt;p&gt;The first post - &amp;quot;scalability! But at what cost&amp;quot;
&lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;www.frankmcsherry.org&amp;#x2F;graph&amp;#x2F;scalability&amp;#x2F;cost&amp;#x2F;2015&amp;#x2F;01&amp;#x2F;15&amp;#x2F;COST.html&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;www.frankmcsherry.org&amp;#x2F;graph&amp;#x2F;scalability&amp;#x2F;cost&amp;#x2F;2015&amp;#x2F;01&amp;#x2F;...&lt;/a&gt;&lt;p&gt;The second post - &amp;quot;bigger data, same laptop&amp;quot;
&lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;www.frankmcsherry.org&amp;#x2F;graph&amp;#x2F;scalability&amp;#x2F;cost&amp;#x2F;2015&amp;#x2F;02&amp;#x2F;04&amp;#x2F;COST2.html&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;www.frankmcsherry.org&amp;#x2F;graph&amp;#x2F;scalability&amp;#x2F;cost&amp;#x2F;2015&amp;#x2F;02&amp;#x2F;...&lt;/a&gt;</text></comment>
<story><title>Do We Need Distributed Stream Processing?</title><url>https://lsds.doc.ic.ac.uk/blog/do-we-need-distributed-stream-processing</url></story><parent_chain></parent_chain><comment><author>cangencer</author><text>Single process &amp;quot;streaming&amp;quot; or basically CEP engines have been around for a very long time and used to be the norm before distributed stream processing engines came around (such as Storm). CEP engines have much richer functionality than distributed stream processing engines because they don&amp;#x27;t need to deal with partitioning or data distribution. I&amp;#x27;m not quite sure I see the appeal of making a non-distributed engine but with the same limitations of a distributed engine.&lt;p&gt;I work on Hazelcast Jet [1], which is a Java based distributed stream processing engine. The core engine is fast enough that it can be used with very good throughput on a single node (several times faster compared to Flink or Spark) but usually several nodes are not only needed strictly for parallelization but also for tolerating node failures and being able to restart where you left off. As others have pointed out, not every computation can be parallelised efficiently. Jet also offers in memory storage, so adding more nodes also increases your storage capacity.&lt;p&gt;Since the core of Jet is small enough (~400kb JAR), we also considered making a non-distributed version that runs strictly in process. Mainly for lightweight usage or embedding but would also offer a path to distributed execution, if it was ever needed.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;hazelcast&amp;#x2F;hazelcast-jet&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;hazelcast&amp;#x2F;hazelcast-jet&lt;/a&gt;</text></comment>