<story><title>My £4 a month server can handle 4.2M requests a day</title><url>https://mark.mcnally.je/blog/post/My%20%C2%A34%20a%20month%20server%20can%20handle%204.2%20million%20requests%20a%20day</url></story><parent_chain><item><author>mynameisash</author><text>What Andy giveth, Bill taketh away.[0]&lt;p&gt;I&amp;#x27;m more than a little annoyed that so much data engineering is still done in Scala Spark or PySpark. Both suffer from pretty high memory overhead, which leads to suboptimal resource utilization. I&amp;#x27;ve worked with a few different systems that compile their queries into C&amp;#x2F;C++ (which is transparent to the developer). Those tend to be significantly faster or can use fewer nodes to process.&lt;p&gt;I get that quick &amp;amp; dirty scripts for exploration don&amp;#x27;t need to be super optimized, and that throwing more hardware at the problem _can_ be cheaper than engineering time, but in my experience, the latter ends up costing my org tens of millions of dollars annually -- just write some code and allocate a ton of resources to make it work in a reasonable amount of time.&lt;p&gt;I&amp;#x27;m hopeful that Ballista[1], for example, will see uptake and improve this.&lt;p&gt;[0] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Andy_and_Bill%27s_law&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Andy_and_Bill%27s_law&lt;/a&gt;&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;apache&amp;#x2F;arrow-datafusion&amp;#x2F;tree&amp;#x2F;master&amp;#x2F;ballista&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;apache&amp;#x2F;arrow-datafusion&amp;#x2F;tree&amp;#x2F;master&amp;#x2F;balli...&lt;/a&gt;</text></item><item><author>nostrademons</author><text>People tend to severely underestimate how fast modern machines are and overestimate how much you need to spend on hardware.&lt;p&gt;Back in my last startup, I was doing a crypto market intelligence website that subscribed to full trade &amp;amp; order book feeds from the top 10 exchanges.  It handled about 3K incoming messages&amp;#x2F;second (~260M per day), including all of the message parsing, order book update, processing, streaming to websocket connections on any connected client, and archival to PostGres for historical processing.  Total hardware required was 1 m4.large + 1 r5.large AWS instances, for a bit under $200&amp;#x2F;month, and the boxes would regularly run at about 50% CPU.</text></item></parent_chain><comment><author>Spooky23</author><text>I get a kick out of stuff like this - I’m mostly an exec these days, but I recently prototyped a small database system to feed a business process in SQLite on my laptop.&lt;p&gt;To my amusement, my little SQLite prototype smoked the “enterprise” database. Turns out that a MacBook Pro SSD performs better than the SAN, and the query planner needs more tlc. We ended up running the queries off my laptop for a few days while the DBAs did their thing.</text></comment>
<story><title>My £4 a month server can handle 4.2M requests a day</title><url>https://mark.mcnally.je/blog/post/My%20%C2%A34%20a%20month%20server%20can%20handle%204.2%20million%20requests%20a%20day</url></story><parent_chain><item><author>mynameisash</author><text>What Andy giveth, Bill taketh away.[0]&lt;p&gt;I&amp;#x27;m more than a little annoyed that so much data engineering is still done in Scala Spark or PySpark. Both suffer from pretty high memory overhead, which leads to suboptimal resource utilization. I&amp;#x27;ve worked with a few different systems that compile their queries into C&amp;#x2F;C++ (which is transparent to the developer). Those tend to be significantly faster or can use fewer nodes to process.&lt;p&gt;I get that quick &amp;amp; dirty scripts for exploration don&amp;#x27;t need to be super optimized, and that throwing more hardware at the problem _can_ be cheaper than engineering time, but in my experience, the latter ends up costing my org tens of millions of dollars annually -- just write some code and allocate a ton of resources to make it work in a reasonable amount of time.&lt;p&gt;I&amp;#x27;m hopeful that Ballista[1], for example, will see uptake and improve this.&lt;p&gt;[0] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Andy_and_Bill%27s_law&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Andy_and_Bill%27s_law&lt;/a&gt;&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;apache&amp;#x2F;arrow-datafusion&amp;#x2F;tree&amp;#x2F;master&amp;#x2F;ballista&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;apache&amp;#x2F;arrow-datafusion&amp;#x2F;tree&amp;#x2F;master&amp;#x2F;balli...&lt;/a&gt;</text></item><item><author>nostrademons</author><text>People tend to severely underestimate how fast modern machines are and overestimate how much you need to spend on hardware.&lt;p&gt;Back in my last startup, I was doing a crypto market intelligence website that subscribed to full trade &amp;amp; order book feeds from the top 10 exchanges.  It handled about 3K incoming messages&amp;#x2F;second (~260M per day), including all of the message parsing, order book update, processing, streaming to websocket connections on any connected client, and archival to PostGres for historical processing.  Total hardware required was 1 m4.large + 1 r5.large AWS instances, for a bit under $200&amp;#x2F;month, and the boxes would regularly run at about 50% CPU.</text></item></parent_chain><comment><author>Waterluvian</author><text>What reminded me of this the other day is how MacOS will grow your cursor if you “shake” it to help you find it on a big screen.&lt;p&gt;I was thinking about how they must have a routine that’s constantly taking mouse input, buffering history, and running some algorithm to determine when user input is a mouse “shake”.&lt;p&gt;And how many features like this add up to eat up a nontrivial amount of resources.</text></comment>