<story><title>We reduced the cost of building Mastodon at Twitter-scale by 100x</title><url>https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the-cost-of-building-twitter-at-twitter-scale-by-100x/</url></story><parent_chain><item><author>endisneigh</author><text>I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks.&lt;p&gt;It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose.&lt;p&gt;What really irks me is that they go on and on bragging about the low LoC count and literally show nothing complete. They should’ve held on this post and released it simultaneously with the code.</text></item><item><author>sdwr</author><text>In a typical architecture, the DB stores data, and the backend calls the DB to make updates and compile views.&lt;p&gt;Here, the &amp;quot;views&amp;quot; are defined formally (the P-states), and incrementally, automatically updated when the underlying data changes.&lt;p&gt;Example problem:&lt;p&gt;Get a list of accounts that follow account 1306&lt;p&gt;&amp;quot;Classic architecture&amp;quot;:&lt;p&gt;- Naive approach. Search through all accounts follow lists for &amp;quot;1306&amp;quot;. Super slow, scales terribly with # of accounts.&lt;p&gt;- Normal approach. Create a &amp;quot;followed by&amp;quot; table, update it whenever an account follows &amp;#x2F; unfollows &amp;#x2F; is deleted &amp;#x2F; is blocked.&lt;p&gt;Normal sounds good, but add 10x features, or 1000x users, and it gets trickier. You need to make a new table for each feature, and add conditions to the update calls, and they start overlapping... Or you have to split the database up so it scales, but then you have to pay attention to consistency, and watch which order stuff gets updated in.&lt;p&gt;Their solution is separating the &amp;quot;true&amp;quot; data tables from the &amp;quot;view&amp;quot; tables, formally defining the relationship between the two, and creating the &amp;quot;view&amp;quot; tables magically behind the scenes.</text></item><item><author>dataangel</author><text>I do C++ backend work in a non-web industry and this entire post is Greek to me. Even though this is targeted at developers, you need a better pitch. I get &amp;quot;we did this 100x faster&amp;quot; but the obvious followup question is &amp;quot;how&amp;quot; but then the answer seems to be a ton of flow diagrams with way too many nodes that tell me approximately nothing and some handwaving about something called P-States that are basically defined to be entirely nebulous because they are any kind of data structure.&lt;p&gt;I&amp;#x27;m not saying there&amp;#x27;s nothing here, but I am adjacent to your core audience and I have no idea whether there is after reading your post. I think you are strongly assuming a shared basis where everybody has worked on the same kind of large scale web app before; I would find it much more useful to have an overview of, &amp;quot;This what you would usually do, here are the problems with it, here is what we do instead&amp;quot; with side by side code comparison of Rama vs what a newbie is likely to hack together with single instance postgres.</text></item></parent_chain><comment><author>nathanmarz</author><text>We are very open in the post that the core concepts are not new:&lt;p&gt;&lt;pre&gt;&lt;code&gt;  Individually, none of these concepts are new. I’m sure you’ve seen them all before. You may be tempted to dismiss Rama’s programming model as just a combination of event sourcing and materialized views. But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity that characterize and overwhelm existing systems.
&lt;/code&gt;&lt;/pre&gt;
Indexes as arbitrary data structures that you shape to perfectly meet your use cases, a powerful computation API that&amp;#x27;s like a &amp;quot;distributed programming language&amp;quot;, and everything being so integrated make a world of difference.&lt;p&gt;I understand the desire to see all the code, and that&amp;#x27;s coming in two weeks. That said, the code in the post isn&amp;#x27;t trivial as it&amp;#x27;s showing almost the complete implementations of two major parts of Mastodon: the social graph and timeline fanout.&lt;p&gt;Next week you&amp;#x27;ll be able to play with Rama when we release a build of it, and the documentation will help with that.</text></comment>
<story><title>We reduced the cost of building Mastodon at Twitter-scale by 100x</title><url>https://blog.redplanetlabs.com/2023/08/15/how-we-reduced-the-cost-of-building-twitter-at-twitter-scale-by-100x/</url></story><parent_chain><item><author>endisneigh</author><text>I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks.&lt;p&gt;It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose.&lt;p&gt;What really irks me is that they go on and on bragging about the low LoC count and literally show nothing complete. They should’ve held on this post and released it simultaneously with the code.</text></item><item><author>sdwr</author><text>In a typical architecture, the DB stores data, and the backend calls the DB to make updates and compile views.&lt;p&gt;Here, the &amp;quot;views&amp;quot; are defined formally (the P-states), and incrementally, automatically updated when the underlying data changes.&lt;p&gt;Example problem:&lt;p&gt;Get a list of accounts that follow account 1306&lt;p&gt;&amp;quot;Classic architecture&amp;quot;:&lt;p&gt;- Naive approach. Search through all accounts follow lists for &amp;quot;1306&amp;quot;. Super slow, scales terribly with # of accounts.&lt;p&gt;- Normal approach. Create a &amp;quot;followed by&amp;quot; table, update it whenever an account follows &amp;#x2F; unfollows &amp;#x2F; is deleted &amp;#x2F; is blocked.&lt;p&gt;Normal sounds good, but add 10x features, or 1000x users, and it gets trickier. You need to make a new table for each feature, and add conditions to the update calls, and they start overlapping... Or you have to split the database up so it scales, but then you have to pay attention to consistency, and watch which order stuff gets updated in.&lt;p&gt;Their solution is separating the &amp;quot;true&amp;quot; data tables from the &amp;quot;view&amp;quot; tables, formally defining the relationship between the two, and creating the &amp;quot;view&amp;quot; tables magically behind the scenes.</text></item><item><author>dataangel</author><text>I do C++ backend work in a non-web industry and this entire post is Greek to me. Even though this is targeted at developers, you need a better pitch. I get &amp;quot;we did this 100x faster&amp;quot; but the obvious followup question is &amp;quot;how&amp;quot; but then the answer seems to be a ton of flow diagrams with way too many nodes that tell me approximately nothing and some handwaving about something called P-States that are basically defined to be entirely nebulous because they are any kind of data structure.&lt;p&gt;I&amp;#x27;m not saying there&amp;#x27;s nothing here, but I am adjacent to your core audience and I have no idea whether there is after reading your post. I think you are strongly assuming a shared basis where everybody has worked on the same kind of large scale web app before; I would find it much more useful to have an overview of, &amp;quot;This what you would usually do, here are the problems with it, here is what we do instead&amp;quot; with side by side code comparison of Rama vs what a newbie is likely to hack together with single instance postgres.</text></item></parent_chain><comment><author>sixo</author><text>The difference is that the materialized-view logic lives naturally in the application code; there&amp;#x27;s no step where they go out of the DB to do computations and then reinsert.&lt;p&gt;Once SQL materialized views aren&amp;#x27;t enough, you might do this by replicating your database into Kafka, implementing logic in Flink or something, and reinserting into the same DB&amp;#x2F;Elasticsearch&amp;#x2F;etc. Very common architecture. (Writ small, could also use a queue processor like RabbitMQ.)&lt;p&gt;Their approach is to instead--apparently--make all of these first-class elements of the same ecosystem, not by &amp;quot;putting it all in the database&amp;quot;, but by putting the database into the application code. Which seems wild, but colocates data, transformation, and view.&lt;p&gt;Seems like it would open up a lot of cans of worms, but if you solve those, sounds great.</text></comment>