<story><title>Billions of Messages a Day – Yelp&apos;s Real-Time Data Pipeline</title><url>http://engineeringblog.yelp.com/2016/07/billions-of-messages-a-day-yelps-real-time-data-pipeline.html?hn=hn</url></story><parent_chain><item><author>z3t4</author><text>100 million reviews&amp;#x2F;year is only 3 reviews per second on average. Sure, they seem to do more then just that, like voting, comments, etc. But it still seems like something an old school stack could handle on a single large instance.&lt;p&gt;Reading between the lines it seems the problem wasn&amp;#x27;t scaling, but programmer productivity. Smaller code bases is often easier to work with so I guess they solved that by dividing it up into many small services. The blog could use a more detailed description of the problem they are actually solving.</text></item></parent_chain><comment><author>TeMPOraL</author><text>Whenever I see headlines like these, &amp;quot;X billion messages per day&amp;#x2F;hour&amp;#x2F;second in some web service Y&amp;quot;, I wonder what the hell is that service doing that it generates so much messages? I.e. I could understand Facebook, with its billion+ of users and built-in messaging platform, could generate billion of &amp;quot;messages&amp;quot; a day. But a mostly read-based service like Yelp?&lt;p&gt;But I finally realized - those messages are probably mostly tracking, ads, more tracking, some infrastructure work and even more ads &amp;amp; tracking. The sausage machine that turns people into money.</text></comment>
<story><title>Billions of Messages a Day – Yelp&apos;s Real-Time Data Pipeline</title><url>http://engineeringblog.yelp.com/2016/07/billions-of-messages-a-day-yelps-real-time-data-pipeline.html?hn=hn</url></story><parent_chain><item><author>z3t4</author><text>100 million reviews&amp;#x2F;year is only 3 reviews per second on average. Sure, they seem to do more then just that, like voting, comments, etc. But it still seems like something an old school stack could handle on a single large instance.&lt;p&gt;Reading between the lines it seems the problem wasn&amp;#x27;t scaling, but programmer productivity. Smaller code bases is often easier to work with so I guess they solved that by dividing it up into many small services. The blog could use a more detailed description of the problem they are actually solving.</text></item></parent_chain><comment><author>nirmel</author><text>100 million is cumulative, not annual.</text></comment>