How would you build a high performance, semi-real-time stats service?
simonw: OK, I'll start. I should emphasize I don't have a particular project in mind just yet, but I do think it's an interesting thought exercise.For my purposes, I'm not particularly interested in saving a full record for every single hit for all time - I've already got that in web server logs. I'd much rather accumulate counters - they take up much less space and should be a lot cheaper to query when I want to generate graphs and reports.MongoDB looks like a tempting option here. It has an atomic increment option and from my initial testing appears to be really fast. Since I'm just incrementing counters, sharding writes across multiple instances should be pretty easy in the future (though that is true no matter what data store I use).Here's one potential scheme. For each page (or whatever is being counted), maintain a counter for every minute for the past two hours. Also maintain a counter for every hour for the past month, and a counter for every day for ever.When a hit comes in, increment the counter for that page on the current day, the current hour and the current minute. That's three counter updates.Every 24 hours, expire any minute counters from more than 24 hours ago, and any hour counters for more than two days ago.You might want to keep track of hits to an individual page, hits to that section of the site (or any of that user's pages) and hits to the site as a whole. That's fine - it just means you'll be updating 9 counters for every hit instead of 3.