<story><title>Storing 50M events per second in Elasticsearch</title><url>https://datadome.co/store-50-million-event-per-second-in-elasticsearch/</url></story><parent_chain></parent_chain><comment><author>jungletime</author><text>3 years ago, I made a simple calendar app in django, and I wanted to use Elasticsearch so users can search and find an event, and to use it to populate an upcoming events list. There&amp;#x27;s only about 10,000 events in the database.&lt;p&gt;I quickly realized what a pain it is to use Elasticsearch, for a simple app like mine.&lt;p&gt;Pain points:&lt;p&gt;1) You have to setup and recreate part of your database in elastic search. So you essentially end up with two databases. Which now you have to keep in sync.&lt;p&gt;2) I was getting unpredictable query results from Elasticsearch, which after a few days, and much head scratching turned out to be that I was running out of memory.&lt;p&gt;3) When a user added a new event, it was not being added to elastcsearch index automatically. I could not figure out how to do this reliably. I could make it work reliably only after a sync of the entire Elasticsearch index. But this meant that it was next to useless, to use for the Upcoming Events List.  
Since I only wanted to sync the index once a day. Confusing the users, as to why their event was not showing up. And I gave up, and just ended up implementing the Upcoming Events List directly from my database in python.&lt;p&gt;4) Elasticsearch came without some security settings not set by default, and after a few months it was hacked.
I had to download a new version and wasted more time.&lt;p&gt;I still use Elasticsearch, but only for search, and not the upcoming event list. And I don&amp;#x27;t think it was worth the complexity that it added to my project.</text></comment>
<story><title>Storing 50M events per second in Elasticsearch</title><url>https://datadome.co/store-50-million-event-per-second-in-elasticsearch/</url></story><parent_chain></parent_chain><comment><author>mistrial9</author><text>it appears in this document:&lt;p&gt;* DataDome is a security company, and gets web traffic in near real-time for clients; a lot of traffic in some cases with very specific numbers given, like daily peak loads.&lt;p&gt;* DataDome only retains records for 30 days, and the most attention is given to the most recent traffic, to detect attacks&lt;p&gt;* an ElasticSearch deployment records all of the traffic records downstream from Apache Flink; a new feature added to ES this year, improves the management of ES indexing, and that solved problems that DataDome was having.. things are better! write an engineering blog post !&lt;p&gt;* re-indexing is done nightly, and implemented in a cloud environment that can handle the (heavy) work to rebuild the indexing.&lt;p&gt;These numbers are impressive. Earlier criticisms of ES are being addressed, and ES is stable and a cornerstone of the architecture. A company called DataDome is providing real services in near real-time. Congratulations to the team and an interesting read.</text></comment>