<story><title>Running a database on EC2? Your clock could be slowing you down</title><url>https://heapanalytics.com/blog/engineering/clocksource-aws-ec2-vdso</url></story><parent_chain><item><author>scarface74</author><text>Why would you run your own Postgres instance on EC2 within AWS? That kind of defeats the purpose of paying for AWS.  Why not use Postgres RDS or Aurora?&lt;p&gt;It makes some sense with Sql Server and Oracle in a few cases because of licensing but hosting your own Postgres instance on AWS is the worse of both worlds -- you&amp;#x27;re paying more than with a cheaper VPS and you have to do all of the maintenance yourself and not taking advantage of all of things that AWS provides -- point in time restores, easy cross region read replicas, faster disk io (Aurora), etc.</text></item></parent_chain><comment><author>malisper</author><text>Hi. I&amp;#x27;m one of the database engineers at Heap. This is a good question. There are several reasons why we use EC2. First of all, I will say I love RDS as a product. We actually do use RDS for a number of our services. We use Postgres on EC2 only for our primary data store. As for reasons why we use EC2:&lt;p&gt;Cost - Our primary data store has &amp;gt;1 Petabyte of raw data stored across dozens of Postgres instances. The amount of data we store is at the point where RDS is too expensive for us. The cost of an instance on RDS is more than twice the cost on EC2. For example, an on-demand r4.8xl on EC2 instance costs $2.13 an hour, while an RDS r4.8xl costs $4.80 an hour.&lt;p&gt;Performance - The only kind of disk available on RDS is EBS. EBS is slow compared to the NVMe the i3s provide. We used to use r3s with EBS and got a major speedup when we switched to i3s. As a side note, the cost of an i3 is also less than the cost of an r3 with an equivalent amount of EBS.&lt;p&gt;Configuration - By using EC2 we can configure our machines in ways we wouldn&amp;#x27;t be able to if we used RDS. For example, we run ZFS on our EC2 instances which compresses our data by 2x. By compressing our data, we get a major cost saving and a major performance boost at the same time! There isn&amp;#x27;t an easy way to compress your data if you use RDS.&lt;p&gt;Introspection - There are times where we&amp;#x27;ve needed to debug performance problems with Postgres and EXPLAIN ANALYZE won&amp;#x27;t suffice. A good example is we used flame graphs to see what Postgres was using CPU for. We made a small change that resulted in a 10x improvement to ingestion throughput. If you are curious, I wrote a blog post on this investigation: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;heapanalytics.com&amp;#x2F;blog&amp;#x2F;engineering&amp;#x2F;basic-performance-analysis-saved-us-millions&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;heapanalytics.com&amp;#x2F;blog&amp;#x2F;engineering&amp;#x2F;basic-performance...&lt;/a&gt;</text></comment>
<story><title>Running a database on EC2? Your clock could be slowing you down</title><url>https://heapanalytics.com/blog/engineering/clocksource-aws-ec2-vdso</url></story><parent_chain><item><author>scarface74</author><text>Why would you run your own Postgres instance on EC2 within AWS? That kind of defeats the purpose of paying for AWS.  Why not use Postgres RDS or Aurora?&lt;p&gt;It makes some sense with Sql Server and Oracle in a few cases because of licensing but hosting your own Postgres instance on AWS is the worse of both worlds -- you&amp;#x27;re paying more than with a cheaper VPS and you have to do all of the maintenance yourself and not taking advantage of all of things that AWS provides -- point in time restores, easy cross region read replicas, faster disk io (Aurora), etc.</text></item></parent_chain><comment><author>aidos</author><text>In my experience you get much better performance outside of RDS and you can inspect and tune it better. Maybe I’m missing something and no doubt I could put more work into it but we’ve actually talked about moving our RDS dbs back to EC2 because there are plenty of queries we do that are embarrassingly slow on RDS when they shouldn’t be.&lt;p&gt;Also, you can’t replicate out of RDS. I like to know where my data is and how to bring it back online during a disaster.</text></comment>