<story><title>When AWS Autoscale Doesn’t</title><url>https://segment.com/blog/when-aws-autoscale-doesn-t/</url></story><parent_chain><item><author>DVassallo</author><text>The way I&amp;#x27;ve been happiest using EC2 Auto Scaling was to have a single cron-job continuously calculating how many instances I should be running, and it sets the desired capacity manually with the Auto Scaling API[1]. This may seem to defeat the purpose of Auto Scaling, but it&amp;#x27;s actually much more convenient than spinning up&amp;#x2F;down EC2 instances with the EC2 API. You get to precisely control how to scale, and won&amp;#x27;t be at the mercy of the Auto Scaling heuristics.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;docs.aws.amazon.com&amp;#x2F;autoscaling&amp;#x2F;ec2&amp;#x2F;userguide&amp;#x2F;as-manual-scaling.html&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;docs.aws.amazon.com&amp;#x2F;autoscaling&amp;#x2F;ec2&amp;#x2F;userguide&amp;#x2F;as-man...&lt;/a&gt;</text></item></parent_chain><comment><author>samstave</author><text>So we had this cause a spectacular outage a few years ago.&lt;p&gt;We were doing exactly this - but we had a flaw: we didnt handle the case when the AWS API was actually down.&lt;p&gt;So we were constantly monitoring for how many running instances we had - but when the API went down, just as we were ramping up for our peak traffic - the system thought that none were running because the API was down - so it just kept continually launching instances.&lt;p&gt;The increased scale of instances pummeled the control plane 
 with thousands of instances all trying to come online and pull down their needed data to get operational -- which them killed our DBs, pipeline etc...&lt;p&gt;We had to reboot our entire production environment at peak service time...</text></comment>
<story><title>When AWS Autoscale Doesn’t</title><url>https://segment.com/blog/when-aws-autoscale-doesn-t/</url></story><parent_chain><item><author>DVassallo</author><text>The way I&amp;#x27;ve been happiest using EC2 Auto Scaling was to have a single cron-job continuously calculating how many instances I should be running, and it sets the desired capacity manually with the Auto Scaling API[1]. This may seem to defeat the purpose of Auto Scaling, but it&amp;#x27;s actually much more convenient than spinning up&amp;#x2F;down EC2 instances with the EC2 API. You get to precisely control how to scale, and won&amp;#x27;t be at the mercy of the Auto Scaling heuristics.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;docs.aws.amazon.com&amp;#x2F;autoscaling&amp;#x2F;ec2&amp;#x2F;userguide&amp;#x2F;as-manual-scaling.html&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;docs.aws.amazon.com&amp;#x2F;autoscaling&amp;#x2F;ec2&amp;#x2F;userguide&amp;#x2F;as-man...&lt;/a&gt;</text></item></parent_chain><comment><author>cdoxsey</author><text>This is how the cluster-autoscaler works in kubernetes. It sets the desired capacity based on the number of pods needing to be scheduled.&lt;p&gt;Coupled with a horizontal pod autoscaler (which sets the replica count based on a metric) you get the best of both worlds.</text></comment>