<story><title>DDoS Attack Against Dyn Managed DNS</title><url>https://www.dynstatus.com/incidents/nlr4yrr162t8</url></story><parent_chain><item><author>tim_armandpour</author><text>I wanted to provide an update on the PagerDuty service. At this time we have been able to restore the service by migrating to our secondary DNS provider. If you are still experiencing issues reaching any pagerduty.com addresses, please flush your DNS cache. This should restore your access to the service. We are actively monitoring our service and are working to resolve any outstanding issues.  We sincerely apologize for the inconvenience and thank our customers for their support and patience. Real-time updates on all incidents can be found on our status page and on Twitter at @pagerdutyops and @pagerduty. In case of outages with our regular communications channels, we will update you via email directly.&lt;p&gt;In addition you can reach out to our customer support team at support@pagerduty.com or +1 (844) 700-3889.&lt;p&gt;Tim Armandpour, SVP of Product Development, PagerDuty</text></item></parent_chain><comment><author>pfarnsworth</author><text>I had the privilege of being on-call during this entire fiasco today and I have to say I was really really disappointed.  It&amp;#x27;s surprising how broken your entire service was when DNS went down.  I couldn&amp;#x27;t acknowledge anything, and my secondary on-call was getting paged because it looked like I wasn&amp;#x27;t trying to respond.  I was getting phone calls for alerts that wasn&amp;#x27;t even showing up on the web client, etc.  Overall, it caused chaos and I was really disappointed.</text></comment>
<story><title>DDoS Attack Against Dyn Managed DNS</title><url>https://www.dynstatus.com/incidents/nlr4yrr162t8</url></story><parent_chain><item><author>tim_armandpour</author><text>I wanted to provide an update on the PagerDuty service. At this time we have been able to restore the service by migrating to our secondary DNS provider. If you are still experiencing issues reaching any pagerduty.com addresses, please flush your DNS cache. This should restore your access to the service. We are actively monitoring our service and are working to resolve any outstanding issues.  We sincerely apologize for the inconvenience and thank our customers for their support and patience. Real-time updates on all incidents can be found on our status page and on Twitter at @pagerdutyops and @pagerduty. In case of outages with our regular communications channels, we will update you via email directly.&lt;p&gt;In addition you can reach out to our customer support team at support@pagerduty.com or +1 (844) 700-3889.&lt;p&gt;Tim Armandpour, SVP of Product Development, PagerDuty</text></item></parent_chain><comment><author>patrickg_zill</author><text>Sorry if this sounds dickish, but renting 3 servers @ $75 apiece from 3 different dedicated server companies in the USA, putting TinyDNS on them, and using them as backup servers, would have solved your problems hours ago.&lt;p&gt;Even a single quad-core server with 4GB RAM running TinyDNS could serve 10K queries per second, based on extrapolation and assumed improvements since this 2001 test, which showed nearly 4K&amp;#x2F;second performance on 700Mhz PIII CPUs:  &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;lists.isc.org&amp;#x2F;pipermail&amp;#x2F;bind-users&amp;#x2F;2001-June&amp;#x2F;029457.html&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;lists.isc.org&amp;#x2F;pipermail&amp;#x2F;bind-users&amp;#x2F;2001-June&amp;#x2F;029457....&lt;/a&gt;&lt;p&gt;EDIT to add: and lengthening TTLs temporarily would mean that those 10K queries would quickly lessen the outage, since each query might last for 12 hours; and large ISPs like Comcast would cache the queries for all their customers, so a single successful query delivered to Comcast would have (some amount) of multiplier effect.</text></comment>