<story><title>Details of yesterday&apos;s Bunny CDN outage</title><url>https://bunny.net/blog/the-stack-overflow-of-death-dns-collapse/</url></story><parent_chain></parent_chain><comment><author>ram_rar</author><text>&amp;gt; On June 22nd at 8:25 AM UTC, we released a new update designed to reduce the download size of the optimization database. Unfortunately, this managed to upload a corrupted file to the Edge Storage.&lt;p&gt;I wonder, if simple checksum verification of the file would have helped in avoiding this outage all together.&lt;p&gt;&amp;gt; Turns out, the corrupted file caused the BinaryPack serialization library to immediately execute itself with a stack overflow exception, bypassing any exception handling and just exiting the process. Within minutes, our global DNS server fleet of close to a 100 servers was practically dead&lt;p&gt;This is exactly, why one needs a canary based deployments. I have seen umpteen amounts of issues being caught in canary, which has saved my team tons of firefighting time.</text></comment>
<story><title>Details of yesterday&apos;s Bunny CDN outage</title><url>https://bunny.net/blog/the-stack-overflow-of-death-dns-collapse/</url></story><parent_chain></parent_chain><comment><author>string</author><text>Good and clear explanation. This is a risk you take when you use a CDN, I still think the benefits outweigh the occasional downtime. I&amp;#x27;m a big fan of BunnyCDN, they&amp;#x27;ve saved me a lot of money over the past few years.&lt;p&gt;I&amp;#x27;m sure I&amp;#x27;d be fuming if I worked at some multi-million dollar company but as someone that mainly works for smaller businesses it&amp;#x27;s not the end of the world, I suspect most of my clients haven&amp;#x27;t even noticed yet.</text></comment>