<story><title>Debugging an evil Go runtime bug: From heat guns to kernel compiler flags (2017)</title><url>https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/</url></story><parent_chain></parent_chain><comment><author>Terr_</author><text>&amp;gt; Over the course of 22 kernel builds, I managed to simplify the config so much that the kernel had no networking support, no filesystems, no block device core, and didn’t even support PCI (still works fine on a VM though!).&lt;p&gt;Flashbacks to a job where I was asked to figure out why a newer kernel was crashing. This was a very frustrating time, because I had (have) basically zero real C&amp;#x2F;C++  experience but I&amp;#x27;d helped out with Bitbake recipes and everyone else was busy or moved to other projects.&lt;p&gt;To cut a multiweek tale of dozens of recompilations short: The kernel was fine. The headless custom hardware was fine. The problem was a  hypervisor misconfiguration,  overwriting part of the kernel address space. All of our kernels have been corrupt, but this was the first one where the layout meant it mattered.&lt;p&gt;A month of frustration, two characters to fix, the highest ratio I&amp;#x27;ve encountered so far.&lt;p&gt;My reward for struggling through a complex problem I was unqualified for? &amp;quot;Great, now we need to backport security patches from the main Linux kernel to the SoC vendor&amp;#x27;s custom fork...&amp;quot;</text></comment>
<story><title>Debugging an evil Go runtime bug: From heat guns to kernel compiler flags (2017)</title><url>https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/</url></story><parent_chain></parent_chain><comment><author>BobbyJo</author><text>This is honestly wild. 99% of devs would have found a work around and moved on. Going so far as to create a multi-kernel test bench to narrow down the source of the instability is a level of dedication I have not personally seen, and I respect it.</text></comment>