What was the worst bug you've ever solved ?
jacquesm: Ok, here's one of mine, it's only fair.Jasper L. systems administrator of an early web hosting company calls up one evening, there is a problem with the paging system.A certain host is being paged as 'offline' but when checking the machine works fine.So I go there bit by bit we check out the software, everything works fine, but sure enough every 10 minutes or so the machine (called 'chopper', I'll remember that for the rest of my life) gets reported as 'down' again.But there is absolutely nothing wrong with it.After ruling out all the software bits we figure it must be hardware somehow. The way the supervisor works is it sends a ping packet to the machine, and if the machine responds it is deemed to be up. But chopper misses one ping out of every 5 or so, and sometimes several in a row.We swap out the computer, move the scsi drive to another box and boot it. Chopper registers as off-line, but works just fine.More confusion, finally, out of desperation we start messing with the network. This is all 10Base T, coaxial cables with little T connectors on the machines and a terminator at the end of the line on the last 'T' to make sure the impedance is right.The terminator was on, so that wasn't it.We went for a break at that point, we'd been at it for hours. Finally, by elimination we figure the only things we haven't changed yet are the drive and the 'T' connector, but surely that can't be it.And it was... somehow that T connector was a pretty good filter for some bits in the incoming or outgoing ping packet
 changing one of the bits, causing the IP checksum to fail. No returned ping... we replaced it back and forth 3 more times just to make sure we weren't seeing things.