I just finished the nastiest debugging experience of my career–nearly 3 weeks on a single bug. After days and days of staring at code, swearing at core dumps, tailing logs, connecting to gdbserver via a multi-hop ssh tunnel from inside a secure environment, and general programmer misery, I felt like doing cartwheels when I finally figured it out, tweaked a few lines of code, and observed stability again.

Hindsight teaches me this lesson: undocumented, unhandled constraints waste enormous amounts of time and energy. If you’re interested in writing good code, you must know your limits, and you must communicate them. This especially matters when the constraints are obscure or surprising.

Naive optimism

My bug seemed simple enough at first blush: Continue reading →