wiredfool

Some days you win, some days Murphy does. And sometimes Murphy takes all your processor, memory, and disk bandwith.

My day job uses systems that (among other things) can send lots of email quickly. It’s designed so that for each mailing, at most one person will get a duplicate. (for those of you wondering, it’s all extremely opt in. period.) This is all fine and dandy until the database stops saving, starts eating memory and blows up at 2 am taking the rest of the server down with it. Then I get to reconstruct it from transaction logs. That’s why they’re there, but they’re generally used to ensure that the minutes between saves aren’t lost to Murphy, instead of 8 hours.

At least it’s recoverable. It could always be worse. It’s all ok as long as I can unplug a network cable or get console access. It’s just a real bummer when it’s 2am and the network cable is 15 minutes away. But on the bright side, UPS brought batteries yesterday, so at least the power was clean.