It's always better to learn from someone else's mistakes than from your own. This weekend we feature Kyle Rankin and Bill Childers as they tell stories from their years as systems administrators. It's a win-win: you get to learn from their experiences, and they get to make snide comments to each other. 

by Kyle Rankin and Bill Childers

I was suffering, badly. We had just finished an all-night switch migration on our production Storage Area Network while I was hacking up a lung fighting walking pneumonia. Even though I did my part of the all-nighter from home, I was exhausted. So when my pager went off at 9am that morning, allowing me a mere four hours of sleep, I was treading dangerously close to zombie territory...

by Kyle Rankin and Bill Childers

Sometimes events and equipment conspire against you and your team to cause a problem. Occasionally, however, it's lack of understanding or foresight that can turn around and bite you. Unfortunately, this is a tale of where we failed to spot all the possible things that might go wrong.

by Kyle Rankin and Bill Childers

I was now at the next phase of troubleshooting: prayer. Somewhere around this time, I had my big breakthrough...

by Kyle Rankin and Bill Childers

As much as I love working with Linux and configuring software, one major part of being a sysadmin that always has appealed to me is working with actual hardware. There's something about working with tangible, physical servers that gives my job an extra dimension and grounds it from what might otherwise be a completely abstract job even further disconnected from reality. On top of all that, when you get a large shipment of servers, and you view the servers at your company as your servers, there is a similar anticipation and excitement when you open a server box as when you open Christmas presents at home. This story so happens to start during the Christmas season...