entombed in the shrine of zeroes and ones

Nagios schmagios
Ron Echeverri's phone
wakes him up late with a
clamorous beep

Stumbling half-naked, he
somnambulistlically
fixes the problem and
goes back to sleep

In a painfully typical version of "en casa de herrero, cuchillo de palo", and despite two previous disk crashes, i finally made a backup of my primary domain server earlier this week on a disk i procured about two months ago.

From O'Reilly's LDAP System Administration:

Acknowledgements

At the end of every project, I am acutely aware that I could never have reached the end without the grace provided to me by God through my Savior, Jesus Christ. I hope He is proud of how I have spent my time.

While reading this book, you may find yourself feeling a little like a sky diver who has just jumped out of an airplane. As you approach the ground, things come more into focus. As you squint and try to make out the color of that house far below, you suddenly realize that you are plummeting closer and closer toward the very thing you are trying to observe.

While reading this book, you may find yourself feeling a little like you are about to die.

rone: Hmm, the production mail server is running an ancient version of Postfix. I need a newer version that supports milters. But this version of CentOS doesn't have a newer version of Postfix in its yum repository. Well, perhaps some enterprising person on the Net has filled my need. Let's ask Google.

Google: Here you go, a mailing list archive for CentOS.

Person with my same need: Hey, mailing list. Anyone have an RPM that fits my needs that i can use with yum?

Filthy nerd #1: If this is a dedicated mailserver, i prefer installing postfix from source.

Filthy nerd #2: If you are going to compile everything, why not use Gentoo?

rone: I am going to set these motherfuckers on fire.

I would like to introduce to Six Apart the concept that those of us in systems administration call a "staging environment".  In it, one stages new releases on a reasonable facsimile of the production environment (that is, the one with live users), in order to avoid exposing the production environment to any late bugs that arise during deployment.  Please run it by your Director of Operations.

(If they'd hired me a year and a half ago, i could've given them a head start on this.  Too bad, so sad.)

OK, you know when a bug is filed and steps to reproduce it are actually, helpfully, miraculously attached?  It would be in one's best interest to FOLLOW THEM.  Instead of taking the developers at their word and starting down a path that involves rewriting cookies at the load balancer as well as editing the application server's configuration, when faced by my cosysadmins' emphatic insistence that the load balancer cookie is not used by the application, i suggested that we reproduce the bug.  And as we take the steps, i see that the link that's being followed is going OFF-SITE, to one of our static content sites.  SO OF COURSE IT'S NOT GOING TO WORK.  NOW I SET EVERYONE ON FIRE.

It's a little difficult to track down errors in your application when, given three integration servers and four QA servers, no two servers have the same Java and Oracle memory settings.

It's my day off, seeing as how i spent an hour yesterday shutting down and unplugging all office equipment due to a planned building-wide power outage, and five hours today bringing it all back up.  Four power supplies decided they liked the rest too much to get back up again, so i had to go vulture some from the ranks of the undead computers in the back office.  One CAT-5 cable decided to spontaneously die, too, and the rigor mortis didn't let me extract it from its jack without specialized tools (a knife).  For bonus points, the outage was scheduled from 16:00 Saturday to 04:00 today, and they kindly called me just after 04:00 to let me know it all went well, and i spent the remaining hours of sleep dreaming of wholesale failure at work during today's activities.  To top it off, i woke up with a stiff neck, which made lugging around dead computers extra fun.

At least yesterday wasn't too bad; it was only an hour, Kim kept me company during that time, and after that we went for Peruvian food at Machu Picchu Restaurant.  I had anticuchos (skewered chunks of marinated beef heart), sopa a la minuta (soup with ground beef, noodles, potato, and a few spinach leaves), and fried plantains.  Kim had a steak dish where the steak was cut into thin strips and tasted just like the steak my mom makes.  Good food.

Today i was at my company's colocation cage.  The task set before me and my coworker was to rack two Sun Fire 445 servers.

The Problem
The rails are one and one-third rack units high.  There are two holes on top and bottom, and two pins in the middle.  The idea is that the holes allow the rails to be screwed in, and the pins align directly with the two unused screw holes in between.

In practice, you can see a bend at the end of the rail.  This causes the rail to not allow the pins to align with the holes.  Thus, the rail is effectively useless.

The Solution

I managed to procure a hammer at my favorite restaurant, Hunan Taste, where my coworker and i had lunch.  It was either that or attempt to unbend the cheap metal rail with a pair of vise grips.

The Result

Once again, a judicious and clever application of violence saves the day.