entombed in the shrine of zeroes and ones

Gene Kim is a guy who's accomplished a lot more in his career than i have.  I kinda-sorta met him in passing at an LSPE Meetup and he seemed like a nice guy.  But his recent writeup, "Why We Need DevOps Now", which title i agree with, is just not good at all (i used the word "terrible" on Twitter, which someone called me on, so maybe i'll try to dial back the hyperbole).

As my friend John Willis told me after I dismissed DevOps as just another marketing fad, “DevOps is the best chance at relevance that IT Operations has had in thirty years.” I immediately realized that he was right.

BOSS: "We're going to try something called agile programming.  That means no more planning and no more documentation.  Just start writing code and complaining." WALLY: "I'm glad it has a name." BOSS: "That was your training."

Act I begins with IT Operations, where we’re supporting a large, complex revenue generating application. The problem is that everyone knows that the application and supporting infrastructure is... fragile.

In Act 2, our life gets worse when the business starts making even bigger commitments to Wall Street, often dreamed up by art or creative writing majors

We all know that there must be better way, right? DevOps is the proof that it’s possible to break the core, chronic conflict, so we can deliver a fast flow of features without causing chaos and disruption to the production environment.

"Before you can solve a complex problem, you must first have empathy for the other stakeholders."

... is when work takes a turn onto Mad Shit Lane but you are enjoined, on pain of death, from talking about it with any level of detail.

And, no, this is not an invitation to play Twenty Questions.

We can safely do 150 writes/sec/storage node. IR alone was attempting 330 writes/sec/storage node. We dialed back IR traffic and stopped seeing 503s for writes. However, reads were still seeing 503s. This was because we were doing full table scans.

Two weeks, i hit LISA for the first time in 12 years.  I can't really explain why it took me so long to return, other than the typical excuse of indolence (and, really, from a professional standpoint, that is simply bad form).  It was good to see _nicolai_, pir, gothgeekgrrl, and gallifreyan, as well as Steve VanDevender, ex-coworkers Grant Talarico and Aaron Fraser, and talk.bizarre's own George William Herbert (pure chance encounter), and i indulged in the opportunity to show them around downtown San José for lunch at Morocco's, Hanuman, and Mezcal.

One thing i learned at LISA was from ARIN regarding the imminence of IPv4 address space exhaustion; if everything breaks right, the last available /8s will be handed out by IANA to the regional internet registries sometime in the second quarter of 2012.  So, adding IPv6 is moving out of the "hardcore early adopter" stage into the "prudent proactive user" stage (but feel free to disagree and call me names).  I also picked up some good tips regarding interviewing, résumé writing, and becoming a better senior sysadmin.

As of this week, i've been working during the day, filling in for someone else in the group who's out (the group is running rather lean as of late).  It's been nice to see people and actually get to have questions answered without having to wake someone up (although i get the impression today that far too many people have checked out early for Thanksgiving).  I hope that i'll be on days permanently soon.

Programmer: I need access to the frobnicator tools on one of the test frobnicator hosts.

Programmer's manager: I approve this request.

rone: Access granted.  I will share the process with the rest of my group.

Impatient pessimist: You did it wrong; you needed to add him to the frobnicator group.  I will undo your work and instead do what i think is right.

rone: The frobnicator group allows access to the frobnicator tools on all of the test frobnicator hosts, as well as all of the production frobnicator hosts.  This is not what the user requested nor is it what his manager approved.  I have undone your changes and reset the changes i made.  I also invite you to choke on my fuck.

It's nearly my six-month anniversary at Netflix, which means that i've been working graveyard shift for 21 weeks.  Despite a few hiccups along the way, sleeping during the day hasn't been too bad; i've settled into taking a melatonin+valerian capsule and half a Benadryl before sleep (occasionally washed down with hot chocolate instead of water).  I usually hit the sack soon after i see Kim off to work in the morning, so that i can be awake by the time she gets home.  On Friday, i eschew the sleeping aids and just settle into a four-hour sleep period; when i wake up, i can attend one of our team meetings in the afternoon, and then proceed with having a normal weekend (although lately my sleep has been cut short at night, so i've been taking a full Benadryl on Friday nights).  On Sunday evenings, i have another 4-hour nap, and upon waking put out the trash and then head to work.

At times i feel a little uncomfortable about the pill-taking, but given that this isn't going to last forever, i'll keep it up for now.

It's always comical when a sysadmin in the Real Sysadmin group, as opposed to us fakey-fake sysadmins in the NOC, openly admits that email notifications of ticket updates that we file are "basically ignored" and that "no one on the SA team has ever paid attention to them," and that instead we should file JIRA tickets for any issues that require their investigation.  And then it's doubly hilarious when his boss follows up with, "[Our move to JIRA] is not a widely communicated change, so it's not expected that you or anyone outside the SA team would know this."

Our IT all-hands meetings have a raffle at the end.  In the last meeting, i won a basket of Godiva chocolate.  Today, i won a bottle of Etude 2006 pinot noir.

The best part is that i didn't go to either meeting (a coworker puts in the names of her off-shift teammates).

And here you thought i'd go all December without updating my journal...

After about two months and a half into the job, i am enjoying it a fair amount.  One of the big pluses is that my group has as many women as men in it, and i've been enjoying working with them a lot (more so than the men, who are, in turn, an impatient pessimist, a taciturn geek, and a French Canadian).  The graveyard shift has been a bit tough on me (although swing shift is worse, because i don't get to see Kim at all except on the weekend) but it's also nice and quiet, which gives me time to analyze our procedures and processes, which are often, er, questionable.  Right now i'm working our holiday shift, where each of us works 3 or 4 12-hour shifts over the last two weeks of December, which leaves a lot of free time.  It's not a bad arrangement.

Doing NOC work is certainly a bit of a step down for me, but at the same time, the NOC needs a lot of work and that's why they're hiring experienced people to do that work.  We got a new director about a month ago, and a new manager two weeks ago, and so far i like what they're doing and the questions they're asking.  In addition, two of my ex-bosses from Macromedia are in the sysadmin group (one as a sysadmin and the other as his boss's boss) and we're in the same headspace regarding getting the job done right.  So i'm feeling fairly optimistic about getting some good changes implemented in the next six months.

It's usually a bad sign when a contrarian reverses position, right?