rone: (nose)

@[livejournal.com profile] ronebofh: DevOps in the Time of Cholera

@[livejournal.com profile] palecur: Memories of my Melancholy NOCMonkeys

@[livejournal.com profile] palecur: Chronicles of an Outage Foretold

@[livejournal.com profile] ronebofh: One Hundred Years of On-Call

@[livejournal.com profile] palecur: No One Writes to the Kernel

@[livejournal.com profile] ronebofh: The General in His LDAP-rinth

@[livejournal.com profile] palecur: Of Love and Other Daemons

rone: (bofh)

In my efforts to find a new job, i've signed up for an AWS Bootcamp class, because THE CLOUD is one of the New Hotnesses that i lack.

So far, i am dismayed by the difficulty some of the people in this Intermediate class are having with finding stuff on a page, following instructions, and figuring stuff out.

rone: (bofh)

Gene Kim is a guy who's accomplished a lot more in his career than i have.  I kinda-sorta met him in passing at an LSPE Meetup and he seemed like a nice guy.  But his recent writeup, "Why We Need DevOps Now", which title i agree with, is just not good at all (i used the word "terrible" on Twitter, which someone called me on, so maybe i'll try to dial back the hyperbole).

As my friend John Willis told me after I dismissed DevOps as just another marketing fad, “DevOps is the best chance at relevance that IT Operations has had in thirty years.” I immediately realized that he was right.
Is that really all it took?

"John, it's a fad."
"Gene, it's our last, best hope."
"OMG UR RITE!!"


DevOps is real (this article says everything that needs to be said about it) and also a fad, in the same way that Agile Software Development is real and a fad:

BOSS: "We're going to try something called agile programming.  That means no more planning and no more documentation.  Just start writing code and complaining." WALLY: "I'm glad it has a name." BOSS: "That was your training."
A good business avails itself of forward-looking approaches in order to contend with the firehose of change that is our industry's lifeblood.  But we cannot mistake these approaches for anything other than a tool.
Act I begins with IT Operations, where we’re supporting a large, complex revenue generating application. The problem is that everyone knows that the application and supporting infrastructure is... fragile.
I smell a setup.  The scenario described is clearly the result of bad management, who failed to see this sort of undesirable performance down the road and act upon it to preclude the fragility, and it has been that way every time i've encountered it in my career.  It is not, as implied, a result of traditional IT operations.
In Act 2, our life gets worse when the business starts making even bigger commitments to Wall Street, often dreamed up by art or creative writing majors
I have no idea what he's talking about here.  In my experience, publicly-traded businesses (which would be the ones who make a commitment to shareholders, not Wall Street) don't usually have art or creative writing majors making major business commitments; the more likely case is that those positions are staffed with MBAs.  Privately held businesses, on the other hand, tend to be beholden to venture capital firms, whose presence can become far more unwelcome and meddlesome than that of your typical shareholder.
We all know that there must be better way, right? DevOps is the proof that it’s possible to break the core, chronic conflict, so we can deliver a fast flow of features without causing chaos and disruption to the production environment.
YOU NEED MANAGEMENT BUY-IN.  YOU NEED THE SUPPORT OF THE PEOPLE WHO CONTROL THE MONEY TO HIRE THE RIGHT PEOPLE AND PURCHASE THE RIGHT EQUIPMENT, THE PEOPLE WHO SET THE COMPANY'S EXTERNAL AND INTERNAL STRATEGIES, THE PEOPLE WHO HAVE YOUR BACK WHEN YOU TELL THE COMPANY, "THIS WILL HURT, BUT IN 12 MONTHS, WE WILL NO LONGER LIMP."  Painting DevOps as a panacæa does all of us who believe in DevOps and work in DevOps a tremendous disservice.  To unfuck a fucked company, you need to fix the culture.  This is why Netflix is a shining example — not because of their use of DevOps, but because their culture enabled them to use their talent in a massively constructive and creative fashion.
"Before you can solve a complex problem, you must first have empathy for the other stakeholders."
Before you can solve a complex problem, you must first understand it.  One of the important factors in understanding it is empathizing with the other parties.  It is not, however, the first thing to do.

Perhaps i'm being too harsh on what's less of a thoughtful article and more a plug for his book.  But i do not find the appeal in being sold on DevOps by starting off with an elaborate strawman.

s-m-r-t

Feb. 14th, 2011 05:17 pm
rone: (asplode)

Last week, i had a job interview (i'm looking because i'm tired of the night shift and i'm bored as hell and there hasn't been an opening in either the day shift of my group or in another group that i'm qualified for; that might change in 2 or 3 months, but i'm not willing to wait longer than that), and when i asked two of the guys who had just started working there what they liked best about their new employer, they both talked about how smart everyone was.  This made me wince on the inside, because in my experience, smarts only get you so far.  In systems operations, it's much more important to have people who work hard and play well with others than to have some super-sharp person bursting at the seams with cleverness and grand ideas, especially in an industry where it's often too easy to find a brilliant yet intractable coworker who's tolerated because of what they can do whenever they stop being an asshole.  Perhaps i'm biased, but i'd rather see the smart kids doing the programming so that us worker bees can get on with making the infrastructure hum.  I'm so glad i'm a Beta.  Or something.

rone: (bofh)

Two weeks, i hit LISA for the first time in 12 years.  I can't really explain why it took me so long to return, other than the typical excuse of indolence (and, really, from a professional standpoint, that is simply bad form).  It was good to see [livejournal.com profile] _nicolai_, [livejournal.com profile] pir, [livejournal.com profile] gothgeekgrrl, and [livejournal.com profile] gallifreyan, as well as Steve VanDevender, ex-coworkers Grant Talarico and Aaron Fraser, and talk.bizarre's own George William Herbert (pure chance encounter), and i indulged in the opportunity to show them around downtown San José for lunch at Morocco's, Hanuman, and Mezcal.

One thing i learned at LISA was from ARIN regarding the imminence of IPv4 address space exhaustion; if everything breaks right, the last available /8s will be handed out by IANA to the regional internet registries sometime in the second quarter of 2012.  So, adding IPv6 is moving out of the "hardcore early adopter" stage into the "prudent proactive user" stage (but feel free to disagree and call me names).  I also picked up some good tips regarding interviewing, résumé writing, and becoming a better senior sysadmin.

As of this week, i've been working during the day, filling in for someone else in the group who's out (the group is running rather lean as of late).  It's been nice to see people and actually get to have questions answered without having to wake someone up (although i get the impression today that far too many people have checked out early for Thanksgiving).  I hope that i'll be on days permanently soon.

rone: (bofh)

There has been serious attrition in the Systems group (3 sysadmins, plus the group's director, for whom i worked at Macromedia) over the last couple of months, so i asked my boss as well as the SA manager about the likelihood of my moving to that group.  Both responded very positively, so the next move is going through the interview process... which of course means i'll have to alter my sleeping schedule slightly.  I hope i don't pass out between interviewers.

I'm experiencing some trepidation due to the recent combustibility, but i do think that it's settled down now, and i'm pretty sure that there's little more i can do for the NOC right now, and i think that i can do more for the group if i change cubes to the other side of the building.

rone: (bofh)

It's always comical when a sysadmin in the Real Sysadmin group, as opposed to us fakey-fake sysadmins in the NOC, openly admits that email notifications of ticket updates that we file are "basically ignored" and that "no one on the SA team has ever paid attention to them," and that instead we should file JIRA tickets for any issues that require their investigation.  And then it's doubly hilarious when his boss follows up with, "[Our move to JIRA] is not a widely communicated change, so it's not expected that you or anyone outside the SA team would know this."

rone: (brock)

Working at Netflix combines the joy of working for a large company that has a corporate culture laden with baggage that results from their vaunted "Rules Annoy Us" core value, with the joy of working for a startup where people are scrabbling to get things done because they can't be arsed to improve the process because everything is changing at a stupidly fast pace.  Tomorrow i get to find out whether they truly value my courage, honesty, and passion, because after this week i'm seriously tired of seeing people in my group either running around like headless chickens herding cats, or clicking buttons like experimental monkeys.

I was told when i was interviewed that the goal of hiring senior-level sysadmins for a NOC (there are a few of us here, and we agree on what i'm bitching about here) was part of a grand plan to raise the NOC's level to something impressive and worthy of, i dunno, a company like Netflix.  Tomorrow i'm gonna ask what the plan is and when we get started on it; i know that one big part of it is getting a monitoring system that isn't a piece of shit, but the putative replacement has been in the wings for six months and just semi-officially got off the ground last week, and, well, we were told to make sure that the new system's alerts matched the old system's alerts, and they aren't even fucking close.  If someone doesn't get fired over this fiasco (and Netflix loves to fire people), i think that i will have to seriously consider whether my sanity is worth my very generous salary.

Now that i'm at a point in my career where i actually, finally, know what i want to do and where i want to go, i really don't want to spin my wheels, and i need to find out whether my shiny new boss (who has already given pause to some in the group) and boss's boss (who is a Netflix veteran but has only been with our group for two months) are going to tell me something good.  If not, i know that there's a recent opening in the sysadmin group, and i'll talk to the boss over there to see what my options are.

rone: (yikes)

Because i could not stop for death, i bought some to go.


What the world needs is a first-person shooter version of Dig Dug.


Working in the data center is like hanging out in Darth Vader's chest.


When i die, i hope to have a funny look on my face.  When the rigor mortis sets in, people coming to my funeral will say, "Wow, Mom was right."


You know the human race is a fucking failure when there's such a thing as a "blogging Lifetime Achievement Award".


Yo mama's so ugly, she can't masturbate anymore `cause her clitoris got a restraining order against her.


There is no 'i' in 'team', but there is an 'eat' and a 'me'.


Consider, if you will, the potential of Dr. Phil/Jim Cramer erotica.


Dear President Obama: stretching out a hand to Joe Lieberman should only happen when your arm fully extends as you backhand him across his Droopy face.

Thank you for reading.  Check back again in three years for more Twitturds.

rone: (Default)

Because i could not stop for death, i bought some to go.


What the world needs is a first-person shooter version of Dig Dug.


Working in the data center is like hanging out in Darth Vader's chest.


When i die, i hope to have a funny look on my face.  When the rigor mortis sets in, people coming to my funeral will say, "Wow, Mom was right."


You know the human race is a fucking failure when there's such a thing as a "blogging Lifetime Achievement Award".


Yo mama's so ugly, she can't masturbate anymore `cause her clitoris got a restraining order against her.


There is no 'i' in 'team', but there is an 'eat' and a 'me'.


Consider, if you will, the potential of Dr. Phil/Jim Cramer erotica.


Dear President Obama: stretching out a hand to Joe Lieberman should only happen when your arm fully extends as you backhand him across his Droopy face.

Thank you for reading.  Check back again in three years for more Twitturds.

rone: (simian)

Nix wrote:

One person at work keeps a file named -i in important directories on Solaris boxes, just to catch rm *'s in them.

Do they keep an extra pair of underpants in their car in case they shit themselves on the way to work?

rone: (Default)

Nix wrote:

One person at work keeps a file named -i in important directories on Solaris boxes, just to catch rm *'s in them.

Do they keep an extra pair of underpants in their car in case they shit themselves on the way to work?

rone: (brock)

[livejournal.com profile] 2wanda has said before (and i admit that she, as my wife, might be biased) that she thinks that computer systems administrators should unionize because we're the tech workers who are most often abused by companies.  I have mixed feelings about unions, but there's little doubt that the industry takes us for granted far too often.  And yet, in all of my career, i have never encountered a situation so dire as in my current place of employ.  If they'd offered me a full-time position, i would have turned it down and fled screaming; however, the contract lets me stick to fulfilling tasks instead of worrying about my future with the company, or the company's future.

Right now, IT ("Ops" is reserved for another part of the company that fulfills a different role) is composed of the Director, who is a senior sysadmin with a managerial title, and one junior sysadmin.  Normally, that's a bit light for a small startup, but not egregious.  The guy they hired before me absconded with a new Mac laptop and an iPhone after four days there.  The guy they hired before that lasted two weeks before they realized that he was a complete charlatan (he was hired over the Director's objection, too).  The guy who was the boss when the current Director was hired was such an idiot that he recompiled the kernel on the phone system server in the middle of the afternoon, rebooted it, and when it failed to come up, took a few half-hearted stabs at fixing it before leaving for an appointment, leaving the phones down until the next day.  This was normal for him, i was told; he was too often worse than no help at all.

Aside from the personnel problems, the infrastructure is a complete cockup.  A few examples: the whole office is on a /24 (a network segment with 256 IP addresses) that has been out of available addresses for months, predictably yielding IP conflicts; the Ops team works in an office that is restricted to people with an Ops keycard for security reasons, but their network is fully reachable from anywhere in the company; the office has both wired and wireless pipes to the Internet, but they are not set up for failover, nor are services such as VPN set up to use either one; production has been running on network equipment that dates from the Clinton administration, and development is running on servers from a manufacturer that stopped making hardware in 2001; the source control repository that contains the code for the company's applications has not been backed up in months.

The CEO introduced herself to me on Wednesday, and told me, "I'm glad you're here.  We really need you.  As a SaaS company, infrastructure is really important to us."  You know what, lady?  You don't really believe that.  If you did, you wouldn't keep hiring morons, you wouldn't be running your company on spit and baling wire, and you wouldn't be telling your Director of IT that you can't afford to hire the people he needs to keep your fucking company from imploding.

I cannot fathom why so many Internet technology companies run their business on a suboptimal infrastructure that is always the result of an abject failure of prioritization.  They get the best possible developers because they want a top-notch product, they get the best possible sales and marketing because they want to have as many customers as possible, but when it comes to building the base on which everything runs, well, the founders just slapped together some whitebox PCs from Fry's, and the product's been running on that just fine, so why change it?  It's this and eleven billion other idiotic excuses that come from people who Don't Fucking Understand how to build an infrastructure and are, somehow, Utterly Incapable of hiring someone who does.

Why?  It can't be that we're scary; yes, it's easy to lean on the "sysadmins and netadmins are all disturbed or disturbing" stereotype, but you know what, every FUCKING job req out there demands "excellent writing and speaking skills", and people get hired, so maybe we're not nearly as bestial as people think.  In the end, i think it's simply a complete lack of respect for what we do; we're almost never given the resources that we need to do the job right, and when the resource starvation produces less than stellar results, the blame always comes quickly at us.  We need advocacy.  I don't get the impression that SAGE has any idea or interest in it, but they damn well should.  Maybe i'll go yell at them for a while.  Yelling at people is always an effective communication tool.

rone: (Default)

[livejournal.com profile] 2wanda has said before (and i admit that she, as my wife, might be biased) that she thinks that computer systems administrators should unionize because we're the tech workers who are most often abused by companies.  I have mixed feelings about unions, but there's little doubt that the industry takes us for granted far too often.  And yet, in all of my career, i have never encountered a situation so dire as in my current place of employ.  If they'd offered me a full-time position, i would have turned it down and fled screaming; however, the contract lets me stick to fulfilling tasks instead of worrying about my future with the company, or the company's future.

Right now, IT ("Ops" is reserved for another part of the company that fulfills a different role) is composed of the Director, who is a senior sysadmin with a managerial title, and one junior sysadmin.  Normally, that's a bit light for a small startup, but not egregious.  The guy they hired before me absconded with a new Mac laptop and an iPhone after four days there.  The guy they hired before that lasted two weeks before they realized that he was a complete charlatan (he was hired over the Director's objection, too).  The guy who was the boss when the current Director was hired was such an idiot that he recompiled the kernel on the phone system server in the middle of the afternoon, rebooted it, and when it failed to come up, took a few half-hearted stabs at fixing it before leaving for an appointment, leaving the phones down until the next day.  This was normal for him, i was told; he was too often worse than no help at all.

Aside from the personnel problems, the infrastructure is a complete cockup.  A few examples: the whole office is on a /24 (a network segment with 256 IP addresses) that has been out of available addresses for months, predictably yielding IP conflicts; the Ops team works in an office that is restricted to people with an Ops keycard for security reasons, but their network is fully reachable from anywhere in the company; the office has both wired and wireless pipes to the Internet, but they are not set up for failover, nor are services such as VPN set up to use either one; production has been running on network equipment that dates from the Clinton administration, and development is running on servers from a manufacturer that stopped making hardware in 2001; the source control repository that contains the code for the company's applications has not been backed up in months.

The CEO introduced herself to me on Wednesday, and told me, "I'm glad you're here.  We really need you.  As a SaaS company, infrastructure is really important to us."  You know what, lady?  You don't really believe that.  If you did, you wouldn't keep hiring morons, you wouldn't be running your company on spit and baling wire, and you wouldn't be telling your Director of IT that you can't afford to hire the people he needs to keep your fucking company from imploding.

I cannot fathom why so many Internet technology companies run their business on a suboptimal infrastructure that is always the result of an abject failure of prioritization.  They get the best possible developers because they want a top-notch product, they get the best possible sales and marketing because they want to have as many customers as possible, but when it comes to building the base on which everything runs, well, the founders just slapped together some whitebox PCs from Fry's, and the product's been running on that just fine, so why change it?  It's this and eleven billion other idiotic excuses that come from people who Don't Fucking Understand how to build an infrastructure and are, somehow, Utterly Incapable of hiring someone who does.

Why?  It can't be that we're scary; yes, it's easy to lean on the "sysadmins and netadmins are all disturbed or disturbing" stereotype, but you know what, every FUCKING job req out there demands "excellent writing and speaking skills", and people get hired, so maybe we're not nearly as bestial as people think.  In the end, i think it's simply a complete lack of respect for what we do; we're almost never given the resources that we need to do the job right, and when the resource starvation produces less than stellar results, the blame always comes quickly at us.  We need advocacy.  I don't get the impression that SAGE has any idea or interest in it, but they damn well should.  Maybe i'll go yell at them for a while.  Yelling at people is always an effective communication tool.

rone: (bofh)

(Before you panic, i am still gainfully employed and have no reason to doubt this will continue for a while.)

I was recently contacted by a recruiter who found my résumé on the Web.  The dotcom's job req began: "You're smart, you're fast, and most importantly, you always get it done.  Admit it, you're a rockstar."

I had to suppress the urge to backhand my craptop's screen.  So instead i wrote this and asked the recruiter to pass it onto the responsible party:

This is embarrassing.  I wouldn't hire someone who calls himself "a rockstar" any more than i'd want to work somewhere that looks for one.  It sets the wrong tone for this sort of work, implying that this person is going to hog the spotlight. The only worse term i can imagine is "cowboy".
You know what?  Systems administration is hard work, and it gets harder every day.  Companies want you to know at least a bit of everything and to be awesome at something.  The line between network and systems administration has blurred, and the line between DBA and sysadmin is blurring as well.  Staying current is a tough chore, especially when you've been working all day.  So spare me your "rockstars" because the last thing i need is to deal with some strung-out narcissist with root access.

rone: (bofh)

(Before you panic, i am still gainfully employed and have no reason to doubt this will continue for a while.)

I was recently contacted by a recruiter who found my résumé on the Web.  The dotcom's job req began: "You're smart, you're fast, and most importantly, you always get it done.  Admit it, you're a rockstar."

I had to suppress the urge to backhand my craptop's screen.  So instead i wrote this and asked the recruiter to pass it onto the responsible party:

This is embarrassing.  I wouldn't hire someone who calls himself "a rockstar" any more than i'd want to work somewhere that looks for one.  It sets the wrong tone for this sort of work, implying that this person is going to hog the spotlight. The only worse term i can imagine is "cowboy".
You know what?  Systems administration is hard work, and it gets harder every day.  Companies want you to know at least a bit of everything and to be awesome at something.  The line between network and systems administration has blurred, and the line between DBA and sysadmin is blurring as well.  Staying current is a tough chore, especially when you've been working all day.  So spare me your "rockstars" because the last thing i need is to deal with some strung-out narcissist with root access.

rone: (imminent destruction)

Speaking of systems administration in The Other Place:

rone: I'd love the job more if the industry weren't overrun with fuckheads at all levels.

Nix: But that's true of *all* jobs, and all humanity, everywhere, forever.

rone: No, man, what we have here in Silicon Valley is far from fucking normal.

[livejournal.com profile] allbery: Which is why you need to fund my new startup based around XML modelling of Facebook relationships, through which it's possible to build a metadata repository of all interesting people in the world and hire only perfect employees!  I call it WebHire 3.0!  It will revolutionize the way that people work by letting them conduct all interpersonal relationships through Twitter!

rone: (Default)

Speaking of systems administration in The Other Place:

rone: I'd love the job more if the industry weren't overrun with fuckheads at all levels.

Nix: But that's true of *all* jobs, and all humanity, everywhere, forever.

rone: No, man, what we have here in Silicon Valley is far from fucking normal.

[livejournal.com profile] allbery: Which is why you need to fund my new startup based around XML modelling of Facebook relationships, through which it's possible to build a metadata repository of all interesting people in the world and hire only perfect employees!  I call it WebHire 3.0!  It will revolutionize the way that people work by letting them conduct all interpersonal relationships through Twitter!

rone: (sleep)

Safari and TouchTerm SSH allow me to deal with most of the problems that crop up during my on-call period from the safety and warmth of my bed.

rone: (Default)

Safari and TouchTerm SSH allow me to deal with most of the problems that crop up during my on-call period from the safety and warmth of my bed.

Profile

rone: (Default)
entombed in the shrine of zeroes and ones

December 2022

S M T W T F S
    123
45678910
11121314151617
18192021222324
252627282930 31

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Jul. 10th, 2025 02:55 pm
Powered by Dreamwidth Studios