where am i going, and why am i in this handbasket?

[rone]

2wanda has said before (and i admit that she, as my wife, might be biased) that she thinks that computer systems administrators should unionize because we're the tech workers who are most often abused by companies.  I have mixed feelings about unions, but there's little doubt that the industry takes us for granted far too often.  And yet, in all of my career, i have never encountered a situation so dire as in my current place of employ.  If they'd offered me a full-time position, i would have turned it down and fled screaming; however, the contract lets me stick to fulfilling tasks instead of worrying about my future with the company, or the company's future.

Right now, IT ("Ops" is reserved for another part of the company that fulfills a different role) is composed of the Director, who is a senior sysadmin with a managerial title, and one junior sysadmin.  Normally, that's a bit light for a small startup, but not egregious.  The guy they hired before me absconded with a new Mac laptop and an iPhone after four days there.  The guy they hired before that lasted two weeks before they realized that he was a complete charlatan (he was hired over the Director's objection, too).  The guy who was the boss when the current Director was hired was such an idiot that he recompiled the kernel on the phone system server in the middle of the afternoon, rebooted it, and when it failed to come up, took a few half-hearted stabs at fixing it before leaving for an appointment, leaving the phones down until the next day.  This was normal for him, i was told; he was too often worse than no help at all.

Aside from the personnel problems, the infrastructure is a complete cockup.  A few examples: the whole office is on a /24 (a network segment with 256 IP addresses) that has been out of available addresses for months, predictably yielding IP conflicts; the Ops team works in an office that is restricted to people with an Ops keycard for security reasons, but their network is fully reachable from anywhere in the company; the office has both wired and wireless pipes to the Internet, but they are not set up for failover, nor are services such as VPN set up to use either one; production has been running on network equipment that dates from the Clinton administration, and development is running on servers from a manufacturer that stopped making hardware in 2001; the source control repository that contains the code for the company's applications has not been backed up in months.

The CEO introduced herself to me on Wednesday, and told me, "I'm glad you're here.  We really need you.  As a SaaS company, infrastructure is really important to us."  You know what, lady?  You don't really believe that.  If you did, you wouldn't keep hiring morons, you wouldn't be running your company on spit and baling wire, and you wouldn't be telling your Director of IT that you can't afford to hire the people he needs to keep your fucking company from imploding.

I cannot fathom why so many Internet technology companies run their business on a suboptimal infrastructure that is always the result of an abject failure of prioritization.  They get the best possible developers because they want a top-notch product, they get the best possible sales and marketing because they want to have as many customers as possible, but when it comes to building the base on which everything runs, well, the founders just slapped together some whitebox PCs from Fry's, and the product's been running on that just fine, so why change it?  It's this and eleven billion other idiotic excuses that come from people who Don't Fucking Understand how to build an infrastructure and are, somehow, Utterly Incapable of hiring someone who does.

Why?  It can't be that we're scary; yes, it's easy to lean on the "sysadmins and netadmins are all disturbed or disturbing" stereotype, but you know what, every FUCKING job req out there demands "excellent writing and speaking skills", and people get hired, so maybe we're not nearly as bestial as people think.  In the end, i think it's simply a complete lack of respect for what we do; we're almost never given the resources that we need to do the job right, and when the resource starvation produces less than stellar results, the blame always comes quickly at us.  We need advocacy.  I don't get the impression that SAGE has any idea or interest in it, but they damn well should.  Maybe i'll go yell at them for a while.  Yelling at people is always an effective communication tool.

Comments

[solipsistnation.livejournal.com]

Yup. Over the past few years people have gotten the impression that anyone can just install Red Hat or OS X Server and fire up apache (or, more likely, figure out where to put files that will be served up by the apache process they don't know how to stop and start on its own because, hey, if something goes wrong you can just reboot, right?) and call themselves a sysadmin. Advertising for OS X and linux distributions enforces this belief, too, since they couldn't really push linux on people by saying "This is complicated, easy to do poorly, and difficult to do correctly. You'll want to hire a professional." No, it's all "This is the easiest linux ever!!!!@@!" And things ARE easy enough to get working, you know, _well enough_.

I just dug through and audited an application running her that was written by non-programmers and is being run by non-sysadmins. It's a service that's heavily used and is well-liked by the people who use it, but it's not an "official" service (that is, they offer it as a service but it has no funding, it was never specced, was never audited before I got to poke at it, and it runs on a couple of desktop workstations sitting in somebody's office). It's so fantastically insecure that I found about a dozen ways to do naughty things with it without trying very hard, from SQL injection to uploading files to arbitrary locations to insecure ssh sessions(!) to cross-site scripting stuff, and so on and so forth. But it all works, and it's popular, and hey, it hasn't needed a real sysadmin looking at it for the past couple of years, so why does it need it now all of a sudden? So I have to be very careful not to be The Dick when I point out that if I breathe on it funny it's going to fall over.

Conveniently (if painfully) it failed in a predictable way during a high-profile event last week, so now I do get to offer up ways to fix it and I think large pieces of it will be handed to me to fix, finally. Or (and this is actually more likely these days) we'll just say that it's unfunded and there's no budget (since this is the University of California) and it'll get cut. And that's too bad, because it's a cool service and one that people do use heavily. It just should have been engineered from the start rather than hacked together.

[ronebofh.livejournal.com]

"It just should have been engineered from the start rather than hacked together."

That's it in a nutshell, except for most companies it's not just a "service" but the whole goddamn thing. Why slap something together instead of doing it as right as you can from the start? Why don't they care?

[solipsistnation.livejournal.com]

It works. Why should they worry about it? People who come in and say it'll be complicated and expensive to make it work RIGHT, well, they want to actually be paid. So they hire people who come in and say "Oh this is great, sure" because people who don't know any better are (usually) cheaper to employ.

I blame advertising. "It's totally simple!" "It's 100% reliable!"

I've talked to people about why sysadmins and technical professionals in general don't really organize. I think it's that there's not really incentive (or hasn't been in the past, anyway). Techies make decent money and can always hop from one job to another if they need to, right? Why do you need to worry about not getting screwed over by your employers when you're in such high demand? You only need unions when there's more supply than demand and you need to make sure employers don't exploit that. Otherwise workers would be able to write their own rules. Of course, in THIS economic environment techies are suddenly in the same boat as, I dunno, meatpackers around the year 1900. (Except without as much being accidentally ground up or freezing to death.)

[t0rque.livejournal.com]

Speaking as someone driving timelines these days, my answer is "Because sometimes you need to get something UP and OUT or it doesn't matter if it's well-engineered."

Being successful as a startup, in my observation, requires walking a fine balance between total kludged hacking and overengineering. One way, you fall over when you get into the market; the other way you miss the market window.

[ronebofh.livejournal.com]

Yeah, i realize it's a risk, but i'm not sure that fucking up now versus a slow, painful, four-year fuckup is a real choice, except in the cynical "sell out in three years" plan.

I'm all for cutting corners if there's an easy way to stick the corners back on later.

[xtingu]

I have nothing to add here except nods and groans of understanding.

(Any chance of opening this post to mrlich, my beau and fellow IT surprise buttsecks victim?)

[ronebofh.livejournal.com]

IT IS DONE.