where am i going, and why am i in this handbasket?

[rone]

2wanda has said before (and i admit that she, as my wife, might be biased) that she thinks that computer systems administrators should unionize because we're the tech workers who are most often abused by companies.  I have mixed feelings about unions, but there's little doubt that the industry takes us for granted far too often.  And yet, in all of my career, i have never encountered a situation so dire as in my current place of employ.  If they'd offered me a full-time position, i would have turned it down and fled screaming; however, the contract lets me stick to fulfilling tasks instead of worrying about my future with the company, or the company's future.

Right now, IT ("Ops" is reserved for another part of the company that fulfills a different role) is composed of the Director, who is a senior sysadmin with a managerial title, and one junior sysadmin.  Normally, that's a bit light for a small startup, but not egregious.  The guy they hired before me absconded with a new Mac laptop and an iPhone after four days there.  The guy they hired before that lasted two weeks before they realized that he was a complete charlatan (he was hired over the Director's objection, too).  The guy who was the boss when the current Director was hired was such an idiot that he recompiled the kernel on the phone system server in the middle of the afternoon, rebooted it, and when it failed to come up, took a few half-hearted stabs at fixing it before leaving for an appointment, leaving the phones down until the next day.  This was normal for him, i was told; he was too often worse than no help at all.

Aside from the personnel problems, the infrastructure is a complete cockup.  A few examples: the whole office is on a /24 (a network segment with 256 IP addresses) that has been out of available addresses for months, predictably yielding IP conflicts; the Ops team works in an office that is restricted to people with an Ops keycard for security reasons, but their network is fully reachable from anywhere in the company; the office has both wired and wireless pipes to the Internet, but they are not set up for failover, nor are services such as VPN set up to use either one; production has been running on network equipment that dates from the Clinton administration, and development is running on servers from a manufacturer that stopped making hardware in 2001; the source control repository that contains the code for the company's applications has not been backed up in months.

The CEO introduced herself to me on Wednesday, and told me, "I'm glad you're here.  We really need you.  As a SaaS company, infrastructure is really important to us."  You know what, lady?  You don't really believe that.  If you did, you wouldn't keep hiring morons, you wouldn't be running your company on spit and baling wire, and you wouldn't be telling your Director of IT that you can't afford to hire the people he needs to keep your fucking company from imploding.

I cannot fathom why so many Internet technology companies run their business on a suboptimal infrastructure that is always the result of an abject failure of prioritization.  They get the best possible developers because they want a top-notch product, they get the best possible sales and marketing because they want to have as many customers as possible, but when it comes to building the base on which everything runs, well, the founders just slapped together some whitebox PCs from Fry's, and the product's been running on that just fine, so why change it?  It's this and eleven billion other idiotic excuses that come from people who Don't Fucking Understand how to build an infrastructure and are, somehow, Utterly Incapable of hiring someone who does.

Why?  It can't be that we're scary; yes, it's easy to lean on the "sysadmins and netadmins are all disturbed or disturbing" stereotype, but you know what, every FUCKING job req out there demands "excellent writing and speaking skills", and people get hired, so maybe we're not nearly as bestial as people think.  In the end, i think it's simply a complete lack of respect for what we do; we're almost never given the resources that we need to do the job right, and when the resource starvation produces less than stellar results, the blame always comes quickly at us.  We need advocacy.  I don't get the impression that SAGE has any idea or interest in it, but they damn well should.  Maybe i'll go yell at them for a while.  Yelling at people is always an effective communication tool.

Comments

[ltempt.livejournal.com]

Doing things the right way costs money. Hiring good people costs money. My impression of most startups, especially ones in nebulous industries like SaaS is they're driven by development and everyone else is seen as needless overhead.

The "sysadmins and netadmins are all disturbed or disturbing" stereotype is dated. It's more sysadmins and netadmins are incompetant until proven otherwise, and probably faking it even then. Sometimes, useless morons get hired because there's just nobody else applying for the gigs.

Shabby environments mean good sysadmins and netadmins get out and either find a new job or find a cushy enterprise where they can park for a long time. The few that stay in the industry get to wallow in the mess left by the useless until they decide to get out, too.

As far as I can see, there's no solution. That's why more and more shops are using crappier and crappier infra designs - they're easier to support. Running production on a single whitebox with Ubuntu means a lot less sysadmin skill than an HA cluster on UNIX. A few crappy routers and some crappier switches is easier to maintain than a complex network design with failover and MPLS and VLANs and QOS and whatnot.

I guess the only plus-side of a gig like this is you might be able to make a difference.

[eejitalmuppet.livejournal.com]

Looking in from the outside, i suspect that there is a lot of common or garden ignorance. The PC with Windows that the Finance Director bought from Dell (or whoever) Just Works, (and they don't have much of importance on it so they don't get too upset if they lose some data, or they pay some guy to fix it), so they assume that a cube farm with a hundred or more PCs will Just Work as well. They can't understand that it's the diffeence between driving their Acura (which they don't do very well, but that's what their insurance is for) and driving an 18-wheeler...

[solipsistnation.livejournal.com]

Yup. Over the past few years people have gotten the impression that anyone can just install Red Hat or OS X Server and fire up apache (or, more likely, figure out where to put files that will be served up by the apache process they don't know how to stop and start on its own because, hey, if something goes wrong you can just reboot, right?) and call themselves a sysadmin. Advertising for OS X and linux distributions enforces this belief, too, since they couldn't really push linux on people by saying "This is complicated, easy to do poorly, and difficult to do correctly. You'll want to hire a professional." No, it's all "This is the easiest linux ever!!!!@@!" And things ARE easy enough to get working, you know, _well enough_.

I just dug through and audited an application running her that was written by non-programmers and is being run by non-sysadmins. It's a service that's heavily used and is well-liked by the people who use it, but it's not an "official" service (that is, they offer it as a service but it has no funding, it was never specced, was never audited before I got to poke at it, and it runs on a couple of desktop workstations sitting in somebody's office). It's so fantastically insecure that I found about a dozen ways to do naughty things with it without trying very hard, from SQL injection to uploading files to arbitrary locations to insecure ssh sessions(!) to cross-site scripting stuff, and so on and so forth. But it all works, and it's popular, and hey, it hasn't needed a real sysadmin looking at it for the past couple of years, so why does it need it now all of a sudden? So I have to be very careful not to be The Dick when I point out that if I breathe on it funny it's going to fall over.

Conveniently (if painfully) it failed in a predictable way during a high-profile event last week, so now I do get to offer up ways to fix it and I think large pieces of it will be handed to me to fix, finally. Or (and this is actually more likely these days) we'll just say that it's unfunded and there's no budget (since this is the University of California) and it'll get cut. And that's too bad, because it's a cool service and one that people do use heavily. It just should have been engineered from the start rather than hacked together.

[ronebofh.livejournal.com]

"It just should have been engineered from the start rather than hacked together."

That's it in a nutshell, except for most companies it's not just a "service" but the whole goddamn thing. Why slap something together instead of doing it as right as you can from the start? Why don't they care?

[solipsistnation.livejournal.com]

It works. Why should they worry about it? People who come in and say it'll be complicated and expensive to make it work RIGHT, well, they want to actually be paid. So they hire people who come in and say "Oh this is great, sure" because people who don't know any better are (usually) cheaper to employ.

I blame advertising. "It's totally simple!" "It's 100% reliable!"

I've talked to people about why sysadmins and technical professionals in general don't really organize. I think it's that there's not really incentive (or hasn't been in the past, anyway). Techies make decent money and can always hop from one job to another if they need to, right? Why do you need to worry about not getting screwed over by your employers when you're in such high demand? You only need unions when there's more supply than demand and you need to make sure employers don't exploit that. Otherwise workers would be able to write their own rules. Of course, in THIS economic environment techies are suddenly in the same boat as, I dunno, meatpackers around the year 1900. (Except without as much being accidentally ground up or freezing to death.)

[t0rque.livejournal.com]

Speaking as someone driving timelines these days, my answer is "Because sometimes you need to get something UP and OUT or it doesn't matter if it's well-engineered."

Being successful as a startup, in my observation, requires walking a fine balance between total kludged hacking and overengineering. One way, you fall over when you get into the market; the other way you miss the market window.

[ronebofh.livejournal.com]

Yeah, i realize it's a risk, but i'm not sure that fucking up now versus a slow, painful, four-year fuckup is a real choice, except in the cynical "sell out in three years" plan.

I'm all for cutting corners if there's an easy way to stick the corners back on later.

[xtingu]

I have nothing to add here except nods and groans of understanding.

(Any chance of opening this post to mrlich, my beau and fellow IT surprise buttsecks victim?)

[ronebofh.livejournal.com]

IT IS DONE.

[catbear.livejournal.com]

So. You have nothing to lose. Ask the CEO to sit down with you in a coffee shop and explain it to her. Systems administration is 20% making sure everything is working now and 80% making sure that when something fails (and it will) that it fails without taking the business down with it. How do you explain this? You make a diagram showing the current infrastructure and you use big red arrows for points of single critical failure and big yellow arrows indicating areas of stupidity and inefficiency. And then you are ready to tell them what current failure modes are, and how much it will cost to reduce those failure modes to non-critical levels. And then you bill $400 an hour.

[ronebofh.livejournal.com]

Ideally, yeah, that'd be the way to go, but things are so hosed right now even that's too much to put on them. There's no reason for me to believe that anything i could suggest could be implemented in a reasonable timeframe. Also, much of this has already been explained by the current dude, and the reply has been, "We understand, but you'll have to make do with what you have for now." He is, by far, the most stressed sysadmin i have ever met, and i wish i could do more for him. I believe that this job is killing him.

[syringavulgaris.livejournal.com]

It seems to me, then, that the task (in between running around with fire buckets) is to manage the expectations of the Powers that Be. "Here are the most critical / most likely failure points, here's what we can do to mitigate the potential damage armed only with what we have, but here's what we CAN'T do anything about", etc.

[ronebofh.livejournal.com]

Except that everyone else thinks their "emergency" is the most important thing in the company and won't leave the guy alone. Like i said, every annoying thing in the annals of systems administration are happening all at once in this place, with the exception of getting paid on time.

Man, that's something else, too: in a culture where startups get lots of goodies for their people, this place has nothing except for coffee and instant hot chocolate. It's a disgrace.

[2wanda.livejournal.com]

Maybe one way you could help him is for someone else to sit down with the CEO, completely unprompted by him, and tell her exactly what he's been telling her. She might need to hear it from a different perspective.

[ronebofh.livejournal.com]

Yeah, i'd like to do that, but probably after more than having been there a week. I want to be sure to experience some of this stuff first-hand.

[2wanda.livejournal.com]

It's also possible that your good looks and charm will win her over to the plight of the sysadmin. You never know!

[syringavulgaris.livejournal.com]

Which part of this catspew were you actually hired to fix? (please don't say "all of it" please please)

[ronebofh.livejournal.com]

One thing at a time, but i'm there for whatever the other dude wants me to do. If i don't know how to do it, i figure it out. In a way, it's rather refreshing and enjoyable because i'm stretching myself.

[bittercrackbaby.livejournal.com]

Both sides are to blame. IT is to blame because the people don't have the proper planning, risk management, and business skills to go along with questionable technical skills that go obsolete every 18 months. Management is to blame because they consider IT a cost center and won't touch anything until a major outage occurs or an employee burns out, and they do a poor job of idenifying talent and scoping requirements. Think about how many shops where IT and Management can effectively discuss core application business requirements establishing how many 9's it should have.

And you know my opinion of unions. I would argue there's a significantly higher ratio of bad IT folks in the industry as compared to bad school teachers, and not being able to wash out bad teachers is one of the major beefs I have with the education industry. Nobody good in the technical field will tolerate that at all.

[ronebofh.livejournal.com]

100%. But how can we make it better?

[eagle]

I'm afraid that the short answer is that nearly everything in our current world is based on the principle of the cheapest thing that could possibly work.

It's not just IT infrastructure. It's house construction, road construction, most consumer electronics, most computer hardware, clothing, furniture, packaging... it's everywhere. That's where the incentive structure is aimed. Proper design is an up-front cost with a multi-year payoff. I highly doubt the company is planning for a multi-year payoff from any of its computing infrastructure that isn't the code they're developing. They'll probably get bought out and throw out all their infrastructure anyway. So why put any time or money into it?

[ronebofh.livejournal.com]

Maybe it's just that i feel personally and professionally dissed by this crap, but it feels to me like it's more than just lowest-bidder culture. It's a willful sort of ignorance.

It can be demonstrated that effort put into the infrastructure will make any company a more attractive acquisition target (if that's the goal). So either it isn't being demonstrated, or the demonstration has no effect on the decision makers.

[dr-strych9.livejournal.com]

Yes, well I'd agree with tuwanda about the unionism thing, but I tend to favor the industrial unionism approach over the craft unionism approach.

[2wanda.livejournal.com]

Thanks, but its 2wanda That other bitch took my name and hasn't posted since forever. The nerve!

[dr-strych9.livejournal.com]

Crap. My apologies. I'd explain how I made that mistake, but it's tedious and not worth recounting.

[2wanda.livejournal.com]

No worries. I know it's confusing considering my email address is tuwanda. I've asked that woman to hand over my nickname on LJ a couple times, since she hasn't used it in 5 years, but she keeps ignoring me. Oh well.

I so support you ...

[drieuxster.livejournal.com]

I wonder about the false drieuxs who are out there corrupting the moral fibre of innocent children!!!

[nixzusehen.livejournal.com]

In addition the "IT is a cost center" mentality, I think there's also a huge number of SAs -- well, maybe not huge, but still way too many -- who see their work as an extension of their own messing with computers for fun, and treat it that way. That means they try to and cobble things together, do it on the cheap, etc.. It also means they don't understand the first law of systems administration (according to me): If it ain't broke, don't fix it.

I consider myself really lucky to have a boss who understands that that is not what you want in an infrastructure. When I say, "disk is cheap, but storage is expensive" he knows that and is quite willing to buy storage, even though we're a small company, ditto with servers. (Case in point: we have a $12,000 Netapp in a company with ten employees).

I've never been sure what the right solution to the training issue is there -- I have doubts about certification (just because you know what buttons to push or commands to type doesn't mean you have the sense to know you shouldn't be typing that right-fucking-now). The downside of Unions is that they tend to protect the crap, too.

Jay Fields briefly proposes this in in his blog post on net negative producing programmers (http://blog.jayfields.com/2009/01/cost-of-net-negative-producing.html): "Lastly, you could suggest moving developer compensation to be based on the success of the project. A royalties model would be really interesting." Maybe there's something be said for that... but of course, royalties have less obvious correlates in the SA world, and I can't imagine how one would measure success reliably.

Oh, and this quote sounds like a summary of my entire career: "[...] most good programmers spend the majority of their time fixing problems created by terrible programmers."

*sigh*

[ronebofh.livejournal.com]

The problem with "If it ain't broke, don't fix it" is that 'broke' is delightfully subjective. I think too many companies adopt IIABDFI far too liberally: "Is the site up? Good! So what are complaining about? If it breaks, fix it — that's your job."

Fields: "When (permanently employed) programmers screw up they get let-go." That's far too optimistic; in my experience, programmers who screw up get far more lenience than administrators who screw up. Yes, it may well be due to the fact that programmer screwups are less immediately visible and often have a lesser impact, but them's the facts nonetheless. In any case, damn good article.

[nixzusehen.livejournal.com]

It's true. I suppose a great deal of being a good sysadmin is knowing how to define "broke". Unfortunately, for far too many SAs "broke" includes "is .0.1 revs behind the bleeding edge."

I would tend to agree with you about programmers, but I think SAs *can* get too much slack, usually owing the fact that what we do is a black box to most management. They're used to their crappily written desktop apps crashing all the time, so have trouble distinguishing between that and actual SA created failures – which could also be bad when SAs get blamed for software failures, but that seems to be less common than the other way around. The public has been trained to expect crap software and we're all the poorer for it - literally and figuratively.

[ronebofh.livejournal.com]

Yeah, completely agree, and am even guilty of having taken advantage of that when things are busy.

[vatine]

My general guideline has always been "does not cause hassle". If a system (or a wodge of systems) are needing poking and prodding, it may just be normal, but it prompts me to look if ethre's something that can be done to improve stability or what-have-you.

Of course, these days the "systems" I mostly poke tend to be labelled "routers" and "switches" and they're simpler beasts, although their malfunctoins tend to have larger impacts.

I'm like totally missing your point here?

[drieuxster.livejournal.com]

Clearly as we all know, the best of all possible worlds has been created by the freedome of the free market to provide the bestest and the brightestest of free market entrepreneurs who understand that there is no problem in the whole of the great wide world that can not be solved by some scripting, and getting those lazy no good lay about sysadmin types to learn to put in a full 28hour day like jesus intended them!!!

All of this Terrorist Talk about Unionizing, is merely the Hatred of Freedom that comes from those who OPPOSED the greatness of BoingBoingBarbieBou and the true freedomes of the market!!! You should feel plesased that you live in america where the outsourcing of jobs is what makes the greatness great, that and our willingness to nuke unbelievers to defend the GodFearingWhiteChristianAmericanDollar!!!

So when you are willing to do your partriotic duty, your little problems will all go away in a puff of commitment to the True War President who is bringing the freedomes of the market to everyone everywhere!!!

[deirdremoon.livejournal.com]

That entire rant about where they are NOW is just horrifying. I'm glad, at least, that you are employed, and that this isn't a job where you have your heart set on being hired permanently... that sounds SO fuxxored.

Well crap.

[asienieizi.livejournal.com]

I guess I missed the memo that said you found another job. Belated congratulations on that, anyway.
(I used to be very anti-union until I spent a major portion of my life working in Right To Work States. The RTW concept seems to naturually flow into: Work For Less Pay, Eat Our Crap and We Will Fire Your Ass At Will For No Particular Reason Just Because We Can.)

That is just bad management

[toropeza.livejournal.com]

Sounds like a badly managed company - and no union could fix that. Just make it so they hires less disturbed sys admins :-)

We used to have a lot of those low-lifes. We fired them. Sorry they are still out there making a bad name for the good ones.

I do not like unions and I do not think they would improve life only add another cost onto the price of sys admins making the pay scale lower actually.

Show them was a great job an IT Professional can do!!

T