Kaboom

So I’ve been there about three months.  I’m learning, getting adjusted to the culture, trying to get comfortable.  Everyone goes through this when they get a new job. You try to make the unfamiliar familiar, which takes effort and leaves you feeling off-balance at all times.

But I’m starting to feel like, OK.  It’s going to be fine here.  People like me well enough, and I’m slowly becoming assimilated into the collective.

Suddenly Statler and Waldorf are out on the same day.  And we have a trading system outage.  I’m the only one who can take it.

Mr. Manager asks me to join a conference bridge, where I’m met with about ten other people who are all wondering what’s going on.  Most of the voices on the call I don’t recognize.  And they’re all looking to me to tell them what’s happening.

I’m not sure.  I fumble a bit.  I ask them to wait while I look at the systems.  Everything looks OK from my perspective, and I relate this information on the call.

But <livingafi>, everything isn’t ok.  Traders A through X are having trouble doing <some application thing.>

OK, let’s restart everything.  Can we do that?

Sure.

Systems come back online five minutes later but the users are seeing the same problems.  Mr. Manager physically appears in my office and asks how it’s going.

Not good.  I’m not sure where else to look.  Logs are clean and a restart didn’t help.

Do you understand the problem?

Yes and no.  Some specific feature of the app isn’t working.

Should we call Statler?

Well, he’s sick right?   Actually, who cares.  I just heard another VP join the conference bridge.  Yeah, call him, I’m not sure what to do next.

Ten minutes later and a very tired sounding Statler joins the call.  We explain the situation and he asks us to check a few things.  Another ten minutes later, and he’s restored functionality.

Whoever says muppets aren’t awesome is an idiot.

Later that afternoon, Mr. Manager calls me into his office and asks why I wasn’t able to resolve the issue. Was I not getting enough training? What did I need that I wasn’t getting?

I wasn’t sure. I told him I was doing everything I could to learn the environments.

He asks me to think it over and get back to him. He’s disappointed that he had to call one of his guys to help out. And the business was not happy with the length of time it took to restore full application functionality — nearly an hour end to end.  I don’t want to consider the cost estimates of the outage.  I’m sure it’s more than I’d make in a hundred years working for FinancialCompany.

Statler is back in the office the next day. I’m completely depressed and anxious at the same time, hopped up on coffee but simultaneously feeling like I need a day off myself to recover from my own failure. But I force myself to talk to him and guide me through what he did to troubleshoot and fix the issue.

It ends up being a constructive experience, and I learn additional things to check in order to solve problems. Still, this was my first taste of trying to lead troubleshooting efforts on a large conference bridge. And I failed.

I had nothing on Sherlock Holmes.  Hell, I wasn’t even Watson.  My best estimate was that my technical sleuthing abilities most closely resembled Sherlock Hemlock, the bumbling Holmes parody from Seasame Street.

Uh, not sure what you’re looking for up there, Hemlock.

Even though Statler was great about it – kept telling me not to worry about it, I’m still super-new and everyone understands that I can’t solve everything – I felt deflated and low for days.  Thoughts raced through my head like “You’re a phony and a fraud.  You made them hire you even though you’re not qualified and now you’re being exposed.  Serves you right.”

It wasn’t rational thinking, but I couldn’t help it.

Feeling like a failure became a staple of the job because for every ten issues you resolved, there was one you couldn’t.  And because of the way that people work — we tend to reflexively focus on the negative — the issues that stuck out in peoples’ minds and memories were the ones you had a lot of difficulty with.

Sure, YOU remember that tricky son-of-a-bitch that you fixed with a Eureka moment that would have made Newton proud.  But no one else remembers it, because that’s the outage that you resolved in two minutes flat — barely a blip for the company.

I will say that the missteps were usually a good thing overall, because they provided motivation to shore up whatever holes you had in your knowledge that prevented you from initially resolving the issue. Absolutely, they made you stronger in the end.  All of the cliches about failure being an essential component of success are completely true.

But all of that being said, it doesn’t change the fact that it’s really hard to focus on the positive side of screwing up during epic fails that are aired out on public calls, your fuckuppery visible for all the world to see. While the act is in progress, it just sucks.  Your hands shake.  Your voice trembles.  And you feel like a miserable pile of worthless donkey shit.

Or maybe that’s just me.

Sorry, Master, but I call bullshit on this.  Trying is the path toward Doing.

I should point out that I eventually became more accustomed to these sorts of things after a while. It took perhaps another year or so to gain confidence and stop dreading them completely.  Call it ‘toughening up’ or ‘learning’ or ‘hardening,’ whatever you like — it takes time coupled with additional experience, and in turn this greatly reduces stress.  I stopped trying and started doing, an important developmental shift.

You can see, though, that the memories around these types of system-down events are not all that fun to recall — and they happened constantly. When it came to resolving major outages, it doesn’t matter if you’re in year 1 or year 100.  The Job Experience was never a good time.