The Job Experience, IT Plumbing: Year #5, Part 2

Kaboom

failure

So I’ve been there about three months.  I’m learning, getting adjusted to the culture, trying to get comfortable.  Everyone goes through this when they get a new job. You try to make the unfamiliar familiar, which takes effort and leaves you feeling off-balance at all times.

But I’m starting to feel like, OK.  It’s going to be fine here.  People like me well enough, and I’m slowly becoming assimilated into the collective.

Suddenly Statler and Waldorf are out on the same day.  And we have a trading system outage.  I’m the only one who can take it.

Mr. Manager asks me to join a conference bridge, where I’m met with about ten other people who are all wondering what’s going on.  Most of the voices on the call I don’t recognize.  And they’re all looking to me to tell them what’s happening.

I’m not sure.  I fumble a bit.  I ask them to wait while I look at the systems.  Everything looks OK from my perspective, and I relate this information on the call.

But <livingafi>, everything isn’t ok.  Traders A through X are having trouble doing <some application thing.>

OK, let’s restart everything.  Can we do that?

Sure.

Systems come back online five minutes later but the users are seeing the same problems.  Mr. Manager physically appears in my office and asks how it’s going.

Not good.  I’m not sure where else to look.  Logs are clean and a restart didn’t help.

Do you understand the problem?

Yes and no.  Some specific feature of the app isn’t working.

Should we call Statler?

Well, he’s sick right?   Actually, who cares.  I just heard another VP join the conference bridge.  Yeah, call him, I’m not sure what to do next.

Ten minutes later and a very tired sounding Statler joins the call.  We explain the situation and he asks us to check a few things.  Another ten minutes later, and he’s restored functionality.

Whoever says muppets aren’t awesome is an idiot.


Later that afternoon, Mr. Manager calls me into his office and asks why I wasn’t able to resolve the issue. Was I not getting enough training? What did I need that I wasn’t getting?

I wasn’t sure. I told him I was doing everything I could to learn the environments.

He asks me to think it over and get back to him. He’s disappointed that he had to call one of his guys to help out. And the business was not happy with the length of time it took to restore full application functionality — nearly an hour end to end.  I don’t want to consider the cost estimates of the outage.  I’m sure it’s more than I’d make in a hundred years working for FinancialCompany.

Statler is back in the office the next day. I’m completely depressed and anxious at the same time, hopped up on coffee but simultaneously feeling like I need a day off myself to recover from my own failure. But I force myself to talk to him and guide me through what he did to troubleshoot and fix the issue.

It ends up being a constructive experience, and I learn additional things to check in order to solve problems. Still, this was my first taste of trying to lead troubleshooting efforts on a large conference bridge. And I failed.

I had nothing on Sherlock Holmes.  Hell, I wasn’t even Watson.  My best estimate was that my technical sleuthing abilities most closely resembled Sherlock Hemlock, the bumbling Holmes parody from Seasame Street.

If I can assign slightly derogatory nicknames to my teammates, it's only fair that I do it to myself, too.

Uh, not sure what you’re looking for up there, Hemlock.

Even though Statler was great about it – kept telling me not to worry about it, I’m still super-new and everyone understands that I can’t solve everything – I felt deflated and low for days.  Thoughts raced through my head like “You’re a phony and a fraud.  You made them hire you even though you’re not qualified and now you’re being exposed.  Serves you right.”

It wasn’t rational thinking, but I couldn’t help it.

Feeling like a failure became a staple of the job because for every ten issues you resolved, there was one you couldn’t.  And because of the way that people work — we tend to reflexively focus on the negative — the issues that stuck out in peoples’ minds and memories were the ones you had a lot of difficulty with.

Sure, YOU remember that tricky son-of-a-bitch that you fixed with a Eureka moment that would have made Newton proud.  But no one else remembers it, because that’s the outage that you resolved in two minutes flat — barely a blip for the company.

I will say that the missteps were usually a good thing overall, because they provided motivation to shore up whatever holes you had in your knowledge that prevented you from initially resolving the issue. Absolutely, they made you stronger in the end.  All of the cliches about failure being an essential component of success are completely true.

But all of that being said, it doesn’t change the fact that it’s really hard to focus on the positive side of screwing up during epic fails that are aired out on public calls, your fuckuppery visible for all the world to see. While the act is in progress, it just sucks.  Your hands shake.  Your voice trembles.  And you feel like a miserable pile of worthless donkey shit.

Or maybe that’s just me.

I have to call bullshit on this, Yoda -- I tried pretty hard.

Sorry, Master, but I call bullshit on this.  Trying is the path toward Doing.

 


I should point out that I eventually became more accustomed to these sorts of things after a while. It took perhaps another year or so to gain confidence and stop dreading them completely.  Call it ‘toughening up’ or ‘learning’ or ‘hardening,’ whatever you like — it takes time coupled with additional experience, and in turn this greatly reduces stress.  I stopped trying and started doing, an important developmental shift.

You can see, though, that the memories around these types of system-down events are not all that fun to recall — and they happened constantly. When it came to resolving major outages, it doesn’t matter if you’re in year 1 or year 100.  The Job Experience was never a good time.

This entry was posted in Backstory. Bookmark the permalink.

8 Responses to The Job Experience, IT Plumbing: Year #5, Part 2

  1. Dwayne Hoover says:

    “After a while certain things build up in your mind over time like plaque in your arteries and the only way to flush them out is to get up and walk to a new place. I could feel the bits of congealed frustration breaking loose and getting cleared out.”

    This part is a punch to the gut. B2B sales role for 4 years, but I’m about to take a non-selling role with my company if I can last another 30 days or so without quitting. The closer I get, the harder this job is. Probably twice a week, I’ll wake up in the middle of the night, after basically have a nightmare about a technical issue with a customer – some former, some current, some a figment of my imagination. Can’t wait to clear my arteries. First thing I’ll be doing is changing my cell phone number (which my customers have).

    You’ve got a really unique voice. Keep up the awesomeness.

    • livingafi says:

      Hey Dwayne, really sorry to hear about the job situation. That totally sucks and as you know from my blog posts, I can completely relate, especially to the sleeping bit. Too many nights like that and you know it’s time for a change or you’re going to start getting ulcers. You can definitely make it for a month though, and when you get there you’ll feel SO much better, trust me. That big fat job-change reset button will get hit on the stress. Sure, your new position will be challenging but good-bye clotted up old junk. A good trade.

  2. Gamergirl says:

    I can’t believe they chastised you for bringing your lunch to work!

    And I agree with you about dressing up for work when you work in Accounting or IT is stupid. I turned down several job offers at places with strict dress codes.

  3. DS says:

    Really appreciate this series! Found your blog through either MMM or BNL and have been reading for 3 months or so. Just hit my one year mark in IT (basic project mgmt and data analysis) at a financial-based company. Between the incident counting, meeting sitting, and progress reporting, I have related a little too much to what you’ve written. Wishing every day I was hiking or doing anything but sitting in that office. Thank you again – I will continue to read, and I hope to reach FI some day soon!

    • livingafi says:

      Hey DS, thanks for stopping by. You’ll definitely make it — the only thing really required for FI is a good savings rate, and consistency i.e. sticking to your plan over time. Looking outside your window on nice days when you’re stuck in the office will help keep your motivation up and there’s no doubt you’ll stay on track. GL!

  4. Raffy says:

    I’m a new fan! I’m enjoying reading through this series. This week I had to deal with the disappointing feeling of failure and being a fake when everything is riding on you. I’m learning to accept it and move on… it’s just work after all, and the harshest critic is always yourself. Keep up the excellent writing!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s