The Test System That Wasn't - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
IT Life
Commentary
9/12/2014
08:07 AM
Grumpy IT Guy
Grumpy IT Guy
Commentary
50%
50%

The Test System That Wasn't

Why is Grumpy IT Guy extra grumpy today? It's crazy vendor day!

[Editor's Note: Please welcome our new IT Life columnist, Grumpy IT Guy. We are keeping his real name secret to let him speak the unfiltered truth to power and the clueless. This sage veteran leads an IT organization somewhere in America. Ask him questions using the comments field below. You may not like the answer, but you may need it.]

Grumpy IT Guy: We have a system that was put in years ago. Company paid big money for enterprise app vendor to install a parallel separate test environment. Best-practice, right? Test environment means stupid vendor patches kill test environment, not production.

But recently, sysadmin gets a patch. Sysadmin installs the patch into the "test" environment. Boom! Production environment for hundreds grinds to a halt. Which part of "separate test environment" did our vendor not understand?

Vendor: These things happen.

Grumpy IT Guy: Are you serious?

Vendor: Sorry?

Grumpy IT Guy: Test systems should NEVER blow up production systems.

Vendor: Oops.

Grumpy IT Guy: So what are you going to do about it?

Vendor: Try again?

Grumpy IT Guy: NO.

Vendor: Make a new patch and try again?

Grumpy IT Guy: NO.

Vendor: Surely you don't want us to stop randomly throwing code over the fence to you.

Grumpy IT Guy: That would be nice.

Vendor: Surely you don't want us to actually figure out how the two systems got connected and how to separate them.

Grumpy IT Guy: Arrrrgh!

[Do you love your job? See What Willie Nelson Taught Me This Summer.]

Question is now whether the vendor is going to charge for system reconfig, or if my system admin is going to do it. The vendor is optimistic -- would be glad to charge us to rebuild everything. So now the quizzing down of the sys admin begins.

Sysadmin: Um, can I have money to pay the vendor?

Grumpy IT Guy: How could this happen?

Sysadmin: I don't know.

Grumpy IT Guy: Isn't it your system? Why don't you know?

Sysadmin: I don't know why I don't know.

Grumpy IT Guy: Why wasn't this a problem before?

Sysadmin: Do you want me to guess?

Grumpy IT Guy: Did something change? Did we do something? Did the vendor do something?

Sysadmin: Um.

Grumpy IT Guy: Could you, maybe, kind of... FIND OUT?

Sysadmin: (Flees)

Who wants to take bets about what the problem was?

Will it be the database of doom, where someone decided to migrate a database to the wrong server? Will it be the application server of annihilation, where someone bonked the load balance order? Will it be the Web server of woe, where the change manager of confusion approved the wrong Web wrangling? I guarantee you that it will be human error, regardless of the vendor and the system admin vocalizing things like "software issues" and "load balancer problems." I know they privately agree with me, because they are both doing some low-key finger pointing at each other.

Will it never end? Take your guess in the comments section below.

If the world wasn't changing, we might continue to view IT purely as a service organization, and ITSM might be the most important focus for IT leaders. But it's not, it isn't and it won't be -- at least not in its present form. Get the Research: Beyond IT Service Management report today. (Free registration required.)

Grumpy IT Guy avoided historic disasters and clueless people while working his way up the IT ranks, but he retained his keen sense of humor. He now leads an IT organization somewhere in America, as part of the FBI's Grump Protection Program. Need advice? Send questions to ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Page 1 / 2   >   >>
dr2127dr
50%
50%
dr2127dr,
User Rank: Strategist
9/17/2014 | 12:45:08 AM
Re: It's Never the Vendor's Fault...
Susan,

 

Not generally - the one I mentioned seems to be the exception. On the other hand we have cloud with the #1 provider of data warehouses and they are the opposite - very zealous on the dev, test and production  process. They are new to cloud but have been around forever. If you know data warehouseing systems you know who I'm talking about.
Susan_Nunziata
50%
50%
Susan_Nunziata,
User Rank: Strategist
9/16/2014 | 6:51:38 PM
Re: It's Never the Vendor's Fault...
@dr2127dr: Are you seeing a more lax attitude about test systems from SaaS/Cloud vendors than you did from vendors selling on-premise systems? Or is this an across-the-board issue in IT?

And, to answer your question: when all of us old dogs bag it who is going to manage this kind of crazy behavior?

I'd say we are all doomed.
Susan_Nunziata
50%
50%
Susan_Nunziata,
User Rank: Strategist
9/16/2014 | 6:48:15 PM
Re: Here's one guess
@moarsauce123: I wish you were makiing up the part about the Sys Admin not knowing what Firefox and Chrome are....that is truly frightening. Excellent story, though. I did laugh out loud reading that, even though it must have entirely frustrated you when it happened.

But really, why can't we just make all our customers visit our website only using IE?

<grin>
moarsauce123
IW Pick
100%
0%
moarsauce123,
User Rank: Ninja
9/13/2014 | 10:30:08 AM
Here's one guess
OK, my guess does not really fit because no vendor is involved, but here it goes because it just happened to me yesterday:

Me: Hosting sys admin, customers complain that they see a certification warning when they hit our web sites. I can reproduce this and it appears as that the certificate is expired.

Sys admin: Ah yea, that is possible, I got a bunch of emails throughout the past weeks from the issuer.

Me: And you ignored them?

Sys admin: Well...the cert was still good.

Me: Until today when the customers start complaining.

Sys admin: Put in a helpdesk ticket so that we can schedule the work

Me: What???? You go and fix this right now, we are dead in the water! You've gotten enough emails to have you schedule that work!

Sys admin: Well...I talk to my manager and see what we can do.

....some hours later....

Sys admin: OK, we managed to squeeze this work order in. It is all set.

Me: Let me try it right now.....works fine in IE, but not with Firefox.

Sys admin: Forefox? What is that?

Me: It is a web browser, it is the second most used browser of our customers after Chrome.

Sys admin: Like chrome on a car?

Me: Exactly! *groans* No, they are both web browsers. Now go back to your manager and tell him to tell you to fix it!

Sys admin: Can you submit a helpdes...

Me: NO!!!

Sys admin: Alright...I'll ask.

...some hours later....

Sys admin: You are all set. Someone misconfigured IIS and the new cert now works for other browsers as well. Would have been easier to tell all customers to stop using these wonky browsers and go with what is the best on the market: Internet Explorer.

Me: Won't fly...but thanks for fixing it. I can confirm that it works now.

...a few minutes later...

Me: Hey, sys admin, customers are still complaining, they again get the expired cert warning. What the heck?

Sys admin: Uhmmmm.....helpdesk ticket??

Me: No, manager, pronto, fix it!!

...a few minutes later...

Sys admin: Oh, this is now all fixed. We spun up a test server to check out different certs and it happened to be an exact clone of your web server so we got an IP conflict.

Me: *cries* Thanks!
Joe Stanganelli
50%
50%
Joe Stanganelli,
User Rank: Author
9/13/2014 | 7:44:38 AM
Testing
The sad part is that good test processes do not begin and end with a testbed; vendors need to be doing more to test their products and patches.

As an IT old-timer told me in a recent interview: "One QA lady I knew rolled a Coke can across the keyboard as her first test—if your application failed to recover, you got it back right then."
Joe Stanganelli
50%
50%
Joe Stanganelli,
User Rank: Author
9/13/2014 | 7:41:19 AM
Re: It's Never the Vendor's Fault...
@dr2127dr: I wouldn't worry.  That's how new dogs become old dogs -- by dealing with this kind of garbage!  ;)

Technocrati
50%
50%
Technocrati,
User Rank: Ninja
9/12/2014 | 10:57:38 PM
Misunderstood Pencil Pushers and Web Surfers ?

Wow !     I really enjoyed this new section of Grumpy IT Guy.     This could have been called "Tech -Guy Uncut" as well  but I think grumpy is a good description.    Because let's face it, even though the IT Guy's or Girl's main purpose in life is to "protect and serve" .  Everyone is entitled to a bad day, week or month even.   I know many would say a month is pushing it, well as a result of this view -  the Grump IT Guy really starts to fester. Because at this point, you are surly talking to someone outside of tech, and most season tech guys know that is a losing battle.

To hear this type of concern from a "pencil pusher" or the proverbial "professional "web surfer " ( Is it time to go home yet ? ) only servers to raise the blood pressure.  And at this point, even the effort to defend one's  self only serves to produce more stress.  There used to be only two certainites - death and taxes.  Well I think we can add this situation to it as well.

The situation depicted in the first episode of Grumpy IT Guy, happens to IT people at least once a year.  And If near disaster does not almost strike  year after year - there would be no grump IT Guy. 

It is refreshing to consider that  No.  We ( IT professionals )  don't always have to come running with a smile because we are probably dealing with something that doesn't require one.   There is probably nothing to smile about and I wish - the web-surfer and the pencil pusher would be aware enough to understand that.  

But apparently I am asking for too much because I have never seen this understanding extended for an extended period of time.  Because "bad stuff" just doesn't happen in their worlds.  Even after the pencil pusher lays you off - they don't consider that "bad stuff".  And if you are lucky enough you might even catch them smiling after they do it.  Ah the irony, pencil pushers and Web Surfers are good for that - always.

So here's to the Grumpy IT Guy - I am sure he will be telling it like it is and that is good because I lost my rose covered glasses years ago. 

dr2127dr
IW Pick
100%
0%
dr2127dr,
User Rank: Strategist
9/12/2014 | 7:18:12 PM
Re: It's Never the Vendor's Fault...
This is prefect. We have a supply chain platform that we moved from on premise to SaaS / cloud. Same vendor - this was an upgrade.

So when I asked where is the test system - they said what test system? - I said, you know the system where we test things before cut over and or move changes into production for one of the most critical systems in a retail enterprise. They said - we don't have a test system - we test the code and then move it to to your production system - I said - really and if it goes bad then what? They said - oh we fix it ...... and non of our other custoemrs have asked for this. Really - not even the largest jean manufacture on the planet?

you get the picture - in the end we provided the test system.....  This stuff happens a lot more than everyone would admit. My question - when all of us old dogs bag it who is going to manage this kind of crazy behavior? 
Somedude8
100%
0%
Somedude8,
User Rank: Ninja
9/12/2014 | 2:27:18 PM
My guess
I am going to put my money on a screwed up config that points some little deep dark hidden piece of both environments to a production database. The config mistake happened a long time ago. The recent patch made some schema changes (that were probably pretty questionable to begin with, but vendor ideas of good database design is another train wreck story for another day...).

Someone is going to work a 70-80 hour week without overtime to figure this out and fix it, and it won't be the person who caused it. At least 3 C-level people will get involved, certain that if they are assertive enough that the problem will get fixed. One mid level manager is refreshing his resume right now and checking LinkedIn for the first time in ages, certain that at least one of his 11 contacts must know of an opening somewhere, and will be on Dice.com by the end of the day, which will trigger some alert based on them accessing known job hunting URLs from the office. That alert will never be read by anyone.

Yes, I have seen this movie.
TerryB
100%
0%
TerryB,
User Rank: Ninja
9/12/2014 | 1:24:07 PM
Re: No consequences
@Kevin, I think you need to work with better class of vendor and sys admin. That is an exception experience, not a "day in the life".

I think the real msg here is just how complicated things have gotten in systems. It's a wonder to me anyone gets anything to work anymore in these multi server, multi role implementations. And it is huge factor in implementing security properly.

But doesn't appear to be going away. Now everyone wants to go to "cloud", that magical place where everything is secure and just works. But nobody is exactly sure how or why.....
Page 1 / 2   >   >>
Slideshows
What Digital Transformation Is (And Isn't)
Cynthia Harvey, Freelance Journalist, InformationWeek,  12/4/2019
Commentary
Watch Out for New Barriers to Faster Software Development
Lisa Morgan, Freelance Writer,  12/3/2019
Commentary
If DevOps Is So Awesome, Why Is Your Initiative Failing?
Guest Commentary, Guest Commentary,  12/2/2019
White Papers
Register for InformationWeek Newsletters
State of the Cloud
State of the Cloud
Cloud has drastically changed how IT organizations consume and deploy services in the digital age. This research report will delve into public, private and hybrid cloud adoption trends, with a special focus on infrastructure as a service and its role in the enterprise. Find out the challenges organizations are experiencing, and the technologies and strategies they are using to manage and mitigate those challenges today.
Video
Current Issue
Getting Started With Emerging Technologies
Looking to help your enterprise IT team ease the stress of putting new/emerging technologies such as AI, machine learning and IoT to work for their organizations? There are a few ways to get off on the right foot. In this report we share some expert advice on how to approach some of these seemingly daunting tech challenges.
Slideshows
Flash Poll