SupportSQL

The CEO corners me…

…with a quip and a pat on the shoulder, comes a little closer, stares me in the eyes and quietly asks a simple question, “Is our data protected?” It’s a common question for those with liability. What’s the answer given I don’t have regular update meetings with him, and he doesn’t want the full story of the entire disaster recovery implementation and its operational condition, the failures and achievements up to now. He just needs a simple “yes”, and only a “yes” will do. 

This simple question any superior executive could ask of anyone in IT. The answer is not so simple, but always “yes”. Caution: to say “yes” is to give a guarantee, and if I say it, it’s my guarantee. For the bounty he calls “our data” is what so much stands upon, and without which so much would be gone. Merely saying “yes” brings an obligation to be responsible for that valuable bounty.

If securing, or “not losing” the data means as it usually does, at all costs, then there are many critical facets that have to all be successful, to protect and secure data reliably. Ranging from tape storage vendors to corporate security intelligence, cryptography  and master keys, on and on. It’s a team effort and full of trial and error. Daily checklists with numerous repetitive critical tasks, a bullet proof auditing process, and a process to audit that process, it’s well beyond compliance. Then mitigate the un-knows derived from all the human involvement. The guarantee that goes with a simple “yes”, would make me cough and sputter if I didn’t have full knowledge over the entire current disaster recovery process and it’s immediate audit results.

Whoever is in charge of the backups and restores during a disaster recovery event has the highest awareness of this, or hopefully does. Taking this seriously is realizing ahead of time that the restore task we hope is never coming - is coming. It means being ready to do what’s been tasked and doing it at a time when maybe the whole company is depending,  and much worse, breathing down a neck, to do it perfectly and swiftly without fail. Don’t imagine it because it’s coming, be ready. To be ready is to experience it. That requires to “have full knowledge over the entire current disaster recovery process and it’s immediate audit results” and then fail it with planned purpose, and see what happens to the controls in place. If everything tests perfect that only means the odds are in your favor. Testing regularly improves those odds and brings peaceful sleep.

The persons directing and managing IT, the hardware team, communications, even maintenance and building security all have a part in the process, and most would need disclosure of the current process to be prepared for their part when inevitability knocks. It seems easy to treat the daily validation process in a compliance-like process manner, which tends to follow suit by falling back in daily priorities, as compliance driven goals tend to. It’s not about compliance, no matter the process resemblance. Rather sheer survival, and for many not just one.

I recall… I was hired at an engineering consulting company, decent size 2000+ employees global. My very first day, working where 85% of employees were engineers, a super smart workplace and I was happy to be there. But that first day I spent three

hours in the large board room, along with all the others in that office, everyone. No one could work, document processing was their whole normal day, but not today, for we all watched as six would have been my peers in IT were let go, leaving 2 and me. Not let go, I mean fired, publicly, and with prejudice after a 1hr+ lecture where heads hung low, in front of everyone. I never saw or heard of anything like that before, but I must say, it didn’t strike me as odd or extreme.

The other 150 employees in the room were not shocked, and not appalled, neither was I, but I was stunned while many were grieving. Not for their six peers, because of them. It was over a year since anyone had checked to see if the backup job that uses the tape the receptionist replaces, and has every night on rotation, was actually running. This morning when the server failed they checked and found it hadn’t, for a year. The work of over 200 employees and consultants, all email, the accounting system and payroll, was all gone. They all six shared the same. Publicly humiliated, lectured, made to face their friends turned victims, then ostracized. All because one scheduled job, one command didn’t do it’s thing, oh, for a year, and nobody checked, oops. People were mad is an understatement, some losing literally months of work that the clients already paid for, scientific analyses that could fill a warehouse, missing legal deliverables with serious implications, they were in trouble far beyond retyping everything. I learned a valuable lesson that day, the easy way. Know what is important in that which is daily taken for granite, it may not be granite. Its easy to see data like Niagara Falls, always flowing forever, but data is more like the light in a light bulb, when it goes, its gone in a flash. It’s comforting to see responsibility shared as the norm, but when things go really bad, heads always roll. Even if not fair. Be prepared.

Almost every company’s IT dept has a nightmare story or two of full nights or weekends working and for sure praying to get back data. Costs are never an issue then, and so are not now either. Fearing data loss = nightmares, at the very least impending gigantic embarrassment. The only way to sleep good, is to
“have full knowledge over the entire current disaster recovery process and it’s immediate audit results.”

The day is coming. Tell me, is your data protected?

Alek Kirstein

Shoot me an email or comment to this article.
Thanks for reading and participating.

 alek@supportsql.com

(Disclaimer: I have not tried to discuss the process definition or scope of a reliable disaster recovery process. Only the nature of it’s value.
Anything contained in this article is of my opinion alone and does not represent the opinions of the content providers or my employers.)