The NSA Teaches the Importance of Testing

The NSA Teaches the Importance of Testing

October 31

We try our best to hammer the point home, but despite our constant prodding and cajoling, it seems our partners still have trouble testing backup and disaster recovery plans with their clients.

According to a recent survey conducted by StorageCraft, the number two most challenging thing about an IT professional’s backup and disaster recovery offering is performing the actual failover and recovery procedures. We also asked clients to tell us in their own words what the most difficult thing about recovering a failed machine is. A large number of them said that remembering how to perform the procedures is the most difficult thing. Failure is common, but evidently not common enough for our clients to be well-practiced at recovering machines quickly. The good news is that there’s a remedy to this: test your plan.

A recent report about the NSA could offer you some help while you’re thinking about testing. A recent Desert News article reported that the new NSA data center in Bluffdale, Utah had an electrical failure. According to another article on PCWorld, the data center has actually been plagued by chronic electrical surges for a while—not particularly surprising considering the $1.4 billion data center continuously uses 65 megawatts of electricity, enough to garner a power bill of over $1 million a month. That’s a lot of electricity to manage safely and effectively.

The article also cites a report by the Wall Street Journal claiming that the facility has suffered ten meltdowns in the past thirteen months, resulting in the damage of hundreds of thousands of dollars’ worth of machinery. The fact that these failures happened is actually a good thing because the data center isn’t open yet. The surges actually occurred during regular testing of the facility. By understanding the problems ahead of time, officials were able to fix them before it’s time to actually open the data center.

So of course, how can you know if something will work unless you test it first? The Deseret News reported that the new NSA data center was delayed by over a year due to these meltdowns, but the issues have been fixed. Soon enough the data center will hum to life and start monitoring everything you do, but you’ll rest securely knowing it has been tested for optimum privacy invasion.

So what’s the big lesson? When you’re creating a disaster recovery plan, you’re somewhat like the NSA building the data center. All the pieces seem to fit and everything seems to be fine on the surface. If you were to test the plan, you’d find all of the issues you need to fix and be ready for when show time arrives. I’ll drive the point home again: you can only be sure your plan will work if you’ve tested it ahead of time. Take time on a weekend or after regular business hours to make sure you can perform any necessary failover tasks. Your clients (and your reputation) can’t afford to suffer downtime while you figure out what you’re doing, so practice your disaster-resistance techniques, and practice them often.

The NSA data center is pretty neat– did you know it doesn’t actually contains backup data rather than unique data? Our article illustrates.

Photo Credit: albertogp123 via Compfight cc