Earthquakes, tsunamis, tornadoes, fires, and floods.
The devastation of such natural disasters made 2011 a challenging year for many IT and network administrators throughout the world, who fought the elements to keep their systems running and available to the millions of users who depend on these systems.
But for Anita Gingras, Network Administrator for Greenfield Savings Bank (GSB) in Massachusetts, U.S., 2011 was a reminder of another major cause of server crashes: user error.
“After an incorrect password was entered, the domain administrator account for our Exchange server locked up in the middle of a transaction, which caused corruption in the database. Then, free space ran out, and everything stopped, so users had no access to their emails, calendars, nothing.” (Keep reading below or view the PDF).
Looking For A Solution That Works
Prior to this disaster, GSB had a mainstream system recovery product to back up their servers, but Anita had been dissatisfied with its recovery function, particularly with its granular restore capabilities for Exchange.
“We’d get backups that look fine, but when we tried to restore a server, an Exchange mailbox or individual emails, it’d fail. So we decided to look for another solution, and our vendor recommended that we try StorageCraft® ShadowProtect®.
ShadowProtect is best-in-class backup and disaster recovery software, with the ability to restore any Windows server, regardless of different hardware.”
While the competing system recovery and backup and disaster recovery products used by GSB previously touted many of the same backup features, ShadowProtect focuses on fast and reliable recovery. Gingras continued:
“When the disaster happened, we happened to be in the second week of evaluating ShadowProtect, and had taken backup images of our Exchange server, but no one was trained yet on how to use it. So I immediately called StorageCraft to figure out what we could do to get the server back up and running.”
Smiling, she added, “Brett was the lucky guy who got my case.”
Brett Twiggs, StorageCraft Senior Sales Engineer at the time, explained what happened:
“Since GSB was evaluating ShadowProtect and didn’t have a maintenance agreement with us yet, I supported them on their issues as part of our pre-sales engagement.”
Racing Against the Clock
Anita’s first priority was to restore Exchange database access to GSB users.
At 100 GB in size and a limited throughput to the server, it would take approximately fourteen hours to restore just the Exchange database. Moreover, Microsoft Support recommended a bare metal restore (BMR) of the Exchange server, which would add more time to the recovery process.
Working remotely from Utah by phone, Twiggs made several suggestions to take advantage of fast recovery options in ShadowProtect.
“A BMR of the physical Exchange server could mean users wouldn’t have access for up to several days, which would be disastrous for the bank. So to speed things up, we decided to use VirtualBoot.”
The VirtualBoot feature of ShadowProtect lets you quickly access a server’s data by booting a ShadowProtect backup image as a virtual machine.
After locating spare hardware to host a virtualized Exchange server, Anita’s team of network and IT administrators started the VirtualBoot quick recovery function. They soon realized, however, that because of the amount of data stored on the mail relay appliance, there was not enough free space on the C: drive to complete the process. According to Twiggs,
“We moved the temporary VM directory to a different drive with more free space. This allowed us to spin up a complete virtual Exchange server on the spare hardware. Once the hardware was in place and the temporary directory was moved, the VirtualBoot process of creating a temporary VM took about eight minutes.”
Using the various tools included in ShadowProtect, including Hardware Independent Restore, which allows recovery to different physical or virtual hardware, the backup image of the physical Exchange server was restored on the spare hardware, thereby recovering the Exchange server and giving access to GSB users in under two hours. That was just what Gingras needed.
“The virtualization process was quite time-intensive. But our RTO [Recovery Time Objective] for the Exchange server is 48 hours, and we recovered in under that time, with no data loss, so I was happy.”
But we weren’t done—we still had to rebuild the original Exchange server.
Saving the Day
With access to Exchange restored on the virtual server, and with backups continuing as usual, Anita’s team had time to rebuild the physical Exchange server on new drives with plenty of free space.
With Brett’s guidance, Anita used the HeadStart Restore® (HSR) feature in the complementary StorageCraft ShadowControl™ ImageManager to begin restoring backup images to the original Exchange server while the new “virtual” production server was still running.
ShadowControl ImageManager, which includes HSR, is an optional program that drastically reduces recovery times and management of backup images. According to Twiggs,
“HSR also allows you to break up the volume restore process into multiple stages, which is useful with large volumes that would normally take days to restore.”
According to Gingras, the switch over from the virtual server to the new physical server went seamlessly.
“Using the ShadowProtect tools, we moved the entire Exchange database from the temporary virtual system to the new physical chassis without any of the bank’s users noticing the change. It went very smoothly.”
The Exchange server crash incident helped improve business continuity procedures at GSB.
Now promoted to Senior Network Administrator, Gingras reflects on some of the lessons learned.
“We pay more attention to the free space requirements of our systems. And we take full advantage of the tools and features available in ShadowProtect.”
Twiggs summarized the experience like this:
“Taking backups is important but worthless if you can’t recover quickly and reliably when the unexpected happens.”
Image source: pshutterbug