Case Study: Mechanics Bank
The Mechanics Bank Moves Backups to Disk and Reaps Large Rewards
Located 16 miles northeast of San Francisco, the Richmond headquarters of The Mechanics Bank no longer bears much resemblance to its humble beginnings in 1905. It has grown to offer the modern conveniences of Internet Banking, ATMs, and Debit Card services. Holding over $2.5 billion dollars in assets, this independent community bank now employs 640 people and consists of a total of 33 retail and commercial banking offices in California—from Napa and Marin to San Francisco and Sacramento.
The Mechanics Bank’s Vice President of Information Technology Richard Lewis has made a career in knowing when it’s time to evolve the bank’s internal processes and technologies. He recently recognized that the days were numbered for its tape-based data backup system.
Lewis came across an EVault white paper that described the idea of performing “virtual” full backups on disk—without tape. “I read it and said, ‘That’s what I want.’” It wasn’t long thereafter that Lewis began envisioning a better backup and disaster recovery paradigm for The Mechanics Bank.
For some time, The Mechanics Bank had taken a traditional route to backup and disaster recovery. Using the bank’s own internal tape backup processes, administrators were often tasked with rotating tapes in and out of the bank’s tape changer, labeling and tracking rotation sets and sending them periodically offsite.
When it came to performing annual tests at an offsite disaster recovery facility, however, problems invariably arose. One year it was the challenge of performing bare-metal restores for offsite servers with tapes as the starting point. “Another year, we ran a catalog of what was on the tape for one of our large imaging and cold systems, only to discover that the tape was empty,” said Lewis. “That wasn’t a great year, either.”
The tape issues during each year’s DR test were compounded by the lengthy time involved in managing tapes and restoring individual files. Combined, such issues soon led Lewis to one conclusion. “We just said, ‘enough is enough is enough’ when it came to tape. Do a little math and add up how much those tapes cost. Then add the fact that it’s not the most trustworthy of media. We said, ‘We’re getting out of this game.’” Lewis began to investigate disk-based backup solutions like EVault Software as a replacement for tape. He hoped such solutions could:
- Shorten data restore times for individual files or bulk data
- Minimize the amount of time spent overseeing backup and recovery
- Allow the bank to back up more server data offsite than before
- Improve security for managing data offsite
- Transfer more control in-house for offsite disaster recovery
- Help in the development of a multi-tiered backup architecture
Beyond the obvious issues that arose during annual DR testing, The Mechanics Bank’s IT team had also grown accustomed to fielding panicked calls on almost a daily basis from employees who had inadvertently deleted or corrupted an important file. “If you were lucky, that information would be in the database,” said Lewis. “But you still needed to find the right tape, load it if it wasn’t already in the changer, and do the restore. It was time-consuming from the perspective of our administrator.” If the data had made it to tape, Lewis said they could usually restore it, albeit slowly.
Extra-long backup jobs that cut into the next morning’s productive work time were also common. “We’d come in on a Monday and the system would still be writing to tape,” he said. The Mechanics Bank had gone through several iterations of tape backup architectures, from servers—each with its own internal tape drive — to a dedicated backup host on the network with a giant tape changer inside. “We did it all,” said Lewis, when it came to trying to make tape work for their growing data needs.
In the end, however, it wasn’t just the slowness of tape-based backups that prompted a change. “My biggest concern wasn’t just the inefficiencies. I was concerned about the integrity of the data on those tapes and the ability to encrypt them. As the cost of disk drives started to drop, I just didn’t understand economically why someone would stay with tape-based backups when you could gain so many benefits and save money by going with a solution like EVault,” he said.
Once he saw EVault Software and EVault DualVault in action and put the solutions through their paces, it wasn’t long before Lewis had mapped out a way to eliminate tape for backup in most of the bank’s data protection areas—with a move instead to EVault disk-based backups.
Now, The Mechanics Bank uses EVault Software with its DualVault configuration to ensure rapid recovery from any localized failure, or a broader, site-wide disaster. There’s a disk-based backup “vault” or repository at the bank’s primary data center in Hercules, CA, and a second disk-based backup vault at the bank’s offsite DR facility in Sacramento.
After getting its feet wet with EVault and disk-to-disk backup, Lewis has since evolved his DR portfolio further to include a tiered data protection architecture that now includes EVault, asynchronous, SAN-based data replication technology and server-based virtualization from VMware.
Today, EVault serves as a primary backup solution for all of the bank’s “Tier 2” data, which includes large file and print servers, and other infrastructure services like Lotus Domino messaging/email servers. In this regard, Lewis notes EVault has played a large role in allowing the bank to also back up certain supporting infrastructure servers even though they don’t necessarily deliver critical data. He admits this goes a long way to make life easier during the recovery process.
EVault also serves as a secondary backup for what Lewis qualifies as the bank’s “Tier 1” (or most critical) data requiring protection. Here, EVault is used in conjunction with The Mechanics Bank’s SAN-based asynchronous mirroring software to ensure fast recovery of the bank’s large “imaging and cold” server data (used to store checks, statements, and the business units’ daily reports). EVault also provides secondary recovery of a handful of VMware virtual machines that are also replicated asynchronously.
Lewis is particularly fond of a few things EVault has brought to the table. One is EVault’s early ability to encrypt data—both “across the wire” and while it resides on one of its disk-based backup vaults. The other is EVault’s DeltaPro™ technology that only backs up the smaller blocks of data that have changed since the prior backup. The impact of the latter for saving on WAN bandwidth is substantial, says Lewis. “Because EVault handles just the deltas, we can recover many more servers over that pipe than if they were doing a full backup every night.”
Thanks to his team’s positive experiences with EVault, Lewis remains sold on the concept of disk-based backup and restores. That’s been a factor in helping the bank to move forward with its aggressive, multi-tiered data protection architecture.
According to Lewis, the bank continues to benefit from its use of EVault in a number of ways. “EVault lets us recover more servers in a shorter timeframe. For example, you don’t need to catalog the tapes up-front, or worry about having tapes arrive from an offsite storage facility. This means you can start the true recovery process sooner in the game.”
Individual file restores are also now lightening-quick. “About a week after we had installed EVault, one senior manager involved in the decision called us and said he’d mistakenly deleted some data,” said Lewis. “We were able to restore that data while the manager was on the phone with us. That never happens with tape. In a concrete way, it was a validation of the decision to move from tape.”
Lewis also speaks glowingly of the change EVault has brought to his administrators’ workloads. “The number of hours spent in any given week on the backup process has been diminished. We don’t spend as much time swapping tapes or arranging for those tapes to be rotated offsite,” he says. “All these things add up to letting our network administrator focus on other issues that are more critical.”