Fixing the fragmented approach to Backup, Archiving, Disaster Recovery & Business Continuity

Despite decades of research and development into backup and data protection, enterprise customers are generally unsatisfied with the status quo when it comes to data protection and recovery processes. Business continuity, disaster recovery, backup and archiving are common, overlapping data center operations, and are repeatedly targeted for replacement.

Rather than simply moving from one vendor to another, enterprise customers are seeking a new paradigm for data and workload protection and recovery, one that provides a complete view of all their applications and systems, is verifiably effective, cost effective and resilient by design to various calamities which might strike the enterprise.

According to research done by Gartner:

By 2013, at least 20% of organizations will have changed their primary backup vendor due to frustration over cost, complexity and/or capability, up from the typical single-digit percentage shifts today. By 2013, more than 50% of midsize organizations and more than 75% of large enterprises will implement tiered recovery architectures.– “Best Practices for Addressing the Broken State of Backup”, 27 August 2010

The Broken State of Enterprise Recovery

Data Growth

The largest single contributor to dissatisfaction with enterprise recovery is the massive increase in the data to be protected. Estimated by industry analysts at around 45% growth per annum, data is growing so rapidly that many IT leaders are facing an unsustainable situation. Richer digital media (pictures, video, and sound), increased collaboration and connectivity, increasing use of “big data” by marketers, versioning, replication and the monolithic nature of application design contribute to the rising tide of enterprise data growth.

Beyond the growth of the data, the amount of data in protection/recovery systems is growing even faster, as companies add more systems to the list of ones that have to be protected. Additionally, protection requirements are changing from simply keeping a week’s worth of daily backups to keeping a day’s worth of 15-minute snapshots, a week’s or month’s worth of dailies, and years’ worth of monthlies, sometimes for ten years.

Backups Are Broken

Too much data means fewer and poorer backups, which creates risk to the organization – the risk of downtime, lost productivity, lost data, and financial loss. Expectations from business leaders for service levels on recovery times and the volume of single occurrence data loss have increased to the point where near-zero system downtime is a commonly accepted goal.

The very idea of the “backup window” has been obsoleted by the 24×7 nature of business. An outdated paradigm for backup and recovery where systems are offline once a week for a full backup, offline daily for incremental backups, and where data is copied to long term storage medium, has become nearly impossible to accomplish successfully.

Fragmented Approach to Recovery

Further complicating matters, companies have built up, over time, fragmented approaches to recovery. First there was the backup system with tapes being driven to a vault. Then perhaps a do-it-yourself (DIY) disaster recovery plan was implemented at a co-location facility other than the primary one, with manually managed system upgrades and tests, if the DR plan is tested at all. Some applications may have their own DR plan, like the inline SaaS offerings for email. Some corporate data may, of late, be in a cloud storage or cloud backup service.

In short, things are a mess, with no one quite sure where all the data is or how to get at all of it if necessary. A lack of a comprehensive approach leaves the IT team with a training and personnel issue which ultimately adds to cost and complexity. What’s called for is a standard way of backing up and recovering all services.

While data loss and system outage are unacceptable, so too are the escalating costs of managing all this data. While a company’s IT systems can be an effective differentiator and a strategic investment, data protection and recovery is not viewed as such. This has led to a growing realization among IT leaders that scarce resources (primarily people) would be better off focused on strategic IT initiatives that differentiate the organization from its competitors, rather than being consumed with mundane tasks such as backup and recovery.

We’ll continue this topic in a future post…

Unfortunately, this is just a baby step towards something that already exists: a unified platform that provides continuity, disaster recovery (DR) and archiving, obviating the need for backup at all.

We call this Three Phase Workload Protection. “Workload”, not “data”, because what’s being protected is the entire server: OS, application, configuration and data. It’s a super-set of data protection. Lose a file? No problem, mount a saved workload as a disk and retrieve the file. But lose a whole server? Also no problem – spin up the saved workload as a virtual server and get back to work.

The key to the Three Phase approach is that the primary (production server) is snapshotted frequently, and the snapshots then automatically proceed through the phases, with no manual intervention needed.

Why are Continuity and DR different? In our view, it’s about what you need to recover, and the best place to recover from. There are three things you can lose:

1. Data
2. A server
3. The whole data center

The first two can and should be recovered locally, from a saved workload that’s on your LAN. No big, hairy failover needed. Self-service, fast, no latency. We call that “Continuity”. (In the case of a data recovery, that’s “backup”, which is simply a feature of the Three Phase approach.)

Backup has been relegated to feature status

If you lose your whole data center due to blackout, hurricane, flood, or what have you, you need to failover to a remote facility. That’s Disaster Recovery.

When the workload images get old enough that they are no longer if use for continuity or DR, but still need to be retained for regulatory or policy reasons, they are automatically moved to archival storage. And of course, old workload snapshots can be automatically discarded based on policy.

Sure, we exaggerate when we say that “backup is dead.” It isn’t, it’s simply been relegated to feature status in the much more comprehensive Three Phase Workload Protection model.

But Mother Nature earns all her headlines for good reason. The big, capital-D, Disasters also matter. And there’s a lot of them out there.

A recent story I read pointed back to a map that the New York Times published last year, “Where to Live to Avoid a Natural Disaster”. It is not encouraging. Between hurricanes, tornadoes and earthquakes, there aren’t a lot of businesses in the US that can take the chance of not having a DR solution to handle The Big One.

I did find a small corner of Wyoming that looks pretty safe, so if you’re a fan of winter on the Great Plains, I guess you could move your business there.

To reinforce the point, the Insurance Information Institute just published their data for 2011, and to the surprise of almost no one, it was a record-setting year for natural disasters.

This isn’t intended as fear mongering, but as a reality check. Back when disaster recovery was complex and costly, you could understand why a business might try to bravely ignore the risk from natural disasters. But the time for whistling past the graveyard is over. Modern, cloud-based recovery services have seriously altered the economics of DR. It’s time for businesses to stand up to Mother Nature.

Methods of Data Protection

This brings us to the issue of choosing data protection technologies. If you have one of the rare, homogeneous environments, this can be a straightforward matter, as simple as using the replication technology available with your virtualization platform of choice, like VMware’s Site Replication Manager.

More typical data centers with physical and virtual servers, Windows and Linux, and so forth, will need a replication solution with broad OS and application support, like FalconStor’s data protection solution, with host agents for a wide variety of systems. Look for a solution that can do frequent snapshots, as this will determine the recovery point objective (RPO) that’s possible. Any solution can offer a 4-hour RPO, but 15-minutes is the new gold standard, so do your research.

Some critical business applications are starting to build in their own data protection capabilities, and using them might be the best approach. One example is Microsoft Exchange 2010, with its Database Availability Group (DAG) feature. It lets multiple Exchange mailbox servers be configured in a group, with each having both active and passive mailbox databases. DAGs can be configured by some RaaS providers with passive databases hosted by the provider while the active copies are on the customer’s premises. It’s a clean approach that leverages what Microsoft has built, but not all RaaS providers can accommodate it.

The bottom line is that you have choices for how to go about protecting your data center. The best choice of a provider will be the one who can provide a broad array of data protection technologies and accommodate heterogeneous environments.

Testing

We come to testing last, but it is the most important issue here. For some companies, it is their DR “Waterloo”, where they lose the war against business disruption. In the past, DR solutions were separate systems, disconnected from the production environment. Managing “change skew” – the continual stream of software updates, hardware upgrades and OS patches, become duplicate work and was often neglected. Without updates, tests become worthless and were discontinued. In the event of a disaster, the recovery site was essentially useless.

Snapshotting/replication technologies have solved the first issue, keeping up with change skew. By automatically and continuously doing block-level replication of protected servers, all system changes are captured automatically. Install a patch, and minutes later, those changes are captured as a delta snapshot and replicated to the RaaS site.

This makes testing the DR plan and the RaaS solution practical, and test you must! The reality of the typical mid-size or larger company is that there are many critical applications, some of which have multiple components running on separate servers (eg: an e-commerce site with separate web, app and database servers). Failing over to the DR site in a disaster isn’t a simple matter of starting everything up all at once. Having a comprehensive and up-to-date run book is key, and so is exercising it periodically. Starting servers in the right order, making sure that all core services like DNS and DHCP work, that IP address assignment happened properly – all of this needs to be de-bugged before the next disaster hits.

Conclusion

Recovery-as-a-Service is changing the business continuity and disaster recovery market. RaaS makes real DR available and affordable to mid-size companies that couldn’t afford it and didn’t have the staff to implement it in the past. There are now multiple RaaS providers with widely different solutions, so customers have to do their homework. It’s not enough to know what RPO and RTO the provider offers, or where their data centers are located. You need to understand some of the core technologies they use and set of services they offer to make their offering a complete, turnkey DR solution.

Prioritize Servers and Applications

The same mentality that leads companies to use backup and forego DR also negatively affects the way they think about prioritizing their protection requirements. Thinking about recovery in terms of which files to protect is the wrong way to go; you end up lost in the weeds and not looking at things systematically.

With RaaS, you’re protecting entire servers, so the focus should be on which servers are critical to the company’s survival, which are important but not critical, and which ones don’t require a DR plan. If you are a law firm, you’ll probably put your email and document management system in Tier 1; if you are an online retailer, your e-commerce site (web server, app server, payments) should be Tier 1. In any case, the point is to match both your Recovery Point Objective and Recovery Time Objective to the importance of the server.

The reason for prioritizing may itself be phasing out. In traditional DR solutions, companies had to rent co-lo space, buy hardware, bandwidth, etc., and then watch it sit idle until there was a disaster. Companies became cost conscious, and started prioritizing their business applications, protecting just the most critical ones, in order to save money. But RaaS changes the economics of DR solutions. By taking advantage of the elasticity provided by virtualization, and the economies of scale that public cloud based over-provisioning allow, RaaS solutions often cost just a fraction of traditional methods. This has allowed organizations to include more of their applications in their DR plans, making their businesses safer.

Dealing with a Heterogeneous Environment

Protecting your servers would be easy if they were all the same. A lot of customers and vendors talk about DR as if all their servers were Windows OS running on VMware on an x86 box. And indeed, a great number of business applications today run this way. RaaS solutions that only work on these kinds of servers seem well suited to the very few companies that have nothing but this type of environment.

But in practice, companies don’t have such homogeneous data centers. Their Oracle database runs on Windows, but their Apache web server runs on Linux. The Active Directory server runs virtually, but the ERP system runs on a physical server, and it’s Solaris. The home office is standardized, but a couple of the remote offices have their own non-standard servers. Et cetera. The point is that when making a DR plan and selecting a RaaS provider, you need to include all the critical business applications, regardless of platform.

I’ll use a few of my upcoming posts to serialize the paper…

Five Things You need to Know to Get Raas Right – Part I

Recovery-as-a-Service (RaaS) has emerged in the last three years as a viable and attractive alternative to traditional methods. Recovery-as-a-Service, also called “recovery in the cloud”, and “cloud based recovery”, takes advantage of the resource elasticity inherent in cloud computing to lower the cost of establishing and maintaining a target environment for business continuity and disaster recovery.

Most IT departments are now familiar with RaaS and the basic questions they should ask providers. They know to look at locations and quantity of data centers the provider has, and if the data centers are SAS-70 certified. They know to look at RTO and RPO capabilities. And they know to compare the often complex pricing, based on number of servers, amount of storage consumed, etc.

But there are more nuanced issues that customers need to understand to determine if RaaS is right for them, and to select the best provider for their needs.

Backup Is Not Disaster Recovery

Online, or “cloud”, backup services are popular with consumers as a way to protect personal data. Unfortunately, companies are starting to use them to protect their business continuity. While their low prices are appealing, trusting your company’s ability to stay open and survive a disaster to these kinds of services is misguided. The contract and service level agreement available with these solutions are often so minimal as to be pointless. So called “best effort” services – the provider will make its best effort to keep its service running at all times – just aren’t appropriate for business use. The cloud backup solutions also create concerns around data security. But most importantly, they are only backup solutions, and backup isn’t disaster recovery.

Backup solutions, whether new breed cloud backup, or traditional tape- and disk-based, on-premises solutions, are inherently limited in the level of recovery options they provide. Backup solutions let you find the file, or disk image, you want, mount it, and copy data back to a server. If you had a server hardware failure, you first have to acquire new hardware, and rebuild it from the operating system on up. This can take hours or days, far too long for a business to be without its critical applications.

On the other hand, disaster recovery solutions provide a super-set of backup’s functionality. In addition to allowing for data recovery, DR also lets you recover a whole server (physical or virtual), with OS, application, settings and data, in an environment where end-users can connect to the server and keep working. RaaS technologies allow this to happen in 15 minutes, sometimes even less. If the disaster affects your entire data center, a DR solution will allow you to failover and keep running all of the protected servers.

Next time: Prioritization

As our CEO, Mark Hadfield put it,

“For nScaled to deliver enterprise-class cloud infrastructure to our risk-averse customers, it’s vital that we build our Remote Cloud data centers in the best available co-location facilities. In addition to providing world-class IBX facilities, Equinix also has the global reach we need as we expand our business worldwide.”

The network density, standardization and consistency of services provided by Equinix allow nScaled to maintain an extremely high level of service to its customers.

Equinix is a great partner for nScaled, providing us with first class data centers all over the world. We’re looking forward to a great relationship with them.

Shein does a great job of reviewing the pressing need that all businesses have for DR, and cites some scary statistics about the paltry uptake of DR among SMBs.

The most important part of her story covers the need for companies to have a DR plan. This is great advice, and comes before selecting a technology provider to help implement that plan. We couldn’t agree more.

Shein also repeats our mantra, “Test, test test.” It’s good advice, and bears repeating because DR testing is the kind of eat-your-spinach advice that is easy to ignore. But if you don’t test your DR plan, then you don’t have a DR plan.

I called this post “Disaster Recovery 201” because I’d like to add a few things to Shein’s piece that are a bit more advanced.

if you don’t test your DR plan, then you don’t have a DR plan

First, I want to point out that using a cloud storage or cloud backup solution is NOT disaster recovery. There are many scenarios where all you will need to do is recover some data, but true DR means being able to spin up servers that end-users can connect to, to keep working, and the cloud backups stuff can’t do that.

Second, Shein quotes someone in the story as advising triaging what gets backed up by file type. Again, this advice, while well meant (prioritizing what to protect is important), doing so at the file type level is the wrong level of granularity.

What corporate IT needs to is classify their servers as tier 1, 2 or 3, based on the impact to the company if a server becomes unavailable to the business and its employees. Email is tier 1 for almost all businesses. ERP is for manufacturing firms; document management for law firms; web servers for e-commerce, etc.

Once you determine the tiers of servers, and protect them accordingly (RPO and RTO that meet the needs of your business in a disaster), then you implement a data protection / snapshotting technology that captures everything on the server – the OS, app, settings and data – and protects it. Fool around with file types and you will be lost in the weeds in no time.

With any insurance policy, the subscriber tries to calculate what a fair premium to pay is. The calculation is based on the likelihood of making a claim, the size of a claim, and the premium payments. As long as the sum of all premium payments is less than the expected value of the policy (likelihood x amount), then the insurance is worth its price.

You’re dealing with imperfect information, of course, because you can’t predict the future, and don’t know what claims you might make. The best you can do is look to case studies for guidance. And maybe learn new ways to estimate insurance’s value to you.

So here’s a case study about the value of DR:

If the company had been shut down for those three days, they would have lost $900,000 in revenue.

nScaled has a customer on the US East Coast that was hit by Hurricane Irene this past August. This customer has an annual subscription with nScaled’s disaster recovery services that costs about $50,000 per year to protect all of their tier 1 and tier 2 servers.

Irene caused the company to be without power at their main data center for three days. When Irene hit, the company was able to failover to the nScaled Remote Cloud in about three hours total for all servers to be activated. During the three days of power outage, the firm was up and running, and they lost no work productivity as a result. They continued to serve their customers and earn money.

If the company had been shut down for those three days, they would have lost $900,000 in revenue.

…an ROI of 1,700%

The company had been using nScaled for about one year, so they paid $50,000 and received $900,000 in exchange, for an ROI of 1,700%.

($900,000 – $50,000) / $50,000 = 1,700%

That’s a hell of an investment!

But, you say, the company experienced a disaster just one year after subscribing to nScaled’s services. The high ROI is attributable to good timing. This is true. (Although trying to guess when to start paying for DR so you don’t pay more than you need to is a sucker’s game.)

This is where you have to understand the likelihood – or frequency – of making a claim. Of course, Irene was a once-every-25-years storm. But when you add up all the storms, blackouts, equipment failures, human errors, and hacker attacks that an IT environment is subject to, the frequency of a “disaster” goes up considerably. Many of our customers expect that they will need to failover part or all of their environment once a year.

Suppose we assume that this same customer doesn’t need to failover to nScaled’s data centers for another ten years, and when that happens, it would involve a similar avoided loss of $900,000. In that case the ROI of the nScaled solution would be 260%, still a spectacular investment by any standard.

($1,800,000 – $500,000) / 500,000 = 260%

So, if your company is without a DR plan because someone at the top of your organization thinks it’s too expensive, or doesn’t like to pay insurance premiums, present it to them as the smartest investment the company can make. What other project in your company (not just in IT, but the entire company) can boast an ROI of 260% or more?

In this era of savings accounts that pay 0.25%, and long-term bonds that pay 7% (if you’re willing to lend money to Italy), investing in your business’ continuity, and worker productivity, makes a hell of a lot of financial sense.