It has been a turbulent week on the East Coast.  We have had a rare 5.9 earthquake and have been hit by a Category 1 hurricane that have left millions without power and has caused major flooding. So naturally I have been thinking about Disaster Recovery.  It really takes extreme cases like the past week to get people thinking about disasters and Disaster Recovery.  But the truth is that disasters happen every day.  There are fires and floods and explosions that impact businesses every day.

But being that large powerful eye opening events really get people thinking about disasters, I will seize the moment and use it to help get people start thinking about Disaster Recovery planning.  Disaster Recovery planning is not easy. The exercise is trying to plan for something you don’t know exists and can’t anticipate environmental, physical and human actions. But before you get discouraged, you can start planning for things that you think might happen even if you don’t know the exact chain of events.

Categories of disasters

When I look at Disaster Recovery planning I like to split the disaster into 1 of 2 categories.  The first category is a temporary disruption in a business’ ability to access their server/network infrastructure.  This could be the result of an extended power outage that shuts the servers down. Or may be the result of a flood that makes travel to the office for employees impossible but also disrupts the network communication and remote access such as a failed T1, DSL or cable modem.  Both of these scenarios leave a business and employees temporarily without access to the network, data and applications.  The second category is more serious and involves destruction of a business’ server/network infrastructure.  This could be the result of a fire, flood, explosion, earthquake, etc.  The business’ servers and network are permanently destroyed.

You will notice that splitting the disasters into 2 categories allow for planning of multiple scenarios but without having to know the exact cause of the disaster.  It makes the Disaster Recovery planning much easier.

Data replication

One of the key parts of ensuring that you have a Disaster Recovery plan is to figure out how you are going to access critical data in the event that your servers/network are either temporarily or permanently inaccessible.  In this post I go into detail on Disaster Recovery planning which includes data replication and utilizing alternate locations to run duplicate infrastructure. The details of the post will give you good insight into some of the alternatives.

Communications

But another key part of Disaster Recovery planning is much less high tech.  In fact it is probably very low tech and almost as important.  In a disaster one of the worst outcomes is that a business’ employees may not have the ability to communicate with each other.  For example if there is a widespread power outage and your business primarily relies on email to communicate, your email server may be down and this will not be an option.  Secondly as more and more people move away from landline phones (Verizon, AT&T, etc.) to voice over IP (VoIP) such as Vonage and phone service through Cable companies, FIOS, etc, power outages cause people to lose their home phone access. When the power is out, Internet and phone are also out.  The third point is that as we rely on cell phones more and more for communications we are very susceptible to a disruption in cell service.  After the recent earthquake, millions rushed to their cell phones to makes calls only to find that calls would not go through. Unfortunately the reality is our cell phone infrastructure has major problems with extremely high volumes of calls and in disasters that is exactly the amount of volume to expect.  So a business might face the scenarios where email is down and employees can’t be reached via home and cell phones. The issue is critical if you cannot communicate with employees.

Let’s take a low tech approach to communications and see if some basic planning can help.  Prior to the recent hurricane, Entegration did some basic planning to ensure that all employees could communicate in the event of a disaster.  Here are some of the steps we took:

1. Ensure that we had an up to date contact list with all home phone numbers, cell phone numbers and home addresses (yes driving to a person’s house is a viable option if there is no other way to communicate with them).
2. Every employee setup an alternate email address (via Gmail, Hotmail, Yahoo mail, etc.).  We set up the address as First Name Last Name Company Name .  For example ArtGEntegration@hotmail.com.  In the event our primary email server went down and we could not communicate via Exchange/Outlook or our smartphones, we could still communicate via alternate email providers.  These email services are free and very easy to setup. And with smartphones, tablets and wireless network ability, access to these services are very straightforward and easy even in the event of a power outage.  We ensured that our contact list as mentioned in bullet 1 had both the primary and secondary email address for each employee.

Social Networks

Other alternatives are to utilize social networks such as Facebook, Twitter, LinkedIn and Google+ to communicate.  Adding social networks to the above options increase your chances of being able to communicate.

Summary

So hopefully this will get you thinking about Disaster Recovery planning.  In summary:

1. Break disasters into categories (temporary and permanent disruptions of service).
2. Focus on communication strategies that will enable all employees to communicate in the event of a disaster.
3. Plan data replication and alternate locations to run critical business functions.

Image via Flickr posted by www.gisuser.com

Joplin, MO was hit by a massive tornado on Sunday evening that did extensive damage to the St. John’s Regional Medical Center hospital. There are reports that x-rays from the hospital have been found in driveways 70 miles east of the hospital.

On Twitter Steven Waldren sheds some very interesting and insightful perspectives:

Steven’s quotes gets to the bottom of Disaster Recovery.  When an actual disaster hits and your servers are destroyed how do you get to your data? Having tape backups or offsite backups are fine but if your servers are gone where do you restore the data?

Disaster Recovery (DR) planning is more than ensuring you have a backup of your data. It is about ensuring that your organization can still function and get to critical systems even when your primary systems have been destroyed. With cloud-based Disaster Recovery solutions the cost of implementing DR has been significantly lowered. All healthcare organizations should be looking into some sort of DR that will not only ensure that data is properly backed up but will allow for access to critical data in the event of a real disaster.

Contingency planning and DR planning are required under the HIPAA Security Rule:

STANDARD § 164.308(a)(7)Contingency Plan

The purpose of contingency planning is to establish strategies for recovering access to EPHI should the organization experience an emergency or other occurrence, such as a power outage and/or disruption of critical business operations. The goal is to ensure that organizations have their EPHI available when it is needed. The Contingency Plan standard requires that covered entities:

“Establish (and implement as needed) policies and procedures for responding to an emergency or other occurrence (for example, fire, vandalism, system failure, and natural disaster) that damages systems that contain electronic protected health information.”

DISASTER RECOVERY PLAN (R) – § 164.308(a)(7)(ii)(B)

The Disaster Recovery Plan implementation specification requires covered entities to:

“Establish (and implement as needed) procedures to restore any loss of data.” Some covered entities may already have a general disaster plan that meets this requirement; however, each entity must review the current plan to ensure that it allows them to recover EPHI

A final takeaway is that the time to think about Disaster Recovery is before a disaster hits. Implementing DR is not only required under HIPAA but is critical to any business to ensure that the organization can continue to operate even when primary systems are destroyed.

You have just implementing a new electronic health records (EHR) system, congratulations!  You probably spent anywhere from $75,000 – $500,000+ on hardware, software, licenses,and implementation labor.
Hopefully you qualify for EHR meaningful use incentive funds to offset some of those expenses. While you are looking to stop spending money and to start recouping some of the expense, I am going to tell you 3 additional products and services that you must consider.

The 3 products and services are:

1. Offsite data backup
2. HIPAA Security
3. Disaster Recovery

I realize those 3 items are not sexy and will not help increase your revenue. I think that is one of the reasons that many medical practices don’t sign up for these services. The 3 services are about protecting your EHR, your data, your patient’s information and protecting your practice.

Offsite Data Backup

“Why do I need offsite data backup when we are backing up to a tape drive?”

I can’t tell you how many times I have had this conversation. Backing up your data nightly to a tape drive is a good practice but unfortunately backup tapes are not completely reliable. Every time we have to restore a file, database or other data from a backup tape, I hold my breath and pray that the data is on the tape and we can retrieve it successfully.

If you are backing up to tape the responsibility to switch tapes on a daily basis is usually assigned to an individual in the practice. From experience we have seen that people forget to switch tapes (trust me this happens more then you can imagine). In addition, tapes are used over and over and eventually they lose their ability to successfully read and write data. Hence the praying comment that when we need the data, the tape will not be at the point where we can not successfully retrieve the data.

Offsite data backup is a very straightforward process and very similar to backing data up to tape. On a nightly basis the data is backed up but instead of being backed up to tape it is backed up to a server in a vendor’s data center. Here is how it works.

1. On the system that you are backing up, there is a backup agent (software program) that starts to backup the data.
2. The backup agent makes a secure encrypted connection via the Internet to a server(s) at a vendor’s data center.
3. The data is copied to the servers and is stored on the vendor’s server is a secure encrypted format.

As you can see it is critical to have an Internet connection in order to perform the offsite backup.  The offsite data backup is scheduled and runs automatically so there is no human intervention required. This eliminates the issue with someone forgetting to change the backup tape.

My recommendation to most practices is to use offsite data backup as a supplemental service in addition to doing nightly tape backups.  If you do both then you have your data in 2 different places and you increase your chances that the data will be available if and when you need it.

On an average, offsite data backup costs around $2/GB.  So if you are backing up your EHR and you have 20GB of data it will cost you around $40/mo. I think that is a very reasonable amount to help ensure that your data is protected. To help convince you that offsite backup is worth the additional expense let’s look at a scenario that I have seen happen multiple times.

There is a really bad storm with heavy rain and lightning. The storm knocks out power to your office and although your EHR server is on a uninterrupted power supply (UPS) the server does not shut down cleanly (immediately loses power) and in the process it corrupts the EHR database. When power is eventually restored and the server comes back online the EHR program generates errors stating that it can not read the EHR database (it is corrupt). Imagine that you have been using the EHR for 1 month and every patient that you have seen is in your EHR (go ahead and imagine you have been using it for over a year and the amount of records would be even scarier). Your IT company comes in to help restore the EHR database from tape and get you back up and running.  When the IT company inserts the backup tape they can not locate the EHR database.   It turns out that the person who was responsible for changing the tape forgot to do it the last 2 evenings. They are able to restore the database from 2 days ago but all the data that was entered for the past 2 days is lost.  Think about having to recreate that data. You are using an EHR so do you have notes on each patient? Probably not. The amount of time and effort you and your staff will have to use to recover from the lost data makes the $40 look cheap.

HIPAA Security

The second service I urge you to consider is HIPAA Security. You are using an EHR and all of your patient information moving forward will be electronically stored. You may also have interfaces with vendors for electronic lab results, digital x-rays, ultrasounds, etc. For each patient there is a lot of electronic information that has to be protected.

Most EHR vendors do not address HIPAA security when they are training employees on the new EHR. If they do it is not in depth and there is a good chance that your employees will not understand what is required by HIPAA to protect patient information.

HIPAA security is about protecting patient data in electronic format. I am recommending you sign up for a HIPAA security service not only to comply with the HIPAA regulations but to ensure that your entire staff is educated on what exactly is required to protect patient data and to understand the best practices for protecting data. More importantly HIPAA security is a defensive measure to help protect your patients and your practice against a data breach. A lost laptop or USB drive with patient information could have serious financial impact on an organization.  Imagine a data breach that costs your practice $1,500,000. If you think that number too high consider the regulatory fines, patient breach notification expenses, lost revenue from patients leaving the practice, IT related expenses to re-mediate the breach, etc.  Even if the expense is half of that at $750,000 it can have a significant impact to an organization. And if you are thinking that your general liability insurance policy will cover most of those expenses you should check your coverage. Most policies do not cover HIPAA related expenses (although there are supplemental insurance policies that do cover HIPAA and cyber expenses).

There are many HIPAA security services on the market but on a whole you should look to accomplish the following:

1. Implement policies and procedures to ensure that patient information is properly protected
2. Perform a risk assessment to understand where you are at risk in protecting patient information and what additional security measures you should implement to better protect the information.
3. Train your entire staff on exactly what is HIPAA security, what they should be doing to protect patient data and what they should not be doing that could put patient data at risk.

HIPAA security will range in costs but for some real numbers this service will cost $1,750 to provide the 3 items above. (Full disclosure, HIPAA Secure Now! is a service of Entegration, Inc.).

As with the justification for offsite data backup, spending $1,750 to help protected you from fines and expenses that could be up to 100 times more expensive seems like a good investment.

Disaster Recovery

The third and final service I will urge you to consider is disaster recovery for your EHR and network.

I will start off by acknowledging that the odds of a disaster are slim but yet we have seen the affects of earthquakes and tornadoes in the past few months. And disasters are not only confined to natural disasters.  Fires and floods occur all the time.  Broken water pipes and sprinkler systems can destroy servers and computing equipment.

What exactly is disaster recovery?  Simply stated it is the ability to continue to utilize your applications in the event that your primary servers, network and applications are either destroyed or made unavailable by some event. Disaster recovery is ensuring that you can run your EHR on another server and access that server in the event of a disaster.

I wrote a detailed blog article on cheap disaster recovery which you should read.  But from a high level view, disaster recovery is:

1. Ensuring that you have another server(s) in another physical location that you can use in the event your primary server is unavailable
2. Data needs to be copied and kept up to date on the server(s) that you will use for disaster recovery
3. A method of accessing the disaster recovery server must be established
4. A detailed procedure must be in place that defines exactly what is needed to utilize the disaster recovery server(s) and what your employees need to do to operate in disaster recovery mode.

If you go back to the blog article that I wrote on cloud based disaster recovery the prices range from around $100/month/server.  So if you need to ensure that have your EHR server and your Domain Controller available in the event of a disaster then it will cost you around $200/mo.

Again let me define a scenario that helps justify the expense.

Let’s assume a water pipe bursts in the office above you and overnight hundreds of gallons of water leak onto your servers, destroying them.  Everything else in your office is wet but usable. After a couple of days of clean up you are ready to see patients but you no longer have functional servers and no functional EHR. You can order new servers from Dell or HP but even with overnight shipping there is a chance you will not receive them for 10-14 days.  Can you go without your EHR for that long? With the cloud based disaster recovery you can be up and running in as little as 4 hours. You can even access the EHR if you need to see patients in another practices’ office while you repair your office. Again I argue that $200/month is worth the expense to provide the safety net and flexibility to recover in the event of a disaster.

Summary

The 3 services that I described will protect your medical practice. Each of the services can be considered a safety net and operational insurance to protect you and to avoid events that can have significant financial impact to your organization. Take a step back and think of how much money you just spent on your EHR. The services that I recommend will cost you under $5,000 the first year (and half of that moving forward) and will help protect your investment in your EHR.

I would love to hear your thoughts and help with any questions you may have. Use the comments section below to give feedback.

Disaster Recovery is usually reserved for enterprise or large businesses.  Disaster Recovery (DR) has historically been too complex and too expensive for small to mid size businesses and medical practices.  In the past if you had multiple servers for different functions like database, email, file services, etc. you would need physical servers at another (backup)  site that duplicated those functions.  In addition, you would need to replicate the data between the primary servers and the DR servers at the backup site.  Replicating data meant that you needed complicated software and high speed communication links between the two sites.  So between the duplicate physical servers and data replication requirements you can see how the expense and complexity can be a deterrent to small to midsize businesses.

Virtual Servers

Over the past few years some of the costs have been lowered through the use of Virtual Servers.  Virtual Server technology that is provided by VMware, Microsoft and Citrix allow a physical server to run multiple Virtual Servers.  So one physical server can be configured with enough memory, processors and hard drives to run multiple servers.  For example, one physical server can run a database, email and file server.  In addition, the cost of hardware including servers have dropped which further contribute to reduced costs.  Virtual Servers are perfect technology for a DR infrastructure.

Double-Take and XOsoft Replication Software

In addition to Virtual Servers and falling hardware costs, modern replication technology from Double-Take and CA XOsoft has greatly reduced the complexity of replicating information from primary servers to DR servers.  The replication software has simplified setting up DR environments but add significant costs. The replication software is licensed per server and can run between $2,500 and $3,500 per server.  Taking the example of 3 physical primary servers and 1 backup server running Virtual Servers the replication software can cost around $12,000-$15,000.  Add in hardware and other licensing costs and you are looking at a $20,000 or more in DR implementation costs.  At that price point it is still elusive to many small to mid-size businesses.

Utilize  Cloud Computing

The next step to making DR more affordable to all companies is it to take advantage of Cloud computing. Amazon’s EC2 infrastructure let companies run Virtual Servers in the Amazon cloud utilizing the massive amount of computing infrastructure that they have built.  Companies can run Virtual Servers for just pennies an hour.  This Virtual Server environment in the Amazon EC2 cloud is a perfect fit for DR.  It enables companies to setup their DR infrastructure without any hardware expenses.

DR for everyone

Double-Take software has partnered with Amazon to develop a product / service called Double-Take Cloud. This service allows companies to rent the Double-Take software on a monthly basis for under $100 per server. Factor in the computing costs for utilizing the Amazon infrastructure and a company’s DR costs for the 3 physical server DR environment can be setup for around $350/month.  This monthly expenses requires no upfront hardware or software expenses.  At this price point Disaster Recovery is now affordable to all companies.

If you don’t have a Disaster Recovery plan in place for your business or medical practice, you should seriously consider looking into it.