Disaster recovery planning. Protecting patient care, no matter what.
If you live in Houston, you probably keep hurricane prep supplies within easy reach when a storm is brewing in the Gulf. But have you ever intentionally pulled the circuit breaker at home on a Saturday to practice in case a Category 3 knocks the power out? Probably not.
Well that’s exactly what IT does twice a year to ensure our ability to maintain patient care during a natural disaster or other unexpected outage. Since 2020, IT has been performing drills to make sure we are prepared in the event of a disaster.
“Extensive planning and training make it possible to continue patient care, no matter what,” said Jim Francis, vice president of Technology and Shared Services.
“First, do no harm.” This phrase forms part of the Hippocratic oath physicians take before ever beginning to treat patients. Similarly in IT, patient safety is our number one goal, and disaster recovery planning ensures our ability to maintain the critical systems that enable our physicians and clinicians to care for our patients.
Practice drills save patient lives.
Twice a year, IT teams gather remotely on a Saturday to practice recovering our systems. After intentionally bringing systems down, the teams immediately begin working to restore foundational hardware and apps. These include things like phones, network, email, Epic and MARS, to name just a few.
“Our disaster recovery planning is a structured approach,” Francis said. “We document our plan, and it includes all the detailed steps we take to restore our critical services, including applications, data, hardware and more. Practice drills help us make sure we have the right people, in the right place, at the right time.”
The numbers tell the story about the complexity involved in maintaining patient care during a disaster. “We run 300 to 400 systems throughout all of our hospitals and facilities, and we recover about 80 to 85 of those quickly following an unexpected disaster or outage,” Francis said.
Documented protocols and processes in our playbook ensure that everyone knows what to do in these cases. Additionally, during practice drills, the teams often find a better way to handle something, and these changes are incorporated into the planning for upcoming drills.
“Our goal is sustaining excellent patient care in any situation. We want to practice enough that the steps become like muscle memory for the team,” said Jay Kumar, IT director of data and analytics, who also leads the semi-annual disaster recovery drills.
Data centers preserve systems and information.
In addition to a data center in Houston, there is an additional data center in Austin. If the main data center is damaged or loses power, IT teams can switch to the Austin data center, where duplicates of our critical IT systems, applications and information are maintained.
“In a real disaster scenario, during the time it takes to transition to the Austin data center, we ask operations to shift to their planned downtime procedures,” Kumar said. “This enables them to use certain downtime computers to continue caring for our patients and recording patient information in the charts.”
Tracking every incident. Continuous improvement.
In addition to practice drills and a backup data center, disaster preparation includes closely examining all incidents — like if an application experiences a brief interruption or isn’t functioning properly — anytime these occur.
Our teams remain constantly alert to resolve any issues, as well as monitor and debrief on all incidents. “We examine every incident, large or small, on Fridays,” Francis said. “We identify the problem, find out what caused it and learn how to prevent it from happening again. Is it a software bug, a workflow issue or something else? This is what we try to determine.”
Applying what we learned from drills. Reaping an award.
During a major power outage last year, we put into practice the knowledge and skills gained from our disaster recovery drills. We restored critical IT systems to normal operations in a timely manner and even received global recognition for this accomplishment. The Disaster Recovery Institute International awarded us with its Response and Recovery of the Year Award for our quick recovery of critical systems.
Protecting our ability to care for our patients.
Providing uninterrupted patient care during natural disasters and other unanticipated events is at the heart of our preparation, which includes conducting drills twice a year. “Our focus is on restoring services while maintaining patient care,” Francis said. With extremely detailed protocols and practice drills, we continually improve the disaster recovery processes that ensure our ability to care for our patients.
We are vigilant, at all times, anticipating the worst to ensure that we’re fully prepared for what may come with fail-safe programs and processes in place — all with one goal, protecting our ability to care for our patients, no matter what.