3PV – Third Party Verification (Booth 316) is showcasing disaster-recovery planning at the CompTel/ASCENT Fall show by using real-life experience gleaned from surviving Hurricane Charley, which ripped through southern and central Florida in August, causing damage with an estimated cost of at least $15 billion.
A Category 4 hurricane packs winds in excess of 131 miles an hour. Fortunately, storms of this intensity are rare in central Florida the last storm of a similar magnitude was Hurricane Donna in 1960.
Charley suddenly veered and grew in intensity during the early morning hours of Friday, Aug. 13, putting central Florida directly in its path. Executives at 3PV, based just north of Orlando in Altamonte Springs, knew one thing was clear: Their disaster-recovery planning was about to get a major workout.
According to 3PVs CEO David Brinkman, the response to Charley began long before the storm arrived when the company selected its facility in 1999. We are a company that takes enormous pride in delivering outstanding service levels and excellent customer care to every client, he says. We do not accept downtime, and we incorporated that philosophy into our disaster-recovery plan.
Step one was finding the right facility. Because 3PV specializes in speech-enabled third party verification, maintaining system uptime was the top requirement. 3PV had a lengthy wish list for its new headquarters: A facility with a full building power generator so it could operate for prolonged periods even if utilities were down; in-ground phone lines; fiber through SONET rings to provide system redundancy; and the ability to deploy multiple carriers, ensuring failover. It was a tall order, but we found everything we were looking for in our current facility at East Central Parkway in Altamonte Springs, says Brinkman.
He knew that decision was about to be tested when the forecast for Charley shifted from bad to worse. 3PV began the process of battening down the hatches on Aug. 12, when all of Florida was declared under a state of emergency. Team members at 3PV reviewed their disaster-preparedness plans and discussed how they would work together to ensure system uptime during the worst of the storm.
We knew that we had already fought and won half the battle because of our location, says Brinkman. I had total confidence in our technology and the facility. I also knew we have talented, dedicated and expert staff, who were ready to do whatever it took to keep the system up and running. When I heard the forecast and the broadcasts out of Port Charlotte, I knew we had done everything we could. I only hoped it was enough.
Essential staff stayed until 2 p.m. Friday, securing the facility. Charley hit hard in the early evening hours. The 3PV facility was right on Charleys path the eye of the storm passed within 10 miles of the building, bringing heavy rain and sustained wind exceeding 100 miles per hour.
Brinkman inspected the building on Saturday and found hundreds of downed trees, an extensive power failure and the 3PV office up and running, complete with air conditioning. All in all, 3PV performed flawlessly, he says. We did lose one of our Internet providers, but because of automatic failover to another provider, we didnt lose a single call. The building infrastructure, the disaster-recovery planning and the team effort allowed business as usual during the height of the hurricane.
Many 3PV employees did not fare as well at home. Most were without power, water and sewer services for an extended period of time. One employee lost her home and her car when her apartment building collapsed. 3PV was operating with a reduced staff for a few days during the clean-up.
Brinkman identifies some room for improvement. The only real operating difficulty we experienced was with our employees, he says. Many employees, including some of our live agents, were not able to arrive at the office due to impassable roads, downed trees, and other effects of the storm aftermath. The only thing we would have done differently in retrospect is to transfer the entire load from our live agents to partners in the days immediately after the storm.
Brinkman adds, Hopefully, it will be another 40 years before we get to test out our revised disaster-recovery plan.