Hurricanes, floods, fiber cuts keep IT pros on their toes

Many companies are prepared for a disaster, with plans and technologies in place to help sustain operations in the wake of severe weather, hardware failures, human error, and accidents. But not every company is covered. In a survey conducted by Network World and IT management software maker SolarWinds, 25% of respondents admitted their companies don't have a disaster preparedness and response plan in place. Among those without a plan, 57% said they'll create one in the next 12 months.

PAYDAY: Who are tech's highest paid CEOs?

There's good reason to make that happen: Among the 230 IT pros who responded to our survey, 27% said they've been unable to go into the office because of a disaster of some kind. Of those, 34% missed a week or more of work because of the incident.

"During a severe storm an electrical surge caused our UPS to shut down, taking down our data center," recalled one survey participant. "After this we added a whole building surge suppression system."

Another company lost connectivity in its office for a week after a backhoe cut through the building's phone lines and fiber. The company quickly moved its servers and set up shop in an employee's home. "We are a software developer, [so we] were able to bring up our web site at an employee's house within half a day. Clients didn't know we had a problem," the IT pro said, but it was a "big headache for the network guys, cleaning, buying, rebuilding."

Another respondent cited multiple outages at his company. In one instance, a server room was exposed to smoke from a fire and "all the equipment had to be removed and cleaned, stopping the business for a week." Another time, a flood crippled the business -- 1,500 people across 50 sites -- for nearly three days because voice, WAN and Internet were all routed through a single fiber connection to the carrier's exchange, which was powered down due to flooding.

Despite the horror stories, it can be difficult for IT pros to convince senior management of the risks of insufficient business continuity planning. When asked about the attitude of senior non-IT management toward disaster preparedness and response investments, just 31% of survey respondents said it's a top or high priority. Another 29% said it's a medium priority, and 40% said it's a low priority or management is unconcerned.

Sometimes, it takes a disaster to get the message across.

In one instance, corporate finance nixed IT's plans to deploy a backup AC system during data center construction. When temperatures in the data center soared past 90 degrees, it resulted in a shutdown of nearly all server resources. The result? A "new, dedicated backup AC installed [and] connected to the building's backup generator," the respondent said.

"Flooding can be an issue during hurricanes, nor'easters and other similar events," said another IT pro whose company is located in a low-lying coastal area. "We justified our demands that the servers not get shoved in the basement at our main site when it was flooded out during [hurricane] Irene. Nothing learned the hard way, thankfully."

Disaster readiness

Disaster preparedness is a moving target; plans require testing, updates and modernizations on a regular basis. In our survey, 44% of respondents said they update their plans about once a year, and another 15% do updates every two years.

Some companies test plans even more frequently. "All foundation applications are housed off-site with campus network presence. Tapes are sent off daily for other systems and can be recovered within 12 to 18 hours," one respondent said. "We test this twice a year."

In general, respondents have deployed a wide range of technologies that play a role in their disaster preparedness plans, including VPNs, offsite backup/storage, remote access technologies, and company-supported mobile devices.

CEO FIRST JOBS: Licorice maker, housekeeper, scuba diver and more

In particular, virtualizing applications and putting apps in the cloud as a disaster preparedness measure are common tactics among respondents. Email/messaging infrastructure is most often virtualized or put in the cloud (cited by 58% of respondents), followed by web site/ecommerce systems (48%), database servers (46%), Web servers for critical online applications (44%), and CRM/ERP systems (32%). Twenty percent of respondents said none of their apps are virtualized or in the cloud.

On the data-protection front, nearly one-quarter of respondents (23%) said 100% of their company data is backed up at an offsite location at least 50 miles away from the main office. At the other extreme, 23% of respondents said none of their data is backed up offsite. The remainder fall somewhere in between, with the majority backing up at least 40% of their data offsite.

If a disaster were to strike, respondents have varying opinions about how well their organizations would fare. When asked if they're confident their organization could recover in a reasonable amount of time (12 to 18 hours) if a significant disaster were to make their main data center inaccessible, 45% said they're certain or very confident, 25% are somewhat confident, and 30% said they're not at all confident.

"I'm confident we could restore our data and servers. Not so sure about getting hardware shipped and loaded within 12 to 18 hours. I'd say at least a week," one respondent estimated.

Some are fully aware of the limitations in their current business continuity plans.

"It'd take 140+ hours to restore a full backup from tape," one respondent said.

"We have way too much data backed up on tape -- restoration would take days if not weeks," another echoed.

"The biggest problem will be DNS settings. While there are technical 'best practices,' we can't afford many of them, either because of license costs or recurring bandwidth charges," acknowledged another respondent.

Experience with data center disasters, however painful, can sometimes pay off in the future.

"Given that we have already seen one incident, the office has adopted cautious and diligent attention to prevention and management of similar incidents in the future," said one IT pro. "Back-up facilities have been upgraded and a new data center has been constructed."

Ann Bednarz covers IT careers, outsourcing and Internet culture for Network World. Follow Ann on Twitter at @annbednarz and check out her blog, Occupational Hazards. Her e-mail address is abednarz@nww.com.

Read more about infrastructure management in Network World's Infrastructure Management section.

Show Comments