IBM-owned cloud company SoftLayer has been caught on the back foot in response to the critical Xen bug published today, kicking off its maintenance 15 hours after the bug was made public.
For an IBM brand promising more than a “commodity” cloud, SoftLayer could have been quicker to kick off instance reboots across its global data centres to patch a security flaw in the Xen hypervisor that was made public on Wednesday.
Like SoftLayer, parts of Amazon Web Services’ and Rackspace' clouds rely on Xen. But unlike AWS and RackSpace, which told customers last week that they would reboot instances across the globe, SoftLayer waited until yesterday.
Amazon has since said that 10 percent of EC2 instances were affected while Rackspace today said 200,000-plus customers were impacted by the reboot.
Both companies staggered the reboots across availability zones, which took around 48 hour to complete. Amazon finished its last maintenance reboot on September 30.
Some Amazon Web Services customers were not pleased with 48 hours notice of a potentially disruptive maintenance and only a sparse explanation for why it was occurring. Despite those complaints, AWS importantly finished up ahead of midnight UTC Wednesday October 1 — the time and date that details of the Xen bug were scheduled to be published.
The Xen project on Wednesday revealed the bug affects a Xen virtualisation type known as “hardware-assisted virtual machine” (HVM), which maintains reserved hardware for each OS. It explained today, “A buggy or malicious HVM guest can crash the host or read data relating to other guests or the hypervisor itself.”
Only Xen 4.1 and upwards are vulnerable if they’re running on x86 systems, but not ARM systems. The other virtualisation type called Xen “para virtualisation” was not affected.
Rackspace's update has left some customers at its London and Sydney data centres bruised, but technically it too beat the embargo. SoftLayer on the other hand told customers it wouldn't commence its reboot earlier than 1500 UTC October 1.
Read more: Amazon AWS algorithms watch for cloud-based hacks
“The maintenance schedule will begin on October 1st and is being finalized at this time…. We will begin upgrading the hosts a data center at a time in the following order starting at approximately, and no sooner than, 1-OCT-2014 15:00 UTC,” the company said.
SoftLayer has several data centres in the US, as well as Singapore, Canada, the Netherlands, Hong Kong, London. More recently, it opened a data centre in Melbourne, although this data centre is not included in its global maintenance schedule.
CSO has asked IBM Australia if Australian customers there can expect to be affected by similar maintenance work.
In an apology to customers on Wednesday, Racksapce CEO and president Taylor Rhodes explained why it was so important to beat the embargo.
“We were faced with the difficult decision of whether to start our reboots over the weekend, with short notice to our customers, or postpone it until Monday. The latter course would not allow us to sufficiently stagger the reboots. It would jeopardize our ability to fully patch all the affected servers before the vulnerability became public, thus exposing our customers to heightened risk,” wrote Rhodes.
SoftLayer, like its rivals, faced a tight deadline to prepare patches ahead of the reboot, however it was likely aware of the issue before Tuesday.
On Wednesday AWS evangelist Jeff Bar explained the Xen community’s two-stage disclosure process. The Xen Project community provides early disclosure to a select list of providers, offering them a “limited time to make accommodations and apply updates before it becomes widely known” ahead of full disclosure on the public disclosure date.
And according to the Xen project, SoftLayer is on that early disclosure list.