"Never waste a crisis," says Adobe's product security chief Brad Arkin. He says that with a smile today, but when he joined the company in late 2008 the crisis was very real and the outcome unknowable.
Through the first decade of the 21st century, Microsoft had steadily improved the security of Windows and Office — the direct result of Bill Gates' trustworthy computing memo and a new software development process called the Secure Development Lifecycle (SDL). So the bad guys just moved up the stack.
Adobe had the "good fortune", as Arkin wryly puts it, to have two of the most widely-deployed non-Microsoft applications: Reader and Acrobat. Both had an old, complicated code base. Both had serious security flaws.
Adobe became the new target.
It was Arkin's job to deal with that.
"PDFs were really attractive because [attackers] could email a link or IM a link, and then you get the web-hosted attack vector, and then you have the email attachment attack vector. These things made it really perfect for attacking corporate types," Arkin told CSO Online.
These techniques would eventually be used in the high-profile Operation Aurora attacks against Google, Juniper Networks and even Adobe itself.
"We were experiencing a lot of pain, and we had a plan, but at the time we didn't know if it was going to work. It was merely the best we could come up with. And that was a really scary thing because there were moments where it felt like the challenges we were up against were too great and the speed at which bad guys were changing tactics was always going to stay a step ahead of us," Arkin said.
"In hindsight it's easy to just pay attention to the choices we made that were effective, and then we tend to forget about all the dumb ideas that we wasted time on."
One of those dumb ideas was trying to fix all the bugs.
"If we can do that faster than the bad guys are finding them and exploiting them then we'll be all set," Arkin said.
"It just doesn't work."
Chasing all the bugs is simply trench warfare. For all that time spent on industrial-scale automated bug-finding tools called fuzzers, for every bug fixed there's still another latent bug waiting to be found.
Arkin blames his own consulting background for that mistake. It's the perfect job for a consultant because it never ends. There's always more bugs to find.
"The definition of a crisis isn't people running around screaming, but it's a turning point where things are decided for better or for worse," Arkin said. "So you leverage that moment of crisis and you say, 'Well, this is our chance to make sure things are better going forward.'"
What made things better was forgetting about all the bugs and concentrating on driving up the cost for attackers to exploit those bugs to the point where it wasn't worth their while.
That in turn meant making better use of the available programming tools and developing the sandbox that was first introduced in Reader and Acrobat version X in November 2010.
"The bad guys have a certain MO where they want to get on the machine and install a trojan of some sort and then move laterally, and version X made it really hard to install software," Arkin said. "Other types of things that might still be straightforward are not interesting and so they just moved on."
The improvements in version X are holding up, said Arkin.
"I was hoping we could make it the first 24 hours without anyone pointing out an obvious goof or like, 'Oh, what were they thinking?' And then we made it through the first day and then the first week, the first month. I said, 'OK, this is pretty good.' Nothing glaring was left open."
"We've been very happy, and also a little surprised, that there hasn't been any bad-guy activity against version X. We made it as hard as we could to attack, but... there's certain things our design didn't seek to prevent... To date we still haven't seen any malware that is affective against X, and not just installing trojans but anything at all."
Adobe's problem now is getting everyone who's still running old versions of the software to upgrade.
"We've pretty much scooped up just about everyone who is awake and paying attention, and now we're stuck on this machine replacement rate," Arkin said. "We're not trying to be sneaky, but we're trying to get the people who aren't paying attention."
That's a legal problem. Adobe can't just upgrade people to the new version automatically because in many jurisdictions that would be a cybercrime.
"We're not making much progress there," Arkin said.
Nevertheless, the problems with Reader and Acrobat are slowly going away. Adobe's next challenge will be bringing the same security improvements to Flash Player.
That's more complex. Flash Player has to deal with many different web browser environments as well as different operating systems. Adobe's experience fixing Reader and Acrobat should make things easier — as will the newer Pepper plug-in architecture that replaces the old Netscape Plugin API standard.
"It's fun now, because I can give this advice to other people, and help them avoid my mistakes," Arkin said.
And then comes Adobe's move into the cloud.
"You've got all these other layers you've got to worry about. There's all sorts of new ways to screw up."
Contact Stilgherrian at Stil@stilgherrian.com or follow him on Twitter at @stilgherrian
______________________________________________________________________________________
CSO Announcement
Register Today. Hear from Rob Livingstone, Michael Barnes, Steve Quane and Dave Asprey amongst others on the Evolution, Trends, Solutions and the Future of Cloud Security, limited seats register today through CSO.
______________________________________________________________________________________