Heartbleed: Lessons learnt from first contact

One thing I insist on in any security testing or activity is the need for continuous learning or understanding. When the Heartbleed vulnerability exploded, I had the opportunity to go through such a process.

April 9, 2014. It’s a balmy Wednesday morning in Sydney. I had started working for a new company the week before. That morning I walked in to an energetic boss who exclaimed ‘the internet is on fire!’ It was Heartbleed and developments overnight had been interesting. I had started following Heartbleed on Tuesday afternoon but had no idea that it would blow up like it did. Exploits had evolved and become widely available, mailing lists had fired up and for the next 24 hours things would be interesting.

Short of a zombie apocalypse, this is the event that any IT security enthusiast gets motivated by − an in the wild, reliable exploit that a lot of organisations had next to no time to respond to. My excitement was not about the cataclysmic ‘end of the world’ scenario that so many people had proclaimed; rather, it was about the fact there was work to be done, research to be completed, people to help and lessons to learn.

At 9am on the Wednesday, a team meeting set the goals for the day, consulting work was halted and we trawled through old reports to help our customers prepare for the dangers ahead. The internet had indeed burst into flame and our own chat services were busy as our coordinated effort to identify and help existing customers evolved into well organised crisis management. We found that:

• Approximately 20% of our existing customer base was vulnerable
• The majority of this number were unaware they were running the vulnerable version of OpenSSL
• Information was still unclear in the minds of many as to the nature and risk of the vulnerability

As we progressed into the afternoon, I had found a payment gateway that reported version 1.0.1e of OpenSSL in the HTML header. I noted it and called the client, only to hear the response that a change request had been planned on the weekend when it wouldn't affect their core business.

Despite my best efforts, I could not sway this person from their firmly held belief that a patch couldn't be installed except on weekends. Needless to say, their payment gateway was still reporting the vulnerable version during late night shopping on Thursday.

Based on my experience of April 2014, I have learned that:

1. First and foremost, organisations need to know their environment. Just this year I have successfully acquired administrative credentials for a system because no one at the customer was aware that the system needed to be patched.
2. Everyone needs to keep abreast of what is happening outside your environment. Subscription to alerting services or even reading mail lists and tweets provides a valuable source of information.
3. Security patching according to schedule or ‘within 48 hours’ is no longer applicable. Patch management must now incorporate an appreciation of the threat outside risk ratings or vague descriptions of consequence.
4. Detach emotion and be as objective as possible: the fastest way to burn your mind is to head down the self destructive path that many gain solace from in information security. The more you remove yourself from the ‘world is burning’ or ‘safeguard the homeland’ mentality the more effective you’ll be at fixing problems.

Since these events, I have observed these lessons have not necessarily been taken onboard by many. Emotion, salesmanship and a lack of awareness in information security continue to undermine our ability to protect the environments we’ve been charged with. Many sales teams since the events of April 2014 have claimed that their products protected customers against Heartbleed, instead of the patch, and oothers have neglected to learn from these events.

About the author

Edward Farrell is a seasoned penetration tester and information security consultant with nearly 10 years’ experience. In 2015 Edward sought to go out on his own and created Mercury Information Security Services. Edward’s new organisation provides customised information security services and advice for Australian businesses.

This article is brought to you by Enex TestLab, content directors for CSO Australia.