The worst attacks are the ones you don't even know to look for

Network-security tools have long focused on identifying compromises that they recognise from past encounters, but what do you do about the attacks that you’ve never seen before – or even thought to look for?

This question is guiding the development and refinement of a new generation of security-intelligence tools that complement the search for well known and well understood attacks with advanced data analytics that are designed to identify threats by finding anomalous behaviour within an organisation’s IT environment.

It's a more flexible approach that Mike O'Keeffe, Product Director for Financial Crime and Cyber with New Zealand data-analytics success story Wynyard Group, says is proving remarkably good at finding the 'unknown unknowns' of network and user behaviour – the threats that you not only can't detect, but don't even know to look for.

“Organisations are currently using technologies that are great at stopping the things that people know about using preventative technologies,” he explains, “but they're not great at stopping the things that people don't know about. Those are the things that can cause the organisation to have a ‘very bad day’.”

Identification of those unknown unknowns happens through the application of unsupervised machine-learning algorithms against standard log files that represent user behavior, network activity and data movement. These algorithms – which Wynyard Group has extrapolated from years developing expertise in the highly specialised field of forensic data matching for law-enforcement authorities around the globe – were recently built into a new proactive monitoring tool called Advanced Cyber Threat Analytics (ACTA).

When applied to a corporate IT environment, ACTA uses unsupervised machine learning to build a baseline of activity that is considered normal, and then flag deviations from these patterns. These ‘anomalies’ may not be necessarily be malicious – a user who suddenly logins in from overseas may simply be on holiday, but equally his login identity could be compromised.

These machine-learning algorithms have proved astute at picking up anomalous behaviour that can often be attributed to previously unknown, zero-day compromises, compromised user accounts and suspicious data movement “If you're telling the computer what it is that it needs to look for, essentially you're going down the same route as rules and signatures,” O'Keeffe says. “We want to let the machine figure out what's unusual for itself. The natural consequence of that is that we will find specific sets of activities that can be attributed to particular sets of attacks.”

A trial with an unnamed UK-based Risk Consultancy identified a potential internal compromise that had been carried out by a specific user who had downloaded a potentially unwanted program “that may have left the network open to being attacked,” O'Keeffe says. “We're finding stuff that organisations are not aware of.”

“Real time” versus meaningful time

Many cyber analytics products claim to operate on the network in ‘real time’ but ACTA adopts a different philosophy, collecting log data for longer periods to build up a meaningful understanding of normal behaviour.

This approach favours slow, careful and deliberate analysis over wire-speed data capture that is often quite limited in its capabilities as a result.

Drawing from Wynyard Group's heritage in after-the-fact forensic data analysis – which requires collection of large volumes of data before analysing them – the application of this technique to cybercriminal activity reflects the need to maintain a bigger-picture view of ongoing network activities.

“When you operate in real time you can only use a specific set of data to be processed through your machine-learning models to get a result,” O'Keeffe explains. “Taking a long term approach, the analysis process is more deliberative and more logical. You can take more variables into account – and when you generate a number of threats for investigation, you can be more confident that they are prioritised and focused on the things that investigators need to be concerned with.”

This approach is particularly important given the “gigantic” volumes of data being generated by network-security logging tools, which O'Keeffe says make it “absolutely impossible to keep pace with monitoring that type of data. There’s simply too much data to monitor; even using rules and signatures or trained models, it's very difficult to find serious compromises simply because of the volume of data.”

The application of special-purpose algorithms to massive data repositories has become a defining feature of the new economy, helping organisations make sense of their fast-accumulating information in a meaningful way. Gartner calls this trend the 'algorithm economy' and has highlighted its importance in helping companies apply advanced analytical techniques to their data.

“Proprietary algorithms that solve specific problems that translate into actions – will be the secret sauce of successful organizations in the future,” the firm's analysts have written, noting that algorithms “promise a brave new world of opportunities: software that thinks and does.

Cognitive software that drives autonomous machine-to-machine interactions. Artificial intelligence.” This prediction directly addresses the type of machine-learning technology that Wynyard Group is already offering within its ACTA tool – and O'Keeffe says customers have warmed quickly to the opportunities that better security-profiling analysis offers.

“We're already having conversations with large financial institutions and telecommunications providers, with Telstra being an early adopter customer,” he explains. “People are already very advanced in their thinking and they have this threat-hunting mentality where they are putting discovery teams together.”

“These teams focus on hunting for threats that they accept have breached their network, using a combination of advanced analytics and specific discovery tools to explore the analytics results and hunt through the forest of data in a targeted manner to find the threats.”

By keeping their minds open to new and potentially unknown issues, O'Keeffe says, those teams are hastening “the death of the use case” – a common conceptual paradigm that has the unintended effect of limiting the scope of analytic searches.

Instead, open-ended analysis allows for the nailing down of those 'unknown unknowns'. “Wherever they find the deviation from normal, that's where those discovery teams can go and look more closely,” O'Keeffe says. “If our set of highly-tuned algorithms can help them find answers to questions they didn't know about, then it will have been successful.”