How to become a master threat hunter

By Kane Lightowler, Carbon Black

Usually an IT professional who has been threat hunting in an environment for six months, a year, or more, will be regarded as a senior. He/she is expanding skills and knowledge, building and using tools, and beginning to mentor others – in fact becoming a master threat hunter.

His hunt findings will strengthen an organisation’s overall posture as follows.

✓ Improved defences: By chasing down intruders and deny their return, he is closing down one vulnerability after another. Over time, this begins to limit severely the techniques that can be used for successful intrusions.

✓ More detection: Defences are updated based on learning from previous incidents. Each time an intruder is caught, the new attack vectors are catalogued for gaining immediate visibility into subsequent attempts.

✓ Infrastructure familiarity: By chasing intruders all over the organisation’s environment, the threat hunter becomes intimately familiar with it, perhaps more so than its own designers and engineers. As an expert in defence, he is able to impart several useful suggestions to tighten things up from an architectural perspective. He will have an understanding of where the organisation may be weak in detection or response capabilities and be able to offer suggestions for additional tools that could be used to allow for a better overall defence.

✓ Better instincts: With experience in threat hunting, the hunter begins to build an instinct for spotting abnormal activity as well as the way in which the next intruders might attempt strike . . . and he will be there to catch them. This continuous improvement is partly about the organisation and its improved defences, and the rest is about growing prowess as a black belt threat hunter.

Achieving master threat hunter status doesn’t signify arrival. Rather, it represents a hunter’s outlook and discipline. He knows the enemies and how they work, and is determined to continue learning, to be one step ahead of them and anticipate their next moves. It requires constant vigilance and focus.

Be embedded

With the hunting tools at his disposal and the ability to look deeply into any server or endpoint in the organisation, the hunter is certainly embedded in the technical environment. The focus here needs to be about how to work with others in the organisation. While threat hunting can sometimes be depicted as the activities of a solitary threat hunter, more often than not he is a collaborator, known across IT and involved in its many varied teams.

So the hunter needs to work with teams across all of IT as they discuss current projects. To defend the organisation’s environment, he must work closely with these teams as they build and run the IT environment. Mainly this is because:

✓ He needs to understand what they built. As he observes system operation, interaction, and data movement, he needs to work with people who understand how systems were designed, built and implemented. This knowledge helps to better distinguish anomalies from legitimate operations.

✓ He needs to understand what they’re building. Given that most IT environments grow organically, he must be involved in this change. In working with teams in IT and building trust with them, they’ll explain more about their projects. There are two reasons for needing to be involved:

• The hunter needs to understand how their new systems work, so his understanding of what’s normal is accurate.

• He may need to advise them to make design enhancements based on his knowledge of the current threats and adversaries facing the organisation, so those new systems will be more secure by design.

Relationships with the teams in IT serve him well. As he works with these teams over months and years, his role as a subject matter expert will foster trust, and these teams will rely on him to provide accurate and reasonable guidance for improving the environment’s defences. They’ll be more apt to take advice and incorporate more and better security practices into new projects. This is why the threat hunter is there: to help everyone in IT build and administer systems and networks that have better defences.

Research

A master threat hunter needs an insatiable desire to learn more. To know about the newest exploit or that latest tool. The more he knows, the more he wants to learn, so he does some of his own research and runs experiments to see how things work. He builds his your own lab environments and test ranges. This process can include probing captured malware to play with an exploit kit he found, or reviewing experimental changes in systems to make them more resistant to attacks.

He might be building newer and more complex queries with threat hunting toolsets and trying to see if there are any new ‘hits’ against a dataset containing a new batch of attack vectors. With experience he will constantly be thinking of new ways that intruders can try to penetrate the environment, and how to stop them.

Pragmatically, research helps a hunter to design better techniques to validate those suspicions. He knows where the weak points are and it’s up to him to discover new ways to watch them. These methods include new traps, triggers and filters that may be used to tighten the environment a bit more. On rare occasions, research might lead to the discovery of a previously unknown zero‐day.

Developing intuition

A master threat hunter develops a sixth sense in hunting. Eventuaslly he sees attack patterns emerge out of a collection of seemingly unrelated data points. He begins to recognise reconnaissance and the intended activities behind the exploit and dropper tools that adversaries are using. At times, this can lead to the threat hunter being able to predict what intruders might do next so they can be stopped.

Using his intuition the threat hunter can put himself in an attacker’s shoes, see the environment as a potential target, and anticipate the attacker’s next move given this understanding of how he sees the defender. Thinking like an attacker separates the master threat hunters from the rest.

Educated hunches

Threat hunting is more than taking blind leaps, it’s also about making educated hunches, triggered by new pieces of intelligence that showed up in a threat feed or something the hunter read about like a new exploit in the wild. Follow leads in other ways too, including reviewing indicators from monitoring tools like an intrusion prevention system (IPS) that can alert personnel to traffic and discovering low‐reputation IPs or endpoint anti-malware sandboxes firing off notifications about an application pivoting in a way that it shouldn’t.

OODA

Intuition is also about OODA: observe, orient, decide, act. This is the military’s way of responding to situations in combat operations. The threat hunter is in combat too — on the cyber battlefield. An example of OODA   would go something like this:

✓ Observe: Collect data from sensors on your endpoints and events in the network.

✓ Orient: Discern what this data means in context. How does this information relate to other information and what could it mean? Could command and control (C&C) traffic be occurring, or could an endpoint be under attack from a ransomware variant?

✓ Decide: Make a decision about what to do. When you have a clear picture of an incident, determine a course of action. Typically this is the containment phase in which your incident response strategy will kick in. Only after the breach has been scoped should you proceed to the eradication and subsequent recovery and feedback stages to prevent similar intrusions from recurring.

✓ Act: Execute the plan to shut down the intrusion, harden the organisation’s security posture, and enhance detection.

Repeat

While many hunts might return ‘empty’ and no intrusion will be discovered that leverages that particular vulnerability, the knowledge created is incredibly valuable because you’ve created a series of processes and detection mechanisms that serve to harden the organisation against future potential incursions.

One way to expand knowledge about systems and data in an environment is to mentally build a model representing how they work and interact together. The same principle holds true in learning how an intruder might attack an organisation:

Study and develop models that represent how these actors operate. In developing security acumen, you may notice a tendency to stand inordinately firm in certain beliefs and opinions:

✓ Operating systems always open files like this.

✓ Intruders would never attack this program.

The mental models in a threat hunter’s subconscious help him understand complex topics and navigate them with ease.

While these constructs can be helpful to simplify certain concepts, never become too entrenched in a certain way of thinking because you blind yourself from new ways of thinking. This case holds doubly true in the security field where, especially with new technology, the only constant is change.

Be open to changing your understanding about things when new information comes in. This is known as strong opinions, loosely held, which is the safety valve that helps you recognise new facts that may change the way you think about things. For instance, how operating systems and applications do what they do and how attackers do what they do.

Cling to your time‐honoured beliefs too tightly and your hunts may suffer - you might not only return with no prey, but could become the prey.

While many hunts might return ‘empty’ and no intrusion discovered that leverages a particular vulnerability, the knowledge created is incredibly valuable because the hunter has created a series of processes and detection mechanisms that serve to harden the organisation against future potential incursions.

Developing personal tools

Master threat hunters don’t rely solely on vendors’ tools and interfaces.They view these resources as a starting point and work to engineer ways to extend and correlate the data and capabilities of these tools to build a system in which the whole is greater than the sum of its parts:

✓ Custom data collection scripts and analyst tools: Occasionally master threat hunters may need to write their own scripts to collect or analyse data. One example, writing a simple WMI script to collect various instances of persistence in the Windows registry.

Another could be building a python utility to generate analytics on a set of metrics to discover anomalous data points. Typically, master threat hunters are no strangers to leveraging powerful instruments like pivot tables and regular expressions to twist collections of data for a specific purpose.

✓ Custom integrations: Likely there are a lot of tools in the environment, many of which may have APIs or interfaces, that can be used to acquire or distribute information.

A trigger in an endpoint detection tool could activate the creation of a new IPS or firewall rule used to block a particular network connection. Or information from a threat feed could be filtered and fed into a tool to update its own rules that could then action a ticket to the help desk or even isolate a system on the network.

Master hunters are also builders and often they’ll act as both the problem finder and the problem solver. They must be able to understand how new attacks work and how to ‘stitch’ together various pieces of information available in the environment to enhance visibility and defences.

Setting landmines

A master threat hunter thinks ahead and anticipates what a known or a potential adversary might do. In this scenario, hunters can set landmines for attackers. Such methods attempt to attract attackers so that an alarm can be raised to alert security that illegitimate activity may be occurring.

Using incident detection and response tools means setting up queries for events that might happen. Here it’s critical to fuel passion to learn about new, clever attack vectors. As the hunter continues to develop his mental cyber armoury, he learns how to probe sections of the environment that were previously invisible.

In addition to standard hunting tools, he can leverage more advanced resources such as honeypots in an attempt to lure malicious actors into attacking a decoy target loaded with intrusion detection monitoring sensors. Instead of housing legitimate data, a honeypot is built to impersonate critical assets while having extremely sensitive monitoring and alerting configured.

In certain organisations, the intrepid hunter might go one step further to create honey accounts, which contain one or more honeypots, and set up user accounts that follow certain naming conventions for VIP users, and monitor for any access attempts (meanwhile, the VIP users are assigned other legitimate logins).

Tags IT professionalsviruscyber attacksIT environmentsthreat hunting

Show Comments