"The analysts are on the left. They are performing the monitoring and analysis," he said. "And on the right are the security engineers. They are responsible for fault configuration performance management of our services. That is, any firewall policy changes, any patching of systems, and any outages on a system that a client might need."
The system provides checks and balances, he noted. Analysts determine if there is a problem worth responding to but are unable to change anything. The engineers take action, if necessary.
The SOC is only one part of the managed security system. Symantec also has network of sensors deployed called "Deep Site." Users can download the agent and see a quick snapshot of current attack and threat trends. And there are response labs. In the labs, employees dissect malware to understand its methodology, how severe it is and then push it back out to customers in the form of products.
That dissection process includes 2 million decoy email accounts, or "honeypot networks," according to Geyer. They are decoy email accounts set up to gauge new kinds of spam. And there are also regional considerations that come into play because malware threats that affect some parts of the world are often unheard of in other countries.
"Vulnerability data is very different from malware which is very different from attack trends. And spam and phishing data are different. So, unless you have purposefully set up ways of getting slices of data, you miss the multidimensional aspect of security threats."
Looking for a needle in a needle stack
Of the 2 billion security logs analyzed by the SOCs each day, there are many incidents that look very bad but that are benign, said Geyer. In fact, about 3,300 are incidents that merit further investigation. But there are many which look benign, that are very bad. About 100 per day end up being severe incidents that need action, which is why Geyer likens the process to looking for a needle in a needle stack.
"If you were to automate this, judging by number of logs we analyze, you would miss most of the problems. It takes an expert to analyze it to see if there is something malicious going on."
We see this process at work by visiting the desk of Analysis Supervisor Tracy Williams, who is reviewing logs and making decisions about what needs further attention.