In putting together awareness programs for dozens of clients, the potential to integrate phishing simulations always comes up. For the most part, it seems like a staple of awareness programs. But when the concept of phishing is raised, I always ask, “Why?”
Yes, the question potentially costs me money. Also while most people perceive the phishing simulations as a direct way to decrease phishing susceptibility, the decrease might not be relevant or significant. So when I looked at a recent CSO article that asked security experts what they thought “success” meant when it came to phishing simulations, I was a frustrated.
The comments from security experts mostly focused on a reduction in clicking on simulated phishing messages. I assume people believe that if fewer people click on a simulated phishing message, fewer people will click on a real message. That is not necessarily the case. This discussion is actually much more complicated than it appears, and it involves dispelling many myths and specious beliefs about phishing.
What is security success?
Before looking at success in phishing simulations, we must first consider what is success for overall security efforts. First off, there is no such thing as security. The dictionary defines security as freedom from risk. There will always be risk, so security is unattainable. An implementable definition of security is risk management.
Risk management is essentially the act of cost effectively mitigating loss. In short, security efforts are successful if you reduce your loss by more money than your security countermeasures cost. For example, if you invest $500,000 in anti-malware software, and you reduce the costs of loss due to malware by more than $500,000, your security program is successful. If you reduce loss by less than $500,000, your program, or at least anti-malware, failed.
There is a general problem with this measure, as most organizations do not adequately track security-related losses. Without the appropriate metrics, it is hard to prove success. However, the principle is straightforward. If you plan in advance, you should at least attempt to gather the appropriate metrics.
The problems with phishing simulations
There are several critical issues with implementing phishing simulations. The first one is the actual receipt of the messages. With all services, you have to white list the messages to ensure they get to the recipients. So, you are testing people with phishing messages that they would never receive, as the white listing is implemented to avoid the messages getting sent to spam files or from being deleted, before reaching the recipients.
[ MORE ON CSO: How to avoid phishing attacks ]
Then there is the fact that just because a user does not click on one phishing message, it doesn’t mean they will not click on others. Some people might not click on cat videos, while they would click on a shipping message.
Then there is the sophistication of phishing messages to consider. I can purposefully manipulate the user response rate, if I choose. For example, if I want to show success in the program, I can create a very sophisticated message that uses inside information and is related to some timely event, and get a very high response rate. I would then follow it up with a more generic phishing message, such as a shipping message with poor grammar, and would get a very low rate.
The referenced article states that if phishing simulations get a 10 percent response rate, the effort is a success. As the previous paragraph highlights, a 10 percent response rate can mean little in actual effectiveness, depending upon the simulated phishing message used. However, even if you assume it is the most sophisticated simulated phishing message ever, that means that a significant number of people within an organization will still respond to the message.
More frequently, users begin to recognize the simulated phishing messages and do not respond, not because they are more aware of phishing concerns, but because they are aware of the simulations. Another common occurrence is that if one person detects a phishing message in an organization, they may then warn their coworkers about the message. The coworkers will then know to proactively delete the messages. In more than one simulation I was involved in, companies proactively warned employees that they will receive a simulated phishing message within a given time period for political reasons.
Phishing messages require technical failures to be successful
While security professionals seem to attribute responses to phishing messages as a demonstration of poor security awareness, it is actually a much more complicated issue. Again, there had to be a technical failure for messages to get to the user. More important, just because a user responds to a message, it does not mean that there should actually be a loss.