Security testing firm NSS Labs has publically defended itself against furious accusations by security firm FireEye that a cool assessment of the security vendor's breach detection technology published last week was based on a flawed methodology.
Testing security products is a complex undertaking riven with uncertainties about whether any assessment can possibly simulate real-world attacks, which doesn't, of course, stop security vendors quoting these results when they do well.
Less frequently, when a vendor does badly - or just not as well as its rivals - the fur can start to fly. And so it was when last week NSS Labs' Breach Detection Systems Comparative Analyst Report gave FireEye's Web MPS 4310 and Email MPS 5300 systems a lower rating on its Security Value Map (SVM) compared to equivalent products from SourceFire, Trend Micro, Fortinet and Fidelis.
NSS Labs' assessment could be described as relatively stinging, slapping FireEye's product (and one from South Korean firm AhnLab) with a 'caution' while the others received a' recommended'. Anyone who believes that nobody reads these reports, or that they have little effect, might want to ponder the effect on FireEye's share price, which dropped nearly 8 percent on 3 April (although tech stocks were hit anyway the next day).
This would be a troubling day for any security company but for a firm barely six months on from a high-profile and well-subscribed IPO, any bump is unpleasant. Wounded, FireEye senior vice president Manish Gupta came out swinging, criticising the test methodology on a number of counts, in particular the selection of malware against which systems had been assessed, which he believed skewed FireEye's results down.
He also said the firm had "declined to participate in this test because we believe the NSS methodology is severely flawed," and that the "FireEye product they used was not even fully functional, leveraged an old version of our software and didn't have access to our threat intelligence."
It's a high-risk strategy for FireEye because it draws more attention to the results and risks the firm getting drawn into a verbal exchange that attracts even more rubber-neckers who don't understand the complex issues at hand. Sure enough, NSS Labs has today published its rebuttal of Gupta's claims.
In a post Don't Shoot the Messenger NSS Labs' Bob Walder denied that FireEye had not been a willing participant and said the firm's products were installed and configured by its engineers during 2013. Walder also rebutted Gupta's various claims over the testing methodology in some detail.
"In the grand scheme of things, FireEye's results were not that bad. The real issue here is that FireEye now has credible competition in the BDS [breach detection system] market place and the data from this NSS test shows it," wrote Walder.
That bring us to the really contentious thing about this test - on numbers alone FireEye really didn't do that badly, detecting 95 percent of web malware, 96 percent of email malware and 93 percent of exploits, giving an overall detection rate of 94.5 percent and a zero percent false positive rate. Although this is below the roughly 98-99 percent scores achieved by most of its rivals, the real problem NSS Labs found with the FireEye systems was their cost-performance.
This plots the total cost per Mbps protected against security effectiveness, which in the case of FireEye left its product with a figure of $427.85 (£280) against the highest-rated Sourcefire Advanced Malware protection costed at $231.86. In NSS's assessment at least, Sourcefire simply offers more protection for every dollar spent than does FireEye.
Regardless of the arguments on either side of this judgement, it is clear that breach protection security comes at a premium more or less which company is looked at; these are all expensive systems and measuring value for money and effectiveness will remain a black art shrouded in technical complexity. It is also the case that working how good they are at living up to the claims made in the sales brochures is not going to be as easy in 2014 for any firm as it was a year or two ago.