Palo Alto next-gen firewall marked a 'caution' in NSS Labs test

Do independent security tests make or break products? Normally not but the latest assessment of next-generation firewalls (NGFWs) by testing firm NSS Labs will make mixed reading for some of the products with one - Palo Alto's PA-3020 - described as "below average" with a rating of "caution".

Most of the other 11 NGFWs in the test performed pretty well on NSS Labs' Security Value Map (SVM), which with its X and Y axes vaguely resembles a sort of Gartner Quadrant for security systems.

Eight out of twelve of the systems received a 'recommended' status, exceeding 90 percent for security effectiveness - Check Point's 13500, Cisco's 5525-X and Firepower 8350, Dell SonicWALL's SuperMassive E10800, Fortinet's FortiGate-1500D and FortiGate-3600C, McAfee's NGF-1402, and WatchGuard's XTM1525.

Neutral ratings were awarded to Barracuda's F800b, Cisco's 5585-X SSP60 and Cyberoam's CR2500iNG-XP for a variety of reasons varying from below average security effectiveness (Barracuda and Cyberoam) to price-performance (Cisco). Palo Alto's PA-3020 was marked 'below average' on both security effectiveness and price-performance.

The positive news anyone looking to buy one of these products is that the total cost of ownership (measured against Mbps protected) is half what it was in 2013 at only $21.80 - high-end firewalls are getting cheaper for a given throughput.

But some failings are still apparent.

"Evasions continue to be a challenge for the industry," said NSS Labs CEO, Vikram Phatak. "To date, every single NGFW group test has resulted in at least one vendor missing one or more critical evasions.

"If someone uses an evasion to circumvent a security product, you will never know until you are compromised. This is why ongoing independent testing is so important to cyber resiliency," he said.

So what went wrong for Palo Alto's system? The PA-3020 blocked 93.1 percent of attacks against server applications, 92 percent of attacks against clients, giving a 92.5 percent overall score, which sounds quite good. What let it down was its ability to protect against three classes of what are called 'evasions', techniques for disguising an attack to avoid detection.

As NSS Labs states: "many of the techniques used in this test have been widely known for years and should be considered minimum requirements for the NGFW product category."

Techworld received no statement from Palo Alto by press time but in comments to another website the company questioned the NSS Labs methodology.

Although the firm will receive negative headlines for the latest test, in the past NSS Labs has given its products passing marks while calling out other products. As with any class of security product, performance varies over time. The fact that Palo Alto didn't do as well as its rivals in this test doesn't mean that it will do so in the same test in a year's time.

NSS Labs has been here before with other vendors. Earlier this year it dished out some pain to FireEye's Web MPS 4310 and Email MPS 5300 products which were given the same 'caution rating. That generated some heat as FireEye sought to defend its products in public. Perhaps wisely, Palo Alto has yet to copy that tactic.