Next Generation security firewalls offered by many different vendors were recently tested by an Austin, Texas-based security product testing firm, NSS Labs. These results gained a lot of attention and proved to be controversial, specifically regarding the results of one vendor, that is, Palo Alto Network Inc.’s appliance, which got a “recommended” score last year, and this year got a “caution” rating instead.
The report suggested that the firewall failed to protect against certain attacks and underperformed against the firewalls of the competitors. The firewall was tested in comparison with firewalls from other vendors like Check Point Software Technologies, Cisco Systems, Dell, Fortinet, Intel Security (formerly McAfee) and WatchGuard. The results even surprised some analysts doe to the recognition of Palo Alto product as an industry leader. NSS Labs claimed that the Palo Alto PA-3020 Appliance passed stability and reliability tests, and enforced firewall policies. They also confirmed that it successfully enforced complex outbound and inbound policies. However, the appliance fell short in detecting evasion measures which is often used by attackers to bypass firewalls. NSS Labs was able to conduct a bypass, using RPC and IP Fragmentation attacks. It was also noticed that the appliance took a performance hit, earning a 719-Mbps rating while the vendor claimed a 1-Gbps performance. The tests gave the company’s next generation firewalls a 60.9 percent security effectiveness score and a below-average total-cost-of-ownership rating. NSS labs specifically called out the PA-3020’s low overall resistance score, a mere 65% , and its low 64% exploit block rate, which dropped from an 80% rating in last year’s report.
Meanwhile, Palo Alto did not take the results lightly and questioned the reliability of the results and the NSS testing methodology. The senior vice president of product management at Palo Alto Network said that the company did not cooperate in the NSS Labs tests and hence claimed that the poor scores were a result of the fact that unlike competitor products in the tests, Palo Alto did not provide guidance on the configuration and tuning of the device. However, NSS Labs Founder and Chief Research Officer responded to Palo Alto’s claims, noting that the participation of the company is not a factor in how their products are evaluated and that each vendor is treated exactly the same, regardless of whether they make an engineer available to support the testing. Palo Alto argued that it doesn’t understand how NSS could come to these results and claim “a drastically different result compared to the same tests run against the same technology in 2013”. It was noted that Palo Alto’s product ran PAN-OS version 4.1.9 in last year’s test, while this year’s product used version 6.0.3, and that specific changes in the newer platform resulted in the susceptibility of the product to multiple evasion techniques.
Moreover, in a show of good faith, NSS Labs offered detailed information about its findings to Palo Alto customers for free to “allow the company’s customers to at least attempt to improve the protection offered by PAN-OS devices to the best of their ability, until such time as Palo Alto Networks decides to take this issue seriously.” It was also seen that NSS gave “neutral” ratings to firewall models from Barracuda Networks Inc, Cisco and Sophos Cyberoam.
