NETWORK FIREWALL 2011 COMPARATIVE TEST RESULTS

CHECK POINT POWER-1 11065 CISCO ASA 5585 FORTINET FORTIGATE 3950 JUNIPER SRX 5800 PALO ALTO NETWORKS PA-4020 SONICWALL NSA E8500

APRIL 2011 METHODOLOGY VERSION: 3.0

Licensed to: Purchaser (Single-User, INTERNAL USE ONLY) To receive a licensed copy or report misuse, please contact NSS Labs at: +1 (760) 270-9852 or advisor@nsslabs.com.

2011 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the authors. Please note that access to or use of this report is conditioned on the following: 1. 2. The information in this report is subject to change by NSS Labs without notice. The information in this report is believed by NSS Labs to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this report are at the readers sole risk. NSS Labs is not liable or responsible for any damages, losses, or expenses arising from any error or omission in this report. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY THE NSS LABS. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT ARE DISCLAIMED AND EXCLUDED BY NSS LABS. IN NO EVENT SHALL NSS LABS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet the readers expectations, requirements, needs, or specifications, or that they will operate without interruption. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective owners.

3.

4.

5. 6.

CONTACT INFORMATION
NSS Labs, Inc. P.O. Box 130573 Carlsbad, CA 92013 USA +1 (760) 270-9852 info@nsslabs.com www.nsslabs.com

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

EXECUTIVE SUMMARY
Corporate networks and data are under attack more than ever, and the threats continue to change as do vendor solutions. Firewalls will continue to play a key role in layered defenses. An essential part of layered security, firewalls must be stable, fast, and easy to deploy and maintain. During Q1 2011, NSS Labs performed the industrys most rigorous test of leading firewall solutions. This report has been produced for our enterprise subscribers, as part of NSS Labs independent testing information services. Leading vendors were invited to participate fully at no cost, and NSS Labs received no vendor funding. The time required to install and configure each device was recorded for purposes of estimating the total cost of ownership (TCO) calculations. Effectiveness and performance results were obtained with identical policies across products in order to provide comparable results. Key Findings Five of the six products allowed external attackers to bypass the firewall and become an internal trusted machine. Three of the six products tested crashed when subjected to our stability tests. These kind of crashes indicate the existence of a vulnerability which an attacker may be able to exploit in the field, given enough time. This lack of resiliency is alarming, especially considering all three were certified by ICSA Labs and/or Common Criteria certified. Performance claims in vendor datasheets are generally grossly overstated. Measuring performance based upon RFC-2544 (UDP) does not provide an accurate representation of how the firewall will perform in live real-world environments.

Recommendations If you have one of the firewalls that has issues with TCP split handshake, review NSS Labs remediation guidelines and contact your vendor. Before implementing remediations, consider performance impacts of turning on additional protections. Contact NSS Labs for assistance with determining performance requirements and capabilities of devices. If your firewall failed NSS Labs stability tests, encourage your firewall vendor to address stability issues ASAP to avoid exploitation. Consult NSS Labs subsequent firewall test report later in 2011 to ensure your vendor has remediated the issues. If your firewall is crashing, locking up, or displaying other unstable behavior, it may be the subject of an attack. If your organization is extremely risk averse or highly sensitive to down-time, consider migration to one of the more stable firewall platforms in our tests.

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

TABLE OF CONTENTS
1 Introduction ................................................................................. 1 1.1 The Need for Firewalls ...................................................................... 1 1.2 The Need for Testing ........................................................................ 1 1.3 About This Test Methodology and Report .......................................... 1 1.4 Tested Products ................................................................................ 1 1.5 About NSS Labs ................................................................................ 2 2 Security Effectiveness .................................................................. 3 2.1 Firewall Policy Enforcement.............................................................. 3 3 Performance ................................................................................ 8 4 Stability & Reliability.................................................................. 12 5 Total Cost of Ownership & Value ................................................ 15 6 Product Guidance ....................................................................... 17 6.1 Recommend .................................................................................... 18 6.2 Neutral ........................................................................................... 18 6.3 Caution ........................................................................................... 20 7 Test Methodology Elements Overview ........................................ 21 Appendix A: Special Thanks ............................................................ 23

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

TABLE OF FIGURES
Figure Figure Figure Figure Figure Figure 1: Rated Throughput (Mbps) ........................................................................................... 8 2: Maximum Concurrent Connections vs. HTTP Transactions & TCP Connections per Second .... 9 3: Maximum HTTP Connections per Second with Various Size Payloads ............................... 10 4: Maximum THroughput (MBPS) with Various Size Payloads ............................................. 10 5: UDP THroughput (MBPS) .......................................................................................... 11 3 - 3 year TCO ............................................................................................................ 16

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

1 INTRODUCTION
1.1 THE NEED
FOR

FIREWALLS

Firewall technology has been around for at least 25 years, and has undergone several stages of development; from early packet and circuit firewalls to application layer and dynamic packet firewalls. Across these stages, the goal has continued to be to provide a protective barrier between internal and external networks, while allowing for productive communications to pass from one side to the other. As firewalls will be deployed at critical choke-points in the network, the stability and reliability of a Firewall is imperative. Therefore prime directive of any firewall is that it must be stable / reliable. And it must not degrade network performance or it will never be installed. In order to establish a secure perimeter, a firewall must provide granular control based upon the source and destination IP Addresses and ports. The following capabilities are considered essential as part of a firewall: Basic packet filtering Stateful multi-layer inspection NAT Highly Stable Ability to operate at layer 3

1.2 THE NEED

FOR

TESTING

In this Firewall Group Test, NSS Labs objective was to answer the critical questions about product capabilities and limitations that enterprises could not answer for themselves without great effort and investment in time, equipment, and specialized expertise. In the process, we discovered some failures in the way firewalls have been traditionally tested. As a result, we found stability was more of an issue than we had anticipated given that all of the problematic products recently passed through certification with another major lab and/or were Common Criteria certified. And all but one vendor failed to properly handle a type of spoofing called a TCP Split Handshake attack. We believe the results indicate the need for more in-depth testing on a recurring basis. Considering that what gets measured, gets managed, and (hopefully) improved, we look forward to working further with end-users and vendors to continue enhancing corporate defenses.

1.3 ABOUT THIS TEST METHODOLOGY

AND

REPORT

NSS Labs test reports are designed to address the challenges faced by IT professionals in selecting and managing security products. The scope of this report is focused on: Security effectiveness Performance Stability Total Cost of Ownership (TCO)

1.4 TESTED PRODUCTS

NSS Labs testing is known to be the most rigorous in the industry, providing readers with hard-hitting real-world research and analysis. It is understandable that some vendors are concerned about the Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved. 1

marketing impact of potentially poor performance. However, enterprise and government buyers are increasingly asking what is getting through, why, and what to do about it. Vendors who chose to participate should be commended for their commitments to transparency and improvement. In order to garner the greatest participation, and allay any potential concerns of bias, we invited all leading vendors to submit products at no cost. We selected products for inclusion based on enterprise client requests. The following is a current list of the products that were tested, sorted alphabetically: Check Point Power-1 11065 Cisco ASA 5585-40 Fortinet Fortigate 3950B Juniper SRX 5800 Palo Alto Networks PA-4020 Sonicwall NSA E8500

All firewalls were generally available (GA) products. No Beta or otherwise unavailable products were included.

1.5 ABOUT NSS LABS

NSS Labs, Inc. is the worlds leading independent information security research and testing organization. Its expert analyses provide information technology professionals with the unbiased data they need to select the right product for their organizations. Pioneering intrusion detection and prevention system testing with the publication of the first such test criteria in 2001, NSS Labs also evaluates firewall, unified threat management, anti-malware, encryption, web application firewall, and other technologies on a regular basis. The firms real-world test methodology is the only one to assess security products against live Internet threats. NSS Labs tests are considered the most aggressive in the industry and its recommendations are highly regarded by enterprises. Founded in 1991, the company has offices in Carlsbad, California and Austin, Texas.

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

2 SECURITY EFFECTIVENESS
This section verifies that the Device Under Test (DUT) is capable of enforcing a specified security policy effectively. NSS Labs firewall analysis is conducted by incrementally building upon a baseline configuration (simple routing with no policy restrictions and no content inspection) to a complex real world multiple zone configuration supporting many addressing modes, policies, applications, and inspection engines. At each level of complexity, test traffic is passed across the firewall to ensure that only specified traffic is allowed and the rest is denied, and that appropriate log entries are recorded. The firewall must support stateful firewalling either by managing state tables to prevent traffic leakage or as a stateful proxy. The ability to manage firewall policy across multiple interfaces/zones is a required. At a minimum, the firewall must provide a trusted internal interface, an untrusted external/Internet interface, and (optionally) one or more DMZ interfaces. In addition, a dedicated management interface (virtual or otherwise) is preferred.

2.1 FIREWALL POLICY ENFORCEMENT

Policies are rules that are configured on a firewall to permit or deny access from one network resource to another based on identifying criteria such as: source, destination, and service. A term typically used to define the demarcation point of a network where policy is applied is a demilitarized zone (DMZ). Policies are typically written to permit or deny network traffic from one or more of the following zones: Untrusted This is typically an external network and is considered to be an unknown and non-secure. An example of an untrusted network would be the Internet. DMZ This is a network that is being isolated by the firewall restricting network traffic to and from hosts contained within the isolated network. Trusted This is typically an internal network; a network that is considered secure and protected.

The NSS Labs Firewall certification tests performance and the ability to enforce policy between the following: Trusted to Untrusted Untrusted to DMZ

Trusted to DMZNote: Firewalls must provide at a minimum one DMZ interface in order to provide a DMZ or transition point between untrusted and trusted networks.

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

2.1.1 B ASELINE P OLICY

Routed configuration with an allow all policy
Product CheckPoint Power-1 11065 Cisco ASA 5585-40 Fortinet 3950B Juniper SRX 5800 Palo Alto PA-4020 Sonicwall NSA E8500 Result PASS PASS PASS PASS PASS PASS

2.1.2 S IMPLE P OLICIES

Simple outbound and inbound policies allowing basic browsing and e-mail access for internal clients and no external access
Product CheckPoint Power-1 11065 Cisco ASA 5585-40 Fortinet 3950B Juniper SRX 5800 Palo Alto PA-4020 Sonicwall NSA E8500 Result PASS PASS PASS PASS PASS PASS

2.1.3 C OMPLEX P OLICIES

Complex outbound and inbound policies consisting of many rules, objects, and services.
Product CheckPoint Power-1 11065 Cisco ASA 5585-40 Fortinet 3950B Juniper SRX 5800 Palo Alto PA-4020 Sonicwall NSA E8500 Result PASS PASS PASS PASS PASS PASS

2.1.4 S TATIC NAT (N ETWORK A DDRESS T RANSLATION )

Inbound Network Address Translation (NAT) to DMZ using fixed IP address translation with one-to-one mapping.
Product CheckPoint Power-1 11065 Cisco ASA 5585-40 Fortinet 3950B Juniper SRX 5800 Palo Alto PA-4020 Sonicwall NSA E8500 Result PASS PASS PASS PASS PASS PASS

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

2.1.5 D YNAMIC /H IDE NAT (N ETWORK A DDRESS T RANSLATION )

Outbound Network Address Translation (NAT) (from Internal to External) where all outbound traffic hides behind the IP Address of the External Interface of the Firewall utilizing a pool of high ports to manage multiple connections.
Product CheckPoint Power-1 11065 Cisco ASA 5585-40 Fortinet 3950B Juniper SRX 5800 Palo Alto PA-4020 Sonicwall NSA E8500 Result PASS PASS PASS PASS PASS PASS

2.1.6 S YN F LOOD P ROTECTION

The basis of a SYN Flood attack is to not complete the 3-way handshake necessary to establish communication. Specifically the attacker (client machine A in fig. 6) refusing to send the ACK signal to the host server (B) after receiving the SYN/ACK from Host B. Such a connection is called a half open connection.

Instead of sending an ACK, attacker A sends another SYN signal to the victim server. The server again acknowledges it with a SYN/ACK and B again refuses to send the final ACK signal. By repeating this several times the attacker tries to overflow the data structure of the host server. The data structure is built in the memory of the host server with the purpose of keeping records of connections to be completed (or half open connections). Since the data structure is of a finite size, it is possible to overflow it by establishing a large number of open connections. Once overflow occurs the host server will not be able to accept new connections thus resulting in a denial of service. There is however a time-out associated with each of the connections (approximately 3 minutes) after which the host server will automatically drop the half open connections and can start accepting new connections. If the attacker can request connections at a rate higher than the victim servers ability to expire the pending connections then it is possible to crash the server.

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

Thus the objective of SYN flooding is to disable one side of the TCP connection which will result in one or more of the following: The server is unable to accept new connections. The server crashes or becomes inoperative. Authorization between servers is impaired.

The firewall is expected to protect against SYN Floods.

Product CheckPoint Power-1 11065 Cisco ASA 5585-40 Fortinet 3950B Juniper SRX 5800 Palo Alto PA-4020 Sonicwall NSA E8500 Result PASS PASS PASS PASS PASS PASS

2.1.7 IP A DDRESS S POOFING

This test attempts to confuse the firewall into allowing traffic to pass from one network segment to another. Each IP packet header contains the source and destination address of the packet. The source address is normally the address that the packet was sent from. By forging the header so it contains a different source address, an attacker can make it appear that the packet was sent by a different (trusted) machine. The machine that receives spoofed packets will send response back to the forged source address. The firewall is expected to protect against IP Address spoofing.
Product CheckPoint Power-1 11065 Cisco ASA 5585-40 Fortinet 3950B Juniper SRX 5800 Palo Alto PA-4020 Sonicwall NSA E8500 Result PASS PASS PASS PASS PASS PASS

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

2.1.8 TCP S PLIT H ANDSHAKE S POOF

This test attempts to confuse the Firewall into allowing traffic to pass from one network segment to another. The TCP Split handshake blends features of both the three way handshake and the simultaneous-open connection. The result is a TCP Spoof that allows an attacker to bypass the firewall by having the attacker instruct the target to initiate the session back with the attacker. Popular TCP/IP networking stacks respect this handshaking method, including Microsoft, Apple, and Linux stacks, with no modification.1 TCP spoofing attacks have been around for years, and presumed cured by modern firewalls. What makes this particular attack different is that it targets the client. The firewall is expected to protect against TCP Split Handshake spoofing.
Product CheckPoint Power-1 11065 Cisco ASA 5585-40 Fortinet 3950B Juniper SRX 5800 Palo Alto PA-4020 Sonicwall NSA E8500 Result PASS FAIL FAIL FAIL FAIL FAIL

This is a critical failure of five out of six leading firewalls. In order to help our enterprise customers remediate these issues, we contacted the vendors in early February 2011 and provided details of the issues. At the time of printing, April 11, 2011, Juniper, Palo Alto and Sonicwall had provided remediation steps. These are detailed in a separate document available to registered users at no cost at www.nsslabs.com. Cisco and Fortinet do not currently have remediations available, though we are still actively working with Cisco2 and Fortinet states it is working on validating a fix for a future release. Check back with us or the vendor for more information.

1 2

The TCP Split Handshake: Practical Effects on Modern Network Equipment, Tod Alien Beardsley & Jin Cisco Bug ID CSCtn29349 7

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

3 PERFORMANCE
NSS Labs collected extensive performance metrics during this test, according to our established methodology. The volumes of data produced by these tests are designed to capture maximum capacities or the edge of performance that may be obtainable for a given metric. In addition, our real-world traffic mix testing methods enable us to more accurately estimate the performance users can expect in their environments. Due to space considerations and the number of different products, we have summarized some of the most important figures that a network administrator should consider when sizing a deployment.

3.1.1 R ATED T HROUGHPUT

NSS Labs rates product performance based upon the average of three traffic types: 21KB HTTP response traffic, a mix of perimeter traffic common in enterprises, and a mix of internal core traffic common in enterprises. Details of these traffic mixes are available in the Firewall Test Methodology.
Juniper SRX 5800 5,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 45,000 42,000

Cisco ASA 5585-40

12,033

Palo Alto PA-4020

5,207

CheckPoint Power-1 11065

4,567

Fortinet 3950B

4,763

Sonicwall E8500

1,527

FIGURE 1: RATED THROUGHPUT (MBPS)

3.1.2 C ONNECTION D YNAMICS

Beyond overall throughput of the device, connection dynamics can play an important role in sizing a security device that will not unduly impede the performance of a system or an application. Maximum Connection and transaction rates help size a device more accurately than simply looking at throughput. By knowing the maximum connections per second, it possible to predict maximum throughput based upon the traffic mix in a given enterprise environment. For example, if the device maximum HTTP CPS is 2,000, and average traffic size is 44KB such that 2,500 CPS = 1Gbps, then the tested device will achieve a maximum of 800 Mbps ((2,000/2,500) x 1,000 Mbps)) = 800 Mbps. Following is a subset of figures from our performance tests. The aim of these tests is to stress the detection engine and determine how the sensor copes with large numbers of TCP connections per second, application layer transactions per second, and concurrent open connections. All packets contain valid payload and address data and these tests provide an excellent representation of a live network at various connection/transaction rates.

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

Note that in all tests, the following critical breaking pointswhere the final measurements are takenare used: Excessive concurrent TCP connections - latency within the firewall is causing unacceptable increase in open connections on the server-side. Excessive response time for HTTP transactions/SMTP sessions - latency within the firewall is causing excessive delays and increased response time to the client. Unsuccessful HTTP transactions sessions normally, there should be zero unsuccessful transactions. Once these appear, it is an indication that excessive latency within the firewall is causing connections to time out. Maximum Capacity
Theoretical Max. Concurrent TCP Connections w/Data 762,500 1,980,000 1,447,500 14,000,000 523,000 2,500,000 Maximum TCP Connections Per Second 25,900 281,950 31,200 290,000 22,400 36,000 Maximum HTTP Transactions Per Second 99,940 474,570 99,440 440,000 158,000 102,000

Product CheckPoint Power-1 11065 Cisco ASA 5585-40 Fortinet 3950B Juniper SRX 5800 Palo Alto PA-4020 Sonicwall E8500

The following chart depicts the relationship between the maximum concurrent connections and the number of HTTP transactions per second that can be transmitted and received through the device.
16,000,000 14,000,000 12,000,000 10,000,000 8,000,000 6,000,000 4,000,000 2,000,000 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 450,000 500,000 Juniper SRX 5800

Sonicwall E8500 Fortinet 3950B CheckPoint Power-1 Palo Alto PA-4020

Cisco ASA 5585-40

FIGURE 2: MAXIMUM CONCURRENT CONNECTIONS VS. HTTP TRANSACTIONS & TCP CONNECTIONS PER SECOND

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

3.1.3 M AXIMUM HTTP C APACITY

These tests aim to stress the HTTP detection engine in order to determine how the sensor copes with detecting and blocking exploits under network loads of varying average packet size and varying connections per second. By creating genuine session-based traffic with varying session lengths, the sensor is forced to track valid TCP sessions, thus ensuring a higher workload than for simple packetbased background traffic.
450,000 400,000 350,000 300,000 250,000 200,000 150,000 100,000 50,000 Juniper SRX 5800 Cisco ASA 5585-40 Fortinet 3950B Palo Alto PA-4020 CheckPoint Power-1 11065 Sonicwall E8500

HTTP CPS

44KB 165,000 24,130 21,720 18,400 13,000 2,400

21KB 320,000 43,350 23,450 18,600 17,500 4,400

10KB 357,000 69,720 24,900 21,000 18,900 6,400

4.5KB 370,000 72,330 25,500 25,500 20,940 8,400

1.7KB 390,000 75,000 26,100 33,000 21,810 8,600

FIGURE 3: MAXIMUM HTTP CONNECTIONS PER SECOND WITH VARIOUS SIZE PAYLOADS

Each transaction consists of a single HTTP GET request and there are no transaction delays (i.e. the web server responds immediately to all requests). All packets contain valid payload (a mix of binary and ASCII objects) and address data. This test provides an excellent representation of a live network (albeit one biased towards HTTP traffic) at various network loads.
70,000 60,000 50,000 Mbps 40,000 30,000 20,000 10,000 Juniper SRX 5800 Cisco ASA 5585-40 Fortinet 3950B Palo Alto PA-4020 CheckPoint Power-1 11065 Sonicwall E8500 44KB 66,000 9,652 8,688 7,360 5,200 960 21KB 64,000 8,670 4,690 3,720 3,500 880 10KB 35,700 6,972 2,490 2,100 1,890 640 4.5KB 18,500 3,617 1,275 1,275 1,047 420 1.7KB 9,750 1,875 653 825 545 215

FIGURE 4: MAXIMUM THROUGHPUT (MBPS) WITH VARIOUS SIZE PAYLOADS

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

10

3.1.4 R AW P ACKET P ROCESSING P ERFORMANCE (UDP T RAFFIC )

The aim of this test is purely to determine the raw packet processing capability of each in-line port pair of the device. It is not real world, and can be misleading. It is included here primarily for legacy purposes. This traffic does not attempt to simulate any form of real-world network condition. No TCP sessions are created during this test, and there is very little for the detection engine to do in the way of protocol analysis (although each vendor will be required to write a signature to detect the test packets to ensure that they are being passed through the detection engine and not fast-tracked from the inbound to outbound port).
180,000 160,000 140,000 120,000 Mbps 100,000 80,000 60,000 40,000 20,000 Juniper SRX 5800 Fortinet 3950B Palo Alto PA-4020 Cisco ASA 5585-40 CheckPoint Power-1 11065 Sonicwall E8500 128 13,000 8,300 7,300 7,280 1,940 770 256 39,000 12,050 8,000 11,470 3,650 1,400 512 76,000 18,110 8,000 15,330 6,925 2,840 1024 140,000 20,000 8,000 15,530 11,425 5,500 1514 160,000 20,000 8,000 15,930 12,140 6,000

FIGURE 5: UDP THROUGHPUT (MBPS)

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

11

4 STABILITY & RELIABILITY

Long term stability is particularly important for an in-line device, where failure can produce network outages. These tests verify the stability of the DUT along with its ability to maintain security effectiveness while under normal load and while passing malicious traffic. Products that are not able to sustain legitimate traffic (or crash) while under hostile attack will not pass. The DUT is required to remain operational and stable throughout these tests, and to block 100 per cent of previously blocked traffic, raising an alert for each. If any non-allowed traffic passes successfully - caused by either the volume of traffic or the DUT failing to bypass for any reason - this will result in a FAIL.

4.1.1 B LOCKING U NDER E XTENDED A TTACK

The DUT is exposed to a constant stream of security policy violations over an extended period of time. The device is configured to block and alert, and thus this test provides an indication the effectiveness of both the blocking and alert handling mechanisms. A continuous stream of security policy violations mixed with legitimate traffic is transmitted through the device at a maximum of 100Mbps (max 50,000 packets per second, average packet sizes in the range of 120-350 bytes) for 8 hours with no additional background traffic. This is not intended as a stress test in terms of traffic load (covered in the previous section) - merely a reliability test in terms of consistency of blocking performance. The device is expected to remain operational and stable throughout this test, and to block 100 per cent of recognizable violations, raising an alert for each. If any recognizable policy violations are passed - caused by either the volume of traffic or the sensor failing open for any reason - this will result in a FAIL.
Product CheckPoint Power-1 11065 Cisco ASA 5585-40 Fortinet 3950B Juniper SRX 5800 Palo Alto PA-4020 Sonicwall NSA E8500 Result PASS PASS PASS PASS PASS PASS

4.1.2 P ASSING L EGITIMATE T RAFFIC U NDER E XTENDED A TTACK

This test is identical to 4.1.1, where the external interface of the device is exposed to a constant stream of attacks over an extended period of time. The device is expected to remain operational and stable throughout this test, and to pass most/all of the legitimate traffic. If an excessive amount of legitimate traffic is blocked throughout this test caused by either the volume of traffic or the DUT failing for any reason - this will result in a FAIL.

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

12

Product CheckPoint Power-1 11065 Cisco ASA 5585-40 Fortinet 3950B Juniper SRX 5800 Palo Alto PA-4020 Sonicwall NSA E8500

Result PASS PASS PASS PASS PASS PASS

4.1.3 P ROTOCOL F UZZING & M UTATION

This test stresses the protocol stacks of the DUT by exposing it to traffic from various protocol randomizer and mutation tools. Several of the tools in this category are based on the ISIC test suite and the BreakingPoint Stack Scrambler component. Traffic load is a maximum of 350Mbps and 60,000 packets per second (average packet size is 690 bytes). Results are presented as a PASS/FAIL - the device is expected to remain operational and capable of detecting and blocking attacks throughout the test.
Product CheckPoint Power-1 11065 Cisco ASA 5585-40 Fortinet 3950B Juniper SRX 5800 Palo Alto PA-4020 Sonicwall NSA E8500 Result PASS PASS FAIL FAIL PASS FAIL

It is important to note that when a service or device to crashes it is most often due to a software vulnerability. And while not all vulnerabilities can be exploited, most can. Therefore we urge vendors whose devices failed this test to fix their devices at the earliest opportunity. NSS Labs considers a product to have failed this test if it becomes unstable and falls over crashing and not allowing any traffic to flow. In effect, the firewall becomes a doorstop and must be power-cycled to recover. NSS Labs considers a severe fail if upon failure, all traffic is allowed to to pass through the firewall, or if the firewall itself is breached upon failure.

In the case of Fortinet, upon failure the device rebooted and permitted an attacker unauthenticated root access via ssh. Fortinet was notified of this failure condition of build 279 and promptly instructed us to upgrade to build 303, after which, failure did not permit unauthenticated root access. NSS Labs recommends that Fortinet customers running FortiOS 4.0 MR2 Patch 1, build 279 should consider updating at the earliest opportunity.

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

13

If Fortinet had not issued the patch, the device would have automatically garnered a caution rating. However, since Fortinet responded promptly and corrected the issue, we are feel a neutral rating is still appropriate. Further, NSS Labs has determined the most responsible action is to issue a severe fail rating in order to notify Fortinet customers with devices deployed in the field who may be running the unpatched version.

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

14

5 TOTAL COST OF OWNERSHIP & VALUE

Firewall implementations can be complex projects with several factors affecting the overall cost of deployment, maintenance and upkeep. All of these should be considered over the course of the useful life of the solution. 1. 2. Fees paid to the vendor for the initial product and yearly maintenance Labor costs for installation, upkeep and tuning

No two network security products deliver the same security effectiveness or throughput, making apples to apples comparisons extremely difficult. In order to capture the relative value of devices on the market and facilitate such comparisons, NSS Labs has developed a unique metric to enable valuebased comparisons: Price per protected megabit/sec = Cost / (security effectiveness * throughput). Developed in 2009 by NSS Labs, this metric is used extensively in the sections below to evaluate cost of security, throughput and 3-year TCO. The figures here are based on list prices provided by vendors. NSS Labs clients can gain access to our TCO spreadsheets in order to customize comparisons based on special-offer pricing.

5.1.1 L ABOR PER P RODUCT ( IN H OURS )

Firewall solutions are among the most complex products in information security discipline. With the shortage of skilled and experienced practitioners, it is important to consider the required time and resources to properly install, maintain and manage the solution. Failure to do so could result in products not achieving their full security potential. This table estimates the annual labor required to maintain each device. There are three main components to be considered: 1. 2. Installation the time required to take the device out of the box, configure it, put it into the network, apply updates and patches, initial tuning, and set up desired logging and reporting. Upkeep the time required to apply periodic updates and patches from vendors, including hardware, software, and protection (signature/filter/rules) updates.

This table estimates the annual labor required to maintain each device. Since vendors sent their very best engineers to tune, NSS Labs assumptions are based upon the time required by a highly experienced security engineer ($75 per hour fully loaded). This allowed us to hold the talent cost variable constant and measure only the difference in time required to tune.
Product CheckPoint Power-1 11065 Cisco ASA 5585-40 Fortinet 3950B Juniper SRX 5800 Palo Alto PA-4020 Sonicwall NSA E8500 Installation (Hrs) 8 8 8 16 8 8 Upkeep / Year (Hrs) 25 25 25 50 25 25

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

15

5.1.2 P URCHASE P RICE AND T OTAL C OST OF O WNERSHIP

TCO incorporates the labor costs for each product over three years as described above with the fees paid to the vendor for purchase and maintenance licensing. Calculations are made as follows: Value 1 Year TCO 2 Year TCO 3 Year TCO Description of Calculation Initial Purchase Price + Maintenance + (Installation + Upkeep) * Labor rate ($/hr) 1 Year TCO + (Upkeep hours) * Labor rate ($/hr) 2 Year TCO + (Upkeep hours) * Labor rate ($/hr)

Each vendor provided pricing information. When possible, we selected the 24/7 maintenance and support option with 24-hour replacement as this is the option most organizations will select.
Product CheckPoint Power-1 11065 Cisco ASA 5585-40 Fortinet 3950B Juniper SRX 5800 Palo Alto PA-4020 Sonicwall NSA E8500 Purchase $60,000 $139,995 $130,495 $1,236,320 $35,000 $39,995 Maintenance / year $12,000 $23,519 $10,500 $67,930 $4,200 $4,275 1 Year TCO $74,475 $165,989 $143,470 $1,309,200 $41,675 $46,745 2 Year TCO $88,350 $191,383 $155,845 $1,380,880 $47,750 $52,894 3 Year TCO $102,225 $216,777 $168,220 $1,452,560 $53,825 $59,044

5.1.3 T HREE -Y EAR T OTAL C OST OF O WNERSHIP PER P ROTECTED M BPS

The following table illustrates the relative cost per unit of work performed. This is helpful in understanding value (Mbps of protected traffic). The least expensive product will not be the best value if it does not block attacks nor provide sufficiently high throughput.
Product Palo Alto PA-4020 Cisco ASA 5585-40 CheckPoint Power-1 11065 Juniper SRX 5800 Fortinet 3950B Sonicwall NSA E8500 Throughput 5,207 12,033 4,567 42,000 4,763 1,527 3 Year TCO $53,825 $216,777 $102,225 $1,452,560 $168,220 $59,044 Price / Mbps-Protected $10 $18 $22 $35 $35 $39

FIGURE 6 - 3 YEAR TCO

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

16

6 PRODUCT GUIDANCE
NSS Labs issues summary product guidance based on evaluation criteria that is important to information security professionals. The evaluation criteria are weighted as follows: 1. Security effectiveness - The primary reason for buying a firewall is to separate internal trusted networks from external untrusted networks while allowing select controlled traffic to flow between trusted and untrusted. Resistance to Evasion- Failure in any evasion class permits attackers to circumvent protection. Stability - Long term stability is particularly important for an in-line device, where failure can produce network outages Performance Correctly sizing a firewall is essential Value Customers should seek low TCO and high effectiveness and performance rankings.

2. 3. 4. 5.

Product Guidance Summary NSS Labs recommendations are based solely on empirical test data, validated over multiple iterations. Failure to resist all evasion attempts prevents a firewall from achieving Recommended status. Consult the detailed product guidance section for more information about each products rating. Products are listed alphabetically within their guidance rating groups. Rating Products Check Point Power-1 11065 Cisco ASA 5585-40 Fortinet Fortigate 3950B Juniper SRX 5800 Palo Alto Networks PA-4020 Sonicwall NSA E8500

Recommend

Neutral

Caution

Evasion Techniques: A firewalls effectiveness is significantly handicapped if its policies can be circumvented using obfuscation or evasion techniques, and our product guidance is adjusted to reflect this. Only one product passed our evasion testing. Security Effectiveness & Cost: Buyers should not only consider the initial purchase price of a given product, but also the total cost of ownership and relative value of the product.

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

17

6.1 RECOMMEND
A Recommend rating from NSS Labs indicates that a product has performed well and deserves strong consideration. Only the top technical products earn a Recommend rating from NSS Labsregardless of market share, company size, or brand recognition. Full evasion resistance required.

6.1.1 C HECK P OINT P OWER -1 11065

CheckPoint is one of the best known names in the firewall space, having pioneered stateful inspection technology back in the 90s. CheckPoints acquisition of Nokias Security Appliance group in April 2009 has enabled the company to produce a much more cohesive and simplified product. The new Power-1 11000 series platforms, combined with the newly released R75 is a robust and stable firewall. Product Power-1 11065 Max UDP Throughput 12,140 Mbps NSS Labs Rated Throughput 5,207 Mbps

CheckPoint Power-1 11065 was one of only three products that was able to withstand our stability test and remain functional. In addition, it was the only product that properly handled the TCP Split Handshake attack.

6.2 NEUTRAL
A Neutral rating from NSS Labs indicates that a product has performed reasonably well and should continue to be used if it is the incumbent within an organization. Products that earn a Neutral rating from NSS Labs deserve consideration during the purchasing process.

6.2.1 C ISCO ASA 5585-40

Cisco is one of the marketshare leaders in the enterprise firewall space. The ASA appliances are Ciscos intended replacement for the popular PIX firewalls. NSS Labs rates the ASA at 12,033 Mbps. Product ASA 5585-40 Max UDP Throughput 15,930 Mbps NSS Labs Rated Throughput 12,033 Mbps

Cisco ASA was one of only three products that was able to withstand our stability test and remain functional. However, it failed to properly handle the TCP Split Handshake attack. This is a serious weakness which leaves customers unprotected and should be remedied at the earliest opportunity. See NSS Labs remediation guide for additional information.

6.2.2 F ORTINET F ORTIGATE 3950B

Fortinet is a major player in the UTM space which for the past few years has been making a push into the Enterprise in the firewall market. NSS Labs rates the Fortigate 3950B at 4,763 Mbps. Product FortiGate 3950B Max UDP Throughput 20,000 Mbps NSS Labs Rated Throughput 4,763 Mbps

The Fortigate 3950B failed our stability test; upon failure the device rebooted and permitted an attacker unauthenticated root access via ssh. Fortinet was notified of this failure condition of build 279 and promptly instructed us to upgrade to (build 303). Once more we subjected the (patched) Fortigate 3950B to our stability test. The device crashed hard essentially locking up to the point

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

18

that it was unresponsive even via console requiring a manual power cycle. However, this time it did not permit unauthenticated remote login. NSS Labs recommends that Fortinet customers running FortiOS 4.0 MR2 Patch 1, Build 279 should consider updating at the earliest opportunity. Further, the Fortigate failed to properly handle the TCP Split Handshake attack. This is a serious weakness which leaves customers unprotected. According to a Fortinet representative, there is currently no fix, but one will be included in the next release which is scheduled for the third week in May. See NSS Labs remediation guide for additional information.

6.2.3 J UNIPER SRX 5800

Juniper is one of the marketshare leaders in the firewall space, having acquired NetScreen in 2004. The Juniper SRX platform is a relatively new multi-function gateway and is a sister to the stand-alone firewall products from Juniper. Product SRX 5800 Max UDP Throughput 160,000 Mbps NSS Labs Rated Throughput 42,000 Mbps

The SRX-5800 has been rated by Juniper as passing a whopping 140,000 Mbps of UDP traffic. We found that to be a slightly conservative number based upon NSS Labs rating of 160,000 Mbps using 1514 byte packets. However, overall we rated the SRX 5800 as a 42,000 Mbps firewall based on realworld traffic still very impressive, yet roughly 25% of the 140,000 Mbps rating. The SRX failed our stability test; it crashed denying all traffic requiring a reboot of the blade. However, we were able to do so without requiring a full manual power cycle of the entire chassis by logging into the management blade and rebooting the service blade. Further, the SRX 5800 failed to properly handle the TCP Split Handshake attack. This is a serious weakness which leaves customers unprotected and should be remedied at the earliest opportunity. See NSS Labs remediation guide for additional information.

6.2.4 P ALO A LTO N ETWORK PA-4020

Palo Alto Networks is a new entrant in the firewall category, focusing primarily on the Next-Gen firewall market. The PA-4020 has been rated by Palo Alto as a 2 Gbps firewall, a very conservative number based upon NSS Labs rating of 5,207 Mbps. Product PA-4020 Max UDP Throughput 8,000 Mbps NSS Labs Rated Throughput 5,207 Mbps

The PA-4030 was one of only three products that was able to withstand our stability test and remain functional. However, it failed to properly handle TCP Split Handshake attack. This is a serious weakness which leaves customers unprotected and should be remedied at the earliest opportunity. See NSS Labs remediation guide for additional information.

6.2.5 S ONICWALL E8500

Sonicwall is a challenger in the enterprise firewall category, having entered the market at the low end. The NSA E8500 has been rated by Sonicwall as as a 2.2 Gbps firewall. However, NSS Labs rates the E8500 at 1,527 Mbps. Product NSA E8500 Max UDP Throughput 6,000 Mbps NSS Labs Rated Throughput 1,527 Mbps

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

19

While the NSA E8500 comes with 8 Gigabit Ethernet ports, the device does not come with a dedicated management port. Therefore one of the eight ports must be used for management rendering the device with 7 usable ports (assuming out-of-band management). The Sonicwall NSA E8500 failed our stability test; it crashed requiring a full manual power cycle to recover. Further, the NSA E8500 failed to properly handle the TCP Split Handshake attack. This is a serious weakness which leaves customers unprotected and should be remedied at the earliest opportunity. See NSS Labs remediation guide for additional information.

6.3 CAUTION
A Caution rating from NSS Labs indicates that a product has performed poorly. Organizations using one of these products should review their security posture and other threat mitigation factors, including possible alternative configurations and replacement. Products that earn a Caution rating from NSS Labs should not be short-listed or renewed.

6.3.1 O THER F IREWALL V ENDORS

Our testing continues to show wide variations in security effectiveness and product reliability. Thus, buyers should view untested products with a fair amount of caution until they can be evaluated.

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

20

7 TEST METHODOLOGY ELEMENTS OVERVIEW

The following table lists the individual tests NSS Labs performed on each of the products. Direct references are provided to NSS Labs Test IDs from Sections 3 through 6 of NSS Labs Firewall Test Methodology v3.0.
Test ID 3 3.1 3.1.1 3.1.2 3.1.3 3.1.4 3.1.5 3.1.6 3.1.7 3.1.8 3.1.9 4 4.1 4.1.1 4.1.2 4.1.3 4.1.4 4.1.5 4.2 4.2.1 4.2.2 4.2.3 4.2.4 4.2.5 4.3 4.3.1 4.3.2 4.3.3 4.3.4 4.3.5 4.3.6 4.4 4.4.1 4.4.2 4.4.3 4.4.4 4.4.5 4.5 4.5.1 4.5.2 5 5.1 5.2 5.3 6 6.1 6.1.1 Description Security Effectiveness Firewall Policy Enforcement Baseline Policy Simple Policy Complex Policy Static NAT Dynamic / Hide NAT Syn Flood Protection Address Spoofing Protection Session Hijacking Protection TCP Split Handshake Performance Raw Packet Processing Performance (UDP Traffic) 128 Byte Packets 256 Byte Packets 512 Byte Packets 1024 Byte Packets 1514 Byte Packets Latency - UDP 128 Byte Packets 256 Byte Packets 512 Byte Packets 1024 Byte Packets 1514 Byte Packets Maximum Capacity Theoretical Max. Concurrent TCP Connections Theoretical Max. Concurrent TCP Connections w/Data Stateful Protection at Max Concurrent Connections Maximum TCP Connections Per Second Maximum HTTP Connections Per Second Maximum HTTP Transactions Per Second HTTP Capacity With No Transaction Delays 2.500 Connections Per Second 44Kbyte Response 5,000 Connections Per Second 21Kbyte Response 10,000 Connections Per Second 10Kbyte Response 20,000 Connections Per Second 4.5Kbyte Response 40,000 Connections Per Second 1.7Kbyte Response Real World Traffic Real World Protocol Mix (Perimeter) Real World Protocol Mix (Core) Stability & Reliability Blocking Under Extended Attack Passing Legitimate Traffic Under Extended Attack Protocol Fuzzing & Mutation Total Cost of Ownership & Value Ease of Use Initial Setup (Hours) Result

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

21

Test ID 6.1.2 6.2 6.2.1 6.2.2 6.2.3 6.2.4 6.3 6.3.1 6.3.2 6.3.3 6.3.4

Description Time Required for Upkeep (Hours per Year) Expected Costs Initial Purchase Ongoing Maintenance & Support (Annual) Installation Labor Cost (@$75/hr) Management Labor Cost (per Year @$75/hr) Total Cost of Ownership Year 1 Year 2 Year 3 3 Year Total Cost of Ownership

Result

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

22

APPENDIX A: SPECIAL THANKS

Special thanks go to our test infrastructure partners who provide much of the equipment, software, and support that make this testing possible:

Network Firewall Comparative Test Results 2011 NSS Labs, Inc. All rights reserved.

23