NSS Group logo

Unified Threat Management (UTM) 

Foreword

The NSS Group is pleased to present the results of the first round of  Unified Threat Management (UTM) testing, the first test of its kind. 

The NSS UTM test evaluates the performance, reliability, security effectiveness, and usability of UTM products. The testing covers the seven main security modules typically included in UTM products: Firewall, VPN, IDS/IPS, Anti Virus, Anti Spam, URL Filtering, and Content Filtering

Overall, the brand new test suite contains hundreds of individual tests, many of which are run multiple times, to provide the most thorough and complete evaluation of UTM products available anywhere today.  

It is important for readers to recognise, however, that we would normally spend a significant amount of time on testing a single IPS, IDS, Anti Spam, VPN or firewall product. Given that the same amount of time will be allocated to test each UTM product as we would normally spend on a dedicated device, it is clearly impractical for us to test each module of a UTM device to the same extent as we would the equivalent dedicated device.  

Where vendors wish to prove that a particular module performs in a specific way, it is possible to submit a UTM device for an additional security-specific test - NSS has already developed full test suites for all of the seven security engines listed above. For example, some vendors have already submitted a product to both the IPS test AND the UTM test, and in the past, many firewall vendors also submitted their product for separate VPN testing.  

Readers should therefore not rely on NSS approval of a UTM device to decide whether that device can operate as a dedicated, single-function security device (such as a firewall or IPS alone). If a reader is particularly interested in a UTM device as an IPS appliance or an Anti Virus gateway, they should look for a product with NSS Approved certification for both UTM and the specific technology in which he/she is interested.  

It is worth pointing out that not every product submitted for testing receives an NSS Approved award. Standards are very high, and only two products have received NSS Approved awards in the first round of testing.  

We believe that our UTM test methodologies will become the de facto standard for testing in-line UTM devices, and the NSS Approved logo an essential item on the list of requirements when purchasing these products.  

We also believe that these reports are essential reading for anyone considering deploying UTM devices in their networks, either in a test or live situation, and we hope that you find them both informative and useful in making your purchasing decisions. Details of all the products which completed the first round of UTM testing can be viewed on-line at www.nss.co.uk/utm 

Bob Walder

Table of Contents

Introduction

UTM Product Reviews

Fortinet FortiGate-3600
Executive Summary
Architecture
FortiGate-3600 System
Web Manager

FortiManager
Logging & Reporting
Performance
Security Effectiveness
Usability
Installation
Configuration
Policy Management
Alert Handling
Reporting and Anaylsis
Verdict
Contact Details

ISS Proventia M50 V3.2
Executive Summary
Architecture
Proventia Appliance
SiteProtector
Event Collector
SiteProtector SecurityFusion Module
SiteProtector Console
Local Management Interface (LMI)
High Availability
Performance
Security Effectiveness
Usability
Installation
Configuration

Policy Management
Alert Handling
Reporting and Analysis
Verdict
Contact Details

UTM Testing Methodology
The Test Environment
The Tests
Section 1 - Firewall
Section 2 - VPN
Section 3 - IPS
Section 4 - Content Filtering

Section 5 - Anti Virus
Section 6 - Anti Spam
Section 7 - All Modules

UTM Test Results

Appendix A - The Test Equipment
Spirent Communications SmartBits SMB-6000/SMB-600
SmartBits Applications
Spirent Communications Avalanche
Adtech AX/4000
Cisco Catalyst 6500 Series Switches 

Top         Home

Security Testing

NSS Awards

Group Test Reports

Articles/White Papers

Contact

Home

Click here to purchase electronic (PDF) and hard copy versions of this report

Send mail to webmaster with questions or 
comments about this web site.

Copyright � 1991-2005 The NSS Group Ltd.
All rights reserved.