Test Description

High Severity attacks

These are recent attacks which are deemed to be serious and which can result in remote server compromise of systems which are widely deployed and generally Internet-facing (i.e. Microsoft OS, IIS, Apache, etc.). Also included are some older remote compromise attacks which are included in readily-available attack tools such as Metasploit, Core Impact, etc. It is possible to detect these attacks with a high degree of confidence, leading to low incidence of false positives. These attacks should therefore be enabled in block mode by default.

Medium Severity attacks

These are recent attacks which are deemed to be of lower severity than those in Test 1.1.1, but which nevertheless could have serious consequences - for example: remote server compromise of systems which are not generally Internet-facing, or DOS conditions which are easily recoverable and do not lead to server compromise. Because it is not always possible to detect these attacks with 100 per cent confidence, or because they are not considered critical, these attacks may be rated differently by different vendors, and may be enabled in either block or log mode.

Low Severity attacks

These are recent attacks which are deemed to be of lower severity than those in Test 1.1.2 by virtue of the fact that the systems are not generally Internet-facing or are not widely deployed. These attacks may be rated differently by different vendors, and may be enabled in either block or log mode.

Audit only

These are recent and older attacks which are deemed to be of low severity and will have little or no impact on target hosts. Some older attacks are used here to determine when vendors are “shedding” older signatures for performance reasons - these will typically be those types of attacks which are used most often in widely available vulnerability scanner software, and for which we believe alerts should still be raised as an indication that reconnaissance is being performed against protected systems. Alerts are raised for information purposes only. Policy tuning is allowed to ensure all attacks in this section are covered.

Reconnaissance

These are system probes and enumeration attempts which are of no serious threat, and whose sole purpose is to elicit information from target hosts. Alerts are raised for information purposes only. Policy tuning is allowed to ensure all attacks in this section are covered

DOS/DDOS

These are serious Denial of Service attempts using SYN Flood techniques (up to 100Mbps on a single in-line port pair). Two types of SYN Flood attack are used - one with a single source IP address (DOS), and one with multiple source IP addresses (DDOS). During the attacks, latency will be measured and attempts made to create legitimate sessions to servers through the DUT. The effect on the server(s) under attack will also be measured.

P2P (optional)

This is peer-to-peer file-sharing traffic from common P2P programs. Not all products will offer the ability to filter this traffic (this is a policy rather than pure security issue), and therefore failure to detect and block will not be seen as cause for failure in the test. However, because of potential for false positives and the possible impact on legitimate traffic (especially if P2P traffic is permitted), this traffic should not be blocked using the default policy or recommended settings

Spyware (optional)

This is traffic from common Spyware/Adware programs which are rated as “critical” or “high” severity (i.e. potential to cause remote compromise), and test cases will be created to include initial download/infection and subsequent “phone home” traffic. Not all IPS products will offer the ability to filter this traffic (some vendors may consider this more of an Anti Virus or dedicated Anti Spyware issue), and therefore failure to detect and block will not be seen as cause for failure in the test, and detection requirements are lower (for the moment) than other types of exploits.

Server-to-Client Exploits (optional)

Most HTTP exploits occur in client requests to servers (attempted buffer overflows, malformed requests, and so on). Given the amount of data that is returned by the server, many devices do not monitor this side of the traffic flow for performance reasons. Some vendors do not include such signatures because they also offer host-based IPS solutions which are often better suited to preventing this type of exploit. For the moment, this section remains optional, though given the increasing use of such exploits to compromise client systems, this section will become mandatory in future tests. This section is composed of test cases where the exploit traffic is “reversed”, from server to client. Where the DUT does provide server-to-client signatures, they will be tested here - given the lack of coverage of this type of signature in the IPS industry at present, however, there is no minimum detection requirement. The vendor will also be given the option to disable these signatures for the main performance tests, but this will be noted in the report. For this test, we leave it to the reader or potential purchaser to decide if this feature is essential to them, and, if included, whether the DUT should be capable of detecting such exploits at full performance.

False Positives

Trace files of normal traffic with “suspicious” content are sent across the DUT, together with several “neutered” exploits which have been rendered completely ineffective. In addition, for every malicious test case used in Test 1.1, normal traffic is sent using the same mechanism. If a signature has been coded for a specific piece of exploit code rather then the underlying vulnerability, or if it relies purely on pattern matching, or it is too generic in its detection methods, some of these false alarms could be alerted upon

False Negatives

A number of common exploits are modified such that, whilst still accomplishing their intended task (i.e. to gain a root shell) their “signature” looks quite different. If a signature has been coded for a specific piece of exploit code rather then the underlying vulnerability, or if it relies purely on pattern matching, some of these false negative cases could evade the IPS and succeed in their purpose

Baseline Attack Replay

A number of common exploits are replayed across the DUT to ensure that they are detected in their unmodified state. These will be chosen from a suite of older/common basic exploits for which NSS is certain that all vendors will have signatures. None of the exploits used in Section 1 will be used as evasion baselines, to ensure that vendors are not provided with any information on the content of the Section 1 test� suite.

2.2.2

Ordered 8 byte fragments, duplicate last packet

Ordered 1 byte segments, interleaved duplicate segments with requests to resync sequence numbers mid-stream

2.3.8

2.4.2

Canvas Reference Implementation Level 2 (MS)

Test ID

2.5.1

2.5.2

2.5.3

2.5.4

2.5.5

2.5.6

2.5.7

2.5.8

2.5.9

2.5.10

2.5.11

2.5.12

2.5.13

2.5.14

2.5.15

Test ID

2.6.1

UTF-16 character set encoding (big-endian)

The UTF-16 character set specifies a 2-byte sequence for most characters and a 4-byte sequence for the others (a small percentage). Recoding an HTML document in UTF-16 changes its appearance significantly. A document that contains just the ASCII subset of characters will appear to have a null byte between every one of the original characters. There are also two different forms of the UTF-16 encoding depending on whether the null high byte comes first (big-endian) or second (little-endian) - this test uses big-endian byte ordering

2.6.2

UTF-16 character set encoding (little-endian)

As for Test 2.6.1, but using little-endian byte ordering

2.6.3

UTF-32 character set encoding (big-endian)

This UTF-32 character set specifies a 4-byte sequence. Like the UTF-16 character set encoding there are two variations - big-endian and little-endian - and this test case uses big-endian byte ordering

2.6.4

UTF-32 character set encoding (little-endian)

As for Test 2.6.3, but using little-endian byte ordering

2.6.5

UTF-7 character set encoding

The UTF-7 character set encodes most ASCII characters as themselves. However, in addition to recoding non-English characters as other encodings do, it also recodes many punctuation symbols, including many of the symbols that are important to the HTML specification. Therefore, recoding an HTML document in UTF-7 significantly changes its appearance

2.6.6

Chunked encoding (random chunk size)

Chunked encoding allows the server to break a document into smaller chunks and transmit them individually. The server needs only to specify the size of each chunk before it is transmitted and then indicate when the last chunk has been transmitted. Since chunked encoding intersperses arbitrary numbers (chunk sizes) with the elements of the original document, it can be used to greatly change the appearance of the original document as observed “on the wire”. In addition, the server can choose to break the document into chunks at arbitrary points. This makes it difficult for simple pattern matching systems to reliably identify the original HTML document from the raw data on the network

2.6.7

Chunked encoding (fixed chunk size)

As for Test 2.6.6 but with fixed 8 byte chunk sizes

2.6.8

Chunked encoding (chaffing)

As for Test 2.6.6 but with “chaffing” (arbitrary numbers inserted between chunks)

2.6.9

Compression (Deflate)

Per RFC 2616, the HTTP protocol allows the client to request and the server to use several compression methods. These compression methods not only improve performance in many circumstances, they completely change the characteristic size and appearance of HTML documents. Furthermore, small changes in the original document, can greatly change the final appearance of the compressed document. This property of these algorithms could be used to obfuscate hostile content for the purpose of evading detection. The deflate compression method is a Lempel-Ziv coding (LZ77), specified in RFC 1951

2.6.10

Compression (Gzip)

As Test 2.6.9, but using gzip compression as specified in RFC 1952

2.6.11

Combination

This test applies a combination of UTF-7 encoding (Test 2.6.5), Gzip compression (Test 2.6.10) and chunked encoding with random chunk sizes (Test 2.6.6) to the malicious Web page.

Test ID

2.7.1

2.7.2

2.7.3

2.7.4

2.7.5

2.7.6

2.7.7

2.7.8

2.7.9

Test ID

2.8.1

2.8.2

2.8.3

Test ID

3.1.1

Theoretical Maximum Concurrent TCP Connections

This test is designed to determine the maximum concurrent TCP connections of the DUT with a 8 byte object size. This response size would not typically be found on a normal network, but it provides the means to determine the maximum possible concurrent connections figure.

Client and server are using HTTP 1.0 with keep-alive, and the client will open a TCP connection, send 10 HTTP requests, and close the connection. This ensures that TCP connections remain open until all ten HTTP transactions are complete, and a “user think time” of 30 seconds between every HTTP Transaction ensures a high number of concurrent connections. Load is increased until one or more of the defined breaking points is reached (the concurrent TCP connections breaking point does not apply to this test). Avalanche load specification is Connections Per Second.

3.1.2

Maximum Concurrent TCP Connections with 1K Response

This test is designed to determine the maximum concurrent TCP connections of the DUT with a 1Kbyte object size.

Maximum Concurrent TCP Connections with 5K Response

This test is designed to determine the maximum concurrent TCP connections of the DUT with a 5Kbyte object size. This is more typical of a “normal” network

Test ID

Test Description

3.2.1

Attack Detection - Light Load

This test determines if the sensor is able to detect new exploits as the number of open sessions reaches 10 per cent of the maximum determined in Test 3.1.1

3.2.2

Attack Blocking - Light Load

This test ensures that the sensor continues to block new exploits as the number of open sessions reaches 10 per cent of the maximum determined in Test 3.1.1

3.2.3

State Preservation - Light Load

This test determines if the sensor maintains the state of pre-existing sessions as the number of open sessions reaches 10 per cent of the maximum determined in Test 3.1.1.

A legitimate HTTP session is opened and the first packet of a two-packet exploit is transmitted. As the number of open connections approached the maximum, the initial HTTP session is then completed with the second half of the exploit and the session is closed. If the sensor is still maintaining state on the original session, the exploit will be recorded. If the state tables have been exhausted, the exploit string will be seen as a non-stateful attack, and will thus be ignored.

Both halves of the exploit are required to trigger an alert - a product will fail the test if it fails to generate an alert after the second packet is transmitted, or if it raises an alert on either half of the exploit on its own

3.2.4

Pass Legitimate Traffic - Light Load

This test ensures that the sensor continues to pass legitimate traffic as the number of open sessions reaches 10 per cent of the maximum determined in Test 3.1.1

3.2.5

Attack Detection - Heavy Load

This test determines if the sensor is able to detect new exploits as the number of open sessions approaches the maximum determined in Test 3.1.1

3.2.6

Attack Blocking - Heavy Load

This test ensures that the sensor continues to block new exploits as the number of open sessions approaches the maximum determined in Test 3.1.1

3.2.7

State Preservation - Heavy Load

This test determines if the sensor maintains the state of pre-existing sessions as the number of open sessions approaches the maximum determined in Test 3.1.1. Method of execution is identical to Test 3.2.3

3.2.8

Pass Legitimate Traffic - Heavy Load

This test ensures that the sensor continues to pass legitimate traffic as the number of open sessions approaches the maximum determined in Test 3.1.1

Attack Detection - Maximum Exceeded

This test determines if the sensor is able to detect new exploits as the number of open sessions exceed the maximum determined in Test 3.1.1

3.2.10

Attack Blocking - Maximum Exceeded

This test ensures that the sensor continues to block new exploits as the number of open sessions exceed the maximum determined in Test 3.1.1

3.2.11

State Preservation - Maximum Exceeded

This test determines if the sensor maintains the state of pre-existing sessions as the number of open sessions exceed the maximum determined in Test 3.1.1. Method of execution is identical to Test 3.2.3

Pass Legitimate Traffic - Maximum Exceeded

This test ensures that the sensor continues to pass legitimate traffic as the number of open sessions exceed the maximum determined in Test 3.1.1

Test ID

3.3.1

Stateless Attack Replay - Trigger Packet Only

Packets containing valid exploit “triggers” are transmitted via the DUT, but without first establishing a valid session with the target server. The session tear down and acknowledgement packets are also removed so that the sensor can not “infer” that a valid connection was made. This is typical of the traffic generated by tools such as Stick and Snot.

3.3.2

Stateless Attack Replay - Trigger Packet Plus Session Tear-Down

Packets containing valid exploit “triggers” plus session tear down and acknowledgement packets are transmitted via the DUT, but without first establishing a valid session with the target server.

Test ID

Test Description

4.1.1

128 Byte Packets

Maximum 740,000 Packets Per Second. Repeated with 250Mbps, 500Mbps, 750Mbps and 1000Mbps of background traffic.

4.1.2

256 Byte Packets

Maximum 370,000 Packets Per Second. Repeated with 250Mbps, 500Mbps, 750Mbps and 1000Mbps of background traffic

4.1.3

512 Byte Packets

Maximum 185,000 Packets Per Second. Repeated with 250Mbps, 500Mbps, 750Mbps and 1000Mbps of background traffic.

This test provides a reasonable indication of the ability of a device to process packets from the wire on an “average” network.

4.1.4

1024 Byte Packets

Maximum 92,500 Packets Per Second. Repeated with 250Mbps, 500Mbps, 750Mbps and 1000Mbps of background traffic.

This test has been included mainly to demonstrate how easy it is to achieve good results using large packets - beware of test results that only quote performance figures using similar packet sizes.

Test ID

Test Description

4.2.1

Maximum TCP Connections Per Second (8 byte Response)

This test is designed to determine the maximum TCP connection rate of the DUT with an 8 byte HTTP response size. The response size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header. An 8 byte response size is designed to provide a theoretical maximum connections per second rate.

Client and server are using HTTP 1.0 without keep alive, and the client will open a TCP connection, send one HTTP request, and close the connection. This ensures that all TCP connections are closed immediately the request is satisfied, thus any concurrent TCP connections will be caused purely as a result of latency within the DUT. Load is increased until one or more of the breaking points defined in Test 3.1 is reached. Avalanche load specification is Connections Per Second.

4.2.2

Maximum TCP Connections Per Second (1Kbyte Response)

This test is designed to determine the maximum TCP connection rate of the DUT with a 1000 byte HTTP response size. The object size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header.

Client and server are using HTTP 1.0 without keep alive, and the client will open a TCP connection, send one HTTP request, and close the connection. This ensures that all TCP connections are closed immediately the request is satisfied, thus any concurrent TCP connections will be caused purely as a result of latency within the DUT. Load is increased until one or more of the breaking points defined in Test 3.1 is reached. Avalanche load specification is Connections Per Second.

4.2.3

Maximum TCP Connections Per Second (5Kbyte Response)

This test is designed to determine the maximum TCP connection rate of the DUT with a 4700 byte HTTP response size. The object size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header. A 5Kbyte response size is designed to provide an indication of connections per second rate on a typical network.

Client and server are using HTTP 1.0 without keep alive, and the client will open a TCP connection, send one HTTP request, and close the connection. This ensures that all TCP connections are closed immediately the request is satisfied, thus any concurrent TCP connections will be caused purely as a result of latency within the DUT. Load is increased until one or more of the breaking points defined in Test 3.1 is reached. Avalanche load specification is Connections Per Second.

4.2.4

Maximum HTTP Transactions Per Second (8 Byte Response)

This test is designed to determine the maximum HTTP transaction rate of the DUT with an 8 byte HTTP response size. The object size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header. An 8 byte response size is designed to provide a theoretical maximum connections per second rate.

Client and server are using HTTP 1.1 with persistence, and the client will open a TCP connection, send ten HTTP requests, and close the connection. This ensures that TCP connections remain open until all ten HTTP transactions are complete, thus eliminating the maximum connection per second rate as a bottleneck (one TCP connection = 10 HTTP transactions). Load is increased until one or more of the breaking points defined in Test 3.1 is reached. Avalanche load specification is Transactions Per Second.

4.2.5

Maximum HTTP Transactions Per Second (1Kbyte Response)

This test is designed to determine the maximum HTTP transaction rate of the DUT with a 1000 byte HTTP response size. The object size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header.

Client and server are using HTTP 1.1 with persistence, and the client will open a TCP connection, send ten HTTP requests, and close the connection. This ensures that TCP connections remain open until all ten HTTP transactions are complete, thus eliminating the maximum connection per second rate as a bottleneck (one TCP connection = 10 HTTP transactions). Load is increased until one or more of the breaking points defined in Test 3.1 is reached. Avalanche load specification is Transactions Per Second.

Maximum HTTP Transactions Per Second (5Kbyte Response)

This test is designed to determine the maximum HTTP transaction rate of the DUT with a 4700 byte HTTP response size. The object size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header. A 5Kbyte response size is designed to provide an indication of connections per second rate on a typical network.

Client and server are using HTTP 1.1 with persistence, and the client will open a TCP connection, send ten HTTP requests, and close the connection. This ensures that TCP connections remain open until all ten HTTP transactions are complete, thus eliminating the maximum connection per second rate as a bottleneck (one TCP connection = 10 HTTP transactions). Load is increased until one or more of the breaking points defined in Test 3.1 is reached. Avalanche load specification is Transactions Per Second.

Test ID

4.3.1

44Kbyte Response

Max 2,500 new connections per second - average packet size 1000 bytes - maximum 120,000 packets per second. Repeated with 250Mbps, 500Mbps, 750Mbps and 1Gbps of background traffic. With relatively low connection rates and large packet sizes, we expect all sensors to perform well throughout this test

4.3.2

21Kbyte Response

Max 5,000 new connections per second - average packet size 540 bytes - maximum 225,000 packets per second. Repeated with 250Mbps, 500Mbps, 750Mbps and 1Gbps of background traffic. With average connection rates and average packet sizes, this is a good approximation of a real-world production network, and we expect all sensors to perform well throughout this test.

4.3.3

11Kbyte Response

Max 10,000 new connections per second - average packet size 440 bytes - maximum 275,000 packets per second. Repeated with 250Mbps, 500Mbps, 750Mbps and 1Gbps of background traffic. With average packet sizes coupled with very high connection rates, this is a strenuous test for any sensor, and represents a very heavily used production network.

4.3.4

5Kbyte Response

Max 20,000 new connections per second - average packet size 360 bytes - maximum 320,000 packets per second. Repeated with 250Mbps, 500Mbps, 750Mbps and 1Gbps of background traffic. With small packet sizes and extremely high connection rates this is an extreme test for any sensor. Not many sensors will perform well at all levels of this test

Test ID

4.4.1

21Kbyte Response With Delay

Max 5,000 new connections per second - average packet size 540 bytes - maximum 225,000 packets per second. Repeated with 250Mbps, 500Mbps, 750Mbps and 1Gbps of background traffic. 10 second transaction delay resulting in a maximum of 50,000 open connections during the test. With average connection rates and average packet sizes, this is a good approximation of a real-world production network, and we expect all sensors to perform well throughout this test.

4.4.2

11Kbyte Response With Delay

Max 10,000 new connections per second - average packet size 440 bytes - maximum 275,000 packets per second. Repeated with 250Mbps, 500Mbps, 750Mbps and 1Gbps of background traffic. 10 second transaction delay resulting in a maximum of 100,000 open connections during the test. With average packet sizes coupled with very high connection rates, this is a strenuous test for any sensor, and represents a very heavily used production network.

Test ID

4.5.1

“Real World” Protocol Mix

Traffic is played across the DUT comprising the following protocol mix:

60% HTTP
20% SMTP
10% POP3
5% FTP
5% DNS

Max 5,000 new connections per second - average packet size 540 bytes - maximum 200,000 packets per second. Repeated with 250Mbps, 500Mbps, 750Mbps and 1Gbps of background traffic. Maximum of 1,000 open connections during the test.

With lower connection rates, average packets sizes, and a common protocol mix comprising protocols which all require inspection by the IPS engine, this is a good approximation of a heavily-used production network. We expect all sensors to perform well throughout this test.

Test ID

5.1.1

Latency With No Background Traffic

SmartFlow traffic is passed across the infrastructure switches and through one in-line port pair of the DUT (the latency of the basic infrastructure is known and is constant throughout the tests). The packet loss and average latency are recorded at each packet size (128, 256, 512 and 1000 bytes) and each load level from 250Mbps to 1Gbps (in 250Mbps steps)

5.1.2

Latency With Background Traffic Load

The Avalanche devices are configured to generate HTTP traffic through each in-line port pair of the DUT up to 50 per cent of the maximum rated bandwidth of the DUT - maximum 2Gbps (500Mbps per port pair) - maximum 10,000 new connections per second - average packet size 540 bytes - maximum 450,000 packets per second.

A trickle (less than 1Mbps) of bi-directional SmartFlow traffic at various packet sizes (128, 256, 512 and 1000 bytes) is then passed across the infrastructure switches and through the DUT, and the packet loss and average latency are recorded.

5.1.3

Latency Under DDOS Attack

The Spirent WebSuite software is used to generate a fixed load of DOS/DDOS traffic of 10 per cent of the maximum rated bandwidth (maximum 100Mbps) on any one of the in-line port pairs of the DUT. A trickle (less than 1Mbps) of bi-directional SmartFlow traffic at various packet (128, 256, 512 and 1000 bytes) is then passed across the infrastructure switches and through the DUT, and the packet loss and average latency are recorded. The DUT should be configured to detect/block/mitigate the DOS attack by the most efficient method available.

Test ID

5.2.1

Web Response With No Background Traffic (21Kbyte Response)

The Avalanche devices are configured to generate HTTP traffic through the DUT up to 50 per cent of the maximum rated bandwidth of the DUT - maximum 500Mbps - maximum 2,500 new connections per second - average packet size 540 bytes - maximum 112,500 packets per second.

The minimum, maximum and average page response times and number of failed connections for each port pair (plus overall DUT performance) are recorded by Avalanche to provide an indication of the expected response times under normal traffic conditions.

5.2.2

Web Response Under DDOS Attack (21Kbyte Response)

The Avalanche devices are configured to generate HTTP traffic through the DUT as for Test 5.2.1. The Spirent WebSuite software is then used to generate DOS/DDOS traffic up to 10 per cent of the maximum rated bandwidth (maximum 100Mbps) of the DUT.

The minimum, maximum and average page response times and number of failed connections are recorded by Avalanche to provide an indication of the expected response times when the device is under attack.

Test ID

Test Description

6.1.1

Blocking Under Extended Attack

For this test, the external interface of the device is exposed to a constant stream of exploits over an extended period of time. The device is configured to block and alert, and thus this test provides an indication the effectiveness of both the blocking and alert handling mechanisms.

A continuous stream of exploits mixed with some legitimate sessions is transmitted through the device at a maximum of 100Mbps (max 50,000 packets per second, average packet sizes in the range of 120-350 bytes) for 8 hours with no additional background traffic. This is not intended as a stress test in terms of traffic load - merely a reliability test in terms of consistency of blocking performance.

The device is expected to remain operational and stable throughout this test, and to block 100 per cent of recognisable exploits, raising an alert for each. If any recognisable exploits are passed - caused by either the volume of traffic or the sensor failing open for any reason - this will result in a FAIL.

6.1.2

Passing Legitimate Traffic Under Extended Attack: This test is identical to 6.1.1, where we expose the external interface of the device to a constant stream of alerts over an extended period of time.

The device is expected to remain operational and stable throughout this test, and to pass most/all of the legitimate traffic. If an excessive amount of legitimate traffic is blocked throughout this test - caused by either the volume of traffic or the sensor failing closed for any reason - this will result in a FAIL.

ISIC/ESIC/TCPSIC/UDPSIC/ICMPSIC

This test attempts to stress the protocol stack of the device under test by exposing it to traffic from the ISIC test tool. The ISIC test tool host is connected to the switch on the external interfaces of the DUT, and the ISIC target to the switch on the internal interfaces. ISIC traffic is transmitted through a single in-line port pair of the DUT and the effects noted.

Traffic load is a maximum of 350Mbps and 60,000 packets per second (average packet size is 690 bytes). Results are presented as a simple PASS/FAIL - the device is expected to remain operational and capable of detecting and blocking exploits throughout the test to attain a PASS.

Policy Push

The Avalanche devices are configured to generate HTTP traffic through each in-line port pair of the DUT up to 50 per cent of the maximum rated bandwidth of the DUT - maximum 500Mbps - maximum 2,500 new connections per second - average packet size 540 bytes - maximum 112,500 packets per second.

A new policy is pushed to the DUT during the test, and the minimum, maximum and average page response times and number of failed connections are recorded by Avalanche. These results can be compared with Test 5.2.1 to provide an indication of the effect of pushing policies on legitimate traffic.

6.1.5

�

Power Fail

The Avalanche devices are configured to generate HTTP traffic through each in-line port pair of the DUT up to 50 per cent of the maximum rated bandwidth of the DUT - maximum 500Mbps - maximum 2,500 new connections per second - average packet size 540 bytes - maximum 112,500 packets per second.

Power to the DUT is cut during the test, and the minimum, maximum and average page response times and number of failed connections are recorded by Avalanche. If the device is configured to fail open, there should be minimal loss of legitimate sessions throughout the test (over and above the baseline loss expected through switch renegotiation). If the device is configured to fail closed, no traffic should be passed once power has been cut.

6.1.6

�

Redundancy

Does the DUT include multiple redundant critical components (fans, power supplies, hard drive, etc.) (YES/NO/OPTION)

Fail Open (Power Fail)

Does the DUT provide the ability to fail open with minimal/zero loss of legitimate traffic (either via built-in, or optional hardware bypass) following power failure (YES/NO/OPTION).

Fail Open (Resource Issues)

Does the DUT provide the ability to pass all traffic when resources are exhausted or it is no longer possible to analyse traffic for any reason (i.e. packet rate exceeds device capabilities)

Fail Closed (Power Fail)

Does the DUT provide the ability to fail closed following power failure (YES/NO/OPTION)

Fail Closed (Resource Issues)

Does the DUT provide the ability to block all traffic when resources are exhausted or it is no longer possible to analyse traffic for any reason (i.e. packet rate exceeds device capabilities)

6.1.11

�

High Availability (HA) Option (Stateful)

Is an HA option available for this device, providing fully stateful active-active or active-passive failover between devices (YES/NO)

6.1.12

�

High Availability (HA) Option (Non-stateful)

Is an HA option available for this device, providing any form of failover between devices where existing connections may be lost during failover (YES/NO)

Persistence Of Data

The DUT should retain all configuration data, policy data and locally logged data once restored to operation following power failure.

IPV6

The DUT should be capable of detecting exploits over both IPV6 and IPV4.

Test ID

7.1.1

Open Ports Required

The vendor will list the open ports and active services on the management interface along with their use.

7.1.2

Open Ports Detected

The management port will be scanned to determine ports/services visible on the management interface. If any ports additional to those listed in Test 7.1.1 are discovered, this will result in an automatic FAIL.

7.1.3

7.1.4

ISIC/ESIC/TCPSIC/UDPSIC/ICMPSIC Detection

Are ISIC attacks detected by the DUT even though targeted at the management port (YES/NO)

Test ID

7.2.1

Transparent Mode

The DUT must be capable of running in transparent bridge mode, with no IP address assigned to detection ports. Detection ports should ignore all direct connection attempts.

7.2.2

Management Port

The DUT should feature a dedicated management port, separate from detection ports. Although this is the preferred configuration, lack of a management port (requiring DUT to be managed via one of the detection ports) will not be cause for failure providing management connection and communication is securely encrypted.

7.2.3

Management Protocol

Connection from management console to DUT should be protected by a minimum of a user name/password combination or multi-factor authentication system, and all communications between should be securely encrypted.

Where a three-tier management architecture is employed, all communication between console and management server(s), and between management server(s) and sensor(s) should be securely encrypted.

7.2.4

Authentication

Access to management console should be protected by a granular user authentication system which allows for separation of read only and read-write access, preventing users who require reporting access only from modifying device parameters, etc.

No access to administrative functions should be permitted (using either direct or centralised administration capabilities) without proper authentication.

7.2.5

Enterprise Authentication

Access to management console should be protected by a granular user authentication system which allows for restriction of individual users to specific devices, ports, reports, alerts and security policies. Authenticated users should be unable to access devices/ports/policies/alerts/reports/etc. restricted to other users of the system.

7.2.6

Direct Device Management (Optional)

Direct access to the DUT should be provided (either via command line or Web interface) for single-device management.

7.2.7

Centralised Device Management

A centralised management system should be provided to manage one or more sensors from a single point, including centralised device configuration, policy definition, alert handling and reporting for all sensors under the control of the management system. This should be scalable to large numbers of sensors.

7.2.8

Pass-Through Mode (Optional)

It should be possible to place the DUT into a mode whereby all traffic is allowed to pass through the device, but data will be logged according to the policy in place at the time (thus, the DUT will log alerts and state whether the packets would have been dropped, session terminated, etc., but without enforcing those actions on the traffic processed).

This should be via a single system-wide operation via the management console or DUT command line (i.e. it is not permitted to achieve this by requiring that all BLOCK signatures be amended to LOG ONLY, or by switching policies - it must be achieved without affecting the current policy in force).

7.2.9

Signature Update

The vendor should demonstrate access to a vulnerability research capability (either in-house or via a recognised third-party) which is able to provide timely and accurate signature updates at regular intervals.

7.2.10

Secure Sensor Registration

Initial registration of sensor to central management console should be in a fully secure manner (it is permitted to offer a less secure/rapid option, but this should not be the default).

Documentation

Adequate documentation should be provided for both installation, and day-to-day management.

Test ID

Test Description

7.3.1

Sensor Configuration

The management system should provide the means to configure one or more sensors from a central location, assigning signatures, sensor settings, etc.

7.3.2

Policy Definition

The management system should provide the means to define and save multiple security policies, consisting of:

General sensor configuration
System-wide parameters
Signatures enabled/disabled
Actions to take when malicious traffic discovered

7.3.3

Recommended Settings

The vendor should provide a default policy or suite of recommended settings which comprises the optimum configuration for a typical network (including which signatures are enabled/disabled, which are enabled in blocking mode, required actions, etc.)

7.3.4

Custom Attack Signatures

It should be possible for the administrator to be able to define custom attack signatures for use in standard policies

7.3.5

Bulk Operations

It should be possible to search quickly and easily for individual signatures or groups/classes of signatures, and subsequently to apply one or more operations to an entire group in a single operation (for example, to enable or disable a group of signatures, or to switch a group from block mode to log mode, etc.)

7.3.6

Granularity

The DUT should be capable of blocking or creating exceptions based on IP address, application, protocol, VLAN tag, etc. (i.e. never block HTTP traffic between two specific IP addresses, always block FTP traffic to one specific IP address, etc.).

7.3.7

Policy Association

Once policies have been defined, it should be possible to associate them with specific devices or groups of devices.

7.3.8

Inheritance

It should be possible to create groups and sub-groups of devices such that sub-groups can inherit certain aspects of configuration and policy definition from parent groups.

7.3.9

Virtualisation

Once policies have been defined, it should be possible to associate them with specific “virtual” devices or groups of devices, comprising an entire DUT, individual ports, port groups, IP address range, subnet or VLAN.

7.3.10

Policy Deployment

Once policies have been defined, it should be possible to distribute them to the appropriate device(s), virtual device(s), or groups of devices in a single operation.

7.3.11

Policy Auditing

All changes to policies should be logged centrally. Log data should include at a minimum the date/time the changes were made, and the identity of the user who made them. If possible (OPTIONAL) the system should record the actual changes.

Policy Version Control

All changes to policies should be recorded by saving a version of the policy before each change. It should be possible to roll-back to a previous version of any policy via a single operation.

Test ID

Test Description

7.4.1

Required Log Events

The DUT should record log entries for the following events:

Detection of malicious traffic
Termination of a session
Successful authentication by administrator
Unsuccessful authentication by administrator
Policy changed
Policy deployed
Hardware failure
Power cycle

7.4.2

Log Location (Optional)

The log events should be logged on the DUT initially, in a secure manner, and subsequently transmitted to a central repository for permanent storage.

7.4.3

Communication Interruption

Where communications between sensor and console/management server are interrupted, storage capacity on the DUT should be sufficient to hold one week’s worth of log data on a typical network. If it is not possible to restore communication in a timely manner, once the local logs are full, the DUT should either (1) continue passing traffic and overwrite oldest log entries, or (2) stop passing traffic. This option should be configurable by the administrator.

7.4.4

Log Flooding

Mechanisms should be in place (aggregation) to prevent the DUT from flooding the management server/console with too many events of the same type in a short interval. However, it should be possible to disable aggregation/flood protection completely for testing purposes to ensure NSS can see every individual alert.

7.4.5

Alerts

The DUT should record log entries each time it detects malicious traffic. At a minimum (depending on protocol), these log entries should contain:

Unique event ID
Date and time of event
Device ID (includes sensor ID, port ID, etc.)
Direction of traffic (physical/logical source and destination interfaces)
Detection engine which raised the alert (OPTIONAL)
Source IP address
Source port/service (where applicable)
Destination IP address
Destination port/service (where applicable)
ICMP message type and code (where applicable)
Protocol
Unique signature ID
Human-readable description of the event/exploit
CVE reference, Bugtraq ID, or other non-vendor-specific identifier
Action taken by the DUT (block, log, etc.)

7.4.6

Alert Accuracy

The DUT should record log entries which are accurate and human readable without having to use additional reference material. The DUT should attempt to minimise the number of alerts raised for a single event wherever possible.

7.4.7

Centralised Alerts

No matter how many sensors are installed, all alerts should be delivered to, and handled by, a single, central, management console. From that console, it should be possible to view all alerts globally, or select alerts from individual devices (logical or physical).

Alert Delivery Mechanism

At a minimum, the DUT should be able to deliver alerts in a timely manner to a central database for permanent storage, central console for a real-time display, and SMTP server for e-mail alerts.

7.4.9

Alert Actions (Mandatory)

On detecting malicious traffic, the DUT should be able to perform the following actions at a minimum:

Ignore
Log only
Drop packet (no reset)
Drop session (no reset)
E-mail administrator

7.4.10

Alert Actions (Optional)

On detecting malicious traffic, the DUT may optionally be able to perform the following actions:

Send TCP reset (or ICMP redirect) to source only
Send TCP reset (or ICMP redirect) to destination only
Send TCP reset (or ICMP redirect) to both source and destination
Reconfigure external firewall
Reconfigure switch to isolate/quarantine offending port
Page administrator

7.4.11

Forensic Analysis

The DUT should provide the ability to capture individual packets, a range of packets, or an entire session where required (globally, or on a rule-by-rule basis)

7.4.12

Summarise Alerts

The central console should provide the ability to select a particular piece of data from an alert and summarise on that data field (i.e. select a source IP address and view all alerts for that source IP). Alternatively, it should be possible to construct data filters manually in a search form and summarise on the specified search criteria. The preferred scenario is to offer both of these options.

7.4.13

View Alert Detail

The central console should provide the ability to select an individual alert and view the following information at a minimum:

Detailed alert data (including all data mentioned in Test 7.4.5)
Detailed exploit data (description of the exploit research)
Signature/rule
Remediation data/preventative action

7.4.14

View Policy

Having selected an alert, the system should provide the ability to access directly the policy and rule which triggered the event in order to view and/or modify the policy for further fine tuning.

7.4.15

View Packet Contents (Optional)

The central console should provide the ability to select an individual alert and view the contents of the trigger packet or context data for the exploit.

7.4.16

Alert Suppression

The central console should provide the ability to create exception filters based on alert data to eliminate further alerts which match the specified criteria (i.e. same alert ID from same source IP). This does not disable detection, logging or blocking, but merely excludes alerts from the console display.

7.4.17

Correlation (Automatic)

The system should provide the means to infer connections between multiple alerts and group them together as incidents automatically.

7.4.18

Correlation (Manual)

The system should provide the means for the administrator to infer connections between multiple alerts and group them together as incidents manually.

Incident Workflow

The system should provide the ability to annotate and track incidents to resolution.

Test ID

Test Description

7.5.1

Centralised Reports

No matter how many sensors are installed, the system should be capable of reporting on all alerts from a single, central, management console. From that console, it should be possible to report all alerts globally, or to report on alerts from individual devices (logical or physical).

7.5.2

Top Attacks

The system should provide a report listing the top N attacks in the previous hour, day, week, month, year, or custom date range.

7.5.3

Top Sources

The system should provide a report listing the top N source IPs from which attacks have been detected in the previous hour, day, week, month, year, or custom date range.

7.5.4

Top Targets

The system should provide a report listing the top N target IPs at which attacks have been launched in the previous hour, day, week, month, year, or custom date range.

7.5.5

Top Services

The system should provide a report listing the top N target ports/services at which attacks have been launched in the previous hour, day, week, month, year, or custom date range.

7.5.6

Top Protocols

The system should provide a report listing the top N protocols over which attacks have been launched in the previous hour, day, week, month, year, or custom date range.

7.5.7

Custom Reports

The report generator should provide the ability to construct complex data filters in a search form and summarise alerts on the specified search criteria.

7.5.8

Saved Reports

Having defined a custom report filter, it should be possible to save it for subsequent recall.

7.5.9

Scheduled Reports

It should be possible to schedule saved reports for regular unattended runs. The output should be saved as HTML or PDF at a minimum. It should optionally be possible to publish to a central FTP/Web server, and/or e-mail reports to specified recipients.

Log File Maintenance

The system should provide for automatic rotation of log files, archiving, restoring from archive, and reporting from archived logs.