Infosec 2013: University research challenges reliability of IPS
Research shows intrusion prevention systems (IPS) are not as effective at detecting malicious activity as some might think
Research shows that intrusion prevention systems (IPSs) are not as effective at detecting malicious activity as many organisations that have deployed them may think.
Almost 78% of IPS systems tested by the University of South Wales failed to detect 34%-49% of attacks that used advanced evasion techniques (AETs) to hide attacks exploiting a well-known vulnerability.
An AET typically combines IPS evasion techniques to create more stealthy attacks that are dynamic, delivered over several layers of a network and harder to detect and block.
Malicious payloads
By breaking up attack payloads into multiple packets across multiple network layers, most IPSs cannot recognise malicious payloads.
AETs are a relatively new hacking techniques that enable malware, viruses, worms and other security threats to bypass next-generation firewalls and IPSs.
A research report launched at Infosecurity Europe 2013 in London presents the findings of an experiment that tested a number of evasion techniques against a set of well-known IPSs.
Read more about AETs
- Stonesoft releases AET defence system and free test tool
- Stonesoft urges channel to exploit free AET tool
- New batch of IDS, IPS evasion techniques are hitting their targets
- Stonesoft discovers 124 new advanced evasion techniques
- Evasion threat to critical systems goes ignored, says Stonesoft
The experiment made use of the open source Evader tool created by security firm Stonesoft to generate the attacks and their evasions.
The IPSs used in the experiment were drawn from Sourcefire, IBM, PaloAlto, Fortigate, McAfee, Checkpoint, Juniper, Cisco and Stonesoft.
The IPS were all up-to-date and configured, using a best configuration scenario. This ensured all attack attempts against two well-known vulnerabilities were blocked if no AETs were used.
However, when AETs were used against the first vulnerability (CVE-2008-4250) between 0.072% and 6.669% of attacks were successful.
Success rate
Most IPS tools detected 98.5% of the attacks with 1.5% or less of the attacks using evasion techniques being successful.
“While 1.5% may not sound like a lot, it certainly would matter if a multi-million pound piece of IP were at stake,” said Andrew Blyth, co-author of the report and professor of information security at the University of South Wales.
The first experiment used 2,759 attack combinations, which means a 6.669% failure rate equates to 184 successful attacks and 1.5% to around 41. Against the best performing IPS, two attacks were successful.
The best two systems during testing were found to be Cisco, with a success rate for detection of 99.928%; and Stonesoft, with a 99.565% rate.
Results were normalised to allow for the different speeds of the IPS appliances used in the experiment.
| IPS | CVE-2008-4250 Successful evasion(s) | Successful evasion(s) % | Detection rate % |
| Sourcefire | 184 | 6.669% | 93.331% |
| IBM | 41 | 1.486% | 98.514% |
| Palo Alto | 38 | 1.377% | 98.623% |
| Fortigate | 36 | 1.305% | 98.695% |
| McAfee | 30 | 1.087% | 98.913% |
| Checkpoint | 25 | 0.906% | 99.094% |
| Juniper | 12 | 0.435% | 99.565% |
| Stonesoft | 12 | 0.435% | 99.565% |
| Cisco | 2 | 0.072% | 99.928% |
The most dramatic test results emerged with the second vulnerability (CVE-2004-1315) when between 0.265% and 49.431% of AETs were successful, which equates to between seven and 1,304.
Only two suppliers achieved a detection rate of 99% or higher. These were Fortigate and Stonesoft, with a detection rate of 99.242% and 99.735%, respectively.
Comparing the results of experiment one and two show that IPS tools are generally better at detecting standard buffer overrun type attacks than attacks aimed at web services, said Blyth.
Across both experiments, Fortigate – with 98.695% and 99.242% – and Stonesoft – with 99.565% and 99.735% – generally fared best overall, scoring high detection rates.
| IPS | CVE-2004-1315 Successful evasion(s) | Successful evasion(s) % | Detection rate % |
| McAfee | 1304 | 49.431% | 50.569% |
| Juniper | 1303 | 49.393% | 50.607% |
| Palo Alto | 1294 | 49.052% | 50.948% |
| Cisco | 1292 | 48.976% | 51.024% |
| Checkpoint | 1132 | 42.911% | 57.089% |
| Sourcefire | 997 | 37.794% | 62.206% |
| IBM | 900 | 34.117% | 65.883% |
| Fortigate | 20 | 0.758% | 99.242% |
| Stonesoft | 7 | 0.265% | 99.735% |
Cause for concern
The findings provide some cause for concern and should be a warning to organisations that rely on simple and/or outdated implementations of IPS, the research report said.
The report notes that the experiment shows that some of the IPS installed and available offer limited protection against attacks using advanced evasion techniques.
The key requirement of a successful system is to provide the broadest possible protection for a network. Based on the experiments, Stonesoft’s IPS offers the best protection, the report said.
However, the experiments show that using multiple IPS tools in combination is likely to provide the best and most comprehensive protection as none of the systems achieved a 100% detection rate, said Blyth.
“Having multiple protection devices on one network, whether it is firewalls, IPS or routers, is a good network defence principal to adhere to,” he said.
Defence in depth
When it comes to IPS, defence in depth is a good strategy because the experiments show that most IPS tools have a long way to go, particularly in terms of securing web applications, said Blyth.
If organisations were to buy only one IPS, then it should be Stonesoft, but if they are going for defence in depth, the three they should be looking at are Cisco, Fortigate and Stonesoft, the top performers across both experiments, he said.
Blyth said Evader is a standard open source tool that can be downloaded by any organisation and used to replicate the experiments.
“IPS suppliers cannot be held to account in terms of zero-day exploits, but they have a due-diligence requirement in testing their products to ensure that what they are doing is the best in the light of public open source knowledge as tested by Evader,” he said.
