Security suppliers typically compete on performance or throughput of their appliances, but the fastest is not necessarily the most secure.
In fact, the speed a of security appliance may well be an indication of its lack of effectiveness, because speed is typically gained through other cuts.
Studies by NSS Labs have revealed that the default threat mitigation capabilities of seven top security appliances ranged from below 20% to 65%.
The best, by Sourcefire, achieved only 89% effectiveness after default settings were tuned by the supplier, according to the tests conducted in late 2009.
Defining competitive benchmarks
Suppliers shy away from competing on mitigation capabilties, says Anthony Haywood, chief technology officer at technology management firm Idappcom.
It is far too difficult to compete on this basis, he says, so they fall back instead on speed as a metric for comparison.
"The problem is that speed is often achieved through turning off rules and taking other shortcuts," says Haywood.
Other common speed-enhancing techniques include scanning only selected port numbers for threats and inspecting only the first 300bytes of data packets.
"This is dangerous as protections can be rendered powerless by using simple evasion techniques such as using a-typical port numbers for specific attacks," says Haywood.
Scanning only part of packets is also risky, he says, as the signatures used to identify attacks are often much deeper within packets.
Speed distracts from real issues
In buying new security products, organisations should force suppliers to focus on mitigation capabilities rather than speed, says Haywood.
If speed is an issue, he believes it is better to load-balance across several appliances than to compromise on the thoroughness of scanning.
The good news, he says, is that organisations can boost the effectiveness of existing investments simply by adding rules to cover any weaknesses.
"Most security suppliers make good guns, but they lack the effective ammunition of good rules," says Haywood.
He argues that any system can be boosted to near 100% effectiveness in detecting and mitigating threats, but only if organisations assess their security capabilities regularly.
"Continual auditing of systems is essential to prove the real defence capabilities and identify potential weaknesses," says Haywood.
Organisations can then apply additional, high-quality security rules, he says, to increase security capabilities and lower the number of false positives.
Haywood called on security suppliers to be more upfront about the shortcuts they take to improve performance, at an Infosecurity news conference ahead of Infosecurity Europe 2011 in London 19 to 21 April.
"We need more honesty from suppliers, who should make it absolutely clear to customers what they are doing and what level of real protection they are providing," he said.