Security Zone: Testing standards at a crossroads - do changes in the Anti-Malware Testing Standards Organisation reflect better practice in antivirus testing?
Which group is more trustworthy: anti-malware suppliers or anti-malware product testers? David Harley, CITP FBCS CISSP, ESET senior research fellow and a director of AMTSO, looks at both sides.
Which group is more trustworthy: anti-malware suppliers or anti-malware product testers? Three or four years ago, nearly anyone outside the AV industry was likely to assume that all tests were accurate and that supplier complaints were hype or "supplier whining", writes David Harley, CITP FBCS CISSP, ESET senior research fellow and a director of AMTSO.
Then came AMTSO (the Anti-Malware Testing Standards Organization). AMTSO represented something unusual: suppliers and testers joining forces in an attempt to raise the general standard of testing.
Well, not that unusual: mainstream testers and suppliers have always had a symbiotic relationship. The main testers and suppliers have long shared samples and information at many levels. So while each represents a vested interest (we all have to make a living), they were nevertheless able to agree that dynamic testing of web threats draws a better picture of a scanner's performance than static testing (by simply running a scanner against a folder full of malware samples).
In principle, that is. But not every test that claims to be testing dynamically is a good test. Some have actually found that symbios is suspicious, but I am not sure they should.
Testers and suppliers still represent different vested interests, which often puts them into conflict. Yet somehow they still manage to sing (some of the time) from the same hymn sheet, demonstrating a mutual interest in the welfare of the customer as well as their own bottom lines. That is a significant achievement in itself, though the fact that they have also managed to generate some useful guidelines for testers is a pretty substantial bonus.
Antivirus testing for marketing's sake?
But there is a fly in the ointment. The emphasis in those papers is, generally, on what testers should be doing, and that can be - and sometimes is - seen as a supplier majority trying to enforce standardised testing models that suit the industry's marketing models rather than the customer's needs.
Is this really the case? The industry is united (generally) in wanting better testing, but each company also invests in marketing to differentiate between its own products and those of its competitors, in the hope of selling more units.
"Better" testing is not a way of making our lives easier; on the contrary, truly improved testing does not mean we no longer have to try, only that we can invest more in improving products and less in tweaking products in ways that make them look good in tests but are irrelevant in the real world.
Still, the documents are only guidelines, even where they contain quite detailed recommendations. Testers have time and resource constraints that often mean they have to compromise on how far they go to implement "best practice", though hopefully they always try to comply with the high level principles that offer an ethical framework rather than a set-in-stone template for implementation.
Supplier support for testers
The most positive outcome of a recent AMTSO meeting in Prague is what may be the most important document to come out of AMTSO in years. Rather than lecturing testers on how they should test, it suggests ways in which suppliers can make a tester's life easier by providing support for better logging, automation and so on.
AMTSO's recent history has not been a smooth progression - there have been mistakes, missed opportunities, misconceptions and misadventures. Yet this paper suggests to me a still-young organisation with some hope of maturing into a "really useful adult".
This was first published in May 2011