Add human touch to black box tests

Vulnerability testing software promises much, but what can it actually deliver? The answer depends on how much human expertise you can put in as well

Web products continue to be a weak link in security. I could spend many hours discussing why this is so and proposing solutions, but I want instead to talk about the tools we are being sold to assess online product risk and the dangers of relying on them to produce plans of remediation.

"Black box" is a term used for testing the security of a web product when there is no knowledge about the source code or architecture apart from what can be deduced by going online and performing an analysis. This is the hacker's view.

Most vulnerability testing software takes this view and has simplified the process of performing a black box test. In fact, it has been simplified so much that what used to be a specialist task often requires no more than entering the IP address of the website to be tested and hitting the scan button.

To add value to their products, many suppliers of vulnerability testing software include pre-formatted compliance reporting. So, for example, if you want to assess your web product for compliance with the Payment Card Industry (PCI) scheme, you select the relevant test plan, scan, and print out the resulting reporting.

The problem is that this is not enough. At best, scanning software will uncover some vulnerability that may or may not be important, serious, or relevant. The issue becomes whether your organisation has the skills to verify the report and weed out the myriad of false positives. At worst, the compliance report will highlight areas of non-compliance that are simply inaccurate.

Black box testing is a learned skill. It takes tenacity and an in-depth knowledge about how web products work beyond whizzing robot-like through all the web links an application contains. It requires a human mind to analyse responses to web requests, not a software response to an error message to advise you that you have a critical non-PCI compliant security hole that is really nothing more than an unhandled error that the software did not recognise.

The opposite case is just as important. If you have a web product that is processing credit card payments and needs to be PCI-compliant, there is a very real danger that scanning the product using automated vulnerability tools could result in a "compliant" status when there are serious issues present that should have been identified.

This is even true if you are using a third party supplier to perform the scanning work - if their assessment is based solely on the results of an automated scan, you cannot rely on the report as an accurate assessment of security status.

If the product is important enough for you to be considering spending the thousands of pounds that the security software suppliers charge for their products, you should be doing the job properly. This entails a thorough internal end-to-end review that covers the full product lifecycle.

There is a clear correlation between a product that lacks defined security requirements and application development standards and its overall security status. Web products have become extremely complex, and hitting the scan button is not enough when it comes to assessing risk and taking into account the requirements placed on businesses to comply with legislation and protect critical data.

The best automated black box test will not review the password policy and will not be able to determine whether good encryption has been used. It will also not identify poorly implemented database connectivity or the fact that none of the developers understand what is meant by the term "SQL injection".

A black box test is also out of date the moment a new change gets put into production without due consideration being given to the security implications of doing so.

This last point is the most important of all because it means that to be secure we go right back to looking at the reasons why web products end up being insecure in the first place. And that is quite another story.

Stuart King is a senior information security practitioner at Reed Elsevier specialising in addressing risks associated with online products

Read more on Antivirus, firewall and IDS products