Happy Thanksgiving (and more on vulnerability scanners)

Happy Thanksgiving day! Many of my colleagues are American and so today should be a quiet one on the email front – although you can bet there will always be at least one of them sneaking a message out on the blackberry whilst on a trip into the garage to get some more beer.

I mentioned a couple of days ago that I was looking at a new version of a well known web product vulnerability scanner. As previously reported I’m none too enthusiastic about the whole automated testing area – I think it’s lazy at best and inaccurate at worst. So, this week I’ve had a chance to do something I rarely get time for these days, and that’s roll up my sleeves and do some fun stuff with web site pen testing.

To say I’m an accomplished hacker would be very far from the truth: I know a few tricks learnt from people far more accomplished in this area than myself but just enough to be dangerous.

So, rubber gloves on, I fired up the interface of the scanning software and was pleasently surprised that it appears far more intuitive to use than the old version I was familiar with and contained a richer set of tests. But I’m afraid to say that it was still just as useless! OK – that’s an unfair statement, it did correctly identify an unencrypted view_state but the subsequent reports then had the nerve to tell me that this made the product fall out of compliance with SOX, PCI, the European Data Protection legislation, and probably also the rulings of the intergalactic council. This was wrong – the parameter in question contained no data of importance whatsoever, being as it was on the product’s public facing homepage. But for those organisations that place reliance on the reporting features and who don’t have the resources available to validate what they are presented with then this could be terrifying.

The problem is the same one faced by anything programmed to do an automatic job – you can set up a robot to press a button but it wont know whether the result of doing so is an egg sandwich or a nuclear explosion. In much the same way, the application scanner blithely does the task for which it has been programmed, pressing buttons and putting the output into various buckets, but without any context on which to base the sorting. Us, the consumers, are expected to know which bits to tip out of the buckets and what to discard as rubbish (and some of what we decide to throw away might actually be important).

So, my views have not been swayed by this experience and if I have a product that needs testing then I’d rather go to the professionals than trust the task to a button labelled “scan”.

What I will talk about in more detail at some point in the future is unit testing tools, because these really are useful.