TT: How have penetration tests changed over time?
Amit: They've changed heaps. There is a greater focus on the application side of the testing nowadays alongside the network / host / device and OS pen testing. Figures are showing most of the vulnerabilities are application focused and we're certainly seeing things like that - particularly in terms of the critical business threatening issues we find. Keep in mind while app level vulnerabilities are on the increase in terms of what is being reported, you can probably add many fold more that aren't. You don't generally see advisories for one-off systems/applications or smaller niche applications so do the sums. It's all happening on that front.
Pen tests have changed a lot over the last few years but there's still people and organisations out there that think running Nessus or some application testing tools is sufficient. Situations like that could almost make things worse by causing a false sense of security. Is it better to know you may have weaknesses but be aware of that and be alert, or believe you're okay and not worry?
Tools only find so much and application level tools probably even less. Pen testing is still a manual process that is supported by tools but driven by the expertise of the tester.
The skill of the tester is key.
Brian: Fundamentally, they haven't changed much at all. All the same concepts, just a little shift in technology and tools to better audit the target. There has been a big shift to application assessments, doing in depth evaluation of web sites and portals, looking for many classes of vulnerabilities in custom written applications.
Adam: Because it's more common for organisations to receive on-going vulnerability assessments using automated, yet fairly high-level tools, penetration testers have had to become smarter. It used to be whipper-snapper "consultants" were pushed out to client sites by big consulting firms. They'd turn up with laptops full of tools only to later deliver reports full of badly interpreted results and findings, providing no value at all.
It's a good thing organisations now understand that a light-touch "ethical hack" service provided by a generalist IT shop just isn't sufficient. They are now seeking deeper penetration-testing style security assessments that actually leave them with some form of security assurance.