Bug bounties not a silver bullet, Katie Moussouris warns

Triage difficult to staff consistently

Reiterating that bug bounties are not a magical fix, Moussouris said that while bug bounty companies promise that they will clean the feed of bugs and mimic the professionalism, safety and risk aversion of a traditional penetration test, they are missing the fact that triage is among the most difficult jobs to staff consistently in information security.

“And even with all the triage support in the world, sometimes there are just too many bugs and the backend infrastructure just can’t handle it,” she said, that one of the reasons Microsoft created its bug bounty programme was because of large volume of bug reports it was getting.

And while the bug bounty programme was able to shape and focus the attention of the hacker community, the volume of reports was just as great after the programme was reduced as it was before.

Moussouris warned that by offering cash rewards for vulnerabilities, bug bounty programmes have resulted in some “perverse incentives”.

“In creating this market we have gotten carried away with incentives instead of looking at the full picture,” she told Computer Weekly.

“Bug bounties have a role to play. As we did at Microsoft, bug bounties that focus incentives on the exact areas you want to discover security issues is a good thing in general, but it is a bad thing if you don’t understand who might have these skills and how you might want to use them in terms of your overall workforce. As a result, it is a bad idea just to seek out individuals with particular skills using high six-figure values,” she said.

Bug bounty payouts

In the past year, Moussouris said there has been a sharp increase in bug bounty payouts. “Top payouts at a quarter of a million dollars for – certainly rare type of finds – but the problem is that you need those people with the rare skills wanting to apply to work inside your companies, not holding out and going for a quarter of a million dollars after the fact because by nature it is too late, especially when it comes to architecture issues.

“We don’t want the next Meltdown and Spectre to be found through bug bounties. We want it to be prevented inside of those chip companies.”

This has a huge potential to damage severely the skills pipeline for organisations that need to be thinking about the architecture for security for the future, said Moussouris.

“Hiring managers inside those companies are worried that with $250,000-incentives on offer, with only a six-month waiting period before former employees of the company can collect them, that they will in fact decimate and gut their own talent pipeine,” she said.

“We have to think that this ‘perverse incentive’ is a real thing, and the offence will always be able to outbid the defence market, so we must not focus on price.”

Virtue signalling

Another concern Moussouris highlighted in her presentation was that paying for bugs has become a form of “virtue signalling” for taking security seriously. “As we look at the future, I would worry about the term ‘bug bounty’ replacing ‘penetration testing’ as we saw in the Google search stats,” she said.

Most bug bounties are good at finding “low-hanging” fruit, said Moussouris, with the majority of bug bounties being paid out for things like cross site scripting vulnerabilities, which suggests very little progress has been made in the past 20 years in secure development to prevent this type of vulnerability.

“One of the things I said to the senate, to give them advice about the future, was that we needed to create more secure coders instead of cranking out people who write the same bugs over and over again,” she said, adding that bug bounties are mainly finding these kinds of bugs that could be found with commercially available tools and even free tools, and not fewer vulnerabilities of greater complexity that teach defenders something new, which was the direction the industry as a whole was supposed to go in.

“When organisations think about starting a bug bounty programme, they are typically not thinking it through. They are not thinking how to prevent vulnerabilities in the place, how they can enhance internal teams to help identify and prevent these issues, and how they can attract more talent to come work inside as opposed to putting out incentives and hoping for the best from the outside.”

In her work with the UK’s National Cyber Security Centre (NCSC), Moussouris said the focus is on building capacity within organisations. “We don’t work on this idea that eventually you are going to offer cash. It’s just not necessary, as proved by the fact that Microsoft’s bug bounty did not increase the number of bug reports they received, they were just more focussed.”

The truth, she said, is that bug bounties are not a replacement for penetration testing, and she warned against “over romanticising” of zero day vulnerabilities and of bug hunting in general, without regard to the entire workforce that needs to be prepared for the future.

“Remember, there needs to be an army behind all security companies to process this information and to be able to do something with it, and once the patches are out, every IT organisation still has to deploy them.”

Exceptional tooling

According to Moussouris, research has shown that one of the most effective things organisations can do is to close the skills gap between defence and offence using “exceptional tooling” that defenders need to assess, at whatever volume, the vulnerabilities that are coming in.

Any organisations that run bug bounty programmes, she said, should think about ways of focussing the attention of the hacking community and putting thresholds in place to cap the number of reports they receive to reduce the signal to noise ratio and ensure they can handle the number of reports they receive.

In closing, Moussouris reiterated the warning that if industry and governments continue to over romanticise bug hunting for cash rewards, they will decimate their workforce of the future. She also reiterated the need to ensure that coders are taught secure development.

“We need to be wary of initiatives that put forward bug hunting as the answer, and instead start looking at the labour market as a whole. And as we look to the future, we need to look for opportunities not to just incentivise destruction, but to build more resilient creators and to support and celebrate the maintainers, instead of calling it ‘the worst job in science’.”