peterzayda - stock.adobe.com
Debugging bug bounty programmes
Bug bounty programmes have recently become a popular method of vulnerability management, but poor programme management can lead to development teams becoming overwhelmed and bugs being missed
Bug bounty programmes have become a popular technique for code reviews; either in conjunction with, or instead of, penetration testing. Rather than an organisation relying on their own internal teams to review code for vulnerabilities, bug bounty programmes can be used to outsource the testing to external researchers.
The origin of bug bounty programmes can be traced back to 1988, when the hacker community testified before the US Senate about internet vulnerabilities. Since then, bug bounties have become a formal process, with client organisations being able to comply with the ISO standard for vulnerability disclosure (ISO/IEC 29147:2018).
Bug bounty programmes are similar to “wild west” wanted posters, in that financial rewards are paid to external parties, in this case for the successful identification of vulnerabilities in code. The advantage of using bug bounty programmes over an in-house penetration testing team is that the talent pool is much larger, bringing in a wider skill set, which allows for continuous reviews as the code is developed.
“We think it is really difficult to have 100% confidence that you have found everything, even with internal controls,” says Adrian Ludwig, chief information security officer for Atlassian. “This is why bug bounty programmes become a really valuable way to tap into resources that you could not [otherwise] bring to bear before you ship.”
One of the key advantages of bug bounty programmes is the diversity of thinking they provide. This offers benefits that are broader than just a range of skill-sets, as different cultures tend to have different problem-solving techniques.
There may also be local factors that code developers are unaware of. “Researchers in one country may have one version of the product that is localised in a local language and as a result they pick up on different behaviour,” says Ludwig.
However, an over-reliance on bug bounty programmes should be avoided. As Luta Security CEO and bug bounty pioneer Katie Moussouris recently warned in an interview with Computer Weekly, bug bounties should not be considered as a replacement for penetration testing.
Read more about bug bounties
- Teen becomes first millionaire through HackerOne bug bounties.
- GitHub security bug bounty program stretches to enterprise cloud.
- Google Play bug bounty hunts RCE vulnerabilities.
- Most infosec professionals would trust hacker-tested products.
Ludwig observes: “The reality of security assurance is that there is no silver bullet. There are a lot of different technologies; all of them are getting better over time. But it is really important that you deploy as many overlapping technologies as possible.”
Despite the advantages, bug bounty programmes carry with them inherent challenges that need to be addressed in order to be fully effective and worthwhile. A poorly implemented bug bounty programme can lead to development teams becoming swamped by notifications, resulting in vulnerabilities being missed and a harassed workforce.
That said, these challenges can be mitigated as follows:
1. Perform a pre-review pass with machine learning tools
Prior to initiating a bug bounty programme, and especially when the project is already in development, organisations should take the time to review the code using machine learning tools to identify any quick-fix bugs.
Not only will this reduce the number of bug bounties to be paid, it will also allow the researchers to focus their time on searching for the more subtle and less obvious bugs.
“Machine-based tools are good way to speed everything up during a first pass, so the human has more time for the deep dives and to look at the more unusual things,” says Colin Tankard, managing director of Digital Pathways.
2. Enact strict versioning rules
One of the key routines a development team should adhere to is following a strict versioning system, thus ensuring the latest version is shared for review. Complications can arise if the development team is working on one version of the code while the review team is examining another.
As well as being inefficient, conflicting versions of code can cause confusion when researchers are referencing parts of the code that are potentially no longer there, or have been substantially changed.
Organisations need to ensure their development teams adhere to management-led, organisation-wide, versioning procedures. This needs to be performed in conjunction with confirming that the latest versions of the code are published for review, thereby ensuring that everyone is working on the same code.
“We have experienced where the code is being developed by an internal team and it only takes somebody to pull the wrong version or not to upload it in time, for it to start to become out of synch,” says Tankard.
3. Introduce bug bounties sooner rather than later
Ideally, like conventional internal testing, bug bounty programmes should be implemented as the code is being developed. This allows bugs to be spotted sooner, allowing them to be resolved with minimal impact on the overall coding.
Waiting until development is nearly completed carries with it the potential for deeply embedded flaws to be uncovered. This could require significant changes to the code, which could have been mitigated had they been discovered sooner.
4. Scale up the bounties
One of the greatest challenges when first introducing a bug bounty programme is that organisations can be inundated with results. This can be overwhelming and demoralising for a development team. To mitigate this, organisations should start with lower-priced bounties, thereby reducing the number of bugs likely to be reported in the early stages of the bug bounty programme.
As the bug bounty programme continues, and the more easily discovered bugs have been found, organisations can start to increase the bounties. This incentivises a greater number of researchers to spend more time examining the code for subtler bugs.
“We try to guide the bug bounty programme to areas that are particularly interesting or complex, by looking at how much we pay,” says Ludwig. “If a product is not getting as much attention as we would like, we increase the amount we will pay for specific issues.”
5. Gradually increase the number of researchers reviewing the code
Organisations can also consider opening the programme only to a limited number of researchers initially, thereby preventing a surge of reports relating to the more obvious bugs.
As with restricting the initial rewards, organisations can begin to increase the number of researchers examining the code for potential vulnerabilities as the bug bounty programme matures.
An extension of this would be to completely change the researchers examining the code once the number of bug reports start slowing down, allowing new people, with different backgrounds and skill-sets, to examine the code.
“The idea is to slowly ramp up not only the customer development but also the researchers,” says David Baker, chief security officer of Bugcrowd. “You train both sides of the market place to provide value to each other.”
6. Curate the results
Often the bugs that have been found will be duplicates of what others researchers discover at the same time. Rather than having the results fed directly to the development team, and potentially distracting them with the same bugs being repeatedly reported, organisations should ensure the feedback is curated.
It is also important to check that the submissions cannot lead to the introduction of vulnerabilities into the code. These could occur as a genuine mistake, but equally – especially in fintech industries – potentially be deliberate malfeasance by malicious actors wishing to subvert the code.
“We score bug reports using the common vulnerability scoring system, which is an industry standard way of scoring those issues, and have a service level agreement process,” says Ludwig. “There is a rate at which issues must be resolved, based on the severity of the issue, we bracket them as low, medium, high or critical issues.”
7. Have the bug bounty programme form part of the development cycle
Having a bug bounty in place is one thing, but the feedback needs to disseminated in an orderly and concise fashion. Rather than bombarding development teams with endless notifications when bugs are found, distracting them and potentially causing another bug to be created, organisations should incorporate the feedback into team meetings to ensure the information is adequately disseminated.
“Two main challenges exist when integrating bug bounties into an existing cycle. First, a team doesn't know when new reports appear, so it's not possible to plan proactively,” says Tim Douglas, an operations engineer at DuckDuckGo. “Second, the reported vulnerabilities can be severe, becoming the top priority and delaying other work.”
Although organisations can distribute the feedback as a regular email detailing all of the discovered bugs, it may be better to introduce the feedback in a meeting, to promote discussion and encourage creative thinking.
The key point here is to ensure that the development team is regularly updated with the latest feedback in a concise format without causing unnecessary distraction. This will allow the development team to work more efficiently, by highlighting which parts of the code should be prioritised.
8. Ensure reported bugs are resolved swiftly
Not only can it be costly for organisations to pay for previously identified bugs that have not yet been fixed, but researchers can be left feeling frustrated and unappreciated when they are repeatedly finding the same bugs.
A strict development cycle ensures any reported bugs are swiftly resolved, without interrupting the developers with endless notifications. A code review meeting at the commencement of the week, discussing bugs that need to be resolved, allows development teams to plan when and how the bugs are resolved.
Read more about vulnerability management
- Four key steps to managing software vulnerabilities.
- Vulnerability management programs need an upgrade for the cloud era.
- The vulnerability management process after Equifax.
- Pentagon hack possible due to bad vulnerability management.
Also, scheduling time for debugging, or assigning part of the development team to focus on this, can further help with resolving bugs before the next version goes live.
“Even if you have been penetration-tested, if you do not have a means by which to take a vulnerability presented to the development team to have fixed and prioritised according to its criticality, any pen test or bug bounty is not going to be effective,” says Baker.
9. Maintain project confidentiality
One of the main benefits of relying on internal penetration testing is that it reduces the likelihood of exposure to malicious actors. However, with the appropriate security protocols, these risks can be mitigated.
Rather than utilising an open bug bounty platform, where anybody can view the code, organisations can use what is called a closed bug bounty platform, in which only certain researchers are invited to take part.
Researchers can be curated based on being positively identified, passing a background check, and/or signing a non-disclosure agreement.
While restricting entry to a bug bounty programme can limit one of their core benefits, namely a wide skill-set through the broad range of researchers, it does allow an organisation to have controlling measures in place to maintain project confidentiality and security.
10. Ongoing review, rather than short-term assessment
Ideally, bug bounty programmes should be an ongoing review throughout the lifecycle of the product, rather than a short-term review.
As many online applications have ongoing support and regular patches to update the code against the vulnerabilities, these cause the code to evolve and expand in ways that were unforeseen by the development team. These updates have the potential to create inadvertent vulnerabilities, which organisations may be unaware of if they do not have a bug bounty programme.
Furthermore, having a bug bounty programme as part of the update cycle allows for the code to be constantly reviewed by external parties. This, in turn, allows organisations to focus more of their resources on development projects, thereby allowing new products to be developed faster.
Having these new products available on the market sooner will allow organisations to cover the costs of their ongoing bug bounty programmes for their existing products.
Bug bounty programmes allow organisations to better manage their own resources and expand the potential talent pool of security researchers examining their code. However, these programmes need to be carefully managed and curated, so that development teams are not overwhelmed.
The key is for organisations to remain open to constructive criticism and accept that they will never – internally – spot every vulnerability. “One of the things companies hang up on is inviting random researchers to look at their security. They want more control of the situation,” concludes Ludwig.
“When you have a bug bounty programme, you have some definition of what that engagement should look like. It actually ends up being more transparent, but also more controlled.”