A recent publication, New Solutions for Cybersecurity, published by MIT analyzes 61 bug bounty programs over the course of two years. Their conclusions point out some items myself and colleagues in the industry have noticed:

• The top 1% of bug bounty hackers collect most bounties
• Top bounty hackers received pay between $16k-$34k a year

For Western security researchers, that pay looks more like a monthly than a yearly salary. Since the top hackers work full-time hours, it’s necessary that they come from areas with lower costs of living.

This data begs the question: Is it worth running a bug bounty program and outsourcing some security work to hackers based abroad?

Risks of the Purchasing Power Parity

Many organizations in sensitive areas, such as politics and healthcare, do not allow their data to be accessed by those based abroad. A main reason for this policy is access to legal remedies. For example, if a person decides to post the database on the open internet, they can be subject to a civil suit or even criminal charges in the US.

If a bounty hacker is located in a country that does not have a Mutual Legal Assistance Treaty (MLAT) with the United States, there’s little recourse if they don’t behave ethically. Legal action is still possible if the person is based in an MLAT country, but effectiveness plummets while the cost skyrockets.

Many organizations provide isolated instances for use by bug bounty hackers. However, if a vulnerability discovered there can be used in production, those hackers must be trusted not to disclose it before you’ve patched it.

Companies like Zerodium purchase zero-day vulnerabilities for up to two million dollars. So, you’re trusting the hacker to prefer a relatively small bounty to a potentially large windfall.

It’s worth noting some Bug Bounty platforms offer a service utilizing only thoroughly-vetted hackers based in the US. However, this option tends to raise the cost of the bug bounty program well beyond the price of a comparable pentest.

Compliance Hasn’t Caught Up

During a recent SOC2 audit, I informed the auditors that the company had an active bug bounty program. They explained that while the bug bounty program is positive and would be noted in the report, it did not directly satisfy any requirements. This is because bug bounty programs don’t follow a comprehensive pentest methodology like testing all API endpoints; typically it’s just various hackers using ad-hoc tactics. Fortunately, the company had conducted a traditional pentest and continues to do so annually to stay compliant.

Individual bug bounty programs vary greatly in terms of what vulnerabilities are in-scope or not, the amount of vulnerability reports received, and the amounts offered/spent on bounties. Given the uneven landscape, it’s difficult for auditors to assess the effectiveness of a bug bounty program as a security control.

Right now, there’s no explicit mention of bug bounty programs in compliance standards like CSA STAR, ISO27001, HIPAA and SOC2.

So, while bug bounty programs certainly help with real-world application security, right now they don’t help much with compliance.

Price Comparison

Generally, I’ve found the cost of licensing a bug bounty program and paying out bounties to be more expensive than conducting a traditional penetration test.

The reason is that the platforms tend to charge a hefty licensing fee, which itself rivals the cost of a single penetration test. From there, every vulnerability discovered carries a price tag tied to its severity. It’s like paying for a penetration test one line item at a time.

Hidden Costs: Engineer Time

Beyond the up-front costs, there’s the hidden costs of internal staff time. Vulnerability reports are typically confirmed between 5–20% of the time; meaning that there’s a lot of triage work looking into vulnerabilities which really don’t exist. Unlike a pentest that occurs during a set time period, this causes a stream of issues needing confirmation to tie up engineers’ time throughout the year. You’ll have to budget this cost, and the realistic availability of your developers, into whether these programs will work for your company.

Real-World Security

If there’s an area where bug bounty programs excel and deliver return on investment, it’s finding vulnerabilities that might stop a real data breach. The hackers looking for bounties try some of the same techniques you can expect real hackers to. They often discover vulnerabilities not seen on a pentest or code review.

If your web application is open to the public, chances are some hackers have already been trying to find vulnerabilities. Having a bug bounty program gives them an opportunity to make money from it above-board, rather than try to find ways to monetize your ill-gotten data.

If reports are triaged and patched effectively, bounty programs certainly help improve application security on an ongoing basis.

Conclusions

Given this information, what is a CISO or security manager to do? I’d suggest:

• Prioritize regular penetration testing, alternating vendors
• Recognize Bug Bounty programs augment pentesting, but don’t replace it
• Review compliance requirements and risks present in using hackers based abroad
• Ensure you have buy-in from engineering to confirm/resolve bug reports
• Establish a Bug Bounty program to augment an already healthy application security program

Bug Bounty programs should be treated as a feather in the cap of your information security program. Properly managed, they will help you fix vulnerabilities before malicious hackers discover them. However, despite marketing to the contrary, they won’t solve compliance problems and shouldn’t replace penetration testing.