The AI Era Is Creating a Bug Hunting Arms Race

The AI Era Is Creating a Bug Hunting Arms Race 🚀

Vulnerability disclosure and bug bounty programs have represented a paradigm shift years in the making. When Apple finally announced a bug bounty in 2016, the top reward was $200,000. It rose to $1 million in 2019 and $2 million last year. But all of that is about to change again. As agentic AI models become more adept at autonomously identifying software vulnerabilities and developing exploits for them—identifying weaknesses and creating hacking tools—vulnerability disclosure programs are being flooded just as organizations are finding more bugs than ever themselves. This abundance is changing the economics of bug bounties for both institutions soliciting submissions and researchers. And, crucially, the field is changing in lockstep for attackers, too.

“I’ve probably submitted three times more bugs than I did last year at this time. I would suspect that a company like Google is going to spend two to 10 times as much on bug payouts as they did last year,” says independent security researcher Joseph Thacker. Tech giants, he adds, “can handle that pressure, but most companies can’t. Developers may start to feel even more pressure to quickly release patches—potentially speeding longstanding and hard-won standards like 90-day disclosure deadlines. As security researcher Himanshu Anand wrote earlier this month, “The 90-day responsible disclosure window was built for a world where bug finders were rare and exploit development was slow. That world is gone. LLMs have compressed both timelines.”

The urgency of real-world attacks facilitated by AI seems to be growing. In findings published earlier this month, Google researchers said that they had observed “prominent cyber crime threat actors” attempting to exploit a zero-day—or previously unknown—vulnerability that they had developed using AI tools to bypass two-factor authentication on an open source system administration platform. “We all assumed it was already happening, and this is our first evidence that it is happening,” John Hultquist, Google Threat Intelligence Group chief analyst, says of attackers using AI to discover novel vulnerabilities and create exploits. Hultquist adds, “Zero-day use by criminal actors has been fairly limited, and the ones that do use them tend to be really successful, so I think we shouldn’t underestimate the impact of more criminals with a zero day in their hands.”

For researchers making money through bug hunting, though, times are changing. The command-line tool Curl ended its bug bounty program in January after being inundated with low-quality submissions generated by AI. “We have concluded the hard way that a bug bounty gives people too strong incentives to find and make up ‘problems’ in bad faith that cause overload and abuse,” the group wrote. Last week, Linux creator and lead developer Linus Torvalds wrote that the famed Linux security mailing list has become “almost entirely unmanageable” because of high volume and duplicate AI bug reports. In April, though, Daniel Stenberg, the founder and lead developer of Curl, said that the quality of submissions had improved. “Instead we get an ever-increasing amount of really good security reports, almost all done with the help of AI. They’re submitted in a never-before-seen frequency and put us under serious load.”

At the end of April, Google announced that it was overhauling its Vulnerability Reward Programs for Chrome and Android and lowering payouts for some classes of bugs while increasing others. The company wrote, “As the security research landscape evolves with AI, we’re making changes in our programs to ensure we’re rewarding the most challenging and impactful vulnerabilities in our products.” Increasingly, some researchers argue that structural defenses are necessary to address accelerating vulnerability discovery. “You can’t patch your way out of this,” says longtime security engineer and researcher Niels Provos. “You need to build infrastructure that makes as many bugs as possible irrelevant.”

Read full article