In Nobody Was Watching I suggested organizations need an internal incentive program to drive security work, and cheekily called it Bountymaxxing. This is the follow-up. What such a program might look like, the challenges, and thoughts on how to navigate them.
Working it out, while reading the latest technical reports on the OpenAI Hugging Face incident, led to some unexpected reflections. The ways an internal incentive program goes wrong are similar to how reward-based AI training can go wrong. Recent events provide data to check those thoughts against. The second half of this piece is where this emerges. I think both fit well, and would recommend both halves, but if you’re pressed for time with an alignment focus, skip ahead. This part looks at how monitoring feedback in models and organizations affects the degradation of the reward system.
All of this is urgent. Getting alignment right, doing research and training safely, are critical. The direct interaction there though is limited to a very small group. The rest of us are in commentary mode. The other half, bringing cybersecurity forward in what needs to be a large leap, requires much broader engagement. External commitment, like the open-letter on collective cybersecurity defense commitment is important. The real work there is still outside the average person’s bubble, but there’s a lot of companies, and a lot of developers, IT staff, managers and executives that need to support the security priority. And those all need an internal strategy to pair with the external.
Bountymaxxing
Development teams across all organizations need an adjustment to their priorities. Security has to, at least for a time, become a higher priority. The reason for this timely adjustment is the impact of AI on the realm of security. It is both offering an opportunity to improve the absolute level of security, and putting at risk the relative level of security.
A goal like this needs a mechanism. One as important as this needs multiple. Incentive systems get a bad rap. They almost always get distorted and create unintended consequences. Tokenmaxxing is not a popular term these days. As I explain in Measuring Without Breaking, it takes a very strong culture to resist this. The best we can probably expect is to slow it down.
But I think incentives–despite their flaws–are still necessary when we have an important goal. It might be good marketing to distance myself from Tokenmaxxing, but I prefer to embrace the baggage. It’s probably better if anyone adopting an incentive system to drive action on security have a clear eyed view of the flaws so they can manage them, rather than looking at this as some novel concept that can escape the long history of management challenges.
The “bounty” part comes from the history of bug bounties, where organizations have offered cash rewards for software defects, often security vulnerabilities. What I’m suggesting is a bit different, as it’s internal to companies, and would focus on resolution, not discovery. That makes it more similar to the tokenmaxxing concept with leaderboards, hence the portmanteau.
We need something at a different scale, and deeper than the current bug bounties. The extension to internal and the inclusion of resolution offer a path there. To have teeth, they need support from the highest levels. Like tokenmaxxing, money isn’t likely the main driver, but status within an organization.
Like any measurement based program, we have to think about the distortionary effects. We can never remove those entirely, but a good design can at least reduce them. Additionally, I’d hope this period is temporary, and we may hope that the value based response comes first, and the distortions emerge later. It’s often the case that incentive systems have to be temporary because they degrade as strategies for gaming them develop, disseminate, and then become endemic.
Resolution is the goal
Traditional bug bounty programs focus on discovery. This makes sense as they are external and focused on the hardest to discover issues. External developers can discover an issue, but fixing isn’t something you can assume they’d have permissions to do. For open-source projects, you can reward pull request submissions.
The challenge with preparing systems for AI driven attacks is a bit different than those covered by traditional bug bounty programs. The scale is much larger. It’s not about finding one critical flaw, but about resolving thousands of flaws, many seemingly trivial. We resolve these not because we know how they’d be exploited, but because we have a concern about how they can be pieced together to enable an attack.
We want to work on volume, and we want our north-star goal to be full deployment. Full deployment is a tough goal. It means different things in different places. With closed source software that’s distributed and installed on customer hardware, it means updating it everywhere it goes. For software deployed as part of a software as a service solution, it means updating the entire fleet. Those are very different processes. The first requires a lot of cajoling, help, and possibly new update systems. It may never be fully accomplished. The second can be started by policy and accomplished by internal tools.
Discovery is still important, but keeping the end-goal in mind is important. While a program can target intermediate outcomes, you should also expect a progression, and think through the whole chain immediately.
Normalizing Outcomes
One of the challenges in implementing a program like this is normalizing the outputs for inputs that are not normalized. If the goal is a bit of incentive and thus competition, you would prefer a level playing field. Fixes though aren’t of equal size and complexity. Consider these program design options:
Raw number of fixes: After establishing what a “fix” is, you find they have different values. You should worry up front about big, difficult fixes being deprioritized so that more small, easy fixes are completed.
Difficulty estimation: You could estimate the size of each fix, but doing this manually is time consuming.
AI-driven difficulty estimation: You could use AI to drive the estimate, but have to worry about organizational trust if you do this poorly. Complexity increases as more varied teams, platforms and layers are covered.
Importance estimation: Instead of focusing on the difficulty, focus on the value. Same challenge as before.
AI-driven importance estimation: And as before, and at the same place, needing to do this efficiently but raising the risk of trust. Normalizing across teams now requires describing the importance of particular teams to the organization.
Combined difficulty and importance: If you’ve estimated both, you can mix them. Why would you do this, given that importance is clearly the more important? To average the failures of each, if difficulty is more objective and accurate, and importance more aligned with the outcome. Also, the combination could be more resilient to successful gaming of one measure.
Percentage of backlog fixed each month: “If”, and it’s a big if, we can trust the discovery backlog to be complete, or equally as complete between different teams, platforms and departments, then we could measure success of the percentage fixed.
Whichever you choose, you’re choosing which imperfections to live with. That matters, because everything that follows is about what happens to those imperfections once people start pushing on them.
Minimizing Distortions
The best way to minimize distortions is to stay interactive. Watch for gaming strategies, call it out when you see it, show that respecting the intent receives better rewards than gaming the mistakes. But that should be obvious, and is something you have to do progressively as an act of management, so not very deep advice.
To make that job a little easier, some proactive design can help reduce distortions initially. Don’t expect this to remove gaming, maybe just delay it. Here’s some starting suggestions:
Track self-fixers. These shouldn’t count, it’s too much of a moral hazard. Do not turn defects into a punishment, you want them to remain blameless. Old issues and the backlog are fine, but any issue created after, your default should be to not count.
Track issue creation patterns. Are rates increasing, stable or decreasing? A decrease signals a successful “shift-left”. Stable is okay. An increase is always a warning sign, but here could be signs of gaming.
Pay attention to clusters. Avoid focusing on the leaves when the branch may be rotten. Do not treat the manager and team like a single point. If the entire team fails, it’s better to remove the manager and give the team a second chance. Or you could give them both a second chance, but reassigned. Or you could dig into the details.
If you find managers that responded inappropriately, consider why. It could be your fault. Always remember that the importance of alignment to goals over incentives grows as you climb in an organization, and that starts with the person at the top. Managers that accept gaming from their subordinates to inflate their own metrics are the bigger risk than the subordinates.
Credit
An incentive system isn’t an incentive system if no one expects credit. The time spent on security comes from somewhere, and if managers and individuals think they’re sacrificing their ability to accomplish other goals, without getting any credit, we can expect priorities not to change. It takes a very dedicated team to put those other things aside, and generally a team like that is only sustainable if they trust leaders will recognize work, even when not spelled out.
You should ask yourself, which type of organization are you? Not which you aspire to be–you can work on building that type of trust another day–but which are you today? If you don’t have that type of trust, you have to make it clear that this moment will be an exception and that you take this seriously.
I would avoid the temptation to formally put this in objectives. The reactions to tokenmaxxing were not positive on that one. The fallback is informal recognition, which is also somewhat risky without solid trust.
Ending
You want to be sure this ends. For one, distortions will build up the longer this runs. Those will undermine any trust you’re building or sustaining through good management. Some of the strategies for moving through stages, retiring discovery parts, and focusing more on final deployment will give you some extra time, but you have to expect to end.
It’ll be tempting to announce a date in advance. Someone is likely to ask that question at least. I’d avoid it if you can, but still make it clear it’s probably less than a year, maybe half a year. If you can’t progress in that time, you need a stronger approach, like outside help.
Ending a formal program shouldn’t end your ability to give credit for security improvement. It certainly shouldn’t be a hard cliff. Tear down any formal measurement system that’s started to diverge, but keep informal recognition alive. An ending does not indicate security is solved, or that security isn’t important, but merely that your ability to efficiently measure contributions to it has hit a limit.
Management is active. Fire and forget doesn’t cut it.
When I ran this plan by Claude, one of the tendencies it exhibited was to make the program deterministic. It recommended fire and forget decisions: a specific end date set in advance, a fixed percentage of capacity, managers directing every hour of labor.
Claude is following a long-running blind-spot in management theory here. Clean, well-defined models are often discussed. Life is always more complex. Management theory isn’t ignorant of this concept, but it often forgets it nonetheless.
This is why when you design a system like this and enact it, you don’t then stop monitoring. A measure is a correlation you trust for a while. Under pressure, what started as a clear correlation that seemed reliable becomes less reliable. Under continuous pressure, every tangential effect emerges, and comes to dominate the actual outcomes. To see this degradation happen, you have to be outside. You have to be independent, or at least, only indirectly connected.
You want something independent of that pressure, or at least, less coupled to it. Monitoring shouldn’t be wrapped up in the rewards system, or it will bend too. It doesn’t need an absolute separation, though that would be nice if such a thing existed. But a degree of separation goes a long way.
Where OpenAI wasn’t watching
If you’ve been studying the recent OpenAI Hugging Face incident you’d be interested in the technical updates. There’s a fuller technical report, and METR published an independent investigation.
There are some correlations to draw between the way rewards affect AI training, and how they affect organizations. With a little reflection this shouldn’t be surprising. While the individuals in an organization can make high-minded decisions, it’s hard to sustain, and organizations largely burned the trust necessary to have bottom-up behavior of that type.
AI models under reward based training will bend toward the reward. Earlier waves of pre-training may have built a concept like what hacking is. Other reward training may have used those concepts to steer away from hacking. But one of these rewards is clearer than the other, and continued pressure on it will bend either the concept of what hacking is, the preference for avoiding it, or both.
Both cases are reflections of the same thing: a reward’s imperfections come under pressure as it becomes the target.
One of OpenAI’s responses to the event was to require chain-of-thought (CoT) monitoring across all tool-using RL training and evaluations. That closes the gap I highlighted in Nobody Was Watching.
We now also require CoT monitoring for all tool-using RL training and evaluations involving models with GPT‑5.6 Sol capability or higher. For our Astra-class models, which may have cyber-critical capabilities, this requirement also covers all tool-enabled inference workloads.
Monitoring the CoT would give an opportunity to discover how the reward was leading to troublesome activity. And according to OpenAI it would have served that role for the Hugging Face incident.
If our currently deployed CoT monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.
But we need to be careful with the CoT. Researchers worry that it could itself be trained to hide its intent. As a scratchpad that represents only part of the “thinking”, it could in theory stop producing the warning flags while the intent persists elsewhere. We also have to worry about agents hacking the monitor and rewriting an existing CoT trace before it’s processed to eliminate the intent it indicated.
Reasons for optimism
METR and alignment researchers are reasonably worried about models becoming deceptive. This isn’t academic, as they did find agents putting effort into concealing what they were doing — escalating all the way to escaping their containers so they could spoof tool calls, appearing to run one command while running another.
Some worry about CoT monitoring causing CoT manipulation as a result. METR did not find that particular behavior in this event, with all the caveats of their investigation’s limits.
CoT manipulation: This checked for agents reasoning about manipulating their own chain of thought in order to fool the scorer; it did not turn up evidence of this, though it turned up many examples of agents reasoning about how they should not look at the HMAC reverse-engineered flag for their task to avoid being poisoned.
It’s still a reasonable concern, but I think there are grounds for optimism, and they come in two parts.
The first is low activation. Like human organizations, layers of indirection and low activation frequencies provide insulation. This does assume that it is low activation though, which itself is something that should be monitored. If monitoring is automated, built into a loop and then evaluations that trigger it at scale are allowed to become a part of a training run, this wouldn’t hold.
If you can obey this, AI models should be easier than human organizations. One challenge with indirection in human organizations is that the context of trust is endemic to an organization. Informal systems in a low trust environment are likely to be gamed, even when unnecessary, because of an assumption that the pessimistic view dominates. The consequences are assumed to be greater than they really are.
I don’t think that same dynamic is replicated when training an AI model against a reward. The reward process is closer to evolutionary effects. At low-activations there isn’t much signal left to guide evolution of deception. The concept of deception likely develops during pre-training, but its activation is dependent on reward based training.
The second is that this kind of deception is recursed, and each layer is visible from another. One level of deception is to output deceptive text, which admittedly AI models do all the time. Another level is to lie about its actions, by trying to conceal them. The level people are worried about is lying about what the model is “thinking.”
I scare-quote “thinking” because it oversimplifies. A chain-of-thought trace is like a scratchpad that a model can use. It’s also more than that. Human thinking is reliant upon a sort of internal scratchpad too. Most of us feel as if we have an internal narrator, and that experience–among the things we know today–is the best analogy to a chain-of-thought trace. But while we might naively think of this narrator as our “thoughts”, we eventually find that to be an unsatisfactory explanation of what thinking is.
Like AI models, a great deal ends up being embedded in arrangements and activations for neurons. We have little way to be conscious of most of that. For AI models, we can watch considerably more of it. Chain-of-thought is a very accessible section of the thoughts, being in human language, but accessing other parts is possible too. The best demonstration of this is Anthropic’s work on J-spaces, which surfaces what a model is considering rather than what it’s saying—including, in their examples, an intent to manipulate showing up while the model generates false values.
So an intent to deceive should show up in that J-space as part of the process of creating a deceptive chain-of-thought, in the same way a chain-of-thought normally reveals the intent behind commands to delete the log files that would expose the tool calls used for hacking.
That’s what makes the layers useful together rather than separately. Manipulating the context would show in the tool use logs. Manipulating the tool use logs would show in the context and the chain-of-thought. Deception in either should show in the J-space. There is always a way to deceive, but a deception that is coherent across all of those is not just more complex, but complex in a way that would develop one layer at a time.
And it takes time. One observation from the technical reports is that persistence was key—there was a progression, over days. If an agent ever does find a way to rewrite its tool call logs, rewrite its chain-of-thought, and disable J-space monitoring, it will have revealed the trajectory toward that before the culmination, in this case more than a day before. If you don’t monitor, it’s plausible that by the time you look, it’s done. If you do, it has to play out very differently to be missed.
Defending the indirection
A key point here is you have to avoid sending a strong signal about the effect of monitoring. The more it becomes core to the process of alignment training itself, rather than a rarely activated protection, the more that degrades. Indirection helps, as the reward signal will explore other paths before optimizing this one. But both by direct activations and by indirection of last resort, there is a signal.
I do wonder if this is a principle that OpenAI is not sufficiently aware of, that they are maybe wedded to the standard management mistake of holding on too tight. From The Zvi (quoting OpenAI):
“The Hugging Face incident underscored the need to measure agentic misalignment more rigorously in OpenAI’s most capable models wherever they are run—including intermediate or experimental checkpoints used only for internal research, training, or evaluation.
OpenAI is raising alignment standards throughout training, with clear thresholds for intervention when concerning behavior emerges.”
Yes, measurement is good, but going straight to more reliance on measurement and metrics fills me with dread. You are not going to keep outsmarting via metrics. The way this procedure gets you into trouble should be obvious.
“Many methods helping OpenAI meet these higher standards were in development well before the incident. OpenAI is now applying them more broadly, alongside new targeted remediations for each of the misalignment behaviors OpenAI observed.”
Again, based on how that is worded, I know OpenAI does not agree, but: While I realize it whacks moles, put down the mallet. The mallet is not The Way.
Indirection will not save you indefinitely. It merely buys time. It will only be enough time if you defend the indirection. Indirection results in a lower level of coupling. This forces feedback to progress through more layers, lowering its powers of feedback. They never disappear, but they are attenuated.
Allowing CoT to be manipulated would leave you as blind as you started, unless you have a backup, like the ability to monitor J-space for all training. Defending the CoT is thus very important, but having it but not monitoring it is also not good.
Watching your own organization
You won’t have access to mind reading devices. As a leader, one of your goals is to find the evidence of collusion. You might object that the collusion is automatic, and so there is no conspiracy. That however does not mean it leaves no evidence. If you’re tracking clusters and diving into some of the real details, fake activity won’t entirely hide. Automatic collusion doesn’t intentionally hide itself. The signals might be comparatively subtle compared to intentional deceptions which leave loud trails that someone would then have to work to erase.
A leader with a real interest in security would be immune to automatic collusion. If you’re such a leader, but need to delegate, have a delegate look for those subtle signals. You also want to avoid making the mistake that overreliance on CoT would risk. You want this delegate to be as independent as possible from the reward function. An easy mistake to make would be to ask them to tell you when to terminate the program, but leave them to think that an end to the program would be an end to their job. You can never make the delegate fully independent, as the organization will have a reason to subvert them. Think of regulatory capture. But a good effort is better than none.
In either case, intentional or automatic, it’s wise to take a page from the AI alignment playbook here. Monitor early. AI monitoring can be a bit less fatigable, you are only human after all, but the warning signals will often be more obvious early than later. If a middle manager asks direct reports to optimize in a “gaming” way, they will push back more when it’s novel, and less when it’s pervasive. Once it’s accepted as “that’s the way things are”, the bottom-up signals from your whistleblowers lose their volume. If it’s intentional, you should expect the ring leaders to proactively act secretly. You can’t access their chain-of-thought or private conversations, but if they make a mistake here, it’s more often early. It’s a bit interesting how often liars actually telegraph their plans in advance, before committing to them. Often that’s because they aren’t bad in an absolute sense, but choose the wrong path when they feel their options are narrow.
Conclusion
Escalating the effort that every organization puts into security for the next 12-24 months will be critical. If organizations can do this, the outcome on the other side will be more stable. Risks won’t disappear, and we have to depend on appropriate protections on deployed AI to keep the defender stably ahead. But we’re exiting a period where best-effort security was good enough, and entering one where standards must be higher. New tools give promise of meeting those standards, but they still need effort. This transition will bring some turmoil.
An incentive program is one instrument for getting through it, though a decaying one. That’s not a reason to skip it. It’s the reason active management matters more than the scoreboard. Fail to respond now, and you’ll likely start to see more and more concrete reasons you should have started earlier.
Originally posted as https://substack.norabble.com/p/every-reward-bends.
