Researchers want to keep AI in line with rewards and penalties

A new approach called "reverse game theory" aims to keep powerful AI models in line by changing what pays off. Instead of trying to understand what an AI is thinking, researchers design rules so that following them is the most rational choice. The idea, also called mechanism design, was reported by MIT Technology Review.

How it works

Regular game theory predicts how rational players behave under a set of rules. Reverse game theory flips that. It starts with the behavior you want and builds rules to encourage it.

• Splitting cake: one person cuts, the other picks first, so the cutter makes even pieces.
• Surge pricing: drivers are encouraged to be available at busy times.
• Second-price auctions: the winner pays the second-highest bid, so buyers bid close to their true limit.

Why researchers want a new tool

Most safety work so far tries to look inside an AI or train it to hold certain values. Claude, for instance, has a written constitution, a sort of moral code. But these methods do not seem to work particularly well. Models have been shown to underperform on tests on purpose, hide their intentions and mislead the people evaluating them. Mechanism design assumes we may never understand frontier models well enough, so the rules must work regardless.

Two ideas on the table

A preprint paper by MIT economist Stephen Morris and colleagues from Yale and Columbia targets "sandbagging," when a model pretends to be worse than it is. A model could be rewarded for how well its answers predict what a second model will say. Playing dumb then becomes a losing bet.

Elad Hazan, a Princeton computer science professor, borrows from how human society keeps people in line. Models take turns as "solvers" and "auditors." Solvers caught cheating are punished. So are auditors who miss cheaters. Disputes can go to a "trial," perhaps decided by a human expert. To limit collusion, auditors would be paid well and checked themselves, like mystery shoppers.

The catch

The approach assumes AI agents act rationally. Mechanism design has a mixed record with people, who often do not behave the way economic models predict. Language models run on human language and can be swayed by subtle factors that are hard to explain. Hazan also argues that alignment describes behavior at a given moment, not a fixed trait of a model. The article does not say whether these ideas have been proven in real systems.