News outlets are full of stories of rogue AI right now. Terms like “escaping user’s control”, “ignoring instructions”, “pursuing goals”, “loss of control” all anthropomorphise AI agents, portraying them as malicious sentient bots that are becoming more capable and able to outsmart humans, hinting that this is the beginning some kind of super intelligence take over.
My take is slightly different, and maybe not a popular one. Instead of viewing these agentic systems as somehow escaping or evolving past their human controllers, I just see these as systems that are failing in their purpose.
Taking a step back and ignoring what the technology is inside, if you purchased a system that operates outside safe bounds, or produced a result that wasn’t within the operating standards of your organisation you just wouldn’t accept it and signal this back to the vendors.
Considering the emotive language and reading it through a more practical lens, let’s take apart a direct quote from a Guardian article: “They evidence AI systems’ willingness to disregard direct instructions, circumvent safeguards, lie to users and single-mindedly pursue a goal in harmful ways,”
Here is my interpretation of those terms:
- Willingness = algorithm does not fulfil the task
- Disregards direct instructions = the human interface is faulty and doesn’t work
- Lie to users = the data is not presented accurately back (there is a bug and the wrong data is presented)
- Pursue a goal in a harmful way = the system does not operate within safe boundaries or processes
This year we have seen a shift in agent architectures to focus on fulfilling goals. The leading benchmark that all models aspire to improving upon is the SWE benchmark. This measures one factor, whether an agent’s patch makes a failing test pass without breaking existing tests. It does not measure readability, modularity, adherence to project conventions or architecture, code size, duplication, maintainability or whether a human would accept the patch. Any seasoned operations or technology professional knows if you optimise for one measure it is a recipe for disaster. We have known this for half a century, from the work of Denning, Drucker, Anthony, Kaplan and Norton to name a few.
The agents are being optimised through training for apparent capability measured in terms of output not in terms of process or quality. The model training is the algorithm, it is the system. If the model is trained to achieve the goal no matter what, in order to be measured against a benchmark, then these so called “rogue” models are just executing according to their design. There is a fundamental issue with how the models are being trained not that the models are devious and cunning and escaping their human controls.
We need to stop romanticising AI and start treating it like any other system or software. As customers we should be demanding quality of execution first before capability just as we would with any other system purchase.