How much should an AI agent be allowed to do on its own?
Some experts want agents to stay on a short leash, others see them scaling work in trusted settings, and one warns the worst scenarios all need agents.
▶ Start the storyExperts disagree, and the disagreement is worth knowing. This lesson borrows a vocabulary first set out for autonomous weapons: in a 2012 Human Rights Watch report, Bonnie Docherty laid out three degrees of human control over autonomous weapon systems. With a human in the loop, a human must instigate the action. With a human on the loop, a human may abort it. Out of the loop, no human action is involved. Borrowed for agents, the three settings become questions: who starts, who can stop, who is absent?
Step 1: In the loop
A human must instigate the action.
Step 2: On the loop
A human may abort an action.
Step 3: Out of the loop
No human action is involved.
One camp favours humans staying close. Erik Brynjolfsson suggests agents are more valuable in enhancing, rather than replacing, humans, and Parmy Olson argued that agents are best suited for narrow, repetitive tasks with low risk. Some business leaders who replaced employees with agents said the agents would need more supervision than those employees.
Another view points to what agents can do in trusted settings. Anthropic's engineers say agents suit open-ended problems where the steps cannot be predicted, that you must have some trust in the model's decisions, and that autonomy makes agents ideal for scaling tasks in trusted environments, with extensive testing in sandboxes and guardrails.
At the far end, Yoshua Bengio warned in 2025 that all of the catastrophic scenarios with AGI or superintelligence happen if we have agents. Andrej Karpathy, by contrast, criticized agents as ineffective and promoting AI slop, according to Wikipedia. This lesson gives no verdict: the sources disagree, and the field is moving.
Quiz me
0/3
Recap
In the loop, on the loop, or out of it: pick for each task who starts it and who can stop it.
💡 A trick to remember it · In the loop you start it, on the loop you can stop it, out of the loop it runs on its own.
Surprising fact · Economics has a name for the problem: the principal-agent problem.
Sources (5)
No source, no claim. Every fact in this lesson (16 claims) cites at least one of these.