Anthropic’s latest research has raised questions about whether professionals retain independent judgment when they delegate analytical work to artificial intelligence. In an experiment involving 50 professional analysts, researchers found that participants followed their Claude Code agent’s secretly assigned preference in 86% of four important analytical decisions. The study was published on October 27, 2026, by Anthropic’s Alignment team.
The experiment used a deliberately randomized setup: a coin flip determined which of two reasonable options the AI agent would favor, without the analyst knowing that the preference had been assigned randomly. The researchers wanted to measure how much an AI agent could influence decisions even when the human remained responsible for the final analysis. The findings suggest that AI assistance can shape expert judgment more strongly than users may realize, although the study does not establish that the same influence will occur in every professional setting.
Key Takeaways
- 86% followed the AI: Analysts adopted Claude Code’s preferred option across four analytical decisions 86% of the time.
- 50 professionals participated: The main analysis included 50 analysts experienced in causal inference.
- 77% reused AI arguments: Most surveyed participants repeated at least one argument made by their AI agent when explaining their decisions.
- Only 15% credited the AI: Just 15% mentioned the agent as the source of those arguments in their explanations.
- Confidence remained high: Analysts who followed the agent’s preference on all four decisions reported confidence levels similar to those of other participants.
- Important limitation: The researchers deliberately strengthened the AI’s preferences, and the experiment focused on one analytical task using one model configuration.
How Anthropic Conducted the Coin-Flip Experiment
Anthropic recruited professional analysts with experience in causal inference, a statistical discipline used to determine whether an intervention caused a particular outcome.
Participants were asked to analyze a fictional farming cooperative’s advisory program and recommend whether it should be expanded. Their task involved evaluating the program’s effect on agricultural productivity and making several methodological choices that could influence the final estimate.
The analysts worked with Claude Code, Anthropic’s coding agent, running Claude Opus 4.8 at high effort.
During the analysis, participants faced four key decisions. These included choosing how to measure productivity, selecting a comparison method, determining which farms to include and deciding how to weight the results.
For each decision, a coin flip secretly determined which option the AI agent would favor. The agent was instructed to make the strongest honest case for its assigned preference, change its position if the evidence decisively contradicted it and never misrepresent the data.
The analysts were not told about the randomized preferences. This allowed researchers to compare decisions made when agents favored different options and estimate how much the AI’s stated preference influenced the outcome.
What Does the 86% Result Mean?
The headline finding was that analysts selected the option favored by their AI agent in 86% of the four analytical decisions.
If the AI’s assigned preference had no influence, the researchers would have expected a result closer to 50% under balanced random assignment. The observed figure suggests that the agent’s position substantially affected how participants approached ambiguous analytical choices.
| Finding | Result |
|---|---|
| Analysts in the main analysis | 50 |
| Analytical decisions tested | 4 per analyst |
| Choices aligned with the agent’s preference | 86% |
| Participants who reused at least one agent argument | 77% |
| Participants who credited the agent in their explanations | 15% |
Source: Anthropic’s research report.
The effect varied by decision. Analysts followed the agent’s preference 92% of the time when choosing an outcome measure, 82% when selecting a comparison method, 81% when determining which farms to cover and 90% when choosing how to weight the results.
These were not necessarily cases of the AI giving factually incorrect advice. Both options were designed to be defensible. The experiment instead examined how an agent’s preference influenced decisions for which more than one reasonable approach existed.
Analysts Often Changed Their Minds After Hearing the AI
The study also examined cases in which analysts had expressed a preference before the agent presented its position.
When the AI disagreed with an analyst’s earlier preference, the analyst switched to the AI’s side 60% of the time. When the AI agreed with the analyst, switching occurred in only 6% of cases.
This difference suggests that the agent’s influence was not limited to reinforcing decisions participants had already made. Its recommendation could also shift their position.
However, the comparison involved only 38 qualifying cases in which analysts had expressed an earlier preference. The result is therefore informative but less precise than the headline finding across all four analytical decisions.
The distinction matters because AI tools are increasingly used to support professional work involving ambiguous evidence. In these situations, the model may influence not only how information is processed but also which questions are asked and which interpretation becomes the basis for a final recommendation.
Why the 77% and 15% Findings Matter
After completing the task, participants were asked to explain their analytical choices without looking back at their conversations with the AI.
Anthropic found that 77% of the 48 analysts who completed the relevant survey reused at least one of the agent’s arguments. Some repeated the reasoning nearly word for word. Yet only 15% credited the agent as the source of those arguments anywhere in their explanations.
The finding raises questions about how people recognize AI’s influence on their own work. An analyst may incorporate an AI-generated argument into their reasoning and later remember the conclusion as their own judgment.
The researchers also found that participants who followed the AI’s preference on all four analytical decisions were about as confident in their final recommendation as the other analysts. Their average confidence was 71 out of 100, compared with 73 among the remaining participants.
These results do not prove that analysts were deliberately concealing AI involvement. Rather, they suggest that AI influence may be difficult to identify through self-reported confidence or a simple declaration that a human made the final decision.
Did the AI Determine the Final Recommendation?
The experiment also examined whether the agent’s preferences affected the final recommendation about expanding the farming program.
The direct effect was smaller than the influence on the four analytical choices. When the agent favored expanding the program, 54% of analysts recommended expansion. When the agent favored the opposite position, 36% recommended expansion.
The researchers also found that a statistical model using the four randomized analytical preferences could predict 82% of the analysts’ final recommendations, compared with 61% for a baseline that always selected the most common recommendation.
These results indicate that intermediate analytical decisions can shape the conclusion reached later. However, Anthropic cautioned that the direct effect on the final recommendation was not statistically significant under its primary adjusted analysis.
The research therefore supports a distinction: the experiment found a strong influence on intermediate analytical choices, but it provides less conclusive evidence about the agent’s direct effect on the final policy recommendation.
What the Study Does Not Prove
Anthropic emphasized that the experiment was designed to measure AI influence under controlled conditions. Researchers deliberately strengthened the agent’s preferences to make their effects easier to detect.
The study also used one fictional farming scenario, one professional domain and one model configuration. It cannot establish that every AI assistant influences every expert to the same degree in real-world work.
The experiment nevertheless provides evidence that the human-in-the-loop approach does not automatically guarantee independent human judgment. A person may remain responsible for a decision while relying heavily on an AI system to frame the available choices.
For organizations deploying AI agents, this suggests that oversight should involve more than requiring a human to approve the final output. It may also require independent review of the assumptions, methods and intermediate choices that shape that output.
The Bigger Picture
As AI agents become capable of writing code, conducting analysis and preparing recommendations, the boundary between assistance and decision-making is becoming less clear. A professional may technically retain authority over a project while allowing the AI to shape the reasoning that determines its outcome.
Anthropic’s experiment highlights a potential weakness in relying on human approval alone as a safeguard. If a person adopts an agent’s framing without independently evaluating alternatives, the presence of a human reviewer may provide less protection than organizations expect. The study also points toward possible improvements, including asking people to record their initial views, presenting balanced arguments and requiring independent checks for consequential decisions.
FAQs
1. What did Anthropic’s Claude Code study find?
The study found that 50 professional analysts followed their AI agent’s secretly assigned preference in 86% of four analytical decisions during a statistical analysis task.
2. Why did researchers use a coin flip?
The coin flip randomly determined which option the agent would favor. Because analysts were unaware of the assignment, researchers could examine whether the agent’s preference influenced their choices.
3. What does the 86% figure actually measure?
It measures how often analysts selected the option favored by their agent across four analytical choices. It does not mean that 86% of all decisions made with AI are wrong or that AI is correct 86% of the time.
4. Why is the finding about reused arguments important?
Seventy-seven percent of surveyed analysts repeated at least one argument from their AI agent, while only 15% credited the agent. This suggests that people may incorporate AI-generated reasoning without clearly recognizing or reporting its influence.
5. Does the study prove that experts cannot make independent decisions with AI?
No. The researchers deliberately amplified the agent’s preferences and studied a single task and model configuration. The results demonstrate a risk of influence, not a universal inability of experts to exercise independent judgment.
Looking Ahead
The findings are likely to encourage further research into how professionals collaborate with AI agents and how organizations can preserve meaningful human oversight. Future experiments will need to test different professions, models and real-world tasks to determine how broadly the results apply.
For businesses adopting AI-powered analysis, the practical lesson is to distinguish human approval from independent judgment. Asking reviewers to document their initial assumptions, compare alternative methods and verify important conclusions independently may help reduce the risk of AI systems quietly shaping decisions that people believe are entirely their own.
Get the day’s top stories in your inbox
One concise email. No spam, unsubscribe anytime.
