Knowing the Job Beats Knowing the Tool
7 September 2026
Anthropic published a study a couple of months ago, built on roughly 400,000 real Claude Code sessions logged between October 2025 and April 2026.
They weren’t looking at which model writes better code. Instead, the study focused on what separates people who actually get useful work out of an AI agent from people who don’t.
It classified sessions by expertise level, based on how precise the instructions were, how often the person verified the output and how they corrected the AI when it got something wrong.
Some useful insights from this study:
1. Domain knowledge matters more than a coding background
The first thing that stood out is who wins.
Software engineers had a verified success rate of 30% on these coding sessions. Everyone else came in at 29%, and management roles actually edged out software engineering roles slightly.
That suggests the thing predicting success wasn’t simply a coding background. It was task-specific domain knowledge: knowing what the finished thing is actually supposed to look like.
2. The biggest payoff comes early
Going from novice to intermediate roughly doubles the success rate.
After that, the curve goes relatively flat. Intermediate to expert barely adds anything.
So much of the effort people put into becoming a “power user” may bring less benefit than the earlier jump to competent, engaged use.
3. What you do when something breaks changes the outcome
Novices abandon a troubled session 19% of the time.
Intermediate and expert users abandon only 5–7% of the time.
Same failure, completely different response.
Put those three together, and the bottleneck was never just who can write a prompt well or who understands the tool best.
It’s who knows the problem being solved and who stays in the loop long enough to correct the work instead of walking away from it.
Knowing the job still beats knowing the tool.
Reference
