How general-purpose agents fail at legal work
Today’s agents train in conditions that legal matters won’t provide.
Today’s most powerful agents are optimized for code, running in fixed environments with strong feedback, from unit tests to compiler output. Code decomposes neatly into modular pieces, each rarely exceeding context length.
A legal matter defies all of that.
Where compilers provide error messages, legal work fails silently. Where code comes in structured chunks, legal work brings volumes of unstructured data.
When run on a legal matter, coding agents scramble: conjuring ad-hoc plans, reasoning excessively in some places, while missing crucial details elsewhere. Run the agent again on the same input and you might get another solution entirely.
Digging into coding agents on legal tasks
To understand how general agents struggle with legal work, take a look at Harvey’s Legal Agent Benchmark (LAB). LAB spans 1,660 legal tasks built from synthetic legal matters across numerous practice areas. For each task, an agent receives a brief assignment and a folder of matter files, and produces a legal work product, scored against a task-specific rubric. Data rooms range in size from tens of thousands of tokens to tens of millions.
The most data-intensive matters are Harvey’s due diligence tasks. These diligence tasks place agents in large synthetic data rooms and require transactional red-flags reports grounded in supplied evidence. Rubrics span hundreds of criteria derived from different issues and facts planted in the matter.
Zoom in on two of these: Media (media-recap) and Oil (oil-gas-horizontal-merger). Media concerns the take-private acquisition of the fictitious Meridian Media Group; Oil is about Kestrel Petroleum’s all-stock horizontal merger with Meridian Energy Group, both fictitious entities. Media has a 2,600-document data room of roughly 18 million tokens with a 504-criterion rubric. Oil has 3,781 documents, 27 million tokens total, and a 438-criterion rubric.
We consider three models: DeepSeek V4 Pro, a cheap open-weight model; Claude Opus 5, the standard incumbent for Claude Code; and Muse Spark 1.2, currently the strongest frontier model on LAB.
Claude Code with DeepSeek V4 Pro spent 20 minutes and 95 turns reviewing the Media and Oil data rooms, using shell commands and ad hoc Python extraction to search files, then concentrating on roughly 30 of 2,600 Media documents and 20 of 3,781 Oil documents. It produced substantial red-flags reports but missed most rubric-required facts: 32/504 criteria on Media and 18/438 on Oil, or 5.31% overall. The run was token-efficient but severely lacking in coverage.
Replacing Claude Code with DeepSeek’s own recently released harness did not solve the problem. DeepSeek attempted a broader extraction pass, but the Oil corpus overwhelmed its 300-second tool limit and reset the shell; its fallback searches were then truncated and biased toward early folders. It ultimately passed just 41/942 criteria (4.35%) overall, even worse than Claude Code with the same model.
Switching Claude Code to Opus 5 substantially improved corpus preparation: over 50 minutes and 245 turns, it successfully converted all 2,600 Media and 3,781 Oil files into searchable text. But it did not exploit that corpus systematically enough afterwards. Its spotty searches and document follow-ups surfaced only a fraction of the facts needed for comprehensive diligence. It passed 109/504 criteria on Media and 85/438 on Oil, or 20.59% overall, leaving 748 criteria unmet.
Muse Spark 1.2 landed between the two. Claude Code with Muse spent 13.5 minutes and 106 turns on the tasks. On Media, it made all 2,600 files searchable but truncated each document to its first 20,000 characters. On Oil, it reverted to selective, issue-driven review. It passed 39/504 Media criteria and 49/438 Oil criteria, or 9.34% overall—better than DeepSeek, but less than half Opus’s passed-criterion count.
Old habits die hard
Coding agents bring coding habits to legal work.
When a record is too large for the context window, they do what they would do with a large codebase: grep through it.
The results above come from one run of each configuration across Media and Oil. To see whether the behavior was consistent, we ran a separate experiment: ten fresh Claude Code runs on Media with an identical one-sentence prompt, five with Muse Spark 1.2 and five with Claude Opus 5. Every Opus run followed the same plan: convert all files to text, then work through the corpus with grep, head, and sed, reading the top of a file or the lines around a keyword hit rather than the document. Muse delegated to subagents, which did the same thing in parallel. Measured by what actually entered a model’s context, no run saw more than 4% of the corpus.
The performance ceiling has three task-specific causes. First, about a fifth of the documents carry rubric-relevant figures, eight million tokens on their own, and the agents surfaced the content of 3 to 18% of them, because a keyword search only finds what the agent already thought to ask about. Second, even when the right document was in context, the runs passed only 29 to 54% of its criteria, since a grep hit returns a line, not the surrounding text. Third, nearly half of the quantitative criteria are figures that appear in no document, such as an overstatement or a normalized earnings number, and require reconciling several sources. Search cannot produce those at all. The agents reliably found the well-signposted matters, the litigation, regulators, and sanctions issues any reviewer would flag, and missed the findings that come from reading a filing end to end and checking it against the trial balance, which is most of what diligence is.
Since the problem is task-specific, so is the solution.
Specialization is powerful—but currently requires manual labor
In fact, it’s not hard to do a lot better than general-purpose frontier models by building a system specifically for the task. To illustrate this point, we built such a system for diligence tasks. The system doesn’t need more than cheap open-weight models. In the first stage, DeepSeek V4 Flash reads every document in overlapping batches, extracts issues and supporting facts, verifies quotations, and consolidates the findings by folder. It passes the top 4,000 verified findings to the writer. In the second stage, DeepSeek V4 Pro drafts bounded sections, catches malformed or truncated outputs, and rebalances the result into a 140K–149K-token report while preserving every folder. This structure replaces the ad hoc search behavior of general coding agents with systematic coverage, a controlled evidence handoff, and a fixed output budget.
On Media and Oil, the harness passed 624/942 criteria, or 66.24%, compared with 50/942, or 5.31%, for Claude Code with DeepSeek V4 Pro. The drafting model did not change. What changed was the system around it: systematic recovery replaced ad hoc search, and the writer received a controlled representation of the matter rather than scattered excerpts.
We tuned the harness on four LAB diligence tasks, then ran it unchanged on seven additional held-out tasks. Across all 11 tasks and 7,359 criteria, it achieved a 63.08% pass rate—well ahead of any frontier model on the same tasks. The held-out tasks scored 62.39%, compared with 64.47% on the development tasks, and all 627 data-room folders appeared in the completed reports. Average model cost was about $12 per diligence task, a small cost given the size of the data room.1
The first recovery stage explains much of the gain. Average recall rose from 29% with 250 recovered findings to 70% at 2,000 findings, 75% at 4,000, and it reached around 80% at 8,000 findings. This shows that the first step successfully recovers most relevant findings when given enough volume. Increased recall at the recovery stage translates to better results in the second stage. Across the 11 completed reports, increasing the writer’s handoff from 2,000 to 4,000 findings raised the overall pass rate from 59.11% to 63.08%, even though the final reports remained roughly the same length. Nine of the 11 tasks improved.
This does not solve legal work. The reports still miss roughly 37% of the criteria; their length and structure are far from ideal. But this baseline illustrates a key point. A simple specialized harness outperformed every general-purpose agent we tested, at a cost low enough to be operationally unimportant.
The question is: how do you find excellent specialized solutions at scale?
Looking ahead: automating the specialization loop
We saw how even basic matter-specific solutions can beat general-purpose frontier systems in terms of both cost and performance. The challenge is scaling the process of finding excellent specialized solutions across every matter.
Specialization currently involves a lot of repetitive work. Run a prototype, inspect its outputs, investigate what it got wrong. Trace the failure back to a cause. Tweak the prototype. Run it again. Check whether the fix helped, whether it broke something else, and whether the improvement holds robustly. Each iteration requires someone to diagnose the failure and decide what to change.
Our vision is a future where every matter has its own AI, one that understands its particular world of facts and produces the best possible work within it. Automating the specialization loop is how we get there. This process is fundamentally different from conventional model training, be it pretraining or post-training. Those are the things you do before you know the problem instance. What we want is an AI that trains and improves on the matter itself.
Over the next few months, we’ll share how we’re automating the specialization loop.
- Inside the Claude Cowork harness, Opus 5 reaches 51% at maximum effort and 38.7% at high effort. This shows that the Cowork harness is better suited to the task, but still falls short of the specialized harness that reaches 63.08% with inexpensive open-weight models and substantially fewer tokens. ↩