Specialization is powerful. Legal work is diverse, and there are huge benefits to specialization. Unfortunately, the economics are against it. Building a specialized tool today takes a team of forward-deployed engineers and domain experts to handcraft a custom solution, one that depreciates as soon as the model frontier advances. What’s worse, no one can staff a team for every problem. So teams build for a general workflow, not for the specific problem in front of them, and the result is never quite tailored enough.
The only way to get ahead of the problem is to automate specialization itself. Once specialization scales, we can apply it at the most atomic level: the individual legal matter. At grubel, we’re automating the specialization loop for legal work. Our goal is to give every legal matter its own AI, one that works best for the problem at hand.
The data piece
One of the most important pieces of the loop is data: specifically, realistic environments used to train, evaluate, and optimize agentic AI systems. Agentic legal environments are hard to build. The documents a lawyer works from (the data room) are usually not public, and expert legal judgment is needed both to create tasks and to evaluate the deliverables. The usual workarounds each give something up: synthetic data rooms compromise the realism of the tasks, and having experts write tasks and rubrics makes them expensive and hard to scale.
At grubel, we’ve believed in the importance of real data from the start, and we’ve looked for creative ways to gather, refine, and use it at scale, largely automatically.
Today we’re excited to announce a significant step in our data efforts. In collaboration with Harvey, we’re building agentic environments from real federal litigation, which Harvey also announced today. We’ll open-source the pipeline, so that anyone can build environments for the courts and the kinds of litigation they care about.
Agentic legal environments from real data
Lawsuits that a court has already decided provide the ingredients for building legal environments: a realistic task, a data room, and expert briefs and judgment from which rubrics can be built. A federal docket holds the documents the lawyers worked from, the briefs they filed, and the ruling in which the judge decided the dispute. Everything needed to build a task and to grade it is already in the record.
In the spirit of SWE-bench, which builds coding tasks from resolved GitHub issues and checks them against the real fix, we build legal tasks from decided motions and check them against the real filings. Because the ground truth already exists, environments can be built largely automatically, which is exactly what an automated loop needs.
All of this rests on the Free Law Project, whose CourtListener and RECAP archives make federal court records accessible. We are grateful for their work.
An upcoming note describes the methodology we have developed to build these environments. It builds a task for each role around a motion to dismiss (defense counsel, plaintiffs’ counsel, and the law clerk who drafts the bench memo) and grades each deliverable against rubrics derived from the briefs counsel actually filed and the ruling the court actually wrote. Harvey, in contrast, built a neat way to grade briefs, ranking them head to head with its own pairwise grader. A ranking tells you which system is better. Grading against the real work product tells you what a system missed, argument by argument. We’ll discuss both evaluation methodologies in upcoming work with Harvey.
Looking ahead
Data is only one part of the loop. Over the coming weeks, we’ll share what we’re building for the other parts.
Harnesses shaped for legal work
Today’s agent harnesses work great for coding, but they bring coding habits to legal work: they truncate long documents, page through them, and keep everything in one growing context. Even when the documents fit, models fight the harness to read them, and by the time they write, they stop one step short. In our runs, most of the criteria a strong model failed were issues it had found but did not carry through to the relevant argument or deliverable.
Our second upcoming note describes a harness we built from the ground up for legal work, in the spirit of MapReduce. Fresh subagents each take a part of the task and write their work to a shared workspace; a fresh agent then reads all of it together and writes the deliverable. On a curated version of Harvey’s Legal Agent Benchmark, it reaches the highest all-pass rate we know of. And where more reasoning effort did not reliably help in a coding harness, in ours it does.
Surgical model specialization
Legal data rooms can hold thousands of documents and tens of millions of tokens. No agent keeps all of that in its context, so the question is how to make sure it sees what matters. Having a general agent repeatedly scan the whole room for relevant pieces is slow and expensive.
Our third upcoming note describes early research results on specializing models for legal work. We specialize a decision model to improve retrieval quality in legal work and show how this improvement lifts the quality of the deliverables. Rather than writing a judgment about each passage, as a general LLM would, a decision model chooses from a fixed set of options, here how significant the passage is for the task. We start from an open decision model and fine-tune a small set of adapters on a handful of legal environments.
We integrate the decision model into our legal harness to work effectively with large data rooms. On due-diligence tasks from Harvey’s Legal Agent Benchmark, with data rooms of thousands of documents, it selects the critical legal evidence far better than both general-purpose LLMs and off-the-shelf decision models, at up to an order of magnitude less cost and time than having an LLM read the room. This demonstrates that targeted adaptation of a model can lift the performance of the agentic system at low cost.
Closing the loop
Once the pieces are in place, we can begin to close the loop. Environments built automatically from real records let a system adapt its model and its harness to a matter, check the result, and iterate, without a team rebuilding it each time. That is the loop we are closing.