Law students who reviewed legal complaints with AI highlighting finished about 30% faster, and their answers were just as accurate. An AI-written summary, on the other hand, saved no time at all, and giving students both tools was slower than highlighting alone.
The experiment was run by Aileen Nielsen of Harvard Law School, Stavroula Skylaki and Milda Norkute of Thomson Reuters Labs in Zug, Switzerland, and Alexander Stremitzer of ETH Zurich’s Center for Law and Economics. It was published online in the Journal of Empirical Legal Studies on November 17, 2024, and appeared in the December 2024 issue (read the paper).
Four ways to read a complaint
Between January and April 2022, the team recruited 206 students from nine US law schools, ranging from a top-five school to one ranked in the 70s. Each student read three legal complaints in a custom interface and answered questions about their contents and quality.
Students were randomly assigned to one of four setups: no AI, an AI-generated one-sentence summary of each complaint, AI-generated highlighting of passages within the complaint, or both. The AI came from a commercial tool Thomson Reuters already uses to help attorneys write one-sentence summaries of complaints. ETH funded the study, and the authors say their institutions contracted for full academic freedom to publish the results.
Highlighting saved six minutes, summaries saved none
Students with no AI took an average of 1,338 seconds, a little over 22 minutes, to work through the three complaints. Those with highlighting only took under 16 minutes, a 29.7% reduction. Students with the summary alone were no faster than those without AI, and students with both tools landed in between, at about 19 minutes.
Speed did not come at the cost of quality. Accuracy on the factual questions was high and nearly identical across groups, from 85.4% with both tools to 87.7% with the summary only, and the authors found no change in any of the quality indicators they measured.
The authors call the highlighting result surprising. The highlights were never designed to help human readers; they are a byproduct of training the model to write summaries, and they could easily have been a distraction. Students with highlighting also did no better on questions whose answers happened to be highlighted, which suggests the tool did not work simply by pointing to answers. The authors think it worked through a mix of cognitive and emotional effects, and found no sign that it made students overconfident.
Too much help can slow people down
The slower result for the combined setup is the study’s warning: more AI is not automatically better, and the form of assistance needs fine-tuning. The authors note that their participants were law students, and earlier studies have found that students and judges do not always behave alike in legal tasks.
They also point out that, at the time of writing, popular consumer chatbots did not offer this kind of in-document highlighting. Their recommendation to legal AI developers is direct: build it.
Study Details:
- Title: Building a better lawyer: Experimental evidence that artificial intelligence can increase legal work efficiency
- Authors: Aileen Nielsen, Stavroula Skylaki, Milda Norkute, Alexander Stremitzer
- Journal: Journal of Empirical Legal Studies
- Publication Date: November 17, 2024
- DOI: 10.1111/jels.12396