Passing the bar exam is the gate to practicing law in almost every U.S. state, but a study of new lawyers in Nevada found that their scores had little to do with how effective they later proved to be in the eyes of supervisors, peers and judges.
The study was led by Jason M. Scott of the AccessLex Institute and Stephen N. Goggin of San Diego State University, with co-authors from the Nevada Board of Bar Examiners, the State Bar of Nevada, UC College of the Law San Francisco, the University of British Columbia and UNLV. It was published open access in the Journal of Law and Empirical Analysis in November 2024. The authors describe it as the first study to measure the link between bar scores and new lawyers’ effectiveness.
Bar results matched with workplace ratings
The team obtained first-attempt bar scores for all 1,414 lawyers admitted in Nevada from 2014 through early 2020 who had not been licensed elsewhere. That included the 200-question multiple-choice Multistate Bar Examination (MBE), the Multistate Performance Test and Nevada’s own essays, one of them on legal ethics.
In 2021, 524 of those lawyers took part in a survey, a 37% response rate, and named supervisors, peers and judges who knew their work. Those raters scored each lawyer on 26 lawyering skills developed by Marjorie Shultz and Sheldon Zedeck, from research and writing to client relationships, conflict resolution and character. Most participants, 453, got at least one outside evaluation, and lawyers also rated themselves on five skills.
The analysis controlled for age, gender, race and ethnicity, and time since taking the exam.
Small effects, mostly not practical
On their own, first-time bar passage or bar score explained no more than 12% of the variation in any of the effectiveness ratings. The MBE explained no more than 10%, the essays no more than 2% and the performance test no more than 3%.
Lawyers who passed on their first try got ratings about 1% higher from themselves and from peers, 5% higher from supervisors and 6% higher from judges. Only the supervisor difference was statistically significant.
A one standard deviation higher overall bar score went with no change in peer ratings, 2% higher supervisor ratings and 4% higher judge ratings, the largest effect in the study. For the ethics essay, peer ratings actually dipped by 1%.
“Our results suggest that the bar exam does not meaningfully predict the demonstration of the competencies to be an effective lawyer,” the authors write. Some results reached statistical significance, they note, but significance alone does not mean an effect is large enough to matter.
Why it matters for who gets licensed
The authors point to large gaps in pass rates. In 2021, white law graduates passed on their first attempt at a rate 24 percentage points higher than black graduates, according to American Bar Association data. A test that limits entry so unevenly, they argue, should face frequent validity testing.
They also note that time since the exam was positively linked to effectiveness, though they caution against reading much into a control variable. That fits with recommendations from a Nevada Supreme Court commission, which the authors support: an early knowledge test after the first year of law school, a Nevada-built performance test and a period of supervised practice, much like a medical residency.
Study Details:
- Title: Putting the Bar to the Test: An Examination of the Predictive Validity of Bar Exam Outcomes on Lawyering Effectiveness
- Authors: Jason M. Scott, Stephen N. Goggin, Rick Trachok, Jenny S. Kwon, Sara Gordon, Dean Gould, Fletcher S. Hiigel, Leah Chan Grinvald, David Faigman
- Journal: Journal of Law and Empirical Analysis, Vol. 1, No. 2
- Publication Date: November 2024
- DOI: 10.1177/2755323X241296683