Does LLM Assistance Improve Healthcare Delivery? An Evaluation Using On-site Physicians and Laboratory Tests

Jason Abaluck; Robert Pless; Nirmal Ravi; Anja Sautmann; Aaron Schwartz

doi:10.3386/w34660

Does LLM Assistance Improve Healthcare Delivery? An Evaluation Using On-site Physicians and Laboratory Tests

Jason Abaluck, Robert Pless, Nirmal Ravi, Anja Sautmann & Aaron Schwartz

Working Paper 34660

DOI 10.3386/w34660

Issue Date January 2026

We deployed large language model (LLM) decision support for health workers at two outpatient clinics in Nigeria. For each patient, health workers drafted care plans that were optionally revised after LLM feedback. We compared unassisted and assisted plans using blinded randomized assessments by on-site physicians who evaluated and treated the same patients, as well as results from laboratory tests for common conditions. Academic physicians performed blinded retrospective reviews of a subset of notes. In response to LLM feedback, health workers changed their prescribing for more than half of the patients and reported high satisfaction with the recommendations, and retrospective academic reviewers rated LLM-assisted plans more favorably. However, on-site physicians observed little to no improvement in diagnostic alignment or treatment decisions. Laboratory testing showed mixed effects of LLM-assistance, which removed negative tests for malaria but added them for urinary tract infection and anemia, with no significant increase in the detection rates for the tested conditions.

The authors gratefully acknowledge the research assistance of Nina (Zi Wei) Low, Lisa Timm, Eduardo Arancón Simal, Grady McPeak, Kylie Kim, Owen Hart, and Tedi Totojani. We thank Fatimah Howeidy, Attat Yohannes, Adham Hallal for project support, Ogochukwu Maduabum and George Ohia for building the electronic medical record. Drs. Fatimah Tsiga- Ahmed, Rabiu Jalo and Aminatu Kwaku reviewed the patient notes. This research was funded by the Gates Foundation Grand Challenges Initiative (INV-081875) and a World Bank Research Support Budget grant. Dr. Schwartz received additional support from the University of Pennsylvania Matt Slap Research Award and the Department of Veterans Affairs Career Development Award initiative (IK2HX003634). This trial was pre-registered in the AEA RCT registry under trial number AEARCTR- 0015226 and approved by the Yale University IRB (protocol ID 2000035990) and the Kano State Health Research Ethics Committee (SHREC/2024/5464). The findings, interpretations, and conclusions expressed in this article are entirely those of the authors and do not represent the views of the World Bank and its affiliated organizations, nor of the Executive Directors of the World Bank or the governments they represent, nor or the U.S. government or the Department of Veterans Affairs, nor of the National Bureau of Economic Research.

Aaron Schwartz
Dr. Schwartz reports that a sibling of his is an employee of OpenAI. The sibling and the company played no role in the funding, design, execution, or writing of the research study.
Copy Citation

Jason Abaluck, Robert Pless, Nirmal Ravi, Anja Sautmann, and Aaron Schwartz, "Does LLM Assistance Improve Healthcare Delivery? An Evaluation Using On-site Physicians and Laboratory Tests," NBER Working Paper 34660 (2026), https://doi.org/10.3386/w34660.

Download Citation

MARC RIS BibTeΧ
- randomized controlled trials registry entry

Does LLM Assistance Improve Healthcare Delivery? An Evaluation Using On-site Physicians and Laboratory Tests

Related

Topics

Programs

More from the NBER