Is Model Accuracy Enough? A Field Evaluation Of A Machine Learning Model To Catch Bogus Firms
We investigate the use of a machine learning (ML) algorithm to identify fraudulent non-existent firms. Using a rich dataset of tax returns over several years in an Indian state, we train an ML-based model to predict fraudulent firms. We then use the model predictions to carry out field inspections of firms identified as suspicious by the ML tool. We find that the ML model is accurate in both simulated and field settings in identifying non-existent firms. Withholding a randomly selected group of firms from inspection, we estimate the causal impact of ML driven inspections. Despite its strong predictive and field performance, the model driven inspections do not yield a significant increase in enforcement as measured by the cancellation of fraudulent firm registrations and tax recovery. We provide two rationales for this discrepancy based on a close analysis of the tax department’s operating protocols: selection bias, and institutional friction in integrating the model into existing administrative systems. Our study serves as a cautionary tale for the application of machine learning in public policy contexts and of relying solely on test set performance as an effectiveness indicator. Field evaluations are critical in assessing the real-world impact of predictive models.
-
-
Copy CitationTaha Barwahwala, Aprajit Mahajan, Shekhar Mittal, and Ofir Reich, "Is Model Accuracy Enough? A Field Evaluation Of A Machine Learning Model To Catch Bogus Firms," NBER Working Paper 32705 (2024), https://doi.org/10.3386/w32705.Download Citation
Published Versions
Taha Barwahwala & Aprajit Mahajan & Shekhar Mittal & Ofir Reich, 2024. "Is Model Accuracy Enough? A Field Evaluation of a Machine Learning Model to Catch Bogus Firms," ACM Journal on Computing and Sustainable Societies, vol 2(3), pages 1-43.
Related
Programs
More from the NBER
In addition to working papers, the NBER disseminates affiliates’ latest findings through a range of free periodicals—the NBER Reporter, the NBER Digest, the Bulletin on Health, the Bulletin on the Economics of Alzheimer's, and the Bulletin on Entrepreneurship—as well as research spotlights, lectures, and panel videos.
