AI Agents and Prompt Engineering in Econometric Coding
We study how large language models write the code for statistical and econometric analysis. We assign six major models 40 different tasks in Stata, R, and Python, with different prompts and different degrees of agency. The prompt is either only the statement of the task or a prompt that has a related code example. The least agentic version, the chatbot, just writes the code, and a more agentic version can run and revise its code and decide when to stop. We find that more agency raises success from 84 to 97 percent. Letting the chatbot revise its script achieves most of this gain. The example and the language matter more for the chatbot than for the agent, which also narrows the differences between models. An agent’s runs cost 3.4 times as much as chatbot runs, and the cost of an additional correct answer ranges from $0.02 to $1.13, depending on the model. The agent removes almost all loud failures, which occur when the script does not run or does not return every result. On the other hand, it removes only about half of the silent failures, those in which the script runs but returns a wrong value. As a result, most of the failures that remain are silent, so researchers who use AI agents still need to check the code, even when it seems to work.
-
-
Copy CitationSebastian Galiani, Federico Ariel López, and Raul A. Sosa, "AI Agents and Prompt Engineering in Econometric Coding," NBER Working Paper 35588 (2026), https://doi.org/10.3386/w35588.Download Citation
-