Designing Human-AI Collaboration: A Sufficient-Statistic Approach

We develop a sufficient-statistic approach to designing collaborative human-AI decision-making policies in classification problems, where AI predictions can be used to either automate decisions or selectively assist humans. The approach allows for endogenous and biased beliefs, and effort crowd-out, without imposing a structural model of human decision-making. We deploy and validate our approach in an online fact-checking experiment. We find that humans under-respond to AI predictions and reduce effort when presented with confident AI predictions. AI under-response stems more from human overconfidence in own-signal precision than from under-confidence in AI. The optimal policy automates cases where the AI is confident and delegates uncertain cases to humans while fully disclosing the AI prediction. While both automation and human judgement are valuable, the incremental benefit over selective automation of assisting humans with AI predictions is negligible. The sufficient-statistic approach accurately predicts the performance of out-of-sample collaboration policies, suggesting it can be a useful guide to designing collaborative systems.