Business Utility Testing for LLM Data Analysis Agents

LLM-based data analysis agents need evaluation beyond average accuracy because inconsistent exploratory results increase verification cost and weaken managerial trust. The paper benchmarks 15 model variants across a simulated consumer-goods supply chain task where agents infer supplier-product causes of quality and sales loss from indirect traces. It compares mean score, coefficient of variation, condition sensitivity, and a proposed Business utility metric that discounts performance by instability. Technology leaders should evaluate EDA agents for repeatability and robustness before allowing autonomous analytical use.