Today, we're introducing Decision-Bench. It tests whether frontier AI models can turn realistic enterprise data into decisions that improve the bottom line.

Decision-Bench grew out of our experience helping enterprises turn AI deployments into measurable business results. Its two hundred ten tasks reflect that work, from forecasting costs and investigating fraud to deciding where to cut spending. The benchmark brings four properties together.

First, enterprise scale. Models work across seven point four nine billion rows and two hundred thirty-five tables covering orders, payments, courier compensation, and accounting. Second, real business systems. The warehouse is modeled on Oracle E-Business Suite, with custom operational tables. Models must connect records across the systems a company uses to run its business.

Third, decisions with consequences. Models choose where to cut spending, which accounts to close, and how much to budget. Cutting one expense can raise another. Closing the wrong account can destroy legitimate business. And fourth, measurable outcomes. We grade each submission against what the work is meant to accomplish, including whether a decision saves the business money or leaves it worse off.

Existing data benchmarks stop short of the decisions enterprises need AI to make. BIRD tests whether generated S Q L returns the expected result. Dab step tests multi-step analysis against predetermined answers. Both measure whether a model can answer questions about a business. Neither measures what happens when the business acts on its answer.

Business decisions, however, are less prescribed. A user can specify an objective, such as reducing costs or preventing fraud, without knowing the best action to take. The model has to identify its options, weigh the tradeoffs, and choose. Success depends on what that choice accomplishes. A correct query is one step toward that result, not evidence that the result was achieved.

Benchmarks that evaluate action address part of this gap. Vending-Bench asks agents to make purchasing and pricing decisions and measures their financial results. But running a vending machine does not test whether an agent can make those decisions from billions of records across an enterprise’s financial and operational systems.

Decision-Bench brings the consequences of a decision into an enterprise-scale evaluation. Models must work through interconnected financial and operational records, choose an action, and account for its effects. We measure whether that work achieves the business objective.

Decision-Bench Leaderboard.

The strongest evaluated model reached the solved threshold on thirty-four point eight percent of tasks. Claude Opus five point five leads the version one point one results with a best reported average score of fifty-nine point five out of 100. G P T-6 Astra and Claude Sonnet five point five both averaged fifty-one point eight after rounding.

The leaderboard lists each model at its highest-scoring tested reasoning setting. It shows the average score alongside the share of tasks that reached the solved threshold.

Performance across reasoning settings.

The curves illustrate the results another way: how much performance changes as we give each model more room to reason, and what that improvement costs. Each point represents a tested reasoning setting, plotted against its average score and model A P I cost per task.

Here’s how to read the benchmarks:

Start with one model. Follow its curve through its reasoning settings, such as low, medium, and high. These settings differ across providers, so “high” does not represent the same amount of computation for every model.

Compare performance and cost. A curve that climbs as it moves left shows that extra spending bought a better result. At the same score, the point farther right is cheaper.

Keep score and accuracy separate. A score of 60 means an average of 60 points under the task-specific grading rules. It does not mean 60 percent accuracy or that the model completed 60 percent of tasks.

Check the solved rate. Average scores include partial credit. We count a task as solved when it earns at least 95 out of 100. Hover over a point to see its solved rate, score, cost, and reasoning setting.

The lines connect settings we tested; they do not predict performance between those points. The cost axis includes model A P I charges. We examine warehouse costs separately below.

Overview.

Decision-Bench places models inside a simulated New York City food delivery business operating over the course of 2024. The business gives every task a shared context: customers place orders, couriers make deliveries, restaurants receive payouts, and the company keeps the books.

Each model acts as an agent, using tools to query data, run analysis, and submit work for evaluation. We give models the same tools and instructions so we can compare how they approach the work.

1. Enterprise scale.

The evaluation world contains 81 million orders from three point four million active customers, recorded in seven point four nine billion rows across two hundred thirty-five tables. Agents work across the operational and financial records of the business.

That is a different scale of investigation from many established analysis benchmarks. Dab step’s published payments dataset, for example, contains one hundred thirty-eight thousand transactions. At enterprise scale, the agent’s choices about what to inspect and query affect both its ability to finish the work and the cost of doing so.

2. Real business systems.

A single delivery creates records for the order, the customer’s payment, the courier’s pay, the restaurant’s payout, and the accounting entries behind each. Understanding the delivery means connecting those records.

We designed the warehouse around those connections. Working with three experienced E R P consultants, we modeled it on Oracle E-Business Suite, one of the enterprise resource planning systems companies use to run their operations and keep their books.

The platform also extends its E R P system to support its own operations. Its two hundred thirty-five tables have distinct structures, including 76 custom tables for dispatch, courier incentives, delivery delays, and other parts of the delivery business. Agents have to understand how those custom records connect to the standard financial records.

Raw table count can hide how much of that work a task requires. A warehouse may contain thousands of tables that repeat the same structure across dates or regions. In our analysis of Spider 2.0, three-quarters of the questions with public reference S Q L touched no more than two logical tables after we grouped repeated tables together. The forecasting example in our paper requires six distinct tables across three E R P modules just to reconstruct courier pay before estimating the following month’s costs.

3. Decisions with consequences.

Agents submit work the business could use: forecasts, published dashboard data, account bans, and spending plans. Each submission is evaluated according to its purpose.

The business also changes while those decisions are being made. In April, the platform responds to a minimum-pay increase modeled on the change introduced in New York City in 2024. The policy changes courier scheduling and compensation. An agent forecasting costs for May has to account for that change instead of assuming earlier patterns will continue.

Other tasks require agents to weigh the costs of acting. A fraud investigation can prevent losses, but closing legitimate accounts also damages the business. A spending cut can reduce one expense while creating another. The agent must account for both effects.

4. Measurable outcomes.

Historical records leave some of those effects hidden. They show the fraud a company caught, but not every fraud that occurred. They show what happened under an existing policy, but not what would have happened under an alternative.

The simulator lets the evaluator observe those missing outcomes. For a forecast, the agent might receive records through April while the evaluator can inspect May’s results. For a spending plan, the evaluator can measure how the business responds to the proposed allocation.

The agent still has everything it needs to do the work. Every task has an executable reference solution based only on the warehouse available to the agent. The evaluator’s additional information makes grading possible; solving the task does not require access to it.

Methodology.

World construction.

We built the business using public datasets, regulatory disclosures, company filings, and industry research. The simulator generates every order, courier shift, and payment. No warehouse record is generated by a language model.

We checked the resulting economics against reported New York City figures. The simulation’s per-delivery economics stay within 5 percent in 13 of 16 quarterly comparisons. The exception is courier pay after the April minimum-pay increase, when the simulator runs six to nine percent above the reported figures.

Task construction.

The two hundred ten tasks cover five parts of an analyst’s job. We report results separately so a model’s strengths and weaknesses remain visible.

Forecasting. Estimate future demand, pay, or costs from the available records, and provide a range that reflects uncertainty.

Fraud and risk. Investigate suspicious activity and decide which accounts to ban or payouts to hold, accounting for the cost of acting against legitimate users.

Finance and accounting. Reconstruct balances, reconcile records, and produce financial schedules that agree with the books.

Dashboards. Build and publish the tables behind dashboard views, with the right definitions, level of detail, and values.

Compliance and analytics. Check business activity against requirements such as minimum-pay rules, identify shortfalls, and report operational measures.

Each task has an executable reference solution that establishes the work can be completed from the available records. Agents submit their own work through a common interface called Mission Control.

Evaluation harness.

The evaluation in the paper ran 12 models through a common Inspect AI harness, the software that runs each task and records the agent’s steps.

Inspect AI gives models the same instructions, tools, and execution limits, reducing the chance that differences in setup drive differences in scores. Its records also let us inspect how an agent reached a decision and where the work went wrong.

Models can inspect tables, query data, and execute Python analysis. They work in isolated environments with statistical, machine-learning, and optimization libraries, without internet access. Each run can use up to 500 model turns and three hours.

Grading and private evaluation.

Task-specific rules grade the submitted work, without another language model acting as judge. Dashboard data must contain the required values. Forecasts are evaluated against future outcomes and a reference forecast. Budget plans are scored on their simulated results. Many fraud tasks weigh prevented losses against the cost of incorrect bans.

These rules produce scores from 0 to 100, with partial credit where the task allows it. Each category score averages the tasks in that category; the overall score averages all two hundred ten tasks.

Official evaluations use a separately generated version of the business. Its entities and answer keys differ from those in the public dataset, although the rules governing the business are shared. Memorizing public answers will not reproduce the private world’s results. The simulator, graders, and answer keys remain private.

Results.

The version one point one leaderboard provides the latest headline comparison. The detailed results below come from the evaluation reported in the paper, before those updates.

Performance varies across business areas.

Opus led four of the five reporting categories in the paper’s evaluation; Astra led forecasting. A model’s overall position therefore did not describe its performance on every kind of work.

Those differences matter when choosing a model for a particular job. Reconstructing last month’s financial records tests different capabilities from forecasting next month’s costs. Category results make those differences visible.

More reasoning has uneven returns.

More reasoning improved some models considerably. Opus’s average score rose from forty-one point three at low effort to fifty-six at medium and fifty-nine point three at high. Several other models showed no further improvement above medium effort.

More model calls did not consistently produce better results either. Gemini three point eight Flash and Muse Spark one point three averaged 197 and 174 calls per task, respectively, compared with 54 for Opus, while scoring lower overall.

Those investigations also incurred database costs. G P T-6 Luna used about four cents in model A P I fees per task at high effort, but its total rose to about fifty-two cents when BigQuery scans were included. Across the full experiment, warehouse scans added three thousand eight hundred eleven dollars to the eight thousand six hundred sixteen dollars spent on model A P I calls.

The cost of reaching an answer includes the data an agent reads as well as the model it uses. Comparing token prices alone misses a substantial part of the bill.

Average scores conceal important failures.

Many dashboard submissions looked structurally correct but contained wrong values. Of the submitted data sources that passed the format checks, seventy-two point eight percent failed the numerical checks. A data source passes only when every required value is correct within the specified tolerances.

Forecasts exposed a different problem. We asked agents to provide both an estimate and a range intended to contain the eventual outcome 80 percent of the time. Across two thousand eight hundred ninety-four forecast series, those ranges contained the outcome only forty-one point eight percent of the time.

A business planning staffing or cash requirements needs an estimate and a realistic account of uncertainty. An estimate can be close to the final number while its accompanying range gives a misleading impression of how much the outcome could vary.

We therefore score forecasts on both their estimates and their uncertainty ranges. We also report how far estimates missed and how often their ranges contained the outcome.

A fraud review that cost more than the fraud.

Support noticed that some “never arrived” refund claims came from the same customers, against the same couriers, again and again. The company refunds those claims in full. We asked agents to find the partnerships behind the fake claims and remove both the courier and the customer.

The brief also warned about lookalikes: a courier with a bad week, a customer whose building has a package thief, or a claimant who was right. Those far outnumber real partnerships.

Claude Opus five point five at medium reasoning ran a careful test. It expected about five point five repeat pairs by chance over the year and found seven hundred forty-nine. It noted that most of its couriers had a single partner who had claimed twice, and banned them anyway, “because the pair statistics are just as strong.” It deactivated six hundred seventy-four couriers and closed seven hundred twenty-three customer accounts. Of those couriers, four hundred eighty-seven were honest, and four hundred eighty-five of them had a single regular whose order went missing twice.

At high reasoning, the same model left that group alone, writing that their claims “look more like a package thief or a genuine claim.” It deactivated two hundred forty-four couriers, two hundred forty-two of them in a scheme. Its plan saved one hundred fifty-four thousand nine hundred fifty-three dollars, while the medium plan cost one hundred seventy-five thousand one hundred ninety-eight dollars more than doing nothing. The two runs differed by one dollar and fifty-eight cents in model fees.

Three more in brief.

Half the cost of a referral.

Each referral pays the referrer a bonus and gives the new customer a ten-dollar credit. In nine of thirty frontier runs, agents counted only the referral table and reported about ten dollars. The ledger shows nineteen dollars and ninety cents. Claude Opus five point five found it in all eight runs.

The permit holder was removed.

Two storefronts shared one city permit. Most agents assumed it belonged to whichever joined the app first, and removed the real holder in fifty-one of sixty-two runs. The best run checked the city’s inspection records.

The top ambassador never ordered.

Of sixty frontier runs, fifty-one enrolled a twenty-three-day-old account with no orders and one hundred forty-five referrals in twelve days. Claude Opus five point five ranked it first, at four thousand nine hundred fifty dollars. It referred no one after June.

From benchmarks to better decisions.

Decision-Bench tests whether an agent’s work achieves a business objective. That same evaluation is necessary to make agent-driven decisions a repeatable part of how a company operates. Before delegating a recurring decision, a business needs a way to judge its quality and measure its consequences.

This is the step from conversational analytics and business intelligence toward decision intelligence: agents use data to propose actions, and we evaluate what those actions accomplish. Repeated across workflows, that becomes a decision factory, with a defined objective and a measurable result for each job.

Reinforcement learning closes the loop. The outcomes used to evaluate an agent can also become rewards for training it. A spending agent can learn to reduce total costs, including expenses its cuts create elsewhere. A fraud agent can learn to prevent losses while accounting for legitimate business it disrupts.

Decision-Bench gives us a controlled environment for studying that process. The next step is applying the same discipline to enterprise workflows, so progress means better decisions with measurable business value.

Existing Text Q L customers can get in touch with their account rep to explore evaluating a recurring business decision and training agents against a specific P and L objective.