OpenAI's Decisions API Won't Make Your AI Agent Cheaper by Itself

OpenAI's Decisions API went to general availability on October 22, priced on input tokens only^[2]. That puts OpenAI in the category TypeSafe's Jev started: models that pick from a fixed set of answers and return a confidence score, fast and cheap^[1].

The pitch is easy to follow. Swap the expensive LLM calls that only choose something for a cheap judge, and your AI system gets faster and cheaper.

A controlled test published five days before the launch points somewhere less tidy. One researcher put Jev inside a coding agent, and the agent cost 19% more per attempt and passed exactly as often^[3]. Jev was the best judge in that test. The extra cost came from where it was asked.

What is a decision model in AI?

A decision model is a model that does not write text. You give it a question and a fixed set of allowed answers, and it returns a choice with a probability for each option. It works as a judge, not a writer, and that is why it is cheap: there is no long reply to generate.

TechCrunch describes them as super classifiers: given a set of options, they "can quickly make choices and offer a confidence rating in their decision"^[1]. TypeSafe's Jev created it. OpenAI's version runs on its Luna model, takes text, images and voice, and costs $0.10 per million input tokens with no output charge^[2].

The speed is real. Speak, which builds AI language tutors, told TechCrunch the API cut the time between actions in its app from 1.4 seconds to 0.13 seconds^[1]. For routing a request or classifying an input, that is a good trade, and nothing below argues otherwise.

The open question is what happens when the judge sits inside an agent's loop.

Do decision models make AI agents cheaper?

Not by themselves. In the only controlled test so far of a decision model steering a coding agent, consulting Jev at fixed checkpoints raised the cost per attempt by 19% and left the pass rate unchanged. The judge was good. The checkpoints it was asked at did not turn its answers into savings.

The study is Naman Vats's "Good Judgment, Bad Advice"^[3]. He held one coding agent fixed (Claude Sonnet working in a command line, on Terminal-Bench, a standard test of that kind of work) and changed only the controller that weighed in at four moments:

• after two or more failed commands in a row
• when the agent repeated itself with no progress
• when the agent asked to finish
• at a checkpoint every 10 steps

Five arms ran on 21 held-out tasks, 315 trials in all: no controller, simple rules, a small LLM, Jev, and a placebo that answered at random in Jev's usual proportions. Against no controller, Jev raised cost per attempt by 19% at an identical 73% pass rate. The placebo did not raise cost at all.

The sample is small, and Vats says so: 21 tasks, mostly easy ones, one agent loop and one decision model. The claim it supports is narrow. A good judge, asked on a schedule, made this agent more expensive.

Why did the better judge cost more?

Jev judged well and acted badly because of where it was asked and what it could choose. It ranked the agent's finish requests better than rules or a small LLM, but its confidence ran low, so it kept asking for checks on work that was already right. And it never chose the one answer that saves money: stop.

Scored offline against how each run actually ended, Jev ranked finish requests at 0.78 on a standard ranking score, against 0.62 for rules and 0.60 for the small LLM.

But its average confidence that a run was done was 0.50 when 87% of those runs had passed, so it chose "verify" on 29 of 53 finish requests^[3].

Every one of those verifications cost a model call and changed no outcome.

The money a stopping rule could save, ending a run that is going to fail, never showed up.

A second paper explains why timing matters so much. Taste-Bench, from researchers at City University of Hong Kong and Microsoft, pulls 502 decision forks out of real agent runs and asks models to pick the better branch^[4]. The best model gets 59.7% right.

When the evidence that settles a fork only shows up later in the run, the average across 14 models falls to 21%, near the 25% you would get by guessing, and a bigger reasoning budget does not help.

A judge asked before the evidence exists is guessing. A judge asked every 10 steps is often asked before the evidence exists.

Where does a decision model pay off?

The same Taste-Bench paper shows the other result. A small model trained on a teacher that had seen how runs ended gave advice at each fork before the work started, and an agent's success on 41 real software tasks rose from 14.6% to 33.7%^[4]. Same kind of component, opposite outcome.

Perfect advice would have reached 39%, so the trained judge captured most of what advice can do. The difference from the Vats setup is not the price of the call. It is that the evidence was in view and the answer changed what the agent did.

Put the three cases side by side and the pattern is about placement.

This is the same lesson we keep finding in agent work: the harness decides more than the model. A cheaper component dropped into the wrong place can make the system worse, the way a better model can be worse at your tools.

Is Jev better than an LLM as a judge?

Jev and its peers are far cheaper and more consistent than LLM judges, but not less biased. A study of eight typed decision models (the research name for the category) found every one biased on three to eight of twelve standard bias tests, while costing about 37 times less than 2026 LLM judges.

Cheap and neutral are different properties.

That study, from researchers at the University of Göttingen and Bielefeld University of Applied Sciences, tested these models as judges in the evaluation sense: comparing two answers and picking the better one^[5]. That is a different job from steering an agent.

It still says something about trusting the output. Rewriting the better answer in an emotional tone dropped a typical model's accuracy from 99% to 42%.

The authors still land on a practical yes: these models "can be attractive judges if their biases are tested for the task"^[5]. The price advantage is real. So is the need to check it on your own inputs before it decides anything that matters.

Where should a decision model sit in an agent harness?

Put a decision model at a decision whose answer changes what the system does next, and ask it when the evidence is already in view. Stop or continue, finish or verify, send to a person or not. A judge on a timer, or one whose answer the agent can ignore, adds calls without changing outcomes.

For a team adding one to an agent, four checks follow from the research:

• Pick decisions that change control flow. Stopping a doomed run saves money; a note the agent may ignore only adds a call.
• Ask after the evidence arrives. A test result or a failed command is evidence. A step count is not.
• Log the confidence and check it. Vats found a model that ranked well and was miscalibrated; only logged probabilities against real outcomes show that.
• Measure cost per finished task. A cheaper call that adds calls can raise the bill, which is why we argue for cost per finished task, not cost per token.

The pricing story will keep moving. OpenAI is new to the category, and TechCrunch reports public rankings that put Jev seventh and third on two decision indexes, with OpenAI's model eighth and fourteenth^[1].

Placement is the part that does not change with the price list. The model got cheaper; deciding where it sits is still your call, and that decision sets the bill.