BlogNovember 22, 2026Agentic AI · Reinforcement learning14 min read
Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents
Decision models answer a question with a choice and a probability, so the same model can pick an agent's actions, grade them, or teach a policy. We find that a model which acts poorly can still teach well, as long as it is asked the right question and shown the whole trajectory.

Contents
TL;DR
A decision model answers a question about a context with a typed output, such as a choice among options with a probability for each, instead of free text. Because any question can be asked about any state, one model can choose an agent's next action, evaluate an action already taken, or score every action of a policy during reinforcement learning. So far these models have mostly been judged by how well they act.
We asked whether a model that acts poorly can still teach well. On ALFWorld, JEV completes 42.9% of tasks when it chooses the actions itself. Used instead to score each action of a 1.5B policy during GRPO training, it raises that policy's success over GRPO alone from 72.8% to 82.9% on the same tasks, and from 56.8% to 71.4% on WebShop. The difference comes from timing: an actor must anticipate consequences, whereas a judge that reads the finished episode can see them.
The benefit is not automatic. It appears only when the model is asked whether each action helped achieve the goal, rather than whether the task will succeed, and only when it sees the whole trajectory rather than the steps leading up to the action. Queried this way after each episode, a frozen decision model provides step-level supervision for 22 to 30 USD of API calls per training run, with no step labels and no learned critic.
1Three ways to ask a decision model
A decision model takes a question and a context. For a multiple-choice question it selects an option and assigns a probability to every option; for a yes/no question it returns one score in [0, 1]. We query the same model in three roles that differ only in what it is asked and how its answer is used.
- Actor. At each step the model picks the next action from the valid commands, given the goal, the history and the current observation. The choice is executed.
- Judge. The model scores each action of a recorded trajectory. We vary what it sees, the trajectory up to the action (Prefix) or the whole trajectory (Hindsight), and what it is asked, whether the task will succeed after the action (Value) or whether the action helped achieve the goal (Progress).
- Teacher. The model scores the actions of the policy being trained, and the scores enter GRPO as step-level credit. It stays frozen and is not used at test time.
Figure 1 plays the three roles on the paper's running example: put a heated mug on a table.
Actorpicks the next action
- A
place mug on table - B
go to microwave - C
look… other valid actions
QuestionWhich action should be executed next?
Judgescores a recorded action
Recorded episode: put a heated mug in the cabinet
- 6
move mug to cabinetplaced too early - 7
take mug from cabinet - 8
go to microwave - 9
open microwavetarget - 10
heat mugmilestone - 12
move mug to cabinetgoal complete
QuestionDid this action help achieve the goal?
Teacherscores the policy's own rollouts
πθPolicy being trainedupdated
take mug
take againno-op
heat mugmilestone
place ✓
2
place cold mug
take mug back
heat mug
place ✓
Question, for every actionDid this action help achieve the goal?
Decision model
question + context → typed answer. Here JEV, one of the six models we test.
next_actioncontext goal · history · observationanswer one option + a probability for eachprogresscontext trajectory up to step 9 (Prefix)answer score in [0, 1]progresscontext whole trajectory (Hindsight)answer score in [0, 1]progress, every actioncontext whole rollout (Hindsight)answer one score per actionChoice probabilities
place mug on tableplaced ✓heated ✗Easier subgoal first: the task is left incomplete.
Score for step 9, open microwave
Same question; seeing the mug heated shows why it mattered.
Teacher scores group mean 0.71
The no-op is pushed down; taking the cold mug back is credited.
We evaluate six models that answer questions in this form. Three are released as dedicated decision models and are queried through their own decision APIs. The other three are general LLMs asked the same questions, either by prompting for a JSON answer or by reading the first-token probabilities of the valid answer labels. Two of these general LLMs are the base models of Decider v1.2 and GPT Decisions, so each dedicated model can be compared with its own base model.
| Model | Base LLM | How the decision interface is obtained | Access |
|---|---|---|---|
| JEV | undisclosed | undisclosed | API only |
| Decider v1.2 | Qwen3.8-27B | full fine-tuning | open weights |
| GPT Decisions | GPT-6 Luna | undisclosed | API only |
| Qwen prompted | Qwen3.8-27B | prompting (JSON answer) | open weights |
| Qwen constrained | Qwen3.8-27B | constrained readout (first-token probabilities) | open weights |
| GPT Luna | GPT-6 Luna | prompting (JSON answer) | API only |
We use two text environments in which some requirements must be met in a fixed order. In ALFWorld [1], an agent completes household goals with commands such as go to, take and heat. Its Clean, Heat and Cool tasks have a dependency: the object's state must change before it is placed. Pick2 also has two requirements, two objects to place, but they can be met in either order. In WebShop [2], an agent searches a catalogue and must select the required options before it buys.
2As an actor, an easier subgoal gets in the way
As actors, all six models struggle when one subgoal must be completed before another: an easier subgoal draws them away from the one that has to come first.
Ordered subgoals are where acting breaks
Every model succeeds far less often on tasks with a dependency than on tasks without one (Figure 2). JEV completes 70.8% of Pick2 tasks but at most 18.8% of each dependent task type. Decider v1.2, the strongest actor overall at 63.6%, completes only 18.8% of Heat tasks. Because Pick2 also has two subgoals, the gap points to their order rather than their number as the difficulty.
ALFWorld, acting
Actor success rate by task type
The failed episodes show how the order is broken. Holding the mug at the table, the actor can place it in one action, or walk to the microwave, heat it and come back. In many failures it places the unheated mug and the task can no longer be completed as intended.
Distraction, not ignorance
We collected 116 states, from 66 tasks, in which a single action would satisfy the easier subgoal while the required one was still unmet, and queried each actor there under two goals that differ only in the easier subgoal, for example “put a heated mug on the side table” versus “heat a mug”. Everything else, including the history, observation, candidate actions and remaining budget, is identical.
With the full goal, JEV puts 63.8% of its probability on the easier subgoal and 14.6% on the step that must come first; without it, these become 2.5% and 77.6% (Figure 3, left). All six actors shift the same way. They know the required step; the easier subgoal pulls probability away from it. Reminding them of the dependency barely helps: a generic hint changes success by −2.8 to +5.2 points, and adding a worked ordering example helps some models but not all (Figure 3, right).
JEV, 116 states
Where the probability goes
Share of JEV's probability by action type, %
Same 116 states, continued to the end
Success with an ordering example in the prompt
Original prompt → hint plus a heated-mug example, %
Acting correctly here requires anticipating that the easy action leaves the task unfinished before any consequence is visible. A judge evaluates the action after its consequences have occurred, which is where the next section starts.
3As a judge, the question and the context decide
The same models can recognise contributions they overlook as actors. Seeing the whole trajectory reveals how earlier actions feed later progress, and asking whether each action helped separates those actions from ones that change nothing.
We reused the successful continuations from the 116 states and asked each judge whether the first action taken from the diagnostic state helped achieve the goal, with outcome labels masked. Seeing the whole trajectory instead of the prefix raises the mean score of these actions for all six models; for JEV it goes from 0.41 to 0.59. That raises two questions: does a full trajectory simply make every action look good, and what does the judge actually gain from it?
Ask whether the action helped, not whether the task will succeed
We labelled actions in 462 ALFWorld trajectories as milestones, which achieve a subgoal, no-ops, which leave the environment unchanged, and other. With the whole trajectory visible, the Value question gives the three groups almost the same score: if the episode ends in success, every action in it “leads to success”. The Progress question spreads them apart for every model (Figure 4). For JEV, within a trajectory, Progress scores milestones 0.36 above no-ops, against 0.05 for Value, and it ranks more milestones among each trajectory's top-scored actions, by 8.0 to 16.0 points.
ALFWorld, 462 trajectories, Hindsight context
Mean action score by action type
Show the whole trajectory
To see what the extra context buys, we held the question fixed to Progress and revealed the trajectory step by step: the prefix, then the immediate feedback after the action, then 25%, 50%, 75% and all of what follows. Over 167 trajectories, JEV's ranking of actions within a trajectory moves by 1.95 positions on average once feedback is added, and by 3.30 positions with the full trajectory. Figure 5 shows one of them. Drag the slider, or press play.
JEV as a judge, Progress question
“Put a heated mug in the cabinet”: scores as more of the episode becomes visible
Under the prefix, JEV rates the premature placement (step 6) at 0.79 and going to the microwave (8) and opening it (9) at only 0.23 and 0.20, the same preference for the easier subgoal that the actor showed. Immediate feedback makes it worse: placement rises to 0.90. Only as the continuation comes into view do the microwave steps earn credit; opening the microwave reaches 0.66 once heating is visible and 0.76 with the full trajectory. The repeated look (step 2) stays low throughout, because the prefix alone already shows that it repeats step 1.
The two dedicated decision models whose base LLM is disclosed, Decider v1.2 and GPT Decisions, also judge more accurately than that base LLM given the same question, at 28% to 43% lower cost per score.
4As a teacher, the best judge gives the best signal
The judgments that best separate useful from useless actions also train the strongest policies, and the gains are largest on the tasks the teacher fails as an actor.
From scores to advantages
GRPO gives every action in a rollout the same outcome advantage. For each task the policy samples a group of trajectories; after they finish, the teacher D scores every executed action t of every trajectory τi from the goal and the trajectory alone, with no rewards, success labels or hidden state:
The scores are centred on the mean s̄ over all scored actions in the group and added with weight λ = 0.5. They are deliberately not divided by their standard deviation, so a group in which the teacher hardly distinguishes actions receives only small adjustments. Figure 6 shows what the teacher term does to the two rollouts from Figure 1.
Illustrative group of two rollouts
What the teacher adds to each action's advantage
Results
We train Qwen2.5-1.5B-Instruct with JEV as the teacher under all four combinations of question and context, and compare against GRPO with outcome rewards alone (results from GiGPO [3]). Each policy is evaluated with three seeds. Hindsight + Progress gives the highest success on both environments: 72.8% → 82.9% on ALFWorld and 56.8% → 71.4% on WebShop. It improves on every ALFWorld task type, most on Cool (+23.0) and Look (+20.7). With the prefix only, the two questions perform about the same. The ordering matches the judge analysis: under Hindsight, the Value question gives every action of an episode nearly the same score and so says little about which actions helped.
Qwen2.5-1.5B-Instruct, mean of three seeds
Success rate, teacher-trained policy vs GRPO
The gain holds at 7B, from 77.6% to 83.8% on ALFWorld and from 66.1% to 73.6% on WebShop, and the 1.5B policy taught this way outperforms the 7B policy trained with outcome rewards alone on both environments. Hindsight + Progress also finishes sooner: it completes 74.5% of ALFWorld tasks within 20 actions, against 63.6% to 69.5% for the other settings, and 71.1% of WebShop tasks within 10 actions, against 60.9% to 62.2%.
Learning beyond the teacher
On the same ALFWorld evaluation tasks, JEV completes 42.9% as an actor; the 1.5B policy trained on its scores completes 82.9%. The largest gains are on the dependent tasks, the ones JEV handles worst as an actor: there the policy reaches 86.8%, 6.9 to 16.2 points above the other question and context settings, while JEV itself completes only 14.7% of them (Figure 8). The teacher does not need to solve a task to provide a useful signal for it.
ALFWorld, same evaluation tasks
JEV acting vs the 1.5B policy JEV taught
Cost. We query JEV through OpenRouter at 0.042 USD per million input tokens to score every executed action during training: 150 steps, 16 tasks per batch, 8 rollouts per task. Scoring an entire training run costs between 22.21 and 30.07 USD.
| Hindsight + Progress run | ALFWorld 1.5B | ALFWorld 7B | WebShop 1.5B | WebShop 7B |
|---|---|---|---|---|
| Max actions per rollout | 30 | 30 | 15 | 15 |
| Teacher API cost (USD) | 30.07 | 24.50 | 22.21 | 26.83 |
5Takeaways for using decision models in agents
- Actors know the required step but are drawn to the easier one. They choose the right step when it is the only subgoal, and general reminders do not stop the error. When a decision model must act, restructuring the goal into ordered subgoals is more promising than adding instructions.
- The question sets the granularity of the signal. Asking about the eventual outcome gives every action of a trajectory a similar score and largely repeats the outcome reward; asking about each action's contribution separates them.
- Hindsight targets the actor's own failure. Seeing the whole trajectory lets a judge credit the steps toward the subgoal that must come first. Decision models are best used to score actions after an episode, as in RL training, and are less suited to filtering actions online, where only the prefix exists.
- Screen judges before training. Judging recorded trajectories identified the question and context that trained the best policy, at a small fraction of the cost of an RL run. Choose a teacher by how well it judges, not by how well it acts.
- Step-level supervision becomes affordable. A frozen decision model, queried after each episode and absent at test time, costs tens of dollars per run and lets a small policy outperform a larger one trained on outcomes alone.
Citation
@article{liu2026actor,
title = {Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents},
author = {Liu, Haochen and Zhu, Qianhao and Wang, Zixuan and Li, Dingming and Chen, Tongbo and Qiu, Peng and Qu, Yingqi and Lu, Weiming and Xiao, Jun and Shen, Yongliang},
journal = {arXiv preprint arXiv:2611.17167},
year = {2026}
}References
- Shridhar et al. (2021). ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. ICLR.
- Yao et al. (2022). WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. NeurIPS.
- Feng et al. (2025). Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS.
- Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO).
- Harutyunyan et al. (2019). Hindsight Credit Assignment. NeurIPS.
The full list of references, the prompts and the appendix experiments are in the paper.