BlogNovember 22, 2026Agentic AI · Reinforcement learning14 min read

Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents

Decision models answer a question with a choice and a probability, so the same model can pick an agent's actions, grade them, or teach a policy. We find that a model which acts poorly can still teach well, as long as it is asked the right question and shown the whole trajectory.

Three robots in a watercolour landscape: an amber actor weighing which path to take, a blue judge inspecting footsteps with a magnifying glass, and a green teacher pointing at a board for a smaller student robot
Contents
  1. TL;DR
  2. Three ways to ask a decision model
  3. As an actor
  4. As a judge
  5. As a teacher
  6. Takeaways
  7. Citation

TL;DR

A decision model answers a question about a context with a typed output, such as a choice among options with a probability for each, instead of free text. Because any question can be asked about any state, one model can choose an agent's next action, evaluate an action already taken, or score every action of a policy during reinforcement learning. So far these models have mostly been judged by how well they act.

We asked whether a model that acts poorly can still teach well. On ALFWorld, JEV completes 42.9% of tasks when it chooses the actions itself. Used instead to score each action of a 1.5B policy during GRPO training, it raises that policy's success over GRPO alone from 72.8% to 82.9% on the same tasks, and from 56.8% to 71.4% on WebShop. The difference comes from timing: an actor must anticipate consequences, whereas a judge that reads the finished episode can see them.

The benefit is not automatic. It appears only when the model is asked whether each action helped achieve the goal, rather than whether the task will succeed, and only when it sees the whole trajectory rather than the steps leading up to the action. Queried this way after each episode, a frozen decision model provides step-level supervision for 22 to 30 USD of API calls per training run, with no step labels and no learned critic.

1Three ways to ask a decision model

A decision model takes a question and a context. For a multiple-choice question it selects an option and assigns a probability to every option; for a yes/no question it returns one score in [0, 1]. We query the same model in three roles that differ only in what it is asked and how its answer is used.

  • Actor. At each step the model picks the next action from the valid commands, given the goal, the history and the current observation. The choice is executed.
  • Judge. The model scores each action of a recorded trajectory. We vary what it sees, the trajectory up to the action (Prefix) or the whole trajectory (Hindsight), and what it is asked, whether the task will succeed after the action (Value) or whether the action helped achieve the goal (Progress).
  • Teacher. The model scores the actions of the policy being trained, and the scores enter GRPO as step-level credit. It stays frozen and is not used at test time.

Figure 1 plays the three roles on the paper's running example: put a heated mug on a table.

Actorpicks the next action

ObservationAt the table, holding an unheated mug.
  • Aplace mug on table
  • Bgo to microwave
  • Clook… other valid actions

QuestionWhich action should be executed next?

Judgescores a recorded action

Recorded episode: put a heated mug in the cabinet

  1. 6move mug to cabinetplaced too early
  2. 7take mug from cabinet
  3. 8go to microwave
  4. 9open microwavetarget
  5. 10heat mugmilestone
  6. 12move mug to cabinetgoal complete

QuestionDid this action help achieve the goal?

Teacherscores the policy's own rollouts

πθPolicy being trainedupdated

1 take mug take againno-op heat mugmilestone place ✓ 2 place cold mug take mug back heat mug place ✓

Question, for every actionDid this action help achieve the goal?

Decision model

question + context → typed answer. Here JEV, one of the six models we test.

question next_actioncontext goal · history · observationanswer one option + a probability for each
question progresscontext trajectory up to step 9 (Prefix)answer score in [0, 1]
question progresscontext whole trajectory (Hindsight)answer score in [0, 1]
question progress, every actioncontext whole rollout (Hindsight)answer one score per action

Choice probabilities

A place63.8%
B microwave14.6%
other21.6%
place mug on tableplaced ✓heated ✗

Easier subgoal first: the task is left incomplete.

Score for step 9, open microwave

Prefix0.20
Hindsight0.76

Same question; seeing the mug heated shows why it mattered.

Teacher scores group mean 0.71

1 2
outcome advantage + 0.5 × (score − mean)GRPO

The no-op is pushed down; taking the cold mug back is credited.

Figure 1. One decision model, queried three ways on heated-mug tasks. Actor: JEV’s choice probabilities, averaged over 116 states in which the object still needs heating, cleaning or cooling. Judge: JEV’s score for step 9 of a recorded episode, seeing first the steps up to it and then the whole episode (paper Table 9). Teacher: illustrative scores from the paper’s Figure 1, centred within the group and added to GRPO’s outcome advantage. Use the tabs to jump to a role.

We evaluate six models that answer questions in this form. Three are released as dedicated decision models and are queried through their own decision APIs. The other three are general LLMs asked the same questions, either by prompting for a JSON answer or by reading the first-token probabilities of the valid answer labels. Two of these general LLMs are the base models of Decider v1.2 and GPT Decisions, so each dedicated model can be compared with its own base model.

ModelBase LLMHow the decision interface is obtainedAccess
JEVundisclosedundisclosedAPI only
Decider v1.2Qwen3.8-27Bfull fine-tuningopen weights
GPT DecisionsGPT-6 LunaundisclosedAPI only
Qwen promptedQwen3.8-27Bprompting (JSON answer)open weights
Qwen constrainedQwen3.8-27Bconstrained readout (first-token probabilities)open weights
GPT LunaGPT-6 Lunaprompting (JSON answer)API only

We use two text environments in which some requirements must be met in a fixed order. In ALFWorld [1], an agent completes household goals with commands such as go to, take and heat. Its Clean, Heat and Cool tasks have a dependency: the object's state must change before it is placed. Pick2 also has two requirements, two objects to place, but they can be met in either order. In WebShop [2], an agent searches a catalogue and must select the required options before it buys.

2As an actor, an easier subgoal gets in the way

Finding 1

As actors, all six models struggle when one subgoal must be completed before another: an easier subgoal draws them away from the one that has to come first.

Ordered subgoals are where acting breaks

Every model succeeds far less often on tasks with a dependency than on tasks without one (Figure 2). JEV completes 70.8% of Pick2 tasks but at most 18.8% of each dependent task type. Decider v1.2, the strongest actor overall at 63.6%, completes only 18.8% of Heat tasks. Because Pick2 also has two subgoals, the gap points to their order rather than their number as the difficulty.

ALFWorld, acting

Actor success rate by task type

Figure 2. Success rates (%) of the six models acting on ALFWorld, one run per task on the evaluation tasks later used for the trained policies. Pick2 and the three dependent types both have two subgoals; only the dependent ones fix their order. Switch to “Pick2 vs dependent” to compare those directly; hover a cell for its value.

The failed episodes show how the order is broken. Holding the mug at the table, the actor can place it in one action, or walk to the microwave, heat it and come back. In many failures it places the unheated mug and the task can no longer be completed as intended.

Distraction, not ignorance

We collected 116 states, from 66 tasks, in which a single action would satisfy the easier subgoal while the required one was still unmet, and queried each actor there under two goals that differ only in the easier subgoal, for example “put a heated mug on the side table” versus “heat a mug”. Everything else, including the history, observation, candidate actions and remaining budget, is identical.

With the full goal, JEV puts 63.8% of its probability on the easier subgoal and 14.6% on the step that must come first; without it, these become 2.5% and 77.6% (Figure 3, left). All six actors shift the same way. They know the required step; the easier subgoal pulls probability away from it. Reminding them of the dependency barely helps: a generic hint changes success by −2.8 to +5.2 points, and adding a worked ordering example helps some models but not all (Figure 3, right).

JEV, 116 states

Where the probability goes

Share of JEV's probability by action type, %

Same 116 states, continued to the end

Success with an ordering example in the prompt

Original prompt → hint plus a heated-mug example, %

Figure 3. Left: JEV's action probabilities, grouped into the step toward the subgoal that must come first, the action that completes the easier subgoal, and everything else, with and without the easier subgoal in the goal. Right: success when each episode continues under the original prompt versus a dependency hint plus an ordering example. GPT Luna, which changes little, is omitted because the paper does not report its values in the text.

Acting correctly here requires anticipating that the easy action leaves the task unfinished before any consequence is visible. A judge evaluates the action after its consequences have occurred, which is where the next section starts.

3As a judge, the question and the context decide

Finding 2

The same models can recognise contributions they overlook as actors. Seeing the whole trajectory reveals how earlier actions feed later progress, and asking whether each action helped separates those actions from ones that change nothing.

We reused the successful continuations from the 116 states and asked each judge whether the first action taken from the diagnostic state helped achieve the goal, with outcome labels masked. Seeing the whole trajectory instead of the prefix raises the mean score of these actions for all six models; for JEV it goes from 0.41 to 0.59. That raises two questions: does a full trajectory simply make every action look good, and what does the judge actually gain from it?

Ask whether the action helped, not whether the task will succeed

We labelled actions in 462 ALFWorld trajectories as milestones, which achieve a subgoal, no-ops, which leave the environment unchanged, and other. With the whole trajectory visible, the Value question gives the three groups almost the same score: if the episode ends in success, every action in it “leads to success”. The Progress question spreads them apart for every model (Figure 4). For JEV, within a trajectory, Progress scores milestones 0.36 above no-ops, against 0.05 for Value, and it ranks more milestones among each trajectory's top-scored actions, by 8.0 to 16.0 points.

ALFWorld, 462 trajectories, Hindsight context

Mean action score by action type

Figure 4. Mean Hindsight scores for milestone, other and no-op actions under the two questions; scores are averaged within each trajectory, then across trajectories. The bracket shows the difference between the milestone and no-op means. Pick a judge to compare; Progress separates the groups for all six.

Show the whole trajectory

To see what the extra context buys, we held the question fixed to Progress and revealed the trajectory step by step: the prefix, then the immediate feedback after the action, then 25%, 50%, 75% and all of what follows. Over 167 trajectories, JEV's ranking of actions within a trajectory moves by 1.95 positions on average once feedback is added, and by 3.30 positions with the full trajectory. Figure 5 shows one of them. Drag the slider, or press play.

JEV as a judge, Progress question

“Put a heated mug in the cabinet”: scores as more of the episode becomes visible

Visible context

Figure 5. JEV's Progress scores for all 12 actions of one episode (paper Table 9). The mug is put in the cabinet before it is heated (step 6), taken back (7), heated in the microwave (8 to 10) and put back (12). Each bar is the score of that action when the judge sees the episode up to the action, plus the immediate feedback, plus 25%, 50%, 75% or all of the remaining steps.

Under the prefix, JEV rates the premature placement (step 6) at 0.79 and going to the microwave (8) and opening it (9) at only 0.23 and 0.20, the same preference for the easier subgoal that the actor showed. Immediate feedback makes it worse: placement rises to 0.90. Only as the continuation comes into view do the microwave steps earn credit; opening the microwave reaches 0.66 once heating is visible and 0.76 with the full trajectory. The repeated look (step 2) stays low throughout, because the prefix alone already shows that it repeats step 1.

The two dedicated decision models whose base LLM is disclosed, Decider v1.2 and GPT Decisions, also judge more accurately than that base LLM given the same question, at 28% to 43% lower cost per score.

4As a teacher, the best judge gives the best signal

Finding 3

The judgments that best separate useful from useless actions also train the strongest policies, and the gains are largest on the tasks the teacher fails as an actor.

From scores to advantages

GRPO gives every action in a rollout the same outcome advantage. For each task the policy samples a group of trajectories; after they finish, the teacher D scores every executed action t of every trajectory τi from the goal and the trajectory alone, with no rewards, success labels or hidden state:

si,t = D(g, τi, t) ∈ [0, 1] Ai,t = Aouti,t + λ (si,t − s̄)

The scores are centred on the mean s̄ over all scored actions in the group and added with weight λ = 0.5. They are deliberately not divided by their standard deviation, so a group in which the teacher hardly distinguishes actions receives only small adjustments. Figure 6 shows what the teacher term does to the two rollouts from Figure 1.

Illustrative group of two rollouts

What the teacher adds to each action's advantage

Figure 6. λ(s − s̄) for each action, using the illustrative teacher scores of Figure 1 (group mean 0.71). Bars above zero receive a larger advantage than GRPO alone would give, bars below a smaller one. The outcome advantage itself is the same for every action of a rollout, so only the teacher term distinguishes them. The paper trains with λ = 0.5.

Results

We train Qwen2.5-1.5B-Instruct with JEV as the teacher under all four combinations of question and context, and compare against GRPO with outcome rewards alone (results from GiGPO [3]). Each policy is evaluated with three seeds. Hindsight + Progress gives the highest success on both environments: 72.8% → 82.9% on ALFWorld and 56.8% → 71.4% on WebShop. It improves on every ALFWorld task type, most on Cool (+23.0) and Look (+20.7). With the prefix only, the two questions perform about the same. The ordering matches the judge analysis: under Hindsight, the Value question gives every action of an episode nearly the same score and so says little about which actions helped.

Qwen2.5-1.5B-Instruct, mean of three seeds

Success rate, teacher-trained policy vs GRPO

GRPO (outcome reward only)Hindsight + Progresswhiskers: ± std over seeds
Figure 7. Success rates (%) of the 1.5B policy trained with JEV as the teacher under each question and context, against GRPO. The last two groups are overall ALFWorld success and WebShop success. Switch settings to compare; Hindsight + Progress is the best setting on both environments. The 7B option shows the same recipe on Qwen2.5-7B-Instruct.

The gain holds at 7B, from 77.6% to 83.8% on ALFWorld and from 66.1% to 73.6% on WebShop, and the 1.5B policy taught this way outperforms the 7B policy trained with outcome rewards alone on both environments. Hindsight + Progress also finishes sooner: it completes 74.5% of ALFWorld tasks within 20 actions, against 63.6% to 69.5% for the other settings, and 71.1% of WebShop tasks within 10 actions, against 60.9% to 62.2%.

Learning beyond the teacher

On the same ALFWorld evaluation tasks, JEV completes 42.9% as an actor; the 1.5B policy trained on its scores completes 82.9%. The largest gains are on the dependent tasks, the ones JEV handles worst as an actor: there the policy reaches 86.8%, 6.9 to 16.2 points above the other question and context settings, while JEV itself completes only 14.7% of them (Figure 8). The teacher does not need to solve a task to provide a useful signal for it.

ALFWorld, same evaluation tasks

JEV acting vs the 1.5B policy JEV taught

JEV as actor1.5B policy trained with JEV (Hindsight + Progress)
Figure 8. Success rates (%) by task type. The arrow runs from JEV acting on its own (Table 2) to the policy it taught (Table 3). The policy surpasses its teacher everywhere except Pick2, and by the widest margins on the three dependent types.

Cost. We query JEV through OpenRouter at 0.042 USD per million input tokens to score every executed action during training: 150 steps, 16 tasks per batch, 8 rollouts per task. Scoring an entire training run costs between 22.21 and 30.07 USD.

Hindsight + Progress runALFWorld 1.5BALFWorld 7BWebShop 1.5BWebShop 7B
Max actions per rollout30301515
Teacher API cost (USD)30.0724.5022.2126.83

5Takeaways for using decision models in agents

  1. Actors know the required step but are drawn to the easier one. They choose the right step when it is the only subgoal, and general reminders do not stop the error. When a decision model must act, restructuring the goal into ordered subgoals is more promising than adding instructions.
  2. The question sets the granularity of the signal. Asking about the eventual outcome gives every action of a trajectory a similar score and largely repeats the outcome reward; asking about each action's contribution separates them.
  3. Hindsight targets the actor's own failure. Seeing the whole trajectory lets a judge credit the steps toward the subgoal that must come first. Decision models are best used to score actions after an episode, as in RL training, and are less suited to filtering actions online, where only the prefix exists.
  4. Screen judges before training. Judging recorded trajectories identified the question and context that trained the best policy, at a small fraction of the cost of an RL run. Choose a teacher by how well it judges, not by how well it acts.
  5. Step-level supervision becomes affordable. A frozen decision model, queried after each episode and absent at test time, costs tens of dollars per run and lets a small policy outperform a larger one trained on outcomes alone.

Citation

@article{liu2026actor,
  title   = {Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents},
  author  = {Liu, Haochen and Zhu, Qianhao and Wang, Zixuan and Li, Dingming and Chen, Tongbo and Qiu, Peng and Qu, Yingqi and Lu, Weiming and Xiao, Jun and Shen, Yongliang},
  journal = {arXiv preprint arXiv:2611.17167},
  year    = {2026}
}

References

  1. Shridhar et al. (2021). ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. ICLR.
  2. Yao et al. (2022). WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. NeurIPS.
  3. Feng et al. (2025). Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS.
  4. Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO).
  5. Harutyunyan et al. (2019). Hindsight Credit Assignment. NeurIPS.

The full list of references, the prompts and the appendix experiments are in the paper.