Decision models · Long-horizon agents

Actor, Judge, or Teacher?

Understanding Decision Models in Long-Horizon Agents

Haochen Liu^1,*, Qianhao Zhu^1,*, Zixuan Wang^1,*, Dingming Li¹, Tongbo Chen¹,
Peng Qiu², Yingqi Qu², Weiming Lu¹, Jun Xiao¹, Yongliang Shen^1,†

¹Zhejiang University ²Baidu ^*Equal contribution · ^†Corresponding author

Code Paper Read the story ↓

The story in one line

A decision model that acts poorly can still teach well.

Judging an action after the episode ends reveals the consequences that acting must anticipate. We follow JEV through all three roles on the same ALFWorld tasks with dependency (clean, heat, or cool an object before placing it).

Actor
fails to act

14.7%

success on tasks with dependency when JEV chooses the actions itself
Judge
judges well, if asked right

Milestone Other No-op Value 0.632 0.604 0.577 Progress 0.659 0.440 0.298

Only the Progress question separates milestones from no-ops.
Teacher
teaches what it can’t do

86.8%

success on the same tasks for a 1.5B policy trained with JEV’s scores

Abstract

Decision models answer a question with a typed output, such as a choice among options with a probability for each, so one model can choose an agent's actions, judge actions already taken, or score every action to teach a policy with reinforcement learning. They have been evaluated mainly as actors, where they often fail. We study three decision models and three general LLMs given the same questions in all three roles on ALFWorld and WebShop. Our results suggest that a decision model that acts poorly can still teach well, because judging an action after the episode ends reveals the consequences that acting must anticipate. As actors, all six models are pulled away from a subgoal that must come first by an easier one. As judges that see the whole trajectory and are asked whether each action helped, the same models credit the actions that repair such mistakes, whereas asking whether the task will succeed gives every action nearly the same score. Used this way as a teacher, JEV raises a Qwen2.5-1.5B-Instruct policy over GRPO from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop, with the largest gains on the tasks that JEV rarely completes as an actor.

Actor

Acting fails on tasks with dependency.

As Actors, decision models struggle with dependent subgoals: an easier subgoal draws them away from the subgoal that must be met first.

A mug must be heated before placement; two pencils (Pick2) can be placed in either order. All six models succeed less often when order matters.

Actor success on ALFWorld (%)

Same evaluation tasks as the trained policies. Darker is better.

	No dependency			With dependency
Actor	Pick	Look	Pick2	Clean	Heat	Cool	All
JEV	77.1	46.2	70.8	11.1	18.8	16.0	42.9
Decider v1.2	97.1	53.8	83.3	55.6	18.8	40.0	63.6
GPT Decisions	88.6	76.9	62.5	29.6	25.0	24.0	52.9
Qwen prompted	77.1	53.8	79.2	22.2	31.2	20.0	49.3
Qwen constrained	77.1	46.2	75.0	33.3	6.2	8.0	45.0
GPT Luna	77.1	46.2	37.5	18.5	0.0	4.0	34.3

The models know the required step. We collect 116 states in which the Actor stands at the destination holding an object that still needs heating, cleaning, or cooling. A single action would satisfy the easier placement subgoal. We then remove that subgoal from the goal and keep everything else identical.

Where does JEV put its probability?

At the destination, holding an unheated mug.

14.6% 63.8%

Toward the required state change (heat)
Place the mug now (easier subgoal)
Other actions

With both subgoals in the goal, 63.8% of the probability goes to placing the unheated mug.

All six Actors shift the same way. They can identify the required step when it is the only subgoal, but the easier subgoal pulls probability away from it. Reminding them doesn’t fix it:

To act correctly, the model must anticipate a consequence before it is visible.
A judge looks after the consequence has happened. Can the same model credit the step it skipped?

Judge

Judging works, with the right context and question.

As Judges, the same models can recognize contributions they overlook as Actors. This depends on what they see and what they are asked.

A Judge scores each action of a recorded trajectory with a yes/no question and returns a score in [0, 1]. We vary two things, holding everything else fixed:

Context what it sees

Prefix the trajectory up to the action

Hindsight the whole trajectory

Question what it is asked

Value “Will the task succeed after this action?”

Progress “Did this action help achieve the goal?”

Context: Hindsight reveals what a step was for

On the first actions of successful continuations from the Actor’s diagnostic states, all six Judges give higher contribution scores with the whole trajectory; JEV rises from 0.41 to 0.59. The case below shows why. The task requires a heated mug in a cabinet. The trajectory places the mug too early (6), takes it back (7), and heats it in the microwave (8–10).

Heatmap of JEV scores for steps 2, 3, 6, 7, 8, 9, and 10 under Prefix, plus feedback, plus 25, 50, 75 percent, and Full context. — JEV’s Progress score per step as more of the trajectory becomes visible

Drag the slider, or press play.

Question: ask about the action, not the outcome

Seeing the consequences is not enough. With the whole trajectory visible, every action shares one answer to “will the task succeed?”: the outcome. On 462 ALFWorld trajectories, we compare milestone actions (achieve a subgoal), no-op actions (change nothing), and the rest.

JEV’s mean score by action type (Hindsight context)

Switch the question.

Milestone 0.632

Other 0.604

No-op 0.577

Within-trajectory gap, milestone − no-op: 0.05. Value scores track the outcome, so every action looks about the same.

Progress also ranks more true milestones at the top of each trajectory than Value, by 8.0 to 16.0 points at every trajectory length. All six Judges show both effects, and the two decision models with a disclosed base LLM judge more accurately than that LLM given the same question, at 28% to 43% lower cost per score.

Same model. Same context, same question.
Now point it at the rollouts of a policy being trained.

Teacher

The same judgments teach the tasks the Actor fails.

The judgments that best distinguish action contributions also provide the most effective supervision, with the largest gains on tasks that the Actor struggles to complete.

1 Policy rolls out Qwen2.5-1.5B samples a group of trajectories per task

2 JEV scores every action Hindsight context
Progress question

3 GRPO update A_i,t = A^out_i,t + λ (s_i,t − s̄_u)

Scores are centered within each rollout group (λ = 0.5). JEV stays frozen during training and is never queried at test time.

The query that judged best teaches best

1.5B policy success on ALFWorld tasks with dependency (%)

Value

Progress

Prefix

75.0ALFWorld 79.0 · WebShop 67.7

79.9ALFWorld 78.3 · WebShop 66.6

Hindsight

70.6ALFWorld 69.3 · WebShop 68.3

Judge’s pick 86.8ALFWorld 82.9 · WebShop 71.4

The Teacher is best exactly where the Actor is worst

Actor

14.7%

JEV choosing the actions itself

Teacher

86.8%

1.5B policy taught by JEV

ALFWorld tasks with dependency (Clean, Heat, Cool).

Per task type: JEV as Actor vs. what Hindsight + Progress adds as Teacher

Sorted by Actor success. Gains in percentage points for the 1.5B policy.

Task JEV as Actor Teacher gain

Clean dep. 11.1% +7.4

Cool dep. 16.0% +6.7

Heat dep. 18.8% +6.3

Look 46.2% +7.7

Pick2 70.8% −8.3

Pick 77.1% −4.7

Against the best of the other three Context × Question settings, Hindsight + Progress wins on every task JEV fails as an Actor, and loses on the two it handles well.

ALFWorld · 1.5B

72.8 → 82.9

GRPO → + JEV Teacher

WebShop · 1.5B

56.8 → 71.4

GRPO → + JEV Teacher

7B policy

77.6 → 83.8

ALFWorld; WebShop 66.1 → 73.6

Small beats large

1.5B > 7B

taught 1.5B beats 7B GRPO on both environments

Main takeaways

Don’t choose a supervisor by how well it acts. Choose it by how well it judges: ask about each action, after the episode ends, and it can teach what it cannot do.

01 Actor
Actors know the required step but are drawn to the easier one.

General reminders don’t prevent the error. When a decision model must act, restructure the goal, e.g. into ordered subgoals, rather than adding instructions to the prompt.

14.6% → 77.6%JEV’s probability on the required step once the easier subgoal is removed
02 Judge
The question sets the granularity of the signal.

Asking about the eventual outcome gives every action nearly the same score and largely repeats the outcome reward. Asking about each action’s contribution separates actions within a trajectory.

0.05 → 0.36milestone − no-op gap, Value → Progress
03 JudgeActor
Hindsight judgments target the Actor’s own failure.

The whole trajectory reveals why prerequisite steps mattered. Use decision models to score actions after an episode ends, as in RL training, not to filter actions online from the prefix.

0.20 → 0.76JEV’s score for “open microwave”, Prefix → Full
04 JudgeTeacher
Screen Judges before training.

Judge measures on recorded trajectories identified the question and context that trained the best policy, at a small fraction of the cost of an RL run. Choose by judging quality, not acting ability.

42.9% → 82.9%JEV as Actor vs. the 1.5B policy it teaches, ALFWorld
05 Teacher
Step-level supervision becomes affordable.

No step labels, no learned critic, no Teacher at test time. Decision models judge more accurately and more cheaply than prompted LLMs, and a taught 1.5B policy beats a 7B one trained on outcomes alone.

$22–30of JEV queries per training run

Code

Train a policy with a decision-model Teacher.

Decision-model interfaces (Decisions API, prompted, constrained readout), action scoring, and Teacher integration with GRPO training on ALFWorld and WebShop, built on verl-agent. See the installation instructions.

View on GitHub ↗

# Qwen2.5-1.5B-Instruct with a JEV Teacher
$ bash examples/decision_model/run_alfworld.sh
$ bash examples/decision_model/run_webshop.sh

Actor, Judge, or Teacher?

A decision model that acts poorly can still teach well.

Actor

Judge

Context

Question

Teacher

Decision-model family

Actor success on ALFWorld (%)

Where does JEV put its probability?

Context what it sees

Question what it is asked

Context: Hindsight reveals what a step was for

JEV’s Progress score per step as more of the trajectory becomes visible

Question: ask about the action, not the outcome

JEV’s mean score by action type (Hindsight context)

The query that judged best teaches best

1.5B policy success on ALFWorld tasks with dependency (%)

The Teacher is best exactly where the Actor is worst

Per task type: JEV as Actor vs. what Hindsight + Progress adds as Teacher

Actors know the required step but are drawn to the easier one.

The question sets the granularity of the signal.

Hindsight judgments target the Actor’s own failure.

Screen Judges before training.

Step-level supervision becomes affordable.

Train a policy with a decision-model Teacher.