BlogOctober 20, 2026Spatial intelligence · Reinforcement learning10 min read

GPD: Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models

Vision-language models misjudge depth and direction because RGB frames carry no geometry. GPD hands the scene's true geometry to a teacher during training, only the parts each question needs, and deploys a model that still sees RGB alone.

Animated drawing: an ink path leaves a torn paper cube, bends at a node and reaches a flat sheet, on which a faint outline of the cube appears
Contents
  1. TL;DR
  2. Errors start in perception
  3. How GPD works
  4. Results
  5. What matters
  6. Is the geometry absorbed?
  7. Citation

TL;DR

Ask a vision-language model how far the sofa is from the door, or whether the microwave is left or right of the chair, and it often gets the geometry wrong, then reasons fluently from the wrong facts. Training with outcome rewards only tells it the final answer was wrong. Adding 3D encoders or tools fixes the input, but costs architecture changes and inference time.

GPD (Geometry-Privileged Distillation) uses geometry where it is cheap: in training. Spatial datasets built from 3D scans already contain depth, object semantics and room layout. GPD turns them into short text, keeps only the cues each question needs, and gives them, with the reference answer, to a teacher that is the same model. While GRPO trains on the outcome, the teacher corrects the student's incorrect rollouts token by token. At deployment the model sees RGB frames and the question, nothing else.

On Qwen3-VL-4B, GPD reaches 57.1 on VSI-Bench and 37.6 on average over MindCube, SPAR-Bench, MMSI-Bench and ViewSpatial, ahead of both GRPO (56.4 and 35.7) and self-distillation with the answer as the only privilege (55.7 and 31.5).

1Spatial errors start in perception

Spatial reasoning asks a model to judge distance, direction and layout: plan a route, grasp the nearest object, estimate the size of a room. Two lines of work address it. Explicit 3D methods add geometric encoders or call reconstruction and perception tools at test time; they gain geometric evidence but change the architecture and slow inference. Training-based methods keep inference RGB-only and fine-tune or reinforce on spatial question answering, but their supervision is the final answer, so a wrong depth order or a flipped left–right relation along the way is never located, let alone corrected.

On-policy self-distillation (OPSD) offers token-level supervision: the same model, given privileged context, re-scores the student's own trajectory. Its usual privilege is the reference answer. That cannot fix a spatial error, because the error is perceptual: a misjudged depth or direction can be corrected only by the scene's true geometry. And different questions rest on different evidence: counting on semantics, distance on depth, layout on a top-down view.

2How GPD works

GPD has two stages. Offline, it builds question-relevant 3D cues for every training example. During training, one VLM plays two roles under two contexts. Figure 1 walks through both.

Route 3D cuesoffline, per question

<scene_context>
Depth near→far: chair(1.39m) → table(2.08m) → sofa(2.64m) → door(3.05m)
Objects: chair [right-bottom], table [center], sofa [left-middle] …
Cam-BEV: chair(left-front, 1.4m), table(front, 2.1m) …
</scene_context>  <reference_answer> A </reference_answer>

StudentRGB frames + question only

“Near the bed, looking at the chair: is the microwave on the chair’s left or right?”

GRPO scores every rollout by its answer. Rollouts with reward R < 0.5 go to the teacher.

Teachersame weights + routed 3D cues + answer

student: A. Rightreference: B. Left

teacher agreesteacher disagrees

Update, then deployone set of weights

L = LGRPO + λ · min(LOPSD on incorrect rollouts, 1.5),  λ = 0.003
RGB framesquestion→GPD model→answer

No depth, segmentation or layout at test time: the geometry lives in the weights.

Figure 1. GPD in four steps. Route: the base VLM chooses which cues each question needs, following the paper's routing analysis (counting uses semantics; absolute distance uses depth; room size uses the top-down view; relative direction uses semantics and the top-down view); the privilege block is the paper's example. Student: the model samples a group of answers from RGB alone (rewards here are illustrative). Teacher: the same model, given the routed cues and the answer, re-scores an incorrect rollout token by token; the example and its shading follow the paper's Figure 3. Update: GRPO plus the gated distillation loss; the deployed model is RGB-only.

3D cues as text, routed per question

For every training example, GPD builds up to three cues once, offline. Where a dataset comes from 3D scans, as VSI and SPAR do from ScanNet, the cues come straight from the annotations; otherwise depth is estimated with Depth Anything 3 and objects with Grounded-SAM-2. A fixed template turns each cue into a few lines of text: depth lists objects from near to far with distances, semantics lists objects with their image regions, and the bird’s-eye view gives top-down positions relative to the camera.

Not every question needs every cue, and injecting all of them dilutes the teacher's signal. The base VLM therefore picks the relevant subset for each question, offline. The routes it picks are intuitive: counting goes entirely to semantics, absolute distance mostly to depth, room size mostly to the top-down view, and relative direction to semantics plus the top-down view.

Text turns out to be the right form for this privilege. In a zero-shot test before any training, given to Qwen3-VL-4B without the answer, routed text is the only form of 3D input that beats RGB alone (Figure 2).

Qwen3-VL-4B, zero-shot, 1,000 VSI-590K questions, no reference answer

Accuracy with different forms of 3D input (%)

Figure 2. The same RGB frames plus 3D cues given as rendered images or as text, either all cues or only the routed ones (paper Figure 5, left). The dashed line is RGB only; the axis starts at 40.

A teacher that corrects mistakes

During training, the student sees pS = (RGB, question) and samples a group of eight answers; GRPO turns their rewards into advantages. The teacher is the same network given pT = (RGB, question, routed cues, answer). For each token tj the student sampled, GPD compares the two log-probabilities, Δ = log π(tj | pT) − log π(tj | pS), and uses eΔ − Δ − 1 as a cheap estimate of the reverse KL.

Two choices keep this signal clean. It applies only to incorrect rollouts (reward below 0.5): on correct ones the teacher can only add noise. And the averaged loss is clipped at 1.5, then added to GRPO with weight λ = 0.003. The teacher thereby shapes the student's own reasoning, along its own trajectory, rather than asking it to imitate text the teacher wrote.

3Results

We train Qwen3-VL-Instruct at 2B and 4B for one epoch on 10K VSI, 4K SPAR and 1K MindCube questions, and compare with GRPO and with OPSD that uses the reference answer as its only privilege, all on the same data, backbone and settings.

Controlled comparison, same data and backbone

VSI-Bench, average accuracy

Figure 3. Base model, GRPO, answer-privileged OPSD and GPD on five spatial benchmarks (paper Tables 1 and 2). “Average of four” is the mean over MindCube, SPAR, MMSI and ViewSpatial. GPD has the best average on both scales, though not on every benchmark: OPSD leads on ViewSpatial at 4B, and GRPO on SPAR at 2B.

On VSI-Bench, GPD-4B reaches 57.1, 1.7 points above the base model, 0.7 above GRPO and 1.4 above OPSD, with its gains on geometry-heavy subtasks such as relative distance and direction. On the four other benchmarks it averages 37.6, 1.9 points above GRPO and 6.1 above OPSD, with the largest gain on MindCube (38.0 against 33.0 for GRPO). The 2B model shows the same ordering on both averages (51.7 and 33.6). Answer-only OPSD often falls below GRPO, a reminder that an answer alone does not say where the reasoning went wrong.

4What matters

We ablate on Qwen3-VL-4B, averaging over all five benchmarks. The answer and the geometry are complementary: adding the reference answer alone gives 40.2, routed 3D text alone 40.5, and both together 41.5, against 39.8 for GRPO. Routing beats more context: injecting all 3D text, with the answer, falls to 39.8, the same as GRPO, and routed cues rendered as images reach 40.4. The weight matters, and so does the gate: λ = 0.003 is best, and applying the teacher to every rollout instead of only incorrect ones costs 0.9 points.

Qwen3-VL-4B, average of five benchmarks

Which privilege, in which form

Same setting

Distillation weight λ and scope

Figure 4. Left: privilege ablation (paper Figure 4, left). Base and OPSD are averaged from the per-benchmark tables. Right: the distillation weight with incorrect-only gating, against GRPO (dashed) and the all-rollouts variant at λ = 0.003 (paper Figure 4, right; values reported in the text).

5Is the geometry absorbed?

If GPD internalizes the cues, giving them back at test time should help it less. That is what happens (Figure 5): with RGB alone, GPD scores 56.3 against 54.9 for GRPO; adding routed 3D text at test time lifts GRPO by 2.1 points but GPD by only 0.6.

Qwen3-VL-4B

Accuracy with RGB only, and with routed 3D text added at test time

Figure 5. Test-time dependence on 3D privilege (paper Figure 7; the axis starts at 50). The smaller gap for GPD suggests the geometric cues were absorbed during training.

The training curves agree. Validation accuracy is similar early on, then GPD keeps improving while GRPO plateaus; GPD's policy entropy peaks higher and earlier, a sign that the teacher's signal broadens exploration before the two settle. In the paper's case study, a four-view question about walking toward a stone fountain, GPD localizes the camera, maps the turn into the scene's frame and finds the fountain across views, while GRPO places the fountain on the wrong side of a jar and answers wrongly.

Citation

@article{li2026gpd,
  title   = {Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models},
  author  = {Li, Hongxing and Li, Yixin and Li, Dingming and Wang, Zixuan and Yan, Yuchen and Zhang, Wenqi and Lu, Weiming and Shen, Yongliang},
  journal = {arXiv preprint arXiv:2610.12355},
  year    = {2026}
}

References

  1. Yang et al. (2025). Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (VSI-Bench).
  2. Li et al. (2025). ViewSpatial-Bench: Evaluating Multi-Perspective Spatial Localization in Vision-Language Models.
  3. Li et al. (2025). SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models.
  4. Zhao et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models.
  5. Shao et al. (2024). DeepSeekMath (GRPO).