BlogOctober 20, 2026Spatial intelligence · Reinforcement learning10 min read
GPD: Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models
Vision-language models misjudge depth and direction because RGB frames carry no geometry. GPD hands the scene's true geometry to a teacher during training, only the parts each question needs, and deploys a model that still sees RGB alone.
Contents
TL;DR
Ask a vision-language model how far the sofa is from the door, or whether the microwave is left or right of the chair, and it often gets the geometry wrong, then reasons fluently from the wrong facts. Training with outcome rewards only tells it the final answer was wrong. Adding 3D encoders or tools fixes the input, but costs architecture changes and inference time.
GPD (Geometry-Privileged Distillation) uses geometry where it is cheap: in training. Spatial datasets built from 3D scans already contain depth, object semantics and room layout. GPD turns them into short text, keeps only the cues each question needs, and gives them, with the reference answer, to a teacher that is the same model. While GRPO trains on the outcome, the teacher corrects the student's incorrect rollouts token by token. At deployment the model sees RGB frames and the question, nothing else.
On Qwen3-VL-4B, GPD reaches 57.1 on VSI-Bench and 37.6 on average over MindCube, SPAR-Bench, MMSI-Bench and ViewSpatial, ahead of both GRPO (56.4 and 35.7) and self-distillation with the answer as the only privilege (55.7 and 31.5).
1Spatial errors start in perception
Spatial reasoning asks a model to judge distance, direction and layout: plan a route, grasp the nearest object, estimate the size of a room. Two lines of work address it. Explicit 3D methods add geometric encoders or call reconstruction and perception tools at test time; they gain geometric evidence but change the architecture and slow inference. Training-based methods keep inference RGB-only and fine-tune or reinforce on spatial question answering, but their supervision is the final answer, so a wrong depth order or a flipped left–right relation along the way is never located, let alone corrected.
On-policy self-distillation (OPSD) offers token-level supervision: the same model, given privileged context, re-scores the student's own trajectory. Its usual privilege is the reference answer. That cannot fix a spatial error, because the error is perceptual: a misjudged depth or direction can be corrected only by the scene's true geometry. And different questions rest on different evidence: counting on semantics, distance on depth, layout on a top-down view.
2How GPD works
GPD has two stages. Offline, it builds question-relevant 3D cues for every training example. During training, one VLM plays two roles under two contexts. Figure 1 walks through both.
Route 3D cuesoffline, per question
<scene_context> Depth near→far: chair(1.39m) → table(2.08m) → sofa(2.64m) → door(3.05m) Objects: chair [right-bottom], table [center], sofa [left-middle] … Cam-BEV: chair(left-front, 1.4m), table(front, 2.1m) … </scene_context> <reference_answer> A </reference_answer>
StudentRGB frames + question only
“Near the bed, looking at the chair: is the microwave on the chair’s left or right?”
GRPO scores every rollout by its answer. Rollouts with reward R < 0.5 go to the teacher.
Teachersame weights + routed 3D cues + answer
student: A. Rightreference: B. Left
Update, then deployone set of weights
No depth, segmentation or layout at test time: the geometry lives in the weights.
3D cues as text, routed per question
For every training example, GPD builds up to three cues once, offline. Where a dataset comes from 3D scans, as VSI and SPAR do from ScanNet, the cues come straight from the annotations; otherwise depth is estimated with Depth Anything 3 and objects with Grounded-SAM-2. A fixed template turns each cue into a few lines of text: depth lists objects from near to far with distances, semantics lists objects with their image regions, and the bird’s-eye view gives top-down positions relative to the camera.
Not every question needs every cue, and injecting all of them dilutes the teacher's signal. The base VLM therefore picks the relevant subset for each question, offline. The routes it picks are intuitive: counting goes entirely to semantics, absolute distance mostly to depth, room size mostly to the top-down view, and relative direction to semantics plus the top-down view.
Text turns out to be the right form for this privilege. In a zero-shot test before any training, given to Qwen3-VL-4B without the answer, routed text is the only form of 3D input that beats RGB alone (Figure 2).
Qwen3-VL-4B, zero-shot, 1,000 VSI-590K questions, no reference answer
Accuracy with different forms of 3D input (%)
A teacher that corrects mistakes
During training, the student sees pS = (RGB, question) and samples a group of eight answers; GRPO turns their rewards into advantages. The teacher is the same network given pT = (RGB, question, routed cues, answer). For each token tj the student sampled, GPD compares the two log-probabilities, Δ = log π(tj | pT) − log π(tj | pS), and uses eΔ − Δ − 1 as a cheap estimate of the reverse KL.
Two choices keep this signal clean. It applies only to incorrect rollouts (reward below 0.5): on correct ones the teacher can only add noise. And the averaged loss is clipped at 1.5, then added to GRPO with weight λ = 0.003. The teacher thereby shapes the student's own reasoning, along its own trajectory, rather than asking it to imitate text the teacher wrote.
3Results
We train Qwen3-VL-Instruct at 2B and 4B for one epoch on 10K VSI, 4K SPAR and 1K MindCube questions, and compare with GRPO and with OPSD that uses the reference answer as its only privilege, all on the same data, backbone and settings.
Controlled comparison, same data and backbone
VSI-Bench, average accuracy
On VSI-Bench, GPD-4B reaches 57.1, 1.7 points above the base model, 0.7 above GRPO and 1.4 above OPSD, with its gains on geometry-heavy subtasks such as relative distance and direction. On the four other benchmarks it averages 37.6, 1.9 points above GRPO and 6.1 above OPSD, with the largest gain on MindCube (38.0 against 33.0 for GRPO). The 2B model shows the same ordering on both averages (51.7 and 33.6). Answer-only OPSD often falls below GRPO, a reminder that an answer alone does not say where the reasoning went wrong.
4What matters
We ablate on Qwen3-VL-4B, averaging over all five benchmarks. The answer and the geometry are complementary: adding the reference answer alone gives 40.2, routed 3D text alone 40.5, and both together 41.5, against 39.8 for GRPO. Routing beats more context: injecting all 3D text, with the answer, falls to 39.8, the same as GRPO, and routed cues rendered as images reach 40.4. The weight matters, and so does the gate: λ = 0.003 is best, and applying the teacher to every rollout instead of only incorrect ones costs 0.9 points.
Qwen3-VL-4B, average of five benchmarks
Which privilege, in which form
Same setting
Distillation weight λ and scope
5Is the geometry absorbed?
If GPD internalizes the cues, giving them back at test time should help it less. That is what happens (Figure 5): with RGB alone, GPD scores 56.3 against 54.9 for GRPO; adding routed 3D text at test time lifts GRPO by 2.1 points but GPD by only 0.6.
Qwen3-VL-4B
Accuracy with RGB only, and with routed 3D text added at test time
The training curves agree. Validation accuracy is similar early on, then GPD keeps improving while GRPO plateaus; GPD's policy entropy peaks higher and earlier, a sign that the teacher's signal broadens exploration before the two settle. In the paper's case study, a four-view question about walking toward a stone fountain, GPD localizes the camera, maps the turn into the scene's frame and finds the fountain across views, while GRPO places the fountain on the wrong side of a jar and answers wrongly.
Citation
@article{li2026gpd,
title = {Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models},
author = {Li, Hongxing and Li, Yixin and Li, Dingming and Wang, Zixuan and Yan, Yuchen and Zhang, Wenqi and Lu, Weiming and Shen, Yongliang},
journal = {arXiv preprint arXiv:2610.12355},
year = {2026}
}References
- Yang et al. (2025). Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (VSI-Bench).
- Li et al. (2025). ViewSpatial-Bench: Evaluating Multi-Perspective Spatial Localization in Vision-Language Models.
- Li et al. (2025). SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models.
- Zhao et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models.
- Shao et al. (2024). DeepSeekMath (GRPO).