BlogNovember 22, 2026World models · Benchmark12 min read
WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models
Video world models can now generate half a minute of explorable world. But turn the camera away and come back: is it still the same world, and did what was happening keep happening? WorldMemBench measures exactly that.
Contents
TL;DR
A video world model generates a world you can move through. Moving means things leave the frame and come back into it. A model that only produces good frames can still forget a person’s face, swap an object, rearrange a room, or freeze an event that should have kept going while the camera looked away.
WorldMemBench tests this directly. In 260 cases across 23 categories, the camera follows a controlled path away from something it has seen and back, once or several times. Static content, people, objects and scenes, is compared at every return against the same reference from the start. Dynamic events, such as pages being turned or ice melting, are checked with case-specific questions about whether they progressed while out of view. Video quality and camera-following accuracy are reported alongside, so that memory scores can be read in context.
Across nine recent world models, the one that remembers static content best, HY-WorldPlay, starts only 31.7% of the events it is asked to show; and every model forgets more with each return. No model leads on all four dimensions.
1What world memory is
Autoregressive video generation has become long and stable enough, and camera and action control precise enough, that “video world models” now let you explore a generated place for tens of seconds. Exploration constantly moves content in and out of view. We call world memory the ability to keep what has been established, and to keep it evolving coherently, across those interruptions.
It has two halves. Static memory: a person’s identity and clothes, an object’s look and shape, a room’s layout and depth should all be the same when you look again. Dynamic memory: something that was happening should have continued while you looked away, and its consequences should be visible when you return. Frame-level quality and local smoothness say nothing about either.
Earlier benchmarks examine pieces of this, such as revisit consistency, or state evolution while unobserved, and one recent benchmark tests recall across repeated loops. WorldMemBench puts them together: long revisits, repeated revisits, state evolution and physical consistency, with camera control and video quality measured in the same rollouts.
2Leave, and come back
Every case starts from a conditioning frame, the initial view V. The camera then follows a prescribed path that takes it away and brings it back: a departure Di followed by a revisit Ri. Rotations keep the camera in place and turn it left and right (yaw) or up and down (pitch); translations keep its heading and walk a square. Figure 1 plays real rollouts from the benchmark, all generated by LingBot World v2.
260 cases, 23 categories
The benchmark has 60 Human, 70 Object and 70 Environment cases, which form the 200-case static-memory set, and 60 Dynamic cases covering seven kinds of state change: motion, assembly, removal, deformation, transfer, reaction and thermal change. Video quality is measured on all 200 static rollouts, and camera following on 85 of them whose trajectories suit pose estimation.
Composition
Four families, 23 categories
Scoring what stays
For static content, each revisit is compared with a fixed reference from the initial view, never with the previous return, so that errors cannot hide by drifting slowly. Scores are averaged over revisits and then over targets, and a prescribed revisit with no valid observation counts as zero.
- People: identity from ArcFace face embeddings, and appearance from DINOv2 features inside the person’s mask.
- Objects: the target is found again at each return by detection and video segmentation; appearance from DINOv2 features in the mask, and geometry as the IoU of the silhouettes after normalizing position and scale.
- Environments: visible content from global DINOv2 features at the return frame whose camera orientation is closest to the start; relative depth order as the Spearman correlation of depth maps; and 3D structure by projecting both depth maps onto a shared spherical grid and comparing the reconstructed surfaces.
Scoring what changes
A dynamic event cannot be scored by similarity to the first frame, because the right answer is a different frame. Each dynamic case instead specifies an initial state, an action and the expected next state. A vision-language model (Qwen3.8-27B) first decides whether the event visibly starts at all; if it does not, the case scores zero. Otherwise it answers eight yes/no questions with fixed reference answers, two each on return progress, persistence, final state and causal or physical consistency. Figure 3 shows a real case.
3Nine models, four dimensions
We evaluate HY-WorldPlay, Matrix-Game 3 and 3.5, SANA-WM, YUME, LingBot World and LingBot World v2, Evoke and DreamX World, each with the defaults of its official release and fixed seeds, converting the benchmark trajectories into each model’s own control format.
WorldMemBench, nine models
All metrics, coloured by rank within each metric
Static and dynamic memory are different skills. HY-WorldPlay leads all seven static-memory metrics, from human identity (0.828) to environment reconstruction (0.619), yet its dynamic-memory score is 0.294: it starts only 31.7% of the prescribed events. LingBot World (0.808) and YUME (0.775) lead dynamic memory while sitting in the middle on static metrics. No model leads on both: keeping the world unchanged and keeping it moving pull in different directions (Figure 5).
Each point is a model
Static memory (mean of seven metrics) against dynamic memory
Camera following varies independently: Matrix-Game 3 has the lowest translation error (0.424) and HY-WorldPlay the lowest rotation error (0.417), and DreamX World, with the highest of both, also has the lowest environment depth and reconstruction scores. Video quality, meanwhile, varies far less across models than memory does, and says little about it.
4Memory fades with every return
A good first return shows that a model can recover a view once. It does not show that the recovery lasts. Under LRLRLR, the nine-model average falls from the first to the third return on all seven static metrics: object appearance by 0.229, human identity by 0.217, object geometry by 0.205 and environment reconstruction by 0.196. Per model, every one is worse at the third return than the first on the human, object, visible and reconstruction metrics.
Two things compound. The returning viewpoint drifts: mean rotation error grows from 0.430 to 0.502 to 0.530 over the three returns. And every model generates autoregressively, so early errors feed later frames. The decline survives restricting the analysis to the 431 environment rollouts whose departures and returns meet the pose criteria: visible content, depth order and reconstruction still fall by 0.152, 0.135 and 0.139.
One environment case, all nine models
Visible memory at each return
Changing direction is harder still. LRLR and LRUD share their first return but differ in the second: LRLR turns left and right again, LRUD pitches up and down. Returning along a new axis costs more for every static metric on average, and for 50 of the 63 model–metric pairs; environment visible memory drops an extra 0.152.
5What to keep in context
Because the benchmark separates dimensions, it can be used to diagnose design choices. Holding LingBot World v2 and all generation settings fixed, we compared three training-free ways of choosing which past frames the model attends to: the full history (FullKV), only the recent window (Sliding), and recent frames plus view-relevant history retrieved by field of view (FoV), the last two under the same budget.
LingBot World v2, model and settings fixed
Memory scores under three context policies
FoV beats the full history on all four dynamic sub-metrics and both human metrics, while full history keeps an edge on object appearance, object geometry and visible environment content. Both budgeted policies are faster: 262.7 and 260.4 seconds per video against 322.2 for full history, about 1.23× and 1.24×. Mixing recent context with view-relevant history looks like a good trade, and the benchmark shows which kinds of memory each choice helps.
6Do the scores agree with people?
For static memory, three annotators judged 504 blinded pairs of rollouts. The automatic metrics agree with the human majority on 78.5% to 92.5% of decisive pairs, and model rankings correlate with human win rates up to ρ = 0.988 for visible environment memory. For dynamic memory, three annotators answered all 400 checklist items of 50 cases; the VLM judge’s overall scores correlate with theirs at Spearman ρ = 0.732 (PLCC 0.808). Causal and physical questions are the hardest to judge, at ρ = 0.401.
| Static metric | Agreement with human majority | Model-level Spearman ρ |
|---|---|---|
| Environment visible | 92.5% | 0.988 |
| Human appearance | 88.8% | 0.952 |
| Human identity | 85.5% | 0.929 |
| Object appearance | 83.8% | 0.699 |
| Environment depth | 81.8% | 0.301 |
| Object geometry | 78.5% | 0.606 |
Citation
@article{zhou2026worldmembench,
title = {WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models},
author = {Zhou, Nan and Gao, Jianxiong and Wang, Qiuyu and Wang, Wen and Wang, Jiahao and Yuan, Yufeng and Li, Shuaiting and Ouyang, Hao and Shen, Yujun and Lu, Weiming and Xiao, Jun and Zhuang, Yueting and Shen, Yongliang},
journal = {arXiv preprint arXiv:2610.19063},
year = {2026}
}References
- Sun et al. (2025). WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling.
- Huang et al. (2025). Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion.
- Yu et al. (2025). Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval.
- Huang et al. (2024). VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR.
- Duan et al. (2025). WorldScore: A Unified Evaluation Benchmark for World Generation.
Inference settings for every model, the dynamic-memory prompts and the full per-trajectory results are in the paper; rollouts, scores and the evaluation code are on Hugging Face and GitHub.