BlogNovember 22, 2026World models · Benchmark12 min read

WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models

Video world models can now generate half a minute of explorable world. But turn the camera away and come back: is it still the same world, and did what was happening keep happening? WorldMemBench measures exactly that.

A generated rollout from WorldMemBench (an object case, generated by Evoke): one Rubik’s cube on a table at the start, a row of them later on. Catching this kind of drift is what the benchmark is for.

Contents
  1. TL;DR
  2. What world memory is
  3. Leave, and come back
  4. Nine models
  5. Memory fades with every return
  6. What to keep in context
  7. Do the scores agree with people?
  8. Citation

TL;DR

A video world model generates a world you can move through. Moving means things leave the frame and come back into it. A model that only produces good frames can still forget a person’s face, swap an object, rearrange a room, or freeze an event that should have kept going while the camera looked away.

WorldMemBench tests this directly. In 260 cases across 23 categories, the camera follows a controlled path away from something it has seen and back, once or several times. Static content, people, objects and scenes, is compared at every return against the same reference from the start. Dynamic events, such as pages being turned or ice melting, are checked with case-specific questions about whether they progressed while out of view. Video quality and camera-following accuracy are reported alongside, so that memory scores can be read in context.

Across nine recent world models, the one that remembers static content best, HY-WorldPlay, starts only 31.7% of the events it is asked to show; and every model forgets more with each return. No model leads on all four dimensions.

1What world memory is

Autoregressive video generation has become long and stable enough, and camera and action control precise enough, that “video world models” now let you explore a generated place for tens of seconds. Exploration constantly moves content in and out of view. We call world memory the ability to keep what has been established, and to keep it evolving coherently, across those interruptions.

It has two halves. Static memory: a person’s identity and clothes, an object’s look and shape, a room’s layout and depth should all be the same when you look again. Dynamic memory: something that was happening should have continued while you looked away, and its consequences should be visible when you return. Frame-level quality and local smoothness say nothing about either.

Earlier benchmarks examine pieces of this, such as revisit consistency, or state evolution while unobserved, and one recent benchmark tests recall across repeated loops. WorldMemBench puts them together: long revisits, repeated revisits, state evolution and physical consistency, with camera control and video quality measured in the same rollouts.

2Leave, and come back

Every case starts from a conditioning frame, the initial view V. The camera then follows a prescribed path that takes it away and brings it back: a departure Di followed by a revisit Ri. Rotations keep the camera in place and turn it left and right (yaw) or up and down (pitch); translations keep its heading and walk a square. Figure 1 plays real rollouts from the benchmark, all generated by LingBot World v2.

Loading the rollout…
V

Figure 1. The departure-and-revisit protocol on one street scene (environment case 010, generated by LingBot World v2). LRLRLR: the camera stays in place and turns 80° left and back three times (1° per frame, 80 frames per turn), returning at frames 160, 320 and 480. LRLR: two 90° turns (0.75° per frame, 120 frames per turn). LRUD: a 90° left–right cycle, then a 90° up–down cycle, so the second return comes along a new axis. Translation (a different case): the camera keeps its heading and walks forward, right, back and left around a square; the dot’s position is approximate. Each return is scored against the same initial view V; the thumbnails fill in as the video reaches each return.

260 cases, 23 categories

The benchmark has 60 Human, 70 Object and 70 Environment cases, which form the 200-case static-memory set, and 60 Dynamic cases covering seven kinds of state change: motion, assembly, removal, deformation, transfer, reaction and thermal change. Video quality is measured on all 200 static rollouts, and camera following on 85 of them whose trajectories suit pose estimation.

Composition

Four families, 23 categories

Figure 2. Case families and their categories (paper Figure 2). Hover a block for its family.

Scoring what stays

For static content, each revisit is compared with a fixed reference from the initial view, never with the previous return, so that errors cannot hide by drifting slowly. Scores are averaged over revisits and then over targets, and a prescribed revisit with no valid observation counts as zero.

  • People: identity from ArcFace face embeddings, and appearance from DINOv2 features inside the person’s mask.
  • Objects: the target is found again at each return by detection and video segmentation; appearance from DINOv2 features in the mask, and geometry as the IoU of the silhouettes after normalizing position and scale.
  • Environments: visible content from global DINOv2 features at the return frame whose camera orientation is closest to the start; relative depth order as the Spearman correlation of depth maps; and 3D structure by projecting both depth maps onto a shared spherical grid and comparing the reconstructed surfaces.

Scoring what changes

A dynamic event cannot be scored by similarity to the first frame, because the right answer is a different frame. Each dynamic case instead specifies an initial state, an action and the expected next state. A vision-language model (Qwen3.8-27B) first decides whether the event visibly starts at all; if it does not, the case scores zero. Otherwise it answers eight yes/no questions with fixed reference answers, two each on return progress, persistence, final state and causal or physical consistency. Figure 3 shows a real case.

hold
Figure 3. A dynamic case (case 023, generated by LingBot World v2): a hand turns the pages of a book. The camera holds, turns 60° away, holds while the pages should keep turning off-screen, turns back, and holds on the result. The eight checklist questions and their reference answers are the case’s released checklist; each lights up when the part of the video it is judged on is playing.

3Nine models, four dimensions

We evaluate HY-WorldPlay, Matrix-Game 3 and 3.5, SANA-WM, YUME, LingBot World and LingBot World v2, Evoke and DreamX World, each with the defaults of its official release and fixed seeds, converting the benchmark trajectories into each model’s own control format.

WorldMemBench, nine models

All metrics, coloured by rank within each metric

Figure 4. Full results (paper Table 2). Darker means better within a row; for the two camera errors (↓), lower is better. Hover a cell for its rank.

Static and dynamic memory are different skills. HY-WorldPlay leads all seven static-memory metrics, from human identity (0.828) to environment reconstruction (0.619), yet its dynamic-memory score is 0.294: it starts only 31.7% of the prescribed events. LingBot World (0.808) and YUME (0.775) lead dynamic memory while sitting in the middle on static metrics. No model leads on both: keeping the world unchanged and keeping it moving pull in different directions (Figure 5).

Each point is a model

Static memory (mean of seven metrics) against dynamic memory

Figure 5. The horizontal axis is our average of the seven static-memory metrics in Table 2; the vertical axis is the Dynamic Memory Overall score. Point size is the imaging-quality score.

Camera following varies independently: Matrix-Game 3 has the lowest translation error (0.424) and HY-WorldPlay the lowest rotation error (0.417), and DreamX World, with the highest of both, also has the lowest environment depth and reconstruction scores. Video quality, meanwhile, varies far less across models than memory does, and says little about it.

4Memory fades with every return

A good first return shows that a model can recover a view once. It does not show that the recovery lasts. Under LRLRLR, the nine-model average falls from the first to the third return on all seven static metrics: object appearance by 0.229, human identity by 0.217, object geometry by 0.205 and environment reconstruction by 0.196. Per model, every one is worse at the third return than the first on the human, object, visible and reconstruction metrics.

Two things compound. The returning viewpoint drifts: mean rotation error grows from 0.430 to 0.502 to 0.530 over the three returns. And every model generates autoregressively, so early errors feed later frames. The decline survives restricting the analysis to the 431 environment rollouts whose departures and returns meet the pose criteria: visible content, depth order and reconstruction still fall by 0.152, 0.135 and 0.139.

One environment case, all nine models

Visible memory at each return

Figure 6. Scores at the first, second and third return for one environment case (paper Figure 4). Hover a line to pick out a model. HY-WorldPlay and Evoke hold their scene; others lose it, sometimes abruptly.

Changing direction is harder still. LRLR and LRUD share their first return but differ in the second: LRLR turns left and right again, LRUD pitches up and down. Returning along a new axis costs more for every static metric on average, and for 50 of the 63 model–metric pairs; environment visible memory drops an extra 0.152.

5What to keep in context

Because the benchmark separates dimensions, it can be used to diagnose design choices. Holding LingBot World v2 and all generation settings fixed, we compared three training-free ways of choosing which past frames the model attends to: the full history (FullKV), only the recent window (Sliding), and recent frames plus view-relevant history retrieved by field of view (FoV), the last two under the same budget.

LingBot World v2, model and settings fixed

Memory scores under three context policies

Figure 7. Eleven memory metrics and end-to-end latency over 660 matched videos (paper Table 3). FoV is best on eight of eleven metrics and beats Sliding on all eleven.

FoV beats the full history on all four dynamic sub-metrics and both human metrics, while full history keeps an edge on object appearance, object geometry and visible environment content. Both budgeted policies are faster: 262.7 and 260.4 seconds per video against 322.2 for full history, about 1.23× and 1.24×. Mixing recent context with view-relevant history looks like a good trade, and the benchmark shows which kinds of memory each choice helps.

6Do the scores agree with people?

For static memory, three annotators judged 504 blinded pairs of rollouts. The automatic metrics agree with the human majority on 78.5% to 92.5% of decisive pairs, and model rankings correlate with human win rates up to ρ = 0.988 for visible environment memory. For dynamic memory, three annotators answered all 400 checklist items of 50 cases; the VLM judge’s overall scores correlate with theirs at Spearman ρ = 0.732 (PLCC 0.808). Causal and physical questions are the hardest to judge, at ρ = 0.401.

Static metricAgreement with human majorityModel-level Spearman ρ
Environment visible92.5%0.988
Human appearance88.8%0.952
Human identity85.5%0.929
Object appearance83.8%0.699
Environment depth81.8%0.301
Object geometry78.5%0.606

Citation

@article{zhou2026worldmembench,
  title   = {WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models},
  author  = {Zhou, Nan and Gao, Jianxiong and Wang, Qiuyu and Wang, Wen and Wang, Jiahao and Yuan, Yufeng and Li, Shuaiting and Ouyang, Hao and Shen, Yujun and Lu, Weiming and Xiao, Jun and Zhuang, Yueting and Shen, Yongliang},
  journal = {arXiv preprint arXiv:2610.19063},
  year    = {2026}
}

References

  1. Sun et al. (2025). WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling.
  2. Huang et al. (2025). Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion.
  3. Yu et al. (2025). Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval.
  4. Huang et al. (2024). VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR.
  5. Duan et al. (2025). WorldScore: A Unified Evaluation Benchmark for World Generation.

Inference settings for every model, the dynamic-memory prompts and the full per-trajectory results are in the paper; rollouts, scores and the evaluation code are on Hugging Face and GitHub.