Skip to content
MemU-Bench Code

Benchmark · Android GUI agents · user memory

MemU-Bench

Benchmarking GUI Agents on Long-Horizon User Interactions

Yizhou Liu1, Zhengxi Lu1, Fei Tang1, Yiwen Qiu1, Yong Du1, Tongbo Chen1, Bofan Chen1, Tao Jiang2, Wenhao Xu2, Yunfeng Shu1, Wenqi Zhang1, Weiming Lu1, Jun Xiao1, Yueting Zhuang1, Yongliang Shen1,†

1Zhejiang University2Ant Group†Corresponding author

Real users rarely spell everything out. They say “delete the alarm I just set” or “save the first call this morning as a contact”. MemU-Bench gives an Android agent the user's raw interaction history and asks it to finish such vague tasks, which can only be solved by recalling what the user did before. Outcomes are checked against the apps' own databases.

Assistantonline
Today

Please delete the alarm I just set.

Save the first call this morning as a contact named Courier in contacts app.

Play my favorite song in retro music app.

Which alarm? Which call? Which song?

User history · earlier today
  1. Clock app with a 7:30 alarm newly setClock
  2. Phone call log with an incoming 8:12 call from 555-0142Phone
  3. Music player playing Sunny DayMusic
  4. Photo galleryGallery
  5. Chat conversationChat
  6. Map showing a parkMap
frames that resolve the vague phrases
118
vague tasks
29
user trajectories, 100–800 steps each
30+
apps, up to 179 app switches in one trajectory
71 / 47
Level-1 / Level-2 tasks (< 400 / > 700 history screenshots)
3
metrics: SR · IE · SPL

Numbers from the paper. The environment builds on AndroidWorld (Pixel 6 emulator, API 33).

The problem

Every instruction leaves a gap that only the user's history can fill

Users omit details they no longer remember, so the missing piece is an objective fact that was visible in their past interactions and is now buried in hundreds of steps of routine phone use. In the code, each task has two forms: agents are evaluated on the vague goal (--eval_setting unclear_goal), while the full goal spells out the missing value. Pick a task to see the gap. All text below is taken from the task files and eval_results/human.json.

vague goal

full goal

How a task runs

From an ambiguous request to a scored episode

One task from the suite, played step by step with the AppMem pipeline. The instruction is the real ambiguous goal of User0Short_CalendarAddEvent and the human step count comes from human.json; the screenshots, memory entries, retrieved text and the agent's 18 steps are illustrative.

User history user_0 · Level-1 · screenshots + actions
  1. Call log with 138 1122 3344 at the top, 08:47 Phoneclick08:47
  2. Homeswipe09:20
  3. Map app, typing Barstow in search Mapstype09:21
  4. Map place card for Barstow with a Save button Mapsclick09:21
  5. Map with a red pin and Marker saved Mapsclick09:22
  6. Alarm list with 8:30 and 9:10 Clockclick09:30
  7. Alarm time picker at 14:00 Clockclick09:31
  8. Alarm list with 8:30, 9:10 and 14:00 Clockclick09:31

Level-1 histories have < 400 screenshots, Level-2 > 700. Shown with a 4-frame window; the paper uses W = 10.

Assistantnow

I plan to have a picnic at the location I just marked on the mapBarstow from 2 p.m. to 6 p.m. the day after tomorrow. Please help me add this schedule to the simple calendar app.

PhoneSMSMapsClockContactsFilesCalendarExpense
Calendar app: New event form with Picnic, Barstow, 2:00 to 6:00 PM Event saved
? Nothing on the current screen says which location was marked.
#1 open_app("Calendar")
AppMem memory one entry per app
phonefinished

Answered an incoming call from an unsaved number in the morning.

Details · 138 1122 3344
osmandongoingmerged

Searched the map for Barstow and opened the result.

Details · Barstow
clockfinished

Created a new afternoon alarm.

Details · 14:00
App-guided retrieval round 1 of up to 3
  1. Application inference phoneosmandclock
  2. Memory retrieval Read only the osmand entry (1 of 3).
  3. Information filtering “The user searched for Barstow in OsmAnd and saved it as a map marker.”
  4. Sufficiency verification Sufficient Execute now; otherwise drop the explored apps and infer again.
Evaluation eval_checkpoint.py → eval_metrics.py

Rule-based check · calendar database before vs. after

  • ✓ title “Picnic”
  • ✓ location “Barstow”
  • ✓ 2:00 PM – 6:00 PM, day after tomorrow

is_successful = 1 → counts toward SR

Steps against the human reference

human16
agent18

16 / max(18, 16) = 0.89 → this task's term in IE (averaged over successful tasks) and SPL (averaged over all tasks)

Proposed method

AppMem: memory written per app, retrieved per app

AppMem is a training-free framework that turns the raw history into an app-centric memory bank, keeps interrupted intents together across app switches, and answers a vague instruction by first inferring which apps could hold the missing context. Every step is run by the evaluated model itself; in the code both stages use the model set in .env and either can be swapped for your own method.

1 Memory generation

A fixed-size window (W = 10 screenshot–action pairs) slides over the history. Inside each window three stages turn the raw trace into memory entries, one per app:

  • Current event summarization — summarize the window into entries ⟨app, event, details, tstart, tend, status⟩, where details keep on-screen facts such as prices and status is ongoing or finished.
  • Latest memory fetching — for each new entry, find the most recent entry of the same app in the memory bank.
  • Ongoing intent continuation — if that entry is still ongoing, merge the two; if it is finished or absent, append the new entry. Each app's ongoing activity stays in one entry and finished events are never overwritten.
App:        osmand
Event:      searched the map for Barstow, saved a marker
Details:    Barstow            # hard data: numbers, names, sizes
State:      ongoing | finished
Start Time: …   End Time: …

Field layout from memory_generation/memory/base.py; the values are illustrative.

2 App-guided retrieval

Instead of searching the whole history, retrieval first infers which apps could hold the missing context and reads only their entries, in up to K = 3 rounds.

  1. Application inferencefrom the vague instruction, infer a subset A* of the apps that appear in memory
  2. Memory retrievalkeep only the entries of A*, an app-specific sub-bank
  3. Information filteringdistill those entries into the key information κ for the task
  4. Sufficiency verificationif κ suffices, execution begins; otherwise drop the explored apps from the candidates and infer again

In the code, κ is saved as user_history next to the goal and given to the agent at every step as “generated information about the user's previous operations”.

AppMem overview: a sliding window over the frame sequence produces memory through current event summarization, latest memory fetching and ongoing intent continuation; app-guided retrieval runs application inference, memory retrieval, information filtering and sufficiency verification before the GUI agent executes on the emulator.
AppMem overview (Figure 4 of the paper).

Benchmark design

Noisy long histories, restored phones, database-checked outcomes

Long, noisy histories

Operators followed intent-level sketches on an emulator, using it like a real phone: accidental touches, redundant app switches, sessions spread over hours. Screenshots and actions were logged every 0.3 s and identical frames removed.

Level-1: < 400 history screenshots · Level-2: > 700

Restored device

After each trajectory the emulator state is frozen: system time recorded and every involved app's database pulled. Restoring that snapshot (e.g. snapshot_0_short) reproduces the exact phone the operator left behind.

Checked results

A cascade of checks: rule-based verification of the app's database records; an LLM judge only for free-text fields such as message bodies; a VLM judge on screenshots for the few tasks whose database is inaccessible. Human step counts give the efficiency reference.

Where the tasks are carried out

Tasks per target app (Table 7 of the paper; the same counts as app_name in task_evals/fixed_tasks/).

    By app category (Figure 3 of the paper)

    MemU-Bench overview: (a) benchmark architecture with apps, vague tasks, user trajectories and user databases; (b) evaluation pipeline where memory generated from trajectories is retrieved for the GUI agent, which controls the emulator, followed by rule-based and LLM-as-judge evaluation.
    Benchmark architecture and evaluation pipeline (Figure 2 of the paper).

    Metrics

    Did it finish, and how many taps did it take?

    Success alone is not evidence of memory: an agent can finish a vague task by searching app after app. IE and SPL therefore compare the agent's steps with a human's on the same task (eval_results/human.json, 2 to 19 steps per task). eval_metrics.py also reports SR and SPL separately for Level-1 and Level-2.

    SR

    Success Rate

    Share of the N tasks that end in success; C is the set of successful tasks.

    SR=|C|N
    SPL

    Success weighted by Path Length

    Success and efficiency together: every task counts, failures score 0, successes are discounted when the agent takes more steps than the human.

    SPL=1N∑i=1N𝟙i∈C⋅himax(ai,hi)
    IE

    Interaction Efficiency

    Step efficiency over successful tasks only, in [0, 1]. A low IE means the agent reached the goal by exploring for clues instead of recalling them.

    IE=1|C|∑i∈Chimax(ai,hi)

    hi: human steps on task i · ai: agent steps on task i.

    Results

    Without history, agents wander; with AppMem, they remember

    Nine backbones, each run three ways: Vanilla (no history), w/ Summary (the same window iteratively summarizes the history, and key information is retrieved from the summary) and w/ AppMem. The evaluated model performs all memory and summary operations itself; W = 10, K = 3, temperature 0, one complete run per model.

    16.4%best SPL without history (Gemini-3.1-Pro)
    < 5%SPL of every end-to-end GUI agent without history
    15.7 → 66.9GPT-5.5 SPL with AppMem (+51.2)
    +9.3SPL over the Summary baseline (GPT-5.5)

    Table 2 of the paper. SR and SPL in %, IE in [0, 1]; higher is better, except for the L1 − L2 gap (δ = SPLL1 − SPLL2, computed from the table; closer to 0 means more robust to long histories). Bold marks the best value within each backbone; differences are against Vanilla.

    Which part of AppMem matters

    Ablation on Qwen3.5-397B-A17B (Table 3). MS: memory size; TE: retrieval tokens per successful task; CT: retrieved tokens added at execution. Lower is better for MS, TE, CT.

    SettingSPLMSTECT
    Full AppMem47.18.8K6.1K0.10K
    w/o intent continuation42.415.3K10.7K0.11K
    w/o app inference48.58.8K17.8K0.15K
    w/o info filtering41.88.8K9.7K2.89K
    w/o sufficiency check45.38.8K4.9K0.09K

    Intent continuation keeps memory complete and compact (SPL −4.7 and memory 8.8K → 15.3K tokens without it); information filtering keeps the agent's context small; app inference mainly cuts retrieval cost (17.8K → 6.1K tokens) at nearly the same SPL.

    App-guided retrieval vs. alternatives

    Qwen3.5-397B-A17BIESRSPL
    w/ Inquiry (asks the user)0.6239.024.1
    AppMem, dense RAG retrieval0.8743.237.5
    AppMem, app-guided0.9052.547.1

    Replacing app-guided retrieval with a bge-m3 retriever over the top-3 entries costs 9.3 SR and 9.6 SPL (Table 4); on GPT-5.5 it drops from 66.9 to 50.5 SPL. Asking the user for coarse clues instead of using history lifts SR, but extra steps keep IE at 0.62 (Table 6). Retrieval is also cheaper than with a summary: 8.5K → 4.2K tokens per successful task for Gemini-3.1-Pro, 12.0K → 9.9K for Claude-Opus-4.8 (Table 5).

    The bottleneck is memory, not execution: given precise instructions, MAI-UI-8B reaches 61.9% SR, a 55.1-point gap over its 6.8% with summarized history. In a manual error analysis of the Summary baseline, 57.6% of tasks fail with the key information missing, against 6.8% that fail although it was retrieved.

    Quick start

    Run the AppMem pipeline

    Download the data from Hugging Face (apks/, user_database/, user_history/), create the MemUBenchAvd emulator (Pixel 6, API 33) and set MODEL, API_KEY, BASE_URL in .env. Full instructions are in the README.

    1. Initialize the emulator once

      python batch_restore_and_snapshot.py --install-apps --apk-dir ./apks
    2. Generate memory from the logs

      python memory_generation/memory/generate_memory.py \
          --session_paths ./user_history \
          --output_path ./user_information/memory/${MODEL_NAME}.json \
          --env_file .env
    3. Retrieve information per task

      python retrieve_information/retrieve_information.py \
          --method memory \
          --path ./user_information/memory/${MODEL_NAME}.json \
          --output ./relevant_information/memory/${MODEL_NAME}.json \
          --log_path ./retrieve_log/memory/${MODEL_NAME}.txt \
          --env_file .env
    4. Run the agent on the emulator

      python run_memubench.py \
          --suite_family memory_suite \
          --checkpoint_dir ./checkpoint_results/${MODEL_NAME}/memory/run1 \
          --method with_history \
          --user_information_path ./relevant_information/memory/${MODEL_NAME}.json \
          --eval_setting unclear_goal \
          --env_file .env
    5. Score against the human reference

      python eval/eval_checkpoint.py \
          --checkpoint_dir ./checkpoint_results/${MODEL_NAME}/memory/run1 \
          --output ./eval_results/${MODEL_NAME}_memory.json
      python eval/eval_metrics.py \
          --ref ./eval_results/human.json \
          --eval ./eval_results/${MODEL_NAME}_memory.json \
          --output ./metric_results/memory.json

    Bring your own memory method

    Replace the memory generator with any script and its output can take any shape. Add a retrieval module with get_filtered_history(goal, path, …), register it in retrieve_information/__init__.py and pass --method your_own_method. Execution and scoring stay unchanged as long as the retrieval output looks like this:

    {
      "results": [
        {
          "goal": "the ambiguous task instruction",
          "subdir": "short",
          "user_history": "the retrieved information for this task"
        }
      ]
    }

    BibTeX

    @article{liu2026memubench,
      title   = {MemU-Bench: Benchmarking GUI Agents on Long-Horizon User Interactions},
      author  = {Liu, Yizhou and Lu, Zhengxi and Tang, Fei and Qiu, Yiwen and Du, Yong and Chen, Tongbo and Chen, Bofan and Jiang, Tao and Xu, Wenhao and Shu, Yunfeng and Zhang, Wenqi and Lu, Weiming and Xiao, Jun and Zhuang, Yueting and Shen, Yongliang},
      journal = {arXiv preprint arXiv:2611.00191},
      year    = {2026}
    }
    
    @misc{chen2026knowubench,
          title={KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation},
          author={Tongbo Chen and Zhengxi Lu and Zhan Xu and Guocheng Shao and Shaohan Zhao and Fei Tang and Yong Du and Kaitao Song and Yizhou Liu and Yuchen Yan and Wenqi Zhang and Xu Tan and Weiming Lu and Jun Xiao and Yueting Zhuang and Yongliang Shen},
          year={2026},
          eprint={2604.08455},
          archivePrefix={arXiv},
          primaryClass={cs.AI},
          url={https://arxiv.org/abs/2604.08455},
    }