Reflect
Diagnose the executed trajectory under strict information isolation.
EvoSkill-GUI: Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents
Live skill stateS(i)
Revise
Edit only the responsible file through a restricted tool interface.
Reuse
Retrieve verified procedural knowledge for related future tasks.
skill://plan.md updated: verify attachment before send
+16.2pts
MobileWorld
+6.0pts
AndroidWorld
+10.5pts
OSWorld
One backbone. No training.
Skills evolve directly from deployment-time feedback.
01 / Overview
Skills should be living procedural knowledge.
GUI agents operate in non-stationary environments where pop-ups, delayed loads, and relocated widgets routinely invalidate plans fixed before execution.
Existing skill-based agents largely treat skills as static artifacts. EvoSkill-GUI instead represents every skill as a structured, editable package and turns failed execution into supervision for improving that package. During rollout, the executor can revise local mismatches instantly. After failure, the same backbone acts as an isolated critic, diagnoses the trajectory, and guides targeted skill-file edits.
Across MobileWorld, AndroidWorld, and OSWorld, EvoSkill-GUI consistently improves general-purpose and GUI-specialized models without additional training. The evolved library continues to benefit related tasks instead of rebuilding procedural knowledge from scratch.
02 / Method
A closed loop from execution to reusable skill.
EvoSkill-GUI alternates between in-rollout adaptation, information-isolated diagnosis, and targeted file revision. Successful packages return to a metadata-indexed skill library.
01
Execute + adapt
The executor follows the current package and performs instant revisions when the observed interface contradicts the plan.
tau(i) = Phi(S(i), E)
02
Reflect in isolation
The same model inspects only the instruction, observations, and actions: no hidden reasoning, skill body, or ground truth.
Ccritic ∩ {S, CoT, GT} = ∅
03
Revise precisely
Structured critique identifies the failure step, direct cause, and a concrete edit to the responsible skill component.
S(i+1) ~ pi(S(i), c(i), tau(i))
04
Retrieve + reuse
Metadata retrieval matches intent, app, platform, keywords, arguments, history, and verification status.
score(q, S) ≥ thetar
Inside a skill package
Six files. Six distinct responsibilities.
A monolithic skill entangles planning, localization, and recovery. EvoSkill-GUI separates them so each failure updates only what broke.
Restricted tools
readwriteappendlistsearchcreate_failure{
"intent": "send email with attachment",
"app": "Gmail",
"platform": "Android",
"keywords": ["compose", "attach", "send"],
"status": "verified"
}03 / Results
Consistent gains across mobile and desktop.
All evaluations keep the package format and retrieval strategy fixed. Tasks receive up to 50 interaction steps and at most two post-failure revision rounds.
Best absolute gain+16.2Qwen3.6-Plus
Every evaluated backbone improves on the GUI-only split, from general closed models to GUI-specialized open models.
Full reuse gain+6.0Seed 42 variants
Phase 1 constructs and evolves packages. In Phase 2, every task retrieves the evolved library built on related variants.
37.9% reuse rate
68.1 base → +2.6library evolves
100% reuse rate
55.2 base → +6.0Desktop gain+10.5Qwen3-VL-8B
The same skill package design transfers to heterogeneous workflows across browsers, creative tools, office apps, email, media, and development environments.
| Backbone | Base | EvoSkill-GUI | Gain |
|---|---|---|---|
| GUI-Owl-1.5-8B | 46.7 | 54.8 | +8.1 |
| Qwen3-VL-8B-Instruct | 23.8 | 34.3 | +10.5 |
Largest application-level gain: +42.7 on VLC with GUI-Owl-1.5-8B.
04 / What the evolution learns
The gains accumulate, transfer, and cost less than repeated sampling.
Across rounds
Skills improve with repeated use.
Most gains arrive within three rounds. Qwen3.6-Plus reaches 68.6% by Round 3 and 69.5% by Round 5.
Three-round EvoSkill-GUI
94.75M tokens for 69.5% accuracypass@3 baseline103.00M / 62.8%
Initially failed executions recovered
27 / 89 30.3% overall recovery ratevia reused skills19 / 65 repaired
Across task streams
The library grows into reusable infrastructure.
On 116 + 116 AndroidWorld tasks, the library reaches 98 skills and a 56.1% reuse ratio while maintaining higher cumulative success.
Ablation
Each design choice protects performance.
Structured files support targeted edits; instant revision prevents local mismatches from cascading; information isolation keeps critique evidence-grounded.
05 / Case studies
A failure becomes the next successful procedure.
Case studies show how a package changes after execution feedback: missing verification steps become explicit checks, and premature stopping becomes complete traversal.
Revision learned
Locate → verify → attach
The failed skill stopped after finding the file. EvoSkill-GUI writes a verification checkpoint into plan.md so the executor confirms the attachment before sending.
Revision learned
Traverse → verify completion → report
The original procedure reported too early. The revised package adds an explicit completeness check before calculating and returning the top-three item total.
06 / Citation
Build on skills that keep learning.
If EvoSkill-GUI helps your research, please cite our arXiv paper using the BibTeX entry.
View repository@misc{chen2026reflectrevisereusetrainingfree,
title = {Reflect, Revise, Reuse:
Training-Free Skill Evolution for GUI Agents},
author = {Bofan Chen and Boxuan Zhang and Fei Tang and
Zhengxi Lu and Yong Du and Tongbo Chen and
Weiming Lu and Jun Xiao and Yueting Zhuang and
Yongliang Shen},
year = {2026},
eprint = {2609.17653},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2609.17653}
}