REAL Lab logo CUA-JEV GitHub

OPEN REFERENCE FRAMEWORK · WINDOWS FIRST

Jev for
computer use.

An extensible framework for Jev-guided Windows computer use. Four verified workflows show Jev choosing between GUI and structured tools.

Watch the cases See the framework

Four curated workflows and one experimental open-task pilot. The recorded pilot uses text and DOM; optional one-window screenshot/VLM grounding is now integrated but not live-validated. Arbitrary Windows tasks are not yet supported.

CUA × JEV

How Jev powers computer use

For a supported task, Jev chooses from actions supplied by its adapter.

  1. 01

    Adapter observes

    Read structured app state and offer the actions available now.

  2. 02

    Jev chooses

    Select one offered GUI or tool route for the next step.

  3. 03

    Runtime acts & checks

    Guard the choice, run the tool, and verify the new state.

Repeat with the new state until the task is done.
4verified workflows
6action channels
0router training runs
—best paired speedup

THE FRAMEWORK

A reusable Jev implementation for CUA.

Each supported task defines its structured state, legal actions, and success checks. Jev selects the next action; the runtime guards, executes, and verifies it.

01 · OBSERVE

Frame a typed decision

A task-specific adapter turns structured app state into the next set of legal choices for Jev.

Jev’s role
Receives the current subgoal, structured state, and only the legal actions available now.
Runtime’s role
Reads DOM, UI Automation, COM, terminal, or filesystem state through the task adapter.
01

CUA implementation framework

Build a task adapter with observation, candidate actions, execution, and verification under one runtime contract. The four examples are not general app automation.

02

Fast hybrid action selection

Jev chooses between visible GUI work and faster structured tools directly. Tool-rich workflows avoid long chains of clicks without SFT, RL, or router distillation.

VERIFIED CASES

Four Windows workflows cases.

Paired Hybrid and GUI Only runs start from the same task and use the same terminal verifier.

MEASURED RESULTS

Hybrid avoids the GUI bottleneck.

Jev keeps GUI available when it is necessary, then switches to MCP, CLI, COM, DOM, scripts, or APIs when they are faster and verifiable.

One decision surface

GUI and structured tools compete as typed candidates under the same policy.

No routing model to train

Jev makes constrained selections directly from the current state and legal actions.

End-to-end evidence

Wall time, action mix, receipts, and terminal verification come from real runs.

Two comparisons · one task

Loading

01 · ACTION SPACE

Hybrid vs GUI Only

Same Jev policy. Compare completion time.

—Hybrid speedup
Jev routeWall time
02 · AGENT BASELINE

Jev vs Codex Computer Use

Both Hybrid. Compare completion time and model cost.

Hybrid agentWall timeModel cost (USD)

Published Jev API and Codex Enterprise token rates; actual bills may differ.

OPEN-TASK PILOT · REAL EXECUTION

A model finds the next move.
Jev chooses how to execute it.

The model interprets live application state and proposes a grounded intent. The framework turns that intent into safe, typed GUI and structured-tool routes; Jev selects one. Execution and independent state checks close the loop.

Live state feeds a model planner; the framework builds legal routes; Jev selects one; the runtime executes, verifies, and reobserves.
  1. ObserveRead live DOM, accessibility, and file state; optional one-window VLM targets.
  2. Model plansPropose the next intent using current evidence.
  3. Framework builds routesOffer only legal GUI and structured-tool actions.
  4. Jev choosesSelect one typed candidate for execution.
  5. Act and verifyCheck the changed state, then repeat.

Recorded case Text-model planning over live DOM; Jev selection across real execution channels. The video below does not use a VLM.

New · experimental In one selected Windows window, an optional VLM can turn a screenshot into bounded GUI targets when UI Automation is weak. Jev chooses from the guarded candidates; offline tests pass, but no completed live VLM task or broad generalization is claimed.

ONE VERIFIED CASE

Model + Jev: a successful multi-app workflow.

Starting from the official Python tutorial index, the text model proposed the next intent at each step; Jev selected its execution route. The run found five chapter pages, collected one exact quote per page, wrote a sourced note, and opened it in VS Code. No chapter URL or click sequence was encoded in the agent.

12Model plans
12Jev decisions

Real Windows execution · 155.9 s wall time · video played at 2.5× speed.

Explore the 12 Jev decisions
  1. DOM · Edge Open “Whetting Your Appetite”.
  2. Internal · evidence Save a source quote.
  3. DOM · Edge Open “Using the Python Interpreter”.
  4. Internal · evidence Save a source quote.
  5. DOM · Edge Open “An Informal Introduction to Python”.
  6. Internal · evidence Save a source quote.
  7. DOM · Edge Open “More Control Flow Tools”.
  8. Internal · evidence Save a source quote.
  9. DOM · Edge Open “Data Structures”.
  10. Internal · evidence Save a source quote.
  11. File API · local Write the cited study guide.
  12. CLI · VS Code Open the verified result.

This is one successful text-and-DOM case, not a repeated benchmark or a vision demo. A separate checker revisited all five source pages and confirmed the quoted text, URLs, Jev decisions, and output file; it did not grade the written synthesis. Future work includes live VLM task evaluation, broader tasks, reliability, latency, and cost.

Implementation, reproduction, and limits
REAL Lab logo

OPEN SOURCE · APACHE 2.0

Build the next Jev-powered CUA capability.

Explore the code ↗