JohnXu22786/computer-control

简体中文

computer-control — Desktop Control Plugin (for dsh)

Lets an agent operate the computer desktop directly: screenshot observation, mouse/keyboard injection, semantic actions via the accessibility tree (semantics first, pixel-coordinate fallback), with built-in safety guards — emergency stop, allow/deny rules, confirmation flow and idle standby.

The plugin is self-contained and loadable by dsh directly: it declares tools and events via manifest.json, speaks line-delimited JSON-RPC 2.0 (stdio) as its transport protocol, and python -m computer_control serve is the entry point.

---

Feature Overview

Capability	Description
Screenshot screen.capture	Full-screen or region capture, PNG/JPEG, scaling, grayscale — token costs under control
Mouse pointer.*	Move, left/middle/right click (double/triple), drag, scroll (horizontal/vertical)
Keyboard keyboard.*	Single keys, chords (scancode injection, layout-independent), arbitrary Unicode text input
Wait wait.pause	Pause between actions so the UI can settle before the next screenshot
Semantic actions a11y.*	Hierarchical accessibility-tree summaries (skeleton/standard/full), semantic activate/input — UIA mode first, automatic fallback to bounding-box pixel clicks
Batch batch.execute	Run several actions in one call to reduce model round trips; confirmable as a whole, can continue on error
Capability broadcast	tools.list / system.status report per-backend availability; semantic tools are only exposed when UIA is available

Safety guards (see the "Safety" section):

• Emergency stop: triple trigger — global hotkey (default Ctrl+Alt+F12, configurable), protocol command, panic file; a STOP banner shows in the screen corner
• Allow/deny rules: matched by tool name and arguments, deny always wins; whitelist mode switchable
• Confirmation flow: high-risk actions (including chords with win or ctrl+alt) wait for human approval, auto-denied on timeout
• Idle standby: no activity past a threshold puts the session into standby, rejecting all actions until resumed
• Dry-run mode: platform: "dry-run" logs actions without executing them, for safe rehearsals

Directory Structure

computer-control/
├── manifest.json            # dsh plugin manifest (tools/events/entry/transport)
├── pyproject.toml           # packaging metadata and dependency declarations
├── requirements.txt         # core dependencies (Pillow only)
├── requirements-optional.txt# optional capability deps (UIA/mss/hotkey)
├── README.md
├── docs/                    # integration / protocol / actions / configuration
├── examples/                # config examples, session examples, demo scripts
├── computer_control/        # plugin implementation (Python package)
│   ├── cli.py __main__.py   # entry: serve / check / list
│   ├── session.py           # session lifecycle and serial action execution
│   ├── engine.py            # action execution engine (coordinate mapping, events)
│   ├── policy.py            # safety gate: rules/confirmation/panic/watchdog
│   ├── actions.py           # action registry and argument validation
│   ├── geometry.py          # model canvas <-> physical pixel mapping
│   ├── protocol.py server.py# JSON-RPC routing and stdio/HTTP transports
│   ├── client.py            # client for harness/scripts
│   ├── overlay.py           # emergency-stop visual banner (optional, tkinter)
│   ├── drivers/             # execution-layer abstraction + windows impl + dry-run driver
│   └── a11y/                # accessibility-tree summaries + Windows UIA bridge
└── tests/                   # pure-logic and protocol tests (no real hardware)

Installing in DSH

dsh plugin --profile demo add github:JohnXu22786/computer-control

Remove with:

dsh plugin --profile demo remove computer-control

dsh bundle

The repo also ships a Cordis bundle so anything that consumes dsh.bundle manifests can install it the same way: package.json declares dsh.bundle.patch pointing at cordis.patch.yml, and index.js is the bridge a dsh profile loads. It spawns python -m computer_control serve as a child process and re-exposes every tool from manifest.json to the harness over the stdio protocol — the Python core is left untouched. Profile-level settings can be pinned through the patch row's config (e.g. platform.name: dry-run for a rehearsal-only install).

Requirements for the bridge: node >= 18 and a Python 3.9+ interpreter with this package installed (pip install -r requirements.txt is enough for the core tools).

Installation

Requirements: Python 3.9+, Windows 10/11 (full functionality); other platforms see "Platform Support".

# Core (screenshot + input injection)
pip install -r requirements.txt

# Optional capabilities (recommended):
#   comtypes  -> accessibility-tree (UIA) semantic actions
#   mss       -> faster multi-monitor screenshot backend
pip install -r requirements-optional.txt

Environment self-check:

python -m computer_control check

Prints diagnostics for platform, DPI mode, virtual-desktop geometry, screenshot backend, UIA availability, hotkeys, etc.

Quick Start

# List declared tools and events
python -m computer_control list

# Start the plugin service (stdio, for dsh to load)
python -m computer_control serve

Run a rehearsal session with examples/demo.py (defaults to dry-run, touches nothing on the real desktop):

python examples/demo.py

dsh Integration (Summary)

Full integration notes: docs/integration.md.

1. Loading: dsh reads manifest.json, starts the process via entry.command, and establishes a stdio pipe (UTF-8, one JSON object per line).
2. Lifecycle: send session.start (optionally with config) → wait for the session.started event, then call tools; send session.stop at the end.
3. Calling actions: tools.call (single) or tools.call_batch (batch). Responses use a unified {ok, result, error, meta} envelope.
4. Events: the server pushes action.started/finished, safety.confirmation_requested, etc. as event notifications.
5. Confirmation flow: high-risk actions return awaiting_confirmation and emit a confirmation event; the harness should surface a human approval prompt, then reply via session.confirm; timed out confirmations are auto-denied.

The recommended model loop: screen.capture to observe → execute pointer.* / a11y.* on canvas coordinates → wait.pause (if needed) → a fresh screenshot to verify the result.

Action Reference (Summary)

All actions, parameters and examples: docs/actions.md.

Action	Purpose	Risk
screen.capture	Screenshot (region/format/scale/grayscale)	none
pointer.move	Move pointer	moderate
pointer.click	Single/double/triple click, optional position	moderate
pointer.drag	Press and drag	moderate
pointer.scroll	Wheel (horizontal/vertical)	moderate
keyboard.press	Single key	moderate
keyboard.combo	Chord (upgrades to high when it includes win or ctrl+alt)	moderate/high
keyboard.type	Text input (Unicode)	moderate
wait.pause	Pause	none
a11y.snapshot	Hierarchical accessibility-tree summary	none
a11y.activate	Semantic activation (pattern first, pixel fallback)	moderate
a11y.input	Semantic text input (value pattern first)	moderate
batch.execute	Batch execution	max of items

Coordinate Contract

The model sees a canvas, not the raw screen: screenshots are scaled to an aspect-preserving canvas whose width is display_width_px (default 1920), and coordinates returned by the model live on that canvas; the plugin maps them back to physical pixels uniformly via scale = physical width / canvas width. The screen.capture result carries a canvas field along with display_width_px/display_height_px as the source of truth. Multi-monitor setups (including negative-coordinate regions left/above the primary display) and per-monitor DPI are handled in the execution layer.

Safety

See docs/configuration.md and the points below:

• Emergency stop (triple): default global hotkey Ctrl+Alt+F12 (configurable via safety.emergency_hotkey, set empty to disable); protocol method control.panic; panic file (safety.panic_file, presence stops everything). After a panic all actions return safety_stopped; session.resume or pressing the hotkey again restores operation. A red STOP banner shows in the screen corner while engaged (safety.visual_indicator).
• Allow/deny rules (safety.rules): {match: {tool: "keyboard.*", argument: {name, matcher, value}}, effect: "deny"}; deny rules always win over allow rules. safety.default_rule: "deny" switches to whitelist mode (anything not explicitly allowed is denied). Rules can be adjusted at runtime via session.configure.
• Confirmation flow: safety.confirm_threshold (default high) decides which risk levels need human approval; safety.confirm_timeout_s (default 30s) auto-denies on timeout. Approved actions execute normally and emit action.finished.
• Idle standby: enabled when safety.idle_timeout_s > 0; after no activity for the timeout the session enters standby (emits session.idle), session.resume restores it; idle_action: "none" only emits the event without pausing.
• Dry-run mode: with platform: "dry-run" every action is logged but never executed — handy for integration debugging and safety rehearsals.

Platform Support

Platform	Driver	Notes
Windows	drivers/windows.py	Full implementation: SendInput scancode injection, per-monitor DPI awareness, virtual-desktop coordinates, mss/Pillow screenshots, UIA semantic layer
macOS / Linux	interface abstracted	drivers/base.py defines the complete driver contract (capture/pointer/keys/a11y/hotkey); implement it per platform to plug in (drivers/windows.py is a complete example). Until implemented, platform: "auto" fails with a clear error
Any platform	drivers/dummy.py	Dry-run driver: records every action, touches no hardware

Dependencies and degradation:

• Pillow (required): screenshots and encoding. The plugin refuses to start without it.
• mss (optional): faster and more reliable on multi-monitor Windows; falls back to Pillow ImageGrab when missing.
• comtypes (optional): UIA semantic actions. When missing, a11y.* tools are marked unavailable in tools.list and calls return backend_unavailable — the pixel-coordinate path (screenshot + click) is unaffected.
• keyboard (optional, reserved): global hotkeys on macOS/Linux drivers will depend on it; only the Windows and dry-run drivers ship today, and Windows uses built-in GetAsyncKeyState polling, so the package is not needed.

Known limitations:

• Secure attention sequences (e.g. Ctrl+Alt+Del) cannot be triggered via input injection — an OS-level protection that the plugin cannot bypass either.
• UIA depends on the target program exposing an accessibility interface; programs that do not (some games, self-drawn UIs) can only use the pixel path.
• Keyboard scancode injection works better with DirectInput/raw-input programs, but some anti-cheat programs may still reject it (normal protective behavior).

Testing

python -m unittest discover -s tests -v

Tests use only pure logic and the dry-run driver — no real input is injected, no real hardware is touched; the real Windows chain is verified manually via python -m computer_control check and examples/demo.py --live.

License

MIT — see LICENSE. Copyright (c) 2026 JohnXu22786.