zzy6-a/vision-use

English | 中文

Repo: vision-use · Package: dsh-vision

Let the DeepSeek Harness agent actually see your screen and operate the Windows desktop — with a Codex-style blue overlay (press Esc to abort at any time).

---

What it does

Capability	Tool	Description
👁 See	view_screen	Capture the whole Windows screen → straight into the model's vision channel (real pixels, not OCR)
👁 See	view_image	Push any image file (Linux or Windows path) into the vision channel
🖱 Act	computer_move	Glide the cursor smoothly to (x, y)
🖱 Act	computer_click	Left / right / double click with a click ripple
⌨️ Act	computer_type	Type text. method: auto (default) sends real VK keystrokes for ASCII, and switches to clipboard — saving and restoring your original clipboard — when the text contains CJK or exceeds 120 chars. Also supports uia (ValuePattern), clipboard, sendinput
⌨️ Act	computer_key	Key combinations (ctrl+t, enter, alt+f4, …)
🎛 Control	computer_overlay	Start / stop / status the overlay (configurable idle auto-close)
📊 Control	computer_status	Overlay state / Esc flag / current mode / cursor position
🎯 Target	uia_windows	List top-level windows (title / pid / class / rect / which is foreground)
🎯 Target	uia_tree	Dump a window's control tree: type, name, AutomationId, rect, supported patterns
🎯 Target	uia_find	Find controls by name (also AutomationId / ClassName) — no pixel math
🎯 Target	uia_locate	Return a close-up crop centred on a control; the reply carries no coordinates
🎯 Target	uia_click	Click a control by name through UI Automation patterns (invoke / toggle / select / expand) — does not move your mouse; returns a before/after UI fingerprint
⚡ Batch	computer_sequence	Run several actions in one call: click:x,y|type:text|key:enter|wait:ms|shot:path.png — saves 3-5 model round-trips

Visual feedback (Codex style)

┌───────────────────────────────────────────────┐
│  ╔═════════════════════════════════════════╗  │ ← soft blue border (per-pixel alpha)
│  ║  ● DeepSeek Harness is operating  [Esc]  ║  │ ← top status badge
│  ║                                         ║  │
│  ║         typing → blue ring + caret       ║  │
│  ║         click  → 0.7s ripple (pinned)    ║  │
│  ║         move   → ring follows; fades 1s  ║  │
│  ╚═════════════════════════════════════════╝  │
└───────────────────────────────────────────────┘

• One cursor set (arrow / I-beam / hand) in a unified blue-and-white gradient, while keeping Windows' native context switching
• Overlay always on: every hands operation (move/click/type/key) is forced to bring up the overlay first — no silent control
• Esc abort: the keypress writes a cancel flag → the overlay exits → every subsequent action is rejected
• Task-level keep-alive: while the agent is still working in this turn (thinking or running other tools), the overlay stays up; it packs up about 4 seconds after the turn ends. The overlay's own idle timeout remains as a fallback (idle_seconds, default 30, 0 = disable fallback)
• Status pill under the composer: colored dot + CU idle/move/type/click + a Stop button

---

Install

Option 1: GitHub Release (recommended)

dsh plugin --profile web add https://github.com/zzy6-a/vision-use/releases/download/v0.2.0/dsh-vision-0.2.0.tgz

Option 2: git source

dsh plugin --profile web add github:zzy6-a/vision-use

After installing, restart DSH (the client.js half registers at boot). Refresh the browser and the status pill appears below the composer.

---

Quick start

Just tell the agent:

Take a look at my screen

Search for the DeepSeek website in Bing and open it

Open Notepad and type something

The agent will automatically: start the overlay → view_screen to look → computer_* to act → look again to verify. You can press Esc to stop it at any moment.

---

Requirements

Item	Requirement
Host (auto-detected)	Windows native or Windows + WSL2; the plugin detects where DSH runs and picks the matching path/subprocess strategy
Interop when on WSL	WSL interop enabled (/proc/sys/fs/binfmt_misc/WSLInterop = enabled); the plugin uses \\wsl.localhost\<distro> UNC paths automatically
Windows side	PowerShell 5.1 (built in) + .NET Framework (System.Drawing/WinForms)
DSH	>= 0.1.5-rc.1
Node	The plugin itself has zero runtime dependencies (pure ESM + PowerShell child processes)
Linux / macOS	Desktop control is unsupported; view_screen and hands tools fail with a clear error, while view_image still works

---

Architecture

vision-use/
├── lib/index.js        host half: 8 tools + 2 API routes + systemPrompt capability declaration
├── lib/client.js       browser half: composer-dock status pill
├── cordis.patch.yml    bundle layer: inserts the plugin row into the profile tree
├── dsh.plugin.json     plugin manifest (for dsh-market / the plugin manager UI)
└── scripts/            Windows-side toolkit (self-contained, shipped with the package)
    ├── cu.ps1          mouse/keyboard executor + Esc sentinel
    ├── overlay-v2.ps1  overlay: border / badge / ring / ripple + C# 60fps animation engine
    ├── wocr.ps1        Windows native OCR (zero-API-cost fallback, optional)
    └── TOOLKIT.md      toolkit documentation + 10 field-tested rules

Environment auto-detection

At startup the plugin inspects process.platform, WSL_DISTRO_NAME / WSL_INTEROP, and /proc/version:

Detected	Behavior
Windows native	calls powershell.exe directly; flags/state/heartbeat go to %TEMP%
WSL + Windows	calls powershell.exe through interop; flags/state/heartbeat go to /tmp and are exposed as \\wsl.localhost\<distro>\tmp for PowerShell
Linux / macOS	desktop tools return a clear unsupported error; view_image keeps working

PowerShell children learn the cross-boundary flag directory through the DSH_VISION_FLAG_DIR env var (automatically added to WSLENV on WSL).

Vision: images are returned as content: [{type:'image', attachment}]; the host's collectImageRefs() recursively collects them and sends them with the next request — this is DSH's official image channel, so screenshots count as normal model vision tokens (on the DeepSeek official route, roughly 369 tokens per image, capped at 384).

Animation: the overlay uses UpdateLayeredWindow for true per-pixel alpha glow; the hot path runs entirely in compiled C# (PowerShell only keeps the beat), at about 8.8% of a single core.

---

Configuration

Parameter	Default	Description
computer_overlay start idle_seconds	30	Overlay idle-timeout fallback; during an active task the plugin keeps refreshing the heartbeat, so it will not vanish while the agent is thinking (0 = disable fallback)
-MaxSeconds	0	Maximum overlay lifetime (0 = unlimited)
-NoCursorChange	off	Do not install the blue cursor set (keep the system default)

Privacy note: screenshots enter the current session context as image attachments (equivalent to pasting an image manually). If you do not want them to reach the model, use scripts/wocr.ps1 from this repo for local OCR (zero API cost, text only).

---

Billing & privacy

• The plugin itself makes zero network requests: it calls no API; it only stores screenshots in the local attachment store and hands the image blocks to the host
• Images are ultimately sent to the model route selected for the current session (DSH's model adapter performs the request)
• It fully follows the model you choose: the plugin does not choose or change routes; screenshots always go to the model selected in your model picker (pick opencode-go and it goes to opencode-go; pick the official route and it goes to the official route)
• Vision-capability check: if the selected model does not declare image-input support, view_screen / view_image fail with a clear error instead of letting the request die halfway
• On the official route, vision billing is about 369 tokens per image (DSH caps at 384); self-hosted / subscription routes follow their own terms
• If you do not want screenshots to reach the model, use scripts/wocr.ps1 (Windows local OCR, zero API cost, text only)

Known limitations

• Chromium ignores KEYEVENTF_UNICODE injection: type into Edge/Chrome/WeChat with typevk (real VK keystrokes, works across apps)
• Chinese / non-ASCII can only go through the clipboard on this host: typevk drops CJK (VK has no CJK keycodes), SendChar/KEYEVENTF_UNICODE is silently ignored by Windows 11 Notepad and Chromium, and UIA ValuePattern is not supported by every control. method: auto therefore routes CJK (and >120-char ASCII) through the clipboard, saving and restoring your original clipboard content
• clipboard mode is explicit: method: 'clipboard' forces it; it also restores the previous clipboard content
• Pinyin + candidate selection (computer_key pinyin → space/digit) still works but is unreliable — candidate order depends on the IME dictionary and context
• WinUI (Notepad, etc.) does not accept injected Ctrl combinations: computer_key may not work in those apps
• UIA coordinates are unreliable in custom-drawn UIs (Edge tab bar, Windows 11 Notepad tabs return ∞ or wrong offsets): use view_screen to locate targets visually first
• Software with its own drawn cursor (games, some Electron apps) cannot be overridden by the system-level cursor replacement
• Linux / macOS hosts: only Windows native and WSL + Windows are supported; view_image still works, desktop tools fail clearly

---

What's new in 0.3.0

Area	Change
UI Automation	uia_windows / uia_tree / uia_find / uia_locate / uia_click — locate controls by name instead of guessing pixels; uia_click uses invoke/toggle/select/expand patterns, so it does not move your mouse
Batch actions	computer_sequence runs click / type / key / wait / shot in one PowerShell process and one model round-trip — typically saves 30-60 s per task
Capture	view_screen gained hwnd (PrintWindow — captures a covered window), region, grid (coordinate rulers) and marker_x/marker_y (crosshair to verify a target)
Input	computer_type gained method: auto — ASCII stays on real keystrokes; CJK and long text fall back to the clipboard, which is saved and restored
Safety	Desktop actions are serialized (parallel tool calls can no longer fight over the cursor); Esc only blocks mutating actions, so computer_status still answers afterwards; injection into the DSH host window is refused unless -Force
Fixes	Temp-dir leak (%TEMP%\dsh-vision-* accumulated one dir per screenshot), typevk taking 53 s per 1000 chars, seq unable to type a literal |

Related projects

• dsh-upgrade-guard — DSH upgrade compatibility guard (companion plugin by the same author)

Contributors

• zzy6-a — author
• DeepSeek V4.1 — architecture, implementation, testing, and release workflow

License

MIT