AI UI automation via Midscene — android_ui and web_ui tools that tap, type, query, and assert on a real Android device or an already-running Chrome through the ctx.midscene seam.
Install
# from npm (prebuilt)
dsh plugin --profile web add dsh-plugin-midscene
# from GitHub (first run asks for allowBuilds approval — follow the hint, retry)
dsh plugin --profile web add github:ciky20171114/dsh-plugin-midscene
Any plugin you install runs third-party code with your own permissions — it can read your files, use your credentials, and reach the network, and tool approvals don’t sandbox it. GitHub-sourced plugins also run build scripts at install time — pnpm blocks those until you allow them, so an install can stop with ERR_PNPM_GIT_DEP_PREPARE_NOT_ALLOWED or ERR_PNPM_IGNORED_BUILDS; dsh prints the exact key to add under allowBuilds in your profile’s pnpm-workspace.yaml, and the install works on the next run. Allowing a build is a trust decision: only install sources you trust, and pin a commit (github:owner/repo#sha).
README
English | 中文
AI-driven UI automation for DeepSeek Harness (DSH), powered by Midscene. The model sees the screen, locates elements by natural-language description, and acts on real targets — a real Android device or a real Chrome browser.
One capability seam (ctx.midscene), two providers, two tools:
| Provider entry | Tool | Target | |
|---|---|---|---|
| Android | dsh-plugin-midscene/android |
android_ui |
one ADB-connected device |
| Web | dsh-plugin-midscene/web |
web_ui |
the active page of an already-running Chrome |
Each tool is a single tool with an action parameter (like str_replace_editor): the model picks an action (tap / act / input / query / assert / boolean / back) and the tool dispatches internally — no tool-surface bloat.
Requirements
- DSH (
dshCLI) with a profile - Android:
adb devicesshows the device - Web: Chrome started with
--remote-debugging-port=9222 --user-data-dir=<dir>; the provider connects, never launches - A Midscene-compatible vision model, configured through environment variables (see Model configuration)
Install
dsh plugin --profile mysetup add dsh-plugin-midscene
The bundle's default layer registers the two tools. They read ctx.midscene opportunistically, so they appear even before a provider is configured — calling one without a provider fails with a message naming the missing row.
Then add exactly one provider row to your profile's cordis.patch.yml (~/.dsh/profiles/mysetup/cordis.patch.yml; both providers cannot own ctx.midscene in the same context):
Android
- insert:
- id: midscene-android
name: dsh-plugin-midscene/android
config:
deviceId: '' # empty: first device from getConnectedDevices()
aiActionContext: '' # free-form context for aiAct planning, e.g. app conventions
Web
- insert:
- id: midscene-web
name: dsh-plugin-midscene/web
config:
browserWSEndpoint: 'ws://127.0.0.1:9222/devtools/browser/<id>'
aiActionContext: ''
Ready-to-paste provider rows live in examples/.
Get the endpoint from http://127.0.0.1:9222/json/version → webSocketDebuggerUrl. Note the id changes every time that Chrome restarts — update the row and restart dsh.
On teardown the provider destroys its agent and then disconnect()s — never close()s — the browser: the Chrome process belongs to your deployment and keeps running.
Start:
dsh --profile mysetup # add --port 3081 if 3080 is occupied
Install troubleshooting
dsh plugin add fails with ERR_PNPM_IGNORED_BUILDS naming sharp / @ffmpeg-installer/linux-x64: pnpm ≥ 10 blocks those transitive install scripts (pulled in by @midscene/*) until they are declared. Fix: open ~/.dsh/profiles/<name>/pnpm-workspace.yaml, set the keys pnpm printed under allowBuilds to false (the plugin works without them — flip to true only if you want the sharp/ffmpeg binaries for real-device capture), then re-run the add command. This is a one-time fix per profile.
Tool reference
android_ui and web_ui share one shape:
action |
Other parameters | Result |
|---|---|---|
tap |
prompt (element description) |
ack |
act |
prompt (goal description) |
ack + the agent's own result text, if any |
input |
prompt (element) + value (text to type) |
ack |
query |
demand (what to extract) |
extracted JSON |
assert |
prompt (assertion) + optional msg |
pass/fail + optional thought |
boolean |
prompt (yes/no question) |
true/false |
back |
— | ack (Android: system back; Web: history back) |
Cross-field rules the schema cannot express (e.g. value required for input, demand for query) are enforced in execute with named error messages. A failed assertion is a successful pass: false result — the error path is reserved for infrastructure failures (device gone, websocket refused).
Model configuration
Midscene's vision model is configured through @midscene/*'s own conventions — environment variables, not DSH's ctx.llm:
export MIDSCENE_MODEL_NAME=glm-4.6v
export MIDSCENE_MODEL_BASE_URL=https://open.bigmodel.cn/api/paas/v4/
export MIDSCENE_MODEL_API_KEY=<your key>
export MIDSCENE_MODEL_FAMILY=glm-v
(Any OpenAI-compatible multimodal endpoint works — set the matching variables.)
Design boundary: no policy, no recovery
The providers are deliberately thin transports: no retry, no precondition checks, no automatic recovery from unexpected UI state (stray popups, unwanted navigation, re-login). Callers that need that behavior build it on top — for example a constraint/harness layer that checks app state before each write action.
Known limitations
- One target per provider instance — one device or one browser per context; fan out with isolated compositions.
- No reconnect — a mid-session disconnect surfaces as a rejected call.
- Pinned SDK versions —
@midscene/android/@midscene/webat exactly1.11.0; upgrading is a deliberate version bump. puppeteeris a peer (web) — resolved by your deployment's pnpm; Chrome itself is supplied by the deployment, never downloaded by this plugin.
Development
git clone https://github.com/ciky20171114/dsh-plugin-midscene
cd dsh-plugin-midscene
pnpm install # native/browser install scripts are denied by default; tests mock the SDKs
pnpm test # wiring-only: mocked @midscene/*, puppeteer, stub seam behind the real tool registry
pnpm build # tsc emit to lib/ (also runs as `prepare` on git installs)
Layout:
src/service.ts MidsceneService definition — the ctx.midscene seam (7 operations)
src/android.ts Android provider (AndroidDevice + AndroidAgent, ADB)
src/web.ts Web provider (puppeteer.connect + PuppeteerBrowserAgent, connect-only)
src/tool.ts android_ui + web_ui tools (one shared definition, action branching)
tests/ 31 wiring tests — never a real device or browser
Install a local checkout into a profile while developing:
dsh plugin --profile dev add /path/to/dsh-plugin-midscene
Community and support
Feel free to submit feedback or bug reports through GitHub Discussions. This repository carries the dsh-plugin topic for discoverability.
License
MIT
Links
More in this category
Tencent/WeKnora#dsh-weknora★ 37827
Four read-only tools over a WeKnora knowledge base: list knowledge bases, hybrid passage search, reassemble one document's chunks in order, and WeKnora's own cited RAG or ReAct-agent answer with a resumable session id.
DayuanJiang/next-ai-draw-io#dsh-plugin★ 37082
Draws draw.io diagrams in DeepSeek Harness through the next-ai-drawio MCP server: the link in the reply opens a live preview in the right sidebar, you can edit the diagram by hand, and each AI change is a version you can restore. Node.js 18+; npx downloads the MCP server on first use.
superdesigndev/treg★ 8541
Tool catalog for agents: search ~2,600 external endpoints (SEO and SERP, backlinks, social, people and company enrichment, ad libraries, scraping) by the task you want done, read each one's parameters and per-call price, then call it with the credential injected server-side. Ships the skill plus an MCP row that stays disabled until TREG_TOKEN is set.
T-Auto/dsh-ops★ 5947
Provides bash, PowerShell 7, and high-performance Rust tools for DSH on Windows to reduce token usage.
sixtysevenlf/dsh-blender-plugin★ 2207
Lets a DSH agent drive a local Blender over a direct TCP channel with 15 tools: viewport frames, custom-angle renders, Python execution, headless and job offload, transaction rollback, and a planning layer of 228 ops across 34 families.
helloHub1212/dsh-windows-mcp★ 1894
Bridges the CursorTouch/Windows-MCP desktop automation server into DSH through the official MCP client bridge, registering its screenshot, UI tree snapshot, mouse and keyboard input, clipboard, process and registry tools as mcp__windows__* tools for the model, and adds a windows_mcp_doctor diagnostic tool. Computer Use is off by default and is switched on from the plugin panel.
Community comments
Comments are public GitHub Discussions. Loading them connects to GitHub and Giscus; a GitHub account is required to post.