lhbsaa/dsh-visibridge

Host-level vision plugin for DeepSeek Harness (dsh) — adds image recognition to text-only models (DeepSeek, GLM, etc.).

The analyze_image tool sends an image to a configured vision model (local Ollama / Xiaomi MiMo cloud / any OpenAI-compatible endpoint) and returns structured JSON evidence (full OCR text, layout, scene, entities, uncertainty), so a text-only model answers from real evidence instead of guessing.

• Host-level bundle plugin: auto-loaded on every dsh restart, available in any session and any preset
• No proxy process, no hooks, no changes to harness core config
• Images stay local by default (zero upload with the Ollama backend)

---

Features

Capability	Description
Image input	Local file path or http(s) URL
Camera capture	capture_image tool: USB camera photo + instant recognition (structured evidence + saved image); the model can call it autonomously in debug loops to observe changes ("change → shoot → look → change")
Structured evidence	summary, ocr (full_text + lines), layout (regions), semantics (scene + entities), visual (colors + style), uncertainty
Backend presets	One-line switch: "ollama" (local) / "xiaomi" (Xiaomi MiMo cloud)
More backends	"deepseek" (DeepSeek official vision model) preset added; any OpenAI-compatible endpoint via "custom"
Any OpenAI-compatible endpoint	"custom": own baseUrl/model/apiKey (qwen-vl, GLM, SiliconFlow, OpenRouter, vLLM, …)
Xiaomi MiMo adaptation	Auto-detects xiaomimimo.com: api-key auth header + max_completion_tokens field
Ollama structured output	Local backend sends response_format: json_schema to force valid JSON (critical for small models); auto-degrades when unsupported
Model keep-alive	Local Ollama gets keep_alive: 30m to avoid repeated cold loads
Key safety	API keys are redacted ([REDACTED]) in any error message
Transport	Native fetch + node:fs (full Node runtime; no extra process overhead)

---

Architecture

DeepSeek V4 (text-only, cannot see images)
   │ calls analyze_image tool
   ▼
dsh-visibridge (host plugin, loaded at dsh startup)
   ├─ reads dsh-vision-config.json (backend/model/key ref)
   ├─ local image → base64 data URL; URL passed through
   ├─ POST /v1/chat/completions → vision model
   │    ├─ Ollama: json_schema enforced output + keep_alive
   │    └─ Xiaomi MiMo: api-key header + max_completion_tokens
   └─ tolerant JSON extraction + normalization → structured evidence → back to V4

Mounted as a profile bundle (same mechanism as ModLens / dshmarket):

dsh starts → reads dsh.profile.bundles → loads dsh-visibridge's dsh.bundle.patch
          → plugin row inserted into host composition → analyze_image registered globally

---

Requirements

• DeepSeek Harness (web profile)
• Node.js ≥ 22 (native fetch, AbortSignal.timeout)
• A vision backend (choose one):
  • Local Ollama with a vision model (recommended: minicpm-v4.5)
  • DeepSeek official vision model (deepseek-v4-flash-vision-exp, DEEPSEEK_API_KEY)
• USB camera (optional, for capture_image): UVC plug-and-play, autofocus, min focus distance ≤ 10 cm (see Camera capture section)
  • Xiaomi MiMo API key (Xiaomi Open Platform)

---

Installation

Windows examples below (dsh profile web, Harness home ~/.dsh).

1. Place the plugin package

If the plugin is published to npm, skip the manual copy and use the official command below ("Install from npm").

Copy the whole dsh-visibridge directory into the profile's node_modules:

$src = 'path\to\dsh-visibridge'

# Target 1: profile node_modules
Copy-Item $src '%USERPROFILE%\.dsh\profiles\web\node_modules\dsh-visibridge' -Recurse -Force

# Target 2: global npm node_modules (so dsh's loader resolves from either context)
Copy-Item $src '%APPDATA%\npm\node_modules\dsh-visibridge' -Recurse -Force

⚠️ Both locations matter: dsh's plugin loader resolves bare package names from the installation directory, while resolveBundleDir resolves from the profile. Two copies are the safest.

Install from npm (recommended if published)

dsh plugin --profile web add dsh-visibridge

dsh plugin initializes the profile (with @deepseek-ai/dsh-base as the first layer), installs the package via pnpm, and — because the package declares dsh.bundle — automatically appends dsh-visibridge to the profile's bundles list (no manual manifest editing). Verify and start as in steps 3-4 below.

Local-directory install (unpublished / development): dsh plugin --profile web add path\to\dsh-visibridge works too. Note that with a link:, the plugin modules are resolved from the source directory, so the runtime deps @deepseek-ai/dsh-tools / @deepseek-ai/schemastery must resolve inside the repo — run pnpm install in the repo first (this project declares them via peerDependencies + devDependencies). Once published to npm, pnpm handles this automatically.

2. Register the bundle

Edit the profile manifest ~\.dsh\profiles\web\package.json:

{
  "name": "dsh-profile-web",
  "private": true,
  "dependencies": {
    "dshmarket": "^1.5.0",
    "dsh-visibridge": "link:./node_modules/dsh-visibridge"
  },
  "dsh": {
    "profile": {
      "bundles": [
        "@deepseek-ai/dsh-base",
        "@deepseek-ai/dsh-web-app",
        "dshmarket",
        "dsh-visibridge"
      ]
    }
  }
}

(Append "dsh-visibridge" to the bundles array.)

3. Verify the composed config (optional)

npx -y @deepseek-ai/dsh --profile web --dump-config | Select-String 'dsh-visibridge'

You should see - id: dsh-visibridge / name: dsh-visibridge.

4. Restart dsh

Fully restart (close → confirm node processes are gone → start again). After restart analyze_image is registered automatically — no manual activation needed.

5. Configure the vision backend

Config priority (low → high):

built-in defaults → cordis.yml plugin row config (Config schema) → workspace dsh-vision-config.json

cordis.yml values act as deployment-level defaults; the workspace dsh-vision-config.json can override at runtime (edit-and-go, no restart, and the agent may edit it too).

Optional: configure in the profile's cordis.patch.yml (visible via --dump-config):

- id: dsh-visibridge
  name: dsh-visibridge
  config:
    backend: ollama
    model: minicpm-v4.5

Or create dsh-vision-config.json in your workspace root:

{
  "backend": "ollama",
  "model": "minicpm-v4.5",
  "baseUrl": "http://localhost:11434/v1",
  "apiKeyRef": "VISION_API_KEY",
  "timeoutMs": 300000,
  "maxTokens": 8192,
  "maxBytes": 8388608
}

backend decides the defaults:

backend	baseUrl	model	apiKeyRef
ollama	http://localhost:11434/v1	auto (pick a vision model; explicit value overrides)	VISION_API_KEY
xiaomi	https://api.xiaomimimo.com/v1	mimo-v2.5	XIAOMI_MIMO_API_KEY
deepseek	https://api.deepseek.com/v1	deepseek-v4-flash-vision-exp	DEEPSEEK_API_KEY
custom / omitted	explicit fields only	explicit	explicit

An explicit model overrides the preset default (e.g. "minicpm-v4.5" under ollama).

6. Configure cloud API keys (Xiaomi etc.)

Credentials file ~\.dsh\.credentials.yaml (YAML, 0600):

XIAOMI_MIMO_API_KEY: sk-xxxxxxxx

Or set an environment variable of the same name (credentials service prefers env). The file is watched — hot reload, no restart needed.

---

Usage

1. Put the image in the session workspace (or provide an http(s) URL)
2. Say it: 看下这张图 / 分析 xxx.png / 识别 https://example.com/pic.png
3. DeepSeek V4 calls analyze_image → answers from structured evidence

Switch backend (no restart; edit the config file):

{ "backend": "xiaomi" }   // Xiaomi MiMo cloud (image leaves the machine — mind privacy)
{ "backend": "ollama" }   // local Ollama (image never leaves the machine)

Or just tell the AI "switch to Ollama / Xiaomi" and let it edit the config.

---

Camera capture (capture_image)

Connect a USB autofocus camera and the model can take photos and recognize them automatically — for observing screens, boards, or documents during debugging.

Usage (just tell the AI; it calls the tool itself):

"Take a photo of the phone screen / board / this document"
"Check what error the screen shows now"

Tool parameters (optional):

Parameter	Description
question	Focus direction (e.g. "what error is on screen")
camera	Camera device index (default 0)
flip	Software flip none/h/v/b — for upside-down/mirrored mounting

Returns: structured evidence + capture metadata (saved image path .captures/cap-<ts>.jpg, device, flip, time). Timestamped shots let the model compare changes across calls.

Hardware requirements:

Item	Requirement
Interface	USB (UVC plug-and-play)
Autofocus	AF supported (UVC CAP_PROP_AUTOFOCUS, program-controlled — no touch focus needed)
Min focus distance	≤ 10 cm (key for phone screens / small displays)
Resolution	1080p is enough (script defaults to 1920×1080)

Notes:

• Phone/screen shots may show slight moiré (screen pixel matrix + PWM dimming); key info is still readable. Books/documents are perfectly clear.
• Without a camera, capture_image errors clearly and suggests analyze_image with a file path.
• Captures are saved to workspace .captures/ (no auto-cleanup by default).

Configuration reference

Field	Default	Description
backend	—	"ollama" / "xiaomi" / "custom" (omitted = explicit fields)
baseUrl	http://localhost:11434/v1	OpenAI-compatible endpoint
model	auto	Explicit model name (auto works on Ollama only)
apiKeyRef	VISION_API_KEY	Credential reference (~/.dsh/.credentials.yaml or env var)
timeoutMs	300000	Request timeout (ms)
maxTokens	8192	Vision model output cap
maxBytes	8388608	Image byte cap (8 MB)
authStyle	auto	"bearer" / "api-key" (auto-detected per endpoint)
keepAlive	30m local	Ollama keep-alive; false disables (JSON file only)
structuredOutput	on for local	false disables json_schema enforcement (JSON file only)
allowPrivateHosts	false	Allow baseUrl to point at private/reserved addresses (blocked by default — see Security)
configFile	dsh-vision-config.json	Name of the workspace override file (set in cordis.yml)

backend can also be set in cordis.yml (deployment default); a workspace JSON backend takes precedence.

---

Recommended local models (Ollama)

Model	Size	Role
minicpm-v4.5	8B (~6 GB)	Recommended: strongest local OCR / document recognition (ollama pull minicpm-v4.5)
qwen3-vl:4b	4B	Lightweight all-rounder
minicpm-v4.6	1.3B	Ultra-light, low-end machines
glm-ocr	1.1B	OCR-focused

---

Security notes

• baseUrl is validated by default: localhost (local Ollama), known vendors (xiaomimimo.com) and public hosts are allowed; private/reserved addresses are rejected (RFC1918, link-local, CGNAT, multicast, .local, …) to prevent data exfiltration and SSRF. To use an internal endpoint explicitly set "allowPrivateHosts": true.
• Workspace config file: dsh-vision-config.json is read from the session workspace — be careful opening untrusted repositories that ship their own copy: it can redirect the vision backend (image content is sent there). Only place this file in directories you trust.
• Cloud backends send images out: with xiaomi or a custom cloud endpoint, local images are uploaded as base64 to that endpoint. The local Ollama backend uploads nothing by default.
• Remote image URLs: http(s) image links are handed to the vision backend as-is; with local Ollama the fetch happens on your machine — don't use this tool on URLs from untrusted sources.
• Secrets: API keys are read via ctx.credentials or environment variables and are redacted ([REDACTED]) in error messages; they are never written to logs or session records.

---

Troubleshooting

Symptom	Fix
No analyze_image tool after restart	Confirm both node_modules copies exist, bundles contains dsh-visibridge, and dsh was fully restarted (all node processes killed)
[dsh-visibridge] tools service not available	The plugin must use inject: ['tools', ...] (already built in — don't remove)
Result says "degraded extraction"	Vision model didn't output JSON; Ollama uses json_schema to force it — if it persists, check model support
Xiaomi API errors	Confirm XIAOMI_MIMO_API_KEY is configured (credentials.yaml or env var)
Image over limit	Raise maxBytes or compress the image first
Error "baseUrl points at private/reserved address"	Default policy rejects internal endpoints; set "allowPrivateHosts": true if you really need it

---

Development & tests

Pure helpers (base64, tolerant JSON extraction, evidence normalization, endpoint validation, config merging) live in lib/pure.js with zero runtime dependencies, so they are directly unit-testable:

npm test

Requirements: Node ≥ 22.19; after editing lib/index.js run node --check lib/index.js first.

---

Uninstall / rollback

1. ~\.dsh\profiles\web\package.json: remove "dsh-visibridge" from bundles
2. Delete both node_modules\dsh-visibridge directories
3. Restart dsh

---

Compatibility

• Transport uses native fetch (Node ≥ 22)
• Runtime dependency @deepseek-ai/dsh-tools (provided by the dsh install, auto-resolved)
• No dependency on agent-scoped services (fs/subprocess/sandboxPolicy); runs at host top level

License

MIT