DeepSeek Harness Plugin

qishuilalala/dsh-voice-mode#dsh-voice-mode

Stars ★ 35 Downloads (30d) 13,611 Category Voice & Audio Added 2026-08-24 npm dsh-voice-mode

Full-duplex voice mode for the DeepSeek Harness Web UI: toggle (2s-pause auto-send) or hold-to-talk dictation into an editable draft with zipformer2 streaming ASR, optional wake word; sentence-by-sentence Edge TTS read-aloud with live captions, and speaking interrupts playback and the running turn (true barge-in). On-device ASR, no API key.

Install

# from npm (prebuilt)

dsh plugin --profile web add dsh-voice-mode

# from GitHub (first run asks for allowBuilds approval — follow the hint, retry)

dsh plugin --profile web add github:qishuilalala/dsh-voice-mode#path:/plugin/dsh-voice-mode

Any plugin you install runs third-party code with your own permissions — it can read your files, use your credentials, and reach the network, and tool approvals don’t sandbox it. GitHub-sourced plugins also run build scripts at install time — pnpm blocks those until you allow them, so an install can stop with ERR_PNPM_GIT_DEP_PREPARE_NOT_ALLOWED or ERR_PNPM_IGNORED_BUILDS; dsh prints the exact key to add under allowBuilds in your profile’s pnpm-workspace.yaml, and the install works on the next run. Allowing a build is a trust decision: only install sources you trust, and pin a commit (github:owner/repo#sha).

README

CI

Full-duplex voice conversation mode for DeepSeek Harness (dsh): speak, get a spoken answer. Streamed zipformer2 ASR → editable draft → auto send → the final reply is read out sentence-by-sentence via Edge TTS, and your voice interrupts playback and the running turn. No additional speech-service API key is needed for the default path; the host LLM has its own configuration and costs.

English · 中文

dsh-voice-mode full-duplex voice conversation

Real recording: streaming transcription → auto-send → sentence-by-sentence read-aloud with live captions

Current status: Supports 7 built-in online TTS presets with per-preset API keys, Xiaomi MiMo TTS, editable post-ASR replacement rules, a 4-tab settings panel, localized keyboard/touch guidance (including macOS keycap badges and 220ms keyboard push-to-talk), and cross-platform worker isolation. Check npm and Releases for published versions.

Compatibility & verification boundaries: Client injection anchors, host/client typechecks, and isolated runtime checks pass through dsh 0.2.2-rc.5. See the compatibility contract for the full host/OS/browser matrix and unverified layers, the changelog, and the roadmap. CI does not replace real-device microphone or acoustic acceptance; the declared dsh range is >=0.1.1-rc.2 <0.3.0-0, and untested host versions are not guaranteed.


🤝 Fork enhancements (this repo)

  • Edge cloud TTS by default; local TTS optional (privacy-first): local VITS (Chinese) and local Kokoro (zh+en, 103 voices; int8 default ~109 MB / fp32 ~311 MB with a larger model; native sherpa-onnx-node addon, no WASM memory limits) run in an isolated child process;
  • 103 Kokoro voices (F0-measured gender labels, 4 favourite male voices pinned), browsed with a ◀▶ stepper;
  • Delta transport: partials upload only the new 0.9 s — long push-to-talk segments finalize in seconds;
  • Interaction: a mode-switch button next to the mic (continuous ⇄ hold, persisted); hold mode records only while held;
  • Long segments: continuous listening stitches consecutive segments into one message (internally chunked at 30 s and concatenated across chunks); 1500 ms silence split by default; hold keeps pauses from splitting;
  • Hardening: session-existence check, loopback + Origin guards, endpoint-specific rate limits, pinned digests for primary/listed model files (auxiliary directories use manifest/presence checks), download-host allowlist.

ℹ️ The demo-voice-flow.gif at the top is a recorded example of the UI (voice mode → live transcription → pause auto-send → sentence-by-sentence read-aloud with live captions), produced by driving the real pipeline via screenshots/scripts/capture-demo.mjs; the UI keeps evolving, so the actual interface prevails.


✨ Features

  • Voice mode: toggle with the microphone button in the input toolbar or the global shortcut Ctrl+Shift+V (⌃ Control + ⇧ Shift + V on macOS — note Control, not ⌘ Command); with the caret in the composer, Alt+Enter (⌥ Option + Return on macOS) toggles it too, without inserting a newline; globally single-active (only one session is in voice mode at a time; switching sessions yields automatically)
  • Three interaction modes (switchable in settings, right-click menu, or touch ⋯ menu):
    • toggle (default) continuous listening: RMS VAD segmentation → streaming zipformer2 ASR (words appear as you speak, live caption preview) → automatic sentence split after 1500 ms of silence into the draft, consecutive segments joined into one message, then auto-sent after ~1.2 s more of silence (≈2.7 s total); hold Ctrl (⌃ Control on macOS) while speaking to force an immediate send
    • hold push-to-talk: from idle, just hold the mic button to start recording and release to send (no need to tap in first; a short tap still enters/exits); while voice mode is active, you can also hold Ctrl (⌃ Control on macOS, ≥220 ms) to record by keyboard and release to send; after sending, voice mode exits on its own once the reply finishes (read aloud, if on); with auto-send off, releasing only leaves the text in the composer (not sent) and exits; if you hold without speaking it exits after ~25 s; swipe up, Esc, window blur, or pressing another key combo cancels the segment; pauses do not split while held
    • once one-shot: talk like toggle; after sending, the mic turns off and voice mode exits once the reply finishes naturally; holding the mic button or pressing Ctrl (⌃ Control on macOS) interrupts the reply, resumes the mic, and keeps voice mode on for another utterance (with auto-send off: exits after the utterance, text stays in the draft)
  • Read-aloud scope (readAloud): voice (default) only replies in voice mode; always every reply in the visible session, typed ones included (with the session open in several tabs, only the last-viewed tab plays; switching sessions immediately stops the previous session and discards its queued speech); off never
  • The mic button is the status panel (PR #16 by @dyguests): green bars follow your voice while listening, blue bars follow playback while reading, a spinner while once waits for the reply, a check mark before it exits; orange dot = speech detected, red dot = error (hover for details). Right-click (or keyboard menu key) on desktop, or tap the ⋯ button (or 450 ms idle long-press outside hold mode) on touch devices for quick settings. Turn off the status bar and captions to keep all voice UI on the button
  • Wake word (optional, off by default): after setting wakeWord, entering voice mode starts in standby, and recognition only begins once the wake word is spoken (e.g. 你好小D). You may say it together with your command ("wake word, check the weather") — the wake word is stripped from the transcript/caption and never sent; matching is fault-tolerant (homophone substitutions, a wrong first char, leading fillers); 3-4 characters recommended; in standby the status bar live-shows "say '' · what it heard". Boundaries (important): toggle mode only (inactive in hold mode; works normally when idle under manual or detect fallback); it must be repeated after each completed utterance or barge-in (if you only said the wake word, the engine keeps listening so you may pause before the command); saying it while the agent is reading does not trigger (barge-in is VAD-based and unrelated to the wake word)
  • Caption tiers: captionFontSize 4 levels (0=12px / 1=14px / 2=18px / 3=24px) + captionMaxWidth 3 levels (0=50vw / 1=70vw / 2=90vw); adapts to narrow viewports and safe areas, with recent-sentence history after playback
  • Yield semantics (ADR-0008): backchannelYield on by default — saying 嗯 / 对 / right while the AI is reading auto-pauses for 1.5 s; genuine speech still triggers hard barge-in; lets the LLM yield the turn
  • Output pipeline: only the final answer's text-delta is read (reasoning/tool calls are skipped), streamed sentence-by-sentence (Edge cloud by default; local VITS / Kokoro, int8/fp32, optional) with a live caption overlay at the bottom-right; tool calls trigger a beep; the full text is still written to the chat; in voice mode a spoken-format system prompt is injected (short natural sentences, no Markdown decoration), and the reader side strips markers as well for a smoother listening experience
  • Barge-in: three sensitivity levels of voice-onset detection → local mute + host synth queue invalidation (epoch) + running turn cancellation (the half-finished part is kept and naturally flows into your new message). With a wake word set, barge-in remains speak-to-interrupt (the gate is the VAD, not the wake word; the engine returns to standby afterwards)
  • Lazy model download with progress: the zipformer2 Chinese streaming model (~160 MB, .part resumable) is downloaded on first use with live progress in the status bar; npm run prefetch can pre-download it
  • Resilience and diagnostics: mic-denied red hint, visible model-download failure, TTS unreachable status hint (auto retry), failed submit keeps the text in the draft, SSE auto-reconnect; the status bar also hints when the recognition / barge-in detect channel fails several times in a row or the state sync drops (self-healing transient failures stay quiet); when SenseVoice finalization takes over 2 s the status bar counts up "recognizing… Ns"; a "copy diagnostics" button copies version / runtime state / settings snapshot / phase timings in one click — never any transcript text (privacy)
  • Settings: location depends on your dsh version (see Settings below), with voice / rate / interrupt sensitivity / silence pause / idle timeout / model mirror / auto send / interaction mode / wake word / ITN / caption tiers / yield semantics / yield window / barge-in salvage / LLM correction; voices are previewable (the "试听/Preview" button synthesizes and plays the current voice at the current rate (first use may download or load a model), no need to enter voice mode; custom ShortNames are previewable too)
  • Interface language: follows dsh's language setting (Settings → General → Language) and switches instantly, no reload — the settings page, voice status bar, captions, voice names/gender/accent and error messages all follow it, and the LLM spoken-style prompt is chosen in English or Chinese accordingly. Dictionaries are registered in dsh's official locale service under the voice-mode namespace, so community language packs can add more languages (missing ones fall back to English). The plugin's name and description in the Plugins page are localized too.
  • Idle exit: auto-exit and mic release after 5 minutes of inactivity (reading, speech and a running turn renew the timer, subject to a two-hour cap since the last user activity)

⌨️ Interaction gestures & shortcuts

Speaking to trigger a hard interruption stops playback and requests cancellation of the current running turn. Pressing Esc once (or clicking stop) stops read-aloud only; double-pressing Esc exits voice mode.

Action Windows / Linux macOS Behaviour
Enter / exit voice mode Ctrl+Shift+V ⌃ Control + ⇧ Shift + V App-wide toggle (note: Control on macOS, not ⌘ Command; customizable or clearable in Settings → Shortcuts). Clicking the mic button does not steal composer focus
Toggle from composer Alt+Enter ⌥ Option + Return Toggles voice mode while the caret is in the chat input without inserting a newline; leaves Ctrl/⌘ + Enter intact for sending
Continuous listening (toggle) Speak directly; hold Ctrl to send now Speak directly; hold ⌃ Control to send now Splits into draft after ~1500 ms silence and auto-sends after ~2.7 s total quiet; holding Ctrl / ⌃ Control ≥250 ms while speaking sends immediately on release
Push-to-talk (hold) — pointer/touch Hold the mic button Hold the mic button From idle, hold the mic button directly to record and release to send (auto-exits once the reply finishes); swipe up or press Esc to cancel; short tap enters/exits
Push-to-talk (hold) — keyboard Hold Ctrl (≥220 ms) Hold ⌃ Control (≥220 ms) While in voice mode, hold Ctrl / ⌃ Control to record and release to send; pressing Esc, blurring the window, or pressing another key combo discards the segment
Interrupt / stop reading Speak / hold Ctrl / Esc Speak / hold ⌃ Control / Esc Under auto, speaking interrupts immediately; under manual (or detect without native AEC), hold mic or Ctrl / ⌃ Control during playback to interrupt; Esc stops reading, double-Esc exits
Quick settings menu Right-click mic (or Menu key) Right-click / two-finger click mic On touch devices, tap ⋯ next to the mic (or 450 ms idle long-press outside hold mode)
Type in the input box Type in composer Type in composer Auto-exits voice mode while preserving the current draft

🚀 Quick Start (model download time varies)

Requirements: dsh web (the plugin declares Node ≥18; the host may require a newer version), a modern browser (Chrome / Edge / Firefox, supporting getUserMedia and Web Audio).

# Option 1: from npm (recommended)
dsh plugin --profile web add dsh-voice-mode
# Equivalent via npx if the dsh CLI is not installed locally:
npx -y @deepseek-ai/dsh plugin --profile web add dsh-voice-mode

# Option 2: local tarball
dsh plugin --profile web add ./dsh-voice-mode-<version>.tgz

# Option 3: from source
git clone https://github.com/qishuilalala/dsh-voice-mode.git
cd dsh-voice-mode/plugin/dsh-voice-mode && pnpm install && pnpm build
dsh plugin --profile web add .

Bundle plugins require a dsh restart to take effect (restart methods by platform):

  • Linux with a configured systemd service: restart the actual service (for example, systemctl restart dsh only if that is its name). The install command above targets the web profile; use your actual profile for other deployments.
  • Windows / macOS / manual hosting: restart your dsh process (kill and run dsh web again, or restart it in your service manager)

Optional: pre-download the ASR model to reduce the download wait on the first voice-mode entry:

npm run prefetch          # run inside the plugin dir; writes to the platform cache dir
# or specify the cache location: node scripts/prefetch.mjs --cache-dir /where/ever/models

First run:

  1. Click the mic button in the input toolbar (or press Ctrl+Shift+V) to enter voice mode; a status bar appears above the input box
  2. Choose how to speak: just talk and let the ~1500 ms pause split and ~2.7 s of quiet auto-send (toggle); or hold the mic button and release to send (hold)
  3. The AI answer is read sentence-by-sentence with a caption overlay at the bottom-right; click "Skip" or just start speaking to interrupt
  4. Click "Exit" in the status bar (or press Ctrl+Shift+V again) to leave voice mode

On first entry the recognition model is downloaded; the status bar shows Loading model… <file> <percent>%.

If a wake word is configured, you land in standby first (the status bar prompts Say "<wake word>" to start), and recognizing starts after you speak the wake word.


⚙️ Settings

Works on every dsh version: Settings → Voice Mode (the plugin registers its own page in the Settings dialog). Secondary entry points (same card) vary by version:

  • dsh ≤ 0.1.5: Settings → Plugins → plugin config → voice mode
  • dsh 0.1.6-alpha and later (incl. 0.1.7 / 0.2.0): left sidebar Plugins → the installed dsh-voice-mode → the "Voice Mode" card on its detail page
  • On dsh 0.1.7+ the values are stored in the plugin's own file ~/.dsh/voice-mode.settings.json (under $DSH_HOME) and take precedence over same-named keys in the profile config. The first run migrates the old voice-mode: block from settings.yaml(.imported) once. To reset, set the file's content to {} (do not delete it).

Full 35-key settings table

Key Default Description
ttsEngine edge Read-aloud engine: edge Microsoft cloud (default, fast) / vits local Chinese / kokoro local zh+en / openai any OpenAI-compatible server / mimo Xiaomi MiMo (independent mimoTts.apiKey, see below); applies live
kokoroModel int8 Kokoro precision: int8 (default, 109 MB, CPU/low-bandwidth) / fp32 (311 MB, better quality, GPU/large memory); same 103 voices; applies live
voice per engine Voice: 5 VITS speakers; 103 Kokoro voices (◀▶ stepper; 62/68/75/76 favourite males pinned); Edge ShortNames below. The inline "试听" button previews it at the current rate
rate 1.1 Reading speed multiplier (0.5 slow ~ 2.0 fast), applies live
interruptLevel 0 Barge-in sensitivity (host-side VAD frame detection + echo gate): 0 high threshold (3 frames) / 1 medium (2 frames) / 2 low (1 frame)
bargeInMode detect Barge-in mode: detect auto-probes native echo cancellation (default; switches playback to hold-to-interrupt when inactive) / auto force interrupt as soon as you speak (headphones / quiet rooms) / manual manual interrupt (recommended on loudspeakers: speak directly or say the wake word when idle, and hold mic/Ctrl to interrupt during playback; see ADR-0006)
echoGateDb 6 Fallback echo-gate threshold when native AEC is unavailable. Inactive with native AEC. Only on the fallback path, a higher value is more conservative; verify the device and barge-in mode before tuning
silenceMs 1500 Silence pause in ms that marks the end of a complete sentence
idleTimeoutMinutes 5 Minutes of inactivity before auto-exiting voice mode (reading, speech and a running turn renew the timer, subject to a two-hour cap since the last user activity)
modelHost default Model download host (use https://hf-mirror.com on mainland networks)
autoSend true Auto-send once quiet (consecutive segments join into one message); when off, text only goes to the draft (hold Ctrl / release in hold mode still sends)
mode toggle Interaction mode: toggle continuous listening + 1500 ms silence split; hold push-to-talk, release to send (short tap exits); once one utterance, then exit after the reply finishes naturally; manual interruption resumes the mic and keeps voice mode on
readAloud voice What gets read aloud: voice only replies in voice mode; always every reply in the visible session, typed ones included; switching sessions stops the previous read-aloud; off never
showStatusBar true Voice status bar above the composer; when off, status lives on the mic button (timer below it, notices beside it)
showCaptions true Floating caption while reading aloud
autoResume false Auto-restore voice mode when switching back to your last voice session (off by default). When on: the next time you enter that session it auto-enters voice mode and restores it; when off, press Ctrl+Shift+V manually. While it is off, switching back to your last voice session shows a short grey hint about this next to the input box, once per browser
shortcut Ctrl+Shift+V Shortcut to enter / exit voice mode (modifiers Ctrl/Shift/Alt/Meta + one letter / digit / Enter / Space / Tab). In the "Shortcuts" group of the settings page, click "Record" and press the combo (Esc cancels, Backspace clears; a missing modifier or an unsupported key such as arrows / F-keys is flagged); empty = disabled, use the mic button instead; applies immediately
composerShortcut Alt+Enter Shortcut that toggles voice mode while the caret is in the composer (Option+Enter on Mac), no newline inserted; same format and recording as shortcut; empty = disabled; the settings card's own inputs are not intercepted; Cmd/Ctrl+Enter is not used; applies immediately. "Restore defaults" resets both shortcuts to Ctrl+Shift+V and Alt+Enter
wakeWord empty (off) Wake word (e.g. 你好小D): speak it after entering to activate; empty = off. May be said together with your command — the wake word is stripped and never sent; fault-tolerant matching (edit distance ≤1 + a 3-char leading window absorbs homophones/fillers); 3-4 characters recommended (single-char words get no tolerance; a 2-char word also absorbs any same-first-char 2-char word); must be repeated after each utterance split or barge-in; inactive in hold-to-talk mode (works normally when idle under manual or detect fallback); not triggered while the agent is reading
spokenFormat true Spoken-format system prompt: inject "short natural sentences, no Markdown decoration" into voice-mode replies only, applies live
senseVoice true Re-transcribe the finalized utterance with SenseVoice (punctuation + number normalization, more accurate; on by default). Turning it off saves ~228 MB of models and uses streaming recognition only (faster, less accurate)
toolBeep false Tool-call beep (off by default): when on, a short beep plays each time the agent calls a tool; off = silent
senseITN true SenseVoice inverse text normalization (number / date / currency; on by default)
openaiPreset empty (manual) Online TTS preset id (siliconflow-cosyvoice2 / siliconflow-fish / openrouter-openai / openai-official / groq-playai / selfhosted-kokoro / selfhosted-vllm, or custom). Fills baseUrl/model and falls back to the preset voice when voice is empty; keys prefer settings card > openaiTts.apiKeys[preset] > openaiTts.apiKey; applies live
openaiBaseUrl empty (preset/config) Online TTS endpoint override configurable directly in the settings card (priority: settings > preset > openaiTts.baseUrl); pair with custom preset for any OpenAI-compatible server; applies live
openaiModel empty Online TTS model override (same priority as openaiBaseUrl); applies live
openaiApiKey empty (config fallback) Online TTS API key entered in the settings card; stored in the host-side 0600 settings file and never sent back to the browser (GET /voice-mode/settings returns empty string + keys.* booleans); priority: per-preset key > global key > openaiTts.apiKeys[preset] > openaiTts.apiKey; applies live
openaiApiKeys {} (none) Per-preset API keys: {presetId: key} map in the settings card; whichever preset is selected uses its own key without 401 mismatches when switching presets; empty entries fall back to the global key; booleans only, never plaintext; applies live
mimoApiKey empty (config fallback) Xiaomi MiMo API key (api-key header) entered in the settings card; host-only 0600 storage, never sent back to the browser; falls back to mimoTts.apiKey; applies live
asrReplacements [] Post-recognition correction table [{from:[misheard], to:wanted}] (≤200 rules / ≤12 KB UTF-8); applies to both partials and finals, editable in the settings card with hit counters; applies live
captionFontSize 0 Caption font tier 0=12px / 1=14px / 2=18px / 3=24px (default 0 is byte-equivalent to legacy)
captionMaxWidth 1 Caption width tier 0=50vw / 1=70vw / 2=90vw
backchannelYield true Yield semantics (ADR-0008); saying 嗯 / 对 while reading auto-pauses for 1.5 s; genuine speech still triggers hard barge-in; off = behavior identical to pre-change
bargeInSalvage true Barge-in salvage (ADR-0012): when you speak to interrupt while the agent is reading aloud, keep what you said from speech onset to break confirmation and merge it into recognition (prevents swallowed break-in speech; off = discard pre-break audio; recommended off only on speakers without echo cancellation)
llmCorrect false LLM correction (off by default): before sending, run one LLM pass over the whole transcribed text (typos/homophones/punctuation only, meaning preserved; reuses the host configured model). On = about one extra second per message; falls back to the original text if correction fails or the model answers you instead of correcting
yieldMs 1500 Yield-window length (ms, 500-3000): how long TTS frames are dropped after backchannelYield triggers; if you really want to speak in the window the normal hardBreak takes over, and playback resumes when the window ends

Effect timing, in four tiers: ① immediately — shortcut / composer shortcut / interaction mode / read-aloud scope / auto-send / captions / status bar / engine / voice / rate / spoken format (applies to later replies) / model mirror (next download attempt); ② next recognition — finalization re-transcription (SenseVoice) / ITN; ③ next utterance — silence threshold (endpoint; the send wait stays a fixed 1.2 s); ④ next time you enter voice mode — interrupt sensitivity / barge-in mode / echo gate / auto-resume / wake word / tool beep / caption font / caption width / backchannel yielding / yield window / barge-in salvage / LLM correction / idle timeout. Saved-but-not-yet-effective items are listed at the top of the settings card with their current effective values. Defaults come from the plugin config (base layer) — they follow the config unless explicitly changed.

Common voices (full list: node scripts/list-voices.mjs)

Desktop (Electron) note: the official desktop host runs inside Electron, whose V8 forbids the external buffers returned by Kokoro's native add-on. The plugin automatically runs Kokoro in a real Node.js instead — it needs Node.js >= 18 on PATH (or set DSHVM_NODE to its path). Without it Kokoro reports an error; use the default Edge engine or local VITS (both work on desktop).

ShortName Description
zh-CN-XiaoxiaoNeural Xiaoxiao · Female (default)
zh-CN-XiaoyiNeural Xiaoyi · Female
zh-CN-YunxiNeural Yunxi · Male
zh-CN-YunjianNeural Yunjian · Male
zh-CN-YunyangNeural Yunyang · Male
zh-CN-YunxiaNeural Yunxia · Male
zh-CN-liaoning-XiaobeiNeural Xiaobei · Northeastern Mandarin · Female
zh-CN-shaanxi-XiaoniNeural Xiaoni · Shaanxi Mandarin · Female
zh-HK-HiuMaanNeural HiuMaan · Cantonese · Female
zh-HK-WanLungNeural WanLung · Cantonese · Male
zh-TW-HsiaoYuNeural HsiaoYu · Taiwanese Mandarin · Female
zh-TW-YunJheNeural YunJhe · Taiwanese Mandarin · Male
en-US-AriaNeural Aria · English · Female
en-US-GuyNeural Guy · English · Male

🔧 Configuration (bundle patch / settings.yaml)

You can also edit the voice-mode: section of ~/.dsh/settings.yaml directly (the GUI card and RPC write to the same document layer):

- id: voice-mode
  name: dsh-voice-mode
  config:
    enabled: true                 # false = disables voice mode entirely (toggle rejects)
    cacheDir: ~/.cache/dsh-voice-mode/models   # overridable; platform default otherwise
    # Defaults seeded for the settings (the settings panel overrides; the panel is authoritative):
    voice: zh-CN-XiaoxiaoNeural
    rate: 1.1
    interruptLevel: 0
    silenceMs: 1500
    idleTimeoutMinutes: 5
    modelHost: https://huggingface.co

Note: the effective values of voice/rate/interruptLevel/silenceMs/idleTimeoutMinutes/modelHost/autoSend come from the settings panel; the bundle config only seeds defaults for those keys (enabled/cacheDir remain bundle-config-only). The plugin HTTP namespace is fixed to /voice-mode (matching the client bundle contract; not configurable).

OpenAI-compatible & Xiaomi MiMo TTS (ttsEngine: openai / mimo)

Any server that implements OpenAI's POST /v1/audio/speech can read replies aloud: a self-hosted one on a GPU machine (vLLM-Omni, Kokoro-FastAPI, LocalAI, ...) or a hosted API such as SiliconFlow, OpenAI, OpenRouter, or Groq PlayAI. You can pick one of the 7 built-in presets (or custom) and enter openaiBaseUrl / openaiModel / openaiApiKey directly in the Settings card (priority: settings card > preset > bundle config), or seed them in the bundle config below; keys remain host-only (0600) and are never returned to the browser. Xiaomi MiMo (ttsEngine: mimo) uses its own independent key (mimoApiKey in the card or mimoTts.apiKey in config) so keys never leak across providers:

- id: voice-mode
  name: dsh-voice-mode
  config:
    ttsEngine: openai
    voice: vivian                       # voice name on the server; empty = openaiTts.voice
    openaiTts:
      baseUrl: http://gpu-box:8091/v1   # OpenRouter: https://openrouter.ai/api/v1, OpenAI: https://api.openai.com/v1
      apiKey: ''                        # Bearer key, empty for servers without auth
      model: Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice
      voice: vivian
      responseFormat: mp3               # mp3 | wav
      timeoutMs: 8000                   # per sentence; slower counts as unavailable
      fallback: kokoro                  # none | vits | kokoro: local engine while the server is down or slow
      fallbackVoice: zf_xiaobei         # empty = that engine's default voice
      instructions: ''                  # optional style text for models that accept it

Each sentence is one request (the queue already synthesizes sentence by sentence while the reply is still streaming, so in-sentence streaming would not start playback any sooner). rate is sent as speed when it is not 1. With a fallback, a failed or timed-out request makes that sentence (and the next 30 s) use the local engine, then the server is tried again, so a dead server does not cost a timeout per sentence. The text to be spoken goes to baseUrl, so pick a server you trust with your replies.

Fixing misheard words (asrReplacements)

Recognizers keep mishearing some names and English words inside Chinese speech (a name such as "Sakina" can come out as "ski男"). The usual fix, hotwords, doesn't fit here: sherpa-onnx supports hotwords only for transducer models, the streaming zipformer works on Chinese characters and can't spell English words, and the SenseVoice model that produces the final text has no hotword input. Until a model with hotwords covers this, a replacement list is a stopgap: it rewrites the text after recognition, in partial and final results alike, so captions, the draft and the wake word all see the corrected text.

- id: voice-mode
  config:
    asrReplacements:
      - from: [ski男]          # what the recognizer produced, one or more spellings
        to: Sakina             # what you want

Matching is case-insensitive and ignores whitespace between Latin and CJK text ("ski 男" matches too). A spelling that starts or ends with a Latin letter or digit only matches a whole word, so sina leaves Sinatra alone; CJK text has no word boundaries, so a Chinese spelling matches as a substring. The longest spelling wins at each position and replaced text is not matched again. Keep entries to what you have actually seen misheard: a short or common spelling will also rewrite real words. The list is read from the plugin config, so a change takes effect with the next recognition request after the profile reloads.


🌐 API

Route Description
GET /voice-mode/stream SSE: event: audio ({sessionId, seq, text, audio(base64 MP3)}), event: mode (global single-active ownership), event: tool (beep), event: asr-progress / asr-ready / asr-error / tts-error
POST /voice-mode/toggle {sessionId, on} enter/exit voice mode (globally single-active)
POST /voice-mode/asr Raw f32 LE 16k PCM payload → {text} (streaming zipformer2); returns 202 {loading} until the model is ready; ?reset=1 discards the in-flight segment (used on wake-word hit)
POST /voice-mode/cancel {sessionId} invalidates the TTS queue and drops the in-flight ASR segment
POST /voice-mode/preview {voice, rate?} one-shot synthesis preview → audio/mpeg (400 missing voice / voice too long; 502 synthesis failure, e.g. invalid ShortName; 403 when the plugin's enabled=false). Does not require voice mode to be active; uses an isolated synthesis connection and does not affect the reading queue
GET /voice-mode/config Client bootstrap parameters (silence threshold / sensitivity / voice and rate, etc.) — includes the ASR-side fields senseITN / senseVoice / captionFontSize / captionMaxWidth / backchannelYield / yieldMs / bargeInSalvage / llmCorrect
POST /voice-mode/correct {text} one LLM correction pass before sending (typos/homophones/punctuation only; reuses the host-configured model) → {text}; on failure/timeout/no model it falls back to the original text and never blocks sending
GET /voice-mode/voices Full Edge voice list (fetched when Edge is selected in settings; the client falls back to the common list on network failure)
GET /voice-mode Health check {ok, name, enabled, active}

💾 Model & cache

  • Recognition model: csukuangfj/sherpa-onnx-streaming-zipformer-zh-int8-2025-06-30 (encoder ≈154 MB / decoder / joiner / tokens, ~160 MB total), running host-side via sherpa-onnx (Node WASM, Apache-2.0, natively cross-platform)
  • Cache directory defaults by platform:
    • Windows: %LOCALAPPDATA%\dsh-voice-mode\models
    • macOS / Linux: ~/.cache/dsh-voice-mode/models
    • both overridable via cacheDir
  • Downloads use .part resume (single-file models only); huggingface.co falls back to hf-mirror.com on failure (configurable via modelHost)
  • A 120 s idle watchdog guards downloads: any received bytes reset it, a stalled upstream is aborted and the next source tried, and closing the panel aborts in-flight downloads
  • Main model files are SHA256-pinned; within one process a file whose size+inode+ctime is unchanged is not re-verified (the cache is in memory only, so the first run after a restart verifies in full); the Kokoro espeak-ng-data tree is judged complete by manifest file sizes / file count, and once first downloaded Kokoro works fully offline

🏛️ How it works

flowchart LR
    subgraph Client[Browser Client]
        Mic[Microphone 16kHz<br/>AudioWorklet<br/>echoCancellation:true] --> VAD[Client-side VAD<br/>RMS segmentation]
        VAD -->|partial 0.9s| UI[Statusbar + caption overlay]
    end

    subgraph Host[dsh.host]
        ASR[zipformer2 streaming ASR<br/>host-side WASM]
        SV[SenseVoice finalization<br/>+ ITN + punctuation]
        Tap[llm/stream tap<br/>observe-only]
        Seg[sentence segmenter]
        Q[TtsQueue<br/>epoch barge-in]
        TTS{Engine}
        Edge[Edge cloud]
        Vits[Local VITS WASM]
        Kokoro[Local Kokoro<br/>native addon]
        OpenAI[OpenAI-compatible TTS]
        MiMo[Xiaomi MiMo TTS]
    end

    UI -->|audio f32 PCM<br/>POST /voice-mode/asr| ASR
    ASR --> SV
    SV --> Draft[composer draft<br/>autoSend]
    Draft --> Tap
    Tap --> Seg
    Seg --> Q
    Q --> TTS
    TTS -->|edge| Edge
    TTS -->|vits| Vits
    TTS -->|kokoro| Kokoro
    TTS -->|openai| OpenAI
    TTS -->|mimo| MiMo
    Edge -.->|SSE audio frame| UI
    Vits -.->|SSE audio frame| UI
    Kokoro -.->|SSE audio frame| UI
    OpenAI -.->|SSE audio frame| UI
    MiMo -.->|SSE audio frame| UI
    VAD -.->|wake-word / barge-in| Q
  • Speech and reading only happen for the session pointed to by the global single-active pointer activeVoiceSession; other sessions pass through llm/stream with zero overhead (mode isolation)
  • The llm/stream tap is lossless: every chunk passes through unchanged; segmentation/synthesis only observe and never block the model stream
  • ASR runs host-side (sherpa-onnx WASM, zipformer2 Chinese streaming + SenseVoice finalization which adds punctuation); the browser only captures audio (getUserMedia 16k mono) and does endpoint detection
  • Local TTS (VITS / lazily-loaded native Kokoro) runs in an isolated child process (fork, auto-restart); barge-in kills the in-flight synthesis to release its worker resources
  • The TTS queue is per-session with an epoch version: old frames are all invalidated after a barge-in, so it is truly silent

🔍 Historical comparison with the early upstream implementation

Dimension Early upstream (historical) dsh-voice-mode (this plugin)
Recognition model Cloud API (needs key) Local zipformer2 + SenseVoice (zero key)
Multi-language English-first SenseVoice auto-recognition + ITN
TTS engine Cloud TTS Edge cloud + local VITS/Kokoro + OpenAI-compatible / Xiaomi MiMo
Barge-in detection Basic VAD 3 sensitivity levels + echo gate + yield semantics
Caption a11y None 4 font tiers + 3 width tiers + theme-following
Wake word None Lightweight streaming match + prefix filler whitelist
dsh compatibility — Declared range >=0.1.1-rc.2 <0.3.0-0; consult the tested-version matrix

🔐 Permissions and data flow

An honest disclosure of what the plugin does (mapped to the awesome-dsh-plugin capability scan: network / fs-read / fs-write / shell / env):

Category What it does Where data goes
network (1) Default Edge TTS sends spoken reply text over WebSocket to Microsoft speech.platform.bing.com; (2) ttsEngine: openai sends spoken text to the selected preset or configured openaiBaseUrl / openaiTts.baseUrl; (3) ttsEngine: mimo sends spoken text to mimoTts.baseUrl (https://api.xiaomimimo.com/v1 by default); (4) local VITS / Kokoro synthesize offline once models are downloaded from huggingface.co (or configured hf-mirror.com) with SHA256 digest checks; (5) registers loopback HTTP routes /voice-mode/* on the dsh host (loopback only by default; LAN needs allowLan) Browser audio is sent only to the connected dsh host for local recognition (no third-party cloud ASR). Submitted transcripts and optional llmCorrect follow the host LLM configuration
fs-read / fs-write Model cache (Linux/macOS ~/.cache/dsh-voice-mode/models/, Windows %LOCALAPPDATA%\dsh-voice-mode\models\); the dsh 0.1.7+ settings overlay $DSH_HOME/voice-mode.settings.json (0600, API keys remain host-only and are never returned to the browser); never touches your workspace files All local
shell (child processes) Local TTS synthesis runs in a separate child process (child_process.fork → lib/tts-vits-worker.cjs); when the host is Electron (the official desktop app), it additionally probes and launches a real Node.js for Kokoro (node -p … to check the version). It never executes user input and never builds shell command strings Local
env Uses directory/runtime environment settings such as DSH_HOME, LOCALAPPDATA, PATH and optional DSHVM_NODE; this default speech path does not require an extra speech-service API key —

Also: the on-device recording fixture used for debugging is off by default; the plugin sends no telemetry. See SECURITY.md for the threat model and how to report vulnerabilities.

Third-party models and services

  • Local models are not shipped in the package: the ASR (streaming zipformer, SenseVoice re-transcription), VAD and local TTS (VITS / Kokoro) models are downloaded on first use from Hugging Face (or your configured mirror) into a local cache; they come from export repositories in the sherpa-onnx ecosystem (csukuangfj/*). Each model is licensed by its original authors — this plugin's MIT license does not cover them; verify the upstream licenses before commercial use (the model list is under "Model & cache"; the constants are in src/asr-host.ts / src/tts-local.ts).
  • The Edge cloud read-aloud is a third-party online service: the default engine uses the Microsoft Edge browser "Read Aloud" endpoint via msedge-tts. It is not an officially supported Microsoft API — availability, rate limits and terms are Microsoft's and may change at any time. For stability/compliance-sensitive use, switch to local VITS / Kokoro (offline).
  • Licenses of the third-party code inlined into the published package (msedge-tts and its dependencies) are in THIRD_PARTY_NOTICES.md; pnpm audit checks known high+ advisories and does not establish the absence of undisclosed or untracked vulnerabilities (CI audit job keeps watching).

Compatibility declaration (what the plugin market's version filter reads)

package.json declares engines.dsh = ">=0.1.1-rc.2 <0.3.0-0". This is an eligibility range, not evidence that every version in it has passed tests. The compatibility record lists tested releases through 0.2.2-rc.5 and the forms actually exercised. The default set deduplicates the support endpoints, the latest verified release on each version line, and upstream latest/next: currently 9 versions for anchors and host/client typechecks, with 8 runtime cores. Historical evidence remains available; use verify-dual.sh --full or explicit versions to rerun it. Compatibility-only verification does not trigger a release. The daily workflow compares upstream tags and opens an issue for uncovered versions; it does not automatically rerun the full matrix. peerDependencies lists @deepseek-ai/cordis and react supplied by the host.

🚧 Known limitations

  • Barge-in relies on browser echo cancellation (echoCancellation); loud speaker volume may leak into the mic (a self-built NLMS canceller only kicks in as a fallback when native AEC is unavailable)
  • Ctrl+Shift+V overrides the browser's "paste as plain text" shortcut (normal Ctrl+V paste still works)
  • The recognition model prioritizes Simplified Chinese; recognition quality is affected by ambient noise
  • Browser autoplay policy: reading requires prior user interaction on the page (clicking the mic satisfies it); if the browser blocks playback and the status bar shows no hint, make sure the page is foregrounded and not muted
  • The wake word is a lightweight implementation (text matching on the streaming transcript, not a dedicated KWS engine): it may lag or misfire in noisy environments; the wake word itself never enters the chat (the buffer is dropped on hit)
  • In hold mode, switching windows/tabs while holding abandons the segment (prevents continuous recording); come back and hold again
  • The hero (new-session empty state) has no voice entry: voice mode is a session-level feature; enter a session first and use the mic button in the input toolbar
  • Entering/exiting voice mode, reading/writing settings and preview all rely on AbortSignal.timeout (Chrome 103+ / Firefox 100+ / Safari 16+); on older browsers these actions immediately show a failure hint — an expected degradation
  • Cloud read-aloud (Edge) depends on a Microsoft online service: the server side may change its handshake or limits at any time (it did on 2026-10-15 — see CHANGELOG 0.7.24). If several sentences in a row fail, the status bar/toast suggests switching to local VITS / Kokoro, which the settings panel does in one click
  • spokenFormat is injected through the official system-prompt/assemble waterfall; if the current session uses a complete prompt (persona complete: true agent preset) it is not injected (the official complete contract wins)
  • Local Kokoro needs about 1 s to rebuild its engine before the next sentence after a barge-in (the price of killing in-flight synthesis)
  • Apple Safari / iOS: needs HTTPS or localhost (a secure context is mandatory; the mic is unavailable on http:// LAN IPs); grant the mic on first entry (iOS: Settings → Safari → Microphone if denied); recognition and read-aloud pause while backgrounded or locked and resume on return (sentences may be lost)
  • Security note: the plugin's HTTP surface (/voice-mode/*) follows the host's security model — do not expose the dsh port to the public internet; behind a reverse proxy, authenticate at the proxy (e.g. basic auth)

🛠️ Troubleshooting

Symptom Fix
Mic click does nothing, red hint in the status bar The browser denied mic permission: allow it in the address bar and retry
Status bar stuck on Loading model… x% Check the network; the model is large (160 MB) — npm run prefetch first; on mainland networks set modelHost to https://hf-mirror.com
Status bar shows Model download failed (<file>) Both mirrors are unreachable: check network/proxy and re-enter voice mode (resumable)
Local Kokoro on the official desktop app shows "failed" and asks for Node.js The desktop host is Electron, where Kokoro's native add-on cannot run: install Node.js >= 18 on PATH (or set DSHVM_NODE to its path) and retry, or use the default Edge / local VITS (unaffected on desktop)
No sound and no captions on the default (Edge cloud) engine Edge read-aloud depends on a Microsoft online service. If several sentences fail in a row you will see "cloud read-aloud unavailable" — switch the engine to local VITS / Kokoro in settings. v0.7.23 and earlier are silent since 2026-10-15 (server-side handshake rejection) — upgrade to ≥0.7.24
Caption appears (overlay) but no sound Check system volume/output; if autoplay is blocked, click anywhere on the page and retry
Status bar shows 朗读连接失败:正在重试… Edge TTS unreachable (overseas service), auto-retries; if it persists, check network/proxy
Poor recognition Reduce background noise and compare partial/final text. interruptLevel controls barge-in, not recognition accuracy
Hold mode has no effect Make sure hold mode is active and you're in voice mode (button shows 按住说话); the browser window must be in the foreground
Preview button reports synthesis failure Edge TTS unreachable (overseas) or the ShortName doesn't exist: verify the name (node scripts/list-voices.mjs lists all) and retry later
Caption is hidden behind the input box Default captionMaxWidth=1 (70vw) + captionFontSize=0 (12px) can overlap the bottom input on narrow viewports; raise the tier or click the caption's × to dismiss
Yield behavior is wrong (saying 嗯 doesn't pause / real speech gets hard-barge) Short backchannel words (嗯 / 对) auto-pause 1.5 s then reading resumes; continuing to speak triggers hard barge-in; disable backchannelYield to restore pre-change behavior (ADR-0008)
Says 嗯 but no yield happens Confirm backchannelYield=true (default on); in hold mode, continuing to talk within 1.5 s of release triggers hard barge-in
Auto-exits after 5 minutes idle (don't want) Raise idleTimeoutMinutes (default 5 min; reading, the AI speaking and a running turn all count as activity, and background tabs renew too; it still exits 2 hours after your last real action)
Mic button spins forever The host not answering times out after 8 s; click the mic again while it spins to cancel entering, leaving no voice session behind

🛣️ Roadmap

The current roadmap defines priorities, candidate admission, acceptance gates and rollback conditions.

  1. Close confirmed defects and validate one real ASR → editable draft → send → TTS workflow.
  2. Address the highest-loss experience problems: dropped speech, unintended sends, edit conflicts and recovery.
  3. Make model updates and fallback controlled; evaluate new models or protocols only against a measured bottleneck. A new backend is optional, not a delivery promise.

Client-side VAD, WebSocket migration, voice cloning and broad tool integration are not approved near-term work. The historical backlog preserves earlier research; its old Ready labels and estimates are not current commitments.


🛠️ Development

Use Node.js 22.18+ on the 22.x line and pnpm 11 for source development/tests; macOS builds require macOS 12+. The plugin runtime declaration is not the test-toolchain requirement.

Dependency discipline (important)

  • Runtime dependencies (dependencies): @deepseek-ai/schemastery, sherpa-onnx, sherpa-onnx-node — external at runtime, resolved from the registry.
  • msedge-tts is a build-time devDependency and is

…

Content from the project README on GitHub ↗

Links

More in this category

View the whole category →

Community comments

Comments are public GitHub Discussions. Loading them connects to GitHub and Giscus; a GitHub account is required to post.