iiwish/dsh-testkit

DSH Testkit

Deterministic, real-host lifecycle testing for DeepSeek Harness plugins.

简体中文 · Scenario reference · Architecture · Contributing

DSH Testkit packs your plugin, installs it beside an exact DSH version in a disposable environment, boots the real host, exercises deterministic capabilities, uninstalls the plugin, reboots the same profile, and retains reviewable evidence. No model call or model API key is required.

resolve -> install-dsh -> package -> install-plugin -> assemble -> boot -> register
        -> exercise -> update? -> uninstall -> reboot -> recover? -> cleanup

Quick Start

Requirements: Node.js 22 or newer and Docker.

pnpm add -D dsh-testkit
pnpm dsh-test . --dsh 0.1.0-rc.6 \
  --expect-row tool-my-plugin \
  --expect-tool my_tool

Docker is the default runner. A successful run produces report.json, junit.xml, report.md, sanitized command logs, and stage evidence in .dsh-testkit/runs/.

The current adapter supports exactly @deepseek-ai/dsh@0.1.0-rc.6. Unknown versions stop before runner creation with exit code 4, so host drift is not misreported as a plugin failure.

What It Proves

Signal	How it is tested
Package integrity	Local directories go through npm pack; links and unpublished files cannot hide packaging defects.
Real registration	Rows come from DSH --dump-config; services and tool schemas come from an in-process Cordis probe.
Deterministic exercise	The baseline runtime probe and declared tool calls run through the real tool runtime without model selection.
Clean removal	The same profile is rebooted after uninstall and checked for bundles, capabilities, processes, ports, and owned-path residue.
Repeatability	--suite full runs five isolated attempts and returns flaky when semantic outcomes disagree.
Observer limits	Unavailable coverage is disclosed; a required unavailable observer returns unsupported, never a false pass.

It does not prove that arbitrary executable code is safe or that a plugin produces high-quality model output.

Choose The Right Check

These tools are complementary, not competing replacements.

Need	Best fit
Fast static checks, manifest diagnostics, dependency guidance	dsh-plugin-doctor or a plugin preflight check
Conflicts among several bundles before or during assembly	dsh-composition-check
Plugin-owned unit logic	Your test framework
Install, boot, exercise, uninstall, reboot, recovery, residue, and repeatability on a real host	DSH Testkit

DSH Testkit deliberately tests one subject plugin per isolated lifecycle. Multi-plugin state ownership and update order remain outside the current scenario contract until field evidence shows a failure that single-plugin lifecycle testing plus composition checks cannot reproduce.

Scenario As Code

Create dsh-testkit.yaml in the plugin project:

schemaVersion: 1
name: my-plugin-quick
subject:
  source: .
dsh:
  version: 0.1.0-rc.6
expect:
  boot: success
  rows: [tool-my-plugin]
  services: [myService]
  tools: [my_tool]
exercise:
  - tool: my_tool
    arguments:
      value: smoke
observers:
  filesystem: required
  process: preferred
  ports: preferred
  network: off
  canary: preferred

The Scenario Reference covers update targets, expected failures, recovery, timeouts, observer policy, and stage reruns.

CI Evidence

Use the stable moving major tag in GitHub Actions:

- uses: iiwish/dsh-testkit/.github/actions/dsh-test@v0
  with:
    plugin: .
    dsh-version: 0.1.0-rc.6

The Action publishes JUnit and uploads the complete run directory. artifact-name, check-name, output, and retention are configurable; artifact ID, URL, and digest are outputs. GitHub Enterprise Server and other CI systems can invoke the CLI directly because actions/upload-artifact@v4+ is not available on GHES.

Stable exit codes are: 0 passed, 1 lifecycle failure, 2 invalid input, 3 infrastructure error, 4 unsupported capability, and 5 flaky. Published JSON Schemas live at dsh-testkit/schemas/report-v1.json and dsh-testkit/schemas/scenario-v1.json.

DSH-Native Tool

DSH Testkit also ships an optional community-maintained DSH-native Profile Bundle:

dsh plugin --profile web add dsh-testkit@0.2.1
dsh --profile web --dump-config

The bundle registers dsh_test, a thin adapter over the same lifecycle engine. It defaults to the active workspace, requires confirm: true, always uses Docker, ignores implicit repository configuration, rejects paths outside the workspace, and never exposes unsafe-local execution or arbitrary CLI arguments.

This adapter is convenient when DSH is already healthy. Keep the external CLI or CI Action as the independent recovery and release gate because an in-host tool cannot diagnose a host that fails before tool registration.

Community Validation

Maintainers can run an exact-version public cohort with an explicit trust acknowledgement:

pnpm exec dsh-test-community \
  --acknowledge-untrusted-code \
  --dsh 0.1.0-rc.6 \
  --plugin example-plugin@1.2.3 \
  --output /tmp/dsh-testkit-cohort

The runner strips model, npm, GitHub, cloud, and Docker registry credentials from child processes. It keeps named reports locally and writes a subject-free aggregate summary for responsible public reporting.

See the v0.2.1 community validation report for the selection method, aggregate evidence, limitations, and the resulting product decision.

Safety

Plugins are executable code: package scripts and runtime code run during a lifecycle test. Docker narrows the default blast radius but is not a hardened malware sandbox. Use disposable infrastructure for unknown code and never use --runner local --unsafe-local for an untrusted plugin.

The native tool needs Docker daemon access and may execute package scripts with network access inside the runner. Confirmation is a trust decision, not a security certification. Private plugins stay on the CI runner; DSH Testkit has no SaaS dependency and does not upload source or credentials.

See Architecture for trust boundaries and Security Policy for private vulnerability reporting.

Contributing

Bug reports are most useful with the exact plugin version, DSH version, failing stage, report.json, and sanitized logs. Start with Contributing, then use the lifecycle-failure issue template for reproducible host behavior.

DSH Testkit is an independent, unofficial community project released under the MIT License.