initial-d/dsh-plugin-mlquant-benchmark

DeepSeek Harness tools for reproducing the initial-d/ml-quant-trading protocol v1 CPU benchmark.

The point is narrow: make a DSH agent able to run the existing benchmark, read the machine-readable artifact, validate it against the benchmark protocol, and draft an issue-ready report. This plugin does not add a trading agent, does not call market data APIs, and does not configure any model provider.

Why this exists

ml-quant-trading is a good reproducibility target for agent harnesses:

• deterministic synthetic benchmark input;
• fixed protocol v1 command, seed, panel size, repetitions, and thread counts;
• JSON artifact suitable for automated checking;
• public issue template for DeepSeek Harness benchmark reports;
• explicit boundary that benchmark throughput is not trading performance.

Challenge: can DeepSeek Harness reproduce a quant benchmark end to end, preserve the evidence bundle, and avoid turning runtime numbers into alpha claims?

Listed in awesome-dsh-plugin via PR #2573.

Run-To-Report Path

1. Install the plugin from GitHub.
2. Open an initial-d/ml-quant-trading checkout in DSH.
3. Ask DSH to run, validate, summarize, and draft a benchmark report.
4. Submit the drafted report through the dedicated issue template.

That path is intentionally small: the plugin turns DSH attention into a reproducible benchmark report, not an investment or leaderboard claim.

Tools

This package registers four DSH tools:

Tool	Purpose
mlquant_benchmark_v1_cpu	Run the fixed protocol v1 CPU benchmark and write artifacts/benchmark-v1.json.
mlquant_read_benchmark_json	Read the JSON artifact and render a compact Markdown result table.
mlquant_validate_benchmark_json	Check protocol v1 fields, expected cases, fixed parameters, and variance warnings.
mlquant_draft_github_issue	Draft a DeepSeek Harness benchmark issue body from the JSON artifact. It does not post to GitHub.

Install

Install the package in a DeepSeek Harness profile or preset environment:

dsh plugin --profile web add github:initial-d/dsh-plugin-mlquant-benchmark

The package declares a dsh.bundle manifest that inserts:

- id: mlquant-benchmark
  name: dsh-plugin-mlquant-benchmark

If you use a local checkout while developing, add the same row manually:

- id: mlquant-benchmark
  name: file:/path/to/dsh-plugin-mlquant-benchmark

This package is intentionally not published to npm yet. GitHub distribution is enough for the first DSH-facing benchmark reports; npm can come later if there is real usage.

Suggested DSH prompt

Read AGENTS.md, docs/benchmarking.md, and docs/reality_check.md.
Use the mlquant benchmark tools to run the protocol v1 CPU benchmark, validate
and read the JSON artifact, and draft a DeepSeek Harness benchmark report. Keep
the result as an engineering reproducibility benchmark, not a trading-performance
claim.

Public report path

Post the drafted report through the main repository's dedicated template:

https://github.com/initial-d/ml-quant-trading/issues/new?template=deepseek_harness_benchmark.yml

Seed example:

https://github.com/initial-d/ml-quant-trading/issues/61

Independent DSH runs are tracked in the plugin challenge issue:

https://github.com/initial-d/dsh-plugin-mlquant-benchmark/issues/1

Post there if the plugin failed before a valid benchmark-v1.json artifact was created. Post successful or caveated benchmark reports through the main repository template above.

For context and agent-facing guardrails, read the main repository's DeepSeek Harness Recipe and Quant Agent Reproducibility Target.

Development

npm install
npm test

The test loads the plugin with a mock ctx.tools.register, verifies that the four tools register, reads and validates sample artifacts, and drafts an issue body.

Non-goals

• No investment advice.
• No backtest-performance claim.
• No hidden model provider configuration.
• No posting to GitHub from the tool.
• No private data or API keys in artifacts.