Low-cost video understanding tool: a video_understand tool turns a Bilibili link / BV / local video into an AVIS info layer (ASR + scene structure + object tracks + YOLO labels) and returns summary + Q&A. Question-driven dynamic routing across layers (L0 ASR / L1 object tracks / L2 key-frame VLM), semantic-layer reuse for repeat questions, per-question budget cap. Python engine: core layer needs faster-whisper / opencv / yt-dlp (~200-300MB); the optional semantic layer adds ~2GB of torch / transformers / ultralytics. A bundled doctor --fix sets up the venv and installs both.
Install
# from npm (prebuilt)
dsh plugin --profile web add dsh-video-understand
# from GitHub (first run asks for allowBuilds approval — follow the hint, retry)
dsh plugin --profile web add github:ilps2/dsh-video-understand
Any plugin you install runs third-party code with your own permissions — it can read your files, use your credentials, and reach the network, and tool approvals don’t sandbox it. GitHub-sourced plugins also run build scripts at install time — pnpm blocks those until you allow them, so an install can stop with ERR_PNPM_GIT_DEP_PREPARE_NOT_ALLOWED or ERR_PNPM_IGNORED_BUILDS; dsh prints the exact key to add under allowBuilds in your profile’s pnpm-workspace.yaml, and the install works on the next run. Allowing a build is a trust decision: only install sources you trust, and pin a commit (github:owner/repo#sha).
README
This plugin publishes its README in Chinese only.
低成本视频理解插件:给 dsh agent 注册 video_understand 工具——B站链接 / BV 号 / 本地视频 → AVIS 信息层(ASR 转写 + 场景结构 + 运动对象轨迹 + YOLO 语义)→ 摘要+问答。采用 Python 引擎:核心层需 faster-whisper / opencv / yt-dlp(约 200-300MB),可选语义层另需约 2GB 的 torch / transformers / ultralytics,内置 doctor --fix 一键建 venv 并装齐两者。
安装
# npm 安装
npm install dsh-video-understand
# 或 dsh plugin add dsh-video-understand
装完即用:首次调用 video_understand 时自动创建插件本地隔离环境(.venv)并安装核心依赖(优先 uv,回退 venv+pip 清华镜像)——无需手动执行任何命令,不污染系统 Python。
引擎已内含,无需额外克隆外部仓库。
环境自检
npx dsh-video-understand doctor # 逐项检测 + 给出修复命令
npx dsh-video-understand doctor --fix # 一键自动修复(建环境 + 装依赖)
前置条件仅两个:ffmpeg(macOS brew install ffmpeg / Ubuntu sudo apt install ffmpeg)和一个 LLM API key(见下表)。语义层依赖(torch/CLIP/YOLO,约 2GB)为可选,仅建完整语义层时再装:pip install -r engine/requirements-layer.txt。
⚠️ 数据流披露
| 级别 | 数据流向 | 说明 |
|---|---|---|
| L0(默认) | 完全本地 | ASR + 场景分类 + 运动检测,不上传任何数据 |
| L1 | MiMo API | 视频帧发送至 MiMo 服务器进行 VLM 分析 |
| L2 | MiMo API | 视频帧发送至 MiMo 服务器进行 VLM 分析 |
- L0 级别(默认)仅使用本地 ASR + 场景分类 + 运动检测,不涉及云服务
- L1/L2 级别会将视频帧(JPEG 编码)发送至 MiMo API 进行视觉理解
- 帧数据仅用于单次 VLM 推理,不会被存储或用于训练
环境变量
| 变量 | 必需 | 说明 |
|---|---|---|
LLM_API_KEY |
✅* | LLM API 密钥(默认 MiMo:sk-xxxxx) |
LLM_API_URL |
❌ | API endpoint(默认 https://api.xiaomimimo.com/v1/chat/completions) |
LLM_MODEL |
❌ | 模型名(默认 mimo-v2.5) |
VIDEO_UNDERSTAND_PYTHON |
❌ | Python 解释器路径(自动检测有依赖的 Python) |
BILI_DOWNLOAD_SCRIPT |
❌ | bilibili-downloader 脚本路径(下载 B站视频用) |
* 未设置
LLM_API_KEY时,自动从~/.dsh/.credentials.yaml读取(优先XIAOMI_API_KEY,兼容DEEPSEEK_API_KEY)。一个 key 搞定。
工具
video_understand(target, questions?, noDownload?, level?, window?)
| 参数 | 类型 | 说明 |
|---|---|---|
| target | string | B站 URL / BV 号 / 本地视频绝对路径 |
| questions | string[] | 可选,自定义问题(默认 3 问) |
| noDownload | boolean | 本地文件置 true |
| level | string | l0(默认) / l1(+3-5帧VLM视觉摘要) / l2(+时间窗密集帧证据) |
| window | string | L2 时间窗,如 10-30 或秒数(auto=轨迹最活跃30s) |
| budgetCny | number | 单次问题预算上限(元),视觉成本估算超预算自动降级(拦截 L2 用 L0/L1 回答) |
返回 JSON:video / duration_s / token_compression_pct / cost_cny / answers[]。
结构
dsh-video-understand/
├── package.json
├── cordis.patch.yml
├── dsh/
│ └── index.js # host 端:注册 video_understand 工具
├── engine/ # Python 引擎(核心层 + 可选语义层依赖)
│ ├── understand_video.py
│ ├── avis.py
│ ├── visual_level.py
│ ├── frame_prep.py
│ └── livestream-highlight/
│ └── asr.py
└── skills/
└── video-understand/SKILL.md
分级实测(2026-08)
| 层级 | 内容 | 数据流向 | 成本(估算*) |
|---|---|---|---|
| L0 信息层 | ASR+场景+轨迹+YOLO → 摘要/问答 | 本地 | 仅 LLM 文本成本 |
| L1 视觉级 | 3-5 帧 VLM → 颜色/姿态/衣着 | MiMo API | +数帧 VLM 成本 |
| L2 证据级 | 时间窗密集帧 → 时间线 | MiMo API | 按窗长 |
* 成本按
engine/understand_video.py的价格常量估算(可配置输入),实际随厂商定价变化,非测量承诺。详见 docs/blog-视频理解性价比实验(方法 + 实测)。
实测:电影解说 L1 补出「白色立领衬衫/神情凝重/暗色调诊室」(L0 完全给不出);舞蹈 L2 逐帧「头部转 15-20°→45°、口型开口→闭合→微笑」。
设计背景
本插件的核心目标是低成本视频理解:用信息层代替逐帧像素喂 LLM,单视频 LLM 调用仅需几千 token(具体成本取决于所选模型定价,见上)。
- LLM 选型:最初选用 DeepSeek v4 Flash 因其性价比最优;近期调价后已迁移至 MiMo,持续追踪成本效益比
- 视觉级(L1/L2):DeepSeek 暂不支持多模态输入,故 L1/L2 使用 MiMo API 进行视觉分析(此前曾使用 DashScope 作为过渡方案)。未来若 DeepSeek 支持多模态或出现更优的本地推理方案,可进一步优化成本
原理
引擎把视频压缩成信息层(ASR 转写 + 场景结构 + 运动对象轨迹 + YOLO 语义,约 1k token)再喂 LLM;同一视频重复理解时信息层缓存复用(内容哈希,二次提问跳过 ASR)。成本与 token 对比的具体测量见 docs/blog-视频理解性价比实验。
License
MIT
Links
More in this category
Tencent/WeKnora#dsh-weknora★ 37764
Four read-only tools over a WeKnora knowledge base: list knowledge bases, hybrid passage search, reassemble one document's chunks in order, and WeKnora's own cited RAG or ReAct-agent answer with a resumable session id.
DayuanJiang/next-ai-draw-io#dsh-plugin★ 37055
Draws draw.io diagrams in DeepSeek Harness through the next-ai-drawio MCP server: the link in the reply opens a live preview in the right sidebar, you can edit the diagram by hand, and each AI change is a version you can restore. Node.js 18+; npx downloads the MCP server on first use.
superdesigndev/treg★ 8452
Tool catalog for agents: search ~2,600 external endpoints (SEO and SERP, backlinks, social, people and company enrichment, ad libraries, scraping) by the task you want done, read each one's parameters and per-call price, then call it with the credential injected server-side. Ships the skill plus an MCP row that stays disabled until TREG_TOKEN is set.
T-Auto/dsh-ops★ 5921
Provides bash, PowerShell 7, and high-performance Rust tools for DSH on Windows to reduce token usage.
sixtysevenlf/dsh-blender-plugin★ 2194
Lets a DSH agent drive a local Blender over a direct TCP channel with 15 tools: viewport frames, custom-angle renders, Python execution, headless and job offload, transaction rollback, and a planning layer of 228 ops across 34 families.
helloHub1212/dsh-windows-mcp★ 1751
Bridges the CursorTouch/Windows-MCP desktop automation server into DSH through the official MCP client bridge, registering its screenshot, UI tree snapshot, mouse and keyboard input, clipboard, process and registry tools as mcp__windows__* tools for the model, and adds a windows_mcp_doctor diagnostic tool. Computer Use is off by default and is switched on from the plugin panel.
Community comments
Comments are public GitHub Discussions. Loading them connects to GitHub and Giscus; a GitHub account is required to post.