RyanShen3/dsh-toutiao-reader

dsh-toutiao-reader

中文 | English

Packaged real-world experience of reading Toutiao (今日头条) articles as a DeepSeek Harness (DSH) plugin: a webfetch tool + a toutiao-reader skill.

model calls webfetch(url)
   ├─ ① direct-fetch        static direct ~1s (domain cache remembers which sites need a browser)
   ├─ ② resident Edge CDP   cold start ~20s once → resident on port 9333, then ~3-6s per fetch
   │     · reuses a persistent tab (saves ~4s)   · blocks images + ByteDance CDNs for speed
   │     · auto grace window for Toutiao challenge pages (安全验证/__ac_signature/acrawler)
   └─ ③ dump-dom fallback   one-shot render when CDP is unavailable

Why

Toutiao's static HTML always trips its anti-bot layer (__ac_nonce/__ac_signature/acrawler) — a plain fetch gets a "security verification" page, and web_search only returns snippets. This plugin fills the gap: give it an article URL, get back title + full plain text. Tuned for Toutiao, and works for ordinary pages (blogs, news sites, anything reachable directly) too.

What you get

Tool: webfetch

Arg	Type	Notes
url	string, required	Article URL (PC pages at www.toutiao.com/article/<id>/ work best; mobile/share pages are usually App-interstitials with no article text)
maxChars	number	Cap on returned text, default 12000, max 60000; sets truncated when cut
outfile	string	Optional: save full text to a file (title + blank line + body); absolute path, or relative to the workspace config

Returns { method, title, text, textLength, truncated, outfile?, error? }

• Failures come back as { error } instead of throwing — fetching never breaks the main flow
• Declares isConcurrencySafe: false (CDP reuses one tab, so calls are forced sequential) and timeoutMs: 120000
• The method field tells you which tier served the request

Skill: toutiao-reader

A bundled experience doc (model-invocable): Toutiao URL forms, reading the method field, challenge-page handling and retry pacing, troubleshooting short/empty bodies (video pages, Q&A pages, App interstitials), managing the resident headless browser (port/pid/stop command), network-environment notes, and the fallback script path.

Install

Option 1: the plugin market (recommended)

DSH Web → Settings → Plugin Market → search "toutiao" → install.

Option 2: CLI

dsh plugin --profile web add dsh-toutiao-reader

Option 3: manual (local development)

# junction into the profile (no admin needed)
New-Item -ItemType Junction -Path "$env:USERPROFILE\.dsh\profiles\web\node_modules\dsh-toutiao-reader" -Target "<plugin dir>"
# profile package.json: add "dsh-toutiao-reader": "link:<plugin dir>" to dependencies and "dsh-toutiao-reader" to dsh.profile.bundles

Restart dsh web to load. Never run pnpm install inside the profile (it duplicates official packages as physical copies).

Configuration

Use the id-override mode in the profile's cordis.patch.yml (id only — never re-insert):

- id: dsh-toutiao-reader
  config:
    workspace: D:/DSH_Workspace  # base for relative outfile paths
    cdpPort: 9333                # CDP debugging port
    browserPath: ""              # Edge/Chrome override; empty = auto-detect
    maxChars: 12000              # default text cap
    domainCachePath: ""          # domain cache path; empty = %TEMP%/webfetch-domains.json

FAQ

Slow fetches? The first cold start of the resident browser takes ~20s; afterwards ~3-6s per fetch. Image blocking and ByteDance CDN filtering in the CDP tier are intentional speedups.

Very short or empty text? Usually a video page, Q&A page, or App interstitial (mobile share pages carry only ~40 chars of promo text in practice). Switch to the PC URL (www.toutiao.com/article/<id>/).

Stuck in a challenge loop? Wait a few seconds and retry once; avoid hammering (stricter risk control). Solved domains are remembered by the domain cache.

Browser in a bad state? Read %TEMP%\webfetch-edge.pid, then taskkill /PID <pid> /T /F; the next call cold-starts a fresh one.

Already using a script? The plugin shares the resident browser and tab with the workspace script node tools/webfetch.mjs <url> [outfile] (same pid/tab/profile files) — they accelerate each other; the script is the fallback path when the plugin is unavailable.

Security & privacy

• Read-only fetching: writes nothing (unless you pass outfile), reads no credentials, phones nowhere
• The resident headless browser listens on localhost only (127.0.0.1)
• The domain cache stores a single string per domain (whether the site needs a browser) — no content is recorded

Compatibility

• Requires local Edge or Chrome (auto-detected; override with browserPath)
• Node.js ≥ 20 (global fetch / WebSocket / AbortSignal.any); tested on DSH web 0.1.0-rc.6+

License

MIT