v0.9: parallelize phase4 and add model/search playbooks

This commit is contained in:
kai
2026-04-24 10:19:48 +08:00
parent c444007f04
commit c93fce5078
13 changed files with 626 additions and 100 deletions
+4 -2
View File
@@ -62,13 +62,15 @@ uv run python scripts/build_glossary.py <slug> --workers 4
```bash
# 先预览
uv run python scripts/apply_glossary.py <slug> --dry-run
uv run python scripts/apply_glossary.py <slug> --input phase4/final_zh.md --dry-run
# 确认无误后应用
uv run python scripts/apply_glossary.py <slug>
uv run python scripts/apply_glossary.py <slug> --input phase4/final_zh.md
```
这会把 glossary 中发现的拼写错误 / 错译直接替换进 `final_zh.md`
如润色后仍需二次复核,可手动对 `final_zh_polished.md` 再运行一次:
`uv run python scripts/apply_glossary.py <slug> --input phase4/final_zh_polished.md --dry-run`
## Step 6: 润色(Python 脚本)
+31 -2
View File
@@ -1,8 +1,8 @@
# Deep Research 系统方案(OpenCode 实现)
> 本文件是整套方案的**单一真实源**,中断后续接时从此文件恢复上下文。
> 最后更新:2026-04-20
> 实施阶段:路径 2 — 最小可用先行(MVP
> 最后更新:2026-04-24
> 实施阶段:v0.9 — OpenCode 全流程 + Phase 4 并发化 + Codex 第一阶段兼容
---
@@ -492,3 +492,32 @@ OpenCode 的坑:如果只是在主会话里装样子地写"让 X agent 做"
- 反驳证据段标题规范化(建议从"反方证据/Counter-Evidence"改为观点化标题如"另一种声音")
- build_glossary 默认放到 Phase 2 阶段运行,在源头拦截错误
- 提示 dr-analyst 加强对公司名/机构名的搜索验证流程
- 2026-04-24 v0.9**Phase 4 并发提速 + 模型/搜索攻略本 + Codex 兼容**
**目标**:在不破坏 OpenCode 主流程的前提下,把 v0.6 Python 化 Phase 4 进一步提速,并补齐跨平台使用说明。OpenCode 仍是主适配器;Codex 第一阶段只复用 `AGENTS.md` 与 Python 脚本,不复刻 OpenCode subagent。
**Phase 4 并发化**
- `scripts/translate.py` 新增 `--workers`,默认 4;设为 1 时回退串行。
- 翻译阶段改为"稳定术语表快照 + 并发 chunk 翻译 + 事后统一合并 glossary patch",避免多线程同时写 `glossary.json`。
- `scripts/polish.py` 新增 `--workers`,默认 4;润色块彼此独立,按完成顺序写 chunk,最终按原始 order 合并。
- `scripts/lib/zenmux_client.py` 增加日志与 usage 聚合锁,避免并发 JSONL 日志交错或 token 统计竞争。
**流程修正**
- 修正 `apply_glossary.py` 默认输入,从 `phase4/final_zh_polished.md` 改为 `phase4/final_zh.md`。
- `/dr-finalize` 明确默认顺序:`translate.py → build_glossary.py → apply_glossary.py --input phase4/final_zh.md → polish.py → build_report.py`。
- 保留二次修正选项:润色后可手动对 `final_zh_polished.md` 再跑一次 `apply_glossary.py --input phase4/final_zh_polished.md --dry-run`。
**模型与搜索攻略本**
- 新增 `docs/model-playbook.md`:定义 premium / balanced / budget / cn-heavy / verifier 五套模型策略。
- 新增 `docs/search-playbook.md`:说明 Tavily / Exa / Brave / Serper / PubMed / ClinicalTrials / FDA/EMA/NMPA / Patents 的使用边界。
- 新增 `configs/model_profiles.yaml` 与 `configs/search_profiles.yaml`,作为跨平台、人类和 agent 共用的策略配置参考;当前不强制重构 `.opencode/agents` 自动读取。
**Codex 兼容**
- 新增 `docs/codex-usage.md`,说明 Codex 下如何遵循 `AGENTS.md`、运行 Phase 4 Python 流水线、检查 git staging,避免误提交 `projects/**` 研究产物。
- Codex v1 定位为"审阅/规划/修补/执行脚本";确定性编排继续放在 Python 脚本,OpenCode subagent 调度暂不移植。
**Git 管理要求**
- 本轮迭代应在独立分支推送到 Gitea。
- 提交范围仅限系统文件和文档:`README.md`、`PLAN.md`、`scripts/**`、`docs/**`、`configs/**`、必要的 `.opencode/commands/**`。
- 不提交 `projects/**`、生成的 PDF/DOCX/TXT、一次性研究产物或本地临时脚本。
+49 -12
View File
@@ -2,8 +2,8 @@
> 生物医药行业的 AI 驱动深度研究流水线。基于 OpenCode 多 agent 协作,以麦肯锡/德勤式方法论产出专业级研究报告(PDF + DOCX)。
**当前状态**MVP(路径 2 — 最小可用先行),仅实现 Phase 1 能力
详见 `PLAN.md` 了解完整方案与迭代路径。
**当前状态**v0.9 核心迭代。OpenCode 全流程可用(Phase 1-4),Phase 4 已切换为 Python 脚本化流水线,并新增 Codex 第一阶段兼容说明
详见 `PLAN.md` 了解完整方案、版本记录与迭代路径。
---
@@ -108,22 +108,23 @@ source scripts/activate.sh
---
## MVP 可用命令
## 可用命令
| 命令 | 功能 | 状态 |
|---|---|---|
| `/dr-init <主题>` | 初始化新研究,启动访谈 | ✅ MVP |
| `/dr-frame [slug]` | Phase 1:生成 8-15 章研究框架 | ✅ MVP |
| `/dr-research` | Phase 2:深度研究(并行) | ⏳ 下一阶段 |
| `/dr-review` | Phase 3:总编审校 | ⏳ 下一阶段 |
| `/dr-finalize` | Phase 4成稿 PDF+DOCX | ⏳ 下一阶段 |
| `/dr-status` | 查看进度 | ⏳ 下一阶段 |
| `/dr-init <主题>` | 初始化新研究,启动访谈 | ✅ 可用 |
| `/dr-frame [slug]` | Phase 1:生成 8-15 章双语研究框架 | ✅ 可用 |
| `/dr-research [slug]` | Phase 2并行深度研究 | ✅ 可用 |
| `/dr-review [slug]` | Phase 3:总编审校 | ✅ 可用 |
| `/dr-finalize [slug]` | Phase 4英文合稿 → 中文翻译/术语核查/润色 → PDF+DOCX | ✅ 可用 |
| `/dr-glossary [slug]` | 术语表事实核查 | ✅ 可用 |
| `/dr-status [slug]` | 查看进度 | ✅ 可用 |
### 典型 MVP 流程
### 典型流程
```
1. /dr-init GLP-1 减重药物市场
→ dr-plan 向你提 6-8 个访谈问题(研究类型、受众、时间范围等)
→ dr-plan 向你提 8 个访谈问题(研究类型、受众、时间范围等)
→ 你回答后,生成 projects/glp1-obesity-market-2026/manifest.json
2. /dr-frame
@@ -135,9 +136,45 @@ source scripts/activate.sh
3. 你审核框架,或提修改意见,或直接确认
→ 确认后,manifest.phase1.approved = true
4. (后续)/dr-research 触发 Phase 2 深研 — 目前未实现
4. /dr-research
→ dr-pm 分批并行调度 dr-analyst 深研
→ dr-verifier 做反方验证
→ 产出 phase2/drafts、evidence、sources.jsonl
5. /dr-review
→ dr-chief-editor 通读审校,产出 phase3/critique.md
6. /dr-finalize
→ dr-editor-in-chief 合并英文终稿
→ Python 脚本执行 translate → glossary → apply_glossary → polish → build_report
→ 产出 final_zh_polished.md、PDF、DOCX
```
### Phase 4 Python 流水线
Phase 4 已不再依赖单个 LLM agent 一次性翻译整篇报告,而是由 Python 控制切块、并发、重试与断点续传:
```bash
uv run python scripts/translate.py <slug> --workers 4
uv run python scripts/build_glossary.py <slug> --workers 4
uv run python scripts/apply_glossary.py <slug> --input phase4/final_zh.md --dry-run
uv run python scripts/apply_glossary.py <slug> --input phase4/final_zh.md
uv run python scripts/polish.py <slug> --workers 4
uv run python scripts/build_report.py <slug>
```
网络不稳或 API 限流时,把 `--workers` 降到 `3``1` 即可断点续跑。
### 多平台兼容
- OpenCode:主适配器,使用 `.opencode/agents``.opencode/commands`
- Codex:第一阶段兼容,使用 `AGENTS.md` + Python 脚本链路,详见 `docs/codex-usage.md`
- Gemini CLI / Claude Code:暂不做强适配,后续以同一套脚本与配置为基础扩展。
模型与搜索 API 选择见:
- `docs/model-playbook.md`
- `docs/search-playbook.md`
---
## 项目结构
+62
View File
@@ -0,0 +1,62 @@
profiles:
premium:
description: "Highest quality for formal deliverables."
roles:
dr_plan: "zenmux-anthropic/claude-opus-4-7"
dr_pm: "zenmux-anthropic/claude-sonnet-4-6"
dr_analyst: "zenmux-anthropic/claude-sonnet-4-6"
dr_chief_editor: "zenmux/google/gemini-3.1-pro-preview"
dr_editor_in_chief: "zenmux-anthropic/claude-opus-4-7"
dr_verifier: "zenmux/openai/gpt-5.4"
translate: "anthropic/claude-sonnet-4.6"
polish: "anthropic/claude-sonnet-4.6"
glossary: "anthropic/claude-haiku-4.5"
balanced:
description: "Default profile for most reports."
roles:
dr_plan: "zenmux-anthropic/claude-opus-4-7"
dr_pm: "zenmux-anthropic/claude-sonnet-4-6"
dr_analyst: "zenmux-anthropic/claude-sonnet-4-6"
dr_chief_editor: "zenmux/google/gemini-3.1-pro-preview"
dr_editor_in_chief: "zenmux-anthropic/claude-opus-4-7"
dr_verifier: "zenmux/qwen/qwen3.6-plus"
translate: "anthropic/claude-sonnet-4.6"
polish: "anthropic/claude-sonnet-4.6"
glossary: "anthropic/claude-haiku-4.5"
budget:
description: "Lower-cost exploratory profile; requires stronger review."
roles:
dr_plan: "zenmux/qwen/qwen3.6-plus"
dr_pm: "zenmux/qwen/qwen3.6-plus"
dr_analyst: "zenmux/deepseek/deepseek-v3.2"
dr_chief_editor: "zenmux/google/gemini-2.5-pro"
dr_editor_in_chief: "zenmux-anthropic/claude-sonnet-4-6"
dr_verifier: "zenmux/minimax/minimax-m2.7"
translate: "anthropic/claude-haiku-4.5"
polish: "anthropic/claude-haiku-4.5"
glossary: "anthropic/claude-haiku-4.5"
cn_heavy:
description: "China-market-heavy research with more Chinese-source verification."
roles:
dr_plan: "zenmux-anthropic/claude-opus-4-7"
dr_pm: "zenmux-anthropic/claude-sonnet-4-6"
dr_analyst: "zenmux-anthropic/claude-sonnet-4-6"
dr_chief_editor: "zenmux/google/gemini-3.1-pro-preview"
dr_editor_in_chief: "zenmux-anthropic/claude-opus-4-7"
dr_verifier: "zenmux/qwen/qwen3.6-plus"
translate: "anthropic/claude-sonnet-4.6"
polish: "anthropic/claude-sonnet-4.6"
glossary: "anthropic/claude-haiku-4.5"
verifier:
description: "Cross-model verification profile; avoid Claude for verifier."
roles:
dr_verifier_primary: "zenmux/openai/gpt-5.4"
dr_verifier_backup_a: "zenmux/qwen/qwen3.6-plus"
dr_verifier_backup_b: "zenmux/minimax/minimax-m2.7"
dr_verifier_backup_c: "zenmux/moonshotai/kimi-k2.5"
defaults:
profile: "balanced"
translate_workers: 4
polish_workers: 4
glossary_workers: 4
+60
View File
@@ -0,0 +1,60 @@
profiles:
biomed_literature:
description: "Mechanism, clinical, systematic review, and biomedical evidence."
order:
- "pubmed_ncbi"
- "clinicaltrials"
- "fda_ema_nmpa"
- "serper_scholar"
- "tavily_exa_gap_fill"
patent_heavy:
description: "IP landscape, freedom-to-operate, and process-route research."
order:
- "google_patents_via_serper"
- "uspto_epo_cnipa"
- "company_disclosures"
- "exa_tavily_family_discovery"
china_market:
description: "China regulatory, company, supply-chain, and market research."
order:
- "nmpa_cde"
- "exchange_disclosures"
- "serper_brave_chinese"
- "exa_company_pages"
- "tavily_gap_fill"
investment:
description: "Public-company, market-size, and transaction-oriented research."
order:
- "sec_exchange_filings"
- "consulting_and_database_reports"
- "company_announcements"
- "serper_news"
- "tavily_gap_fill"
apis:
tavily:
best_for: ["initial_scan", "gap_fill", "llm_friendly_snippets"]
evidence_role: "discovery_only_unless_original_source"
exa:
best_for: ["company_pages", "terminology_check", "long_tail_professional_pages"]
evidence_role: "discovery_or_secondary"
brave:
best_for: ["cross_check", "counter_evidence", "mixed_language_search"]
evidence_role: "discovery_only_unless_original_source"
serper:
best_for: ["google_scholar", "google_patents", "news"]
evidence_role: "discovery_or_secondary"
pubmed_ncbi:
best_for: ["tier1_literature", "systematic_reviews", "meta_analysis"]
evidence_role: "tier1_or_tier2"
clinicaltrials:
best_for: ["trial_registration", "pipeline_status", "endpoint_design"]
evidence_role: "tier1"
regulatory:
best_for: ["approval_status", "label", "safety", "review_documents"]
evidence_role: "tier1"
defaults:
glossary_workers: 4
unstable_network_workers: 3
trace_back_to_tier_1_2: true
+85
View File
@@ -0,0 +1,85 @@
# Codex Usage
> v0.9 的 Codex 兼容是第一阶段:不复刻 OpenCode subagent,而是沿用 AGENTS.md 的研究规则和 Python 脚本流水线。Codex 负责规划、审阅、修补、执行脚本;确定性编排交给脚本。
## Setup
```bash
cd /Users/tankai/Documents/Projects/deep_research
source scripts/activate.sh
```
Codex 会读取项目根目录的 `AGENTS.md`,研究方法、信源分级、输出规范仍与 OpenCode 一致。
## Recommended Codex Workflow
### Phase 1-3
OpenCode 仍是主入口,因为 `.opencode/commands` 与 subagent 调度已经稳定:
```bash
opencode
/dr-init <topic>
/dr-frame <slug>
/dr-research <slug>
/dr-review <slug>
```
Codex 可用于:
- 审阅 `framework.md``critique.md`
- 修正文档、脚本和配置
- 运行静态检查、引用检查和出稿脚本
- 对某个章节做人工式复核建议
### Phase 4
Codex 可直接运行 Python 化流水线:
```bash
uv run python scripts/translate.py <slug> --workers 4
uv run python scripts/build_glossary.py <slug> --workers 4
uv run python scripts/apply_glossary.py <slug> --input phase4/final_zh.md --dry-run
uv run python scripts/apply_glossary.py <slug> --input phase4/final_zh.md
uv run python scripts/polish.py <slug> --workers 4
uv run python scripts/build_report.py <slug>
```
网络不稳时:
```bash
uv run python scripts/translate.py <slug> --workers 1
uv run python scripts/polish.py <slug> --workers 1
uv run python scripts/build_glossary.py <slug> --workers 3
```
## Git Hygiene
本仓库常有大量 `projects/**` 研究产物处于修改状态。Codex 提交系统迭代时必须只 stage 相关系统文件:
```bash
git add README.md PLAN.md docs configs scripts .opencode/commands/dr-finalize.md
git diff --staged --name-only
```
提交前确认 staged 列表不包含:
- `projects/**`
- 已生成 PDF/DOCX/TXT
- 临时检查脚本或一次性研究产物
## What Codex Should Not Do Yet
- 不直接替代 OpenCode 的 `/dr-research` 多 agent 调度。
- 不在 Codex 内复刻 dr-analyst/dr-verifier 的长流程并行。
- 不把普通网页当作 Tier 1-2 证据。
- 不在未检查 staged 列表前 commit。
## Future Platform Adapters
后续 Gemini CLI / Claude Code 兼容应沿用同一原则:
1. 共享 `AGENTS.md` 方法论。
2. 共享 `scripts/` 的确定性流水线。
3. 平台只提供交互、审阅、调度或修补能力。
4. 不为每个平台复制一套研究逻辑。
+80
View File
@@ -0,0 +1,80 @@
# Model Playbook
> v0.9 起,本文件作为模型选择攻略本。`.opencode/opencode.json` 仍是 OpenCode 的模型白名单,`configs/model_profiles.yaml` 是跨平台策略参考。
## Profiles
### premium
用于高价值、长上下文、强叙事任务。成本最高,但最稳。
- dr-planClaude Opus 4.7cache
- dr-editor-in-chiefClaude Opus 4.7cache
- dr-pm / dr-analystClaude Sonnet 4.6cache
- dr-chief-editorGemini 3.1 Pro Preview
- dr-verifierGPT-5.4
- translate / polishClaude Sonnet 4.6
适用:投资报告、管理层决策报告、需要交付 PDF/DOCX 的正式项目。
### balanced
默认推荐。保留关键环节的强模型,把机械任务交给更便宜模型。
- dr-planClaude Opus 4.7cache
- dr-editor-in-chiefClaude Opus 4.7cache
- dr-pm / dr-analystClaude Sonnet 4.6cache
- dr-searcherClaude Haiku 4.5
- dr-verifierQwen3.6 Plus 或 GPT-5.4
- translate / polishClaude Sonnet 4.6`--workers 4`
适用:大多数 10,000-35,000 中文字报告。
### budget
用于探索性研究或内部草稿。牺牲部分风格和深度,换取成本控制。
- dr-planClaude Sonnet 4.6 或 Qwen3.6 Plus
- dr-editor-in-chiefClaude Sonnet 4.6
- dr-analystQwen3.6 Plus / DeepSeek V3.2
- dr-verifierMiniMax M2.7 / Kimi K2.5
- translate / polishQwen3.6 Plus 或 Claude Haiku 4.5(需要人工抽检)
适用:快速扫盘、预研、内部初稿。不建议直接用于最终交付。
### cn-heavy
用于中国市场、NMPA、企业工商/中文媒体密集场景。
- dr-analystClaude Sonnet 4.6 + Qwen3.6 Plus 交叉
- dr-verifierQwen3.6 Plus
- dr-searcherHaiku + Serper/Brave 中文关键词
- translate / polishClaude Sonnet 4.6
适用:中国药企、国产替代、供应链、政策环境、中文专有名词较多的研究。
### verifier
用于交叉模型验证,避免同模型同偏差。
- 首选:GPT-5.4
- 备用:Qwen3.6 Plus
- 低成本:MiniMax M2.7 / Kimi K2.5
规则:dr-verifier 不使用 Claude 家族,除非用户明确接受同源偏差风险。
## Switching Rules
- 高层结构、最终合稿、Executive Summary:优先 Opus。
- 章节深研:Sonnet 是默认;预算敏感时用 Qwen/DeepSeek,但必须加强 verifier。
- 事实核查与术语核查:优先 Haiku + 搜索 API;低置信度条目人工复核。
- 翻译与润色:默认 Sonnet;并发提速优先调 `--workers`,不要盲目换便宜模型。
- 限流时:先降并发,再换模型。
## Operational Defaults
```bash
uv run python scripts/translate.py <slug> --workers 4 --model anthropic/claude-sonnet-4.6
uv run python scripts/polish.py <slug> --workers 4 --model anthropic/claude-sonnet-4.6
uv run python scripts/build_glossary.py <slug> --workers 4 --model anthropic/claude-haiku-4.5
```
+78
View File
@@ -0,0 +1,78 @@
# Search API Playbook
> v0.9 起,本文件作为搜索 API 选择攻略本。搜索返回本身多为发现入口,结论支撑仍以 AGENTS.md 的 Tier 1-2 信源为准。
## Search Sources
### Tavily
- 优点:LLM 友好,摘要质量稳定,适合快速发现方向。
- 用法:初扫、普通网页、报告线索、交叉补漏。
- 风险:不能把普通网页当结论支撑,必须追溯原始来源。
### Exa
- 优点:neural search,对官网、公司页、长尾专业内容召回好。
- 用法:术语核查、公司/产品名纠错、专业网页发现。
- 风险:macOS 代理环境容易 TLS EOF,项目内 `SearchClient` 已使用 `trust_env=False` 绕开系统代理。
### Brave
- 优点:独立搜索引擎,适合与 Tavily/Exa 交叉验证。
- 用法:Phase 1 初扫、反方证据、中文/英文混合搜索。
- 风险:结果质量波动,需要人工筛 Tier。
### Serper
- 优点:Google Search / Scholar / News 代理,免费额度较高。
- 用法:Google Scholar、Google Patents、新闻时效检索。
- 风险:专利是 `site:patents.google.com` 技巧,不等同官方专利库。
### PubMed / NCBI
- 优点:生物医药论文的一手入口。
- 用法:机制、临床、系统综述、meta 分析。
- 风险:无 API key 限流较低;摘要不足以替代全文判断。
### ClinicalTrials.gov / ChiCTR
- 优点:临床试验注册的一手来源。
- 用法:管线、适应症、试验阶段、终点设计、入组状态。
- 风险:注册信息不等于结果;需要结合论文、公司披露、监管文件。
### openFDA / FDA / EMA / NMPA
- 优点:监管公告与标签信息,Tier 1。
- 用法:批准状态、安全性、适应症、审评文件。
- 风险:不同监管地区口径不同,必须注明地区与日期。
### Patents
- 优点:IP 与工艺路线研究的核心证据。
- 用法:Google Patents、USPTO、EPO、CNIPA。
- 风险:专利文本难读,权利要求和实施例要分开判断。
## Recommended Profiles
### biomed_literature
PubMed / NCBI → ClinicalTrials → FDA/EMA/NMPA → Serper Scholar → Tavily/Exa 补漏。
### patent_heavy
Google Patents/Serper → USPTO/EPO/CNIPA → 公司年报/招股书 → Tavily/Exa 补同族专利线索。
### china_market
NMPA/CDE → 港交所/上交所/深交所披露 → 中文专业数据库/媒体 → Brave/Serper 中文搜索。
### investment
SEC/交易所披露 → Evaluate/IQVIA/咨询报告 → 公司公告 → 新闻仅作时效入口。
## Failure Handling
- 大量 SSL/TLS 错误:先把 workers 降到 3,再重跑。
- API 限流:保留缓存结果,断点续跑,不要强制 `--force`
- 搜索返回普通网页:只做线索,继续追原始论文、监管、专利或公司披露。
- 中英文译名冲突:写入 glossary,标 medium/low confidence,交人工复核。
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
"""根据 glossary 的核查结果,在 final_zh_polished.md 上做精确的文本替换。
"""根据 glossary 的核查结果,在 final_zh.md 上做精确的文本替换。
原理:
- build_glossary.py 会给每条术语标 `zh`(正确中文)和 `issue`(有发现问题)
@@ -178,7 +178,7 @@ def main() -> int:
parser = argparse.ArgumentParser(description="用 glossary 修正正文术语")
parser.add_argument("project", help="项目 slug 或路径")
parser.add_argument(
"--input", default="phase4/final_zh_polished.md",
"--input", default="phase4/final_zh.md",
help="待修正的 Markdown",
)
parser.add_argument(
+2 -2
View File
@@ -22,7 +22,7 @@
用法:
uv run python scripts/build_glossary.py <project_slug>
uv run python scripts/build_glossary.py <project_slug> --workers 6
uv run python scripts/build_glossary.py <project_slug> --workers 4
uv run python scripts/build_glossary.py <project_slug> --only "Mabwell,Maywavee"
uv run python scripts/build_glossary.py <project_slug> --force
@@ -184,7 +184,7 @@ def main() -> int:
)
parser.add_argument("--model", default=DEFAULT_MODEL, help="LLM 模型")
parser.add_argument(
"--workers", type=int, default=6, help="并发度(默认 6Exa 限速 5 QPS"
"--workers", type=int, default=4, help="并发度(默认 4;网络不稳时建议降到 3"
)
parser.add_argument(
"--force", action="store_true",
+8 -3
View File
@@ -15,6 +15,7 @@ from __future__ import annotations
import json
import os
import sys
import threading
import time
from dataclasses import dataclass, field
from pathlib import Path
@@ -111,6 +112,8 @@ class ZenMuxClient:
self.log_file = log_file
self.usage = UsageStats()
self._client = httpx.Client(timeout=timeout)
self._log_lock = threading.Lock()
self._usage_lock = threading.Lock()
def close(self) -> None:
self._client.close()
@@ -125,8 +128,9 @@ class ZenMuxClient:
if not self.log_file:
return
self.log_file.parent.mkdir(parents=True, exist_ok=True)
with self.log_file.open("a", encoding="utf-8") as f:
f.write(json.dumps(payload, ensure_ascii=False) + "\n")
with self._log_lock:
with self.log_file.open("a", encoding="utf-8") as f:
f.write(json.dumps(payload, ensure_ascii=False) + "\n")
def chat_complete(
self,
@@ -191,7 +195,8 @@ class ZenMuxClient:
except Exception as e:
raise ZenMuxError(f"invalid JSON from zenmux: {e}; body={resp.text[:500]}")
usage = data.get("usage", {}) or {}
self.usage.add(model, usage)
with self._usage_lock:
self.usage.add(model, usage)
content = ""
choices = data.get("choices") or []
if choices:
+75 -36
View File
@@ -19,6 +19,7 @@
from __future__ import annotations
import argparse
import concurrent.futures
import json
import sys
import time
@@ -141,6 +142,12 @@ def main() -> int:
default=None,
help="最多润色前 N 个未缓存的块(调试用)",
)
parser.add_argument(
"--workers",
type=int,
default=4,
help="并发润色 worker 数(默认 4;设为 1 回退串行)",
)
args = parser.parse_args()
load_secrets()
@@ -167,50 +174,82 @@ def main() -> int:
total_cn = sum(count_chinese_chars(b.content) for b in blocks)
print(f"Source: {src_path.relative_to(project_root)}")
print(f"Blocks: {len(blocks)} | total Chinese chars: {total_cn:,}")
print(f"Model: {args.model} | temperature: {args.temperature}")
workers = max(1, args.workers)
print(f"Model: {args.model} | temperature: {args.temperature} | workers: {workers}")
print()
start = time.time()
translated_this_run = 0
todo: list[MarkdownBlock] = []
cached_count = 0
notes_records: list[dict] = []
for b in blocks:
chunk_path = chunks_dir / f"{b.order:03d}-{b.anchor}.md"
if only_orders is not None and b.order not in only_orders:
continue
if chunk_path.exists() and not args.force:
cached_count += 1
print(f" [ok ] #{b.order:03d} {b.short_title} (cached)")
continue
if args.limit is not None and len(todo) >= args.limit:
continue
todo.append(b)
if todo:
print(f"To polish this run: {len(todo)} blocks | cached: {cached_count}")
print()
with ZenMuxClient(log_file=log_file) as client:
for b in blocks:
def run_one(b: MarkdownBlock) -> tuple[MarkdownBlock, str, str, float]:
chunk_path = chunks_dir / f"{b.order:03d}-{b.anchor}.md"
if only_orders is not None and b.order not in only_orders:
continue
if chunk_path.exists() and not args.force:
print(f" [ok ] #{b.order:03d} {b.short_title} (cached)")
continue
if args.limit is not None and translated_this_run >= args.limit:
continue
label = f"#{b.order:03d} L{b.level} {count_chinese_chars(b.content):>4}{b.short_title}"
print(f" [... ] {label} ", end="", flush=True)
t0 = time.time()
try:
polished, notes = polish_block(
client,
b,
model=args.model,
system_prompt=system_prompt,
temperature=args.temperature,
)
except (ZenMuxError, RuntimeError) as e:
print(f"\n [FAIL] {label}\n {e}")
continue
elapsed = time.time() - t0
polished, notes = polish_block(
client,
b,
model=args.model,
system_prompt=system_prompt,
temperature=args.temperature,
)
chunk_path.write_text(polished + "\n", encoding="utf-8")
cn = count_chinese_chars(polished)
before_cn = count_chinese_chars(b.content)
delta = cn - before_cn
sign = "+" if delta >= 0 else ""
translated_this_run += 1
if notes:
notes_records.append(
{"order": b.order, "anchor": b.anchor, "title": b.short_title, "notes": notes}
)
print(f"\r [done] {label}{cn}字 ({sign}{delta}, {elapsed:4.1f}s)")
return b, polished, notes, time.time() - t0
if workers == 1:
for b in todo:
label = f"#{b.order:03d} L{b.level} {count_chinese_chars(b.content):>4}{b.short_title}"
print(f" [... ] {label} ", end="", flush=True)
try:
block, polished, notes, elapsed = run_one(b)
except (ZenMuxError, RuntimeError) as e:
print(f"\n [FAIL] {label}\n {e}")
continue
cn = count_chinese_chars(polished)
before_cn = count_chinese_chars(block.content)
delta = cn - before_cn
sign = "+" if delta >= 0 else ""
if notes:
notes_records.append(
{"order": block.order, "anchor": block.anchor, "title": block.short_title, "notes": notes}
)
print(f"\r [done] {label}{cn}字 ({sign}{delta}, {elapsed:4.1f}s)")
else:
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool:
futures = {pool.submit(run_one, b): b for b in todo}
for fut in concurrent.futures.as_completed(futures):
b = futures[fut]
label = f"#{b.order:03d} L{b.level} {count_chinese_chars(b.content):>4}{b.short_title}"
try:
block, polished, notes, elapsed = fut.result()
except (ZenMuxError, RuntimeError) as e:
print(f" [FAIL] {label}\n {e}")
continue
cn = count_chinese_chars(polished)
before_cn = count_chinese_chars(block.content)
delta = cn - before_cn
sign = "+" if delta >= 0 else ""
if notes:
notes_records.append(
{"order": block.order, "anchor": block.anchor, "title": block.short_title, "notes": notes}
)
print(f" [done] {label}{cn}字 ({sign}{delta}, {elapsed:4.1f}s)")
# 汇总
merged: list[str] = []
+90 -41
View File
@@ -6,12 +6,12 @@
# 或:
uv run python scripts/translate.py projects/dual-target-rnai-pipeline-2026
断点续传每块翻译完立即写入 `phase4/zh_chunks/<anchor>.md` 术语表 patch
断点续传每块翻译完立即写入 `phase4/zh_chunks/<anchor>.md`术语表 patch 在本轮结束后统一合并
重跑时已存在的块直接跳过只译缺的
设计要点
1. 切块按 H2 粒度单块一般 <600 英文词单次 API 调用远低于 Sonnet output token 上限
2. 术语表累积式更新每块调用传入当前已知术语译完回写 patch保证全文一致
2. 并发翻译使用稳定术语表快照译完后统一合并 glossary patch避免多线程写冲突
3. 失败不会污染最终产物块级文件独立可重跑汇总步骤独立
4. 日志完整每次 API 调用写 `phase4/logs/translate.jsonl`
"""
@@ -19,6 +19,7 @@
from __future__ import annotations
import argparse
import concurrent.futures
import json
import sys
import time
@@ -29,7 +30,6 @@ sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from scripts.lib.markdown_chunker import (
MarkdownBlock,
count_chinese_chars,
merge_blocks,
split_by_headers,
)
from scripts.lib.zenmux_client import ZenMuxClient, ZenMuxError, load_secrets
@@ -112,12 +112,19 @@ def save_glossary(path: Path, glossary: dict[str, str]) -> None:
)
def _glossary_value(value) -> str:
if isinstance(value, dict):
return str(value.get("zh") or value.get("中文") or "")
return str(value)
def build_user_prompt(block: MarkdownBlock, glossary: dict[str, str]) -> str:
glossary_hint = (
"\n".join(f"{en} || {zh}" for en, zh in sorted(glossary.items()))
if glossary
else "(none yet)"
glossary_lines = (
f"{en} || {_glossary_value(zh)}"
for en, zh in sorted(glossary.items())
if _glossary_value(zh)
)
glossary_hint = "\n".join(glossary_lines) or "(none yet)"
level_hint = (
f"H{block.level}" if block.level >= 1 else "frontmatter (no heading)"
)
@@ -195,6 +202,12 @@ def main() -> int:
default=None,
help="最多翻译前 N 个未缓存的块(调试用)",
)
parser.add_argument(
"--workers",
type=int,
default=4,
help="并发翻译 worker 数(默认 4;设为 1 回退串行)",
)
args = parser.parse_args()
load_secrets()
@@ -223,47 +236,83 @@ def main() -> int:
print(f"Source: {src_path.relative_to(project_root)}")
print(f"Blocks: {len(blocks)} | total English words: {total_en_words:,}")
print(f"Glossary loaded: {len(glossary)} terms")
print(f"Model: {args.model} | temperature: {args.temperature}")
workers = max(1, args.workers)
print(f"Model: {args.model} | temperature: {args.temperature} | workers: {workers}")
print()
start = time.time()
translated_this_run = 0
todo: list[MarkdownBlock] = []
cached_count = 0
for b in blocks:
chunk_path = chunks_dir / f"{b.order:03d}-{b.anchor}.md"
if only_orders is not None and b.order not in only_orders:
continue
if chunk_path.exists() and not args.force:
cached_count += 1
print(f" [ok ] #{b.order:03d} {b.short_title} (cached)")
continue
if args.limit is not None and len(todo) >= args.limit:
continue
todo.append(b)
if todo:
print(f"To translate this run: {len(todo)} blocks | cached: {cached_count}")
print()
successful_patches: list[dict[str, str]] = []
with ZenMuxClient(log_file=log_file) as client:
for b in blocks:
def run_one(b: MarkdownBlock) -> tuple[MarkdownBlock, str, dict[str, str], float]:
chunk_path = chunks_dir / f"{b.order:03d}-{b.anchor}.md"
if only_orders is not None and b.order not in only_orders:
continue
if chunk_path.exists() and not args.force:
print(f" [ok ] #{b.order:03d} {b.short_title} (cached)")
continue
if args.limit is not None and translated_this_run >= args.limit:
continue
label = f"#{b.order:03d} L{b.level} {b.word_count:>4}w {b.short_title}"
print(f" [... ] {label} ", end="", flush=True)
t0 = time.time()
try:
translation, patch = translate_block(
client,
b,
glossary,
model=args.model,
system_prompt=system_prompt,
temperature=args.temperature,
)
except (ZenMuxError, RuntimeError) as e:
print(f"\n [FAIL] {label}\n {e}")
continue
elapsed = time.time() - t0
translation, patch = translate_block(
client,
b,
glossary,
model=args.model,
system_prompt=system_prompt,
temperature=args.temperature,
)
chunk_path.write_text(translation + "\n", encoding="utf-8")
if patch:
for k, v in patch.items():
glossary.setdefault(k, v)
save_glossary(glossary_path, glossary)
cn = count_chinese_chars(translation)
translated_this_run += 1
print(f"\r [done] {label}{cn:>4}字 ({elapsed:4.1f}s, +{len(patch)} terms)")
return b, translation, patch, time.time() - t0
if workers == 1:
for b in todo:
label = f"#{b.order:03d} L{b.level} {b.word_count:>4}w {b.short_title}"
print(f" [... ] {label} ", end="", flush=True)
try:
_block, translation, patch, elapsed = run_one(b)
except (ZenMuxError, RuntimeError) as e:
print(f"\n [FAIL] {label}\n {e}")
continue
successful_patches.append(patch)
cn = count_chinese_chars(translation)
print(f"\r [done] {label}{cn:>4}字 ({elapsed:4.1f}s, +{len(patch)} terms)")
else:
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool:
futures = {pool.submit(run_one, b): b for b in todo}
for fut in concurrent.futures.as_completed(futures):
b = futures[fut]
label = f"#{b.order:03d} L{b.level} {b.word_count:>4}w {b.short_title}"
try:
_block, translation, patch, elapsed = fut.result()
except (ZenMuxError, RuntimeError) as e:
print(f" [FAIL] {label}\n {e}")
continue
successful_patches.append(patch)
cn = count_chinese_chars(translation)
print(f" [done] {label}{cn:>4}字 ({elapsed:4.1f}s, +{len(patch)} terms)")
glossary_conflicts: list[tuple[str, str, str]] = []
for patch in successful_patches:
for k, v in patch.items():
if k not in glossary:
glossary[k] = v
elif _glossary_value(glossary[k]) != v:
glossary_conflicts.append((k, _glossary_value(glossary[k]), v))
if successful_patches:
save_glossary(glossary_path, glossary)
if glossary_conflicts:
print(f" [WARN] glossary patch 冲突 {len(glossary_conflicts)} 条,保留既有译法")
# 汇总:按 order 拼接所有 chunk
merged: list[str] = []