v0.9: parallelize phase4 and add model/search playbooks

This commit is contained in:
kai
2026-04-24 10:19:48 +08:00
parent c444007f04
commit c93fce5078
13 changed files with 626 additions and 100 deletions
+4 -2
View File
@@ -62,13 +62,15 @@ uv run python scripts/build_glossary.py <slug> --workers 4
```bash ```bash
# 先预览 # 先预览
uv run python scripts/apply_glossary.py <slug> --dry-run uv run python scripts/apply_glossary.py <slug> --input phase4/final_zh.md --dry-run
# 确认无误后应用 # 确认无误后应用
uv run python scripts/apply_glossary.py <slug> uv run python scripts/apply_glossary.py <slug> --input phase4/final_zh.md
``` ```
这会把 glossary 中发现的拼写错误 / 错译直接替换进 `final_zh.md` 这会把 glossary 中发现的拼写错误 / 错译直接替换进 `final_zh.md`
如润色后仍需二次复核,可手动对 `final_zh_polished.md` 再运行一次:
`uv run python scripts/apply_glossary.py <slug> --input phase4/final_zh_polished.md --dry-run`
## Step 6: 润色(Python 脚本) ## Step 6: 润色(Python 脚本)
+31 -2
View File
@@ -1,8 +1,8 @@
# Deep Research 系统方案(OpenCode 实现) # Deep Research 系统方案(OpenCode 实现)
> 本文件是整套方案的**单一真实源**,中断后续接时从此文件恢复上下文。 > 本文件是整套方案的**单一真实源**,中断后续接时从此文件恢复上下文。
> 最后更新:2026-04-20 > 最后更新:2026-04-24
> 实施阶段:路径 2 — 最小可用先行(MVP > 实施阶段:v0.9 — OpenCode 全流程 + Phase 4 并发化 + Codex 第一阶段兼容
--- ---
@@ -492,3 +492,32 @@ OpenCode 的坑:如果只是在主会话里装样子地写"让 X agent 做"
- 反驳证据段标题规范化(建议从"反方证据/Counter-Evidence"改为观点化标题如"另一种声音") - 反驳证据段标题规范化(建议从"反方证据/Counter-Evidence"改为观点化标题如"另一种声音")
- build_glossary 默认放到 Phase 2 阶段运行,在源头拦截错误 - build_glossary 默认放到 Phase 2 阶段运行,在源头拦截错误
- 提示 dr-analyst 加强对公司名/机构名的搜索验证流程 - 提示 dr-analyst 加强对公司名/机构名的搜索验证流程
- 2026-04-24 v0.9**Phase 4 并发提速 + 模型/搜索攻略本 + Codex 兼容**
**目标**:在不破坏 OpenCode 主流程的前提下,把 v0.6 Python 化 Phase 4 进一步提速,并补齐跨平台使用说明。OpenCode 仍是主适配器;Codex 第一阶段只复用 `AGENTS.md` 与 Python 脚本,不复刻 OpenCode subagent。
**Phase 4 并发化**
- `scripts/translate.py` 新增 `--workers`,默认 4;设为 1 时回退串行。
- 翻译阶段改为"稳定术语表快照 + 并发 chunk 翻译 + 事后统一合并 glossary patch",避免多线程同时写 `glossary.json`。
- `scripts/polish.py` 新增 `--workers`,默认 4;润色块彼此独立,按完成顺序写 chunk,最终按原始 order 合并。
- `scripts/lib/zenmux_client.py` 增加日志与 usage 聚合锁,避免并发 JSONL 日志交错或 token 统计竞争。
**流程修正**
- 修正 `apply_glossary.py` 默认输入,从 `phase4/final_zh_polished.md` 改为 `phase4/final_zh.md`。
- `/dr-finalize` 明确默认顺序:`translate.py → build_glossary.py → apply_glossary.py --input phase4/final_zh.md → polish.py → build_report.py`。
- 保留二次修正选项:润色后可手动对 `final_zh_polished.md` 再跑一次 `apply_glossary.py --input phase4/final_zh_polished.md --dry-run`。
**模型与搜索攻略本**
- 新增 `docs/model-playbook.md`:定义 premium / balanced / budget / cn-heavy / verifier 五套模型策略。
- 新增 `docs/search-playbook.md`:说明 Tavily / Exa / Brave / Serper / PubMed / ClinicalTrials / FDA/EMA/NMPA / Patents 的使用边界。
- 新增 `configs/model_profiles.yaml` 与 `configs/search_profiles.yaml`,作为跨平台、人类和 agent 共用的策略配置参考;当前不强制重构 `.opencode/agents` 自动读取。
**Codex 兼容**
- 新增 `docs/codex-usage.md`,说明 Codex 下如何遵循 `AGENTS.md`、运行 Phase 4 Python 流水线、检查 git staging,避免误提交 `projects/**` 研究产物。
- Codex v1 定位为"审阅/规划/修补/执行脚本";确定性编排继续放在 Python 脚本,OpenCode subagent 调度暂不移植。
**Git 管理要求**
- 本轮迭代应在独立分支推送到 Gitea。
- 提交范围仅限系统文件和文档:`README.md`、`PLAN.md`、`scripts/**`、`docs/**`、`configs/**`、必要的 `.opencode/commands/**`。
- 不提交 `projects/**`、生成的 PDF/DOCX/TXT、一次性研究产物或本地临时脚本。
+49 -12
View File
@@ -2,8 +2,8 @@
> 生物医药行业的 AI 驱动深度研究流水线。基于 OpenCode 多 agent 协作,以麦肯锡/德勤式方法论产出专业级研究报告(PDF + DOCX)。 > 生物医药行业的 AI 驱动深度研究流水线。基于 OpenCode 多 agent 协作,以麦肯锡/德勤式方法论产出专业级研究报告(PDF + DOCX)。
**当前状态**MVP(路径 2 — 最小可用先行),仅实现 Phase 1 能力 **当前状态**v0.9 核心迭代。OpenCode 全流程可用(Phase 1-4),Phase 4 已切换为 Python 脚本化流水线,并新增 Codex 第一阶段兼容说明
详见 `PLAN.md` 了解完整方案与迭代路径。 详见 `PLAN.md` 了解完整方案、版本记录与迭代路径。
--- ---
@@ -108,22 +108,23 @@ source scripts/activate.sh
--- ---
## MVP 可用命令 ## 可用命令
| 命令 | 功能 | 状态 | | 命令 | 功能 | 状态 |
|---|---|---| |---|---|---|
| `/dr-init <主题>` | 初始化新研究,启动访谈 | ✅ MVP | | `/dr-init <主题>` | 初始化新研究,启动访谈 | ✅ 可用 |
| `/dr-frame [slug]` | Phase 1:生成 8-15 章研究框架 | ✅ MVP | | `/dr-frame [slug]` | Phase 1:生成 8-15 章双语研究框架 | ✅ 可用 |
| `/dr-research` | Phase 2:深度研究(并行) | ⏳ 下一阶段 | | `/dr-research [slug]` | Phase 2并行深度研究 | ✅ 可用 |
| `/dr-review` | Phase 3:总编审校 | ⏳ 下一阶段 | | `/dr-review [slug]` | Phase 3:总编审校 | ✅ 可用 |
| `/dr-finalize` | Phase 4成稿 PDF+DOCX | ⏳ 下一阶段 | | `/dr-finalize [slug]` | Phase 4英文合稿 → 中文翻译/术语核查/润色 → PDF+DOCX | ✅ 可用 |
| `/dr-status` | 查看进度 | ⏳ 下一阶段 | | `/dr-glossary [slug]` | 术语表事实核查 | ✅ 可用 |
| `/dr-status [slug]` | 查看进度 | ✅ 可用 |
### 典型 MVP 流程 ### 典型流程
``` ```
1. /dr-init GLP-1 减重药物市场 1. /dr-init GLP-1 减重药物市场
→ dr-plan 向你提 6-8 个访谈问题(研究类型、受众、时间范围等) → dr-plan 向你提 8 个访谈问题(研究类型、受众、时间范围等)
→ 你回答后,生成 projects/glp1-obesity-market-2026/manifest.json → 你回答后,生成 projects/glp1-obesity-market-2026/manifest.json
2. /dr-frame 2. /dr-frame
@@ -135,9 +136,45 @@ source scripts/activate.sh
3. 你审核框架,或提修改意见,或直接确认 3. 你审核框架,或提修改意见,或直接确认
→ 确认后,manifest.phase1.approved = true → 确认后,manifest.phase1.approved = true
4. (后续)/dr-research 触发 Phase 2 深研 — 目前未实现 4. /dr-research
→ dr-pm 分批并行调度 dr-analyst 深研
→ dr-verifier 做反方验证
→ 产出 phase2/drafts、evidence、sources.jsonl
5. /dr-review
→ dr-chief-editor 通读审校,产出 phase3/critique.md
6. /dr-finalize
→ dr-editor-in-chief 合并英文终稿
→ Python 脚本执行 translate → glossary → apply_glossary → polish → build_report
→ 产出 final_zh_polished.md、PDF、DOCX
``` ```
### Phase 4 Python 流水线
Phase 4 已不再依赖单个 LLM agent 一次性翻译整篇报告,而是由 Python 控制切块、并发、重试与断点续传:
```bash
uv run python scripts/translate.py <slug> --workers 4
uv run python scripts/build_glossary.py <slug> --workers 4
uv run python scripts/apply_glossary.py <slug> --input phase4/final_zh.md --dry-run
uv run python scripts/apply_glossary.py <slug> --input phase4/final_zh.md
uv run python scripts/polish.py <slug> --workers 4
uv run python scripts/build_report.py <slug>
```
网络不稳或 API 限流时,把 `--workers` 降到 `3``1` 即可断点续跑。
### 多平台兼容
- OpenCode:主适配器,使用 `.opencode/agents``.opencode/commands`
- Codex:第一阶段兼容,使用 `AGENTS.md` + Python 脚本链路,详见 `docs/codex-usage.md`
- Gemini CLI / Claude Code:暂不做强适配,后续以同一套脚本与配置为基础扩展。
模型与搜索 API 选择见:
- `docs/model-playbook.md`
- `docs/search-playbook.md`
--- ---
## 项目结构 ## 项目结构
+62
View File
@@ -0,0 +1,62 @@
profiles:
premium:
description: "Highest quality for formal deliverables."
roles:
dr_plan: "zenmux-anthropic/claude-opus-4-7"
dr_pm: "zenmux-anthropic/claude-sonnet-4-6"
dr_analyst: "zenmux-anthropic/claude-sonnet-4-6"
dr_chief_editor: "zenmux/google/gemini-3.1-pro-preview"
dr_editor_in_chief: "zenmux-anthropic/claude-opus-4-7"
dr_verifier: "zenmux/openai/gpt-5.4"
translate: "anthropic/claude-sonnet-4.6"
polish: "anthropic/claude-sonnet-4.6"
glossary: "anthropic/claude-haiku-4.5"
balanced:
description: "Default profile for most reports."
roles:
dr_plan: "zenmux-anthropic/claude-opus-4-7"
dr_pm: "zenmux-anthropic/claude-sonnet-4-6"
dr_analyst: "zenmux-anthropic/claude-sonnet-4-6"
dr_chief_editor: "zenmux/google/gemini-3.1-pro-preview"
dr_editor_in_chief: "zenmux-anthropic/claude-opus-4-7"
dr_verifier: "zenmux/qwen/qwen3.6-plus"
translate: "anthropic/claude-sonnet-4.6"
polish: "anthropic/claude-sonnet-4.6"
glossary: "anthropic/claude-haiku-4.5"
budget:
description: "Lower-cost exploratory profile; requires stronger review."
roles:
dr_plan: "zenmux/qwen/qwen3.6-plus"
dr_pm: "zenmux/qwen/qwen3.6-plus"
dr_analyst: "zenmux/deepseek/deepseek-v3.2"
dr_chief_editor: "zenmux/google/gemini-2.5-pro"
dr_editor_in_chief: "zenmux-anthropic/claude-sonnet-4-6"
dr_verifier: "zenmux/minimax/minimax-m2.7"
translate: "anthropic/claude-haiku-4.5"
polish: "anthropic/claude-haiku-4.5"
glossary: "anthropic/claude-haiku-4.5"
cn_heavy:
description: "China-market-heavy research with more Chinese-source verification."
roles:
dr_plan: "zenmux-anthropic/claude-opus-4-7"
dr_pm: "zenmux-anthropic/claude-sonnet-4-6"
dr_analyst: "zenmux-anthropic/claude-sonnet-4-6"
dr_chief_editor: "zenmux/google/gemini-3.1-pro-preview"
dr_editor_in_chief: "zenmux-anthropic/claude-opus-4-7"
dr_verifier: "zenmux/qwen/qwen3.6-plus"
translate: "anthropic/claude-sonnet-4.6"
polish: "anthropic/claude-sonnet-4.6"
glossary: "anthropic/claude-haiku-4.5"
verifier:
description: "Cross-model verification profile; avoid Claude for verifier."
roles:
dr_verifier_primary: "zenmux/openai/gpt-5.4"
dr_verifier_backup_a: "zenmux/qwen/qwen3.6-plus"
dr_verifier_backup_b: "zenmux/minimax/minimax-m2.7"
dr_verifier_backup_c: "zenmux/moonshotai/kimi-k2.5"
defaults:
profile: "balanced"
translate_workers: 4
polish_workers: 4
glossary_workers: 4
+60
View File
@@ -0,0 +1,60 @@
profiles:
biomed_literature:
description: "Mechanism, clinical, systematic review, and biomedical evidence."
order:
- "pubmed_ncbi"
- "clinicaltrials"
- "fda_ema_nmpa"
- "serper_scholar"
- "tavily_exa_gap_fill"
patent_heavy:
description: "IP landscape, freedom-to-operate, and process-route research."
order:
- "google_patents_via_serper"
- "uspto_epo_cnipa"
- "company_disclosures"
- "exa_tavily_family_discovery"
china_market:
description: "China regulatory, company, supply-chain, and market research."
order:
- "nmpa_cde"
- "exchange_disclosures"
- "serper_brave_chinese"
- "exa_company_pages"
- "tavily_gap_fill"
investment:
description: "Public-company, market-size, and transaction-oriented research."
order:
- "sec_exchange_filings"
- "consulting_and_database_reports"
- "company_announcements"
- "serper_news"
- "tavily_gap_fill"
apis:
tavily:
best_for: ["initial_scan", "gap_fill", "llm_friendly_snippets"]
evidence_role: "discovery_only_unless_original_source"
exa:
best_for: ["company_pages", "terminology_check", "long_tail_professional_pages"]
evidence_role: "discovery_or_secondary"
brave:
best_for: ["cross_check", "counter_evidence", "mixed_language_search"]
evidence_role: "discovery_only_unless_original_source"
serper:
best_for: ["google_scholar", "google_patents", "news"]
evidence_role: "discovery_or_secondary"
pubmed_ncbi:
best_for: ["tier1_literature", "systematic_reviews", "meta_analysis"]
evidence_role: "tier1_or_tier2"
clinicaltrials:
best_for: ["trial_registration", "pipeline_status", "endpoint_design"]
evidence_role: "tier1"
regulatory:
best_for: ["approval_status", "label", "safety", "review_documents"]
evidence_role: "tier1"
defaults:
glossary_workers: 4
unstable_network_workers: 3
trace_back_to_tier_1_2: true
+85
View File
@@ -0,0 +1,85 @@
# Codex Usage
> v0.9 的 Codex 兼容是第一阶段:不复刻 OpenCode subagent,而是沿用 AGENTS.md 的研究规则和 Python 脚本流水线。Codex 负责规划、审阅、修补、执行脚本;确定性编排交给脚本。
## Setup
```bash
cd /Users/tankai/Documents/Projects/deep_research
source scripts/activate.sh
```
Codex 会读取项目根目录的 `AGENTS.md`,研究方法、信源分级、输出规范仍与 OpenCode 一致。
## Recommended Codex Workflow
### Phase 1-3
OpenCode 仍是主入口,因为 `.opencode/commands` 与 subagent 调度已经稳定:
```bash
opencode
/dr-init <topic>
/dr-frame <slug>
/dr-research <slug>
/dr-review <slug>
```
Codex 可用于:
- 审阅 `framework.md``critique.md`
- 修正文档、脚本和配置
- 运行静态检查、引用检查和出稿脚本
- 对某个章节做人工式复核建议
### Phase 4
Codex 可直接运行 Python 化流水线:
```bash
uv run python scripts/translate.py <slug> --workers 4
uv run python scripts/build_glossary.py <slug> --workers 4
uv run python scripts/apply_glossary.py <slug> --input phase4/final_zh.md --dry-run
uv run python scripts/apply_glossary.py <slug> --input phase4/final_zh.md
uv run python scripts/polish.py <slug> --workers 4
uv run python scripts/build_report.py <slug>
```
网络不稳时:
```bash
uv run python scripts/translate.py <slug> --workers 1
uv run python scripts/polish.py <slug> --workers 1
uv run python scripts/build_glossary.py <slug> --workers 3
```
## Git Hygiene
本仓库常有大量 `projects/**` 研究产物处于修改状态。Codex 提交系统迭代时必须只 stage 相关系统文件:
```bash
git add README.md PLAN.md docs configs scripts .opencode/commands/dr-finalize.md
git diff --staged --name-only
```
提交前确认 staged 列表不包含:
- `projects/**`
- 已生成 PDF/DOCX/TXT
- 临时检查脚本或一次性研究产物
## What Codex Should Not Do Yet
- 不直接替代 OpenCode 的 `/dr-research` 多 agent 调度。
- 不在 Codex 内复刻 dr-analyst/dr-verifier 的长流程并行。
- 不把普通网页当作 Tier 1-2 证据。
- 不在未检查 staged 列表前 commit。
## Future Platform Adapters
后续 Gemini CLI / Claude Code 兼容应沿用同一原则:
1. 共享 `AGENTS.md` 方法论。
2. 共享 `scripts/` 的确定性流水线。
3. 平台只提供交互、审阅、调度或修补能力。
4. 不为每个平台复制一套研究逻辑。
+80
View File
@@ -0,0 +1,80 @@
# Model Playbook
> v0.9 起,本文件作为模型选择攻略本。`.opencode/opencode.json` 仍是 OpenCode 的模型白名单,`configs/model_profiles.yaml` 是跨平台策略参考。
## Profiles
### premium
用于高价值、长上下文、强叙事任务。成本最高,但最稳。
- dr-planClaude Opus 4.7cache
- dr-editor-in-chiefClaude Opus 4.7cache
- dr-pm / dr-analystClaude Sonnet 4.6cache
- dr-chief-editorGemini 3.1 Pro Preview
- dr-verifierGPT-5.4
- translate / polishClaude Sonnet 4.6
适用:投资报告、管理层决策报告、需要交付 PDF/DOCX 的正式项目。
### balanced
默认推荐。保留关键环节的强模型,把机械任务交给更便宜模型。
- dr-planClaude Opus 4.7cache
- dr-editor-in-chiefClaude Opus 4.7cache
- dr-pm / dr-analystClaude Sonnet 4.6cache
- dr-searcherClaude Haiku 4.5
- dr-verifierQwen3.6 Plus 或 GPT-5.4
- translate / polishClaude Sonnet 4.6`--workers 4`
适用:大多数 10,000-35,000 中文字报告。
### budget
用于探索性研究或内部草稿。牺牲部分风格和深度,换取成本控制。
- dr-planClaude Sonnet 4.6 或 Qwen3.6 Plus
- dr-editor-in-chiefClaude Sonnet 4.6
- dr-analystQwen3.6 Plus / DeepSeek V3.2
- dr-verifierMiniMax M2.7 / Kimi K2.5
- translate / polishQwen3.6 Plus 或 Claude Haiku 4.5(需要人工抽检)
适用:快速扫盘、预研、内部初稿。不建议直接用于最终交付。
### cn-heavy
用于中国市场、NMPA、企业工商/中文媒体密集场景。
- dr-analystClaude Sonnet 4.6 + Qwen3.6 Plus 交叉
- dr-verifierQwen3.6 Plus
- dr-searcherHaiku + Serper/Brave 中文关键词
- translate / polishClaude Sonnet 4.6
适用:中国药企、国产替代、供应链、政策环境、中文专有名词较多的研究。
### verifier
用于交叉模型验证,避免同模型同偏差。
- 首选:GPT-5.4
- 备用:Qwen3.6 Plus
- 低成本:MiniMax M2.7 / Kimi K2.5
规则:dr-verifier 不使用 Claude 家族,除非用户明确接受同源偏差风险。
## Switching Rules
- 高层结构、最终合稿、Executive Summary:优先 Opus。
- 章节深研:Sonnet 是默认;预算敏感时用 Qwen/DeepSeek,但必须加强 verifier。
- 事实核查与术语核查:优先 Haiku + 搜索 API;低置信度条目人工复核。
- 翻译与润色:默认 Sonnet;并发提速优先调 `--workers`,不要盲目换便宜模型。
- 限流时:先降并发,再换模型。
## Operational Defaults
```bash
uv run python scripts/translate.py <slug> --workers 4 --model anthropic/claude-sonnet-4.6
uv run python scripts/polish.py <slug> --workers 4 --model anthropic/claude-sonnet-4.6
uv run python scripts/build_glossary.py <slug> --workers 4 --model anthropic/claude-haiku-4.5
```
+78
View File
@@ -0,0 +1,78 @@
# Search API Playbook
> v0.9 起,本文件作为搜索 API 选择攻略本。搜索返回本身多为发现入口,结论支撑仍以 AGENTS.md 的 Tier 1-2 信源为准。
## Search Sources
### Tavily
- 优点:LLM 友好,摘要质量稳定,适合快速发现方向。
- 用法:初扫、普通网页、报告线索、交叉补漏。
- 风险:不能把普通网页当结论支撑,必须追溯原始来源。
### Exa
- 优点:neural search,对官网、公司页、长尾专业内容召回好。
- 用法:术语核查、公司/产品名纠错、专业网页发现。
- 风险:macOS 代理环境容易 TLS EOF,项目内 `SearchClient` 已使用 `trust_env=False` 绕开系统代理。
### Brave
- 优点:独立搜索引擎,适合与 Tavily/Exa 交叉验证。
- 用法:Phase 1 初扫、反方证据、中文/英文混合搜索。
- 风险:结果质量波动,需要人工筛 Tier。
### Serper
- 优点:Google Search / Scholar / News 代理,免费额度较高。
- 用法:Google Scholar、Google Patents、新闻时效检索。
- 风险:专利是 `site:patents.google.com` 技巧,不等同官方专利库。
### PubMed / NCBI
- 优点:生物医药论文的一手入口。
- 用法:机制、临床、系统综述、meta 分析。
- 风险:无 API key 限流较低;摘要不足以替代全文判断。
### ClinicalTrials.gov / ChiCTR
- 优点:临床试验注册的一手来源。
- 用法:管线、适应症、试验阶段、终点设计、入组状态。
- 风险:注册信息不等于结果;需要结合论文、公司披露、监管文件。
### openFDA / FDA / EMA / NMPA
- 优点:监管公告与标签信息,Tier 1。
- 用法:批准状态、安全性、适应症、审评文件。
- 风险:不同监管地区口径不同,必须注明地区与日期。
### Patents
- 优点:IP 与工艺路线研究的核心证据。
- 用法:Google Patents、USPTO、EPO、CNIPA。
- 风险:专利文本难读,权利要求和实施例要分开判断。
## Recommended Profiles
### biomed_literature
PubMed / NCBI → ClinicalTrials → FDA/EMA/NMPA → Serper Scholar → Tavily/Exa 补漏。
### patent_heavy
Google Patents/Serper → USPTO/EPO/CNIPA → 公司年报/招股书 → Tavily/Exa 补同族专利线索。
### china_market
NMPA/CDE → 港交所/上交所/深交所披露 → 中文专业数据库/媒体 → Brave/Serper 中文搜索。
### investment
SEC/交易所披露 → Evaluate/IQVIA/咨询报告 → 公司公告 → 新闻仅作时效入口。
## Failure Handling
- 大量 SSL/TLS 错误:先把 workers 降到 3,再重跑。
- API 限流:保留缓存结果,断点续跑,不要强制 `--force`
- 搜索返回普通网页:只做线索,继续追原始论文、监管、专利或公司披露。
- 中英文译名冲突:写入 glossary,标 medium/low confidence,交人工复核。
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""根据 glossary 的核查结果,在 final_zh_polished.md 上做精确的文本替换。 """根据 glossary 的核查结果,在 final_zh.md 上做精确的文本替换。
原理: 原理:
- build_glossary.py 会给每条术语标 `zh`(正确中文)和 `issue`(有发现问题) - build_glossary.py 会给每条术语标 `zh`(正确中文)和 `issue`(有发现问题)
@@ -178,7 +178,7 @@ def main() -> int:
parser = argparse.ArgumentParser(description="用 glossary 修正正文术语") parser = argparse.ArgumentParser(description="用 glossary 修正正文术语")
parser.add_argument("project", help="项目 slug 或路径") parser.add_argument("project", help="项目 slug 或路径")
parser.add_argument( parser.add_argument(
"--input", default="phase4/final_zh_polished.md", "--input", default="phase4/final_zh.md",
help="待修正的 Markdown", help="待修正的 Markdown",
) )
parser.add_argument( parser.add_argument(
+2 -2
View File
@@ -22,7 +22,7 @@
用法 用法
uv run python scripts/build_glossary.py <project_slug> uv run python scripts/build_glossary.py <project_slug>
uv run python scripts/build_glossary.py <project_slug> --workers 6 uv run python scripts/build_glossary.py <project_slug> --workers 4
uv run python scripts/build_glossary.py <project_slug> --only "Mabwell,Maywavee" uv run python scripts/build_glossary.py <project_slug> --only "Mabwell,Maywavee"
uv run python scripts/build_glossary.py <project_slug> --force uv run python scripts/build_glossary.py <project_slug> --force
@@ -184,7 +184,7 @@ def main() -> int:
) )
parser.add_argument("--model", default=DEFAULT_MODEL, help="LLM 模型") parser.add_argument("--model", default=DEFAULT_MODEL, help="LLM 模型")
parser.add_argument( parser.add_argument(
"--workers", type=int, default=6, help="并发度(默认 6Exa 限速 5 QPS" "--workers", type=int, default=4, help="并发度(默认 4;网络不稳时建议降到 3"
) )
parser.add_argument( parser.add_argument(
"--force", action="store_true", "--force", action="store_true",
+8 -3
View File
@@ -15,6 +15,7 @@ from __future__ import annotations
import json import json
import os import os
import sys import sys
import threading
import time import time
from dataclasses import dataclass, field from dataclasses import dataclass, field
from pathlib import Path from pathlib import Path
@@ -111,6 +112,8 @@ class ZenMuxClient:
self.log_file = log_file self.log_file = log_file
self.usage = UsageStats() self.usage = UsageStats()
self._client = httpx.Client(timeout=timeout) self._client = httpx.Client(timeout=timeout)
self._log_lock = threading.Lock()
self._usage_lock = threading.Lock()
def close(self) -> None: def close(self) -> None:
self._client.close() self._client.close()
@@ -125,8 +128,9 @@ class ZenMuxClient:
if not self.log_file: if not self.log_file:
return return
self.log_file.parent.mkdir(parents=True, exist_ok=True) self.log_file.parent.mkdir(parents=True, exist_ok=True)
with self.log_file.open("a", encoding="utf-8") as f: with self._log_lock:
f.write(json.dumps(payload, ensure_ascii=False) + "\n") with self.log_file.open("a", encoding="utf-8") as f:
f.write(json.dumps(payload, ensure_ascii=False) + "\n")
def chat_complete( def chat_complete(
self, self,
@@ -191,7 +195,8 @@ class ZenMuxClient:
except Exception as e: except Exception as e:
raise ZenMuxError(f"invalid JSON from zenmux: {e}; body={resp.text[:500]}") raise ZenMuxError(f"invalid JSON from zenmux: {e}; body={resp.text[:500]}")
usage = data.get("usage", {}) or {} usage = data.get("usage", {}) or {}
self.usage.add(model, usage) with self._usage_lock:
self.usage.add(model, usage)
content = "" content = ""
choices = data.get("choices") or [] choices = data.get("choices") or []
if choices: if choices:
+75 -36
View File
@@ -19,6 +19,7 @@
from __future__ import annotations from __future__ import annotations
import argparse import argparse
import concurrent.futures
import json import json
import sys import sys
import time import time
@@ -141,6 +142,12 @@ def main() -> int:
default=None, default=None,
help="最多润色前 N 个未缓存的块(调试用)", help="最多润色前 N 个未缓存的块(调试用)",
) )
parser.add_argument(
"--workers",
type=int,
default=4,
help="并发润色 worker 数(默认 4;设为 1 回退串行)",
)
args = parser.parse_args() args = parser.parse_args()
load_secrets() load_secrets()
@@ -167,50 +174,82 @@ def main() -> int:
total_cn = sum(count_chinese_chars(b.content) for b in blocks) total_cn = sum(count_chinese_chars(b.content) for b in blocks)
print(f"Source: {src_path.relative_to(project_root)}") print(f"Source: {src_path.relative_to(project_root)}")
print(f"Blocks: {len(blocks)} | total Chinese chars: {total_cn:,}") print(f"Blocks: {len(blocks)} | total Chinese chars: {total_cn:,}")
print(f"Model: {args.model} | temperature: {args.temperature}") workers = max(1, args.workers)
print(f"Model: {args.model} | temperature: {args.temperature} | workers: {workers}")
print() print()
start = time.time() start = time.time()
translated_this_run = 0 todo: list[MarkdownBlock] = []
cached_count = 0
notes_records: list[dict] = [] notes_records: list[dict] = []
for b in blocks:
chunk_path = chunks_dir / f"{b.order:03d}-{b.anchor}.md"
if only_orders is not None and b.order not in only_orders:
continue
if chunk_path.exists() and not args.force:
cached_count += 1
print(f" [ok ] #{b.order:03d} {b.short_title} (cached)")
continue
if args.limit is not None and len(todo) >= args.limit:
continue
todo.append(b)
if todo:
print(f"To polish this run: {len(todo)} blocks | cached: {cached_count}")
print()
with ZenMuxClient(log_file=log_file) as client: with ZenMuxClient(log_file=log_file) as client:
for b in blocks: def run_one(b: MarkdownBlock) -> tuple[MarkdownBlock, str, str, float]:
chunk_path = chunks_dir / f"{b.order:03d}-{b.anchor}.md" chunk_path = chunks_dir / f"{b.order:03d}-{b.anchor}.md"
if only_orders is not None and b.order not in only_orders:
continue
if chunk_path.exists() and not args.force:
print(f" [ok ] #{b.order:03d} {b.short_title} (cached)")
continue
if args.limit is not None and translated_this_run >= args.limit:
continue
label = f"#{b.order:03d} L{b.level} {count_chinese_chars(b.content):>4}{b.short_title}"
print(f" [... ] {label} ", end="", flush=True)
t0 = time.time() t0 = time.time()
try: polished, notes = polish_block(
polished, notes = polish_block( client,
client, b,
b, model=args.model,
model=args.model, system_prompt=system_prompt,
system_prompt=system_prompt, temperature=args.temperature,
temperature=args.temperature, )
)
except (ZenMuxError, RuntimeError) as e:
print(f"\n [FAIL] {label}\n {e}")
continue
elapsed = time.time() - t0
chunk_path.write_text(polished + "\n", encoding="utf-8") chunk_path.write_text(polished + "\n", encoding="utf-8")
cn = count_chinese_chars(polished) return b, polished, notes, time.time() - t0
before_cn = count_chinese_chars(b.content)
delta = cn - before_cn if workers == 1:
sign = "+" if delta >= 0 else "" for b in todo:
translated_this_run += 1 label = f"#{b.order:03d} L{b.level} {count_chinese_chars(b.content):>4}{b.short_title}"
if notes: print(f" [... ] {label} ", end="", flush=True)
notes_records.append( try:
{"order": b.order, "anchor": b.anchor, "title": b.short_title, "notes": notes} block, polished, notes, elapsed = run_one(b)
) except (ZenMuxError, RuntimeError) as e:
print(f"\r [done] {label}{cn}字 ({sign}{delta}, {elapsed:4.1f}s)") print(f"\n [FAIL] {label}\n {e}")
continue
cn = count_chinese_chars(polished)
before_cn = count_chinese_chars(block.content)
delta = cn - before_cn
sign = "+" if delta >= 0 else ""
if notes:
notes_records.append(
{"order": block.order, "anchor": block.anchor, "title": block.short_title, "notes": notes}
)
print(f"\r [done] {label}{cn}字 ({sign}{delta}, {elapsed:4.1f}s)")
else:
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool:
futures = {pool.submit(run_one, b): b for b in todo}
for fut in concurrent.futures.as_completed(futures):
b = futures[fut]
label = f"#{b.order:03d} L{b.level} {count_chinese_chars(b.content):>4}{b.short_title}"
try:
block, polished, notes, elapsed = fut.result()
except (ZenMuxError, RuntimeError) as e:
print(f" [FAIL] {label}\n {e}")
continue
cn = count_chinese_chars(polished)
before_cn = count_chinese_chars(block.content)
delta = cn - before_cn
sign = "+" if delta >= 0 else ""
if notes:
notes_records.append(
{"order": block.order, "anchor": block.anchor, "title": block.short_title, "notes": notes}
)
print(f" [done] {label}{cn}字 ({sign}{delta}, {elapsed:4.1f}s)")
# 汇总 # 汇总
merged: list[str] = [] merged: list[str] = []
+90 -41
View File
@@ -6,12 +6,12 @@
# 或: # 或:
uv run python scripts/translate.py projects/dual-target-rnai-pipeline-2026 uv run python scripts/translate.py projects/dual-target-rnai-pipeline-2026
断点续传每块翻译完立即写入 `phase4/zh_chunks/<anchor>.md` 术语表 patch 断点续传每块翻译完立即写入 `phase4/zh_chunks/<anchor>.md`术语表 patch 在本轮结束后统一合并
重跑时已存在的块直接跳过只译缺的 重跑时已存在的块直接跳过只译缺的
设计要点 设计要点
1. 切块按 H2 粒度单块一般 <600 英文词单次 API 调用远低于 Sonnet output token 上限 1. 切块按 H2 粒度单块一般 <600 英文词单次 API 调用远低于 Sonnet output token 上限
2. 术语表累积式更新每块调用传入当前已知术语译完回写 patch保证全文一致 2. 并发翻译使用稳定术语表快照译完后统一合并 glossary patch避免多线程写冲突
3. 失败不会污染最终产物块级文件独立可重跑汇总步骤独立 3. 失败不会污染最终产物块级文件独立可重跑汇总步骤独立
4. 日志完整每次 API 调用写 `phase4/logs/translate.jsonl` 4. 日志完整每次 API 调用写 `phase4/logs/translate.jsonl`
""" """
@@ -19,6 +19,7 @@
from __future__ import annotations from __future__ import annotations
import argparse import argparse
import concurrent.futures
import json import json
import sys import sys
import time import time
@@ -29,7 +30,6 @@ sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from scripts.lib.markdown_chunker import ( from scripts.lib.markdown_chunker import (
MarkdownBlock, MarkdownBlock,
count_chinese_chars, count_chinese_chars,
merge_blocks,
split_by_headers, split_by_headers,
) )
from scripts.lib.zenmux_client import ZenMuxClient, ZenMuxError, load_secrets from scripts.lib.zenmux_client import ZenMuxClient, ZenMuxError, load_secrets
@@ -112,12 +112,19 @@ def save_glossary(path: Path, glossary: dict[str, str]) -> None:
) )
def _glossary_value(value) -> str:
if isinstance(value, dict):
return str(value.get("zh") or value.get("中文") or "")
return str(value)
def build_user_prompt(block: MarkdownBlock, glossary: dict[str, str]) -> str: def build_user_prompt(block: MarkdownBlock, glossary: dict[str, str]) -> str:
glossary_hint = ( glossary_lines = (
"\n".join(f"{en} || {zh}" for en, zh in sorted(glossary.items())) f"{en} || {_glossary_value(zh)}"
if glossary for en, zh in sorted(glossary.items())
else "(none yet)" if _glossary_value(zh)
) )
glossary_hint = "\n".join(glossary_lines) or "(none yet)"
level_hint = ( level_hint = (
f"H{block.level}" if block.level >= 1 else "frontmatter (no heading)" f"H{block.level}" if block.level >= 1 else "frontmatter (no heading)"
) )
@@ -195,6 +202,12 @@ def main() -> int:
default=None, default=None,
help="最多翻译前 N 个未缓存的块(调试用)", help="最多翻译前 N 个未缓存的块(调试用)",
) )
parser.add_argument(
"--workers",
type=int,
default=4,
help="并发翻译 worker 数(默认 4;设为 1 回退串行)",
)
args = parser.parse_args() args = parser.parse_args()
load_secrets() load_secrets()
@@ -223,47 +236,83 @@ def main() -> int:
print(f"Source: {src_path.relative_to(project_root)}") print(f"Source: {src_path.relative_to(project_root)}")
print(f"Blocks: {len(blocks)} | total English words: {total_en_words:,}") print(f"Blocks: {len(blocks)} | total English words: {total_en_words:,}")
print(f"Glossary loaded: {len(glossary)} terms") print(f"Glossary loaded: {len(glossary)} terms")
print(f"Model: {args.model} | temperature: {args.temperature}") workers = max(1, args.workers)
print(f"Model: {args.model} | temperature: {args.temperature} | workers: {workers}")
print() print()
start = time.time() start = time.time()
translated_this_run = 0 todo: list[MarkdownBlock] = []
cached_count = 0
for b in blocks:
chunk_path = chunks_dir / f"{b.order:03d}-{b.anchor}.md"
if only_orders is not None and b.order not in only_orders:
continue
if chunk_path.exists() and not args.force:
cached_count += 1
print(f" [ok ] #{b.order:03d} {b.short_title} (cached)")
continue
if args.limit is not None and len(todo) >= args.limit:
continue
todo.append(b)
if todo:
print(f"To translate this run: {len(todo)} blocks | cached: {cached_count}")
print()
successful_patches: list[dict[str, str]] = []
with ZenMuxClient(log_file=log_file) as client: with ZenMuxClient(log_file=log_file) as client:
for b in blocks: def run_one(b: MarkdownBlock) -> tuple[MarkdownBlock, str, dict[str, str], float]:
chunk_path = chunks_dir / f"{b.order:03d}-{b.anchor}.md" chunk_path = chunks_dir / f"{b.order:03d}-{b.anchor}.md"
if only_orders is not None and b.order not in only_orders:
continue
if chunk_path.exists() and not args.force:
print(f" [ok ] #{b.order:03d} {b.short_title} (cached)")
continue
if args.limit is not None and translated_this_run >= args.limit:
continue
label = f"#{b.order:03d} L{b.level} {b.word_count:>4}w {b.short_title}"
print(f" [... ] {label} ", end="", flush=True)
t0 = time.time() t0 = time.time()
try: translation, patch = translate_block(
translation, patch = translate_block( client,
client, b,
b, glossary,
glossary, model=args.model,
model=args.model, system_prompt=system_prompt,
system_prompt=system_prompt, temperature=args.temperature,
temperature=args.temperature, )
)
except (ZenMuxError, RuntimeError) as e:
print(f"\n [FAIL] {label}\n {e}")
continue
elapsed = time.time() - t0
chunk_path.write_text(translation + "\n", encoding="utf-8") chunk_path.write_text(translation + "\n", encoding="utf-8")
if patch: return b, translation, patch, time.time() - t0
for k, v in patch.items():
glossary.setdefault(k, v) if workers == 1:
save_glossary(glossary_path, glossary) for b in todo:
cn = count_chinese_chars(translation) label = f"#{b.order:03d} L{b.level} {b.word_count:>4}w {b.short_title}"
translated_this_run += 1 print(f" [... ] {label} ", end="", flush=True)
print(f"\r [done] {label}{cn:>4}字 ({elapsed:4.1f}s, +{len(patch)} terms)") try:
_block, translation, patch, elapsed = run_one(b)
except (ZenMuxError, RuntimeError) as e:
print(f"\n [FAIL] {label}\n {e}")
continue
successful_patches.append(patch)
cn = count_chinese_chars(translation)
print(f"\r [done] {label}{cn:>4}字 ({elapsed:4.1f}s, +{len(patch)} terms)")
else:
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool:
futures = {pool.submit(run_one, b): b for b in todo}
for fut in concurrent.futures.as_completed(futures):
b = futures[fut]
label = f"#{b.order:03d} L{b.level} {b.word_count:>4}w {b.short_title}"
try:
_block, translation, patch, elapsed = fut.result()
except (ZenMuxError, RuntimeError) as e:
print(f" [FAIL] {label}\n {e}")
continue
successful_patches.append(patch)
cn = count_chinese_chars(translation)
print(f" [done] {label}{cn:>4}字 ({elapsed:4.1f}s, +{len(patch)} terms)")
glossary_conflicts: list[tuple[str, str, str]] = []
for patch in successful_patches:
for k, v in patch.items():
if k not in glossary:
glossary[k] = v
elif _glossary_value(glossary[k]) != v:
glossary_conflicts.append((k, _glossary_value(glossary[k]), v))
if successful_patches:
save_glossary(glossary_path, glossary)
if glossary_conflicts:
print(f" [WARN] glossary patch 冲突 {len(glossary_conflicts)} 条,保留既有译法")
# 汇总:按 order 拼接所有 chunk # 汇总:按 order 拼接所有 chunk
merged: list[str] = [] merged: list[str] = []