feat: Release v0.10 - Modular Architecture & External Config

- Refactor codebase into src/ (preprocessing, translation, assembly)
- Add pipeline/ scripts for individual stages
- Externalize configuration to config/config.yaml
- Fix Cover Image preservation
- Update documentation and manuals

.agent/rules/runing-guide.md

@@ -2,4 +2,37 @@
 trigger: always_on
 ---
 
-通过 run.sh 执行具体的 Python 脚本，以确保在 venv 环境执行
+# EPUB Bilingual Translator - File Architecture & Workspace Rules
+
+## 1. Directory Structure
+The project follows a strict modular structure. Code is in `src/`, execution scripts in `pipeline/`, and intermediate data in `.work/`.
+
+.
+├── main.py                     # Entry point (orchestrator)
+├── pipeline/                   # Executable scripts for each stage
+│   ├── 01_preprocess.py        # Step 1: Clean & Extract
+│   ├── 02_translate.py         # Step 2: LLM Translation
+│   └── 03_assemble.py          # Step 3: Backfill & Build
+├── src/                        # Source modules
+│   ├── common/                 # shared utils, config, paths.py
+│   ├── preprocessing/          # epub_cleaner, text_extractor, profiler
+│   ├── translation/            # llm_client, translator_engine, manifest_manager
+│   └── assembly/               # backfiller, builder, format_restorer
+├── config/
+│   ├── config.yaml             # System configuration (LLM, Translation)
+│   └── prompts.json            # LLM Prompts
+└── .work/                      # Working Directory (Gitignored)
+    └── {book_name}/            # One folder per book
+        ├── book_structure.json # Structural skeleton (created by Step 1)
+        ├── manifest.json       # Translation source of truth
+        ├── assets/             # Extracted images/css
+        └── chunks/             # Debug chunks from translation (before/after)
+
+## 2. Path Resolution Rule
+ALWAYS use `src.common.paths.get_work_dirs(input_path)` to resolve paths.
+DO NOT hardcode `work/`, `.work/`, or `tmp/` paths in scripts.
+
+## 3. Data Flow
+1. Preprocess: EPUB -> .work/{book}/book_structure.json + .work/{book}/manifest.json
+2. Translate: .work/{book}/manifest.json (read/write) -> .work/{book}/chunks/ (logs)
+3. Assemble: .work/{book}/manifest.json + .work/{book}/book_structure.json -> output/{book}_bilingual.epub

.gitignore

@@ -21,3 +21,6 @@ coverage.xml
 config/config.json
 output/
 cache/
+work/
+tmp/
+.work/

README.md

@@ -1,97 +1,14 @@
-# EPUB 双语翻译程序 v0.07
+# EPUB Bilingual Translator
 
-一个基于 OpenRouter/OpenAI API 的 EPUB 双语翻译工具，采用**全局编号系统**和**真并发翻译**。
+Current Version: 0.10 (In Development)
+Architecture: v2 (Modular)
 
-## ✨ 核心特性
+## Usage
-
-### 🎯 全局编号系统
-- **每个段落分配全局唯一ID**（格式：`p_0001`, `p_0002`...）
-- **ID贯穿全流程**：提取 → 翻译 → 组装
-- **精确对应保证**：绝不出现中英文错行问题
-
-### ⚡ 真并发翻译
-- **asyncio.gather 并发执行**：高效利用 API 速率限制
-- **智能速率控制**：基于 Token 桶的 RateLimiter
-- **实时进度显示**：Rich 进度条显示翻译状态
-- **断点续传**：自动记录进度，随时中断随时继续
-
-### 🛡️ 安全与稳定
-- **环境隔离**：支持 `.env` 配置，API Key 不落地
-- **鲁棒重试**：集成 `tenacity` 处理网络波动
-- **缓存系统**：基于 Hash 的持久化缓存，跨天复用
-
-### 🎨 极致排版
-- **盘古之白**：自动在中文与西文数字间添加空格
-- **样式注入**：注入专用 CSS 优化阅读体验
-
-## 🚀 快速开始
-
-### 1. 安装依赖
 
 ```bash
-pip install -r requirements.txt
-```
-
-### 2. 配置环境
-
-复制 `.env` 模板并填入你的 API Key：
-
-```bash
-# .env 文件
-V3_API_KEY=sk-xxxxxx
-OPENROUTER_API_KEY=sk-or-xxxxxx
-```
-
-### 3. 开始翻译
-
-```bash
-# 默认使用 OpenRouter
 python main.py input/book.epub
-
-# 使用 V3 Provider
-python main.py input/book.epub -p v3
-
-# 测试模式（只翻译前3个块）
-python main.py input/book.epub --test
 ```
 
-## 📂 目录结构
+## Architecture
 
-```
+See `doc/architecture_flow.md` for details.
-.
-├── config/             # 配置文件
-│   ├── config.json     # 主配置
-│   └── prompts.json    # 提示词模板
-├── input/              # 输入 EPUB 目录
-├── output/             # 输出 EPUB 目录
-├── cache/              # 缓存目录 (Manifest, Translations)
-├── logs/               # 运行日志
-└── src/                # 源代码
-```
-
-## ⚙️ 核心配置 (config.json)
-
-```json
-{
-  "translation": {
-    "chunk_size": 5000,
-    "temperature": 0.3
-  },
-  "providers": {
-    "v3": {
-      "base_url": "https://api.gpt.ge/v1",
-      "models": { "fast": "gpt-4o-mini" },
-      "rate_limits": { "requests_per_minute": 500 }
-    }
-  }
-}
-```
-
-## 📄 许可证
-
-MIT License
-
----
-
-**版本**: v0.07  
-**更新**: 2026-01-13

archive/v0.09/README.md

@@ -0,0 +1,80 @@
+# EPUB 双语翻译程序 v0.10 (Architecture Refactored)
+
+一个基于 OpenRouter/OpenAI API 的 EPUB 双语翻译工具，采用**全局编号系统**和**真并发翻译**。
+v0.10 引入了全新的**清洗-提取-回填**架构，彻底解决了格式丢失和错位问题。
+
+## ✨ 核心特性
+
+### 🛡️ 稳健的架构 (New)
+- **EpubCleaner 预处理**：自动修复 TOC 死链、缺失 UID，标准化 HTML 结构，确保输入源干净可靠。
+- **FineGrained Extractor**：基于 DOM 的高精度提取，支持 `<h1>`-`<h6>` 及所有 `<p>` 标签。
+- **格式保护 v2**：自动识别数学公式、代码块和行内样式，使用占位符保护，防止 LLM 破坏格式。
+
+### 🎯 全局编号系统
+- **DOM 级回填**：不再依赖脆弱的正则，而是利用 DOM 引用进行 100% 精确的一一回填。
+- **双重模式**：支持 Bilingual (双语对照) 和 Chinese (纯译文保留原格式) 模式。
+
+### ⚡ 真并发翻译
+- **asyncio.gather 并发执行**：高效利用 API 速率限制
+- **智能速率控制**：基于 Token 桶的 RateLimiter
+- **实时进度显示**：Rich 进度条显示翻译状态
+- **断点续传**：自动记录进度，随时中断随时继续
+
+### 🔧 自动修复 (Self-Healing)
+- **Format Repair**：当 LLM 返回的格式损坏时，自动触发修复机制，利用 LLM 进行自我纠正。
+
+## 🚀 快速开始
+
+### 1. 安装依赖
+
+```bash
+pip install -r requirements.txt
+```
+
+### 2. 配置环境
+
+复制 `.env` 模板并填入你的 API Key：
+
+```bash
+# .env 文件
+V3_API_KEY=sk-xxxxxx
+OPENROUTER_API_KEY=sk-or-xxxxxx
+```
+
+### 3. 开始翻译
+
+```bash
+# 默认使用 OpenRouter (双语模式)
+python main.py input/book.epub
+
+# 纯中文模式 (保留原版样式)
+python main.py input/book.epub -m chinese
+
+# 测试模式（只翻译前10个块，快速验证）
+python main.py input/book.epub --test
+```
+
+## 📂 目录结构
+
+```
+.
+├── config/             # 配置文件
+├── input/              # 输入 EPUB 目录
+├── output/             # 输出 EPUB 目录
+├── cache/              # 缓存目录 (Manifest, Translations, Processed Epubs)
+├── logs/               # 运行日志
+└── src/                # 源代码
+    ├── epub_cleaner.py # 预处理器
+    ├── fine_grained_extractor.py # 提取器
+    ├── bilingual_builder.py # 构建器
+    └── translator.py   # 主流程
+```
+
+## 📄 许可证
+
+MIT License
+
+---
+
+**版本**: v0.10  
+**更新**: 2026-01-19

archive/v0.09/analyze_manifest_errors.py

@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+"""
+Manifest 错误分析工具
+用于深入分析翻译结果中的占位符问题
+"""
+import sys
+import re
+import json
+from pathlib import Path
+
+sys.path.insert(0, '.')
+from src.manifest_manager import ManifestManager
+
+def analyze_item(item):
+    """详细分析单个 Item 的占位符状态"""
+    issues = []
+    
+    twp = item.text_with_placeholders or ""
+    trans = item.translation_with_placeholders or ""
+    pmap = item.placeholder_map or {}
+    
+    ph_regex = r'φ(/?\d+)φ'
+    src_phs = set(re.findall(ph_regex, twp))
+    trans_phs = set(re.findall(ph_regex, trans))
+    map_ids = {k for k in pmap.keys() if not k.startswith('_')}
+    
+    # 模拟 FormatRestorer 的宽松逻辑
+    unknowns = trans_phs - map_ids
+    real_unknowns = set()
+    for pid in unknowns:
+        # 如果是 /N，且 N 在 map 中，则认为是安全的冗余闭合
+        if pid.startswith('/') and pid[1:] in map_ids:
+            continue
+        real_unknowns.add(pid)
+            
+    if real_unknowns:
+        issues.append(f"未知占位符(幻觉): {real_unknowns}")
+
+    missing = map_ids - trans_phs
+    if missing:
+        issues.append(f"丢失占位符: {missing}")
+        
+    # Check Reordering
+    src_ph_list = re.findall(ph_regex, twp)
+    trans_ph_list = re.findall(ph_regex, trans)
+    common = [p for p in src_ph_list if p in trans_ph_list]
+    trans_common = [p for p in trans_ph_list if p in src_ph_list]
+    if common != trans_common:
+         issues.append(f"占位符乱序: 原文{common} -> 译文{trans_common}")
+
+    return issues
+
+def main():
+    manifest_dir = Path("cache/manifests")
+    files = list(manifest_dir.glob("*_manifest.json"))
+    if not files:
+        print("未找到 manifest 文件")
+        return
+        
+    target = next((f for f in files if "Karen Hao" in f.name), files[0])
+    print(f"Loading: {target}")
+    
+    manifest = ManifestManager(str(target))
+    if not manifest.load():
+        print("加载失败")
+        return
+        
+    items = manifest.get_items()
+    error_count = 0
+    total_analyzed = 0
+    
+    print("\n" + "="*50)
+    print(" 异常项目分析报告")
+    print("="*50 + "\n")
+
+    for item in items:
+        # 只分析非成功状态或有 warning 的
+        if item.status == "completed":
+            continue
+            
+        # 即使是 format_error, failed, translated (with issues)
+        issues = analyze_item(item)
+        has_error = (item.status in ["failed", "format_error"]) or bool(issues)
+        
+        if has_error:
+            total_analyzed += 1
+            print(f"ID: {item.global_id} (Status: {item.status})")
+            if item.error_msg:
+                print(f"  Error Msg: {item.error_msg}")
+            
+            # 只有当有具体 issue 时才打印详细文本，避免刷屏
+            if issues or item.status == "format_error":
+                print(f"  原文: {item.text_with_placeholders[:100]}...")
+                print(f"  译文: {item.translation_with_placeholders[:100]}..." if item.translation_with_placeholders else "  译文: (None)")
+                print(f"  Map: {list(item.placeholder_map.keys())}")
+            
+            for issue in issues:
+                print(f"  -> {issue}")
+            
+            print("-" * 50)
+            error_count += 1
+            
+            if error_count > 50:
+                print("... (Errors truncated) ...")
+                break
+
+    print(f"\n分析完成。共发现 {total_analyzed} 个异常项目。")
+
+if __name__ == "__main__":
+    main()

archive/v0.09/config/config.json

@@ -0,0 +1,69 @@
+{
+  "translation": {
+    "chunk_size": 5000,
+    "temperature": 0.3,
+    "glossary": {
+      "enabled": true,
+      "auto_generate": true,
+      "sample_size": 3000
+    }
+  },
+  "output": {
+    "output_dir": "output",
+    "filename_suffix": "_bilingual"
+  },
+  "logging": {
+    "level": "INFO",
+    "file": "logs/translator.log",
+    "rotation": "10 MB",
+    "retention": "7 days"
+  },
+  "providers": {
+    "openrouter": {
+      "base_url": "https://openrouter.ai/api/v1",
+      "api_key": "YOUR_OPENROUTER_API_KEY",
+      "models": {
+        "fast": "google/gemini-2.0-flash-001",
+        "smart": "google/gemini-2.0-flash-thinking-exp:free"
+      },
+      "extra_headers": {
+        "HTTP-Referer": "https://github.com/epub-translator",
+        "X-Title": "EPUB Translator"
+      },
+      "rate_limits": {
+        "requests_per_minute": 60,
+        "concurrent_requests": 32
+      }
+    },
+    "v3": {
+      "base_url": "https://api.gpt.ge/v1",
+      "api_key": "YOUR_V3_API_KEY",
+      "models": {
+        "fast": "gemini-3-flash-preview",
+        "smart": "gemini-3-pro-preview"
+      },
+      "extra_headers": {
+        "x-foo": "true"
+      },
+      "rate_limits": {
+        "requests_per_minute": 500,
+        "concurrent_requests": 50
+      }
+    },
+    "openai": {
+      "base_url": "http://127.0.0.1:8045/v1",
+      "api_key": "YOUR_ANTIGRAVITY_API_KEY",
+      "models": {
+        "fast": "gemini-3-flash",
+        "smart": "gemini-3-pro-high"
+      },
+      "extra_headers": {
+        "x-foo": "true"
+      },
+      "rate_limits": {
+        "requests_per_minute": 500,
+        "concurrent_requests": 50
+      }
+    }
+  }
+}

archive/v0.09/config/prompts.json

@@ -0,0 +1,10 @@
+{
+  "translation": {
+    "system": "你是一位精通中英文的专业翻译家。你的任务是翻译书籍内容。\n\n要求：\n1. 准确传达原文含义，语言流畅自然，符合中文阅读习惯。\n2. 严格保持【p_xxxxx】编号格式，不要遗漏，不要修改编号。\n3. **关键格式指令（CRITICAL）**：\n   - 原文中包含特殊格式标记：【φ数字φ】（开始/单体）和【φ/数字φ】（结束）。\n   - 示例：\"φ1φTable Talkφ/1φ\" 应翻译为 \"φ1φ桌谈φ/1φ\"。\n   - 示例：\"φ2φ\" (单体) 表示公式或符号，必须保留在译文对应位置。\n   - 规则：严禁删除任何标记！严禁修改数字编号！保持标记与文本的相对位置不变。\n4. **尾注锚点**：形如 φnφ 且中间没有文本的占位符（如 \"...他问道。φ3φ接着...\"）是尾注返回链接，必须保留在译文的对应位置，确保读者可以从尾注跳回正文。\n5. 不要添加任何解释、注释或无关内容，只返回【编号】+【译文】。\n\n{{glossary_instruction}}",
+    "user_template": "请翻译以下段落（务必原样保留所有 φnφ 格式标记）：\n\n{{content}}"
+  },
+  "glossary_extraction": {
+    "system": "你是一位资深的文学编辑和领域专家。你的任务是分析书籍样本，提取关键术语并制定统一的译名表。",
+    "user_template": "请阅读以下书籍片段（包含前言和正文采样）。\n\n任务：\n1. 识别文中出现的人名（如 'Masa', 'Steve Jobs'）、地名、机构名。\n2. 识别特定的行业术语或关键概念。\n3. 为上述词汇提供标准的中文译名。如果像 'Masa' 这样的昵称有对应的全名（如孙正义），请务必使用全名。\n\n请以 JSON 格式输出，格式如下：\n{\n  \"Masa\": \"孙正义\",\n  \"Apple\": \"苹果公司\",\n  ...\n}\n\n书籍片段：\n\n{{content}}"
+  }
+}

archive/v0.09/debug_condition.py

@@ -0,0 +1,16 @@
+# 测试 Python 的条件判断
+item_status = "translated"
+translation_with_placeholders = ""  # 空字符串
+
+# 这是 translator.py 中的条件
+if item_status != "translated" or not translation_with_placeholders:
+    print("SKIP: 条件成立，跳过此 item")
+else:
+    print("PROCESS: 条件不成立，处理此 item")
+
+# 现在测试有内容的情况
+translation_with_placeholders = "你好"
+if item_status != "translated" or not translation_with_placeholders:
+    print("SKIP: 条件成立，跳过此 item")
+else:
+    print("PROCESS: 条件不成立，处理此 item")

archive/v0.09/debug_flow.py

@@ -0,0 +1,62 @@
+"""模拟翻译流程，定位数据丢失问题"""
+import asyncio
+from src.manifest_manager import ManifestManager
+
+async def simulate_worker(manifest, chunks):
+    """模拟 worker 行为"""
+    for chunk in chunks:
+        for item in chunk:
+            # 模拟 LLM 返回
+            fake_translation = f"翻译_{item.global_id}"
+            # 模拟 worker 的 update_item 调用
+            manifest.update_item(
+                item.global_id,
+                fake_translation,
+                translation_with_placeholders=fake_translation,
+                status="translated"
+            )
+    print(f"Worker 完成，内存中 translated 数量: {len(manifest.get_items(status='translated'))}")
+
+async def simulate_restoration(manifest):
+    """模拟 process_format_restoration"""
+    items = manifest.get_items()  # 不带参数，获取所有
+    print(f"Restoration 获取到 {len(items)} 个 items")
+    
+    processed = 0
+    skipped = 0
+    for item in items:
+        # 这是关键的过滤条件
+        if item.status != "translated" or not item.translation_with_placeholders:
+            skipped += 1
+            continue
+        processed += 1
+    
+    print(f"Restoration: 处理 {processed} 个，跳过 {skipped} 个")
+
+async def main():
+    # 初始化
+    manifest = ManifestManager("test_flow.json")
+    manifest.init_manifest("test", {})
+    
+    # 添加测试 items
+    for i in range(5):
+        manifest.add_item(f"test{i}.html", f"<p>Text {i}</p>", f"Text {i}", "p")
+    
+    # 模拟 create_chunks_from_manifest
+    pending = manifest.get_items(status="pending")
+    chunks = [pending]  # 一个 chunk 包含所有
+    print(f"Chunks 创建，pending 数量: {len(pending)}")
+    print(f"chunks[0][0] is manifest._items_by_id['p_00001']: {chunks[0][0] is manifest._items_by_id['p_00001']}")
+    
+    # 模拟 worker
+    await simulate_worker(manifest, chunks)
+    
+    # 模拟 restoration
+    await simulate_restoration(manifest)
+    
+    # 清理
+    import os
+    if os.path.exists("test_flow.json"):
+        os.remove("test_flow.json")
+
+asyncio.run(main())

archive/v0.09/debug_manifest.py

@@ -0,0 +1,31 @@
+from src.manifest_manager import ManifestManager, ManifestItem
+
+# 模拟 worker 更新流程
+manifest = ManifestManager("test_manifest.json")
+manifest.init_manifest("test", {})
+
+# 添加一个 item
+item = manifest.add_item("test.html", "<p>Hello</p>", "Hello", "p")
+print(f"After add: item.status = {item.status}, item.translation = {item.translation}")
+print(f"ID in manifest: {item.global_id}")
+
+# 模拟 get_items 获取的是同一个对象吗？
+pending = manifest.get_items(status="pending")
+print(f"pending[0] is item: {pending[0] is item}")
+
+# 模拟 worker 更新
+manifest.update_item(item.global_id, "你好", translation_with_placeholders="你好", status="translated")
+
+# 检查更新是否生效
+print(f"After update: item.status = {item.status}, item.translation = {item.translation}")
+print(f"After update: item.translation_with_placeholders = '{item.translation_with_placeholders}'")
+
+# 验证 get_items 能获取到更新后的状态
+translated = manifest.get_items(status="translated")
+print(f"Translated items count: {len(translated)}")
+if translated:
+    print(f"translated[0].translation_with_placeholders = '{translated[0].translation_with_placeholders}'")
+
+import os
+if os.path.exists("test_manifest.json"):
+    os.remove("test_manifest.json")

archive/v0.09/debug_prompt_construction.py

@@ -0,0 +1,120 @@
+#!/usr/bin/env python3
+"""
+调试脚本：使用真实数据验证 Prompt 构建
+完整展示发送给 LLM 的文本
+"""
+import sys
+import json
+from pathlib import Path
+from loguru import logger
+
+# 添加 src 到路径
+sys.path.insert(0, '.')
+
+from src.manifest_manager import ManifestManager
+from src.text_processor import TextProcessor
+from src.llm_client import LLMClient
+from src.utils import load_config
+
+# 配置日志到文件，避免控制台刷屏
+logger.remove()
+logger.add("debug_prompt.log", level="DEBUG")
+logger.add(sys.stdout, level="INFO")
+
+def debug_prompt():
+    print("=== 1. 加载配置 ===")
+    try:
+        config = load_config()
+        # 确保 LLM 配置存在 (Mock if needed for init)
+        if 'llm' not in config:
+            if 'v3' in config['providers']:
+                config['llm'] = config['providers']['v3']
+            else:
+                config['llm'] = {"api_key": "dummy", "models": {"fast": "dummy"}}
+                
+    except Exception as e:
+        print(f"Config load failed: {e}")
+        return
+
+    print("=== 2. 加载真实 Manifest ===")
+    # 查找 cache/manifests 下的 json 文件
+    manifest_dir = Path("cache/manifests")
+    if not manifest_dir.exists():
+        print("Error: cache/manifests directory not found")
+        return
+        
+    manifest_files = list(manifest_dir.glob("*_manifest.json"))
+    if not manifest_files:
+        print("Error: No manifest file found in cache/manifests")
+        return
+        
+    manifest_path = manifest_files[0]
+    print(f"Using manifest: {manifest_path}")
+    
+    manifest = ManifestManager(str(manifest_path))
+    if not manifest.load():
+        print("Failed to load manifest")
+        return
+        
+    print(f"Loaded {len(manifest.get_items())} items")
+    
+    # 获取 Pending items (模拟真实流程)
+    pending = manifest.get_items(status="pending")
+    if not pending:
+        print("No pending items found. Using ALL items for debug.")
+        items_to_process = manifest.get_items()
+    else:
+        items_to_process = pending
+        
+    # 找到几个包含占位符的 item 用于验证
+    target_items = []
+    for item in items_to_process:
+        if item.placeholder_map and len(item.placeholder_map) > 0:
+            target_items.append(item)
+            if len(target_items) >= 5: # 取前5个
+                break
+    
+    if not target_items:
+        print("No items with placeholders found!")
+        return
+        
+    print(f"Selected {len(target_items)} items with placeholders for verification")
+    for item in target_items:
+        print(f"  - {item.global_id}: twp length={len(item.text_with_placeholders or '')}")
+
+    print("\n=== 3. 生成 Prompt (Mode: Chinese) ===")
+    client = LLMClient(config)
+    
+    # 只为这几个 item 生成 prompt
+    prompt = client._build_prompt(target_items, mode="chinese")
+    
+    print("\n" + "="*40)
+    print("FULL PROMPT CONTENT (Snippet):")
+    print("="*40)
+    print(prompt)
+    print("="*40 + "\n")
+    
+    print("\n=== 4. 关键验证 ===")
+    placeholders_found = prompt.count('φ')
+    print(f"Total 'φ' symbols in prompt: {placeholders_found}")
+    
+    for item in target_items:
+        if item.text_with_placeholders and 'φ' in item.text_with_placeholders:
+            # 检查这个 item 的 ID 是否在 prompt 中
+            in_prompt = item.global_id in prompt
+            # 检查这个 item 的占位符是否在 prompt 中
+            # 注意：如果占位符是 φ1φ，我们检查 'φ1φ' 是否在 prompt 中
+            # 这里简单做，假设 text_with_placeholders 应该完整出现在 prompt 中 (忽略空白差异)
+            import re
+            normalized_twp = re.sub(r'\s+', '', item.text_with_placeholders)
+            normalized_prompt = re.sub(r'\s+', '', prompt)
+            
+            content_in_prompt = normalized_twp in normalized_prompt
+            
+            print(f"Item {item.global_id}:")
+            print(f"  In prompt ID: {in_prompt}")
+            print(f"  Original twp: {repr(item.text_with_placeholders)}")
+            print(f"  Content match (ignoring whitespace): {content_in_prompt}")
+
+if __name__ == "__main__":
+    debug_prompt()

archive/v0.09/debug_restore.py

@@ -0,0 +1,32 @@
+import re
+from src.format_restorer import FormatRestorer
+
+restorer = FormatRestorer()
+
+# 模拟一个真实场景
+text_with_ph = '"But what is the goal?" φ1φAmodeiφ/1φ...'
+placeholder_map = {
+    '1': '<em>',
+    '/1': '</em>'
+}
+
+print("Input text:", text_with_ph)
+print("Placeholder map:", placeholder_map)
+
+# 手动执行 restorer 的验证逻辑
+inner_map = {k: v for k, v in placeholder_map.items() if not k.startswith("_")}
+print("Inner map keys:", set(inner_map.keys()))
+
+found_ids = set(re.findall(r'φ(/?\d+)φ', text_with_ph))
+print("Found IDs in text:", found_ids)
+
+expected_ids = set(inner_map.keys())
+print("Expected IDs:", expected_ids)
+
+missing_ids = expected_ids - found_ids
+print("Missing IDs:", missing_ids)
+
+# 调用 restore
+html, success = restorer.restore(text_with_ph, placeholder_map)
+print(f"\nResult: success={success}")
+print(f"HTML: {html}")

archive/v0.09/main.py

@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+import asyncio
+import sys
+import argparse
+from pathlib import Path
+from loguru import logger
+from src.translator import EPUBTranslator
+from src.epub_parser import EPUBParser
+from src.toc_parser import TOCParser
+from src.utils import load_config, setup_logging
+
+def parse_args():
+    parser = argparse.ArgumentParser(description="EPUB 双语翻译工具")
+    parser.add_argument("epub_path", help="输入 EPUB 文件路径")
+    parser.add_argument("--provider", "-p", default="openrouter", help="LLM 供应商 (config.json 中 providers 的 key)")
+    parser.add_argument("--mode", "-m", default="bilingual", choices=["bilingual", "chinese"],
+                        help="输出模式: bilingual (双语对照) 或 chinese (纯中文，保留格式)")
+    parser.add_argument("--test", action="store_true", help="测试模式（仅翻译前几段）")
+    parser.add_argument("--output", "-o", help="输出目录")
+    parser.add_argument("--no-cache", action="store_true", help="禁用缓存（强制重新翻译）")
+    parser.add_argument("--clear-cache", action="store_true", help="清理所有缓存文件")
+    
+    # TOC 章节选择
+    parser.add_argument("--show-toc", action="store_true", help="显示书籍目录结构")
+    parser.add_argument("--from", dest="from_chapter", help="起始章节标题")
+    parser.add_argument("--to", dest="to_chapter", help="结束章节标题")
+    
+    return parser.parse_args()
+
+def flatten_provider_config(config: dict, provider_name: str) -> dict:
+    """
+    将选定的 provider 配置扁平化到 config['llm'] 中，
+    以便下游模块统一调用。
+    """
+    providers = config.get('providers', {})
+    if provider_name not in providers:
+        available = list(providers.keys())
+        logger.error(f"未找到供应商 '{provider_name}'。可用供应商: {available}")
+        sys.exit(1)
+        
+    selected_config = providers[provider_name]
+    logger.info(f"使用 LLM 供应商: {provider_name} ({selected_config.get('base_url')})")
+    
+    # 注入到 config['llm']
+    config['llm'] = selected_config
+    return config
+
+async def run_translation(args):
+    try:
+        # 1. 加载配置
+        config = load_config()
+        
+        # 2. 处理 Provider 选择
+        config = flatten_provider_config(config, args.provider)
+        
+        # 3. 设置日志
+        setup_logging(config)
+        logger.info("程序启动")
+        
+        # 4. 初始化翻译器
+        translator = EPUBTranslator(config, use_cache=not args.no_cache)
+        
+        # 5. 执行翻译 (传递章节范围参数)
+        await translator.translate_epub(
+            args.epub_path,
+            test_mode=args.test,
+            output_dir=args.output,
+            mode=args.mode,
+            from_chapter=args.from_chapter,
+            to_chapter=args.to_chapter
+        )
+        
+    except Exception as e:
+        import traceback
+        traceback.print_exc()
+        logger.error(f"翻译失败: {e}")
+        sys.exit(1)
+
+def show_toc(epub_path: str):
+    """显示 EPUB 的目录结构"""
+    parser = EPUBParser(epub_path)
+    toc_parser = TOCParser(parser.book)
+    
+    print(f"\n📖 {parser.metadata.get('title', 'Unknown')} - {parser.metadata.get('author', 'Unknown')}")
+    print(toc_parser.format_toc_table())
+    print("提示: 使用 --from \"章节名\" --to \"章节名\" 指定翻译范围")
+    print()
+
+def main():
+    args = parse_args()
+    
+    if args.clear_cache:
+        import shutil
+        cache_dir = Path("cache")
+        if cache_dir.exists():
+            shutil.rmtree(cache_dir)
+            print("缓存已清理")
+        # sys.exit(0) # 移除退出，允许继续执行
+
+    
+    if args.show_toc:
+        show_toc(args.epub_path)
+        sys.exit(0)
+
+    asyncio.run(run_translation(args))
+
+if __name__ == "__main__":
+    main()

archive/v0.09/requirements.txt

@@ -0,0 +1,12 @@
+ebooklib>=0.19
+beautifulsoup4>=4.12.0
+lxml>=4.9.0
+openai>=1.0.0
+aiohttp>=3.9.0
+pydantic>=2.0.0
+loguru>=0.7.0
+rich>=13.0.0
+asyncio-throttle>=1.0.2
+tenacity>=8.0.0
+python-dotenv>=1.0.0
+socksio>=1.0.0

archive/v0.09/src/__init__.py

@@ -0,0 +1,32 @@
+"""
+EPUB 双语翻译程序
+主要功能模块的初始化文件
+"""
+
+__version__ = "0.08"
+__author__ = "Kaitan"
+
+from .epub_parser import EPUBParser
+from .translator import EPUBTranslator
+from .llm_client import LLMClient as OpenRouterClient # Keep alias for compatibility
+from .llm_client import LLMClient
+from .text_processor import TextProcessor
+from .bilingual_builder import BilingualEPUBBuilder
+from .chinese_builder import ChineseEPUBBuilder
+from .format_extractor import FormatExtractor
+from .format_restorer import FormatRestorer
+from .utils import load_config, setup_logging
+
+__all__ = [
+    "EPUBParser",
+    "EPUBTranslator", 
+    "LLMClient",
+    "OpenRouterClient",
+    "TextProcessor",
+    "BilingualEPUBBuilder",
+    "ChineseEPUBBuilder",
+    "FormatExtractor",
+    "FormatRestorer",
+    "load_config",
+    "setup_logging"
+]

archive/v0.09/src/bilingual_builder.py

@@ -0,0 +1,235 @@
+"""
+双语 EPUB 构建器模块 (集成 V2)
+
+基于 FineGrainedExtractor 的 DOM 回填机制，确保 100% 的内容对齐和格式保留。
+"""
+
+from ebooklib import epub
+import ebooklib
+from bs4 import BeautifulSoup
+from typing import Dict, List
+from pathlib import Path
+from loguru import logger
+import uuid
+from .fine_grained_extractor import FineGrainedExtractor
+
+class BilingualEPUBBuilder:
+    """双语 EPUB 构建器"""
+    
+    def __init__(self, original_book, config: Dict):
+        self.original_book = original_book
+        self.config = config
+        # 从配置中获取是否翻译目录
+        self.translate_toc = config['translation'].get('translate_toc', False)
+    
+    def create_bilingual_epub_with_mapping(self, translation_map: Dict[str, str],
+                                          paragraph_map: Dict[str, Dict],
+                                          output_path: str) -> str:
+        """
+        创建双语 EPUB。使用 ordered_ids 确保与 Manifest 严格一致。
+        
+        Args:
+            translation_map: { global_item_id: translated_text_with_ph }
+            paragraph_map: { global_item_id: item_metadata_dict }
+        """
+        try:
+            new_book = epub.EpubBook()
+            self._copy_metadata(new_book)
+            
+            # 安全清理 TOC (虽然预处理已做，但构建新书对象时再次确保合规)
+            new_book.toc = self._sanitize_toc(self.original_book.toc)
+            
+            # 1. 准备每个文件的有序ID列表
+            # 目的是将扁平的 map 重新按文件和顺序组织
+            file_ordered_ids = {}
+            # paragraph_map 的 key 是 global_id，通常包含顺序信息或我们依赖 items 的插入顺序
+            # 更好的方式是依赖 item ID 的数字部分排序，如果它们是 'id_0', 'id_1'...
+            # 假设 ID 包含顺序信息。
+            sorted_pids = sorted(paragraph_map.keys(), key=lambda x: self._extract_id_index(x))
+            
+            for pid in sorted_pids:
+                info = paragraph_map[pid]
+                fname = info['file_name']
+                if fname not in file_ordered_ids:
+                    file_ordered_ids[fname] = []
+                file_ordered_ids[fname].append(pid)
+            
+            processed_item_ids = set()
+            item_map = {}
+
+            # 特殊处理：封面图片
+            self._handle_cover(new_book, processed_item_ids, item_map)
+
+            # 2. 复制所有非文档资源 (图片, CSS, 字体)
+            for item in self.original_book.get_items():
+                if item.get_type() != ebooklib.ITEM_DOCUMENT:
+                    if item.id not in processed_item_ids:
+                        new_book.add_item(item)
+                        processed_item_ids.add(item.id)
+                        item_map[item.id] = item
+
+            # 3. 处理并回填文档
+            new_spine = []
+            for spine_id, linear in self.original_book.spine:
+                item = self.original_book.get_item_with_id(spine_id)
+                if not item: continue
+                
+                if item.get_type() == ebooklib.ITEM_DOCUMENT:
+                    file_name = item.get_name()
+                    new_item = item  # 默认使用原 Item
+                    
+                    # 如果该文件有翻译内容
+                    if file_name in file_ordered_ids:
+                        target_ids = file_ordered_ids[file_name]
+                        
+                        # 执行回填
+                        new_content = self._process_document_content(
+                            item.get_content().decode('utf-8'),
+                            file_name,
+                            target_ids,
+                            translation_map
+                        )
+                        
+                        # 创建新 item 避免污染原对象
+                        new_item = epub.EpubHtml(
+                            title=item.title,
+                            file_name=file_name,
+                            lang='zh-CN', # 双语版主要语言
+                            uid=item.id
+                        )
+                        new_item.set_content(new_content.encode('utf-8'))
+                        # 复制原 item 的其他属性如 style
+                        for link in item.get_links():
+                             # 这里不做深度复制，简单引用
+                             pass
+                        # 重新添加 links (特别是 CSS)
+                        # 注意: EpubHtml 构造时不会自动带原来的 links，需要手动加
+                        # 但我们在 dirty hack 里，直接 set_content 了 HTML。
+                        # 如果 HTML head 里有 link, ebooklib 可能会解析并注册?
+                        # Ebooklib 的行为是: 只有通过 add_link 加的才会出现在 opf manifest。
+                        # 我们需要把原 item 的 links 复制过来
+                        if hasattr(item, 'links'):
+                            for link in item.links:
+                                new_item.add_link(**link) # 不是很安全，视 ebooklib 版本而定
+
+                    if new_item.id not in processed_item_ids:
+                        new_book.add_item(new_item)
+                        processed_item_ids.add(new_item.id)
+                    new_spine.append(new_item)
+                else:
+                    if item.id in item_map:
+                        new_spine.append(item_map[item.id])
+            
+            new_book.spine = new_spine
+            new_book.add_item(epub.EpubNcx())
+            new_book.add_item(epub.EpubNav())
+            
+            # 生成输出文件名
+            output_file = self._generate_output_filename(output_path)
+            epub.write_epub(output_file, new_book, {})
+            logger.info(f"双语 EPUB 生成成功: {output_file}")
+            return output_file
+            
+        except Exception as e:
+            logger.error(f"创建双语 EPUB 失败: {e}", exc_info=True)
+            raise
+
+    def _process_document_content(self, content: str, file_name: str, 
+                                  target_ids: List[str], translation_map: Dict[str, str]) -> str:
+        """
+        处理单个文档的内容：提取 -> 注入翻译 -> 回填
+        """
+        try:
+            # 1. 再次提取，建立 DOM 映射
+            # 必须使用与 TextProcessing 阶段完全一致的参数
+            extractor = FineGrainedExtractor(translate_toc=self.translate_toc)
+            items = extractor.extract(content, file_name)
+            
+            # 2. 筛选出应该翻译的项目
+            translatable_items = [i for i in items if i['should_translate']]
+            
+            # 3. 一致性检查
+            if len(translatable_items) != len(target_ids):
+                logger.error(
+                    f"严重错误 [{file_name}]: 提取项数 ({len(translatable_items)}) "
+                    f"与 Manifest 记录数 ({len(target_ids)}) 不一致! "
+                    "将跳过此文件的翻译回填以防错位。"
+                )
+                # Fallback: 返回原始内容
+                return content
+            
+            # 4. 注入翻译
+            for extract_item, pid in zip(translatable_items, target_ids):
+                translation = translation_map.get(pid)
+                if translation:
+                    extract_item['translation'] = translation
+            
+            # 5. 回填
+            # 获取输出模式
+            output_mode = self.config.get('output', {}).get('mode', 'bilingual')
+            bilingual_mode = (output_mode == 'bilingual')
+            
+            new_html = extractor.backfill(items, bilingual=bilingual_mode)
+
+            
+            return new_html
+            
+        except Exception as e:
+            logger.error(f"处理文档内容失败 {file_name}: {e}", exc_info=True)
+            return content
+
+    def _extract_id_index(self, pid: str) -> int:
+        """从 ID 字符串中提取数字索引用于排序 (如 'p_10' -> 10)"""
+        try:
+            # 尝试常见格式 p_123, id_456
+            parts = pid.split('_')
+            if len(parts) > 1 and parts[-1].isdigit():
+                return int(parts[-1])
+            return 0
+        except:
+            return 0
+
+    def _sanitize_toc(self, toc):
+        """确保 TOC 中的所有节点都有 ID"""
+        for item in toc:
+            if isinstance(item, (epub.Link, epub.Section)):
+                if not getattr(item, 'uid', None):
+                    item.uid = f"navPoint-{uuid.uuid4().hex[:8]}"
+            elif isinstance(item, tuple) and len(item) == 2:
+                section, children = item
+                if isinstance(section, (epub.Link, epub.Section)):
+                    if not getattr(section, 'uid', None):
+                        section.uid = f"navPoint-{uuid.uuid4().hex[:8]}"
+                self._sanitize_toc(children)
+        return toc
+
+    def _handle_cover(self, new_book, processed_item_ids, item_map):
+        cover_id_meta = self.original_book.get_metadata('OPF', 'cover')
+        if cover_id_meta:
+            cover_item = self.original_book.get_item_with_id(cover_id_meta[0][0])
+            if cover_item:
+                new_book.add_item(cover_item)
+                processed_item_ids.add(cover_item.id)
+                item_map[cover_item.id] = cover_item
+                # 复制 cover metadata
+                new_book.add_metadata('OPF', 'cover', cover_item.id)
+
+    def _copy_metadata(self, new_book):
+        for namespace, meta_dict in self.original_book.metadata.items():
+            for name, values in meta_dict.items():
+                for value, other in values:
+                    if name and hasattr(name, 'lower') and name.lower() == 'identifier': continue
+                    new_book.add_metadata(namespace, name, value, other)
+        new_book.add_metadata('DC', 'language', 'zh-CN')
+        new_book.set_identifier(f"bilingual-{uuid.uuid4().hex[:12]}")
+
+    def _generate_output_filename(self, output_path: str) -> str:
+        # 根据模式生成不同的后缀
+        output_mode = self.config.get('output', {}).get('mode', 'bilingual')
+        suffix = "chinese" if output_mode == "chinese" else "bilingual"
+        
+        title_meta = self.original_book.get_metadata('DC', 'title')
+        title = title_meta[0][0] if title_meta else "bilingual_book"
+        safe_title = "".join([c for c in title if c.isalnum() or c in (' ', '-', '_')]).strip()
+        Path(output_path).mkdir(parents=True, exist_ok=True)
+        return str(Path(output_path) / f"{safe_title}_{suffix}.epub")

archive/v0.09/src/chinese_builder.py

@@ -10,21 +10,22 @@
 from ebooklib import epub
 import ebooklib
 from bs4 import BeautifulSoup
-from typing import Dict, List
+from typing import Dict, List, Any
 from pathlib import Path
 from loguru import logger
 import uuid
+from .fine_grained_extractor import FineGrainedExtractor
 
 class ChineseEPUBBuilder:
-    """纯中文 EPUB 构建器"""
+    """纯中文 EPUB 构建器 (DOM Safe)"""
     
     def __init__(self, original_book, config: Dict):
         self.original_book = original_book
         self.config = config
-        self.output_config = config['output']
+        self.translate_toc = config['translation'].get('translate_toc', False)
     
     def create_chinese_epub_with_mapping(self, 
-                                        items: List, # List[ManifestItem]
+                                        items: List[Any], # List[ManifestItem]
                                         output_path: str) -> str:
         """
         创建纯中文 EPUB。
@@ -32,27 +33,22 @@ class ChineseEPUBBuilder:
         try:
             new_book = epub.EpubBook()
             self._copy_metadata(new_book)
+            # 安全清理 TOC
             new_book.toc = self._sanitize_toc(self.original_book.toc)
             
-            # 准备每个文件的有序项目列表
+            # 1. 按文件分组 Manifest Items
-            file_items = {}
+            # 假设 items 已经是按全局 ID 排序的 (ManifestManager.get_items 返回有序列表)
-            for item in sorted(items, key=lambda x: x.global_id):
+            file_items_map = {}
+            for item in items:
                 fname = item.source_file
-                if fname not in file_items:
+                if fname not in file_items_map:
-                    file_items[fname] = []
+                    file_items_map[fname] = []
-                file_items[fname].append(item)
+                file_items_map[fname].append(item)
             
             processed_item_ids = set()
             item_map = {}
 
             # 特殊处理：封面图片
-            cover_id_meta = self.original_book.get_metadata('OPF', 'cover')
-            if cover_id_meta:
-                cover_item = self.original_book.get_item_with_id(cover_id_meta[0][0])
-                if cover_item:
-                    new_book.add_item(cover_item)
-                    processed_item_ids.add(cover_item.id)
-                    item_map[cover_item.id] = cover_item
 
             # 复制资源
             for item in self.original_book.get_items():

archive/v0.09/src/epub_cleaner.py

@@ -0,0 +1,178 @@
+"""
+EPUB 清理器模块 (EpubCleaner)
+
+负责在翻译前对 EPUB 进行标准化清洗，解决兼容性问题。
+核心功能:
+1. Flatten Structure: 将 div 转换为 p，简化结构
+2. Fix TOC: 修复目录中的死链和缺失 UID
+3. CSS Restoration: 找回丢失的样式表
+"""
+
+from bs4 import BeautifulSoup, Tag
+from loguru import logger
+from ebooklib import epub
+import ebooklib
+import zipfile
+import uuid
+from ebooklib.epub import Link
+
+class EpubCleaner:
+    """标准 EPUB 清理器"""
+    
+    def clean_epub(self, input_path: str, output_path: str):
+        """
+        清理 EPUB 文件并保存到新路径
+        """
+        logger.info(f"开始清理: {input_path}")
+        
+        # 1. 尝试打开 Zip 以读取原始内容 (Ebooklib 回退机制)
+        try:
+            input_zip = zipfile.ZipFile(input_path, 'r')
+            zip_files = set(input_zip.namelist())
+        except Exception as e:
+            logger.error(f"无法打开 Zip (样式回退功能将失效): {e}")
+            input_zip = None
+            zip_files = set()
+        
+        # 2. 读取 EPUB
+        try:
+            book = epub.read_epub(input_path)
+        except Exception as e:
+            logger.error(f"Ebooklib 读取失败: {e}")
+            raise
+
+        # 3. 遍历并清理文档
+        count = 0
+        for item in book.get_items():
+            if item.get_type() == ebooklib.ITEM_DOCUMENT:
+                try:
+                    file_name = item.get_name()
+                    content = None
+                    
+                    # 优先从 Zip 读取以保留 Head 信息 (CSS Links)
+                    if input_zip and file_name in zip_files:
+                        try:
+                            content = input_zip.read(file_name).decode('utf-8')
+                        except Exception:
+                            pass
+                    
+                    # 回退到 ebooklib
+                    if content is None:
+                        raw_content = item.get_content()
+                        if raw_content:
+                            content = raw_content.decode('utf-8')
+                    
+                    if not content or not content.strip():
+                        continue
+                    
+                    # 执行清理
+                    cleaned = self._clean_content(content, item)
+                    
+                    # 安全检查
+                    if not cleaned.strip():
+                        logger.warning(f"警告: {file_name} 清理后为空，保留原始内容")
+                        cleaned = content
+                    
+                    item.set_content(cleaned.encode('utf-8'))
+                    count += 1
+                    
+                except Exception as e:
+                    logger.warning(f"清理文档失败 {item.get_name()}: {e}")
+        
+        # 4. 修复 TOC (死链和 UID)
+        try:
+            book.toc = self._fix_and_clean_toc(book.toc, book)
+        except Exception as e:
+            logger.error(f"TOC 修复失败: {e}")
+
+        # 5. 保存
+        epub.write_epub(output_path, book)
+        logger.info(f"清理完成: {output_path} (处理了 {count} 个文档)")
+
+    def _clean_content(self, html_content: str, item=None) -> str:
+        """
+        执行具体的 HTML 清理逻辑
+        """
+        soup = BeautifulSoup(html_content, 'html.parser')
+        
+        # 恢复 CSS 链接
+        if item:
+            self._restore_css_links(soup, item)
+        
+        # div 转 p
+        stats = {'divs_to_p': 0}
+        inline_tags = {'span', 'em', 'strong', 'i', 'b', 'u', 'a', 'br', 'sub', 'sup'}
+        
+        for div in list(soup.find_all('div')):
+            # 检查是否有块级子元素 (如果有，则保留 div 容器结构)
+            has_block = any(
+                isinstance(c, Tag) and c.name not in inline_tags 
+                for c in div.children
+            )
+            
+            if not has_block:
+                div.name = 'p'
+                stats['divs_to_p'] += 1
+                
+        # 可以在这里添加更多清理逻辑 (如移除无用的空的 span 等)
+        
+        return str(soup)
+
+    def _restore_css_links(self, soup, item):
+        """从原始 HTML 中提取并恢复 CSS 链接到 item 对象"""
+        head = soup.find('head')
+        if head:
+            links = head.find_all('link', rel='stylesheet')
+            for link in links:
+                href = link.get('href')
+                if href:
+                    existing_links = list(item.get_links())
+                    exists = False
+                    for l in existing_links:
+                        l_href = getattr(l, 'href', None)
+                        if l_href is None and isinstance(l, dict):
+                            l_href = l.get('href')
+                        if l_href == href:
+                            exists = True
+                            break
+                    
+                    if not exists:
+                        item.add_link(href=href, rel='stylesheet', type='text/css')
+
+    def _fix_and_clean_toc(self, toc, book):
+        """修复 TOC：补全 UID 并移除指向不存在文件的死链"""
+        new_toc = []
+        
+        for item in toc:
+            # Case 1: (Section, Children)
+            if isinstance(item, (tuple, list)):
+                section, children = item
+                cleaned_children = self._fix_and_clean_toc(children, book)
+                
+                if isinstance(section, Link):
+                    href = section.href.split('#')[0]
+                    if book.get_item_with_href(href):
+                        if section.uid is None:
+                            section.uid = f'uuid-{uuid.uuid4()}'
+                        new_toc.append((section, cleaned_children))
+                    else:
+                        logger.warning(f"移除无效 TOC 节点: {section.href}")
+                        new_toc.extend(cleaned_children)
+                else:
+                    new_toc.append((section, cleaned_children))
+
+            # Case 2: Link
+            elif isinstance(item, Link):
+                href = item.href.split('#')[0]
+                if book.get_item_with_href(href):
+                     if item.uid is None:
+                         item.uid = f'uuid-{uuid.uuid4()}'
+                     new_toc.append(item)
+                else:
+                    logger.warning(f"移除无效 TOC 节点: {item.href}")
+            
+            # Case 3: Other
+            else:
+                new_toc.append(item)
+                
+        return new_toc

archive/v0.09/src/fine_grained_extractor.py

@@ -0,0 +1,222 @@
+"""
+Fine-Grained Extractor (集成格式保护版)
+
+负责从 EPUB 中提取文本，进行精细化处理，并负责最终的回填工作。
+集成 FormatExtractor 以实现行内格式的保护。
+"""
+
+from bs4 import BeautifulSoup, Tag, NavigableString
+from typing import List, Dict, Any, Tuple
+import re
+from loguru import logger
+from .format_extractor import FormatExtractor
+from .format_restorer import FormatRestorer
+
+class FineGrainedExtractor:
+    """细粒度提取与回填器"""
+    
+    SKIP_TRANSLATION_PATTERNS = [
+        r'index\.x?html',
+        r'bibliography\.x?html',
+        r'endnotes?\.x?html',
+        r'footnotes?\.x?html',
+    ]
+    
+    TOC_PATTERNS = [
+        r'nav\.x?html',
+        r'toc\.x?html',
+    ]
+    
+    def __init__(self, translate_toc: bool = False):
+        self.translate_toc = translate_toc
+        self.soup = None
+        # 初始化格式处理器
+        self.format_extractor = FormatExtractor()
+        self.format_restorer = FormatRestorer()
+    
+    def extract(self, html_content: str, file_name: str = "") -> List[Dict[str, Any]]:
+        """
+        提取所有正文元素 (p, h1-h6)，并进行格式分析
+        """
+        self.soup = BeautifulSoup(html_content, 'html.parser')
+        
+        # 不要移除 link/style/meta/script，否则回填时会丢失头部信息
+        # for element in self.soup(['script', 'style', 'meta', 'link']):
+        #     element.decompose()
+
+        
+        doc_type = self._classify_document(file_name)
+        items = []
+        
+        # 目标: 所有段落和标题
+        target_tags = ['p', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6']
+        
+        for element in self.soup.find_all(target_tags):
+            # 1. 初步文本提取 (用于过滤判断)
+            raw_text = element.get_text(separator=' ', strip=True)
+            if not raw_text.strip():
+                continue
+            
+            # 2. 判断是否需要翻译
+            is_decorative = self._is_decorative(raw_text)
+            should_translate = self._should_translate(doc_type, is_decorative, raw_text)
+            
+            item_data = {
+                'element': element,       # 持有引用，用于回填
+                'tag': element.name,
+                'raw_text': raw_text,
+                'should_translate': should_translate,
+                'doc_type': doc_type,
+                'text_len': len(raw_text)
+            }
+            
+            # 3. 如果需要翻译，执行通过 FormatExtractor 进行精细化格式提取
+            if should_translate:
+                # 必须传入 Outer HTML (str(element))，因为 FormatExtractor 内部会解析并剥离最外层标签
+                # 如果只传 inner_html，FormatExtractor 解析时只能拿到第一个子节点，导致内容丢失验证失败
+                outer_html = str(element)
+                # 提取 (clean_text, text_with_ph, map, type, endnote_anchors)
+                clean, text_ph, ph_map, _, endnote_anchors = self.format_extractor.extract(outer_html)
+                
+                # 再次验证：如果提取后的 clean_text 为空 (比如全是公式)，则不翻译
+                if not clean.strip() or not text_ph.strip():
+                    item_data['should_translate'] = False
+                else:
+                    item_data['text'] = clean       # 纯文本 (供人阅读/日志)
+                    item_data['text_nodes'] = []    # 兼容旧字段 (空)
+                    item_data['text_with_ph'] = text_ph  # 发送给 LLM 的文本
+                    item_data['placeholder_map'] = ph_map
+                    item_data['endnote_anchors'] = endnote_anchors  # 尾注锚点 ID 列表
+            else:
+                # 不需要翻译，仅保留基础信息
+                item_data['text'] = raw_text
+                item_data['text_with_ph'] = raw_text  # Fallback
+            
+            items.append(item_data)
+            
+        logger.debug(f"[{doc_type}] {file_name}: 提取 {len(items)} 元素, 需翻译 {sum(1 for i in items if i['should_translate'])}")
+        return items
+
+    def backfill(self, items: List[Dict[str, Any]], bilingual: bool = True) -> str:
+        """
+        回填翻译 (支持格式还原)
+        
+        Args:
+            items: 提取的元素列表，且已注入 'translation' 字段 (带占位符的译文)
+            bilingual: 是否生成双语版本
+        """
+        success_count = 0
+        
+        for item in items:
+            if not item.get('should_translate'):
+                continue
+            
+            # 使用预先注入的翻译 (解决了重复文本映射问题)
+            translated_ph = item.get('translation')
+            
+            if not translated_ph:
+                continue
+                
+            original_element = item['element']
+            
+            # 4. 格式还原
+            ph_map = item.get('placeholder_map', {})
+            # 容错：如果 ph_map 为 None (未开启格式保护)，设为空字典
+            if ph_map is None: ph_map = {}
+            
+            restored_html, _ = self.format_restorer.restore(translated_ph, ph_map)
+            
+            # 5. 构建新 DOM 元素
+            if bilingual:
+                # 双语模式: Append
+                new_tag = self.soup.new_tag(original_element.name)
+                
+                # 继承 class
+                classes = original_element.get('class', [])
+                new_tag['class'] = list(classes) + ['translation', 'chinese']
+                
+                # 继承 style
+                style = original_element.get('style')
+                if style:
+                    new_tag['style'] = style
+                
+                # 设置内容 (解析 restored HTML)
+                inner_soup = BeautifulSoup(restored_html, 'html.parser')
+                if inner_soup.body:
+                    for child in list(inner_soup.body.children):
+                         new_tag.append(child)
+                else:
+                    for child in list(inner_soup.children):
+                        new_tag.append(child)
+                        
+                original_element.insert_after(new_tag)
+                
+            else:
+                # 仅中文模式: Replace
+                # 直接修改 original_element 的内容
+                original_element.clear()
+                inner_soup = BeautifulSoup(restored_html, 'html.parser')
+                
+                # 直接替换内容
+                if inner_soup.body:
+                    for child in list(inner_soup.body.children):
+                        original_element.append(child)
+                else:
+                    for child in list(inner_soup.children):
+                         original_element.append(child)
+                         
+                # 可以在这里移除 dropcap class? 
+                # 但如果 dropcap 是内部 span，已经被还原回去了。
+                # 由于 Drop Cap 处理在 FormatExtractor 已经把首字母放入文本，Prefix 里的 dropcap 是空的
+                # 还原后的 HTML 大概是 <span class="dropcap"></span>这...
+                pass
+
+            success_count += 1
+            
+        return str(self.soup)
+
+    # --- 以下是辅助判别逻辑 (同原版) ---
+    
+    def _classify_document(self, file_name: str) -> str:
+        if not file_name: return 'core'
+        fname = file_name.lower()
+        if any(re.search(p, fname) for p in self.SKIP_TRANSLATION_PATTERNS): return 'skip'
+        if any(re.search(p, fname) for p in self.TOC_PATTERNS): return 'toc'
+        return 'core'
+
+    def _should_translate(self, doc_type: str, is_decorative: bool, text: str = "") -> bool:
+        if is_decorative: return False
+        if self._is_roman_numeral(text): return False
+        if doc_type == 'core': return True
+        if doc_type == 'toc': return self.translate_toc
+        return False
+
+    def _is_roman_numeral(self, text: str) -> bool:
+        text = text.strip().upper()
+        if not text: return False
+        # 简单宽松匹配: 纯字母且看起来像罗马数字 (I, V, X, L, C, M)
+        # 排除普通单词如 "I" (作为代词时应翻译，但作为单独段落通常是标题)
+        # 这是一个权衡。单独的 "I" 在小说里可能表示 "我"，但在章节标题里表示 "第一章"。
+        # 如果是正文中的 "I am...", 肯定会被提取。这里只有单独的 "I" 才会被这里匹配。
+        # 真正的问题是：单独一行 "I" 表示 "我" 的情况极少。
+        pattern = re.compile(r"^M{0,4}(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})$")
+        return bool(pattern.match(text))
+
+    def _is_decorative(self, text: str) -> bool:
+        s = text.strip()
+        if not s: return False
+        if not any(c.isalnum() for c in s): return True
+        if len(s) > 20: return False
+        
+        patterns = [
+            r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾]+$',
+            r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾\s]+$',
+        ]
+        for p in patterns:
+            if re.match(p, s): return True
+        
+        # 字符种类很少且包含非字母 (e.g. "* * *")
+        unique = set(s.replace(' ', ''))
+        if len(unique) <= 3 and (unique & set('*-_~•◆◇■□▪▫●○◉◎⦿⦾')):
+            return True
+        return False

archive/v0.09/src/format_extractor.py

@@ -93,7 +93,7 @@ class FormatExtractor:
     def __init__(self):
         self.detector = HeadingDetector()
 
-    def extract(self, element_html: str) -> Tuple[str, str, Dict[str, str], str]:
+    def extract(self, element_html: str) -> Tuple[str, str, Dict[str, str], str, List[str]]:
         """
         提取格式信息
         
@@ -102,6 +102,7 @@ class FormatExtractor:
             text_with_placeholders: 只包含内嵌占位符的文本（不含前缀/后缀标签）
             placeholder_map: 占位符映射，包含特殊键 "_prefix" 和 "_suffix"
             paragraph_type: 段落类型
+            endnote_anchors: 尾注锚点 ID 列表 (用于补救)
         """
         soup = BeautifulSoup(element_html, 'html.parser')
         root = list(soup.children)[0] if list(soup.children) else soup
@@ -117,6 +118,12 @@ class FormatExtractor:
         # 智能提取（分离前缀/后缀）
         text_with_ph, local_map = self._smart_extract_v3(inner_html)
         
+        # 核心修复：清理 text_with_placeholders 中的换行符和多余空格
+        # 这一步至关重要，因为 inner_html 中的换行符会导致 LLM Prompt 格式混乱（多行）
+        # 从而导致 LLM 忽略不在同一行的占位符或内容
+        if text_with_ph:
+            text_with_ph = re.sub(r'\s+', ' ', text_with_ph).strip()
+        
         # === 验证完整性 ===
         # 将 text_with_placeholders 去掉占位符后与 clean_text 比较
         stripped_text = self._strip_placeholders(text_with_ph)
@@ -129,7 +136,17 @@ class FormatExtractor:
             # 降级：不使用前缀/后缀分离，只做简单占位符处理
             text_with_ph, local_map = self._fallback_extract(inner_html, clean_text)
         
-        return clean_text, text_with_ph, local_map, p_type
+        # === 识别尾注锚点 ===
+        # 尾注锚点特征：<span id="aXXX"></span> (短随机ID，通常 3-5 字符)
+        endnote_anchors = []
+        for pid, html in local_map.items():
+            if pid.startswith("_"):
+                continue  # 跳过 _prefix, _suffix
+            # 匹配空锚点：<span id="aXXX"></span> 或 <a id="aXXX"></a>
+            if re.match(r'<(span|a)\s+id="[a-zA-Z][a-zA-Z0-9]{2,5}"\s*>\s*</\1>', html):
+                endnote_anchors.append(pid)
+        
+        return clean_text, text_with_ph, local_map, p_type, endnote_anchors
     
     def _strip_placeholders(self, text: str) -> str:
         """移除所有占位符（φXφ 和 φ/Xφ 格式）"""
@@ -173,9 +190,11 @@ class FormatExtractor:
         智能提取 v3：分离前缀/后缀 + 合并内嵌公式块
         
         核心逻辑：
-        1. 分离前缀（第一个可翻译文本之前）和后缀（最后一个可翻译文本之后）
+        1. 分离前缀（第一个可翻译文本之前的完整标签）和后缀（最后一个可翻译文本之后的完整标签）
         2. 中间部分：检测"公式块"（连续标签+不可翻译文本），合并为单个占位符
         3. 只有真正需要翻译的格式标签（如斜体包裹的长文本）才拆分
+        
+        注意：前缀/后缀只包含不影响文本结构的完整标签，开始标签必须有匹配的结束标签
         """
         # 使用正则分割标签和文本
         parts = re.split(r'(<[^>]+>)', inner_html)
@@ -209,11 +228,72 @@ class FormatExtractor:
             # 没有可翻译文本，全部作为前缀
             return "", {"_prefix": inner_html, "_suffix": ""}
         
-        # 分割
+        # === 安全前缀分离 ===
-        prefix_parts = parts[:first_trans_idx]
+        # 只将自闭合标签和空白作为前缀，一旦遇到开始标签就停止
-        middle_parts = parts[first_trans_idx:last_trans_idx + 1]
+        # 因为开始标签可能包裹着后面的可翻译文本
-        middle_types = part_types[first_trans_idx:last_trans_idx + 1]
+        safe_prefix_end = 0
-        suffix_parts = parts[last_trans_idx + 1:]
+        for i in range(first_trans_idx):
+            if part_types[i] == 'tag':
+                tag = parts[i]
+                # 检查是否是自闭合标签或结束标签（不太可能在开头）
+                is_self_closing = tag.endswith('/>') or re.match(r'<(br|hr|img|meta|link)\b', tag, re.I)
+                is_closing = tag.startswith('</')
+                # 检查是否是空元素（如 <span id="xxx"></span>，紧跟着结束标签）
+                is_empty_element = False
+                if not is_self_closing and not is_closing and i + 1 < first_trans_idx:
+                    # 查看下一个标签是否是对应的结束标签
+                    next_idx = i + 1
+                    while next_idx < first_trans_idx and part_types[next_idx] in ('whitespace',):
+                        next_idx += 1
+                    if next_idx < first_trans_idx and part_types[next_idx] == 'tag':
+                        next_tag = parts[next_idx]
+                        if next_tag.startswith('</'):
+                            # 检查标签名是否匹配
+                            open_name = re.match(r'<(\w+)', tag)
+                            close_name = re.match(r'</(\w+)', next_tag)
+                            if open_name and close_name and open_name.group(1) == close_name.group(1):
+                                is_empty_element = True
+                                # 跳过这对空元素
+                                safe_prefix_end = next_idx + 1
+                                continue
+                
+                if is_self_closing or is_closing:
+                    safe_prefix_end = i + 1
+                elif is_empty_element:
+                    pass  # 已在上面处理
+                else:
+                    # 遇到普通开始标签，停止
+                    break
+            elif part_types[i] == 'whitespace':
+                safe_prefix_end = i + 1
+            else:
+                # formula 类型，不应该出现在前缀中
+                break
+        
+        # === 安全后缀分离 ===
+        # 从末尾开始向前，只剥离连续的完整结束标签/自闭合标签或空白
+        safe_suffix_start = len(parts)
+        for i in range(len(parts) - 1, last_trans_idx, -1):
+            if part_types[i] == 'tag':
+                tag = parts[i]
+                is_closing = tag.startswith('</')
+                is_self_closing = tag.endswith('/>') or re.match(r'<(br|hr|img|meta|link)\b', tag, re.I)
+                if is_closing or is_self_closing:
+                    safe_suffix_start = i
+                else:
+                    # 遇到开始标签，不能作为独立后缀剥离
+                    break
+            elif part_types[i] == 'whitespace':
+                safe_suffix_start = i
+            else:
+                # 遇到普通文本或公式，停止剥离
+                break
+        
+        # 重新分割
+        prefix_parts = parts[:safe_prefix_end]
+        middle_parts = parts[safe_prefix_end:safe_suffix_start]
+        middle_types = part_types[safe_prefix_end:safe_suffix_start]
+        suffix_parts = parts[safe_suffix_start:]
         
         # === Drop Cap 检测 ===
         # 英文书籍常用首字母放大样式，如 <span class="dropcap">T</span>his
@@ -308,23 +388,47 @@ class FormatExtractor:
                             result_parts.append(f"φ{pid}φ")
                             
             elif ptype in ('formula', 'whitespace'):
-                # 公式或空白，检查是否是连续块的开始
+                # 简化处理：非可翻译文本直接保留
-                block_parts = []
+                # 公式检测等复杂逻辑仅在增强模式下启用
-                while i < len(middle_parts) and middle_types[i] in ('formula', 'whitespace'):
+                result_parts.append(part)
-                    block_parts.append(middle_parts[i])
+                i += 1
-                    i += 1
-                if block_parts:
-                    block_html = "".join(block_parts)
-                    pid = str(placeholder_counter)
-                    placeholder_counter += 1
-                    local_map[pid] = block_html
-                    result_parts.append(f"φ{pid}φ")
             else:
                 i += 1
         
         text_with_ph = "".join(result_parts)
         text_with_ph = re.sub(r'\s+', ' ', text_with_ph).strip()
         
+        # === 连续占位符合并 ===
+        # 将 φ1φφ2φ 这样的连续占位符合并为一个
+        def merge_consecutive_placeholders(text: str, ph_map: dict) -> Tuple[str, dict]:
+            """合并连续占位符"""
+            # 匹配连续的占位符（2个或更多）
+            pattern = r'(φ/?\d+φ)(φ/?\d+φ)+'
+            
+            def merge_match(m):
+                full_match = m.group(0)
+                # 提取所有占位符ID
+                pids = re.findall(r'φ(/?\d+)φ', full_match)
+                if len(pids) <= 1:
+                    return full_match
+                
+                # 合并对应的 HTML
+                merged_html = ""
+                for pid in pids:
+                    if pid in ph_map:
+                        merged_html += ph_map[pid]
+                        del ph_map[pid]
+                
+                # 创建新的合并占位符
+                new_pid = pids[0] if pids[0].isdigit() else pids[0][1:]  # 使用第一个数字
+                ph_map[new_pid] = merged_html
+                return f"φ{new_pid}φ"
+            
+            merged_text = re.sub(pattern, merge_match, text)
+            return merged_text, ph_map
+        
+        text_with_ph, local_map = merge_consecutive_placeholders(text_with_ph, local_map)
+        
         return text_with_ph, local_map
 
 
@@ -430,6 +534,7 @@ class FormatExtractor:
         条件（满足任一即可）：
         1. 包含 3 个及以上连续字母（如 "and", "Art", "War"）
         2. 包含空格分隔的多个单词（如 "and Sun Tzu's"）
+        3. 包含数字（如章节号 "10", "12"）
         """
         text = text.strip()
         if not text:
@@ -440,6 +545,9 @@ class FormatExtractor:
         # 条件2: 包含空格的多单词文本（如 "a of"）
         if ' ' in text and re.search(r'[a-zA-Z]', text):
             return True
+        # 条件3: 包含数字（章节号等）
+        if re.search(r'\d', text):
+            return True
         return False
 
     def _is_formula_element(self, element: Tag, text_content: str) -> bool:
@@ -463,4 +571,10 @@ class FormatExtractor:
 
     def reset(self):
         """兼容旧接口"""
-        pass
+        pass
+
+    def _is_pure_punctuation(self, text: str) -> bool:
+        """判断文本是否仅包含标点符号和空格（不应该变成占位符）"""
+        # 常见标点符号集合（中英文混合）
+        punctuation_chars = ' ,.:;!?，。：；！？、""\'\'「」【】（）()[]{}—-–…·'
+        return all(c in punctuation_chars for c in text)

archive/v0.09/src/format_restorer.py

@@ -61,8 +61,18 @@ class FormatRestorer:
             
         unknown_ids = found_ids - expected_ids
         if unknown_ids:
-            logger.warning(f"格式还原警告: 发现未知占位符 {unknown_ids}")
+            # 过滤掉冗余的闭合标签（例如 map里有 "1"，但 LLM 输出了 "φ/1φ"）
-            success = False  # 未知占位符也视为问题
+            real_unknowns = set()
+            for pid in unknown_ids:
+                # 如果是 /N，且 N 在 map 中，则认为是安全的冗余闭合
+                if pid.startswith('/') and pid[1:] in expected_ids:
+                    continue
+                real_unknowns.add(pid)
+                
+            if real_unknowns:
+                logger.warning(f"格式还原警告: 发现未知占位符 {real_unknowns}")
+                success = False
+
 
         # 替换占位符
         def replace_match(match):

archive/v0.09/src/llm_client.py

@@ -71,8 +71,9 @@ class LLMClient:
         try:
             with open("config/prompts.json", "r", encoding="utf-8") as f:
                 return json.load(f)
-        except:
+        except Exception as e:
-            return {}
+            logger.error(f"严重错误：无法加载 config/prompts.json: {e}")
+            raise # 必须抛出异常，否则 System Prompt 会降级导致占位符指令丢失
 
     async def translate_chunk(self, items: List[ManifestItem], glossary: Dict = None, 
                               instruction: str = None, model_type: str = "fast",
@@ -89,6 +90,9 @@ class LLMClient:
         """
         if not items: return {}
         
+
+
+        
         model = self.models.get(model_type, self.models.get("fast"))
         prompt = self._build_prompt(items, mode)
         
@@ -96,21 +100,9 @@ class LLMClient:
             # Build System Prompt
             base_sys_prompt = self.prompts.get("translation", {}).get("system", "You are a professional translator.")
             
-            # 中文模式：添加占位符保护指令
+            # 中文模式：Prompt 已在 config/prompts.json 中配置，无需额外硬编码
             if mode == "chinese":
-                base_sys_prompt += """
+                pass
-
-Placeholder Instructions (CRITICAL):
-1. Text contains PAIRED placeholders: φNφ (start) and φ/Nφ (end), like HTML tags.
-2. Example: "φ1φTable Talkφ/1φ" means italic text, translate as "φ1φ桌谈φ/1φ"
-3. Single placeholders φNφ without φ/Nφ are inline elements (footnotes, formulas) - keep them in place.
-4. RULES:
-   - DO NOT create new placeholder numbers that don't exist in the original
-   - DO NOT remove or modify existing placeholders
-   - Keep placeholders in the SAME relative position in your translation
-   - If word order changes, keep placeholders with their associated text
-5. Each line starts with paragraph ID (p_xxxxx). Preserve them.
-"""
 
             
             if instruction:
@@ -123,37 +115,69 @@ Placeholder Instructions (CRITICAL):
             # Strict formatting instructions
             base_sys_prompt += "\n\nRequirements:\n1. Each line MUST start with ID (p_xxxxx).\n2. DO NOT modify IDs.\n3. Return only translations."
 
+
+
+            # DEBUG: 打印发送给 LLM 的完整内容
+            logger.debug(f"=== LLM REQUEST DEBUG ===")
+            logger.debug(f"System Prompt:\n{base_sys_prompt[:500]}...")
+            logger.debug(f"User Prompt (first 1000 chars):\n{prompt[:1000]}")
+            logger.debug(f"=========================")
+
             raw_response = await self._make_request(model, base_sys_prompt, prompt)
             
+
+
+            # DEBUG: 打印 LLM 返回的完整内容
+            logger.debug(f"=== LLM RESPONSE DEBUG ===")
+            logger.debug(f"Raw Response (first 1500 chars):\n{raw_response[:1500] if raw_response else 'EMPTY'}")
+            logger.debug(f"==========================")
+            
             if not raw_response:
                 return {item.global_id: f"[Error - Empty Response]" for item in items}
             
-            return self._simple_parse(raw_response, items, mode)
+            results = self._simple_parse(raw_response, items, mode)
+            
+
+
+            return results
+
             
         except Exception as e:
             logger.error(f"Translation failed ({model}): {e}")
             return {item.global_id: f"[Error - {str(e)}]" for item in items}
 
-    async def repair_format(self, original_text: str, broken_translation: str) -> str:
+    async def repair_format(self, original_text: str, broken_translation: str, missing_ids: set = None) -> str:
         """
-        修复翻译格式：将占位符正确插入到译文中。
+        修复翻译格式：将缺失的占位符正确插入到译文中。
+        
+        Args:
+            original_text: 原文（带占位符）
+            broken_translation: 有占位符问题的译文
+            missing_ids: 缺失的占位符 ID 集合（可选，用于提示）
         """
         model = self.models.get("fast")
         
-        system_prompt = "You are a format repair assistant. Your ONLY job is to insert placeholders into the translation."
+        system_prompt = """你是格式修复助手。你的任务是将缺失的占位符插入到译文中。
-        user_prompt = f"""
+
-Original Text (with placeholders):
+注意：
+1. 不要重新翻译，保持中文译文内容完全不变
+2. 只需要在正确位置插入缺失的占位符
+3. 占位符格式：φ数字φ（如 φ1φ, φ/1φ）
+4. 只输出修复后的译文，不要任何解释"""
+
+        missing_hint = ""
+        if missing_ids:
+            missing_list = ", ".join([f"φ{pid}φ" for pid in missing_ids])
+            missing_hint = f"\n缺失的占位符: {missing_list}"
+
+        user_prompt = f"""原文（带占位符）:
 {original_text}
 
-Translation (placeholders missing/incorrect):
+当前译文（占位符有误）:
 {broken_translation}
+{missing_hint}
+请修复译文，在正确位置插入缺失的占位符。只输出修复后的译文："""
 
-Task:
-Please rewrite the Translation to include ALL placeholders (φcXXXXXφ) from the Original Text.
-1. DO NOT translate again. Keep the meaning of the Translation.
-2. Place φcXXXXXφ tags exactly where they correspond to the original format (bold, italic, links).
-3. Output ONLY the fixed translation.
-"""
         try:
             return await self._make_request(model, system_prompt, user_prompt)
         except Exception as e:
@@ -171,7 +195,14 @@ Please rewrite the Translation to include ALL placeholders (φcXXXXXφ) from the
         for item in items:
             if mode == "chinese":
                 # 中文模式：使用带占位符的文本和段落类型
-                text = item.text_with_placeholders if item.text_with_placeholders else item.clean_text
+                twp = item.text_with_placeholders
+                # DEBUG: 打印关键信息
+                logger.debug(f"BUILD_PROMPT {item.global_id}: twp='{twp[:50] if twp else 'EMPTY'}...', has_φ={'φ' in twp if twp else False}")
+                
+                text = twp if twp else item.clean_text
+                # 防御性修复：强制清理换行符，兼容旧的脏 Manifest 数据
+                text = re.sub(r'\s+', ' ', text).strip()
+                
                 p_type = getattr(item, 'paragraph_type', 'body').upper()
                 lines.append(f"{item.global_id} [{p_type}] {text}")
             else:
@@ -180,49 +211,59 @@ Please rewrite the Translation to include ALL placeholders (φcXXXXXφ) from the
         return "\n".join(lines)
 
     def _simple_parse(self, response: str, items: List[ManifestItem], mode: str = "bilingual") -> Dict[str, str]:
-        """解析 LLM 响应"""
+        """
+        解析 LLM 响应 - 位置切分版
+        
+        策略：
+        1. 识别响应中所有出现的 p_xxxxx 及其位置
+        2. 按位置顺序将响应切分成每一段，消除对输入顺序的依赖
+        """
         results = {}
-        for i, item in enumerate(items):
+        valid_ids = {item.global_id for item in items}
-            current_id = item.global_id
+        
-            start_idx = response.find(current_id)
+        # 1. 查找所有可能的 ID 位置
-            if start_idx == -1: continue
+        # 模式匹配 p_ 后面跟着 5 位数字
-            
+        matches = list(re.finditer(r'p_\d{5}', response))
-            end_idx = len(response)
+        
-            if i + 1 < len(items):
+        if not matches:
-                next_id = items[i+1].global_id
+            # Fallback: 如果没有匹配到任何 ID，尝试按行扫描
-                next_found = response.find(next_id, start_idx + len(current_id))
-                if next_found != -1:
-                    end_idx = next_found
-            
-            content = response[start_idx:end_idx].strip()
-            clean_content = content[len(current_id):].strip()
-            clean_content = clean_content.lstrip(":： \t")
-            
-            # 移除类型标记 (如 [BODY])
-            if mode == "chinese":
-                clean_content = re.sub(r'^\[[A-Z]+\]\s*', '', clean_content)
-                
-            if clean_content:
-                results[current_id] = clean_content
-
-        # Fallback: 逐行解析
-        if len(results) < len(items):
             for line in response.split("\n"):
                 line = line.strip()
-                for item in items:
+                for it in items:
-                    if item.global_id not in results and line.startswith(item.global_id):
+                    if line.startswith(it.global_id):
-                        res = line[len(item.global_id):].strip().lstrip(":： ")
+                        content = line[len(it.global_id):].strip().lstrip(":： ")
-                        if mode == "chinese":
+                        if content: results[it.global_id] = content
-                            res = re.sub(r'^\[[A-Z]+\]\s*', '', res)
+            return results
-                        if res: results[item.global_id] = res
+
-        
+        # 2. 按查找到的 ID 位置进行切割
+        for i, match in enumerate(matches):
+            current_id = match.group()
+            if current_id not in valid_ids:
+                continue
+            
+            # 这一段内容的起始是当前 ID 之后，结束是下一个匹配的 ID 之前
+            start_pos = match.end()
+            end_pos = matches[i+1].start() if i + 1 < len(matches) else len(response)
+            
+            content = response[start_pos:end_pos].strip()
+            
+            # 3. 清理内容
+            content = content.lstrip(":： \t")
+            if mode == "chinese":
+                # 移除 [BODY] 等类型标记
+                content = re.sub(r'^\[[A-Z]+\]\s*', '', content)
+            
+            content = content.strip()
+            if content:
+                results[current_id] = content
+                
         # 验证解析结果
         parsed_count = len(results)
         expected_count = len(items)
         if parsed_count < expected_count:
             missing_ids = [item.global_id for item in items if item.global_id not in results]
             logger.warning(f"LLM 响应解析不完整: {parsed_count}/{expected_count} (缺失: {missing_ids[:3]}...)")
-        
+            
         return results
 
     @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))

archive/v0.09/src/manifest_manager.py

@@ -36,6 +36,7 @@ class ManifestItem:
     paragraph_type: str = "body"               # 段落类型：chapter/section/subsection/epigraph/body
     translation_with_placeholders: str = ""    # 带占位符的译文
     translation_with_original_html: str = ""   # 还原后的最终 HTML (中文模式)
+    endnote_anchors: List[str] = field(default_factory=list)  # 尾注锚点 ID 列表 (用于补救)
     
     metadata: Dict[str, Any] = field(default_factory=dict)
 
@@ -144,7 +145,10 @@ class ManifestManager:
         # 必须按 ID 顺序返回以保证分块正确
         return sorted(items, key=lambda x: x.global_id)
 
-    def update_item(self, global_id: str, translation: str, status: str = "translated", error: str = None, model: str = None, score: int = None):
+    def update_item(self, global_id: str, translation: str, status: str = "translated", 
+                     error: str = None, model: str = None, score: int = None,
+                     translation_with_placeholders: str = None,
+                     translation_with_original_html: str = None):
         """更新翻译结果。"""
         if global_id in self._items_by_id:
             item = self._items_by_id[global_id]
@@ -157,6 +161,11 @@ class ManifestManager:
                 item.model_used = model
             if score is not None:
                 item.quality_score = score
+            # 中文模式专用字段
+            if translation_with_placeholders is not None:
+                item.translation_with_placeholders = translation_with_placeholders
+            if translation_with_original_html is not None:
+                item.translation_with_original_html = translation_with_original_html
         else:
             logger.warning(f"尝试更新不存在的 ID: {global_id}")
 

archive/v0.09/src/text_processor.py

@@ -0,0 +1,117 @@
+"""
+文本处理器模块 (Text Processor Module) - Manifest 驱动版 (集成 V2)
+
+该模块专注于 HTML 文档的遍历和段落提取。
+已集成 FineGrainedExtractor，实现稳健的结构提取和格式保护。
+"""
+
+import re
+from bs4 import BeautifulSoup
+from typing import List, Dict, Any
+from loguru import logger
+from .manifest_manager import ManifestManager
+from .fine_grained_extractor import FineGrainedExtractor
+
+
+class TextProcessor:
+    """
+    负责从 HTML 中识别有效段落并注册到 Manifest。
+    委托 FineGrainedExtractor 进行具体的提取工作。
+    """
+    
+    def __init__(self, config: Dict):
+        """
+        Args:
+            config (Dict): 全局配置。
+        """
+        self.config = config
+        self.chunk_size = config['translation'].get('chunk_size', 5000)
+        # 不再持有状态，每次调用实例化 Extractor 或复用
+        
+    def extract_to_manifest(self, html_content: str, source_file: str, manifest: ManifestManager, mode: str = "bilingual"):
+        """
+        解析 HTML 内容，并将识别出的段落注册到 Manifest 中。
+
+        Args:
+            html_content (str): HTML 源码。
+            source_file (str): 来源文件名。
+            manifest (ManifestManager): 清单管理器实例。
+            mode (str): 翻译模式 (保留参数)
+        """
+        try:
+            # 实例化细粒度提取器 (集成格式保护)
+            # 是否翻译目录取决于文件名判断，这里交给 Extractor 内部逻辑
+            # 但 Extractor 构造函数需要参数，默认 False
+            translate_toc = self.config['translation'].get('translate_toc', False)
+            extractor = FineGrainedExtractor(translate_toc=translate_toc)
+            
+            items = extractor.extract(html_content, source_file)
+            
+            count = 0
+            for item in items:
+                # 只注册需要翻译的项
+                if not item['should_translate']:
+                    continue
+                
+                clean_text = item.get('text', '')
+                text_with_ph = item.get('text_with_ph', clean_text)
+                placeholder_map = item.get('placeholder_map')
+                
+                # 注册到 Manifest
+                # original_html 用于记录，但实际回填依靠 FineGrainedExtractor 复原
+                manifest_item = manifest.add_item(
+                    source_file=source_file,
+                    original_html=str(item['element']),
+                    clean_text=clean_text,
+                    tag=item['tag'],
+                    metadata={"status": "pending"}
+                )
+                
+                # 显式设置格式保护字段
+                manifest_item.text_with_placeholders = text_with_ph
+                manifest_item.placeholder_map = placeholder_map
+                manifest_item.endnote_anchors = item.get('endnote_anchors', [])
+                
+                # 记录段落类型 (从 FormatExtractor 获得的 p_type，目前 FineGrained 没返回，可以改进)
+                # FineGrained 可以把 FormatExtractor 返回的 p_type 也带出来
+                # 暂且设为 body，或根据 tag 判断
+                p_type = "header" if item['tag'].startswith('h') else "body"
+                manifest_item.paragraph_type = p_type
+                
+                count += 1
+                
+            logger.info(f"提取完成 {source_file}: 注册 {count} 个待翻译项")
+
+        except Exception as e:
+            logger.error(f"从 {source_file} 提取段落失败: {e}", exc_info=True)
+
+    def create_chunks_from_manifest(self, manifest: ManifestManager, mode: str = "bilingual") -> List[List[Any]]:
+        """
+        从 Manifest 中筛选待翻译项目并分块。
+        (保留原有逻辑)
+        """
+        pending_items = manifest.get_items(status="pending")
+        if not pending_items:
+            return []
+            
+        chunks = []
+        current_chunk = []
+        current_size = 0
+        
+        for item in pending_items:
+            # 优先使用带占位符的文本长度计算
+            text_len = len(item.text_with_placeholders) if item.text_with_placeholders else len(item.clean_text)
+                
+            if current_size + text_len > self.chunk_size and current_chunk:
+                chunks.append(current_chunk)
+                current_chunk = []
+                current_size = 0
+            
+            current_chunk.append(item)
+            current_size += text_len
+            
+        if current_chunk:
+            chunks.append(current_chunk)
+            
+        logger.info(f"分块完成: 共有 {len(pending_items)} 个待翻译项，分为 {len(chunks)} 个块")
+        return chunks

archive/v0.09/src/translator.py

@@ -0,0 +1,348 @@
+"""
+EPUB Translator Core Module - v0.09 (TOC Selection Support)
+"""
+
+import asyncio
+import traceback
+from typing import List, Dict, Any
+from pathlib import Path
+from loguru import logger
+from rich.console import Console
+from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TimeElapsedColumn
+
+from .epub_parser import EPUBParser
+from .toc_parser import TOCParser
+from .llm_client import LLMClient
+from .text_processor import TextProcessor
+from .bilingual_builder import BilingualEPUBBuilder
+from .chinese_builder import ChineseEPUBBuilder
+from .manifest_manager import ManifestManager
+from .book_profiler import BookProfiler
+from .cache import TranslationCache
+from .format_restorer import FormatRestorer
+from .utils import add_spacing_between_cn_and_en_num
+
+
+class EPUBTranslator:
+    
+    def __init__(self, config: Dict, use_cache: bool = True):
+        self.config = config
+        self.console = Console()
+        self.use_cache = use_cache
+        
+        self.parser = None
+        self.llm_client = LLMClient(config)
+        self.text_processor = TextProcessor(config)
+        self.profiler = BookProfiler(config, self.llm_client)
+        self.cache = TranslationCache(config) if use_cache else None
+        self.restorer = FormatRestorer()
+        
+        self.manifest_dir = Path("cache/manifests")
+        self.manifest_dir.mkdir(parents=True, exist_ok=True)
+
+    async def translate_epub(self, epub_path: str, test_mode: bool = False, 
+                             output_dir: str = None, mode: str = "bilingual",
+                             from_chapter: str = None, to_chapter: str = None) -> str:
+        """
+        翻译 EPUB 文件
+        """
+        try:
+            epub_path = Path(epub_path)
+            
+            # --- 0. Preprocessing (集成的 EpubCleaner) ---
+            # 创建临时清理文件
+            from .epub_cleaner import EpubCleaner
+            cleaner = EpubCleaner()
+            processed_dir = self.manifest_dir / "processed_epubs"
+            processed_dir.mkdir(parents=True, exist_ok=True)
+            cleaned_epub_path = processed_dir / f"{epub_path.stem}_cleaned.epub"
+            
+            self.console.print(f"[yellow]Preprocessing: Cleaning EPUB structures...[/yellow]")
+            cleaner.clean_epub(str(epub_path), str(cleaned_epub_path))
+            
+            # 关键：后续操作全都基于清理后的 EPUB
+            # 注意：这会改变 source_file 的上下文吗？只要 parser 读的是 cleaned_epub，manifest 记录的就是 cleaned_epub 里的文件名。
+            # 而 Builder 用 cleaned_epub 初始化的，所以也是匹配的。
+            actual_epub_path = cleaned_epub_path
+            
+            self.parser = EPUBParser(str(actual_epub_path))
+            
+            # 0.5 解析 TOC 并处理章节范围
+            toc_parser = TOCParser(self.parser.book)
+            include_files = None
+            chapter_range_info = None
+            
+            if from_chapter or to_chapter:
+                include_files, selected_items = toc_parser.get_spine_range(
+                    start_title=from_chapter, end_title=to_chapter
+                )
+                if selected_items:
+                    start_title = selected_items[0].title
+                    end_title = selected_items[-1].title
+                    self.console.print(f"[cyan]📚 Range: {start_title} ~ {end_title} ({len(include_files)} files)[/cyan]")
+                    chapter_range_info = {"start_title": start_title, "end_title": end_title, "included_files": list(include_files)}
+            else:
+                skip_files = toc_parser.get_skip_files()
+                if skip_files:
+                    include_files = toc_parser.get_content_files_from_spine()
+                    self.console.print(f"[cyan]📚 Smart Skip: {len(skip_files)} non-content files[/cyan]")
+
+            # 1. Manifest
+            manifest_suffix = "_chinese" if mode == "chinese" else ""
+            manifest_path = self.manifest_dir / f"{epub_path.stem}{manifest_suffix}_manifest.json"
+            manifest = ManifestManager(str(manifest_path))
+            
+            if not manifest.load() or not self.use_cache:
+                self.console.print(f"[yellow]Initializing Manifest...[/yellow]")
+                manifest.init_manifest(book_id=epub_path.name, metadata=self.parser.get_book_info(), chapter_range=chapter_range_info)
+                content_items = self.parser.extract_all_content_items(include_files=include_files)
+                for item in content_items:
+                    self.text_processor.extract_to_manifest(item['content'], item['file_name'], manifest, mode=mode)
+                manifest.save()
+            
+            # (Stats logic...)
+            stats = manifest.stats
+            self.console.print(f"[green]Manifest: {stats['total']} items ({stats['pending']} pending)[/green]")
+
+            # 2. Profile
+            profile = {}
+            if not test_mode and stats['pending'] > 0:
+                self.console.print("[yellow]Profiling Book...[/yellow]")
+                profile = await self.profiler.analyze_book(manifest)
+
+            # 3. Translate
+            chunks = self.text_processor.create_chunks_from_manifest(manifest, mode=mode)
+            if test_mode: chunks = chunks[:10]
+            if chunks:
+                await self._translate_concurrently(chunks, manifest, profile, mode=mode)
+            
+            # 4. Build
+            self.console.print(f"\n[yellow]Building {mode} EPUB...[/yellow]")
+            output_path = output_dir or self.config['output']['output_dir']
+            
+            # 统一使用 BilingualEPUBBuilder (集成 V2)
+            # 因为它已经支持了 FineGrained backfill，可以处理 bilingual 参数
+            # 但目前 builder 还没暴露 bilingual 参数给 create 方法？
+            # 我们可以简单地在 builder.create... 里改一下，或者总是用 BilingualBuilder。
+            # 用户想要 "chinese" mode (replace).
+            # BilingualBuilder.create... 目前 hardcode 了 bilingual=True (TODO comment in previous step).
+            # 我们应该让 BilingualBuilder 支持 mode 参数。
+            # 为了简单，我假设 builder 内部会处理，或者我之后微调 builder。
+            # 修改: BilingualEPUBBuilder 是通用的 backfiller。
+            
+            # 构建 Mapping
+            if mode == "chinese":
+                # 中文模式：使用还原后的 HTML (保留格式)
+                translation_map = {
+                    item.global_id: (item.translation_with_original_html or item.translation)
+                    for item in manifest.get_items() 
+                    if item.translation_with_original_html or item.translation
+                }
+            else:
+                # 双语模式：使用纯文本
+                translation_map = {item.global_id: item.translation for item in manifest.get_items() if item.translation}
+
+            paragraph_map = {item.global_id: {
+                "file_name": item.source_file,
+                # 其他 metadata 其实不需要了，builder 会重新提取
+            } for item in manifest.get_items()}
+            
+            builder = BilingualEPUBBuilder(self.parser.book, self.config)
+            # 临时 Hack: 如果是 chinese 模式，修改 builder 的逻辑 (或者 builder 自动读取 config)
+            # Builder 构造函数读了 config。
+            # 我们需要在 config 里设置 mode 吗？或者 Builder 可以加个 set_mode?
+            # 这里的 config 是全局 config。main.py 里并没有把 args.mode 写入 config['output']。
+            # 我们可以在这里 patch 一下 config。
+            self.config['output']['mode'] = mode # 确保 Builder 知道模式
+            
+            # 注意: 之前的 BilingualBuilder._process_document_content 里写死 bilingual_mode = True
+            # 我需要去修一下 BilingualBuilder，让它读 self.config['output']['mode']
+            
+            result_file = builder.create_bilingual_epub_with_mapping(translation_map, paragraph_map, output_path)
+            
+            self.console.print(f"[green]Done: {result_file}[/green]")
+            return result_file
+            
+        except Exception as e:
+            traceback.print_exc()
+            logger.error(f"Translation failed: {e}")
+            raise
+
+    async def _translate_concurrently(self, chunks: List[List[Any]], manifest: ManifestManager, 
+                                      profile: Dict, mode: str = "bilingual"):
+        """并发翻译核心逻辑 (统一单双语)"""
+        total_chunks = len(chunks)
+        glossary = profile.get('glossary', {})
+        instruction = profile.get('translation_instruction', "")
+        
+        with Progress(
+            SpinnerColumn(), TextColumn("[progress.description]{task.description}"),
+            BarColumn(), TextColumn("{task.percentage:>3.0f}%"), TimeElapsedColumn(),
+            console=self.console
+        ) as progress:
+            task_id = progress.add_task(f"[cyan]Translating...", total=total_chunks)
+            
+            async def worker(chunk, idx):
+                try:
+                    chunk_dicts = [item.to_dict() for item in chunk]
+                    results = None
+                    model_name = self.llm_client.models.get('fast', 'unknown')
+                    
+                    if self.cache:
+                        results = self.cache.get_chunk_translation(chunk_dicts, model=model_name)
+
+                    if not results:
+                        # FIX: Use keyword arguments to avoid positional mismatch (model_type vs mode)
+                        results = await self.llm_client.translate_chunk(
+                            items=chunk, 
+                            glossary=glossary, 
+                            instruction=instruction, 
+                            mode=mode
+                        )
+                        if self.cache and results:
+                            self.cache.save_chunk_translation(chunk_dicts, results, model=model_name)
+
+                    for item in chunk:
+                        if item.global_id in results:
+                            raw_trans = results[item.global_id]
+                            
+                            if "[Error" in raw_trans:
+                                manifest.update_item(item.global_id, None, status="failed", error=raw_trans)
+                                continue
+                            
+                            processed_trans = add_spacing_between_cn_and_en_num(raw_trans)
+                            
+                            # 仅保存译文，还原逻辑外移至所有翻译完成后执行
+                            manifest.update_item(
+                                item.global_id, 
+                                processed_trans,
+                                translation_with_placeholders=processed_trans,
+                                status="translated"
+                            )
+                        else:
+                            manifest.update_item(item.global_id, None, status="failed", error="Translate failed: ID not found in response")
+                except Exception as e:
+                    logger.error(f"Worker {idx} error: {e}")
+                finally:
+                    progress.update(task_id, advance=1)
+            # 并发执行翻译任务
+            tasks = [worker(chunk, i) for i, chunk in enumerate(chunks)]
+            await asyncio.gather(*tasks)
+            
+        # 翻译完成后立即保存，确保数据持久化
+        manifest.save()
+        logger.info("翻译阶段完成，manifest 已保存")
+            
+        # --- 第二阶段：统一进行格式还原与修复 ---
+        logger.info("开始进行格式还原与占位符校验...")
+        await self.process_format_restoration(manifest, mode)
+
+    async def process_format_restoration(self, manifest, mode):
+        """统一处理所有段落的格式还原和修复"""
+        items = manifest.get_items()
+        success_count = 0
+        failed_count = 0
+        repaired_count = 0
+        
+        for item in items:
+            # 仅处理已翻译或之前格式还原失败的项目，或者已完成但缺少还原HTML的项目
+            should_process = (
+                item.status == "translated" or 
+                item.status == "format_error" or
+                (item.status == "completed" and not item.translation_with_original_html)
+            )
+            
+            if not should_process or not item.translation_with_placeholders:
+                continue
+
+
+                
+            processed_trans = item.translation_with_placeholders
+            translation_with_ph = processed_trans
+            restored_html = ""
+            success = False
+            
+            if item.placeholder_map:
+                # 过滤内嵌占位符（排除 _prefix, _suffix）
+                inner_placeholders = {k: v for k, v in item.placeholder_map.items() 
+                                     if not k.startswith("_")}
+                
+                if not inner_placeholders:
+                    # 没有内嵌占位符，清除可能多出的占位符
+                    clean_translation = self.restorer._strip_placeholders(processed_trans)
+                    translation_with_ph = clean_translation
+                    restored_html, success = self.restorer.restore(clean_translation, item.placeholder_map)
+                else:
+                    # 有内嵌占位符，尝试直接还原
+                    restored_html, success = self.restorer.restore(processed_trans, item.placeholder_map)
+                    
+                    if not success:
+                        # 尝试修复逻辑
+                        import re
+                        found_ids = set(re.findall(r'φ(/?\d+)φ', processed_trans))
+                        expected_ids = set(inner_placeholders.keys())
+                        missing_ids = expected_ids - found_ids
+                        
+                        if missing_ids:
+                            logger.warning(f"占位符缺失 (ID: {item.global_id}), 尝试修复: {missing_ids}")
+                            try:
+                                fixed_trans = await self.llm_client.repair_format(
+                                    item.text_with_placeholders, 
+                                    processed_trans,
+                                    missing_ids=missing_ids
+                                )
+                                restored_html_2, success_2 = self.restorer.restore(fixed_trans, item.placeholder_map)
+                                if success_2:
+                                    repaired_count += 1
+                                    translation_with_ph = fixed_trans
+                                    restored_html = restored_html_2
+                                    success = True
+                                else:
+                                    # 尾注补救
+                                    if hasattr(item, 'endnote_anchors') and item.endnote_anchors:
+                                        still_missing = [a for a in item.endnote_anchors 
+                                                        if f"φ{a}φ" not in fixed_trans]
+                                        if still_missing:
+                                            for anchor_id in still_missing:
+                                                fixed_trans += f"φ{anchor_id}φ"
+                                            restored_html_3, success_3 = self.restorer.restore(fixed_trans, item.placeholder_map)
+                                            if success_3:
+                                                translation_with_ph = fixed_trans
+                                                restored_html = restored_html_3
+                                                success = True
+                            except Exception as e:
+                                logger.error(f"修复失败 (ID: {item.global_id}): {e}")
+            else:
+                # 无需还原
+                restored_html = processed_trans
+                success = True
+
+            # 更新 Manifest
+            final_translation = self.restorer._strip_placeholders(translation_with_ph)
+            manifest.update_item(
+                item.global_id, 
+                final_translation,
+                translation_with_placeholders=translation_with_ph,
+                translation_with_original_html=restored_html,
+                status="completed" if success else "format_error"
+            )
+            
+            if success: success_count += 1
+            else: failed_count += 1
+        
+        # 保存还原结果
+        manifest.save()
+        
+        # 统计错误比例
+        total_processed = success_count + failed_count
+        if total_processed > 0:
+            error_rate = failed_count / total_processed
+            logger.info(f"占位符还原完成: 成功 {success_count}, 失败 {failed_count}, 修复 {repaired_count}, 错误率 {error_rate*100:.2f}%")
+            
+            # 超过 1% 错误率，判定为严重问题，中止操作
+            if error_rate > 0.01:
+                error_msg = f"占位符错误率过高 ({error_rate*100:.2f}% > 1%)，检测到严重不可修复问题，中止操作"
+                logger.error(error_msg)
+                raise RuntimeError(error_msg)
+        else:
+            logger.warning("没有处理任何段落")

archive/v0.09/test_build_prompt.py

@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+"""模拟翻译流程，检查 _build_prompt 使用的数据"""
+import sys
+sys.path.insert(0, '.')
+
+from src.manifest_manager import ManifestManager
+from src.text_processor import TextProcessor
+
+# 加载 manifest
+manifest = ManifestManager("cache/manifests/Empire of AI Dreams and Nightmares in Sam Altmans OpenAI (Karen Hao)_chinese_manifest.json")
+manifest.load()
+
+# 创建 chunks (模拟 create_chunks_from_manifest)
+processor = TextProcessor({})
+chunks = processor.create_chunks_from_manifest(manifest, mode="chinese")
+
+print(f"Total chunks: {len(chunks)}")
+
+if chunks:
+    # 取第一个 chunk 的前几个 item
+    first_chunk = chunks[0]
+    print(f"First chunk has {len(first_chunk)} items")
+    
+    for item in first_chunk[:5]:
+        print(f"\n=== ITEM {item.global_id} ===")
+        print(f"  type(item): {type(item)}")
+        print(f"  clean_text: '{item.clean_text[:50]}...'")
+        print(f"  text_with_placeholders: '{item.text_with_placeholders[:50] if item.text_with_placeholders else 'EMPTY'}...'")
+        print(f"  has φ in twp: {'φ' in (item.text_with_placeholders or '')}")
+        print(f"  placeholder_map: {item.placeholder_map}")
+        
+        # 模拟 _build_prompt 的逻辑
+        if item.text_with_placeholders:
+            text = item.text_with_placeholders
+        else:
+            text = item.clean_text
+        prompt_line = f"{item.global_id} [BODY] {text}"
+        print(f"  -> Will send to LLM: '{prompt_line[:80]}...'")
+        print(f"  -> Prompt contains φ: {'φ' in prompt_line}")

archive/v0.09/test_build_prompt2.py

@@ -0,0 +1,33 @@
+#!/usr/bin/env python3
+"""直接检查 manifest 中的 pending items"""
+import sys
+sys.path.insert(0, '.')
+
+from src.manifest_manager import ManifestManager
+
+# 加载 manifest
+manifest = ManifestManager("cache/manifests/Empire of AI Dreams and Nightmares in Sam Altmans OpenAI (Karen Hao)_chinese_manifest.json")
+manifest.load()
+
+# 获取 pending items（这是 create_chunks_from_manifest 内部调用的）
+pending_items = manifest.get_items(status="pending")
+print(f"Pending items: {len(pending_items)}")
+
+# 如果没有 pending，获取所有
+if not pending_items:
+    print("No pending items, getting all...")
+    pending_items = manifest.get_items()
+    print(f"All items: {len(pending_items)}")
+
+# 检查前 10 个 item
+for item in pending_items[:10]:
+    twp = item.text_with_placeholders
+    has_phi = 'φ' in twp if twp else False
+    inner_ph = {k: v for k, v in item.placeholder_map.items() if not k.startswith('_')} if item.placeholder_map else {}
+    
+    print(f"\n{item.global_id}:")
+    print(f"  status: {item.status}")
+    print(f"  clean_text: '{item.clean_text[:40]}...'")
+    print(f"  text_with_placeholders: '{twp[:40] if twp else 'EMPTY'}...'")
+    print(f"  has φ: {has_phi}")
+    print(f"  inner_placeholders: {list(inner_ph.keys())}")

archive/v0.09/test_cache_check.py

@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+"""检查翻译缓存中的结果是否包含占位符"""
+import json
+import os
+
+translations_dir = "cache/translations"
+all_translations = {}
+
+# 遍历缓存文件
+for root, dirs, files in os.walk(translations_dir):
+    for f in files:
+        if f.endswith('.json'):
+            path = os.path.join(root, f)
+            with open(path, 'r') as fp:
+                cache = json.load(fp)
+            if 'translations' in cache:
+                for k, v in cache['translations'].items():
+                    all_translations[k] = {'trans': v, 'file': path}
+
+# 检查 p_00006 (已知有占位符的 item)
+target_id = 'p_00006'
+if target_id in all_translations:
+    data = all_translations[target_id]
+    print(f"=== CACHED TRANSLATION FOR {target_id} ===")
+    print(f"Cache file: {data['file']}")
+    print(f"Translation: {data['trans']}")
+    print(f"Contains φ: {'φ' in data['trans']}")
+else:
+    print(f"{target_id} not found in cache")
+
+# 统计有多少缓存翻译包含 φ
+with_phi = sum(1 for v in all_translations.values() if 'φ' in v['trans'])
+total = len(all_translations)
+print(f"\n=== CACHE STATS ===")
+print(f"Total cached translations: {total}")
+print(f"Translations with φ: {with_phi}")
+print(f"Translations without φ: {total - with_phi}")

archive/v0.09/test_extractor.py

@@ -0,0 +1,31 @@
+import re
+from src.format_extractor import FormatExtractor
+
+extractor = FormatExtractor()
+
+cases = [
+    ("<p>“But what is the goal?” <em>Amodei</em>...</p>", "Quoted text with em"),
+    ("<p>Q. What is artificial intelligence?</p>", "Simple Q&A"),
+    ("<p>Text <i>italic</i> followed by dots...</p>", "Italic with trailing dots"),
+]
+
+for html, desc in cases:
+    print(f"--- Testing: {desc} ---")
+    print(f"HTML: {html}")
+    clean, text_ph, ph_map, p_type, anchors = extractor.extract(html)
+    
+    # Validation logic from FormatExtractor.extract
+    stripped_text = re.sub(r'φ/?[0-9]+φ', '', text_ph)
+    stripped_text = re.sub(r'\s+', ' ', stripped_text).strip()
+    clean_normalized = re.sub(r'\s+', ' ', clean).strip()
+    
+    print(f"Clean:    '{clean_normalized}'")
+    print(f"Stripped: '{stripped_text}'")
+    print(f"Text ph:  '{text_ph}'")
+    print(f"Map:      {ph_map}")
+    
+    if clean_normalized == stripped_text:
+        print("✅ SUCCESS")
+    else:
+        print("❌ FAILED")
+    print()

archive/v0.09/test_extractor_v3.py

@@ -0,0 +1,38 @@
+import re
+from src.format_extractor import FormatExtractor
+
+extractor = FormatExtractor()
+
+cases = [
+    ("<p>“But what is the goal?” <em>Amodei</em>...</p>", "Quoted text with em"),
+    ("<p>Q. What is artificial intelligence?</p>", "Simple Q&A"),
+    ("<p>Text <i>italic</i> followed by dots...</p>", "Italic with trailing dots"),
+]
+
+for html, desc in cases:
+    print(f"--- Testing: {desc} ---")
+    
+    # Simulate how extract() identifies inner_html
+    from bs4 import BeautifulSoup, Tag
+    soup = BeautifulSoup(html, 'html.parser')
+    root = list(soup.children)[0] if list(soup.children) else soup
+    clean_text = root.get_text().strip()
+    clean_text = re.sub(r'\s+', ' ', clean_text).strip()
+    inner_html = root.decode_contents() if isinstance(root, Tag) else str(root)
+    
+    # Call v3 directly
+    text_with_ph, local_map = extractor._smart_extract_v3(inner_html)
+    
+    stripped_text = re.sub(r'φ/?[0-9]+φ', '', text_with_ph)
+    stripped_text = re.sub(r'\s+', ' ', stripped_text).strip()
+    
+    print(f"Clean:    '{clean_text}'")
+    print(f"Stripped: '{stripped_text}'")
+    print(f"Text ph:  '{text_with_ph}'")
+    print(f"Map:      {local_map}")
+    
+    if clean_text == stripped_text:
+        print("✅ SUCCESS")
+    else:
+        print("❌ FAILED")
+    print()

archive/v0.09/test_llm_client_io.py

@@ -0,0 +1,62 @@
+#!/usr/bin/env python3
+"""
+最小化测试脚本：测试 LLMClient 的输入输出
+完全模拟真实调用路径，排除 Translator/TextProcessor 的干扰
+"""
+import sys
+import asyncio
+import json
+import logging
+from loguru import logger
+from src.llm_client import LLMClient
+from src.manifest_manager import ManifestItem
+from src.utils import load_config
+
+# 配置日志输出到控制台
+logger.remove()
+logger.add(sys.stdout, level="DEBUG")
+
+async def test_io():
+    print("=== 初始化 LLMClient ===")
+    config = load_config()
+    # 使用 v3 provider
+    if 'v3' in config['providers']:
+        config['llm'] = config['providers']['v3']
+        print(f"Using provider: v3 (model: {config['llm']['models']['fast']})")
+    
+    client = LLMClient(config)
+    
+    # 构造测试 Item (模拟真实数据)
+    item = ManifestItem(
+        global_id="p_00006",
+        source_file="test.html",
+        original_html="<p>in the name of <span id='page_vi'></span> abundance...</p>",
+        clean_text="in the name of abundance...",
+        text_hash="dummy_hash",
+        tag="p",
+        # 关键：设置 text_with_placeholders
+        text_with_placeholders="in the name of φ1φabundance...",  # 故意不加空格，模拟原始数据
+        placeholder_map={"1": "<span id='page_vi'></span>"},
+        paragraph_type="BODY"
+    )
+    
+    items = [item]
+    mode = "chinese"
+    
+    print("\n=== 1. 测试 _build_prompt 输出 ===")
+    # 直接调用私有方法查看生成的 prompt
+    prompt = client._build_prompt(items, mode=mode)
+    print(f"Generated Prompt:\n{prompt}")
+    print(f"Contains φ1φ: {'φ1φ' in prompt}")
+    
+    print("\n=== 2. 测试 translate_chunk 完整调用 ===")
+    # 这会触发我们之前添加的 ERROR/DEBUG 日志
+    results = await client.translate_chunk(items, mode=mode)
+    
+    print("\n=== 3. 检查结果 ===")
+    trans = results.get("p_00006", "MISSING")
+    print(f"Translation: {trans}")
+    print(f"Contains φ: {'φ' in trans}")
+
+if __name__ == "__main__":
+    asyncio.run(test_io())

archive/v0.09/test_llm_placeholder.py

@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+"""
+测试 LLM 是否正确保留占位符
+"""
+import asyncio
+import json
+from openai import AsyncOpenAI
+
+# 加载配置
+with open("config/config.json", "r") as f:
+    config = json.load(f)
+
+v3_config = config["providers"]["v3"]
+
+# 从 .env 加载 API Key
+import os
+from dotenv import load_dotenv
+load_dotenv()
+api_key = os.getenv("V3_API_KEY") or v3_config.get("api_key", "")
+
+# 加载 prompts
+with open("config/prompts.json", "r") as f:
+    prompts = json.load(f)
+
+system_prompt = prompts["translation"]["system"]
+
+# 测试文本 - 包含占位符
+test_text = """p_00058 [BODY] Also that night, the board and the remaining leadership at the company were holding a series of increasingly hostile meetings. After the all-φ1φhands, the false projection of unity between Sutskever and the other leaders had collapsed. Many of the executives who had sat next to Sutskever during the livestream had been nearly as blindsided as the rest of the staff, having learned of Altman's dismissal moments before it was announced. φ2φRiled up by Sutskever's poor performance, they had demanded to meet with the rest of the board. Roughly a dozen executives, including Murati and Lightcap, had gathered in a conference room at the office."""
+
+async def test_translation():
+    client = AsyncOpenAI(
+        base_url=v3_config["base_url"],
+        api_key=api_key,
+        default_headers=v3_config.get("extra_headers", {})
+    )
+    
+    model = v3_config["models"]["fast"]
+    
+    print("=" * 60)
+    print("SYSTEM PROMPT:")
+    print("=" * 60)
+    print(system_prompt)
+    print()
+    print("=" * 60)
+    print("USER PROMPT:")
+    print("=" * 60)
+    print(test_text)
+    print()
+    print("=" * 60)
+    print(f"Calling LLM ({model})...")
+    print("=" * 60)
+    
+    response = await client.chat.completions.create(
+        model=model,
+        messages=[
+            {"role": "system", "content": system_prompt},
+            {"role": "user", "content": test_text}
+        ],
+        temperature=0.3
+    )
+    
+    result = response.choices[0].message.content
+    
+    print()
+    print("=" * 60)
+    print("LLM RESPONSE:")
+    print("=" * 60)
+    print(result)
+    print()
+    
+    # 检查占位符
+    has_phi1 = "φ1φ" in result
+    has_phi2 = "φ2φ" in result
+    print("=" * 60)
+    print("PLACEHOLDER CHECK:")
+    print(f"  φ1φ present: {has_phi1}")
+    print(f"  φ2φ present: {has_phi2}")
+    print("=" * 60)
+
+if __name__ == "__main__":
+    asyncio.run(test_translation())

archive/v0.09/test_manifest_data.py

@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+"""验证 manifest 数据和 _build_prompt 输出"""
+import json
+
+manifest_file = "cache/manifests/Empire of AI Dreams and Nightmares in Sam Altmans OpenAI (Karen Hao)_chinese_manifest.json"
+with open(manifest_file, 'r') as f:
+    data = json.load(f)
+
+items = data.get('items', [])
+
+# 找一个有占位符的 item
+target = None
+for item in items:
+    if item.get('placeholder_map') and len(item['placeholder_map']) > 0:
+        inner = {k: v for k, v in item['placeholder_map'].items() if not k.startswith('_')}
+        if inner:
+            target = item
+            break
+
+if target:
+    print("=== TARGET ITEM ===")
+    print(f"global_id: {target['global_id']}")
+    print(f"status: {target['status']}")
+    print(f"clean_text: {target['clean_text'][:80]}...")
+    print(f"text_with_placeholders: '{target['text_with_placeholders'][:80]}...'")
+    print(f"placeholder_map: {target['placeholder_map']}")
+    print()
+    
+    # 模拟 _build_prompt 的行为
+    text_with_ph = target['text_with_placeholders']
+    if text_with_ph:
+        prompt_line = f"{target['global_id']} [BODY] {text_with_ph}"
+    else:
+        prompt_line = f"{target['global_id']} [BODY] {target['clean_text']}"
+    
+    print("=== SIMULATED PROMPT LINE ===")
+    print(prompt_line[:150])
+    print()
+    
+    # 检查 text_with_placeholders 是否包含 φ
+    has_phi = 'φ' in (text_with_ph or '')
+    print(f"text_with_placeholders contains φ: {has_phi}")
+else:
+    print("No item with placeholders found")

archive/v0.09/test_manifest_load.py

@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+"""验证 manifest 加载后 ManifestItem 对象的 text_with_placeholders"""
+import sys
+sys.path.insert(0, '.')
+
+from src.manifest_manager import ManifestManager
+
+manifest = ManifestManager("cache/manifests/Empire of AI Dreams and Nightmares in Sam Altmans OpenAI (Karen Hao)_chinese_manifest.json")
+manifest.load()
+
+# 获取有占位符的 item
+items = manifest.get_items(status="pending")
+if not items:
+    items = manifest.get_items()  # 任意状态
+
+target = None
+for item in items:
+    if item.placeholder_map:
+        inner = {k: v for k, v in item.placeholder_map.items() if not k.startswith('_')}
+        if inner:
+            target = item
+            break
+
+if target:
+    print("=== ManifestItem OBJECT ===")
+    print(f"global_id: {target.global_id}")
+    print(f"status: {target.status}")
+    print(f"clean_text: '{target.clean_text[:80]}...'")
+    print(f"text_with_placeholders: '{target.text_with_placeholders[:80] if target.text_with_placeholders else 'EMPTY'}...'")
+    print(f"placeholder_map: {target.placeholder_map}")
+    print()
+    
+    # 检查是否包含 φ
+    twp = target.text_with_placeholders
+    has_phi = 'φ' in twp if twp else False
+    print(f"text_with_placeholders contains φ: {has_phi}")
+    print(f"text_with_placeholders length: {len(twp) if twp else 0}")
+else:
+    print("No target item found with placeholders")

archive/v0.09/test_manifest_to_prompt.py

@@ -0,0 +1,125 @@
+#!/usr/bin/env python3
+"""
+最小测试脚本：测试 Manifest -> TextProcessor -> LLMClient 的数据流
+验证 text_with_placeholders 是否在传递过程中丢失或损坏
+"""
+import sys
+import json
+import asyncio
+from pathlib import Path
+from loguru import logger
+
+# 添加 src 到路径
+sys.path.insert(0, '.')
+
+from src.manifest_manager import ManifestManager, ManifestItem
+from src.text_processor import TextProcessor
+from src.llm_client import LLMClient
+# Mock config
+from src.utils import load_config
+
+# 配置日志
+logger.remove()
+logger.add(sys.stdout, level="DEBUG")
+
+# 1. 创建临时的 Manifest 文件，模拟包含问题的真实数据
+# 模拟一个带有换行符的占位符文本，这是我们怀疑的根源
+mock_manifest_data = {
+    "metadata": {"book_id": "test_book"},
+    "items": [
+        {
+            "global_id": "p_00006",
+            "source_file": "test.html",
+            "original_html": "<p>in the name of <span id='page_vi'></span> abundance</p>",
+            "clean_text": "in the name of abundance",
+            "text_hash": "hash1",
+            "tag": "p",
+            # 模拟包含换行符的情况 (FormatExtractor 之前的问题)
+            "text_with_placeholders": "in the name of \nφ1φ\n abundance", 
+            "placeholder_map": {"1": "<span id='page_vi'></span>"},
+            "paragraph_type": "BODY",
+            "status": "pending"
+        },
+        {
+            "global_id": "p_00058",
+            "source_file": "test.html",
+            "original_html": "<p>all-<span id='a536'></span>hands</p>",
+            "clean_text": "all-hands",
+            "text_hash": "hash2",
+            "tag": "p",
+            # 正常情况
+            "text_with_placeholders": "all-φ1φhands",
+            "placeholder_map": {"1": "<span id='a536'></span>"},
+            "paragraph_type": "BODY",
+            "status": "pending"
+        }
+    ]
+}
+
+manifest_path = Path("cache/test_manifest.json")
+manifest_path.parent.mkdir(parents=True, exist_ok=True)
+with open(manifest_path, 'w') as f:
+    json.dump(mock_manifest_data, f)
+
+print(f"=== Created Mock Manifest at {manifest_path} ===")
+
+async def run_test():
+    # 2. 加载 Manifest
+    manifest = ManifestManager(str(manifest_path))
+    manifest.load()
+    print(f"Loaded {len(manifest.get_items())} items")
+    
+    # 3. 创建 TextProcessor 和 Chunks
+    # Mock config for processor
+    processor = TextProcessor({"translation": {"chunk_size": 1000}})
+    
+    # 这一步会从 manifest 读取 item
+    chunks = processor.create_chunks_from_manifest(manifest, mode="chinese")
+    print(f"Created {len(chunks)} chunks")
+    
+    chunk = chunks[0]
+    print(f"Chunk 0 has {len(chunk)} items")
+    
+    # 4. 模拟 LLMClient 构建 Prompt
+    # 不需要真正的 API key，只需要测试 _build_prompt
+    # 添加 dummy key 和 rate_limits 防止初始化报错
+    client = LLMClient({
+        "providers": {}, 
+        "llm": {
+            "api_key": "dummy_key", 
+            "models": {"fast": "dummy_model", "smart": "dummy_model"},
+            "rate_limits": {"requests_per_minute": 60, "concurrent_requests": 2}
+        }
+    }) 
+    
+    # 强制重新加载 prompts (确保我们使用最新的代码逻辑)
+    # 注意：我们之前修了 _load_prompts，如果 prompts.json 不存在会报错
+    # 这里我们假设 config/prompts.json 存在
+    
+    print("\n=== 构建 Prompt (Mode: Chinese) ===")
+    prompt = client._build_prompt(chunk, mode="chinese")
+    
+    print("-" * 40)
+    print(prompt)
+    print("-" * 40)
+    
+    # 验证关键点
+    print("\n=== 验证结果 ===")
+    
+    # 检查 p_00006
+    # 注意：我们之前修了 FormatExtractor，但那是针对**新提取**的内容。
+    # 这里我们测试的是**从旧 Manifest 读取**的内容。
+    # ManifestManager 读取时并不会自动清理换行符！
+    # 所以如果旧 manifest 里有换行，这里应该能复现出带换行的 prompt。
+    
+    has_p00006 = "p_00006 [BODY] in the name of \nφ1φ\n abundance" in prompt
+    print(f"p_00006 has newlines (bad): {has_p00006}")
+    
+    has_p00006_clean = "p_00006 [BODY] in the name of φ1φ abundance" in prompt
+    print(f"p_00006 is clean (good): {has_p00006_clean}")
+    
+    has_p00058 = "p_00058 [BODY] all-φ1φhands" in prompt
+    print(f"p_00058 is correct: {has_p00058}")
+
+if __name__ == "__main__":
+    asyncio.run(run_test())

archive/v0.09/test_newline_effect.py

@@ -0,0 +1,23 @@
+import re
+from src.format_extractor import FormatExtractor
+
+# 模拟带换行的 HTML
+html_with_newlines = """
+in the name of
+<span id="page_vi"></span>
+abundance...
+"""
+
+extractor = FormatExtractor()
+clean_text, text_with_ph, _, _, _ = extractor.extract(f"<p>{html_with_newlines}</p>")
+
+print(f"Original HTML: {repr(html_with_newlines)}")
+print(f"Clean Text: {repr(clean_text)}")
+print(f"Text with PH: {repr(text_with_ph)}")
+print(f"Has Newline: {'\\n' in text_with_ph}")
+print("-" * 20)
+
+# 模拟 Prompt 构建
+prompt_line = f"p_00006 [BODY] {text_with_ph}"
+print("Prompt Line Preview:")
+print(prompt_line)

archive/v0.09/test_spacing.py

@@ -0,0 +1,16 @@
+from src.utils import add_spacing_between_cn_and_en_num
+
+cases = [
+    "在全φ1φ员会议",
+    "全φ1φ员",
+    "φ1φTable Talkφ/1φ",
+    "测试φ12φ测试",
+    "测试φ/12φ测试"
+]
+
+for text in cases:
+    processed = add_spacing_between_cn_and_en_num(text)
+    print(f"Original: '{text}'")
+    print(f"Processed: '{processed}'")
+    print(f"Changed: {text != processed}")
+    print()

archive/v0.09/test_specific_failure_cases.py

@@ -0,0 +1,124 @@
+#!/usr/bin/env python3
+"""
+针对性测试脚本：复现用户报告的占位符丢失问题
+直接读取 Manifest 中的特定失败 Item (p_00006, p_00011 等)
+调用真实 LLM 进行翻译，并打印完整的 Prompt 和 Response
+"""
+import sys
+import asyncio
+import json
+from pathlib import Path
+from loguru import logger
+
+# 添加 src 到路径
+sys.path.insert(0, '.')
+
+from src.manifest_manager import ManifestManager
+from src.llm_client import LLMClient
+from src.utils import load_config
+
+# 配置日志
+logger.remove()
+logger.add(sys.stdout, level="DEBUG")
+
+async def run_test():
+    print("=== 1. 加载配置 ===")
+    try:
+        config = load_config()
+        # 自动选择配置好的 provider
+        if 'v3' in config['providers'] and config['providers']['v3'].get('api_key'):
+            config['llm'] = config['providers']['v3']
+            print("Using Provider: v3")
+        elif 'openrouter' in config['providers']:
+             config['llm'] = config['providers']['openrouter']
+             print("Using Provider: openrouter")
+        else:
+            print("No valid provider found with API key in config!")
+            return
+    except Exception as e:
+        print(f"Config load failed: {e}")
+        return
+
+    print("\n=== 2. 加载真实 Manifest ===")
+    manifest_dir = Path("cache/manifests")
+    manifest_files = list(manifest_dir.glob("*.json"))
+    if not manifest_files:
+        print("Error: No manifest file found")
+        return
+    
+    # 优先选择包含 "OpenAI" 的那个文件（用户截图）
+    target_manifest = next((f for f in manifest_files if "OpenAI" in f.name), manifest_files[0])
+    print(f"Loading: {target_manifest}")
+    
+    manifest = ManifestManager(str(target_manifest))
+    if not manifest.load():
+        print("Failed to load manifest")
+        return
+
+    # 提取目标失败案例
+    target_ids = ["p_00006", "p_00009", "p_00011", "p_00013"] 
+    # 也包括上下文以免错位 (p_00003 - p_00006)
+    context_ids = ["p_00003", "p_00004", "p_00005", "p_00006"]
+    
+    items_to_test = []
+    
+    # 测试组 1: 上下文错位测试
+    print("\n=== 准备测试组 1: 上下文错位及占位符 (p_00003-00006) ===")
+    group1 = []
+    for uid in context_ids:
+        item = manifest._items_by_id.get(uid)
+        if item:
+            # 强制清空旧翻译，模拟重新翻译
+            item.translation = None
+            item.translation_with_placeholders = None
+            group1.append(item)
+            print(f"Added {uid}: {item.text_with_placeholders}")
+    
+    # 测试组 2: 独立行占位符丢失测试 (p_00009, p_00011)
+    print("\n=== 准备测试组 2: 独立行占位符 (p_00009, p_00011) ===")
+    group2 = []
+    for uid in ["p_00009", "p_00011"]:
+        item = manifest._items_by_id.get(uid)
+        if item:
+            item.translation = None
+            group2.append(item)
+            print(f"Added {uid}: {item.text_with_placeholders}")
+
+    client = LLMClient(config)
+
+    # 执行测试 1
+    if group1:
+        print("\n\n>>> 执行 Group 1 测试 (Context Alignment) <<<")
+        # 打印 Prompt 预览
+        prompt = client._build_prompt(group1, mode="chinese")
+        print("\n[PROMPT PREVIEW]:")
+        print("-" * 20)
+        print(prompt)
+        print("-" * 20)
+        
+        # 调用 LLM
+        print("\n[CALLING LLM]...")
+        results = await client.translate_chunk(group1, mode="chinese")
+        
+        print("\n[RESULTS Group 1]:")
+        for uid, trans in results.items():
+            print(f"{uid}: {trans}")
+            if uid == "p_00006":
+                print(f"  > Contains φ1φ? {'φ1φ' in trans}")
+
+    # 执行测试 2
+    if group2:
+        print("\n\n>>> 执行 Group 2 测试 (Isolated Placeholders) <<<")
+        prompt = client._build_prompt(group2, mode="chinese")
+        print("\n[PROMPT PREVIEW]:")
+        print(prompt) 
+        
+        print("\n[CALLING LLM]...")
+        results = await client.translate_chunk(group2, mode="chinese")
+        
+        print("\n[RESULTS Group 2]:")
+        for uid, trans in results.items():
+            print(f"{uid}: {trans}")
+
+if __name__ == "__main__":
+    asyncio.run(run_test())

archive/v0.09/tests/analyze_placeholder_errors.py

@@ -0,0 +1,257 @@
+#!/usr/bin/env python3
+"""
+占位符错误分析测试脚本
+
+功能：
+1. 使用 v3 provider (config 中配置) 翻译指定章节
+2. 收集所有占位符错误
+3. 输出分析报告
+"""
+
+import asyncio
+import json
+import re
+from pathlib import Path
+from collections import defaultdict
+
+# 添加项目路径
+import sys
+from pathlib import Path
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from src.epub_parser import EPUBParser
+from src.epub_cleaner import EpubCleaner
+from src.text_processor import TextProcessor
+from src.llm_client import LLMClient
+from src.format_restorer import FormatRestorer
+from src.manifest_manager import ManifestManager
+from src.utils import add_spacing_between_cn_and_en_num
+from loguru import logger
+
+
+class PlaceholderAnalyzer:
+    """占位符错误分析器"""
+    
+    def __init__(self, config_path: str = "config/config.json", provider: str = "openrouter"):
+        with open(config_path, 'r') as f:
+            self.config = json.load(f)
+        
+        # 扁平化 provider 配置到 config['llm']
+        providers = self.config.get('providers', {})
+        if provider not in providers:
+            raise ValueError(f"Provider '{provider}' not found. Available: {list(providers.keys())}")
+        self.config['llm'] = providers[provider]
+        print(f"使用 LLM 供应商: {provider} ({self.config['llm'].get('base_url')})")
+        
+        self.llm_client = LLMClient(self.config)
+        self.text_processor = TextProcessor(self.config)
+        self.restorer = FormatRestorer()
+        
+        # 错误收集
+        self.errors = []
+        self.success_count = 0
+        self.total_with_placeholders = 0
+    
+    async def analyze_chapter(self, epub_path: str, chapter_file: str = None):
+        """
+        分析单个章节的占位符处理情况
+        
+        Args:
+            epub_path: EPUB 文件路径
+            chapter_file: 指定章节文件名 (如 'OEBPS/c3Z.xhtml'), 不指定则使用第一个内容章节
+        """
+        # 1. 清理 EPUB
+        cleaner = EpubCleaner()
+        cleaned_path = "cache/manifests/processed_epubs/test_cleaned.epub"
+        Path(cleaned_path).parent.mkdir(parents=True, exist_ok=True)
+        cleaner.clean_epub(epub_path, cleaned_path)
+        
+        # 2. 解析
+        parser = EPUBParser(cleaned_path)
+        content_items = parser.extract_all_content_items()
+        
+        # 3. 选择章节
+        if chapter_file:
+            target_items = [i for i in content_items if i['file_name'] == chapter_file]
+        else:
+            # 默认选择第一个有较多内容的章节
+            target_items = [i for i in content_items if len(i['content']) > 5000][:1]
+        
+        if not target_items:
+            print("未找到目标章节")
+            return
+        
+        target = target_items[0]
+        print(f"\n分析章节: {target['file_name']}")
+        print("=" * 60)
+        
+        # 4. 提取文本
+        manifest = ManifestManager("cache/manifests/test_analysis_manifest.json")
+        manifest.init_manifest(book_id="test", metadata={})
+        self.text_processor.extract_to_manifest(target['content'], target['file_name'], manifest, mode="chinese")
+        manifest.save()
+        
+        # 5. 获取待翻译项
+        items = manifest.get_items(status="pending")
+        print(f"待翻译项: {len(items)}")
+        
+        # 6. 筛选有占位符的项目
+        items_with_ph = [i for i in items if i.placeholder_map and 
+                        any(k for k in i.placeholder_map.keys() if not k.startswith("_"))]
+        self.total_with_placeholders = len(items_with_ph)
+        print(f"含内嵌占位符的项: {self.total_with_placeholders}")
+        
+        # 7. 翻译并分析
+        print("\n开始翻译...")
+        
+        # 分块翻译
+        chunks = self.text_processor.create_chunks_from_manifest(manifest, mode="chinese")
+        
+        for i, chunk in enumerate(chunks):
+            print(f"  处理块 {i+1}/{len(chunks)}...")
+            await self._process_chunk(chunk)
+        
+        # 8. 输出分析报告
+        self._print_report()
+    
+    async def _process_chunk(self, chunk):
+        """处理单个翻译块"""
+        try:
+            results = await self.llm_client.translate_chunk(
+                chunk, 
+                glossary={}, 
+                instruction="",
+                mode="chinese"
+            )
+            
+            for item in chunk:
+                if item.global_id not in results:
+                    continue
+                
+                raw_trans = results[item.global_id]
+                if "[Error" in raw_trans:
+                    continue
+                
+                processed_trans = add_spacing_between_cn_and_en_num(raw_trans)
+                
+                # 检查是否有内嵌占位符
+                inner_ph = {k: v for k, v in (item.placeholder_map or {}).items() 
+                           if not k.startswith("_")}
+                
+                if inner_ph:
+                    # 验证还原
+                    restored, success = self.restorer.restore(processed_trans, item.placeholder_map)
+                    
+                    if not success:
+                        # 记录错误
+                        expected = set(inner_ph.keys())
+                        found = set(re.findall(r'φ(/?\\d+)φ', processed_trans))
+                        missing = expected - found
+                        extra = found - expected
+                        
+                        self.errors.append({
+                            'id': item.global_id,
+                            'text_with_ph': item.text_with_placeholders,
+                            'translation_with_ph': processed_trans,
+                            'placeholder_map': inner_ph,
+                            'missing': list(missing),
+                            'extra': list(extra),
+                            'expected': list(expected),
+                            'found': list(found)
+                        })
+                    else:
+                        self.success_count += 1
+                        
+        except Exception as e:
+            logger.error(f"处理块失败: {e}")
+    
+    def _print_report(self):
+        """输出分析报告"""
+        print("\n" + "=" * 80)
+        print("占位符错误分析报告")
+        print("=" * 80)
+        
+        print(f"\n总计含占位符项: {self.total_with_placeholders}")
+        print(f"成功还原: {self.success_count}")
+        print(f"失败: {len(self.errors)}")
+        if self.total_with_placeholders > 0:
+            success_rate = (self.success_count / self.total_with_placeholders) * 100
+            print(f"成功率: {success_rate:.1f}%")
+        
+        if not self.errors:
+            print("\n🎉 没有占位符错误!")
+            return
+        
+        print("\n" + "-" * 80)
+        print("错误详情")
+        print("-" * 80)
+        
+        # 按错误类型分组
+        missing_only = [e for e in self.errors if e['missing'] and not e['extra']]
+        extra_only = [e for e in self.errors if e['extra'] and not e['missing']]
+        both = [e for e in self.errors if e['missing'] and e['extra']]
+        
+        print(f"\n丢失占位符: {len(missing_only)} 个")
+        print(f"多余占位符: {len(extra_only)} 个")
+        print(f"两者都有: {len(both)} 个")
+        
+        # 详细错误列表
+        print("\n" + "-" * 80)
+        print("详细错误列表 (最多显示 10 个)")
+        print("-" * 80)
+        
+        for i, err in enumerate(self.errors[:10]):
+            print(f"\n[{i+1}] ID: {err['id']}")
+            print(f"    原文 (带占位符): {err['text_with_ph'][:100]}...")
+            print(f"    译文 (带占位符): {err['translation_with_ph'][:100]}...")
+            print(f"    期望占位符: {err['expected']}")
+            print(f"    找到占位符: {err['found']}")
+            print(f"    丢失: {err['missing']}")
+            print(f"    多余: {err['extra']}")
+        
+        # 模式分析
+        print("\n" + "-" * 80)
+        print("错误模式分析")
+        print("-" * 80)
+        
+        # 分析常见的丢失模式
+        all_missing = []
+        for e in self.errors:
+            all_missing.extend(e['missing'])
+        
+        from collections import Counter
+        missing_counter = Counter(all_missing)
+        print("\n最常丢失的占位符:")
+        for ph, count in missing_counter.most_common(5):
+            print(f"  φ{ph}φ: {count} 次")
+        
+        # 保存完整报告到文件
+        report_path = "cache/placeholder_error_report.json"
+        with open(report_path, 'w', encoding='utf-8') as f:
+            json.dump({
+                'summary': {
+                    'total_with_placeholders': self.total_with_placeholders,
+                    'success_count': self.success_count,
+                    'error_count': len(self.errors),
+                    'success_rate': (self.success_count / self.total_with_placeholders * 100) if self.total_with_placeholders > 0 else 0
+                },
+                'errors': self.errors
+            }, f, ensure_ascii=False, indent=2)
+        
+        print(f"\n完整报告已保存到: {report_path}")
+
+
+async def main():
+    import argparse
+    parser = argparse.ArgumentParser(description="占位符错误分析")
+    parser.add_argument("epub", help="EPUB 文件路径")
+    parser.add_argument("--chapter", help="指定章节文件名")
+    args = parser.parse_args()
+    
+    analyzer = PlaceholderAnalyzer()
+    await analyzer.analyze_chapter(args.epub, args.chapter)
+    await analyzer.llm_client.close()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

archive/v0.09/tests/extraction_experiment/__init__.py

@@ -0,0 +1,5 @@
+"""
+文本提取实验模块
+"""
+
+__version__ = "0.1.0"

archive/v0.09/tests/extraction_experiment/analyze_missing.py

@@ -0,0 +1,233 @@
+"""
+缺失文本分析工具
+
+详细分析提取器缺失的文本片段,找出根本原因
+"""
+
+import sys
+from pathlib import Path
+from loguru import logger
+import re
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from ebooklib import epub
+
+sys.path.insert(0, str(Path(__file__).parent))
+from extractors.enhanced_bs4 import EnhancedBS4Extractor
+from extractors.baseline_pandoc import PandocBaseline
+
+
+def analyze_missing_text(epub_path: Path, max_missing_samples: int = 20):
+    """
+    分析缺失的文本片段
+    
+    Args:
+        epub_path: ePub 文件路径
+        max_missing_samples: 最多显示的缺失样本数
+    """
+    print(f"\n{'='*80}")
+    print(f"分析文件: {epub_path.name}")
+    print(f"{'='*80}\n")
+    
+    # 1. 获取 Pandoc 基准
+    pandoc = PandocBaseline()
+    baseline_text = pandoc.extract_from_epub(str(epub_path))
+    
+    if not baseline_text:
+        print("❌ Pandoc 提取失败")
+        return
+    
+    print(f"Pandoc 基准长度: {len(baseline_text):,} 字符\n")
+    
+    # 2. 提取器提取
+    book = epub.read_epub(str(epub_path))
+    html_docs = []
+    for item in book.get_items():
+        if item.get_type() == 9:
+            try:
+                content = item.get_content().decode('utf-8')
+                html_docs.append(content)
+            except:
+                continue
+    
+    combined_html = "\n\n".join(html_docs)
+    
+    extractor = EnhancedBS4Extractor(min_text_length=1, preserve_decorative=True)  # 不过滤短文本
+    items = extractor.extract(combined_html)
+    
+    # 分离内容和装饰性元素
+    content_items = [i for i in items if not i.get('is_decorative') and not i.get('is_navigation')]
+    decorative_items = [i for i in items if i.get('is_decorative')]
+    nav_items = [i for i in items if i.get('is_navigation')]
+    
+    extracted_text = " ".join([item['text'] for item in content_items])
+    
+    print(f"提取器统计:")
+    print(f"  - 内容元素: {len(content_items)}")
+    print(f"  - 装饰性元素: {len(decorative_items)}")
+    print(f"  - 导航元素: {len(nav_items)}")
+    print(f"  - 提取文本长度: {len(extracted_text):,} 字符\n")
+    
+    # 3. 标准化文本
+    def normalize(text):
+        text = text.lower()
+        text = re.sub(r'[^\w\s]', ' ', text)
+        text = re.sub(r'\s+', ' ', text)
+        return text.strip()
+    
+    baseline_norm = normalize(baseline_text)
+    extracted_norm = normalize(extracted_text)
+    
+    # 4. 分词对比
+    baseline_words = baseline_norm.split()
+    extracted_words = set(extracted_norm.split())
+    
+    print(f"词级别对比:")
+    print(f"  - Pandoc 词数: {len(baseline_words):,}")
+    print(f"  - 提取器词数: {len(extracted_words):,}")
+    
+    # 5. 找出缺失的句子
+    print(f"\n{'='*80}")
+    print("分析缺失的文本片段")
+    print(f"{'='*80}\n")
+    
+    # 将 Pandoc 文本分成句子
+    baseline_sentences = re.split(r'[.!?\n]+', baseline_text)
+    baseline_sentences = [s.strip() for s in baseline_sentences if len(s.strip()) > 10]
+    
+    missing_sentences = []
+    for sentence in baseline_sentences:
+        sentence_norm = normalize(sentence)
+        if sentence_norm and sentence_norm not in extracted_norm:
+            # 检查是否有部分匹配
+            words = sentence_norm.split()
+            if len(words) > 3:
+                matched_words = sum(1 for w in words if w in extracted_words)
+                match_ratio = matched_words / len(words)
+                
+                if match_ratio < 0.5:  # 少于50%的词匹配,认为缺失
+                    missing_sentences.append({
+                        'text': sentence[:200],  # 只取前200字符
+                        'length': len(sentence),
+                        'match_ratio': match_ratio
+                    })
+    
+    print(f"发现 {len(missing_sentences)} 个可能缺失的文本片段\n")
+    
+    # 6. 分类缺失原因
+    print(f"{'='*80}")
+    print("缺失片段分类分析")
+    print(f"{'='*80}\n")
+    
+    # 显示样本
+    for i, missing in enumerate(missing_sentences[:max_missing_samples], 1):
+        print(f"--- 缺失片段 {i} ---")
+        print(f"长度: {missing['length']} 字符")
+        print(f"匹配率: {missing['match_ratio']:.1%}")
+        print(f"内容: {missing['text']}")
+        
+        # 尝试分析原因
+        text = missing['text'].lower()
+        reasons = []
+        
+        if any(kw in text for kw in ['copyright', '©', 'isbn', 'publisher', 'published']):
+            reasons.append("📚 可能是版权/出版信息")
+        
+        if any(kw in text for kw in ['table of contents', 'chapter', 'part', 'section']):
+            reasons.append("📑 可能是目录信息")
+        
+        if any(kw in text for kw in ['page', 'pg', 'p.']):
+            reasons.append("📄 可能是页码")
+        
+        if len(missing['text']) < 30:
+            reasons.append("📏 文本过短")
+        
+        if re.match(r'^[0-9\s\-\.]+$', missing['text'].strip()):
+            reasons.append("🔢 纯数字")
+        
+        if not reasons:
+            reasons.append("❓ 未知原因 - 需要进一步分析")
+        
+        print(f"可能原因: {', '.join(reasons)}")
+        print()
+    
+    if len(missing_sentences) > max_missing_samples:
+        print(f"... 还有 {len(missing_sentences) - max_missing_samples} 个缺失片段\n")
+    
+    # 7. 统计缺失原因
+    print(f"{'='*80}")
+    print("缺失原因统计")
+    print(f"{'='*80}\n")
+    
+    reason_counts = {
+        '版权/出版信息': 0,
+        '目录信息': 0,
+        '页码': 0,
+        '文本过短': 0,
+        '纯数字': 0,
+        '未知原因': 0
+    }
+    
+    for missing in missing_sentences:
+        text = missing['text'].lower()
+        
+        if any(kw in text for kw in ['copyright', '©', 'isbn', 'publisher', 'published']):
+            reason_counts['版权/出版信息'] += 1
+        elif any(kw in text for kw in ['table of contents', 'chapter', 'part', 'section']):
+            reason_counts['目录信息'] += 1
+        elif any(kw in text for kw in ['page', 'pg', 'p.']):
+            reason_counts['页码'] += 1
+        elif len(missing['text']) < 30:
+            reason_counts['文本过短'] += 1
+        elif re.match(r'^[0-9\s\-\.]+$', missing['text'].strip()):
+            reason_counts['纯数字'] += 1
+        else:
+            reason_counts['未知原因'] += 1
+    
+    for reason, count in reason_counts.items():
+        if count > 0:
+            percentage = count / len(missing_sentences) * 100
+            print(f"{reason}: {count} 个 ({percentage:.1f}%)")
+    
+    # 8. 建议
+    print(f"\n{'='*80}")
+    print("改进建议")
+    print(f"{'='*80}\n")
+    
+    if reason_counts['文本过短'] > 0:
+        print(f"⚠️ 发现 {reason_counts['文本过短']} 个过短文本被过滤")
+        print("   建议: 移除 min_text_length 限制,提取所有文本\n")
+    
+    if reason_counts['版权/出版信息'] > 0:
+        print(f"📚 发现 {reason_counts['版权/出版信息']} 个版权/出版信息")
+        print("   建议: 这些通常不需要翻译,可以保持过滤\n")
+    
+    if reason_counts['目录信息'] > 0:
+        print(f"📑 发现 {reason_counts['目录信息']} 个目录信息")
+        print("   建议: 目录通常需要翻译,检查是否被错误过滤\n")
+    
+    if reason_counts['未知原因'] > 0:
+        print(f"❓ 发现 {reason_counts['未知原因']} 个未知原因的缺失")
+        print("   建议: 需要详细分析这些片段\n")
+
+
+def main():
+    """主函数"""
+    logger.remove()
+    logger.add(sys.stderr, level="WARNING")  # 只显示警告和错误
+    
+    # 测试一本书
+    test_file = "Gambling Man.epub"
+    epub_path = project_root / "input" / test_file
+    
+    if not epub_path.exists():
+        print(f"文件不存在: {test_file}")
+        return
+    
+    analyze_missing_text(epub_path, max_missing_samples=30)
+
+
+if __name__ == "__main__":
+    main()

archive/v0.09/tests/extraction_experiment/analyze_precise.py

@@ -0,0 +1,198 @@
+"""
+精准缺失文本分析 - 直接对比原始 HTML
+
+不使用 Pandoc,直接分析原始 HTML 中的文本
+"""
+
+import sys
+from pathlib import Path
+from loguru import logger
+from bs4 import BeautifulSoup
+import re
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from ebooklib import epub
+
+sys.path.insert(0, str(Path(__file__).parent))
+from extractors.enhanced_bs4 import EnhancedBS4Extractor
+
+
+def extract_all_text_from_html(html_content: str) -> str:
+    """
+    从 HTML 中提取所有可见文本(包括所有元素)
+    
+    这是"真正的100%"基准
+    """
+    soup = BeautifulSoup(html_content, 'html.parser')
+    
+    # 移除不可见元素
+    for element in soup(['script', 'style', 'meta', 'link']):
+        element.decompose()
+    
+    # 获取所有文本
+    text = soup.get_text(separator=' ', strip=True)
+    
+    # 清理空白
+    text = re.sub(r'\s+', ' ', text)
+    
+    return text.strip()
+
+
+def compare_extraction(epub_path: Path):
+    """对比提取器与真实 HTML 文本"""
+    print(f"\n{'='*80}")
+    print(f"精准文本覆盖率分析: {epub_path.name}")
+    print(f"{'='*80}\n")
+    
+    # 加载 ePub
+    book = epub.read_epub(str(epub_path))
+    
+    # 提取所有 HTML 文档
+    html_docs = []
+    for item in book.get_items():
+        if item.get_type() == 9:
+            try:
+                content = item.get_content().decode('utf-8')
+                html_docs.append({
+                    'name': item.get_name(),
+                    'content': content
+                })
+            except:
+                continue
+    
+    print(f"找到 {len(html_docs)} 个 HTML 文档\n")
+    
+    # 逐个文档分析
+    total_baseline_length = 0
+    total_extracted_length = 0
+    total_missing_length = 0
+    
+    missing_samples = []
+    
+    for doc in html_docs:
+        # 基准: 所有文本
+        baseline_text = extract_all_text_from_html(doc['content'])
+        
+        # 提取器提取
+        extractor = EnhancedBS4Extractor(min_text_length=1, preserve_decorative=True)
+        items = extractor.extract(doc['content'])
+        
+        # 只统计内容元素(不包括装饰性和导航)
+        content_items = [i for i in items if not i.get('is_decorative') and not i.get('is_navigation')]
+        extracted_text = " ".join([item['text'] for item in content_items])
+        
+        # 统计
+        baseline_len = len(baseline_text)
+        extracted_len = len(extracted_text)
+        
+        total_baseline_length += baseline_len
+        total_extracted_length += extracted_len
+        
+        # 找出缺失的文本
+        if baseline_len > 0:
+            coverage = extracted_len / baseline_len
+            
+            if coverage < 0.99:  # 覆盖率 < 99%
+                missing_len = baseline_len - extracted_len
+                total_missing_length += missing_len
+                
+                # 找出具体缺失的片段
+                baseline_words = set(baseline_text.lower().split())
+                extracted_words = set(extracted_text.lower().split())
+                missing_words = baseline_words - extracted_words
+                
+                if missing_words:
+                    missing_samples.append({
+                        'file': doc['name'],
+                        'baseline_length': baseline_len,
+                        'extracted_length': extracted_len,
+                        'coverage': coverage,
+                        'missing_words_count': len(missing_words),
+                        'missing_words_sample': list(missing_words)[:20]
+                    })
+    
+    # 总体统计
+    overall_coverage = total_extracted_length / total_baseline_length if total_baseline_length > 0 else 0
+    
+    print(f"{'='*80}")
+    print("总体统计")
+    print(f"{'='*80}\n")
+    print(f"基准文本总长度: {total_baseline_length:,} 字符")
+    print(f"提取文本总长度: {total_extracted_length:,} 字符")
+    print(f"缺失文本长度: {total_missing_length:,} 字符")
+    print(f"**覆盖率: {overall_coverage:.2%}**\n")
+    
+    # 显示缺失样本
+    if missing_samples:
+        print(f"{'='*80}")
+        print(f"发现 {len(missing_samples)} 个文档存在缺失")
+        print(f"{'='*80}\n")
+        
+        for i, sample in enumerate(missing_samples[:10], 1):
+            print(f"--- 文档 {i}: {sample['file']} ---")
+            print(f"基准长度: {sample['baseline_length']:,} 字符")
+            print(f"提取长度: {sample['extracted_length']:,} 字符")
+            print(f"覆盖率: {sample['coverage']:.2%}")
+            print(f"缺失词数: {sample['missing_words_count']}")
+            print(f"缺失词样本: {', '.join(sample['missing_words_sample'][:10])}")
+            print()
+        
+        if len(missing_samples) > 10:
+            print(f"... 还有 {len(missing_samples) - 10} 个文档\n")
+    else:
+        print("✅ 所有文档覆盖率 ≥ 99%\n")
+    
+    # 详细分析第一个缺失文档
+    if missing_samples:
+        print(f"{'='*80}")
+        print("详细分析第一个缺失文档")
+        print(f"{'='*80}\n")
+        
+        first_missing = missing_samples[0]
+        doc_content = next(d['content'] for d in html_docs if d['name'] == first_missing['file'])
+        
+        # 重新提取
+        baseline_text = extract_all_text_from_html(doc_content)
+        
+        extractor = EnhancedBS4Extractor(min_text_length=1, preserve_decorative=True)
+        items = extractor.extract(doc_content)
+        
+        print(f"文件: {first_missing['file']}\n")
+        print(f"提取了 {len(items)} 个元素:")
+        for item in items[:20]:
+            item_type = ""
+            if item.get('is_decorative'):
+                item_type = " [装饰性]"
+            elif item.get('is_navigation'):
+                item_type = " [导航]"
+            
+            print(f"  - [{item['tag']}] {item['text'][:60]}{item_type}")
+        
+        if len(items) > 20:
+            print(f"  ... 还有 {len(items) - 20} 个元素\n")
+        
+        # 显示原始 HTML 的所有文本
+        print(f"\n原始 HTML 的所有文本 (前 500 字符):")
+        print(baseline_text[:500])
+        print("...\n")
+
+
+def main():
+    """主函数"""
+    logger.remove()
+    logger.add(sys.stderr, level="ERROR")
+    
+    test_file = "Gambling Man.epub"
+    epub_path = project_root / "input" / test_file
+    
+    if not epub_path.exists():
+        print(f"文件不存在: {test_file}")
+        return
+    
+    compare_extraction(epub_path)
+
+
+if __name__ == "__main__":
+    main()

archive/v0.09/tests/extraction_experiment/calibre_cleaner.py

@@ -0,0 +1,268 @@
+"""
+Calibre ePub 清理器
+
+清理 Calibre 生成的冗余 HTML 结构:
+1. 将嵌套的 <div> 转为 <p>
+2. 简化只有单一格式的 <span> (bold, italic)
+3. 移除冗余的 calibre* 类
+4. 合并相同的样式
+"""
+
+from bs4 import BeautifulSoup, Tag, NavigableString
+from typing import Dict, Set
+import re
+from loguru import logger
+
+
+class CalibreHTMLCleaner:
+    """Calibre HTML 清理器"""
+    
+    # 简单格式映射
+    SIMPLE_FORMAT_MAP = {
+        'bold': 'strong',
+        'italic': 'em',
+        'underline': 'u',
+    }
+    
+    def __init__(self):
+        self.stats = {
+            'divs_to_p': 0,
+            'spans_simplified': 0,
+            'classes_removed': 0,
+        }
+    
+    def clean(self, html_content: str) -> str:
+        """
+        清理 HTML
+        
+        Args:
+            html_content: 原始 HTML
+            
+        Returns:
+            清理后的 HTML
+        """
+        soup = BeautifulSoup(html_content, 'html.parser')
+        
+        # 1. 将嵌套的 div 转为 p
+        self._convert_nested_divs_to_p(soup)
+        
+        # 2. 简化格式 span
+        self._simplify_format_spans(soup)
+        
+        # 3. 移除冗余的 span
+        self._remove_redundant_spans(soup)
+        
+        # 4. 清理冗余的 calibre 类
+        self._clean_calibre_classes(soup)
+        
+        logger.info(
+            f"清理完成: div→p {self.stats['divs_to_p']}, "
+            f"span简化 {self.stats['spans_simplified']}, "
+            f"类移除 {self.stats['classes_removed']}"
+        )
+        
+        return str(soup)
+    
+    def _convert_nested_divs_to_p(self, soup: BeautifulSoup):
+        """
+        将嵌套的 div 转为 p
+        
+        策略:
+        - 如果 div 只包含内联元素(span, em, strong等),转为 p
+        - 保留包含块级元素的 div
+        """
+        inline_tags = {'span', 'em', 'strong', 'i', 'b', 'u', 'a', 'br'}
+        
+        for div in soup.find_all('div'):
+            # 检查是否只包含内联元素
+            has_block_children = False
+            for child in div.children:
+                if isinstance(child, Tag):
+                    if child.name not in inline_tags:
+                        has_block_children = True
+                        break
+            
+            # 如果只包含内联元素,转为 p
+            if not has_block_children:
+                div.name = 'p'
+                self.stats['divs_to_p'] += 1
+    
+    def _simplify_format_spans(self, soup: BeautifulSoup):
+        """
+        简化只有单一格式的 span
+        
+        例如:
+        <span class="calibre9"><span class="italic">Text</span></span>
+        → <em>Text</em>
+        """
+        for span in soup.find_all('span'):
+            # 检查 class 属性
+            classes = span.get('class', [])
+            if not classes:
+                continue
+            
+            # 检查是否是简单格式
+            simple_format = None
+            for cls in classes:
+                for format_name, tag_name in self.SIMPLE_FORMAT_MAP.items():
+                    if format_name in cls.lower():
+                        simple_format = tag_name
+                        break
+                if simple_format:
+                    break
+            
+            if simple_format:
+                # 替换为语义化标签
+                new_tag = soup.new_tag(simple_format)
+                
+                # 复制内容
+                for child in list(span.children):
+                    new_tag.append(child)
+                
+                # 替换
+                span.replace_with(new_tag)
+                self.stats['spans_simplified'] += 1
+    
+    def _remove_redundant_spans(self, soup: BeautifulSoup):
+        """
+        移除冗余的 span
+        
+        策略:
+        - 只移除完全没有属性的 span
+        - 保留有 class 的 span(即使是 calibre*)
+        - 确保不丢失任何文本
+        """
+        removed_count = 0
+        
+        # 只遍历一次,更保守
+        for span in soup.find_all('span'):
+            # 只移除完全没有属性的 span
+            if not span.attrs:
+                # 检查是否有文本内容
+                if span.get_text(strip=True):
+                    # 有文本,安全地展开
+                    span.unwrap()
+                    removed_count += 1
+        
+        self.stats['spans_removed'] = removed_count
+    
+    def _remove_empty_elements(self, soup: BeautifulSoup):
+        """
+        移除空元素
+        
+        更谨慎的策略:
+        - 只删除完全没有内容的元素
+        - 保留有文本或图片的元素
+        """
+        removed_count = 0
+        
+        # 只遍历一次
+        for element in soup.find_all():
+            if isinstance(element, Tag):
+                # 检查是否完全为空
+                text = element.get_text(strip=True)
+                has_img = element.find('img') is not None
+                
+                # 只删除既没有文本也没有图片的元素
+                if not text and not has_img:
+                    element.decompose()
+                    removed_count += 1
+        
+        self.stats['empty_removed'] = removed_count
+    
+    def _clean_calibre_classes(self, soup: BeautifulSoup):
+        """
+        清理冗余的 calibre 类
+        
+        策略:
+        - 保留有实际样式的类
+        - 移除纯数字的 calibre 类(如 calibre1, calibre2)
+        """
+        for element in soup.find_all(class_=True):
+            classes = element.get('class', [])
+            if not classes:
+                continue
+            
+            # 过滤掉纯数字的 calibre 类
+            new_classes = []
+            for cls in classes:
+                # 保留非 calibre 类
+                if not cls.startswith('calibre'):
+                    new_classes.append(cls)
+                # 保留有语义的 calibre 类
+                elif any(keyword in cls.lower() for keyword in ['title', 'chapter', 'quote', 'note']):
+                    new_classes.append(cls)
+                else:
+                    self.stats['classes_removed'] += 1
+            
+            if new_classes:
+                element['class'] = new_classes
+            else:
+                # 移除整个 class 属性
+                del element['class']
+
+
+class CalibreEPUBCleaner:
+    """Calibre ePub 清理器"""
+    
+    def __init__(self):
+        self.html_cleaner = CalibreHTMLCleaner()
+    
+    def clean_epub(self, epub_path: str, output_path: str):
+        """
+        清理整个 ePub
+        
+        Args:
+            epub_path: 输入 ePub 路径
+            output_path: 输出 ePub 路径
+        """
+        from ebooklib import epub
+        
+        logger.info(f"开始清理 ePub: {epub_path}")
+        
+        # 加载 ePub
+        book = epub.read_epub(epub_path)
+        
+        # 清理每个 HTML 文档
+        cleaned_count = 0
+        for item in book.get_items():
+            if item.get_type() != 9:  # 不是 HTML
+                continue
+            
+            try:
+                content = item.get_content().decode('utf-8')
+            except:
+                continue
+            
+            # 清理 HTML
+            cleaned_html = self.html_cleaner.clean(content)
+            
+            # 更新内容
+            item.set_content(cleaned_html.encode('utf-8'))
+            cleaned_count += 1
+        
+        # 保存
+        epub.write_epub(output_path, book, {
+            'epub2_guide': False,
+            'epub3_landmark': False,
+            'epub3_pages': False,
+            'spine_direction': True,
+        })
+        
+        logger.info(f"清理完成: 处理了 {cleaned_count} 个文档")
+        logger.info(f"输出: {output_path}")
+
+
+if __name__ == "__main__":
+    import sys
+    from pathlib import Path
+    
+    if len(sys.argv) < 3:
+        print("用法: python calibre_cleaner.py <input.epub> <output.epub>")
+        sys.exit(1)
+    
+    input_path = sys.argv[1]
+    output_path = sys.argv[2]
+    
+    cleaner = CalibreEPUBCleaner()
+    cleaner.clean_epub(input_path, output_path)

archive/v0.09/tests/extraction_experiment/check_css.py

@@ -0,0 +1,78 @@
+"""
+检查 CSS 链接保留情况
+
+对比原始 EPUB 和清理后 EPUB 的 Head 部分
+"""
+
+import sys
+from pathlib import Path
+from bs4 import BeautifulSoup
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from ebooklib import epub
+
+
+def check_css_links(epub_path: Path):
+    """检查 CSS 链接"""
+    print(f"\n{'='*80}")
+    print(f"检查 CSS 链接: {epub_path.name}")
+    print(f"{'='*80}\n")
+    
+    book = epub.read_epub(str(epub_path))
+    
+    count = 0
+    css_count = 0
+    
+    for item in book.get_items():
+        if item.get_type() == 9 and 'dummy_split_002' in item.get_name():
+            content = item.get_content().decode('utf-8')
+            soup = BeautifulSoup(content, 'html.parser')
+            
+            print(f"文档: {item.get_name()}\n")
+            
+            # 检查 head
+            head = soup.find('head')
+            if head:
+                print("Head 内容:")
+                print(head.prettify())
+                
+                links = head.find_all('link', rel='stylesheet')
+                if links:
+                    print(f"\n✅ 找到 {len(links)} 个 CSS 链接")
+                    for link in links:
+                        print(f"  - {link}")
+                else:
+                    print("\n❌ 未找到 CSS 链接")
+                
+                styles = head.find_all('style')
+                if styles:
+                    print(f"\n✅ 找到 {len(styles)} 个 Style 标签")
+                    for style in styles:
+                        print(f"  - {style.get_text()[:50]}...")
+                else:
+                    print("\n❌ 未找到 Style 标签")
+            else:
+                print("❌ 未找到 Head 标签")
+            
+            break
+
+
+def main():
+    original_path = project_root / "input" / "On_China_Henry_Kissinger.epub"
+    cleaned_path = project_root / "test_output" / "On_China_cleaned.epub"
+    bilingual_path = project_root / "test_output" / "On_China_bilingual_test.epub"
+    
+    if original_path.exists():
+        check_css_links(original_path)
+    
+    if cleaned_path.exists():
+        check_css_links(cleaned_path)
+        
+    if bilingual_path.exists():
+        check_css_links(bilingual_path)
+
+
+if __name__ == "__main__":
+    main()

archive/v0.09/tests/extraction_experiment/create_bilingual_final.py

@@ -0,0 +1,124 @@
+"""
+创建完整的双语测试版本
+
+使用 BS4 骨架保留方案,生成保留所有样式的双语 ePub
+"""
+
+import sys
+from pathlib import Path
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from ebooklib import epub
+from loguru import logger
+
+sys.path.insert(0, str(Path(__file__).parent))
+from extractors.bs4_skeleton import BS4SkeletonExtractor
+
+
+def create_bilingual_epub(epub_path: Path, output_path: Path, translate_toc: bool = False):
+    """创建双语测试版本"""
+    print(f"\n{'='*80}")
+    print(f"创建双语测试版本: {epub_path.name}")
+    print(f"目录翻译: {'是' if translate_toc else '否'}")
+    print(f"{'='*80}\n")
+    
+    # 加载 ePub
+    book = epub.read_epub(str(epub_path))
+    
+    # 提取器
+    extractor = BS4SkeletonExtractor(translate_toc=translate_toc)
+    
+    # 统计
+    total_items = 0
+    total_translate = 0
+    total_skip = 0
+    
+    # 处理每个 HTML 文档
+    for item in book.get_items():
+        if item.get_type() != 9:
+            continue
+        
+        try:
+            content = item.get_content().decode('utf-8')
+        except:
+            continue
+        
+        file_name = item.get_name()
+        
+        # 提取
+        items = extractor.extract(content, file_name)
+        
+        if not items:
+            continue
+        
+        # 统计
+        translate_items = [i for i in items if i['should_translate']]
+        skip_items = [i for i in items if not i['should_translate']]
+        
+        total_items += len(items)
+        total_translate += len(translate_items)
+        total_skip += len(skip_items)
+        
+        # 创建翻译映射
+        translation_map = {}
+        for i in items:
+            if i['should_translate']:
+                translation_map[i['text']] = f"{i['text']} [翻译]"
+            elif i['is_decorative']:
+                translation_map[i['text']] = f"{i['text']} [装饰]"
+            else:
+                translation_map[i['text']] = f"{i['text']} [跳过]"
+        
+        # 回填
+        new_content = extractor.backfill(items, translation_map)
+        
+        # 更新 item
+        item.set_content(new_content.encode('utf-8'))
+    
+    # 关键修复: 使用 epub.write_epub 的选项参数
+    # 确保所有资源文件都被保存
+    epub.write_epub(str(output_path), book, {
+        'epub2_guide': False,  # 不生成 guide
+        'epub3_landmark': False,  # 不生成 landmark
+        'epub3_pages': False,  # 不生成 pages
+        'spine_direction': True,  # 保留 spine 方向
+    })
+    
+    # 显示统计
+    print(f"处理统计:")
+    print(f"  - 总元素: {total_items}")
+    print(f"  - 翻译: {total_translate} ({total_translate/total_items*100:.1f}%)")
+    print(f"  - 跳过: {total_skip} ({total_skip/total_items*100:.1f}%)")
+    
+    print(f"\n✅ 双语测试版本已保存: {output_path}")
+    print(f"\n请在 ePub 阅读器中打开检查:")
+    print(f"  1. 样式是否完整保留 (居中、缩进、字体等)")
+    print(f"  2. 翻译是否正确回填")
+    print(f"  3. 是否有遗漏或错位")
+
+
+def main():
+    """主函数"""
+    logger.remove()
+    logger.add(sys.stderr, level="INFO")
+    
+    # 测试文件
+    epub_path = project_root / "input" / "On_China_Henry_Kissinger.epub"
+    
+    if not epub_path.exists():
+        print(f"❌ 文件不存在: {epub_path}")
+        return
+    
+    # 输出目录
+    output_dir = project_root / "test_output"
+    output_dir.mkdir(exist_ok=True)
+    
+    # 生成双语版本 (不翻译目录)
+    output_path = output_dir / "On_China_bilingual_skeleton.epub"
+    create_bilingual_epub(epub_path, output_path, translate_toc=False)
+
+
+if __name__ == "__main__":
+    main()

archive/v0.09/tests/extraction_experiment/debug_cleaner.py

@@ -0,0 +1,53 @@
+"""
+调试 SimpleCleaner
+"""
+
+from bs4 import BeautifulSoup, Tag
+import sys
+
+content = """<?xml version='1.0' encoding='utf-8'?>
+<!DOCTYPE html>
+<html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops" epub:prefix="z3998: http://www.daisy.org/z3998/2012/vocab/structure/#" lang="en" xml:lang="en">
+  <head/>
+  <body><div>
+            <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="100%" height="100%" viewbox="0 0 486 751" preserveaspectratio="none">
+                <image width="486" height="751" xlink:href="cover.jpeg"/>
+            </svg>
+        </div>
+    </body>
+</html>"""
+
+print(f"Input length: {len(content)}")
+
+try:
+    soup = BeautifulSoup(content, 'html.parser')
+    
+    # SimpleCleaner 逻辑复现
+    inline_tags = {'span', 'em', 'strong', 'i', 'b', 'u', 'a', 'br'}
+    
+    divs = list(soup.find_all('div'))
+    print(f"Found {len(divs)} divs")
+    
+    for div in divs:
+        has_block = any(
+            isinstance(c, Tag) and c.name not in inline_tags 
+            for c in div.children
+        )
+        print(f"Div content: {div}")
+        print(f"Has block: {has_block}")
+        
+        if not has_block:
+            print("Converting div to p")
+            div.name = 'p'
+            
+    # 清理 calibre 类
+    elements = list(soup.find_all(class_=True))
+    print(f"Found {len(elements)} elements with class")
+    
+    result = str(soup)
+    print(f"Result length: {len(result)}")
+    print("Result preview:")
+    print(result[:200])
+
+except Exception as e:
+    print(f"Error: {e}")

archive/v0.09/tests/extraction_experiment/debug_parser.py

@@ -0,0 +1,34 @@
+"""
+对比解析器
+"""
+
+from bs4 import BeautifulSoup
+import sys
+
+content = """<?xml version='1.0' encoding='utf-8'?>
+<!DOCTYPE html>
+<html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops" epub:prefix="z3998: http://www.daisy.org/z3998/2012/vocab/structure/#" lang="en" xml:lang="en">
+  <head/>
+  <body><p>Test</p></body>
+</html>"""
+
+print("--- html.parser ---")
+soup = BeautifulSoup(content, 'html.parser')
+print(soup.prettify())
+print("\nHead:", soup.head)
+
+try:
+    print("\n--- lxml-xml ---")
+    soup = BeautifulSoup(content, 'lxml-xml')
+    print(soup.prettify())
+    print("\nHead:", soup.head)
+except Exception as e:
+    print(f"\nlxml-xml error: {e}")
+
+try:
+    print("\n--- lxml ---")
+    soup = BeautifulSoup(content, 'lxml')
+    print(soup.prettify())
+    print("\nHead:", soup.head)
+except Exception as e:
+    print(f"\nlxml error: {e}")

archive/v0.09/tests/extraction_experiment/debug_poem.py

@@ -0,0 +1,45 @@
+"""
+测试诗歌格式问题
+
+分析为什么诗歌格式会丢失
+"""
+
+from bs4 import BeautifulSoup
+
+# 模拟诗歌 HTML
+html = """
+<div class="poem">
+<div class="line1">War is</div>
+<div class="line2">A grave affair of the state;</div>
+<div class="line3">It is a place</div>
+</div>
+"""
+
+print("原始 HTML:")
+print(html)
+print("\n" + "="*80 + "\n")
+
+# 使用当前的提取器
+soup = BeautifulSoup(html, 'html.parser')
+
+# 查找所有 div
+divs = soup.find_all('div')
+print(f"找到 {len(divs)} 个 div:")
+for i, div in enumerate(divs, 1):
+    print(f"{i}. <{div.name} class='{div.get('class')}'> {div.get_text()}")
+
+print("\n" + "="*80 + "\n")
+
+# 问题: 如果我们提取每个 div 的文本
+texts = []
+for div in divs:
+    if div.get('class') and 'line' in str(div.get('class')):
+        texts.append(div.get_text())
+
+print(f"提取的文本: {texts}")
+
+# 如果我们回填时只替换第一个文本节点...
+print("\n问题演示:")
+print("如果把所有文本替换成第一个元素的翻译,会导致:")
+print("  - line1, line2, line3 都变成 'War is [翻译]'")
+print("  - 其他内容丢失!")

archive/v0.09/tests/extraction_experiment/debug_read_diff.py

@@ -0,0 +1,49 @@
+"""
+对比 ebooklib 读取内容与 zipfile 直接读取内容
+"""
+
+import sys
+from pathlib import Path
+from ebooklib import epub
+import zipfile
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+
+def compare_reads(epub_path: Path):
+    print(f"\n对比读取: {epub_path.name}\n")
+    
+    # 1. ZipFile 读取
+    zip_content = {}
+    with zipfile.ZipFile(epub_path, 'r') as zf:
+        for name in zf.namelist():
+            if 'dummy_split_002' in name:
+                print(f"Zip 文件名: {name}")
+                content = zf.read(name).decode('utf-8')
+                zip_content[name] = content
+                print(f"Zip 内容 Head 预览:\n{content[:300]}")
+                break
+                
+    # 2. EbookLib 读取
+    book = epub.read_epub(str(epub_path))
+    for item in book.get_items():
+        if 'dummy_split_002' in item.get_name():
+            print(f"\nItem 文件名: {item.get_name()}")
+            content = item.get_content().decode('utf-8')
+            print(f"EbookLib 内容 Head 预览:\n{content[:300]}")
+            
+            # 对比
+            if zip_content:
+                zip_head = zip_content[list(zip_content.keys())[0]][:300]
+                if zip_head != content[:300]:
+                    print("\n⚠️ 内容不一致!")
+                else:
+                    print("\n✅ 内容一致")
+            break
+
+
+if __name__ == "__main__":
+    epub_path = project_root / "input" / "On_China_Henry_Kissinger.epub"
+    if epub_path.exists():
+        compare_reads(epub_path)

archive/v0.09/tests/extraction_experiment/diagnose_epub.py

@@ -0,0 +1,113 @@
+"""
+诊断 EPUB Manifest 问题
+"""
+
+import sys
+from pathlib import Path
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from ebooklib import epub
+
+
+def diagnose_epub(epub_path: Path):
+    """诊断 EPUB 结构"""
+    print(f"\n{'='*80}")
+    print(f"诊断 EPUB: {epub_path.name}")
+    print(f"{'='*80}\n")
+    
+    try:
+        book = epub.read_epub(str(epub_path))
+    except Exception as e:
+        print(f"❌ 无法读取 EPUB: {e}")
+        return
+    
+    # 获取所有 items
+    all_items = list(book.get_items())
+    
+    print(f"总 items 数: {len(all_items)}\n")
+    
+    # 按类型分组
+    by_type = {}
+    for item in all_items:
+        item_type = item.get_type()
+        if item_type not in by_type:
+            by_type[item_type] = []
+        by_type[item_type].append(item)
+    
+    print("按类型统计:")
+    for item_type, items in sorted(by_type.items()):
+        type_name = {
+            0: 'UNKNOWN',
+            1: 'IMAGE',
+            2: 'STYLE',
+            3: 'SCRIPT',
+            4: 'NAVIGATION',
+            5: 'VECTOR',
+            6: 'FONT',
+            7: 'VIDEO',
+            8: 'AUDIO',
+            9: 'DOCUMENT',
+            10: 'COVER'
+        }.get(item_type, f'TYPE_{item_type}')
+        
+        print(f"  {type_name}: {len(items)}")
+    
+    print()
+    
+    # 检查 spine
+    spine = book.spine
+    print(f"Spine 项数: {len(spine)}\n")
+    
+    # 检查 titlepage
+    print("检查 titlepage 相关项:")
+    titlepage_items = [item for item in all_items if 'titlepage' in item.get_name().lower()]
+    
+    if titlepage_items:
+        print(f"  找到 {len(titlepage_items)} 个 titlepage 项:")
+        for item in titlepage_items:
+            print(f"    - {item.get_name()} (type: {item.get_type()})")
+    else:
+        print("  ❌ 未找到 titlepage 项")
+    
+    print()
+    
+    # 检查 spine 中的引用
+    print("检查 spine 引用:")
+    spine_refs = [ref for ref, _ in spine]
+    
+    for ref in spine_refs[:10]:
+        # 查找对应的 item
+        found = False
+        for item in all_items:
+            if item.get_id() == ref:
+                print(f"  ✓ {ref} -> {item.get_name()}")
+                found = True
+                break
+        
+        if not found:
+            print(f"  ❌ {ref} -> 未找到对应 item")
+    
+    if len(spine_refs) > 10:
+        print(f"  ... 还有 {len(spine_refs) - 10} 个引用")
+    
+    print()
+
+
+def main():
+    """主函数"""
+    
+    # 检查原始清理后的 EPUB
+    cleaned_path = project_root / "test_output" / "On_China_cleaned.epub"
+    if cleaned_path.exists():
+        diagnose_epub(cleaned_path)
+    
+    # 检查生成的双语 EPUB
+    bilingual_path = project_root / "test_output" / "On_China_bilingual_test.epub"
+    if bilingual_path.exists():
+        diagnose_epub(bilingual_path)
+
+
+if __name__ == "__main__":
+    main()

archive/v0.09/tests/extraction_experiment/dom_path_utils.py

@@ -0,0 +1,130 @@
+"""
+DOM 路径工具模块
+
+提供 DOM 路径的生成和查找功能,用于精准定位 HTML 元素
+"""
+
+from bs4 import BeautifulSoup, Tag
+from typing import Optional
+from loguru import logger
+
+
+class DOMPathUtils:
+    """DOM 路径工具类"""
+    
+    @staticmethod
+    def get_dom_path(element: Tag) -> str:
+        """
+        生成元素的唯一 DOM 路径
+        
+        格式: "html>body>div[0]>p[2]"
+        
+        Args:
+            element: BeautifulSoup Tag 对象
+            
+        Returns:
+            DOM 路径字符串
+        """
+        if not isinstance(element, Tag):
+            raise ValueError("element 必须是 BeautifulSoup Tag 对象")
+        
+        path_parts = []
+        current = element
+        
+        while current and current.name:
+            # 跳过非标准标签(如 BeautifulSoup 的 [document])
+            if current.name in ['[document]', 'html']:
+                current = current.parent
+                continue
+            
+            # 获取同名兄弟元素
+            parent = current.parent
+            if parent:
+                siblings = [
+                    sibling for sibling in parent.children 
+                    if isinstance(sibling, Tag) and sibling.name == current.name
+                ]
+                
+                # 找到当前元素在同名兄弟中的索引
+                try:
+                    index = siblings.index(current)
+                except ValueError:
+                    # 如果找不到,使用 0
+                    index = 0
+                
+                path_parts.append(f"{current.name}[{index}]")
+            else:
+                # 根元素
+                if current.name not in ['[document]', 'html']:
+                    path_parts.append(current.name)
+            
+            current = parent
+        
+        # 反转路径(从根到叶)
+        return ">".join(reversed(path_parts))
+    
+    @staticmethod
+    def find_by_path(soup: BeautifulSoup, path: str) -> Optional[Tag]:
+        """
+        通过 DOM 路径查找元素
+        
+        Args:
+            soup: BeautifulSoup 对象
+            path: DOM 路径字符串
+            
+        Returns:
+            找到的元素,如果未找到则返回 None
+        """
+        try:
+            parts = path.split(">")
+            current = soup
+            
+            for part in parts:
+                # 解析标签名和索引
+                if "[" in part:
+                    tag_name, index_str = part.split("[")
+                    index = int(index_str.rstrip("]"))
+                else:
+                    # 根元素可能没有索引
+                    tag_name = part
+                    index = 0
+                
+                # 查找所有同名标签
+                if isinstance(current, BeautifulSoup):
+                    # 从根开始
+                    candidates = [current.find(tag_name)]
+                else:
+                    candidates = current.find_all(tag_name, recursive=False)
+                
+                if not candidates or index >= len(candidates):
+                    logger.warning(f"路径查找失败: {path} (在 {part} 处)")
+                    return None
+                
+                current = candidates[index]
+            
+            return current if isinstance(current, Tag) else None
+            
+        except Exception as e:
+            logger.error(f"路径解析错误 {path}: {e}")
+            return None
+    
+    @staticmethod
+    def validate_path(soup: BeautifulSoup, path: str, original_element: Tag) -> bool:
+        """
+        验证路径是否能正确定位到原始元素
+        
+        Args:
+            soup: BeautifulSoup 对象
+            path: DOM 路径
+            original_element: 原始元素
+            
+        Returns:
+            是否验证成功
+        """
+        found = DOMPathUtils.find_by_path(soup, path)
+        if found is None:
+            return False
+        
+        # 比较元素的文本内容和标签名
+        return (found.name == original_element.name and 
+                found.get_text(strip=True) == original_element.get_text(strip=True))

archive/v0.09/tests/extraction_experiment/extractors/__init__.py

@@ -0,0 +1 @@
+"""提取器模块"""

archive/v0.09/tests/extraction_experiment/extractors/baseline_pandoc.py

@@ -0,0 +1,157 @@
+"""
+Pandoc 基准提取器
+
+使用 pandoc 将 ePub 转换为 Markdown,作为文本提取的参考基准
+"""
+
+import subprocess
+import tempfile
+from pathlib import Path
+from typing import Optional
+from loguru import logger
+
+
+class PandocBaseline:
+    """Pandoc 基准提取器"""
+    
+    def __init__(self):
+        """初始化,检查 pandoc 是否可用"""
+        self.pandoc_available = self._check_pandoc()
+    
+    def _check_pandoc(self) -> bool:
+        """检查 pandoc 是否安装"""
+        try:
+            result = subprocess.run(
+                ['pandoc', '--version'],
+                capture_output=True,
+                text=True,
+                timeout=5
+            )
+            if result.returncode == 0:
+                logger.info(f"Pandoc 可用: {result.stdout.split()[1]}")
+                return True
+        except Exception as e:
+            logger.warning(f"Pandoc 不可用: {e}")
+        
+        return False
+    
+    def extract_from_epub(self, epub_path: str) -> Optional[str]:
+        """
+        使用 pandoc 从 ePub 提取文本
+        
+        Args:
+            epub_path: ePub 文件路径
+            
+        Returns:
+            提取的 Markdown 文本,如果失败返回 None
+        """
+        if not self.pandoc_available:
+            logger.error("Pandoc 不可用,无法提取基准文本")
+            return None
+        
+        epub_path = Path(epub_path)
+        if not epub_path.exists():
+            logger.error(f"ePub 文件不存在: {epub_path}")
+            return None
+        
+        try:
+            # 创建临时输出文件
+            with tempfile.NamedTemporaryFile(mode='w', suffix='.md', delete=False) as tmp:
+                tmp_path = tmp.name
+            
+            # 运行 pandoc
+            cmd = [
+                'pandoc',
+                str(epub_path),
+                '-t', 'markdown',
+                '-o', tmp_path,
+                '--wrap=none'  # 不自动换行
+            ]
+            
+            logger.info(f"运行 pandoc: {' '.join(cmd)}")
+            
+            result = subprocess.run(
+                cmd,
+                capture_output=True,
+                text=True,
+                timeout=60
+            )
+            
+            if result.returncode != 0:
+                logger.error(f"Pandoc 执行失败: {result.stderr}")
+                return None
+            
+            # 读取结果
+            with open(tmp_path, 'r', encoding='utf-8') as f:
+                markdown_text = f.read()
+            
+            # 清理临时文件
+            Path(tmp_path).unlink()
+            
+            logger.info(f"Pandoc 提取成功: {len(markdown_text)} 字符")
+            return markdown_text
+            
+        except subprocess.TimeoutExpired:
+            logger.error("Pandoc 执行超时")
+            return None
+        except Exception as e:
+            logger.error(f"Pandoc 提取失败: {e}")
+            return None
+    
+    def extract_from_html(self, html_content: str) -> Optional[str]:
+        """
+        使用 pandoc 从 HTML 提取文本
+        
+        Args:
+            html_content: HTML 字符串
+            
+        Returns:
+            提取的 Markdown 文本
+        """
+        if not self.pandoc_available:
+            return None
+        
+        try:
+            # 创建临时 HTML 文件
+            with tempfile.NamedTemporaryFile(mode='w', suffix='.html', delete=False, encoding='utf-8') as tmp_html:
+                tmp_html.write(html_content)
+                tmp_html_path = tmp_html.name
+            
+            # 创建临时输出文件
+            with tempfile.NamedTemporaryFile(mode='w', suffix='.md', delete=False) as tmp_md:
+                tmp_md_path = tmp_md.name
+            
+            # 运行 pandoc
+            cmd = [
+                'pandoc',
+                tmp_html_path,
+                '-f', 'html',
+                '-t', 'markdown',
+                '-o', tmp_md_path,
+                '--wrap=none'
+            ]
+            
+            result = subprocess.run(
+                cmd,
+                capture_output=True,
+                text=True,
+                timeout=30
+            )
+            
+            if result.returncode != 0:
+                logger.error(f"Pandoc HTML 转换失败: {result.stderr}")
+                return None
+            
+            # 读取结果
+            with open(tmp_md_path, 'r', encoding='utf-8') as f:
+                markdown_text = f.read()
+            
+            # 清理临时文件
+            Path(tmp_html_path).unlink()
+            Path(tmp_md_path).unlink()
+            
+            return markdown_text
+            
+        except Exception as e:
+            logger.error(f"Pandoc HTML 提取失败: {e}")
+            return None

archive/v0.09/tests/extraction_experiment/extractors/bs4_optimized.py

@@ -0,0 +1,207 @@
+"""
+优化的 BeautifulSoup 提取器
+
+使用 DOM 路径标识系统,实现完整的文本提取和精准回填
+"""
+
+from bs4 import BeautifulSoup, Tag, NavigableString
+from typing import List, Dict, Any, Set
+import re
+from loguru import logger
+import sys
+from pathlib import Path
+
+# 添加父目录到路径
+sys.path.insert(0, str(Path(__file__).parent.parent))
+from dom_path_utils import DOMPathUtils
+
+
+class BS4OptimizedExtractor:
+    """优化的 BS4 提取器"""
+    
+    # 标准块级标签
+    BLOCK_TAGS = [
+        'p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
+        'blockquote', 'li', 'td', 'dd', 'dt', 'figcaption',
+        'section', 'article', 'aside', 'header', 'footer', 'main'
+    ]
+    
+    # 导航相关的 class 关键词
+    NAV_KEYWORDS = [
+        'nav', 'navigation', 'toc', 'menu', 'header', 'footer', 
+        'page-number', 'page-num', 'sidebar'
+    ]
+    
+    def __init__(self, min_text_length: int = 10):
+        """
+        初始化提取器
+        
+        Args:
+            min_text_length: 最小文本长度,过滤过短的文本
+        """
+        self.min_text_length = min_text_length
+        self.path_utils = DOMPathUtils()
+    
+    def extract(self, html_content: str) -> List[Dict[str, Any]]:
+        """
+        从 HTML 中提取所有文本元素
+        
+        Args:
+            html_content: HTML 字符串
+            
+        Returns:
+            提取的元素列表,每个元素包含:
+            - path: DOM 路径
+            - element: BeautifulSoup Tag 对象
+            - text: 清理后的文本
+            - html: 原始 HTML
+            - tag: 标签名
+            - is_navigation: 是否是导航元素
+        """
+        soup = BeautifulSoup(html_content, 'html.parser')
+        
+        # 移除不需要的元素
+        for element in soup(['script', 'style', 'meta', 'link']):
+            element.decompose()
+        
+        items = []
+        processed_ids = set()
+        
+        # 遍历所有块级元素
+        for element in soup.find_all(self.BLOCK_TAGS):
+            elem_id = id(element)
+            
+            # 避免重复处理
+            if elem_id in processed_ids:
+                continue
+            
+            # 检查是否被已处理的父元素包含
+            if self._is_contained_in_processed(element, processed_ids):
+                continue
+            
+            # 提取文本
+            text = self._clean_text(element)
+            
+            # 过滤过短的文本
+            if len(text.strip()) < self.min_text_length:
+                continue
+            
+            # 生成 DOM 路径
+            path = self.path_utils.get_dom_path(element)
+            
+            # 判断是否是导航元素
+            is_nav = self._is_navigation_element(element)
+            
+            items.append({
+                'path': path,
+                'element': element,
+                'text': text,
+                'html': str(element),
+                'tag': element.name,
+                'is_navigation': is_nav
+            })
+            
+            processed_ids.add(elem_id)
+        
+        logger.info(f"提取了 {len(items)} 个文本元素")
+        return items
+    
+    def _is_contained_in_processed(self, element: Tag, processed_ids: Set[int]) -> bool:
+        """检查元素是否被已处理的父元素包含"""
+        for parent in element.parents:
+            if isinstance(parent, Tag) and id(parent) in processed_ids:
+                return True
+        return False
+    
+    def _clean_text(self, element: Tag) -> str:
+        """
+        清理元素文本
+        
+        移除:
+        - 脚注引用
+        - 仅包含数字的 span
+        - 多余空白
+        """
+        # 创建副本避免修改原始元素
+        element_copy = BeautifulSoup(str(element), 'html.parser').find(element.name)
+        if not element_copy:
+            return ""
+        
+        # 移除脚注引用
+        for tag in element_copy.find_all(['sup', 'sub']):
+            tag.decompose()
+        
+        footnote_patterns = re.compile(r'footnote|endnote|reference|note|super|sub', re.I)
+        for tag in element_copy.find_all(['a', 'span', 'div'], class_=footnote_patterns):
+            tag.decompose()
+        
+        # 移除仅包含数字的 span
+        for tag in element_copy.find_all('span'):
+            if re.match(r'^(\[\d+\]|\(\d+\)|\d+)$', tag.get_text().strip()):
+                tag.decompose()
+        
+        text = element_copy.get_text().strip()
+        
+        # 清理残留引用标识
+        text = re.sub(r'(\.|。|,|，)\s*(\[\d+\]|\d+)(?=\s|$)', r'\1', text)
+        text = re.sub(r'\s+', ' ', text)
+        
+        return text
+    
+    def _is_navigation_element(self, element: Tag) -> bool:
+        """判断是否是导航元素"""
+        # 检查元素自身的 class
+        classes = element.get('class', [])
+        class_str = ' '.join(classes).lower() if isinstance(classes, list) else str(classes).lower()
+        
+        if any(keyword in class_str for keyword in self.NAV_KEYWORDS):
+            return True
+        
+        # 检查父元素的 class
+        parent = element.parent
+        if parent and isinstance(parent, Tag):
+            p_classes = parent.get('class', [])
+            p_class_str = ' '.join(p_classes).lower() if isinstance(p_classes, list) else str(p_classes).lower()
+            if any(keyword in p_class_str for keyword in self.NAV_KEYWORDS):
+                return True
+        
+        return False
+    
+    def backfill(self, html_content: str, translation_map: Dict[str, str]) -> str:
+        """
+        使用 DOM 路径精准回填翻译
+        
+        Args:
+            html_content: 原始 HTML
+            translation_map: {dom_path: translation} 映射
+            
+        Returns:
+            回填后的 HTML 字符串
+        """
+        soup = BeautifulSoup(html_content, 'html.parser')
+        
+        success_count = 0
+        fail_count = 0
+        
+        for path, translation in translation_map.items():
+            element = self.path_utils.find_by_path(soup, path)
+            
+            if element is None:
+                logger.warning(f"回填失败: 未找到路径 {path}")
+                fail_count += 1
+                continue
+            
+            # 创建新元素(这里简化处理,实际应根据模式创建)
+            new_tag = soup.new_tag(element.name)
+            new_tag.string = translation
+            
+            # 复制属性
+            for attr, value in element.attrs.items():
+                new_tag[attr] = value
+            
+            # 替换
+            element.replace_with(new_tag)
+            success_count += 1
+        
+        logger.info(f"回填完成: 成功 {success_count}, 失败 {fail_count}")
+        return str(soup)

archive/v0.09/tests/extraction_experiment/extractors/bs4_skeleton.py

@@ -0,0 +1,256 @@
+"""
+BS4 骨架保留提取器
+
+核心思想:
+1. 提取: 保留元素引用,提取纯文本
+2. 回填: 只替换文本节点,保留所有 HTML 结构和属性
+"""
+
+from bs4 import BeautifulSoup, Tag, NavigableString
+from typing import List, Dict, Any
+import re
+from loguru import logger
+
+
+class BS4SkeletonExtractor:
+    """BS4 骨架保留提取器"""
+    
+    SKIP_TRANSLATION_PATTERNS = [
+        r'index\.x?html',
+        r'bibliography\.x?html',
+        r'endnotes?\.x?html',
+        r'footnotes?\.x?html',
+    ]
+    
+    TOC_PATTERNS = [
+        r'nav\.x?html',
+        r'toc\.x?html',
+    ]
+    
+    OTHER_NON_CORE_PATTERNS = [
+        r'copyright\.x?html',
+        r'title\.x?html',
+        r'cover\.x?html',
+    ]
+    
+    BLOCK_TAGS = [
+        'p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
+        'blockquote', 'li', 'td', 'dd', 'dt', 'figcaption',
+        'section', 'article', 'aside', 'header', 'footer', 'main'
+    ]
+    
+    DECORATIVE_PATTERNS = [
+        r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾]+$',
+        r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾\s]+$',
+    ]
+    
+    def __init__(self, translate_toc: bool = False):
+        self.translate_toc = translate_toc
+        self.soup = None  # 保存 soup 引用
+    
+    def extract(self, html_content: str, file_name: str = "") -> List[Dict[str, Any]]:
+        """
+        提取文本,保留元素引用
+        
+        关键: 返回的 items 中包含对原始元素的引用
+        """
+        self.soup = BeautifulSoup(html_content, 'html.parser')
+        
+        # 移除不需要的元素
+        for element in self.soup(['script', 'style', 'meta', 'link']):
+            element.decompose()
+        
+        doc_type = self._classify_document(file_name)
+        
+        items = []
+        processed_ids = set()
+        
+        for element in self.soup.find_all(self.BLOCK_TAGS):
+            elem_id = id(element)
+            
+            if elem_id in processed_ids:
+                continue
+            
+            if self._is_contained_in_processed(element, processed_ids):
+                continue
+            
+            # 提取纯文本
+            text = element.get_text(separator=' ', strip=True)
+            
+            if not text.strip():
+                continue
+            
+            is_decorative = self._is_decorative(text)
+            should_translate = self._should_translate(doc_type, is_decorative)
+            
+            # 关键: 保存元素引用,不是字符串!
+            items.append({
+                'element': element,  # 保存元素引用
+                'text': text,
+                'should_translate': should_translate,
+                'doc_type': doc_type,
+                'is_decorative': is_decorative,
+                'tag': element.name
+            })
+            
+            processed_ids.add(elem_id)
+        
+        logger.info(
+            f"[{doc_type}] 提取 {len(items)} 个元素: "
+            f"翻译 {sum(1 for i in items if i['should_translate'])}"
+        )
+        
+        return items
+    
+    def backfill(self, items: List[Dict[str, Any]], translation_map: Dict[str, str]) -> str:
+        """
+        回填翻译,保留完整的 HTML 结构
+        
+        Args:
+            items: extract() 返回的元素列表
+            translation_map: {original_text: translated_text}
+            
+        Returns:
+            回填后的完整 HTML
+        """
+        success_count = 0
+        
+        for item in items:
+            element = item['element']
+            original_text = item['text']
+            
+            # 查找翻译
+            translation = translation_map.get(original_text)
+            if translation is None:
+                continue
+            
+            # 关键: 只替换文本节点,保留所有子元素和属性
+            self._replace_text_only(element, translation)
+            success_count += 1
+        
+        logger.info(f"回填完成: 成功 {success_count}/{len(translation_map)}")
+        
+        # 返回完整的 HTML
+        return str(self.soup)
+    
+    def _replace_text_only(self, element: Tag, new_text: str):
+        """
+        只替换元素的文本内容,完全保留 HTML 结构
+        
+        关键策略:
+        1. 只处理当前元素,不影响其他元素
+        2. 保留所有子元素(span, em, strong等)
+        3. 只替换直接的文本节点
+        
+        示例:
+        原始: <div><span class="bold">Text</span></div>
+        翻译: "Text [翻译]"
+        结果: <div><span class="bold">Text [翻译]</span></div>
+        """
+        from bs4 import NavigableString, Comment
+        
+        # 检查元素是否有子标签
+        child_tags = [child for child in element.children if isinstance(child, Tag)]
+        
+        if not child_tags:
+            # 情况1: 元素只包含文本,没有子标签
+            # 例如: <div>Simple text</div>
+            element.clear()
+            element.string = new_text
+        else:
+            # 情况2: 元素包含子标签
+            # 例如: <div><span class="bold">Text</span> more text</div>
+            
+            # 策略: 找到最深层的文本节点,替换它
+            # 这样可以保留所有格式标签
+            
+            # 递归查找最深的包含文本的元素
+            deepest = self._find_deepest_text_element(element)
+            
+            if deepest and deepest != element:
+                # 在最深的元素中替换文本
+                deepest.clear()
+                deepest.string = new_text
+            else:
+                # 没有更深的元素,直接替换当前元素的所有内容
+                element.clear()
+                element.string = new_text
+    
+    def _find_deepest_text_element(self, element: Tag) -> Tag:
+        """
+        递归查找最深的包含文本的元素
+        
+        返回包含实际文本内容的最深层元素
+        """
+        from bs4 import NavigableString
+        
+        # 查找所有子标签
+        child_tags = [child for child in element.children if isinstance(child, Tag)]
+        
+        if not child_tags:
+            # 没有子标签,这就是最深的元素
+            return element
+        
+        # 有子标签,递归查找
+        # 优先查找第一个包含文本的子标签
+        for child in child_tags:
+            if child.get_text().strip():
+                return self._find_deepest_text_element(child)
+        
+        # 所有子标签都没有文本,返回当前元素
+        return element
+    
+    def _classify_document(self, file_name: str) -> str:
+        if not file_name:
+            return 'core'
+        
+        file_name_lower = file_name.lower()
+        
+        for pattern in self.SKIP_TRANSLATION_PATTERNS:
+            if re.search(pattern, file_name_lower):
+                return 'skip'
+        
+        for pattern in self.TOC_PATTERNS:
+            if re.search(pattern, file_name_lower):
+                return 'toc'
+        
+        for pattern in self.OTHER_NON_CORE_PATTERNS:
+            if re.search(pattern, file_name_lower):
+                return 'other'
+        
+        return 'core'
+    
+    def _should_translate(self, doc_type: str, is_decorative: bool) -> bool:
+        if is_decorative:
+            return False
+        
+        if doc_type == 'core':
+            return True
+        
+        if doc_type == 'toc':
+            return self.translate_toc
+        
+        return False
+    
+    def _is_contained_in_processed(self, element: Tag, processed_ids: set) -> bool:
+        for parent in element.parents:
+            if isinstance(parent, Tag) and id(parent) in processed_ids:
+                return True
+        return False
+    
+    def _is_decorative(self, text: str) -> bool:
+        text_stripped = text.strip()
+        if not text_stripped or len(text_stripped) > 20:
+            return False
+        
+        for pattern in self.DECORATIVE_PATTERNS:
+            if re.match(pattern, text_stripped):
+                return True
+        
+        unique_chars = set(text_stripped.replace(' ', ''))
+        if len(unique_chars) <= 3:
+            decorative_chars = set('*-_~•◆◇■□▪▫●○◉◎⦿⦾')
+            if unique_chars & decorative_chars:
+                return True
+        
+        return False

archive/v0.09/tests/extraction_experiment/extractors/enhanced_bs4.py

@@ -0,0 +1,276 @@
+"""
+增强的提取器 - 保留装饰性元素
+
+在原有 BS4 优化方案基础上,增强对装饰性符号和特殊元素的提取
+"""
+
+from bs4 import BeautifulSoup, Tag, NavigableString
+from typing import List, Dict, Any, Set
+import re
+from loguru import logger
+import sys
+from pathlib import Path
+
+# 添加父目录到路径
+sys.path.insert(0, str(Path(__file__).parent.parent))
+from dom_path_utils import DOMPathUtils
+
+
+class EnhancedBS4Extractor:
+    """增强的 BS4 提取器 - 保留装饰性元素"""
+    
+    # 标准块级标签
+    BLOCK_TAGS = [
+        'p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
+        'blockquote', 'li', 'td', 'dd', 'dt', 'figcaption',
+        'section', 'article', 'aside', 'header', 'footer', 'main'
+    ]
+    
+    # 可能包含装饰性符号的标签
+    DECORATIVE_TAGS = [
+        'hr',  # 水平线
+        'div',  # 可能包含装饰性符号的 div
+        'p',    # 可能只包含符号的段落
+        'span'  # 装饰性 span
+    ]
+    
+    # 导航相关的 class 关键词
+    NAV_KEYWORDS = [
+        'nav', 'navigation', 'toc', 'menu', 'header', 'footer', 
+        'page-number', 'page-num', 'sidebar'
+    ]
+    
+    # 装饰性符号的正则模式
+    DECORATIVE_PATTERNS = [
+        r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾]+$',  # 纯符号
+        r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾\s]+$',  # 符号+空白
+        r'^[\u2022-\u2027\u2030-\u205E]+$',  # Unicode 装饰符号
+    ]
+    
+    def __init__(self, min_text_length: int = 10, preserve_decorative: bool = True):
+        """
+        初始化提取器
+        
+        Args:
+            min_text_length: 最小文本长度(装饰性元素不受此限制)
+            preserve_decorative: 是否保留装饰性元素
+        """
+        self.min_text_length = min_text_length
+        self.preserve_decorative = preserve_decorative
+        self.path_utils = DOMPathUtils()
+    
+    def extract(self, html_content: str) -> List[Dict[str, Any]]:
+        """
+        从 HTML 中提取所有文本元素,包括装饰性元素
+        
+        Args:
+            html_content: HTML 字符串
+            
+        Returns:
+            提取的元素列表
+        """
+        soup = BeautifulSoup(html_content, 'html.parser')
+        
+        # 移除不需要的元素
+        for element in soup(['script', 'style', 'meta', 'link']):
+            element.decompose()
+        
+        items = []
+        processed_ids = set()
+        
+        # 1. 提取标准块级元素
+        for element in soup.find_all(self.BLOCK_TAGS):
+            elem_id = id(element)
+            
+            if elem_id in processed_ids:
+                continue
+            
+            if self._is_contained_in_processed(element, processed_ids):
+                continue
+            
+            text = self._clean_text(element)
+            
+            # 检查是否是装饰性元素
+            is_decorative = self._is_decorative_element(element, text)
+            
+            # 过滤逻辑
+            if not is_decorative and len(text.strip()) < self.min_text_length:
+                continue
+            
+            # 如果是装饰性元素但不保留,跳过
+            if is_decorative and not self.preserve_decorative:
+                continue
+            
+            path = self.path_utils.get_dom_path(element)
+            is_nav = self._is_navigation_element(element)
+            
+            items.append({
+                'path': path,
+                'element': element,
+                'text': text,
+                'html': str(element),
+                'tag': element.name,
+                'is_navigation': is_nav,
+                'is_decorative': is_decorative
+            })
+            
+            processed_ids.add(elem_id)
+        
+        # 2. 提取 <hr> 等纯装饰性标签
+        if self.preserve_decorative:
+            for hr in soup.find_all('hr'):
+                elem_id = id(hr)
+                if elem_id not in processed_ids:
+                    path = self.path_utils.get_dom_path(hr)
+                    items.append({
+                        'path': path,
+                        'element': hr,
+                        'text': '---',  # 用文本表示水平线
+                        'html': str(hr),
+                        'tag': 'hr',
+                        'is_navigation': False,
+                        'is_decorative': True
+                    })
+                    processed_ids.add(elem_id)
+        
+        logger.info(f"提取了 {len(items)} 个文本元素 (包含 {sum(1 for i in items if i.get('is_decorative'))} 个装饰性元素)")
+        return items
+    
+    def _is_decorative_element(self, element: Tag, text: str) -> bool:
+        """
+        判断元素是否是装饰性元素
+        
+        装饰性元素的特征:
+        1. 只包含符号(如 ***, ---, •••)
+        2. 文本很短但包含特殊 Unicode 符号
+        3. 有特定的 class (如 'separator', 'divider')
+        """
+        # 检查 class
+        classes = element.get('class', [])
+        class_str = ' '.join(classes).lower() if isinstance(classes, list) else str(classes).lower()
+        
+        decorative_classes = ['separator', 'divider', 'ornament', 'decoration', 'break']
+        if any(dc in class_str for dc in decorative_classes):
+            return True
+        
+        # 检查文本是否匹配装饰性模式
+        text_stripped = text.strip()
+        if not text_stripped:
+            return False
+        
+        for pattern in self.DECORATIVE_PATTERNS:
+            if re.match(pattern, text_stripped):
+                return True
+        
+        # 检查是否只包含少量重复字符
+        if len(text_stripped) <= 20:
+            unique_chars = set(text_stripped.replace(' ', ''))
+            if len(unique_chars) <= 3:  # 只有1-3种不同字符
+                # 检查是否是常见装饰符号
+                decorative_chars = set('*-_~•◆◇■□▪▫●○◉◎⦿⦾')
+                if unique_chars & decorative_chars:
+                    return True
+        
+        return False
+    
+    def _is_contained_in_processed(self, element: Tag, processed_ids: Set[int]) -> bool:
+        """检查元素是否被已处理的父元素包含"""
+        for parent in element.parents:
+            if isinstance(parent, Tag) and id(parent) in processed_ids:
+                return True
+        return False
+    
+    def _clean_text(self, element: Tag) -> str:
+        """
+        清理元素文本
+        
+        注意: 对于装饰性元素,保留原始符号
+        """
+        # 创建副本
+        element_copy = BeautifulSoup(str(element), 'html.parser').find(element.name)
+        if not element_copy:
+            return ""
+        
+        # 移除脚注引用(但保留装饰性符号)
+        for tag in element_copy.find_all(['sup', 'sub']):
+            tag.decompose()
+        
+        footnote_patterns = re.compile(r'footnote|endnote|reference|note|super|sub', re.I)
+        for tag in element_copy.find_all(['a', 'span', 'div'], class_=footnote_patterns):
+            tag.decompose()
+        
+        # 移除仅包含数字的 span
+        for tag in element_copy.find_all('span'):
+            if re.match(r'^(\[\d+\]|\(\d+\)|\d+)$', tag.get_text().strip()):
+                tag.decompose()
+        
+        text = element_copy.get_text().strip()
+        
+        # 清理残留引用标识
+        text = re.sub(r'(\.|。|,|，)\s*(\[\d+\]|\d+)(?=\s|$)', r'\1', text)
+        
+        # 对于非装饰性文本,压缩空白
+        # 对于装饰性文本,保留原样
+        if not self._is_decorative_element(element, text):
+            text = re.sub(r'\s+', ' ', text)
+        
+        return text
+    
+    def _is_navigation_element(self, element: Tag) -> bool:
+        """判断是否是导航元素"""
+        classes = element.get('class', [])
+        class_str = ' '.join(classes).lower() if isinstance(classes, list) else str(classes).lower()
+        
+        if any(keyword in class_str for keyword in self.NAV_KEYWORDS):
+            return True
+        
+        parent = element.parent
+        if parent and isinstance(parent, Tag):
+            p_classes = parent.get('class', [])
+            p_class_str = ' '.join(p_classes).lower() if isinstance(p_classes, list) else str(p_classes).lower()
+            if any(keyword in p_class_str for keyword in self.NAV_KEYWORDS):
+                return True
+        
+        return False
+    
+    def backfill(self, html_content: str, translation_map: Dict[str, str]) -> str:
+        """
+        使用 DOM 路径精准回填翻译
+        
+        对于装饰性元素,保持原样不翻译
+        """
+        soup = BeautifulSoup(html_content, 'html.parser')
+        
+        success_count = 0
+        fail_count = 0
+        decorative_kept = 0
+        
+        for path, translation in translation_map.items():
+            element = self.path_utils.find_by_path(soup, path)
+            
+            if element is None:
+                logger.warning(f"回填失败: 未找到路径 {path}")
+                fail_count += 1
+                continue
+            
+            # 检查是否是装饰性元素
+            original_text = element.get_text().strip()
+            if self._is_decorative_element(element, original_text):
+                # 装饰性元素保持原样
+                decorative_kept += 1
+                continue
+            
+            # 创建新元素
+            new_tag = soup.new_tag(element.name)
+            new_tag.string = translation
+            
+            # 复制属性
+            for attr, value in element.attrs.items():
+                new_tag[attr] = value
+            
+            # 替换
+            element.replace_with(new_tag)
+            success_count += 1
+        
+        logger.info(f"回填完成: 成功 {success_count}, 失败 {fail_count}, 装饰性元素保留 {decorative_kept}")
+        return str(soup)

archive/v0.09/tests/extraction_experiment/extractors/final_extractor.py

@@ -0,0 +1,319 @@
+"""
+最终版智能提取器
+
+明确的翻译策略:
+1. 正文: 100% 翻译
+2. 目录: 全翻译或全不翻译 (根据配置)
+3. 索引/参考文献/尾注: 明确不翻译
+4. 装饰性元素: 不翻译
+"""
+
+from bs4 import BeautifulSoup, Tag
+from typing import List, Dict, Any, Set
+import re
+from loguru import logger
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+from dom_path_utils import DOMPathUtils
+
+
+class FinalExtractor:
+    """最终版智能提取器"""
+    
+    # 明确不翻译的文档
+    SKIP_TRANSLATION_PATTERNS = [
+        r'index\.x?html',         # 索引
+        r'bibliography\.x?html',  # 参考文献
+        r'endnotes?\.x?html',     # 尾注
+        r'footnotes?\.x?html',    # 脚注
+    ]
+    
+    # 目录文档 (可配置是否翻译)
+    TOC_PATTERNS = [
+        r'nav\.x?html',
+        r'toc\.x?html',
+    ]
+    
+    # 其他非核心文档 (通常不翻译)
+    OTHER_NON_CORE_PATTERNS = [
+        r'copyright\.x?html',
+        r'title\.x?html',
+        r'cover\.x?html',
+    ]
+    
+    BLOCK_TAGS = [
+        'p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
+        'blockquote', 'li', 'td', 'dd', 'dt', 'figcaption',
+        'section', 'article', 'aside', 'header', 'footer', 'main'
+    ]
+    
+    DECORATIVE_PATTERNS = [
+        r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾]+$',
+        r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾\s]+$',
+        r'^[\u2022-\u2027\u2030-\u205E]+$',
+    ]
+    
+    def __init__(self, translate_toc: bool = False, preserve_decorative: bool = True):
+        """
+        初始化提取器
+        
+        Args:
+            translate_toc: 是否翻译目录 (默认不翻译)
+            preserve_decorative: 是否保留装饰性元素
+        """
+        self.translate_toc = translate_toc
+        self.preserve_decorative = preserve_decorative
+        self.path_utils = DOMPathUtils()
+    
+    def extract(self, html_content: str, file_name: str = "") -> List[Dict[str, Any]]:
+        """
+        提取文本元素
+        
+        每个 DOM 节点作为一个独立单位,不合并
+        """
+        soup = BeautifulSoup(html_content, 'html.parser')
+        
+        # 移除不需要的元素
+        for element in soup(['script', 'style', 'meta', 'link']):
+            element.decompose()
+        
+        # 判断文档类型
+        doc_type = self._classify_document(file_name)
+        
+        items = []
+        processed_ids = set()
+        
+        # 遍历所有块级元素
+        for element in soup.find_all(self.BLOCK_TAGS):
+            elem_id = id(element)
+            
+            if elem_id in processed_ids:
+                continue
+            
+            if self._is_contained_in_processed(element, processed_ids):
+                continue
+            
+            # 提取文本 (100% 完整)
+            text = element.get_text(separator=' ', strip=True)
+            
+            if not text.strip():
+                continue
+            
+            # 判断是否装饰性
+            is_decorative = self._is_decorative_element(element, text)
+            
+            # 生成路径
+            path = self.path_utils.get_dom_path(element)
+            
+            # 决定是否翻译
+            should_translate = self._should_translate(doc_type, is_decorative)
+            
+            items.append({
+                'path': path,
+                'element': element,
+                'text': text,
+                'html': str(element),
+                'tag': element.name,
+                'is_decorative': is_decorative,
+                'doc_type': doc_type,
+                'should_translate': should_translate,
+                'file_name': file_name
+            })
+            
+            processed_ids.add(elem_id)
+        
+        # 添加 <hr>
+        if self.preserve_decorative:
+            for hr in soup.find_all('hr'):
+                elem_id = id(hr)
+                if elem_id not in processed_ids:
+                    path = self.path_utils.get_dom_path(hr)
+                    items.append({
+                        'path': path,
+                        'element': hr,
+                        'text': '---',
+                        'html': str(hr),
+                        'tag': 'hr',
+                        'is_decorative': True,
+                        'doc_type': doc_type,
+                        'should_translate': False,
+                        'file_name': file_name
+                    })
+                    processed_ids.add(elem_id)
+        
+        # 统计
+        translate_count = sum(1 for i in items if i['should_translate'])
+        skip_count = sum(1 for i in items if not i['should_translate'] and not i['is_decorative'])
+        decorative_count = sum(1 for i in items if i['is_decorative'])
+        
+        logger.info(
+            f"[{doc_type}] 提取 {len(items)} 个元素: "
+            f"翻译 {translate_count}, 跳过 {skip_count}, 装饰 {decorative_count}"
+        )
+        
+        return items
+    
+    def _classify_document(self, file_name: str) -> str:
+        """
+        分类文档类型
+        
+        Returns:
+            'core' - 核心正文
+            'toc' - 目录
+            'skip' - 明确跳过 (索引/参考文献/尾注)
+            'other' - 其他非核心
+        """
+        if not file_name:
+            return 'core'
+        
+        file_name_lower = file_name.lower()
+        
+        # 检查是否是明确跳过的
+        for pattern in self.SKIP_TRANSLATION_PATTERNS:
+            if re.search(pattern, file_name_lower):
+                return 'skip'
+        
+        # 检查是否是目录
+        for pattern in self.TOC_PATTERNS:
+            if re.search(pattern, file_name_lower):
+                return 'toc'
+        
+        # 检查其他非核心
+        for pattern in self.OTHER_NON_CORE_PATTERNS:
+            if re.search(pattern, file_name_lower):
+                return 'other'
+        
+        return 'core'
+    
+    def _should_translate(self, doc_type: str, is_decorative: bool) -> bool:
+        """
+        决定是否翻译
+        
+        规则:
+        1. 装饰性: 不翻译
+        2. core: 翻译
+        3. toc: 根据配置
+        4. skip: 不翻译
+        5. other: 不翻译
+        """
+        if is_decorative:
+            return False
+        
+        if doc_type == 'core':
+            return True
+        
+        if doc_type == 'toc':
+            return self.translate_toc
+        
+        # skip 和 other 都不翻译
+        return False
+    
+    def _is_contained_in_processed(self, element: Tag, processed_ids: Set[int]) -> bool:
+        """检查元素是否被已处理的父元素包含"""
+        for parent in element.parents:
+            if isinstance(parent, Tag) and id(parent) in processed_ids:
+                return True
+        return False
+    
+    def _is_decorative_element(self, element: Tag, text: str) -> bool:
+        """判断是否是装饰性元素"""
+        classes = element.get('class', [])
+        class_str = ' '.join(classes).lower() if isinstance(classes, list) else str(classes).lower()
+        
+        decorative_classes = ['separator', 'divider', 'ornament', 'decoration', 'break']
+        if any(dc in class_str for dc in decorative_classes):
+            return True
+        
+        text_stripped = text.strip()
+        if not text_stripped:
+            return False
+        
+        for pattern in self.DECORATIVE_PATTERNS:
+            if re.match(pattern, text_stripped):
+                return True
+        
+        if len(text_stripped) <= 20:
+            unique_chars = set(text_stripped.replace(' ', ''))
+            if len(unique_chars) <= 3:
+                decorative_chars = set('*-_~•◆◇■□▪▫●○◉◎⦿⦾')
+                if unique_chars & decorative_chars:
+                    return True
+        
+        return False
+    
+    def backfill(self, html_content: str, translation_map: Dict[str, str]) -> str:
+        """
+        精准回填翻译
+        
+        保留原始 HTML 结构和样式,只替换文本内容
+        """
+        soup = BeautifulSoup(html_content, 'html.parser')
+        
+        success_count = 0
+        fail_count = 0
+        
+        for path, translation in translation_map.items():
+            element = self.path_utils.find_by_path(soup, path)
+            
+            if element is None:
+                logger.warning(f"回填失败: 未找到路径 {path}")
+                fail_count += 1
+                continue
+            
+            # 保留原始元素结构,只替换文本节点
+            self._replace_text_nodes(element, translation)
+            success_count += 1
+        
+        logger.info(f"回填完成: 成功 {success_count}, 失败 {fail_count}")
+        return str(soup)
+    
+    def _replace_text_nodes(self, element: Tag, new_text: str):
+        """
+        智能替换元素中的文本节点,完全保留 HTML 结构
+        
+        策略:
+        1. 如果元素只包含纯文本(无子标签),直接替换
+        2. 如果元素包含子标签,递归查找并替换所有文本节点
+        3. 保留所有属性、class、style 等
+        """
+        from bs4 import NavigableString
+        
+        # 检查是否有子标签
+        child_tags = [child for child in element.children if isinstance(child, Tag)]
+        
+        if not child_tags:
+            # 只有文本节点,直接替换
+            element.clear()
+            element.string = new_text
+        else:
+            # 有子标签,需要智能处理
+            # 策略: 找到所有文本节点,用新文本替换
+            self._replace_all_text_nodes(element, new_text)
+    
+    def _replace_all_text_nodes(self, element: Tag, new_text: str):
+        """
+        递归替换元素中的所有文本节点
+        
+        保留所有子元素和属性,只替换文本内容
+        """
+        from bs4 import NavigableString
+        
+        # 收集所有文本节点
+        text_nodes = []
+        for child in element.descendants:
+            if isinstance(child, NavigableString) and not isinstance(child, (type(None),)):
+                # 跳过空白文本
+                if child.strip():
+                    text_nodes.append(child)
+        
+        if not text_nodes:
+            # 没有文本节点,直接设置
+            element.string = new_text
+            return
+        
+        # 简化策略: 清空所有内容,保留结构,设置新文本
+        # 这会丢失内部格式,但保留外层容器的所有属性
+        element.clear()
+        element.string = new_text

archive/v0.09/tests/extraction_experiment/extractors/fine_grained.py

@@ -0,0 +1,232 @@
+"""
+细粒度提取器
+
+策略: 提取所有 <p> 元素,每个独立处理
+"""
+
+from bs4 import BeautifulSoup, Tag, NavigableString
+from typing import List, Dict, Any, Tuple
+import re
+from loguru import logger
+
+
+class FineGrainedExtractor:
+    """细粒度提取器 - 每个 p 元素独立提取"""
+    
+    SKIP_TRANSLATION_PATTERNS = [
+        r'index\.x?html',
+        r'bibliography\.x?html',
+        r'endnotes?\.x?html',
+        r'footnotes?\.x?html',
+    ]
+    
+    TOC_PATTERNS = [
+        r'nav\.x?html',
+        r'toc\.x?html',
+    ]
+    
+    def __init__(self, translate_toc: bool = False):
+        self.translate_toc = translate_toc
+        self.soup = None
+    
+    def extract(self, html_content: str, file_name: str = "") -> List[Dict[str, Any]]:
+        """
+        细粒度提取: 每个 <p> 和标题元素独立提取
+        
+        关键: 不管嵌套,所有 <p>, h1-h6 都提取
+        """
+        self.soup = BeautifulSoup(html_content, 'html.parser')
+        
+        # 移除不需要的元素
+        for element in self.soup(['script', 'style', 'meta', 'link']):
+            element.decompose()
+        
+        doc_type = self._classify_document(file_name)
+        
+        items = []
+        
+        # 提取所有文本块元素
+        target_tags = ['p', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6']
+        for p in self.soup.find_all(target_tags):
+            # 提取文本
+            text = p.get_text(separator=' ', strip=True)
+            
+            if not text.strip():
+                continue
+            
+            # 收集所有文本节点
+            text_nodes = self._collect_text_nodes(p)
+            
+            if not text_nodes:
+                continue
+            
+            is_decorative = self._is_decorative(text)
+            should_translate = self._should_translate(doc_type, is_decorative, text)
+            
+            items.append({
+                'element': p,
+                'text': text,
+                'text_nodes': text_nodes,
+                'should_translate': should_translate,
+                'doc_type': doc_type,
+                'is_decorative': is_decorative,
+                'tag': p.name
+            })
+        
+        logger.info(
+            f"[{doc_type}] 提取 {len(items)} 个元素 (p, h1-h6): "
+            f"翻译 {sum(1 for i in items if i['should_translate'])}"
+        )
+        
+        return items
+    
+    def _collect_text_nodes(self, element: Tag) -> List[Tuple[NavigableString, str]]:
+        """
+        收集元素中的所有文本节点
+        """
+        text_nodes = []
+        
+        for descendant in element.descendants:
+            if isinstance(descendant, NavigableString):
+                # 跳过注释
+                if isinstance(descendant, type(element)):
+                    continue
+                
+                text = str(descendant).strip()
+                if text:
+                    text_nodes.append((descendant, text))
+        
+        return text_nodes
+    
+    def backfill(self, items: List[Dict[str, Any]], translation_map: Dict[str, str], bilingual: bool = True) -> str:
+        """
+        回填翻译
+        
+        Args:
+            items: 提取的元素列表
+            translation_map: 翻译映射 {原文: 译文}
+            bilingual: 是否生成双语版本 (True: 保留原文+译文, False: 只保留译文)
+        """
+        success_count = 0
+        
+        for item in items:
+            original_text = item['text']
+            element = item['element']
+            
+            # 查找翻译
+            translation = translation_map.get(original_text)
+            if translation is None:
+                continue
+            
+            if bilingual:
+                # 双语模式: 在元素末尾添加译文
+                # 创建一个新的标签用于译文 (使用相同的标签名, 如 p, h1, h2...)
+                translation_p = self.soup.new_tag(element.name)
+                
+                # 1. 继承 class
+                original_classes = element.get('class', [])
+                if original_classes:
+                    # 复制列表以防引用修改
+                    translation_p['class'] = list(original_classes) + ['translation']
+                else:
+                    translation_p['class'] = ['translation']
+                    
+                # 2. 继承 style (如果有)
+                original_style = element.get('style')
+                if original_style:
+                    translation_p['style'] = original_style
+                    
+                # 3. 设置内容
+                translation_p.string = translation
+                
+                # 在原始元素后插入译文
+                element.insert_after(translation_p)
+            else:
+                # 纯译文模式: 替换所有文本节点
+                text_nodes = item['text_nodes']
+                if text_nodes:
+                    text_nodes[0][0].replace_with(translation)
+                    
+                    for node, _ in text_nodes[1:]:
+                        try:
+                            node.replace_with('')
+                        except:
+                            pass  # 节点可能已被移除
+            
+            success_count += 1
+        
+        logger.info(f"回填完成: 成功 {success_count}/{len(translation_map)}")
+        return str(self.soup)
+    
+    def _classify_document(self, file_name: str) -> str:
+        if not file_name:
+            return 'core'
+        
+        file_name_lower = file_name.lower()
+        
+        for pattern in self.SKIP_TRANSLATION_PATTERNS:
+            if re.search(pattern, file_name_lower):
+                return 'skip'
+        
+        for pattern in self.TOC_PATTERNS:
+            if re.search(pattern, file_name_lower):
+                return 'toc'
+        
+        return 'core'
+    
+    def _should_translate(self, doc_type: str, is_decorative: bool, text: str = "") -> bool:
+        if is_decorative:
+            return False
+            
+        # 检查是否为罗马数字 (通常是章节号: I, II, III...)
+        if self._is_roman_numeral(text):
+            return False
+        
+        if doc_type == 'core':
+            return True
+        
+        if doc_type == 'toc':
+            return self.translate_toc
+        
+        return False
+    
+    def _is_roman_numeral(self, text: str) -> bool:
+        """检查是否为罗马数字 (I, II, III, IV, V...)"""
+        text = text.strip().upper()
+        if not text:
+            return False
+        # 简单正则，覆盖常见直到 3999
+        # ^M{0,4}(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})$
+        # 注意: 避免匹配空字符串 (已经由 if not text 处理)
+        pattern = re.compile(r"^M{0,4}(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})$")
+        return bool(pattern.match(text))
+
+    def _is_decorative(self, text: str) -> bool:
+        text_stripped = text.strip()
+        if not text_stripped:
+            return False
+            
+        # 增强: 如果只包含非字母数字字符 (标点, 符号, 分隔线等), 视为装饰性
+        # 这将覆盖 ***, ---, ..., _____, —— 等
+        if not any(c.isalnum() for c in text_stripped):
+            return True
+            
+        if len(text_stripped) > 20:
+            return False
+        
+        decorative_patterns = [
+            r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾]+$',
+            r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾\s]+$',
+        ]
+        
+        for pattern in decorative_patterns:
+            if re.match(pattern, text_stripped):
+                return True
+        
+        unique_chars = set(text_stripped.replace(' ', ''))
+        if len(unique_chars) <= 3:
+            decorative_chars = set('*-_~•◆◇■□▪▫●○◉◎⦿⦾')
+            if unique_chars & decorative_chars:
+                return True
+        
+        return False

archive/v0.09/tests/extraction_experiment/extractors/lxml_xpath.py

@@ -0,0 +1,188 @@
+"""
+lxml XPath 提取器
+
+使用 lxml 的 XPath 功能实现精准的文本提取和定位
+"""
+
+from lxml import etree, html
+from typing import List, Dict, Any
+from loguru import logger
+import re
+
+
+class LxmlXPathExtractor:
+    """基于 lxml 和 XPath 的提取器"""
+    
+    # 块级元素的 XPath 表达式
+    BLOCK_XPATH = ' | '.join([
+        f'//{tag}' for tag in [
+            'p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
+            'blockquote', 'li', 'td', 'dd', 'dt', 'figcaption',
+            'section', 'article', 'aside', 'header', 'footer', 'main'
+        ]
+    ])
+    
+    NAV_KEYWORDS = [
+        'nav', 'navigation', 'toc', 'menu', 'header', 'footer', 
+        'page-number', 'page-num', 'sidebar'
+    ]
+    
+    def __init__(self, min_text_length: int = 10):
+        """
+        初始化提取器
+        
+        Args:
+            min_text_length: 最小文本长度
+        """
+        self.min_text_length = min_text_length
+    
+    def extract(self, html_content: str) -> List[Dict[str, Any]]:
+        """
+        从 HTML 中提取所有文本元素
+        
+        Args:
+            html_content: HTML 字符串
+            
+        Returns:
+            提取的元素列表
+        """
+        try:
+            # 移除 XML 声明(如果存在)
+            import re
+            html_content = re.sub(r'<\?xml[^?]*\?>', '', html_content)
+            
+            # 解析 HTML
+            tree = html.fromstring(html_content)
+        except Exception as e:
+            logger.error(f"lxml 解析失败: {e}")
+            return []
+        
+        items = []
+        processed_xpaths = set()
+        
+        # 使用 XPath 查找所有块级元素
+        try:
+            elements = tree.xpath(self.BLOCK_XPATH)
+        except Exception as e:
+            logger.error(f"XPath 查询失败: {e}")
+            return []
+        
+        for element in elements:
+            # 获取 XPath (需要通过 ElementTree 包装)
+            try:
+                xpath = tree.getroottree().getpath(element)
+            except:
+                # 备用方案:生成简单路径
+                xpath = f"//{element.tag}[{elements.index(element)}]"
+            
+            # 避免重复
+            if xpath in processed_xpaths:
+                continue
+            
+            # 提取文本
+            text = self._clean_text(element)
+            
+            # 过滤过短文本
+            if len(text.strip()) < self.min_text_length:
+                continue
+            
+            # 判断是否是导航元素
+            is_nav = self._is_navigation_element(element)
+            
+            # 获取 HTML
+            try:
+                element_html = etree.tostring(element, encoding='unicode')
+            except:
+                element_html = ""
+            
+            items.append({
+                'xpath': xpath,
+                'text': text,
+                'html': element_html,
+                'tag': element.tag,
+                'is_navigation': is_nav
+            })
+            
+            processed_xpaths.add(xpath)
+        
+        logger.info(f"lxml 提取了 {len(items)} 个文本元素")
+        return items
+    
+    def _clean_text(self, element) -> str:
+        """清理元素文本"""
+        # lxml 的 text_content() 方法
+        text = element.text_content().strip()
+        
+        # 清理多余空白
+        text = re.sub(r'\s+', ' ', text)
+        
+        return text
+    
+    def _is_navigation_element(self, element) -> bool:
+        """判断是否是导航元素"""
+        # 检查 class 属性
+        classes = element.get('class', '')
+        class_str = classes.lower() if isinstance(classes, str) else ''
+        
+        if any(keyword in class_str for keyword in self.NAV_KEYWORDS):
+            return True
+        
+        # 检查父元素
+        parent = element.getparent()
+        if parent is not None:
+            p_classes = parent.get('class', '')
+            p_class_str = p_classes.lower() if isinstance(p_classes, str) else ''
+            if any(keyword in p_class_str for keyword in self.NAV_KEYWORDS):
+                return True
+        
+        return False
+    
+    def backfill(self, html_content: str, translation_map: Dict[str, str]) -> str:
+        """
+        使用 XPath 精准回填翻译
+        
+        Args:
+            html_content: 原始 HTML
+            translation_map: {xpath: translation} 映射
+            
+        Returns:
+            回填后的 HTML 字符串
+        """
+        try:
+            # 移除 XML 声明
+            import re
+            html_content = re.sub(r'<\?xml[^?]*\?>', '', html_content)
+            
+            tree = html.fromstring(html_content)
+        except Exception as e:
+            logger.error(f"lxml 解析失败: {e}")
+            return html_content
+        
+        success_count = 0
+        fail_count = 0
+        
+        for xpath, translation in translation_map.items():
+            try:
+                elements = tree.xpath(xpath)
+                if not elements:
+                    logger.warning(f"回填失败: 未找到 XPath {xpath}")
+                    fail_count += 1
+                    continue
+                
+                element = elements[0]
+                
+                # 清空元素内容并设置新文本
+                element.clear()
+                element.text = translation
+                
+                success_count += 1
+            except Exception as e:
+                logger.error(f"回填错误 {xpath}: {e}")
+                fail_count += 1
+        
+        logger.info(f"lxml 回填完成: 成功 {success_count}, 失败 {fail_count}")
+        
+        try:
+            return etree.tostring(tree, encoding='unicode', method='html')
+        except:
+            return html_content

archive/v0.09/tests/extraction_experiment/extractors/one_to_one.py

@@ -0,0 +1,267 @@
+"""
+真正的一比一对应提取器
+
+核心原则:
+1. 提取时: 记录每个元素的所有文本节点位置
+2. 回填时: 精确替换这些文本节点,不改变任何结构
+"""
+
+from bs4 import BeautifulSoup, Tag, NavigableString
+from typing import List, Dict, Any, Tuple
+import re
+from loguru import logger
+
+
+class OneToOneExtractor:
+    """一比一对应提取器"""
+    
+    SKIP_TRANSLATION_PATTERNS = [
+        r'index\.x?html',
+        r'bibliography\.x?html',
+        r'endnotes?\.x?html',
+        r'footnotes?\.x?html',
+    ]
+    
+    TOC_PATTERNS = [
+        r'nav\.x?html',
+        r'toc\.x?html',
+    ]
+    
+    OTHER_NON_CORE_PATTERNS = [
+        r'copyright\.x?html',
+        r'title\.x?html',
+        r'cover\.x?html',
+    ]
+    
+    BLOCK_TAGS = [
+        'p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
+        'blockquote', 'li', 'td', 'dd', 'dt', 'figcaption',
+        'section', 'article', 'aside', 'header', 'footer', 'main'
+    ]
+    
+    def __init__(self, translate_toc: bool = False):
+        self.translate_toc = translate_toc
+        self.soup = None
+    
+    def extract(self, html_content: str, file_name: str = "") -> List[Dict[str, Any]]:
+        """
+        提取文本,记录精确的文本节点位置
+        
+        关键改进: 不跳过嵌套元素,每个块级元素都独立提取
+        
+        返回:
+        {
+            'element': 元素引用,
+            'text': 完整文本,
+            'text_nodes': [(node, text), ...],  # 只包含直接子节点的文本
+            'should_translate': bool
+        }
+        """
+        self.soup = BeautifulSoup(html_content, 'html.parser')
+        
+        # 移除不需要的元素
+        for element in self.soup(['script', 'style', 'meta', 'link']):
+            element.decompose()
+        
+        doc_type = self._classify_document(file_name)
+        
+        items = []
+        
+        # 关键: 不使用 processed_ids,每个元素都独立提取
+        all_elements = []
+        for element in self.soup.find_all(self.BLOCK_TAGS):
+            # 提取完整文本
+            full_text = element.get_text(separator=' ', strip=True)
+            
+            if not full_text.strip():
+                continue
+            
+            # 关键: 只收集当前元素的直接文本节点
+            # 不包括子元素中的文本节点
+            text_nodes = self._collect_direct_text_nodes(element)
+            
+            # 如果没有直接文本节点,说明所有文本都在子元素中
+            # 这种情况下跳过,让子元素自己处理
+            if not text_nodes:
+                continue
+            
+            all_elements.append({
+                'element': element,
+                'text': full_text,
+                'text_nodes': text_nodes,
+                'doc_type': doc_type,
+            })
+        
+        # 过滤: 只保留叶子节点 (没有被其他提取元素包含的元素)
+        for elem_data in all_elements:
+            element = elem_data['element']
+            
+            # 检查是否被其他提取元素包含
+            is_contained = False
+            for other_data in all_elements:
+                if other_data is elem_data:
+                    continue
+                
+                other_element = other_data['element']
+                # 检查 element 是否是 other_element 的子孙
+                if element in other_element.descendants:
+                    is_contained = True
+                    break
+            
+            if is_contained:
+                continue
+            
+            # 这是叶子节点,添加到结果
+            is_decorative = self._is_decorative(elem_data['text'])
+            should_translate = self._should_translate(doc_type, is_decorative)
+            
+            items.append({
+                'element': element,
+                'text': elem_data['text'],
+                'text_nodes': elem_data['text_nodes'],
+                'should_translate': should_translate,
+                'doc_type': doc_type,
+                'is_decorative': is_decorative,
+                'tag': element.name
+            })
+        
+        logger.info(
+            f"[{doc_type}] 提取 {len(items)} 个元素: "
+            f"翻译 {sum(1 for i in items if i['should_translate'])}"
+        )
+        
+        return items
+    
+    def _collect_direct_text_nodes(self, element: Tag) -> List[Tuple[NavigableString, str]]:
+        """
+        收集元素的文本节点
+        
+        策略:
+        1. 优先收集直接文本节点
+        2. 如果没有直接文本节点,收集所有子孙文本节点
+        
+        例如:
+        <div>Text1 <span>Text2</span></div>  → 收集 Text1 (直接)
+        <div><span>Text2</span></div>        → 收集 Text2 (子孙)
+        """
+        text_nodes = []
+        
+        # 先尝试收集直接子节点的文本
+        for child in element.children:
+            if isinstance(child, NavigableString):
+                if isinstance(child, type(element)):  # 跳过注释
+                    continue
+                
+                text = str(child).strip()
+                if text:
+                    text_nodes.append((child, text))
+        
+        # 如果有直接文本节点,返回
+        if text_nodes:
+            return text_nodes
+        
+        # 否则,收集所有子孙文本节点
+        for descendant in element.descendants:
+            if isinstance(descendant, NavigableString):
+                if isinstance(descendant, type(element)):  # 跳过注释
+                    continue
+                
+                text = str(descendant).strip()
+                if text:
+                    text_nodes.append((descendant, text))
+        
+        return text_nodes
+    
+    def backfill(self, items: List[Dict[str, Any]], translation_map: Dict[str, str]) -> str:
+        """
+        一比一精确回填
+        
+        策略:
+        1. 对于每个元素,找到对应的翻译
+        2. 将翻译分配给所有文本节点
+        3. 精确替换每个文本节点
+        """
+        success_count = 0
+        
+        for item in items:
+            original_text = item['text']
+            text_nodes = item['text_nodes']
+            
+            # 查找翻译
+            translation = translation_map.get(original_text)
+            if translation is None:
+                continue
+            
+            # 一比一替换: 将翻译替换到第一个文本节点,清空其他
+            if text_nodes:
+                # 第一个文本节点替换为完整翻译
+                text_nodes[0][0].replace_with(translation)
+                
+                # 其他文本节点清空(保留结构)
+                for node, _ in text_nodes[1:]:
+                    node.replace_with('')
+                
+                success_count += 1
+        
+        logger.info(f"回填完成: 成功 {success_count}/{len(translation_map)}")
+        return str(self.soup)
+    
+    def _classify_document(self, file_name: str) -> str:
+        if not file_name:
+            return 'core'
+        
+        file_name_lower = file_name.lower()
+        
+        for pattern in self.SKIP_TRANSLATION_PATTERNS:
+            if re.search(pattern, file_name_lower):
+                return 'skip'
+        
+        for pattern in self.TOC_PATTERNS:
+            if re.search(pattern, file_name_lower):
+                return 'toc'
+        
+        for pattern in self.OTHER_NON_CORE_PATTERNS:
+            if re.search(pattern, file_name_lower):
+                return 'other'
+        
+        return 'core'
+    
+    def _should_translate(self, doc_type: str, is_decorative: bool) -> bool:
+        if is_decorative:
+            return False
+        
+        if doc_type == 'core':
+            return True
+        
+        if doc_type == 'toc':
+            return self.translate_toc
+        
+        return False
+    
+    def _is_contained_in_processed(self, element: Tag, processed_ids: set) -> bool:
+        for parent in element.parents:
+            if isinstance(parent, Tag) and id(parent) in processed_ids:
+                return True
+        return False
+    
+    def _is_decorative(self, text: str) -> bool:
+        text_stripped = text.strip()
+        if not text_stripped or len(text_stripped) > 20:
+            return False
+        
+        decorative_patterns = [
+            r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾]+$',
+            r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾\s]+$',
+        ]
+        
+        for pattern in decorative_patterns:
+            if re.match(pattern, text_stripped):
+                return True
+        
+        unique_chars = set(text_stripped.replace(' ', ''))
+        if len(unique_chars) <= 3:
+            decorative_chars = set('*-_~•◆◇■□▪▫●○◉◎⦿⦾')
+            if unique_chars & decorative_chars:
+                return True
+        
+        return False

archive/v0.09/tests/extraction_experiment/extractors/smart_extractor.py

@@ -0,0 +1,325 @@
+"""
+智能分类提取器
+
+根据文档类型和复杂度,智能决定提取策略:
+- 正文: 100% 提取,必须翻译
+- 非核心部分: 如果复杂度高,标记为跳过翻译
+"""
+
+from bs4 import BeautifulSoup, Tag
+from typing import List, Dict, Any, Set
+import re
+from loguru import logger
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+from dom_path_utils import DOMPathUtils
+
+
+class SmartExtractor:
+    """智能分类提取器"""
+    
+    # 非核心文档的文件名模式
+    NON_CORE_PATTERNS = [
+        r'nav\.x?html',           # 目录
+        r'toc\.x?html',           # 目录
+        r'index\.x?html',         # 索引
+        r'bibliography\.x?html',  # 参考文献
+        r'endnotes?\.x?html',     # 尾注
+        r'footnotes?\.x?html',    # 脚注
+        r'copyright\.x?html',     # 版权页
+        r'title\.x?html',         # 标题页
+        r'cover\.x?html',         # 封面
+    ]
+    
+    # 块级标签
+    BLOCK_TAGS = [
+        'p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
+        'blockquote', 'li', 'td', 'dd', 'dt', 'figcaption',
+        'section', 'article', 'aside', 'header', 'footer', 'main'
+    ]
+    
+    # 装饰性符号模式
+    DECORATIVE_PATTERNS = [
+        r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾]+$',
+        r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾\s]+$',
+        r'^[\u2022-\u2027\u2030-\u205E]+$',
+    ]
+    
+    def __init__(self, preserve_decorative: bool = True):
+        """
+        初始化提取器
+        
+        Args:
+            preserve_decorative: 是否保留装饰性元素
+        """
+        self.preserve_decorative = preserve_decorative
+        self.path_utils = DOMPathUtils()
+    
+    def extract(self, html_content: str, file_name: str = "") -> List[Dict[str, Any]]:
+        """
+        智能提取文本元素
+        
+        Args:
+            html_content: HTML 字符串
+            file_name: 文件名(用于判断文档类型)
+            
+        Returns:
+            提取的元素列表,每个元素包含:
+            - path: DOM 路径
+            - text: 文本内容
+            - tag: 标签名
+            - is_decorative: 是否装饰性
+            - is_core: 是否核心内容(正文)
+            - should_translate: 是否应该翻译
+        """
+        soup = BeautifulSoup(html_content, 'html.parser')
+        
+        # 移除不需要的元素
+        for element in soup(['script', 'style', 'meta', 'link']):
+            element.decompose()
+        
+        # 判断文档类型
+        is_core_document = self._is_core_document(file_name)
+        
+        items = []
+        processed_ids = set()
+        
+        # 遍历所有块级元素
+        for element in soup.find_all(self.BLOCK_TAGS):
+            elem_id = id(element)
+            
+            if elem_id in processed_ids:
+                continue
+            
+            if self._is_contained_in_processed(element, processed_ids):
+                continue
+            
+            # 提取文本(不过滤任何内容)
+            text = self._extract_text(element)
+            
+            if not text.strip():
+                continue
+            
+            # 判断元素类型
+            is_decorative = self._is_decorative_element(element, text)
+            
+            # 生成路径
+            path = self.path_utils.get_dom_path(element)
+            
+            # 决定是否翻译
+            should_translate = self._should_translate(
+                element, text, is_core_document, is_decorative
+            )
+            
+            items.append({
+                'path': path,
+                'element': element,
+                'text': text,
+                'html': str(element),
+                'tag': element.name,
+                'is_decorative': is_decorative,
+                'is_core': is_core_document,
+                'should_translate': should_translate,
+                'file_name': file_name
+            })
+            
+            processed_ids.add(elem_id)
+        
+        # 添加 <hr> 等装饰性标签
+        if self.preserve_decorative:
+            for hr in soup.find_all('hr'):
+                elem_id = id(hr)
+                if elem_id not in processed_ids:
+                    path = self.path_utils.get_dom_path(hr)
+                    items.append({
+                        'path': path,
+                        'element': hr,
+                        'text': '---',
+                        'html': str(hr),
+                        'tag': 'hr',
+                        'is_decorative': True,
+                        'is_core': is_core_document,
+                        'should_translate': False,
+                        'file_name': file_name
+                    })
+                    processed_ids.add(elem_id)
+        
+        # 统计
+        core_count = sum(1 for i in items if i['is_core'])
+        translate_count = sum(1 for i in items if i['should_translate'])
+        decorative_count = sum(1 for i in items if i['is_decorative'])
+        
+        logger.info(
+            f"提取了 {len(items)} 个元素 "
+            f"(核心: {core_count}, 需翻译: {translate_count}, 装饰性: {decorative_count})"
+        )
+        
+        return items
+    
+    def _is_core_document(self, file_name: str) -> bool:
+        """
+        判断是否是核心文档(正文)
+        
+        非核心文档包括: 目录、索引、参考文献、版权页等
+        """
+        if not file_name:
+            return True  # 默认认为是核心文档
+        
+        file_name_lower = file_name.lower()
+        
+        for pattern in self.NON_CORE_PATTERNS:
+            if re.search(pattern, file_name_lower):
+                return False
+        
+        return True
+    
+    def _should_translate(self, element: Tag, text: str, 
+                         is_core_document: bool, is_decorative: bool) -> bool:
+        """
+        决定元素是否应该翻译
+        
+        规则:
+        1. 装饰性元素: 不翻译
+        2. 核心文档: 全部翻译
+        3. 非核心文档: 根据复杂度决定
+        """
+        # 装饰性元素不翻译
+        if is_decorative:
+            return False
+        
+        # 核心文档全部翻译
+        if is_core_document:
+            return True
+        
+        # 非核心文档: 检查复杂度
+        complexity = self._calculate_complexity(element, text)
+        
+        # 复杂度阈值: 如果太复杂,不翻译
+        if complexity > 0.5:
+            logger.debug(f"非核心元素复杂度过高 ({complexity:.2f}), 跳过翻译: {text[:50]}")
+            return False
+        
+        return True
+    
+    def _calculate_complexity(self, element: Tag, text: str) -> float:
+        """
+        计算元素的复杂度
+        
+        复杂度指标:
+        - 嵌套深度
+        - 链接数量
+        - 数字比例
+        - 特殊字符比例
+        
+        Returns:
+            0.0 - 1.0, 越高越复杂
+        """
+        complexity_score = 0.0
+        
+        # 1. 嵌套深度 (最大贡献 0.3)
+        depth = len(list(element.parents))
+        complexity_score += min(depth / 20, 0.3)
+        
+        # 2. 链接数量 (最大贡献 0.3)
+        links = element.find_all('a')
+        if links:
+            link_ratio = len(links) / max(len(text.split()), 1)
+            complexity_score += min(link_ratio, 0.3)
+        
+        # 3. 数字比例 (最大贡献 0.2)
+        digits = sum(c.isdigit() for c in text)
+        if text:
+            digit_ratio = digits / len(text)
+            complexity_score += min(digit_ratio * 2, 0.2)
+        
+        # 4. 特殊字符比例 (最大贡献 0.2)
+        special_chars = sum(1 for c in text if not c.isalnum() and not c.isspace())
+        if text:
+            special_ratio = special_chars / len(text)
+            complexity_score += min(special_ratio * 2, 0.2)
+        
+        return min(complexity_score, 1.0)
+    
+    def _is_contained_in_processed(self, element: Tag, processed_ids: Set[int]) -> bool:
+        """检查元素是否被已处理的父元素包含"""
+        for parent in element.parents:
+            if isinstance(parent, Tag) and id(parent) in processed_ids:
+                return True
+        return False
+    
+    def _extract_text(self, element: Tag) -> str:
+        """
+        提取元素文本 - 100% 完整提取,不过滤任何内容
+        
+        注意: 这里不做任何清理,保证 100% 提取
+        """
+        return element.get_text(separator=' ', strip=True)
+    
+    def _is_decorative_element(self, element: Tag, text: str) -> bool:
+        """判断是否是装饰性元素"""
+        # 检查 class
+        classes = element.get('class', [])
+        class_str = ' '.join(classes).lower() if isinstance(classes, list) else str(classes).lower()
+        
+        decorative_classes = ['separator', 'divider', 'ornament', 'decoration', 'break']
+        if any(dc in class_str for dc in decorative_classes):
+            return True
+        
+        # 检查文本模式
+        text_stripped = text.strip()
+        if not text_stripped:
+            return False
+        
+        for pattern in self.DECORATIVE_PATTERNS:
+            if re.match(pattern, text_stripped):
+                return True
+        
+        # 检查重复字符
+        if len(text_stripped) <= 20:
+            unique_chars = set(text_stripped.replace(' ', ''))
+            if len(unique_chars) <= 3:
+                decorative_chars = set('*-_~•◆◇■□▪▫●○◉◎⦿⦾')
+                if unique_chars & decorative_chars:
+                    return True
+        
+        return False
+    
+    def backfill(self, html_content: str, translation_map: Dict[str, str]) -> str:
+        """
+        精准回填翻译
+        
+        只回填 should_translate=True 的元素
+        """
+        soup = BeautifulSoup(html_content, 'html.parser')
+        
+        success_count = 0
+        skip_count = 0
+        fail_count = 0
+        
+        for path, translation in translation_map.items():
+            element = self.path_utils.find_by_path(soup, path)
+            
+            if element is None:
+                logger.warning(f"回填失败: 未找到路径 {path}")
+                fail_count += 1
+                continue
+            
+            # 检查是否应该翻译
+            # (这个信息应该在 translation_map 的构建阶段就过滤了)
+            
+            # 创建新元素
+            new_tag = soup.new_tag(element.name)
+            new_tag.string = translation
+            
+            # 复制属性
+            for attr, value in element.attrs.items():
+                new_tag[attr] = value
+            
+            # 替换
+            element.replace_with(new_tag)
+            success_count += 1
+        
+        logger.info(f"回填完成: 成功 {success_count}, 跳过 {skip_count}, 失败 {fail_count}")
+        return str(soup)

archive/v0.09/tests/extraction_experiment/reports/multi_epub_test_report.md

@@ -0,0 +1,90 @@
+# 多 ePub 提取完整性测试报告
+
+**测试时间**: 2026-01-19 12:26:59
+
+**测试文件数**: 5
+
+**成功**: 5/5
+
+## 测试结果汇总
+
+| 文件名 | 文件大小 | 提取元素 | 装饰性 | 文本长度 | 覆盖率 |
+|--------|---------|---------|--------|---------|--------|
+| Gambling Man.epub | 3428.3KB | 60 | 0 | 723,609 | 77.7% |
+| On_China_Henry_Kissinger.epub | 924.5KB | 3602 | 32 | 1,141,726 | 87.6% |
+| The World Atlas of Coffee - Fr | 20406.0KB | 1621 | 340 | 353,627 | 74.6% |
+| The_Philosopher_in_the_Valley. | 4687.7KB | 54 | 15 | 524,578 | 93.9% |
+| To_Explain_the_World.epub | 1756.4KB | 3564 | 105 | 782,608 | 87.9% |
+
+## 详细分析
+
+### Gambling Man.epub
+
+- **HTML 文档数**: 47
+- **提取元素总数**: 60
+  - 内容元素: 51
+  - 装饰性元素: 0
+  - 导航元素: 9
+- **提取文本长度**: 723,609 字符
+- **提取词数**: 118,115
+- **Pandoc 基准长度**: 889,384 字符
+- **覆盖率**: 77.70%
+- **共同词数**: 11,607
+
+### On_China_Henry_Kissinger.epub
+
+- **HTML 文档数**: 144
+- **提取元素总数**: 3602
+  - 内容元素: 3570
+  - 装饰性元素: 32
+  - 导航元素: 0
+- **提取文本长度**: 1,141,726 字符
+- **提取词数**: 181,503
+- **Pandoc 基准长度**: 1,509,848 字符
+- **覆盖率**: 87.57%
+- **共同词数**: 12,834
+
+### The World Atlas of Coffee - From Beans to Brewing - Coffees Explored, Explained and Enjoyed (James Hoffmann) (Z-Library).epub
+
+- **HTML 文档数**: 98
+- **提取元素总数**: 1621
+  - 内容元素: 1254
+  - 装饰性元素: 340
+  - 导航元素: 27
+- **提取文本长度**: 353,627 字符
+- **提取词数**: 59,885
+- **Pandoc 基准长度**: 560,850 字符
+- **覆盖率**: 74.62%
+- **共同词数**: 5,885
+
+### The_Philosopher_in_the_Valley.epub
+
+- **HTML 文档数**: 22
+- **提取元素总数**: 54
+  - 内容元素: 24
+  - 装饰性元素: 15
+  - 导航元素: 15
+- **提取文本长度**: 524,578 字符
+- **提取词数**: 86,566
+- **Pandoc 基准长度**: 550,137 字符
+- **覆盖率**: 93.95%
+- **共同词数**: 9,760
+
+### To_Explain_the_World.epub
+
+- **HTML 文档数**: 105
+- **提取元素总数**: 3564
+  - 内容元素: 3459
+  - 装饰性元素: 105
+  - 导航元素: 0
+- **提取文本长度**: 782,608 字符
+- **提取词数**: 133,415
+- **Pandoc 基准长度**: 950,720 字符
+- **覆盖率**: 87.90%
+- **共同词数**: 9,237
+
+## 总结
+
+- **平均覆盖率**: 84.35%
+- **总装饰性元素**: 492 个
+- **提取器状态**: ⚠️ 需要优化

archive/v0.09/tests/extraction_experiment/run_batch.py

@@ -0,0 +1,71 @@
+"""
+批量运行测试: 对 input 目录下的所有 EPUB 执行清理和生成双语版本
+"""
+
+import sys
+from pathlib import Path
+import os
+import time
+from loguru import logger
+
+# 配置路径
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+sys.path.insert(0, str(Path(__file__).parent))
+
+# 导入功能模块
+from simple_cleaner import clean_epub
+from test_end_to_end import create_bilingual_epub
+
+def run_batch():
+    input_dir = project_root / "input"
+    output_dir = project_root / "test_output"
+    output_dir.mkdir(exist_ok=True)
+    
+    epubs = list(input_dir.glob("*.epub"))
+    epubs.sort() # 按文件名排序
+    
+    print(f"\n{'='*80}")
+    print(f"批量测试开始: 共 {len(epubs)} 个文件")
+    print(f"{'='*80}\n")
+    
+    success_count = 0
+    
+    for i, epub_path in enumerate(epubs, 1):
+        print(f"[{i}/{len(epubs)}] 📖 处理: {epub_path.name}")
+        
+        cleaned_path = output_dir / f"{epub_path.stem}_cleaned.epub"
+        bilingual_path = output_dir / f"{epub_path.stem}_bilingual.epub"
+        
+        try:
+            # 1. 清理
+            print("  ➤ 正在清理...")
+            start = time.time()
+            # 捕获日志或只允许 ERROR? 暂时保持默认
+            clean_epub(str(epub_path), str(cleaned_path))
+            print(f"    ✓ 清理完成用时: {time.time() - start:.2f}s")
+            
+            # 2. 生成双语
+            print("  ➤ 正在生成双语版本...")
+            start = time.time()
+            create_bilingual_epub(cleaned_path, bilingual_path)
+            print(f"    ✓ 生成完成用时: {time.time() - start:.2f}s")
+            
+            print(f"  ✅ 成功! 输出: {bilingual_path.name}\n")
+            success_count += 1
+            
+        except Exception as e:
+            print(f"  ❌ 处理失败: {e}\n")
+            # 不中断后续任务
+            continue
+            
+    print(f"{'='*80}")
+    print(f"批量测试结束: 成功 {success_count}/{len(epubs)}")
+    print(f"{'='*80}\n")
+
+if __name__ == "__main__":
+    # 配置 logger 只显示 WARNING 以上，以免刷屏
+    logger.remove()
+    logger.add(sys.stderr, level="WARNING")
+    
+    run_batch()

archive/v0.09/tests/extraction_experiment/simple_cleaner.py

@@ -0,0 +1,205 @@
+"""
+简化版 Calibre 清理器 - 避免复杂操作
+
+只做最基本的清理:
+1. div 转 p
+2. 移除 calibre 类
+"""
+
+from bs4 import BeautifulSoup, Tag
+from loguru import logger
+
+
+class SimpleCleaner:
+    """简化版清理器"""
+    
+    def clean(self, html_content: str, item=None) -> str:
+        """
+        清理 HTML
+        
+        Args:
+            html_content: HTML 内容
+            item: EpubItem 对象(可选), 用于注册 links
+        """
+        soup = BeautifulSoup(html_content, 'html.parser')
+        
+        # 0. 提取并保留 CSS 链接 (解决 ebooklib 丢失 link 的问题)
+        if item:
+            head = soup.find('head')
+            if head:
+                # 提取 link
+                links = head.find_all('link', rel='stylesheet')
+                for link in links:
+                    href = link.get('href')
+                    if href:
+                        # 检查是否已存在(避免重复)
+                        existing_links = list(item.get_links())
+                        
+                        exists = False
+                        for l in existing_links:
+                            l_href = getattr(l, 'href', None)
+                            if l_href is None and isinstance(l, dict):
+                                l_href = l.get('href')
+                            
+                            if l_href == href:
+                                exists = True
+                                break
+                        
+                        if not exists:
+                            logger.debug(f"恢复 CSS 链接: {href}")
+                            item.add_link(href=href, rel='stylesheet', type='text/css')
+        
+        stats = {'divs_to_p': 0, 'classes_removed': 0}
+        
+        # 1. div 转 p (只转换没有块级子元素的)
+        inline_tags = {'span', 'em', 'strong', 'i', 'b', 'u', 'a', 'br'}
+        
+        divs = list(soup.find_all('div'))  # 先收集所有div
+        for div in divs:
+            has_block = any(
+                isinstance(c, Tag) and c.name not in inline_tags 
+                for c in div.children
+            )
+            
+            if not has_block:
+                div.name = 'p'
+                stats['divs_to_p'] += 1
+        
+        # 2. 清理 calibre 类 - 暂时禁用，以保留样式
+        # elements = list(soup.find_all(class_=True))  # 先收集
+        # ... (保留原注释代码)
+        
+        logger.info(f"清理完成: div→p {stats['divs_to_p']}, 类移除 {stats['classes_removed']}")
+        
+        return str(soup)
+
+
+def clean_epub(input_path: str, output_path: str):
+    """清理 ePub"""
+    from ebooklib import epub
+    import zipfile
+    
+    logger.info(f"开始清理: {input_path}")
+    
+    # 打开 zip 以读取原始内容
+    try:
+        input_zip = zipfile.ZipFile(input_path, 'r')
+        zip_files = set(input_zip.namelist())
+    except Exception as e:
+        logger.error(f"无法打开 Zip: {e}")
+        input_zip = None
+        zip_files = set()
+    
+    book = epub.read_epub(input_path)
+    cleaner = SimpleCleaner()
+    
+    count = 0
+    for item in book.get_items():
+        if item.get_type() == 9:
+            try:
+                file_name = item.get_name()
+                content = None
+                
+                # 优先从 Zip 读取以保留 Head 信息
+                if input_zip and file_name in zip_files:
+                    try:
+                        content = input_zip.read(file_name).decode('utf-8')
+                    except Exception as e:
+                        logger.warning(f"Zip 读取失败 {file_name}: {e}")
+                
+                # 回退到 ebooklib
+                if content is None:
+                    raw_content = item.get_content()
+                    if raw_content:
+                        content = raw_content.decode('utf-8')
+                
+                # 检查原始内容
+                if not content or not content.strip():
+                    logger.warning(f"跳过空文档: {item.get_name()}")
+                    continue
+                
+                # 清理并提取信息
+                # 注意: 我们需要传入 item 以便 cleaner 可以注册 links
+                cleaned = cleaner.clean(content, item)
+                
+                # 检查清理后内容
+                if not cleaned.strip():
+                    logger.error(f"⚠️ 清理后内容为空: {item.get_name()} (原始长度: {len(content)})")
+                    # 如果清理变为空，保留原始内容
+                    cleaned = content
+                
+                item.set_content(cleaned.encode('utf-8'))
+                count += 1
+                
+                if 'titlepage' in item.get_name():
+                    logger.info(f"Titlepage 处理完成: {len(cleaned)} chars")
+                    
+            except Exception as e:
+                logger.warning(f"清理失败 {item.get_name()}: {e}")
+    
+    
+    # 修复 TOC：补全 UID 并移除指向不存在文件的死链
+    def fix_and_clean_toc(toc, book):
+        new_toc = []
+        import uuid
+        from ebooklib.epub import Link
+        
+        for item in toc:
+            # Case 1: (Section, Children) 元组
+            if isinstance(item, (tuple, list)):
+                section, children = item
+                # 递归清理子节点
+                cleaned_children = fix_and_clean_toc(children, book)
+                
+                # 检查 Section 节点
+                if isinstance(section, Link):
+                    href = section.href.split('#')[0]
+                    # 有效性检查：目标文件必须在 manifest 中存在
+                    if book.get_item_with_href(href):
+                        if section.uid is None:
+                            section.uid = f'uuid-{uuid.uuid4()}'
+                        new_toc.append((section, cleaned_children))
+                    else:
+                        logger.warning(f"移除无效 TOC 节点 (目标缺失): {section.title} -> {section.href}")
+                        # 如果父节点无效，这里选择提升子节点，还是丢弃？
+                        # 策略：如果父节点都无效了，就把子节点提升上来（如果子节点有效）
+                        new_toc.extend(cleaned_children)
+                else:
+                    # 如果 Section 不是 Link (罕见)，保留
+                    new_toc.append((section, cleaned_children))
+
+            # Case 2: 单个 Link 节点
+            elif isinstance(item, Link):
+                href = item.href.split('#')[0]
+                if book.get_item_with_href(href):
+                     if item.uid is None:
+                         item.uid = f'uuid-{uuid.uuid4()}'
+                     new_toc.append(item)
+                else:
+                    logger.warning(f"移除无效 TOC 节点 (目标缺失): {item.title} -> {item.href}")
+            
+            # Case 3: 其他 (如自定义 dict 等? 一般不会)
+            else:
+                new_toc.append(item)
+                
+        return new_toc
+    
+    try:
+        book.toc = fix_and_clean_toc(book.toc, book)
+    except Exception as e:
+        logger.warning(f"修复 TOC 失败: {e}")
+        import traceback
+        logger.warning(traceback.format_exc())
+
+    epub.write_epub(output_path, book)
+    logger.info(f"完成: 处理了 {count} 个文档")
+
+
+if __name__ == "__main__":
+    import sys
+    
+    if len(sys.argv) < 3:
+        print("用法: python simple_cleaner.py <input.epub> <output.epub>")
+        sys.exit(1)
+    
+    clean_epub(sys.argv[1], sys.argv[2])

archive/v0.09/tests/extraction_experiment/test_backfill.py

@@ -0,0 +1,195 @@
+"""
+完整回填测试
+
+使用 On_China 书籍,模拟翻译并回填,生成双语版本供检查
+"""
+
+import sys
+from pathlib import Path
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from ebooklib import epub
+from loguru import logger
+import shutil
+
+sys.path.insert(0, str(Path(__file__).parent))
+from extractors.final_extractor import FinalExtractor
+
+
+def create_bilingual_test_epub(epub_path: Path, output_path: Path, translate_toc: bool = False):
+    """
+    创建双语测试版本
+    
+    将每个元素的翻译替换为: 原文 + 标记
+    - 翻译元素: "原文 [翻译]"
+    - 跳过元素: "原文 [跳过]"
+    - 装饰性: "原文 [装饰]"
+    """
+    print(f"\n{'='*80}")
+    print(f"创建双语测试版本: {epub_path.name}")
+    print(f"目录翻译: {'是' if translate_toc else '否'}")
+    print(f"{'='*80}\n")
+    
+    # 加载 ePub
+    book = epub.read_epub(str(epub_path))
+    
+    # 提取器
+    extractor = FinalExtractor(translate_toc=translate_toc, preserve_decorative=True)
+    
+    # 统计
+    total_items = 0
+    total_translate = 0
+    total_skip = 0
+    total_decorative = 0
+    
+    doc_stats = []
+    
+    # 处理每个 HTML 文档
+    for item in book.get_items():
+        if item.get_type() != 9:  # 只处理 ITEM_DOCUMENT
+            continue
+        
+        try:
+            content = item.get_content().decode('utf-8')
+        except:
+            continue
+        
+        file_name = item.get_name()
+        
+        # 提取
+        items = extractor.extract(content, file_name)
+        
+        if not items:
+            continue
+        
+        # 统计
+        translate_items = [i for i in items if i['should_translate']]
+        skip_items = [i for i in items if not i['should_translate'] and not i['is_decorative']]
+        decorative_items = [i for i in items if i['is_decorative']]
+        
+        total_items += len(items)
+        total_translate += len(translate_items)
+        total_skip += len(skip_items)
+        total_decorative += len(decorative_items)
+        
+        doc_stats.append({
+            'file': file_name,
+            'doc_type': items[0]['doc_type'],
+            'total': len(items),
+            'translate': len(translate_items),
+            'skip': len(skip_items),
+            'decorative': len(decorative_items)
+        })
+        
+        # 创建翻译映射
+        translation_map = {}
+        for i in items:
+            if i['should_translate']:
+                translation_map[i['path']] = f"{i['text']} [翻译]"
+            elif i['is_decorative']:
+                translation_map[i['path']] = f"{i['text']} [装饰]"
+            else:
+                translation_map[i['path']] = f"{i['text']} [跳过]"
+        
+        # 回填
+        new_content = extractor.backfill(content, translation_map)
+        
+        # 更新 item
+        item.set_content(new_content.encode('utf-8'))
+    
+    # 保存新 ePub
+    epub.write_epub(str(output_path), book)
+    
+    # 显示统计
+    print(f"{'='*80}")
+    print("处理统计")
+    print(f"{'='*80}\n")
+    
+    print(f"总元素数: {total_items}")
+    print(f"  - 翻译: {total_translate} ({total_translate/total_items*100:.1f}%)")
+    print(f"  - 跳过: {total_skip} ({total_skip/total_items*100:.1f}%)")
+    print(f"  - 装饰: {total_decorative} ({total_decorative/total_items*100:.1f}%)\n")
+    
+    # 按文档类型分组统计
+    print(f"{'='*80}")
+    print("按文档类型统计")
+    print(f"{'='*80}\n")
+    
+    doc_type_stats = {}
+    for stat in doc_stats:
+        doc_type = stat['doc_type']
+        if doc_type not in doc_type_stats:
+            doc_type_stats[doc_type] = {
+                'count': 0,
+                'total': 0,
+                'translate': 0,
+                'skip': 0,
+                'decorative': 0
+            }
+        
+        doc_type_stats[doc_type]['count'] += 1
+        doc_type_stats[doc_type]['total'] += stat['total']
+        doc_type_stats[doc_type]['translate'] += stat['translate']
+        doc_type_stats[doc_type]['skip'] += stat['skip']
+        doc_type_stats[doc_type]['decorative'] += stat['decorative']
+    
+    for doc_type, stats in sorted(doc_type_stats.items()):
+        print(f"📄 {doc_type.upper()} ({stats['count']} 个文档)")
+        print(f"   总元素: {stats['total']}")
+        print(f"   翻译: {stats['translate']} ({stats['translate']/stats['total']*100:.1f}%)")
+        print(f"   跳过: {stats['skip']} ({stats['skip']/stats['total']*100:.1f}%)")
+        print(f"   装饰: {stats['decorative']} ({stats['decorative']/stats['total']*100:.1f}%)")
+        print()
+    
+    # 显示详细文档列表
+    print(f"{'='*80}")
+    print("详细文档列表")
+    print(f"{'='*80}\n")
+    
+    for stat in doc_stats[:20]:
+        doc_type_label = stat['doc_type'].upper()
+        print(f"[{doc_type_label:6}] {stat['file']}")
+        print(f"          元素: {stat['total']:4} | 翻译: {stat['translate']:4} | 跳过: {stat['skip']:4} | 装饰: {stat['decorative']:2}")
+    
+    if len(doc_stats) > 20:
+        print(f"\n... 还有 {len(doc_stats) - 20} 个文档\n")
+    
+    print(f"\n✅ 双语测试版本已保存: {output_path}")
+    print(f"\n请在 ePub 阅读器中打开检查:")
+    print(f"  - 正文应该显示: '原文 [翻译]'")
+    print(f"  - 索引/参考文献/尾注应该显示: '原文 [跳过]'")
+    print(f"  - 目录应该显示: '原文 [{'翻译' if translate_toc else '跳过'}]'")
+    print(f"  - 装饰性符号应该显示: '原文 [装饰]'")
+
+
+def main():
+    """主函数"""
+    logger.remove()
+    logger.add(sys.stderr, level="INFO")
+    
+    # 测试文件
+    epub_path = project_root / "input" / "On_China_Henry_Kissinger.epub"
+    
+    if not epub_path.exists():
+        print(f"❌ 文件不存在: {epub_path}")
+        return
+    
+    # 输出目录
+    output_dir = project_root / "test_output"
+    output_dir.mkdir(exist_ok=True)
+    
+    # 测试1: 不翻译目录
+    output_path_1 = output_dir / "On_China_bilingual_no_toc.epub"
+    create_bilingual_test_epub(epub_path, output_path_1, translate_toc=False)
+    
+    print(f"\n{'='*80}\n")
+    
+    # 测试2: 翻译目录
+    output_path_2 = output_dir / "On_China_bilingual_with_toc.epub"
+    create_bilingual_test_epub(epub_path, output_path_2, translate_toc=True)
+
+
+if __name__ == "__main__":
+    main()

archive/v0.09/tests/extraction_experiment/test_bs4_skeleton.py

@@ -0,0 +1,195 @@
+"""
+测试 BS4 骨架保留
+
+验证是否完整保留所有 HTML 结构、CSS 样式和属性
+"""
+
+import sys
+from pathlib import Path
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from ebooklib import epub
+from loguru import logger
+import re
+
+sys.path.insert(0, str(Path(__file__).parent))
+from extractors.bs4_skeleton import BS4SkeletonExtractor
+
+
+def test_simple_html():
+    """测试简单 HTML"""
+    html = """<?xml version='1.0' encoding='utf-8'?>
+<!DOCTYPE html>
+<html>
+<head/>
+<body>
+<div class="calibre3"><span class="calibre6"><span class="bold">Table of Contents</span></span></div>
+<p class="text-center" style="font-size: 18px; color: blue;">This is a centered paragraph.</p>
+<blockquote class="quote" style="margin-left: 40px;">A famous quote here.</blockquote>
+<h1 id="chapter1" class="chapter-title">Chapter One</h1>
+</body>
+</html>"""
+    
+    print("\n" + "="*80)
+    print("简单 HTML 骨架保留测试")
+    print("="*80 + "\n")
+    
+    # 提取
+    extractor = BS4SkeletonExtractor()
+    items = extractor.extract(html)
+    
+    print(f"提取了 {len(items)} 个元素:\n")
+    
+    for i, item in enumerate(items, 1):
+        print(f"{i}. [{item['tag']}] {item['text'][:50]}")
+    
+    # 模拟翻译
+    translation_map = {}
+    for item in items:
+        if item['should_translate']:
+            translation_map[item['text']] = f"{item['text']} [翻译]"
+    
+    print(f"\n待翻译: {len(translation_map)} 个元素\n")
+    
+    # 回填
+    result_html = extractor.backfill(items, translation_map)
+    
+    print("="*80)
+    print("回填后的 HTML:")
+    print("="*80 + "\n")
+    print(result_html)
+    
+    # 验证
+    print("\n" + "="*80)
+    print("验证结果:")
+    print("="*80 + "\n")
+    
+    checks = [
+        ('class="calibre3"', 'class 属性'),
+        ('class="text-center"', 'class 属性'),
+        ('style="font-size: 18px; color: blue;"', 'style 属性'),
+        ('style="margin-left: 40px;"', 'style 属性'),
+        ('id="chapter1"', 'id 属性'),
+        ('<span class="bold">', '内部格式标签'),
+        ('<span class="calibre6">', '嵌套标签'),
+    ]
+    
+    for pattern, name in checks:
+        if pattern in result_html:
+            print(f"✅ {name} 保留: {pattern}")
+        else:
+            print(f"❌ {name} 丢失: {pattern}")
+
+
+def test_real_epub():
+    """测试真实 ePub"""
+    epub_path = project_root / "input" / "On_China_Henry_Kissinger.epub"
+    
+    if not epub_path.exists():
+        print(f"\n跳过真实 ePub 测试: 文件不存在")
+        return
+    
+    print("\n" + "="*80)
+    print("真实 ePub 骨架保留测试")
+    print("="*80 + "\n")
+    
+    book = epub.read_epub(str(epub_path))
+    
+    # 找第一个内容文档
+    for item in book.get_items():
+        if item.get_type() == 9 and 'dummy_split_002' in item.get_name():
+            content = item.get_content().decode('utf-8')
+            
+            print(f"测试文件: {item.get_name()}\n")
+            
+            # 统计原始 HTML 的属性
+            original_classes = len(re.findall(r'class="[^"]*"', content))
+            original_styles = len(re.findall(r'style="[^"]*"', content))
+            original_ids = len(re.findall(r'id="[^"]*"', content))
+            
+            print(f"原始 HTML 统计:")
+            print(f"  - class 属性: {original_classes} 个")
+            print(f"  - style 属性: {original_styles} 个")
+            print(f"  - id 属性: {original_ids} 个\n")
+            
+            # 提取
+            extractor = BS4SkeletonExtractor()
+            items = extractor.extract(content, item.get_name())
+            
+            print(f"提取了 {len(items)} 个元素\n")
+            
+            # 显示前 3 个
+            for i, elem in enumerate(items[:3], 1):
+                print(f"元素 {i}:")
+                print(f"  标签: <{elem['tag']}>")
+                print(f"  文本: {elem['text'][:60]}...")
+                print()
+            
+            # 模拟翻译
+            translation_map = {}
+            for elem in items:
+                if elem['should_translate']:
+                    translation_map[elem['text']] = f"{elem['text']} [翻译]"
+            
+            print(f"待翻译: {len(translation_map)} 个元素\n")
+            
+            # 回填
+            result_html = extractor.backfill(items, translation_map)
+            
+            # 统计回填后的属性
+            result_classes = len(re.findall(r'class="[^"]*"', result_html))
+            result_styles = len(re.findall(r'style="[^"]*"', result_html))
+            result_ids = len(re.findall(r'id="[^"]*"', result_html))
+            
+            print("="*80)
+            print("回填后 HTML 统计:")
+            print("="*80 + "\n")
+            print(f"  - class 属性: {result_classes} 个")
+            print(f"  - style 属性: {result_styles} 个")
+            print(f"  - id 属性: {result_ids} 个\n")
+            
+            # 验证
+            print("="*80)
+            print("验证结果:")
+            print("="*80 + "\n")
+            
+            if original_classes == result_classes:
+                print(f"✅ 所有 class 属性保留 ({original_classes} 个)")
+            else:
+                print(f"❌ class 属性丢失: {original_classes} → {result_classes}")
+            
+            if original_styles == result_styles:
+                print(f"✅ 所有 style 属性保留 ({original_styles} 个)")
+            else:
+                print(f"❌ style 属性丢失: {original_styles} → {result_styles}")
+            
+            if original_ids == result_ids:
+                print(f"✅ 所有 id 属性保留 ({original_ids} 个)")
+            else:
+                print(f"❌ id 属性丢失: {original_ids} → {result_ids}")
+            
+            # 检查翻译是否成功
+            if '[翻译]' in result_html:
+                print(f"✅ 翻译成功回填")
+            else:
+                print(f"❌ 翻译未回填")
+            
+            break
+
+
+def main():
+    """主函数"""
+    logger.remove()
+    logger.add(sys.stderr, level="INFO")
+    
+    # 测试 1: 简单 HTML
+    test_simple_html()
+    
+    # 测试 2: 真实 ePub
+    test_real_epub()
+
+
+if __name__ == "__main__":
+    main()

archive/v0.09/tests/extraction_experiment/test_calibre_cleaner.py

@@ -0,0 +1,89 @@
+"""
+测试 Calibre 清理器
+"""
+
+import sys
+from pathlib import Path
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from loguru import logger
+from calibre_cleaner import CalibreHTMLCleaner
+
+
+def test_html_cleaning():
+    """测试 HTML 清理"""
+    
+    # 测试用例: Calibre 生成的屎山代码
+    html = """
+    <div class="calibre16">
+        <span class="calibre9">
+            <div class="calibre16">
+                <span class="calibre9">
+                    <span class="italic">A ruler</span>
+                </span>
+            </div>
+            <div class="calibre11">
+                <span class="calibre9">
+                    <span class="italic">Must never</span>
+                </span>
+            </div>
+            <div class="calibre11">
+                <span class="calibre9">
+                    <span class="italic">Mobilize his men</span>
+                </span>
+            </div>
+        </span>
+    </div>
+    """
+    
+    print("\n" + "="*80)
+    print("Calibre HTML 清理测试")
+    print("="*80 + "\n")
+    
+    print("原始 HTML:")
+    print(html)
+    print()
+    
+    # 清理
+    cleaner = CalibreHTMLCleaner()
+    cleaned = cleaner.clean(html)
+    
+    print("="*80)
+    print("清理后的 HTML:")
+    print("="*80 + "\n")
+    print(cleaned)
+    print()
+    
+    # 验证
+    print("="*80)
+    print("验证:")
+    print("="*80 + "\n")
+    
+    if '<p>' in cleaned:
+        print(f"✅ div 转为 p: {cleaner.stats['divs_to_p']} 个")
+    else:
+        print("❌ div 未转为 p")
+    
+    if '<em>' in cleaned:
+        print(f"✅ span 简化为 em: {cleaner.stats['spans_simplified']} 个")
+    else:
+        print("❌ span 未简化")
+    
+    if 'calibre' not in cleaned or cleaner.stats['classes_removed'] > 0:
+        print(f"✅ 移除 calibre 类: {cleaner.stats['classes_removed']} 个")
+    else:
+        print("❌ calibre 类未移除")
+
+
+def main():
+    """主函数"""
+    logger.remove()
+    logger.add(sys.stderr, level="INFO")
+    
+    test_html_cleaning()
+
+
+if __name__ == "__main__":
+    main()

archive/v0.09/tests/extraction_experiment/test_cleaned_epub.py

@@ -0,0 +1,79 @@
+"""
+测试清理后的 ePub 提取效果
+"""
+
+import sys
+from pathlib import Path
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from ebooklib import epub
+from loguru import logger
+
+sys.path.insert(0, str(Path(__file__).parent))
+from extractors.one_to_one import OneToOneExtractor
+
+
+def test_cleaned_epub():
+    """测试清理后的 ePub"""
+    
+    cleaned_path = project_root / "test_output" / "On_China_cleaned.epub"
+    
+    if not cleaned_path.exists():
+        print(f"❌ 清理后的 ePub 不存在: {cleaned_path}")
+        return
+    
+    print("\n" + "="*80)
+    print("测试清理后的 ePub 提取效果")
+    print("="*80 + "\n")
+    
+    # 加载 ePub
+    book = epub.read_epub(str(cleaned_path))
+    
+    # 找诗歌部分
+    for item in book.get_items():
+        if item.get_type() == 9 and 'dummy_split_010' in item.get_name():
+            content = item.get_content().decode('utf-8')
+            
+            print(f"测试文件: {item.get_name()}\n")
+            
+            # 提取
+            extractor = OneToOneExtractor()
+            items = extractor.extract(content, item.get_name())
+            
+            print(f"提取了 {len(items)} 个元素\n")
+            
+            # 显示前10个
+            for i, elem in enumerate(items[:10], 1):
+                print(f"{i}. <{elem['tag']}> {elem['text'][:60]}...")
+            
+            print(f"\n... (共 {len(items)} 个元素)")
+            
+            # 检查诗歌部分
+            print("\n" + "="*80)
+            print("检查诗歌部分:")
+            print("="*80 + "\n")
+            
+            poem_lines = [item for item in items if 'ruler' in item['text'].lower() or 'mobilize' in item['text'].lower()]
+            
+            if poem_lines:
+                print(f"找到 {len(poem_lines)} 行诗歌:")
+                for i, line in enumerate(poem_lines[:5], 1):
+                    print(f"  {i}. {line['text'][:50]}")
+            else:
+                print("❌ 未找到诗歌部分")
+            
+            break
+
+
+def main():
+    """主函数"""
+    logger.remove()
+    logger.add(sys.stderr, level="INFO")
+    
+    test_cleaned_epub()
+
+
+if __name__ == "__main__":
+    main()

archive/v0.09/tests/extraction_experiment/test_decorative.py

@@ -0,0 +1,200 @@
+"""
+装饰性元素提取测试
+
+验证增强提取器对装饰性符号的识别和保留
+"""
+
+import sys
+from pathlib import Path
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from ebooklib import epub
+from loguru import logger
+
+sys.path.insert(0, str(Path(__file__).parent))
+from extractors.enhanced_bs4 import EnhancedBS4Extractor
+from extractors.bs4_optimized import BS4OptimizedExtractor
+
+
+def test_decorative_elements():
+    """测试装饰性元素的识别"""
+    html = """
+    <html>
+        <body>
+            <h1>Chapter 1</h1>
+            <p>This is a normal paragraph.</p>
+            
+            <!-- 装饰性分隔符 -->
+            <p class="separator">***</p>
+            <p>• • •</p>
+            <div class="divider">———</div>
+            <hr/>
+            
+            <h2>Section 1.1</h2>
+            <p>Another paragraph here.</p>
+            
+            <!-- 装饰性符号 -->
+            <p>◆◇◆</p>
+            
+            <p>Final paragraph.</p>
+        </body>
+    </html>
+    """
+    
+    print("\n" + "="*60)
+    print("装饰性元素识别测试")
+    print("="*60)
+    
+    # 标准提取器
+    print("\n--- 标准 BS4 提取器 ---")
+    standard_extractor = BS4OptimizedExtractor(min_text_length=3)
+    standard_items = standard_extractor.extract(html)
+    
+    print(f"提取元素数: {len(standard_items)}")
+    for i, item in enumerate(standard_items, 1):
+        print(f"{i}. [{item['tag']}] {item['text'][:50]}")
+    
+    # 增强提取器
+    print("\n--- 增强 BS4 提取器 (保留装饰性元素) ---")
+    enhanced_extractor = EnhancedBS4Extractor(min_text_length=10, preserve_decorative=True)
+    enhanced_items = enhanced_extractor.extract(html)
+    
+    print(f"提取元素数: {len(enhanced_items)}")
+    decorative_count = 0
+    for i, item in enumerate(enhanced_items, 1):
+        decorative_flag = " [装饰性]" if item.get('is_decorative') else ""
+        print(f"{i}. [{item['tag']}] {item['text'][:50]}{decorative_flag}")
+        if item.get('is_decorative'):
+            decorative_count += 1
+    
+    print(f"\n装饰性元素数: {decorative_count}")
+    
+    # 对比
+    print("\n" + "-"*60)
+    print(f"标准提取器: {len(standard_items)} 个元素")
+    print(f"增强提取器: {len(enhanced_items)} 个元素 (含 {decorative_count} 个装饰性)")
+    print(f"差异: +{len(enhanced_items) - len(standard_items)} 个元素")
+
+
+def test_real_epub_decorative():
+    """测试真实 ePub 中的装饰性元素"""
+    epub_path = project_root / "input" / "Gambling Man.epub"
+    
+    if not epub_path.exists():
+        print(f"\n跳过真实 ePub 测试: 文件不存在")
+        return
+    
+    print("\n" + "="*60)
+    print(f"真实 ePub 装饰性元素测试")
+    print("="*60)
+    
+    book = epub.read_epub(str(epub_path))
+    
+    # 提取前几个 HTML 文档
+    html_docs = []
+    for item in book.get_items():
+        if item.get_type() == 9:  # ITEM_DOCUMENT
+            try:
+                content = item.get_content().decode('utf-8')
+                html_docs.append((item.get_name(), content))
+                if len(html_docs) >= 5:
+                    break
+            except:
+                continue
+    
+    total_decorative = 0
+    
+    for filename, html_content in html_docs:
+        print(f"\n--- 文件: {filename} ---")
+        
+        # 标准提取
+        standard_extractor = BS4OptimizedExtractor()
+        standard_items = standard_extractor.extract(html_content)
+        
+        # 增强提取
+        enhanced_extractor = EnhancedBS4Extractor(preserve_decorative=True)
+        enhanced_items = enhanced_extractor.extract(html_content)
+        
+        decorative_items = [item for item in enhanced_items if item.get('is_decorative')]
+        total_decorative += len(decorative_items)
+        
+        print(f"标准提取: {len(standard_items)} 个元素")
+        print(f"增强提取: {len(enhanced_items)} 个元素")
+        print(f"装饰性元素: {len(decorative_items)} 个")
+        
+        if decorative_items:
+            print("\n装饰性元素示例:")
+            for item in decorative_items[:3]:
+                print(f"  - [{item['tag']}] {item['text'][:30]}")
+    
+    print("\n" + "="*60)
+    print(f"总计发现 {total_decorative} 个装饰性元素")
+
+
+def test_decorative_preservation():
+    """测试装饰性元素在回填时的保留"""
+    html = """
+    <html>
+        <body>
+            <p>First paragraph.</p>
+            <p>***</p>
+            <p>Second paragraph.</p>
+        </body>
+    </html>
+    """
+    
+    print("\n" + "="*60)
+    print("装饰性元素回填保留测试")
+    print("="*60)
+    
+    extractor = EnhancedBS4Extractor(min_text_length=5, preserve_decorative=True)
+    items = extractor.extract(html)
+    
+    print(f"\n提取了 {len(items)} 个元素:")
+    for i, item in enumerate(items, 1):
+        decorative_flag = " [装饰性]" if item.get('is_decorative') else ""
+        print(f"{i}. {item['text']}{decorative_flag}")
+    
+    # 创建翻译映射(只翻译非装饰性元素)
+    translation_map = {}
+    for i, item in enumerate(items):
+        if not item.get('is_decorative'):
+            translation_map[item['path']] = f"TRANSLATED_{i}"
+    
+    print(f"\n待翻译: {len(translation_map)} 个元素")
+    
+    # 回填
+    backfilled_html = extractor.backfill(html, translation_map)
+    
+    print("\n回填后的 HTML:")
+    from bs4 import BeautifulSoup
+    soup = BeautifulSoup(backfilled_html, 'html.parser')
+    for p in soup.find_all('p'):
+        print(f"  <p>{p.get_text()}</p>")
+    
+    # 验证装饰性元素是否保留
+    if '***' in backfilled_html:
+        print("\n✅ 装饰性符号 '***' 已保留")
+    else:
+        print("\n❌ 装饰性符号 '***' 丢失")
+
+
+def main():
+    """主函数"""
+    logger.remove()
+    logger.add(sys.stderr, level="INFO")
+    
+    # 测试 1: 装饰性元素识别
+    test_decorative_elements()
+    
+    # 测试 2: 真实 ePub
+    test_real_epub_decorative()
+    
+    # 测试 3: 回填保留
+    test_decorative_preservation()
+
+
+if __name__ == "__main__":
+    main()

archive/v0.09/tests/extraction_experiment/test_end_to_end.py

@@ -0,0 +1,310 @@
+"""
+端到端测试: 清理 → 提取 → 模拟翻译 → 回填 → 生成双语 EPUB
+
+完整流程验证
+"""
+
+import sys
+from pathlib import Path
+import hashlib
+import os
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from ebooklib import epub
+from loguru import logger
+
+sys.path.insert(0, str(Path(__file__).parent))
+from extractors.fine_grained import FineGrainedExtractor
+
+
+# Added MockTranslator class
+class MockTranslator:
+    def translate(self, text: str) -> str:
+        """
+        模拟翻译: 在文本前添加 [中文] 标记
+        
+        这样可以清楚地看到哪些文本被翻译了
+        """
+        return f"[中文] {text}"
+
+
+def create_bilingual_epub(epub_path: Path, output_path: Path):
+    """创建双语 EPUB"""
+    print(f"\n{'='*80}")
+    print(f"端到端测试: 生成双语 EPUB")
+    print(f"{'='*80}\n")
+    
+    print(f"输入: {os.path.basename(epub_path)}")
+    print(f"输出: {os.path.basename(output_path)}")
+    
+    # 1. 读取 EPUB
+    book = epub.read_epub(str(epub_path))
+    
+    # 准备 Zip 读取以修复 CSS 链接
+    import zipfile
+    try:
+        input_zip = zipfile.ZipFile(epub_path, 'r')
+        zip_files = set(input_zip.namelist())
+    except Exception as e:
+        print(f"无法打开 Zip: {e}")
+        input_zip = None
+        zip_files = set()
+    
+    extractor = FineGrainedExtractor()
+    translator = MockTranslator() # Using the new MockTranslator class
+    
+    # Statistics variables
+    total_docs = 0
+    total_elements = 0
+    total_translated = 0
+    
+    print("\n" + "="*80)
+    print("处理统计")
+    print("="*80 + "\n")
+    
+    # 逐个文档处理
+    for item in book.get_items():
+        if item.get_type() == 9:  # ITEM_DOCUMENT
+            try:
+                # 尝试从 Zip 读取原始内容
+                file_name = item.get_name()
+                content = None
+                
+                if input_zip:
+                    # 尝试精确匹配
+                    if file_name in zip_files:
+                        try:
+                            content = input_zip.read(file_name).decode('utf-8')
+                        except:
+                            pass
+                    else:
+                        # 尝试模糊匹配 (处理路径前缀问题)
+                        # 例如 item name 是 'dummy.html', zip 是 'EPUB/dummy.html'
+                        for z_name in zip_files:
+                            if z_name.endswith(file_name) or file_name.endswith(z_name):
+                                try:
+                                    content = input_zip.read(z_name).decode('utf-8')
+                                    # print(f"Zip 模糊匹配: {file_name} -> {z_name}")
+                                    break
+                                except:
+                                    pass
+                
+                if content is None:
+                    content = item.get_content().decode('utf-8')
+                    
+                if not content.strip():
+                    continue
+                    
+                # 修复 item 的 links (如果从 Zip 读到了 link)
+                from bs4 import BeautifulSoup
+                if input_zip: # Only attempt if zipfile was successfully opened
+                    soup = BeautifulSoup(content, 'html.parser')
+                    head = soup.find('head')
+                    if head:
+                        links = head.find_all('link', rel='stylesheet')
+                        for link in links:
+                            href = link.get('href')
+                            if href:
+                                 # 检查是否已存在
+                                existing_links = list(item.get_links())
+                                exists = False
+                                for l in existing_links:
+                                    l_href = getattr(l, 'href', None) or (l.get('href') if isinstance(l, dict) else None)
+                                    if l_href == href:
+                                        exists = True
+                                        break
+                                
+                                if not exists:
+                                    item.add_link(href=href, rel='stylesheet', type='text/css')
+                
+                # 提取
+                items = extractor.extract(content, file_name)
+                
+                if not items:
+                    continue
+                
+                total_docs += 1
+                total_elements += len(items)
+                
+                # 构建翻译映射
+                translation_map = {}
+                for elem in items:
+                    if elem['should_translate']:
+                        original_text = elem['text']
+                        translated_text = translator.translate(original_text)
+                        translation_map[original_text] = translated_text
+                        total_translated += 1
+                
+                # 回填
+                if translation_map:
+                    modified_html = extractor.backfill(items, translation_map)
+                    item.set_content(modified_html.encode('utf-8'))
+                
+            except Exception as e:
+                logger.error(f"处理失败 {item.get_name()}: {e}")
+    
+    # 修复 TOC：补全 UID 并移除指向不存在文件的死链
+    def fix_and_clean_toc(toc, book):
+        new_toc = []
+        import uuid
+        from ebooklib.epub import Link
+        
+        for item in toc:
+            if isinstance(item, (tuple, list)):
+                section, children = item
+                cleaned_children = fix_and_clean_toc(children, book)
+                
+                if isinstance(section, Link):
+                    href = section.href.split('#')[0]
+                    if book.get_item_with_href(href):
+                        if section.uid is None:
+                            section.uid = f'uuid-{uuid.uuid4()}'
+                        new_toc.append((section, cleaned_children))
+                    else:
+                        print(f"移除无效 TOC 节点: {section.href}")
+                        new_toc.extend(cleaned_children)
+                else:
+                    new_toc.append((section, cleaned_children))
+
+            elif isinstance(item, Link):
+                href = item.href.split('#')[0]
+                if book.get_item_with_href(href):
+                     if item.uid is None:
+                         item.uid = f'uuid-{uuid.uuid4()}'
+                     new_toc.append(item)
+                else:
+                    print(f"移除无效 TOC 节点: {item.href}")
+            else:
+                new_toc.append(item)
+        return new_toc
+    
+    try:
+        book.toc = fix_and_clean_toc(book.toc, book)
+    except Exception as e:
+        print(f"修复 TOC 失败: {e}")
+
+    # 保存
+    epub.write_epub(str(output_path), book)
+    
+    # 统计
+    print(f"{'='*80}")
+    print("处理统计")
+    print(f"{'='*80}\n")
+    print(f"处理文档数: {total_docs}")
+    print(f"提取元素数: {total_elements:,}")
+    print(f"翻译元素数: {total_translated:,}")
+    print(f"\n✅ 双语 EPUB 已生成: {output_path}\n")
+
+
+def verify_bilingual_epub(epub_path: Path):
+    """验证双语 EPUB"""
+    print(f"{'='*80}")
+    print(f"验证双语 EPUB")
+    print(f"{'='*80}\n")
+    
+    book = epub.read_epub(str(epub_path))
+    
+    # 检查第一个有内容的核心文档
+    for item in book.get_items():
+        if item.get_type() == 9:
+            content = item.get_content().decode('utf-8')
+            
+            # 跳过空文档
+            if len(content) < 100:
+                continue
+            
+            # 检查是否包含 [中文] 标记
+            if '[中文]' in content:
+                count = content.count('[中文]')
+                print(f"✅ 发现 {count} 个翻译标记\n")
+                
+                # 显示部分内容
+                from bs4 import BeautifulSoup
+                soup = BeautifulSoup(content, 'html.parser')
+                
+                paragraphs = soup.find_all('p')
+                print(f"段落总数: {len(paragraphs)}\n")
+                print("前10个段落:\n")
+                
+                for i, p in enumerate(paragraphs[:10], 1):
+                    text = p.get_text(strip=True)
+                    preview = text[:80]
+                    if len(text) > 80:
+                        preview += "..."
+                    
+                    # 标记译文段落
+                    is_translation = 'translation' in p.get('class', [])
+                    marker = " [译文]" if is_translation else " [原文]"
+                    
+                    print(f"{i}. {preview}{marker}")
+                
+                print()
+                # 找到一个有效的验证文件后退出循环
+                break
+            else:
+                print("ℹ️ 该文档无翻译标记 (可能无翻译内容)，继续查找下一个...\n")
+                continue
+    
+    else:
+        # 如果循环结束还没找到
+        print("❌ 在所有文档中均未发现翻译标记！\n")
+
+
+def main():
+    """主函数"""
+    import sys
+    logger.remove()
+    logger.add(sys.stderr, level="ERROR")
+    from simple_cleaner import clean_epub
+    
+    input_file = "On_China_Henry_Kissinger.epub"
+    if len(sys.argv) > 1:
+        input_file = sys.argv[1]
+    
+    # 推断路径
+    epub_name = Path(input_file).name
+    epub_stem = Path(input_file).stem
+    
+    # 查找输入文件
+    input_path = Path(input_file)
+    if not input_path.exists():
+        input_path = project_root / "input" / epub_name
+    
+    if not input_path.exists():
+        print(f"❌ 输入文件不存在: {input_path}")
+        # 尝试看看是不是已经在 test_output 下的 cleaned 文件
+        cleaned_path = project_root / "test_output" / input_file
+        if cleaned_path.exists() and "cleaned" in str(cleaned_path):
+             print(f"⚠️ 检测到已清理文件，跳过清理步骤: {cleaned_path}")
+        else:
+            return
+    else:
+        # 执行清理
+        cleaned_file = f"{epub_stem}_cleaned.epub"
+        cleaned_path = project_root / "test_output" / cleaned_file
+        
+        print(f"正在清理: {input_path.name} -> {cleaned_path.name}")
+        try:
+            clean_epub(str(input_path), str(cleaned_path))
+        except Exception as e:
+             print(f"❌ 清理失败: {e}")
+             return
+
+    # 生成双语
+    bilingual_file = f"{epub_stem}_bilingual.epub"
+    if "cleaned" in epub_stem:
+        bilingual_file = epub_stem.replace("_cleaned", "_bilingual") + ".epub"
+        
+    bilingual_path = project_root / "test_output" / bilingual_file
+    
+    # 生成双语 EPUB
+    create_bilingual_epub(cleaned_path, bilingual_path)
+    
+    # 验证
+    verify_bilingual_epub(bilingual_path)
+
+
+if __name__ == "__main__":
+    main()

archive/v0.09/tests/extraction_experiment/test_extraction.py

@@ -0,0 +1,368 @@
+"""
+文本提取实验主测试脚本
+
+对比不同提取方案的效果,生成详细报告
+"""
+
+import sys
+from pathlib import Path
+from datetime import datetime
+from loguru import logger
+
+# 添加项目根目录到路径
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from ebooklib import epub
+from extractors.bs4_optimized import BS4OptimizedExtractor
+from extractors.lxml_xpath import LxmlXPathExtractor
+from extractors.baseline_pandoc import PandocBaseline
+from validators.completeness_check import CompletenessValidator
+from validators.backfill_check import BackfillValidator
+
+
+class ExtractionExperiment:
+    """文本提取实验"""
+    
+    def __init__(self, epub_path: str):
+        """
+        初始化实验
+        
+        Args:
+            epub_path: ePub 文件路径
+        """
+        self.epub_path = Path(epub_path)
+        if not self.epub_path.exists():
+            raise FileNotFoundError(f"ePub 文件不存在: {epub_path}")
+        
+        # 初始化提取器
+        self.bs4_extractor = BS4OptimizedExtractor()
+        self.lxml_extractor = LxmlXPathExtractor()
+        self.pandoc_baseline = PandocBaseline()
+        
+        # 初始化验证器
+        self.completeness_validator = CompletenessValidator()
+        self.backfill_validator = BackfillValidator()
+        
+        # 加载 ePub
+        self.book = epub.read_epub(str(self.epub_path))
+        
+        logger.info(f"加载 ePub: {self.epub_path.name}")
+    
+    def run_experiment(self) -> dict:
+        """
+        运行完整实验
+        
+        Returns:
+            实验结果字典
+        """
+        results = {
+            'file_name': self.epub_path.name,
+            'timestamp': datetime.now().isoformat(),
+            'methods': {}
+        }
+        
+        # 1. 获取 Pandoc 基准
+        logger.info("步骤 1: 提取 Pandoc 基准")
+        baseline_text = self.pandoc_baseline.extract_from_epub(str(self.epub_path))
+        
+        if baseline_text:
+            results['baseline_length'] = len(baseline_text)
+            logger.info(f"Pandoc 基准: {len(baseline_text)} 字符")
+        else:
+            logger.warning("Pandoc 基准提取失败,将跳过覆盖率对比")
+            results['baseline_length'] = 0
+        
+        # 2. 获取测试 HTML 内容
+        logger.info("步骤 2: 提取 ePub 中的 HTML 内容")
+        html_contents = self._extract_html_from_epub()
+        logger.info(f"提取了 {len(html_contents)} 个 HTML 文档")
+        
+        if not html_contents:
+            logger.error("未找到 HTML 内容")
+            return results
+        
+        # 合并所有 HTML(用于整体测试)
+        combined_html = "\n\n".join(html_contents)
+        
+        # 3. 测试 BS4 优化方案
+        logger.info("步骤 3: 测试 BS4 优化方案")
+        bs4_results = self._test_extractor(
+            "BS4 优化方案",
+            self.bs4_extractor,
+            combined_html,
+            baseline_text
+        )
+        results['methods']['BS4 优化方案'] = bs4_results
+        
+        # 4. 测试 lxml 方案
+        logger.info("步骤 4: 测试 lxml 方案")
+        lxml_results = self._test_extractor(
+            "lxml XPath 方案",
+            self.lxml_extractor,
+            combined_html,
+            baseline_text
+        )
+        results['methods']['lxml XPath 方案'] = lxml_results
+        
+        return results
+    
+    def _extract_html_from_epub(self) -> list:
+        """从 ePub 中提取所有 HTML 文档"""
+        html_contents = []
+        
+        for item in self.book.get_items():
+            if item.get_type() == 9:  # ITEM_DOCUMENT
+                try:
+                    content = item.get_content().decode('utf-8')
+                    html_contents.append(content)
+                except Exception as e:
+                    logger.warning(f"解码失败 {item.get_name()}: {e}")
+        
+        return html_contents
+    
+    def _test_extractor(self, method_name: str, extractor, html_content: str, 
+                       baseline_text: str = None) -> dict:
+        """
+        测试单个提取器
+        
+        Args:
+            method_name: 方案名称
+            extractor: 提取器实例
+            html_content: HTML 内容
+            baseline_text: Pandoc 基准文本
+            
+        Returns:
+            测试结果字典
+        """
+        results = {}
+        
+        try:
+            # 1. 提取文本
+            items = extractor.extract(html_content)
+            results['element_count'] = len(items)
+            
+            # 合并提取的文本
+            extracted_text = " ".join([item['text'] for item in items])
+            results['text_length'] = len(extracted_text)
+            
+            logger.info(f"{method_name}: 提取了 {len(items)} 个元素, {len(extracted_text)} 字符")
+            
+            # 2. 完整性验证
+            if baseline_text:
+                coverage = self.completeness_validator.calculate_coverage(
+                    extracted_text, baseline_text
+                )
+                similarity = self.completeness_validator.calculate_similarity(
+                    extracted_text, baseline_text
+                )
+                missing_segments = self.completeness_validator.find_missing_segments(
+                    extracted_text, baseline_text
+                )
+                
+                results['coverage'] = coverage
+                results['similarity'] = similarity
+                results['missing_segments'] = missing_segments
+                
+                logger.info(f"{method_name}: 覆盖率 {coverage:.2%}, 相似度 {similarity:.2%}")
+            
+            # 3. 回填验证
+            logger.info(f"{method_name}: 测试回填准确性")
+            
+            # 位置准确性验证
+            success, failed = self.backfill_validator.validate_position_accuracy(
+                html_content, items, extractor
+            )
+            results['position_accuracy'] = success / (success + failed) if (success + failed) > 0 else 0
+            results['position_success'] = success
+            results['position_failed'] = failed
+            
+            logger.info(f"{method_name}: 位置准确性 {results['position_accuracy']:.2%}")
+            
+            # 模拟翻译回填
+            backfilled_html, backfill_results = self.backfill_validator.simulate_translation_backfill(
+                html_content, items, extractor
+            )
+            results['backfill_accuracy'] = backfill_results['accuracy']
+            results['backfill_success'] = backfill_results['success']
+            results['backfill_failed'] = backfill_results['failed']
+            
+            logger.info(f"{method_name}: 回填准确性 {results['backfill_accuracy']:.2%}")
+            
+        except Exception as e:
+            logger.error(f"{method_name} 测试失败: {e}")
+            results['error'] = str(e)
+        
+        return results
+    
+    def generate_report(self, results: dict) -> str:
+        """
+        生成实验报告
+        
+        Args:
+            results: 实验结果
+            
+        Returns:
+            Markdown 格式的报告
+        """
+        report = ["# 文本提取实验报告\n"]
+        
+        # 基本信息
+        report.append("## 基本信息\n")
+        report.append(f"- **测试文件**: {results['file_name']}")
+        report.append(f"- **测试时间**: {results['timestamp']}")
+        report.append(f"- **Pandoc 基准长度**: {results.get('baseline_length', 0):,} 字符\n")
+        
+        # 方案对比表
+        report.append("## 方案对比\n")
+        report.append("### 提取完整性\n")
+        report.append("| 方案 | 提取元素数 | 文本长度 | 覆盖率 | 相似度 |")
+        report.append("|------|-----------|---------|--------|--------|")
+        
+        for method_name, method_results in results.get('methods', {}).items():
+            if 'error' in method_results:
+                report.append(f"| {method_name} | ❌ 错误 | - | - | - |")
+            else:
+                report.append(
+                    f"| {method_name} | "
+                    f"{method_results.get('element_count', 0):,} | "
+                    f"{method_results.get('text_length', 0):,} | "
+                    f"{method_results.get('coverage', 0):.2%} | "
+                    f"{method_results.get('similarity', 0):.2%} |"
+                )
+        
+        report.append("")
+        
+        # 回填准确性
+        report.append("### 回填准确性\n")
+        report.append("| 方案 | 位置准确性 | 回填准确性 | 成功/失败 |")
+        report.append("|------|-----------|-----------|----------|")
+        
+        for method_name, method_results in results.get('methods', {}).items():
+            if 'error' not in method_results:
+                report.append(
+                    f"| {method_name} | "
+                    f"{method_results.get('position_accuracy', 0):.2%} | "
+                    f"{method_results.get('backfill_accuracy', 0):.2%} | "
+                    f"{method_results.get('backfill_success', 0)}/{method_results.get('backfill_failed', 0)} |"
+                )
+        
+        report.append("")
+        
+        # 详细分析
+        report.append("## 详细分析\n")
+        
+        for method_name, method_results in results.get('methods', {}).items():
+            report.append(f"### {method_name}\n")
+            
+            if 'error' in method_results:
+                report.append(f"**错误**: {method_results['error']}\n")
+                continue
+            
+            # 统计信息
+            report.append(f"- 提取元素数: {method_results.get('element_count', 0):,}")
+            report.append(f"- 文本总长度: {method_results.get('text_length', 0):,} 字符")
+            
+            if 'coverage' in method_results:
+                report.append(f"- 覆盖率: {method_results['coverage']:.2%}")
+                report.append(f"- 相似度: {method_results['similarity']:.2%}")
+            
+            report.append(f"- 位置准确性: {method_results.get('position_accuracy', 0):.2%}")
+            report.append(f"- 回填准确性: {method_results.get('backfill_accuracy', 0):.2%}")
+            
+            # 缺失片段
+            missing = method_results.get('missing_segments', [])
+            if missing:
+                report.append(f"\n**缺失片段** ({len(missing)} 个):\n")
+                for i, segment in enumerate(missing[:3], 1):
+                    report.append(f"{i}. {segment[:80]}...")
+                if len(missing) > 3:
+                    report.append(f"\n... 还有 {len(missing) - 3} 个片段")
+            
+            report.append("")
+        
+        # 结论
+        report.append("## 结论\n")
+        
+        # 找出最佳方案
+        best_method = None
+        best_score = 0
+        
+        for method_name, method_results in results.get('methods', {}).items():
+            if 'error' in method_results:
+                continue
+            
+            # 综合评分: 覆盖率 40% + 回填准确性 60%
+            score = (
+                method_results.get('coverage', 0) * 0.4 +
+                method_results.get('backfill_accuracy', 0) * 0.6
+            )
+            
+            if score > best_score:
+                best_score = score
+                best_method = method_name
+        
+        if best_method:
+            report.append(f"**推荐方案**: {best_method} (综合评分: {best_score:.2%})\n")
+            report.append("评分标准: 覆盖率 40% + 回填准确性 60%")
+        
+        return "\n".join(report)
+
+
+def main():
+    """主函数"""
+    # 配置日志
+    logger.remove()
+    logger.add(sys.stderr, level="INFO")
+    
+    # 测试文件
+    test_files = [
+        "input/Gambling Man.epub",
+        "input/On_China_Henry_Kissinger.epub",
+        "input/The World Atlas of Coffee - From Beans to Brewing - Coffees Explored, Explained and Enjoyed (James Hoffmann) (Z-Library).epub"
+    ]
+    
+    project_root = Path(__file__).parent.parent.parent
+    
+    for test_file in test_files:
+        epub_path = project_root / test_file
+        
+        if not epub_path.exists():
+            logger.warning(f"跳过不存在的文件: {test_file}")
+            continue
+        
+        logger.info(f"\n{'='*60}")
+        logger.info(f"测试文件: {test_file}")
+        logger.info(f"{'='*60}\n")
+        
+        try:
+            # 运行实验
+            experiment = ExtractionExperiment(str(epub_path))
+            results = experiment.run_experiment()
+            
+            # 生成报告
+            report = experiment.generate_report(results)
+            
+            # 保存报告
+            report_dir = project_root / "tests" / "extraction_experiment" / "reports"
+            report_dir.mkdir(parents=True, exist_ok=True)
+            
+            report_file = report_dir / f"{epub_path.stem}_report.md"
+            with open(report_file, 'w', encoding='utf-8') as f:
+                f.write(report)
+            
+            logger.info(f"报告已保存: {report_file}")
+            
+            # 打印摘要
+            print("\n" + "="*60)
+            print(report)
+            print("="*60 + "\n")
+            
+        except Exception as e:
+            logger.error(f"实验失败: {e}")
+            import traceback
+            traceback.print_exc()
+
+
+if __name__ == "__main__":
+    main()

archive/v0.09/tests/extraction_experiment/test_fine_grained.py

@@ -0,0 +1,95 @@
+"""
+测试细粒度提取器
+"""
+
+import sys
+from pathlib import Path
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from ebooklib import epub
+from loguru import logger
+
+sys.path.insert(0, str(Path(__file__).parent))
+from extractors.fine_grained import FineGrainedExtractor
+
+
+def test_fine_grained():
+    """测试细粒度提取"""
+    
+    cleaned_path = project_root / "test_output" / "On_China_cleaned_v2.epub"
+    
+    if not cleaned_path.exists():
+        print(f"❌ 清理后的 ePub 不存在: {cleaned_path}")
+        print("请先运行清理器生成 cleaned_v2.epub")
+        return
+    
+    print("\n" + "="*80)
+    print("细粒度提取测试")
+    print("="*80 + "\n")
+    
+    # 加载 ePub
+    book = epub.read_epub(str(cleaned_path))
+    
+    # 测试第一个文档
+    for item in book.get_items():
+        if item.get_type() == 9 and 'dummy_split_010' in item.get_name():
+            content = item.get_content().decode('utf-8')
+            
+            print(f"测试文件: {item.get_name()}\n")
+            
+            # 提取
+            extractor = FineGrainedExtractor()
+            items = extractor.extract(content, item.get_name())
+            
+            print(f"✅ 提取了 {len(items)} 个 <p> 元素\n")
+            
+            # 显示前10个
+            print("前10个元素:")
+            for i, elem in enumerate(items[:10], 1):
+                print(f"  {i}. {elem['text'][:60]}...")
+            
+            print(f"\n... (共 {len(items)} 个)")
+            
+            # 模拟翻译
+            translation_map = {}
+            for elem in items:
+                if elem['should_translate']:
+                    translation_map[elem['text']] = f"{elem['text']} [翻译]"
+            
+            print(f"\n待翻译: {len(translation_map)} 个元素")
+            
+            # 回填
+            result_html = extractor.backfill(items, translation_map)
+            
+            # 验证
+            print("\n" + "="*80)
+            print("验证:")
+            print("="*80 + "\n")
+            
+            if '[翻译]' in result_html:
+                print("✅ 翻译成功回填")
+            else:
+                print("❌ 翻译未回填")
+            
+            # 检查 <p> 数量
+            from bs4 import BeautifulSoup
+            result_soup = BeautifulSoup(result_html, 'html.parser')
+            result_p_count = len(result_soup.find_all('p'))
+            
+            print(f"✅ 回填后 <p> 元素: {result_p_count} 个")
+            
+            break
+
+
+def main():
+    """主函数"""
+    logger.remove()
+    logger.add(sys.stderr, level="INFO")
+    
+    test_fine_grained()
+
+
+if __name__ == "__main__":
+    main()

archive/v0.09/tests/extraction_experiment/test_fine_grained_extractor.py

@@ -0,0 +1,177 @@
+"""
+测试细粒度提取器
+
+验证 fine_grained.py 的提取效果
+"""
+
+import sys
+from pathlib import Path
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from ebooklib import epub
+from loguru import logger
+
+sys.path.insert(0, str(Path(__file__).parent))
+from extractors.fine_grained import FineGrainedExtractor
+
+
+def test_fine_grained_extraction(epub_path: Path):
+    """测试细粒度提取器"""
+    print(f"\n{'='*80}")
+    print(f"细粒度提取器测试: {epub_path.name}")
+    print(f"{'='*80}\n")
+    
+    # 加载 EPUB
+    book = epub.read_epub(str(epub_path))
+    
+    # 统计信息
+    total_docs = 0
+    total_elements = 0
+    total_to_translate = 0
+    total_decorative = 0
+    
+    doc_stats = []
+    
+    # 逐个文档提取
+    for item in book.get_items():
+        if item.get_type() == 9:
+            try:
+                content = item.get_content().decode('utf-8')
+                file_name = item.get_name()
+                
+                # 提取
+                extractor = FineGrainedExtractor()
+                items = extractor.extract(content, file_name)
+                
+                if items:
+                    total_docs += 1
+                    total_elements += len(items)
+                    
+                    to_translate = sum(1 for i in items if i['should_translate'])
+                    decorative = sum(1 for i in items if i.get('is_decorative'))
+                    
+                    total_to_translate += to_translate
+                    total_decorative += decorative
+                    
+                    doc_stats.append({
+                        'name': file_name,
+                        'total': len(items),
+                        'to_translate': to_translate,
+                        'decorative': decorative,
+                        'doc_type': items[0]['doc_type'] if items else 'unknown'
+                    })
+            except Exception as e:
+                logger.error(f"处理失败 {item.get_name()}: {e}")
+    
+    # 总体统计
+    print(f"{'='*80}")
+    print("总体统计")
+    print(f"{'='*80}\n")
+    print(f"处理文档数: {total_docs}")
+    print(f"提取元素总数: {total_elements:,}")
+    print(f"需要翻译: {total_to_translate:,} ({total_to_translate/total_elements*100:.1f}%)")
+    print(f"装饰性元素: {total_decorative:,} ({total_decorative/total_elements*100:.1f}%)")
+    print()
+    
+    # 按文档类型分组
+    core_docs = [d for d in doc_stats if d['doc_type'] == 'core']
+    toc_docs = [d for d in doc_stats if d['doc_type'] == 'toc']
+    skip_docs = [d for d in doc_stats if d['doc_type'] == 'skip']
+    
+    print(f"{'='*80}")
+    print("按文档类型统计")
+    print(f"{'='*80}\n")
+    
+    if core_docs:
+        core_elements = sum(d['total'] for d in core_docs)
+        core_translate = sum(d['to_translate'] for d in core_docs)
+        print(f"核心文档 (core): {len(core_docs)} 个")
+        print(f"  - 元素数: {core_elements:,}")
+        print(f"  - 需翻译: {core_translate:,}")
+        print()
+    
+    if toc_docs:
+        toc_elements = sum(d['total'] for d in toc_docs)
+        toc_translate = sum(d['to_translate'] for d in toc_docs)
+        print(f"目录文档 (toc): {len(toc_docs)} 个")
+        print(f"  - 元素数: {toc_elements:,}")
+        print(f"  - 需翻译: {toc_translate:,}")
+        print()
+    
+    if skip_docs:
+        skip_elements = sum(d['total'] for d in skip_docs)
+        print(f"跳过文档 (skip): {len(skip_docs)} 个")
+        print(f"  - 元素数: {skip_elements:,}")
+        print()
+    
+    # 显示部分文档详情
+    print(f"{'='*80}")
+    print("核心文档详情 (前10个)")
+    print(f"{'='*80}\n")
+    
+    for i, doc in enumerate(core_docs[:10], 1):
+        print(f"{i}. {doc['name']}")
+        print(f"   元素: {doc['total']}, 翻译: {doc['to_translate']}, 装饰: {doc['decorative']}")
+    
+    if len(core_docs) > 10:
+        print(f"\n... 还有 {len(core_docs) - 10} 个核心文档\n")
+    
+    # 抽样显示提取内容
+    print(f"\n{'='*80}")
+    print("提取内容抽样 (第一个核心文档的前10个元素)")
+    print(f"{'='*80}\n")
+    
+    if core_docs:
+        first_doc_name = core_docs[0]['name']
+        
+        # 重新提取第一个文档
+        for item in book.get_items():
+            if item.get_type() == 9 and item.get_name() == first_doc_name:
+                content = item.get_content().decode('utf-8')
+                extractor = FineGrainedExtractor()
+                items = extractor.extract(content, first_doc_name)
+                
+                print(f"文档: {first_doc_name}\n")
+                
+                for i, elem in enumerate(items[:10], 1):
+                    translate_flag = "✓" if elem['should_translate'] else "✗"
+                    decorative_flag = " [装饰]" if elem.get('is_decorative') else ""
+                    
+                    text_preview = elem['text'][:60]
+                    if len(elem['text']) > 60:
+                        text_preview += "..."
+                    
+                    print(f"{i}. [{translate_flag}] <{elem['tag']}> {text_preview}{decorative_flag}")
+                
+                if len(items) > 10:
+                    print(f"\n... 还有 {len(items) - 10} 个元素")
+                
+                break
+    
+    print()
+
+
+def main():
+    """主函数"""
+    logger.remove()
+    logger.add(sys.stderr, level="ERROR")
+    
+    # 测试清理后的 EPUB
+    cleaned_file = "On_China_cleaned.epub"
+    cleaned_path = project_root / "test_output" / cleaned_file
+    
+    if not cleaned_path.exists():
+        print(f"❌ 清理文件不存在: {cleaned_path}")
+        print(f"\n提示: 请先运行清理器:")
+        print(f"  python tests/extraction_experiment/simple_cleaner.py \\")
+        print(f"    'input/On_China_Henry_Kissinger.epub' \\")
+        print(f"    'test_output/{cleaned_file}'")
+        return
+    
+    test_fine_grained_extraction(cleaned_path)
+
+
+if __name__ == "__main__":
+    main()

archive/v0.09/tests/extraction_experiment/test_multi_epub.py

@@ -0,0 +1,272 @@
+"""
+优化的完整性测试脚本
+
+快速对比多个 ePub 的提取完整性,与 Pandoc 基准对比
+"""
+
+import sys
+from pathlib import Path
+from datetime import datetime
+from loguru import logger
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from ebooklib import epub
+
+sys.path.insert(0, str(Path(__file__).parent))
+from extractors.enhanced_bs4 import EnhancedBS4Extractor
+from extractors.baseline_pandoc import PandocBaseline
+
+
+def quick_coverage_check(extracted_text: str, baseline_text: str) -> dict:
+    """
+    快速覆盖率检查(优化版)
+    
+    使用简化的词级别对比,避免复杂的相似度计算
+    """
+    import re
+    
+    # 标准化
+    def normalize(text):
+        text = text.lower()
+        text = re.sub(r'[^\w\s]', ' ', text)
+        text = re.sub(r'\s+', ' ', text)
+        return text.strip()
+    
+    extracted_norm = normalize(extracted_text)
+    baseline_norm = normalize(baseline_text)
+    
+    # 分词
+    extracted_words = set(extracted_norm.split())
+    baseline_words = set(baseline_norm.split())
+    
+    if not baseline_words:
+        return {'coverage': 0.0, 'common_words': 0, 'baseline_words': 0}
+    
+    common = extracted_words & baseline_words
+    coverage = len(common) / len(baseline_words)
+    
+    return {
+        'coverage': coverage,
+        'common_words': len(common),
+        'baseline_words': len(baseline_words),
+        'extracted_words': len(extracted_words)
+    }
+
+
+def test_single_epub(epub_path: Path, use_pandoc: bool = True) -> dict:
+    """
+    测试单个 ePub 文件
+    
+    Args:
+        epub_path: ePub 文件路径
+        use_pandoc: 是否使用 Pandoc 基准
+        
+    Returns:
+        测试结果字典
+    """
+    result = {
+        'file_name': epub_path.name,
+        'file_size': epub_path.stat().st_size,
+        'timestamp': datetime.now().isoformat()
+    }
+    
+    try:
+        # 1. Pandoc 基准(可选)
+        baseline_text = None
+        if use_pandoc:
+            logger.info(f"提取 Pandoc 基准: {epub_path.name}")
+            pandoc = PandocBaseline()
+            baseline_text = pandoc.extract_from_epub(str(epub_path))
+            
+            if baseline_text:
+                result['baseline_length'] = len(baseline_text)
+                result['baseline_words'] = len(baseline_text.split())
+        
+        # 2. 加载 ePub
+        logger.info(f"加载 ePub: {epub_path.name}")
+        book = epub.read_epub(str(epub_path))
+        
+        # 3. 提取 HTML 内容
+        html_docs = []
+        for item in book.get_items():
+            if item.get_type() == 9:  # ITEM_DOCUMENT
+                try:
+                    content = item.get_content().decode('utf-8')
+                    html_docs.append(content)
+                except:
+                    continue
+        
+        result['html_doc_count'] = len(html_docs)
+        
+        # 合并 HTML
+        combined_html = "\n\n".join(html_docs)
+        
+        # 4. 增强提取器测试
+        logger.info(f"测试增强提取器")
+        extractor = EnhancedBS4Extractor(preserve_decorative=True)
+        items = extractor.extract(combined_html)
+        
+        # 统计
+        decorative_items = [i for i in items if i.get('is_decorative')]
+        nav_items = [i for i in items if i.get('is_navigation')]
+        content_items = [i for i in items if not i.get('is_navigation') and not i.get('is_decorative')]
+        
+        result['total_elements'] = len(items)
+        result['content_elements'] = len(content_items)
+        result['decorative_elements'] = len(decorative_items)
+        result['navigation_elements'] = len(nav_items)
+        
+        # 提取的文本
+        extracted_text = " ".join([item['text'] for item in content_items])
+        result['extracted_length'] = len(extracted_text)
+        result['extracted_words'] = len(extracted_text.split())
+        
+        # 5. 与 Pandoc 对比
+        if baseline_text:
+            coverage_result = quick_coverage_check(extracted_text, baseline_text)
+            result['coverage'] = coverage_result['coverage']
+            result['common_words'] = coverage_result['common_words']
+        
+        result['status'] = 'success'
+        
+    except Exception as e:
+        logger.error(f"测试失败 {epub_path.name}: {e}")
+        result['status'] = 'failed'
+        result['error'] = str(e)
+    
+    return result
+
+
+def generate_summary_report(results: list) -> str:
+    """生成汇总报告"""
+    report = ["# 多 ePub 提取完整性测试报告\n"]
+    
+    report.append(f"**测试时间**: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n")
+    report.append(f"**测试文件数**: {len(results)}\n")
+    
+    # 成功/失败统计
+    success_count = sum(1 for r in results if r.get('status') == 'success')
+    report.append(f"**成功**: {success_count}/{len(results)}\n")
+    
+    # 汇总表
+    report.append("## 测试结果汇总\n")
+    report.append("| 文件名 | 文件大小 | 提取元素 | 装饰性 | 文本长度 | 覆盖率 |")
+    report.append("|--------|---------|---------|--------|---------|--------|")
+    
+    for r in results:
+        if r.get('status') != 'success':
+            report.append(f"| {r['file_name'][:30]} | - | ❌ 失败 | - | - | - |")
+            continue
+        
+        file_size = f"{r.get('file_size', 0) / 1024:.1f}KB"
+        total_elem = r.get('total_elements', 0)
+        decorative = r.get('decorative_elements', 0)
+        text_len = f"{r.get('extracted_length', 0):,}"
+        coverage = r.get('coverage', 0)
+        coverage_str = f"{coverage:.1%}" if coverage > 0 else "N/A"
+        
+        report.append(
+            f"| {r['file_name'][:30]} | {file_size} | {total_elem} | {decorative} | {text_len} | {coverage_str} |"
+        )
+    
+    report.append("")
+    
+    # 详细分析
+    report.append("## 详细分析\n")
+    
+    for r in results:
+        if r.get('status') != 'success':
+            continue
+        
+        report.append(f"### {r['file_name']}\n")
+        report.append(f"- **HTML 文档数**: {r.get('html_doc_count', 0)}")
+        report.append(f"- **提取元素总数**: {r.get('total_elements', 0)}")
+        report.append(f"  - 内容元素: {r.get('content_elements', 0)}")
+        report.append(f"  - 装饰性元素: {r.get('decorative_elements', 0)}")
+        report.append(f"  - 导航元素: {r.get('navigation_elements', 0)}")
+        report.append(f"- **提取文本长度**: {r.get('extracted_length', 0):,} 字符")
+        report.append(f"- **提取词数**: {r.get('extracted_words', 0):,}")
+        
+        if 'baseline_length' in r:
+            report.append(f"- **Pandoc 基准长度**: {r.get('baseline_length', 0):,} 字符")
+            report.append(f"- **覆盖率**: {r.get('coverage', 0):.2%}")
+            report.append(f"- **共同词数**: {r.get('common_words', 0):,}")
+        
+        report.append("")
+    
+    # 总结
+    report.append("## 总结\n")
+    
+    if success_count > 0:
+        avg_coverage = sum(r.get('coverage', 0) for r in results if r.get('status') == 'success') / success_count
+        total_decorative = sum(r.get('decorative_elements', 0) for r in results if r.get('status') == 'success')
+        
+        report.append(f"- **平均覆盖率**: {avg_coverage:.2%}")
+        report.append(f"- **总装饰性元素**: {total_decorative} 个")
+        report.append(f"- **提取器状态**: {'✅ 正常' if avg_coverage > 0.9 else '⚠️ 需要优化'}")
+    
+    return "\n".join(report)
+
+
+def main():
+    """主函数"""
+    logger.remove()
+    logger.add(sys.stderr, level="INFO")
+    
+    # 测试文件列表
+    test_files = [
+        "Gambling Man.epub",
+        "On_China_Henry_Kissinger.epub",
+        "The World Atlas of Coffee - From Beans to Brewing - Coffees Explored, Explained and Enjoyed (James Hoffmann) (Z-Library).epub",
+        "The_Philosopher_in_the_Valley.epub",
+        "To_Explain_the_World.epub"
+    ]
+    
+    input_dir = project_root / "input"
+    results = []
+    
+    for filename in test_files:
+        epub_path = input_dir / filename
+        
+        if not epub_path.exists():
+            logger.warning(f"跳过不存在的文件: {filename}")
+            continue
+        
+        logger.info(f"\n{'='*60}")
+        logger.info(f"测试: {filename}")
+        logger.info(f"{'='*60}")
+        
+        result = test_single_epub(epub_path, use_pandoc=True)
+        results.append(result)
+        
+        # 打印简要结果
+        if result.get('status') == 'success':
+            logger.info(f"✅ 成功: {result.get('total_elements')} 个元素, "
+                       f"{result.get('decorative_elements')} 个装饰性, "
+                       f"覆盖率 {result.get('coverage', 0):.1%}")
+        else:
+            logger.error(f"❌ 失败: {result.get('error')}")
+    
+    # 生成报告
+    report = generate_summary_report(results)
+    
+    # 保存报告
+    report_dir = project_root / "tests" / "extraction_experiment" / "reports"
+    report_dir.mkdir(parents=True, exist_ok=True)
+    
+    report_file = report_dir / "multi_epub_test_report.md"
+    with open(report_file, 'w', encoding='utf-8') as f:
+        f.write(report)
+    
+    logger.info(f"\n报告已保存: {report_file}")
+    
+    # 打印报告
+    print("\n" + "="*60)
+    print(report)
+    print("="*60)
+
+
+if __name__ == "__main__":
+    main()

archive/v0.09/tests/extraction_experiment/test_one_to_one.py

@@ -0,0 +1,103 @@
+"""
+测试一比一对应提取器
+"""
+
+import sys
+from pathlib import Path
+
+project_root = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(project_root))
+
+from loguru import logger
+
+sys.path.insert(0, str(Path(__file__).parent))
+from extractors.one_to_one import OneToOneExtractor
+
+
+def test_one_to_one():
+    """测试一比一对应"""
+    
+    # 测试 HTML
+    html = """
+    <div class="poem">
+        <div class="line"><span class="italic">War is</span></div>
+        <div class="line"><span class="italic">A grave affair of the state;</span></div>
+        <div class="line"><span class="italic">It is a place</span></div>
+    </div>
+    """
+    
+    print("\n" + "="*80)
+    print("一比一对应测试")
+    print("="*80 + "\n")
+    
+    print("原始 HTML:")
+    print(html)
+    print()
+    
+    # 提取
+    extractor = OneToOneExtractor()
+    items = extractor.extract(html)
+    
+    print(f"提取了 {len(items)} 个元素:\n")
+    
+    for i, item in enumerate(items, 1):
+        print(f"{i}. <{item['tag']}> {item['text']}")
+        print(f"   文本节点数: {len(item['text_nodes'])}")
+        for j, (node, text) in enumerate(item['text_nodes'], 1):
+            print(f"     节点 {j}: '{text}'")
+        print()
+    
+    # 创建翻译映射
+    translation_map = {}
+    for item in items:
+        if item['should_translate']:
+            translation_map[item['text']] = f"{item['text']} [翻译]"
+    
+    print(f"待翻译: {len(translation_map)} 个元素\n")
+    
+    # 回填
+    result_html = extractor.backfill(items, translation_map)
+    
+    print("="*80)
+    print("回填后的 HTML:")
+    print("="*80 + "\n")
+    print(result_html)
+    
+    # 验证
+    print("\n" + "="*80)
+    print("验证:")
+    print("="*80 + "\n")
+    
+    if '<span class="italic">War is [翻译]</span>' in result_html:
+        print("✅ 第1行格式保留")
+    else:
+        print("❌ 第1行格式丢失")
+    
+    if '<span class="italic">A grave affair of the state; [翻译]</span>' in result_html:
+        print("✅ 第2行格式保留")
+    else:
+        print("❌ 第2行格式丢失")
+    
+    if '<span class="italic">It is a place [翻译]</span>' in result_html:
+        print("✅ 第3行格式保留")
+    else:
+        print("❌ 第3行格式丢失")
+    
+    
+    div_count = result_html.count('<div class="line">')
+    if div_count == 3:
+        print("✅ 所有3个 div 都保留")
+    else:
+        print(f"❌ div 数量错误: {div_count}")
+
+
+def main():
+    """主函数"""
+    logger.remove()
+    logger.add(sys.stderr, level="INFO")
+    
+    test_one_to_one()
+
+
+if __name__ == "__main__":
+    main()