kai/epub_bilingual_translator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Release v0.10 - Modular Architecture & External Config

Browse Source 
- Refactor codebase into src/ (preprocessing, translation, assembly)
- Add pipeline/ scripts for individual stages
- Externalize configuration to config/config.yaml
- Fix Cover Image preservation
- Update documentation and manuals
This commit is contained in:
谭凯
2026-01-31 22:49:44 +08:00
parent 9ef82393be
commit 7a93c52b42
306 changed files with 30313 additions and 1071 deletions
Download Patch File Download Diff File Expand all files Collapse all files
archive/v0.09/CHANGELOG.md
+94
View File
@@ -0,0 +1,94 @@
# 更新日志 (CHANGELOG)
## [v0.08] - 2026-01-15 (The Optimizer)
### 🎯 占位符系统优化
- **前缀/后缀标签分离**: 文本首尾的纯格式标签（如 `<sup>`, `<sub>`）不再发送给 LLM，自动回填。
- **公式检测**: 数学变量和公式被识别为单一、不可翻译的占位符，显著降低 LLM 误翻译风险。
- **简化占位符格式**: 从全局唯一 `φcXXXXXφ` 简化为段落局部 `φ1φ`，每个段落独立编号。
### 🎨 纯中文排版模式
- **模式切换**: 支持 `--mode chinese` 生成保留原始排版的纯中文译本（默认 `--mode bilingual`）。
- **Format Extractor (替换法)**:
- 彻底重构格式提取逻辑，放弃 DOM 递归，采用基于正则的"标签序列替换法"。
- 能够完美处理任意深度的嵌套标签，将其合并为单一占位符。
- 100% 保留原始 HTML 属性（class, style, href 等），实现"像素级"格式还原。
- **Format Restorer (自愈系)**:
- 引入 `FormatRestorer` 模块，负责将占位符替换回原始 HTML 代码。
- **自动修复 Agent**: 当检测到 LLM 丢失占位符时，自动触发回退机制进行格式修复。
- **优雅降级**: 如果修复失败，系统会自动降级为纯文本，确保程序不崩溃。
### 🛡️ 深度优化
- **占位符升级**: 从易混淆的 `«c...»` 升级为 `φc...φ`，显著降低 LLM 误翻译概率。
- **容器样式继承**:
- 中文模式：直接替换 `inner_html`，完美保留外层容器属性。
- 双语模式：新建 `<p>` 标签时自动继承原文的 `class` 和其他属性。
- **智能测试**: `--test` 模式逻辑升级，智能识别章节边界，自动翻译完第一章。
### 🔧 修复
- 修复了 `LLMClient` 中正则表达式转义错误导致的 `FutureWarning`。
- 修复了 `FormatExtractor` 循环引用问题。
- 解决了复杂科学书籍中上标/链接嵌套导致的校验失败问题。
---
## [v0.07] - 2026-01-13 (The Refinement)
### 🛡️ 安全与配置
- **环境隔离**: 引入 `.env` 支持，彻底移除了代码库中的硬编码 API Key。
- **配置升级**: `utils.py` 现自动加载 `.env` 并注入到配置中，支持任意 Provider 的环境变量覆盖 (如 `V3_API_KEY`, `OPENROUTER_API_KEY`)。
- **模板化**: 新增 `config.example.json` 和 `.env` 模板，提升部署安全性。
### 🚀 核心改进
- **V3 Provider 支持**: 验证并修复了对 V3 API (OpenAI 兼容格式) 的支持，全流程跑通。
- **EPUB 构建修复**: 解决了 `ebooklib` 在处理 TOC 时因缺少 UID 导致的 `Argument must be bytes or unicode` 崩溃问题。
- **缓存优化**:
- 缓存目录结构调整为 Hash 前缀 (`cache/translations/ab/...`)，解决了按日期分目录导致的缓存频繁失效问题。
- 放宽了缓存验证逻辑，支持部分命中的缓存复用。
- **视觉优化**: 引入“盘古之白” (Pangu spacing)，自动在中文与英文/数字之间添加空格，显著提升阅读体验。
### ⚡ 体验提升
- **断点续传提示**: 启动时自动检测并提示未完成的翻译进度。
- **详细统计**: 翻译完成后展示详细的成功/失败/跳过统计数据。
- **并发优化**: 移除了冗余的信号量控制，完全依赖 `RateLimiter`，逻辑更清晰高效。
---
## [v0.05] - 2026-01-12 (The Arena)
### 🌟 核心突破
- **书籍画像 (Book Profiler)**:
- 自动提取前言和正文采样。
- 生成 `Book Profile`，包含领域 (Genre)、文风 (Style)、目标受众 (Audience) 和翻译指令。
- 生成 `Glossary` (术语表)，并支持自动注入 Prompt。
- **状态绑定**: Profile 和 Glossary 现在直接存储在每本书的 `manifest.json` 中，互不干扰。
- **模型竞技场 (Model Arena)**:
- 自动选取典型 Chunk，让多个候选模型 (Gemini, Llama, Qwen) 同台竞技。
- 引入 `Judge Agent` (基于 Smart 模型)，从准确性、信达雅维度评选最佳模型。
- 自动锁定获胜模型用于全书翻译。
### 🏗️ 架构升级
- **LLM Client 重构**:
- **Syntax Fixes**: 彻底修复了正则构造中的语法错误。
- **Quote Safety**: 移除了所有 f-string 中的复杂正则，改用安全的字符串拼接。
- **Dual RateLimiters**: 引入主/副限流器，防止死锁。
- **配置增强**:
- `config.json` 支持 `arena_models` 和 `judge_model` 配置。
### 🔧 修复与优化
- 修复了 `unhashable type: 'dict'` 错误 (移除了错误的 `{{}}`)。
- 修复了多本书连续翻译时 Profile 串用的问题 (Profile 现已绑定至 Manifest)。
---
## [v0.03] - 2026-01-12
- **极简 ID 锚点系统**: 废弃复杂的 `[p_xxxxx]` 格式，使用纯净 ID，彻底解决残留问题。
- **智能术语一致性**: 引入 GlossaryManager。
- **结构完美保留**: 修复了 EPUB Spine 和 Metadata 丢失问题。
## [v0.02] - 2026-01-12
- **Manifest 驱动架构**: 引入 `ManifestManager` 作为单一真理源。
- **流程解耦**: 提取、翻译、构建三阶段分离。
## [v0.01] - 2026-01-10
- 初始版本，实现基本的并发翻译和 EPUB 解析。
archive/v0.09/CHINESE_MODE_FORMAT_ANALYSIS.md
+163
View File
@@ -0,0 +1,163 @@
# 纯中文模式格式丢失问题 - 深度分析
## 🔴 根本原因：Mode 参数未被正确传递
### 问题概述
当前 `translator.py` 的 `translate_epub()` 方法签名只有 3 个参数：
```python
async def translate_epub(self, epub_path: str, test_mode: bool = False, output_dir: str = None) -> str:
```
**缺失 `mode` 参数！** 这意味着：
1. 无论用户指定 `-m chinese` 还是 `-m bilingual`，都走相同的代码路径
2. 永远使用 `BilingualEPUBBuilder`
3. 格式提取功能 (`format_extractor.py`) 从未被调用
---
## 问题链路分析
### main.py 调用
```python
# main.py:55-59
await translator.translate_epub(
args.epub_path,
test_mode=args.test,
output_dir=args.output,
mode=args.mode # ❌ 此参数被忽略！因为 translate_epub 不接受 mode
)
```
### translator.py 未处理 mode
```python
# translator.py:39 - 缺少 mode 参数
async def translate_epub(self, epub_path: str, test_mode: bool = False, output_dir: str = None) -> str:
# translator.py:53 - 提取时未传 mode
self.text_processor.extract_to_manifest(item['content'], item['file_name'], manifest)
# 应该是: extract_to_manifest(..., mode=mode)
# translator.py:72 - 分块时未传 mode
chunks = self.text_processor.create_chunks_from_manifest(manifest)
# 应该是: create_chunks_from_manifest(manifest, mode=mode)
# translator.py:84 - 永远使用 BilingualBuilder
builder = BilingualEPUBBuilder(self.parser.book, self.config)
# 应根据 mode 选择 ChineseEPUBBuilder
```
---
## 完整的格式保留流程 (应有逻辑)
```
┌─────────────────────────────────────────────────────────────────┐
│ 1. 提取阶段 (format_extractor.py) │
│ Input: <p>This is <strong>bold</strong> text.</p> │
│ Output: │
│ - clean_text: "This is bold text." │
│ - text_with_placeholders: "This is φc00001φboldφc00002φ text." │
│ - placeholder_map: {c00001: "<strong>", c00002: "</strong>"} │
└─────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ 2. 翻译阶段 (llm_client.py) │
│ Prompt: p_00001 [BODY] This is φc00001φboldφc00002φ text. │
│ LLM Response: p_00001 这是φc00001φ粗体φc00002φ文本。 │
│ ⚠️ 问题：LLM 可能丢失/错放占位符！ │
└─────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ 3. 还原阶段 (format_restorer.py) │
│ Input: "这是φc00001φ粗体φc00002φ文本。" │
│ Output: "这是<strong>粗体</strong>文本。" │
└─────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ 4. 构建阶段 (chinese_builder.py) │
│ 使用 translation_with_original_html 替换原始内容 │
└─────────────────────────────────────────────────────────────────┘
```
**当前状态**: 步骤 1、3、4 从未执行！
---
## LLM 占位符丢失的常见原因
即使修复了 mode 传递问题，LLM 仍可能丢失占位符：
| 原因 | 示例 | 解决方案 |
|------|------|----------|
| 占位符被"翻译" | φc00001φ → φ中00001φ | 强调 Prompt: "φ...φ 是代码，禁止修改" |
| 占位符位置错误 | 原: Aφc1φB → 译: φc1φAB | 修复 Agent (repair_format) |
| 占位符完全丢失 | 原: φc1φ → 译: (无) | 后处理：从原文恢复 |
| 幻觉占位符 | 原: (无) → 译: φc99φ | 忽略未知 ID |
---
## 修复建议
### 修复 1: translator.py 添加 mode 参数
```python
async def translate_epub(self, epub_path: str, test_mode: bool = False,
output_dir: str = None, mode: str = "bilingual") -> str:
# ...
# 提取时传递 mode
self.text_processor.extract_to_manifest(item['content'], item['file_name'], manifest, mode=mode)
# 分块时传递 mode
chunks = self.text_processor.create_chunks_from_manifest(manifest, mode=mode)
# 翻译时传递 mode
await self._translate_concurrently(chunks, manifest, profile, mode=mode)
# 构建时选择正确的 Builder
if mode == "chinese":
builder = ChineseEPUBBuilder(self.parser.book, self.config)
result_file = builder.create_chinese_epub_with_mapping(manifest.get_items(), output_path)
else:
builder = BilingualEPUBBuilder(self.parser.book, self.config)
result_file = builder.create_bilingual_epub_with_mapping(...)
```
### 修复 2: llm_client.py 添加占位符指令
```python
if mode == "chinese":
base_sys_prompt += """
Placeholder Instructions:
1. Text contains φcXXXXXφ placeholders representing HTML formatting.
2. These are CODE tokens - DO NOT translate, modify, or remove them.
3. Keep placeholders in corresponding positions in your translation.
4. Example: "This is φc00001φboldφc00002φ text." → "这是φc00001φ粗体φc00002φ文本。"
"""
```
### 修复 3: 增强 format_restorer.py 容错
```python
def restore(self, text_with_placeholders, placeholder_map):
# 如果 LLM 完全丢失占位符，尝试智能合并
if not self.PLACEHOLDER_REGEX.search(text_with_placeholders):
logger.warning("所有占位符丢失，降级为纯文本")
return text_with_placeholders, False
# 现有逻辑...
```
---
## 优先级
| 优先级 | 修复项 | 工作量 |
|--------|--------|--------|
| P0 | translator.py 传递 mode 参数 | 小 |
| P0 | translator.py 选择正确 Builder | 小 |
| P1 | llm_client.py 占位符 Prompt | 小 |
| P2 | format_restorer.py 容错增强 | 中 |
| P2 | repair_format 机制完善 | 中 |
archive/v0.09/CODE_REVIEW_REPORT.md
+260
View File
@@ -0,0 +1,260 @@
# 代码审查报告 v2 - 深度分析
## 1. 核心问题:Builder 与 Manifest 逻辑不一致
### 问题描述
`TextProcessor.extract_to_manifest()` 和 `BilingualEPUBBuilder._create_bilingual_document()` 对同一份 HTML 的处理逻辑**不一致**,导致 ID 映射错位。
### 根本原因
#### TextProcessor.extract_to_manifest (text_processor.py:51-89)
```python
for element in text_elements:
clean_text = self.clean_element_text(element)
if not clean_text:
continue # ❌ 跳过,不添加到 manifest
status = "pending"
if self.is_navigation_element(element):
status = "ignored" # ✅ 添加到 manifest,但标记为 ignored
item = manifest.add_item(...) # 添加
```
**结果**:
- 空文本元素: 不添加
- 导航元素: **添加** (ID: p_00002, status: ignored)
- 普通元素: 添加 (ID: p_00001, p_00003...)
#### BilingualEPUBBuilder._create_bilingual_document (bilingual_builder.py:143-167)
```python
current_para_index = 0
for element in text_elements:
if TextProcessor.is_navigation_element(element):
continue # ❌ 跳过,不增加索引
if not TextProcessor.clean_element_text(element):
continue # ❌ 跳过,不增加索引
target_id = ordered_ids[current_para_index] # 使用索引获取 ID
current_para_index += 1
```
**结果**:
- 空文本元素: 跳过
- 导航元素: **跳过** (索引不增加!)
- 普通元素: 使用索引 0, 1, 2...
### 错位示例
假设 HTML 结构:
```html
<p>段落1</p> <!-- clean_text: "段落1" -->
<div class="nav">导航</div> <!-- is_navigation: true -->
<p>段落2</p> <!-- clean_text: "段落2" -->
```
**Manifest 中的 ID 分配**:
- p_00001 → 段落1 (status: pending)
- p_00002 → 导航 (status: ignored)
- p_00003 → 段落2 (status: pending)
**ordered_ids**: `["p_00001", "p_00002", "p_00003"]`
**translation_map**: `{"p_00001": "Translation1", "p_00003": "Translation2"}`
**Builder 的执行**:
```
遍历 element[0] (段落1):
- 不是导航 ✓
- 有 clean_text ✓
- current_para_index = 0
- target_id = ordered_ids[0] = "p_00001" ✓
- translation = "Translation1" ✓
- 插入翻译 ✓
- current_para_index = 1
遍历 element[1] (导航):
- 是导航 ✗
- continue (跳过)
- current_para_index 仍然是 1 ❌
遍历 element[2] (段落2):
- 不是导航 ✓
- 有 clean_text ✓
- current_para_index = 1
- target_id = ordered_ids[1] = "p_00002" ❌ (应该是 p_00003!)
- translation = translation_map.get("p_00002") = None ❌
- 不插入翻译 ❌
- current_para_index = 2
```
**结果**: 段落2 没有翻译!
---
## 2. 修复方案
### 方案 A: 修改 Builder 逻辑 (推荐)
**原理**: Builder 应该与 Manifest 保持一致,遍历所有元素并正确增加索引。
```python
# bilingual_builder.py:143-167
current_para_index = 0
for element in text_elements:
clean_text = TextProcessor.clean_element_text(element)
# 与 extract_to_manifest 保持一致:跳过空文本
if not clean_text:
continue
# 关键:不再跳过导航元素,而是检查 ID 对应的翻译
if current_para_index < len(ordered_ids):
target_id = ordered_ids[current_para_index]
translation = translation_map.get(target_id)
# 只有非导航元素且有翻译时才插入
if translation and not TextProcessor.is_navigation_element(element):
self._insert_translation(element, translation, soup, element.attrs)
current_para_index += 1 # 无论是否插入,都要增加索引
```
**优点**:
- 逻辑简单,与 Manifest 一致
- 不需要修改 Manifest 或 TextProcessor
**缺点**:
- 需要同时修改 `bilingual_builder.py` 和 `chinese_builder.py`
### 方案 B: 修改 Manifest 逻辑
**原理**: 让 `extract_to_manifest` 也跳过导航元素,不添加到 manifest。
```python
# text_processor.py:51-89
for element in text_elements:
clean_text = self.clean_element_text(element)
if not clean_text:
continue
# 新增:跳过导航元素
if self.is_navigation_element(element):
continue
item = manifest.add_item(...)
```
**优点**:
- Manifest 更干净,不包含 ignored 项
**缺点**:
- 可能破坏现有的缓存/manifest 文件
- 如果将来需要处理导航元素,需要重新设计
---
## 3. 其他发现的问题
### 3.1 错误处理不足
**位置**: `translator.py:176-227`
**问题**:
- `llm_client.translate_chunk()` 返回错误字符串 (如 `"[Error - Timeout]"`)
- 这些错误字符串被当作正常翻译保存到 manifest
- 最终 EPUB 中会包含 `[Error - Timeout]` 作为段落内容
**建议**:
```python
# translator.py:178
raw_translation = results[item.global_id]
# 检测错误
if raw_translation.startswith("[Error"):
logger.warning(f"翻译失败: {item.global_id} - {raw_translation}")
manifest.update_item(item.global_id, None, status="failed", error=raw_translation)
continue
processed_translation = add_spacing_between_cn_and_en_num(raw_translation)
```
### 3.2 RateLimiter 效率问题
**位置**: `llm_client.py:19-37`
**问题**:
- 当前实现在 `acquire()` 时串行化请求发起
- 即使 `concurrent_requests=5`,也无法真正并发
**当前逻辑**:
```python
async def acquire(self):
await self.semaphore.acquire() # 等待并发槽位
async with self._lock:
# 计算等待时间
wait_time = self.min_interval - (current_time - self.last_request_time)
if wait_time > 0:
await asyncio.sleep(wait_time) # ❌ 持有锁时 sleep
self.last_request_time = time.time()
```
**问题**: `_lock` 导致所有协程串行等待,无法并发。
**建议**: 使用 Token Bucket 或 `asyncio-throttle` 库。
### 3.3 注释中的 TODO
**位置**: `bilingual_builder.py:150-158`
大量注释表明代码作者也意识到设计不完善:
```python
# 获取原文属性（如果 Manifest 中有的话，需要通过 paragraph_map 传进来吗？
# 此时 ordered_ids 只是 ID 列表。
# 我们需要让 _create_bilingual_document 访问到 paragraph_map
# ...
# 让我们重构一下：
# _create_bilingual_document(self, original_item, ordered_ids, translation_map, paragraph_map)
```
**建议**: 重构函数签名,传入完整的 `paragraph_map` 而不仅仅是 `ordered_ids`。
---
## 4. 测试建议
### 4.1 单元测试
创建测试用例验证 Builder 与 Manifest 的一致性:
```python
def test_builder_manifest_consistency():
html = """
<p>Para1</p>
<div class="nav">Nav</div>
<p>Para2</p>
"""
# 模拟 extract_to_manifest
manifest_ids = [] # 应该是 [p_1, p_2, p_3]
# 模拟 builder
builder_ids = [] # 应该也是 [p_1, p_2, p_3]
assert manifest_ids == builder_ids
```
### 4.2 集成测试
使用真实 EPUB 测试完整流程,验证:
- 翻译是否对应正确的段落
- 导航元素是否被正确忽略
- 错误处理是否生效
---
## 5. 优先级建议
1. **P0 (立即修复)**: Builder 逻辑不一致 → 方案 A
2. **P1 (重要)**: 错误处理 → 添加错误检测
3. **P2 (优化)**: RateLimiter → 使用 asyncio-throttle
4. **P3 (重构)**: 函数签名 → 传入 paragraph_map
---
**总结**: 核心问题是 Builder 与 Manifest 的遍历逻辑不一致。建议采用方案 A,修改 Builder 使其与 Manifest 保持同步。
archive/v0.09/DEVELOPER_GUIDE.md
+51
View File
@@ -0,0 +1,51 @@
# 开发者避坑指南 (Developer's Survival Guide)
这份文档总结了 EPUB 翻译器开发过程中的血泪教训。在修改代码前，**务必阅读此文档**。
## 🔴 核心原则 (Core Principles)
### 1. 奥卡姆剃刀原则 (KISS)
**不要自作聪明。**
* **错误案例**：为了“美观”或“规范”，给 ID 加上方括号 `[p_001]`，甚至试图让 LLM 返回 JSON 结构。
* **后果**：LLM 经常搞错括号的全角/半角，或者漏掉闭合括号，导致正则解析极其痛苦，甚至产生 `SyntaxError`。
* **最佳实践**：**ID 就用纯文本 `p_xxxxx`。** 解析就用 `find()` 和字符串切片。越简单越不容易出错。
### 2. 单一真理源 (Single Source of Truth)
**不要在模块间传递散乱的数据。**
* **最佳实践**：**Manifest (清单) 是唯一的真理。** Profile, Glossary, Winner Model 都应该直接存储在 Manifest 的 metadata 中，而不是依赖外部临时文件。
---
## 🚫 常见陷阱 (Pitfalls)
### 1. Python 语法陷阱
* **f-string 中的正则**：
* *Bad*: `rf'\[{id}\]'` 或 `rf"[{id}]"`。在 f-string 中使用反斜杠转义非常容易出错，尤其是涉及引号嵌套时。
* *Good*: 使用字符串拼接 `r'\[' + id + r'\]'`。虽然丑一点，但绝对安全。
* **Unhashable Dict**:
* *Bad*: `glossary = profile.get('glossary', {{}})`。双花括号 `{{}}` 在 Python 中会被解释为集合 `{dict()}`，而 dict 是不可哈希的，导致 `TypeError`。
* *Good*: `glossary = profile.get('glossary', {})`。
### 2. Prompt Engineering
* **不要让 LLM "解释" 它的翻译。**
* 它一旦开始解释，解析器就很难把正文抠出来。必须在 System Prompt 中严令禁止。
* **Context Injection**:
* 注入 Glossary 时，格式越简单越好（如 `Term -> Translation`），不要用复杂的 JSON 结构，这会消耗 Token 且容易被模型忽略。
### 3. EPUB 结构处理
* **不要随意丢弃 Item。**
* 默认复制所有非 Document 资源。对于 Document，要么替换为双语版，要么原样保留。
* **不要重建 Spine 顺序。**
* 不要试图自己去猜页面顺序。严格按照 `original_book.spine` 的顺序来构建新书。
---
## ✅ 推荐工作流 (Workflow)
1. **修改提取逻辑时** -> 必须同时检查 `get_valid_text_elements` 是否被 `Builder` 复用。
2. **修改 Prompt 时** -> 必须同步更新 `LLMClient` 的解析逻辑。
3. **调试 LLM 输出时** -> 使用 `raw_chat_completion` 接口进行单元测试。
---
*Last Updated: v0.05*
archive/v0.09/README.md
+80
View File
@@ -0,0 +1,80 @@
# EPUB 双语翻译程序 v0.10 (Architecture Refactored)
一个基于 OpenRouter/OpenAI API 的 EPUB 双语翻译工具，采用**全局编号系统**和**真并发翻译**。
v0.10 引入了全新的**清洗-提取-回填**架构，彻底解决了格式丢失和错位问题。
## ✨ 核心特性
### 🛡️ 稳健的架构 (New)
- **EpubCleaner 预处理**：自动修复 TOC 死链、缺失 UID，标准化 HTML 结构，确保输入源干净可靠。
- **FineGrained Extractor**：基于 DOM 的高精度提取，支持 `<h1>`-`<h6>` 及所有 `<p>` 标签。
- **格式保护 v2**：自动识别数学公式、代码块和行内样式，使用占位符保护，防止 LLM 破坏格式。
### 🎯 全局编号系统
- **DOM 级回填**：不再依赖脆弱的正则，而是利用 DOM 引用进行 100% 精确的一一回填。
- **双重模式**：支持 Bilingual (双语对照) 和 Chinese (纯译文保留原格式) 模式。
### ⚡ 真并发翻译
- **asyncio.gather 并发执行**：高效利用 API 速率限制
- **智能速率控制**：基于 Token 桶的 RateLimiter
- **实时进度显示**：Rich 进度条显示翻译状态
- **断点续传**：自动记录进度，随时中断随时继续
### 🔧 自动修复 (Self-Healing)
- **Format Repair**：当 LLM 返回的格式损坏时，自动触发修复机制，利用 LLM 进行自我纠正。
## 🚀 快速开始
### 1. 安装依赖
```bash
pip install -r requirements.txt
```
### 2. 配置环境
复制 `.env` 模板并填入你的 API Key：
```bash
# .env 文件
V3_API_KEY=sk-xxxxxx
OPENROUTER_API_KEY=sk-or-xxxxxx
```
### 3. 开始翻译
```bash
# 默认使用 OpenRouter (双语模式)
python main.py input/book.epub
# 纯中文模式 (保留原版样式)
python main.py input/book.epub -m chinese
# 测试模式（只翻译前10个块，快速验证）
python main.py input/book.epub --test
```
## 📂 目录结构
```
.
├── config/ # 配置文件
├── input/ # 输入 EPUB 目录
├── output/ # 输出 EPUB 目录
├── cache/ # 缓存目录 (Manifest, Translations, Processed Epubs)
├── logs/ # 运行日志
└── src/ # 源代码
├── epub_cleaner.py # 预处理器
├── fine_grained_extractor.py # 提取器
├── bilingual_builder.py # 构建器
└── translator.py # 主流程
```
## 📄 许可证
MIT License
---
**版本**: v0.10
**更新**: 2026-01-19
archive/v0.09/REFACTOR_SUMMARY.md
+399
View File
@@ -0,0 +1,399 @@
# 🎉 EPUB翻译器 v2.0 重构完成总结
## 📅 重构日期
2026-01-12
## 🎯 重构目标
1. ✅ 修复中英文错行问题
2. ✅ 实现真正的并发翻译
3. ✅ 简化代码架构
4. ✅ 提升翻译效率
---
## 🔧 核心改进
### 1. **全局编号系统**
#### 问题
- 原有缓存以单段落为key，但翻译是chunk级别
- 翻译分割导致内容错位
- 段落对应关系混乱
#### 解决方案
```python
# 每个段落分配全局唯一ID
p_0001, p_0002, p_0003, ...
# 数据流
段落提取 → 分配ID → 分块 → 翻译 → 精确匹配
```
#### 效果
- ✅ 完全杜绝中英文错行
- ✅ 缓存基于ID序列，精确可靠
- ✅ 翻译结果可追溯
---
### 2. **真并发翻译**
#### 问题（原有代码）
```python
# 串行执行
for chunk in chunks:
result = await translate(chunk) # 等待完成
# 下一个才开始
```
**实际并发数：1** （虽然配置了8）
#### 解决方案（新代码）
```python
# 并发执行
tasks = [translate(chunk) for chunk in chunks]
results = await asyncio.gather(*tasks) # 同时执行
```
**实际并发数：8** （受Semaphore控制）
#### 效果
- ✅ 翻译速度提升 **7-8倍**
- ✅ 100个chunks从100秒降到13秒
- ✅ 充分利用API并发能力
---
### 3. **代码架构简化**
#### 删除的冗余代码
1. ❌ 复杂的目录解析逻辑（章节、序言、尾声分类）
2. ❌ 复杂的段落排序算法
3. ❌ 章节边界切割逻辑
4. ❌ 过时的配置参数（max_context_length等）
5. ❌ 多余的文本清理规则
#### 保留的核心功能
1. ✅ 段落提取（简化版）
2. ✅ 全局编号
3. ✅ 智能分块（不切断段落）
4. ✅ 并发翻译
5. ✅ 缓存系统
6. ✅ 双语EPUB构建
#### 效果
- ✅ 代码量减少约 **40%**
- ✅ 逻辑清晰，易维护
- ✅ 专注核心功能
---
### 4. **分块策略优化**
#### 原有策略
- 按章节分组
- 在章节内按chunk_size切割
- 不允许跨章节
- 复杂的边界处理
#### 新策略
```python
# 全局分块，不考虑章节边界
total_paragraphs = [p1, p2, p3, ..., p_n]
↓
chunks = [
[p1, p2, p3], # chunk1: 2850字符
[p4, p5], # chunk2: 2950字符
[p6, p7, p8] # chunk3: 2700字符
]
```
#### 原则
- ✅ 纯粹按字符数分块
- ✅ **严格不切断段落**
- ✅ 允许跨章节（现代LLM完全支持）
- ✅ 简化边界处理
---
### 5. **配置精简**
#### 删除的配置参数
```json
{
"translation": {
"concurrent_requests": 16, // 冗余，未使用
"cache_enabled": true, // 冗余，由cache.enabled控制
"never_fallback_to_original": true, // 冗余，固定策略
"max_context_length": 4000, // 过时，不再需要
"sample_ratio": 0.05, // 已删除术语表生成
"preserve_formatting": false, // 未使用
"max_tokens": 8000 // 固定在代码中
},
"processing": {
"skip_sections": [...], // 删除，不再分类
"include_sections": [...], // 删除，不再分类
"clean_patterns": [...] // 删除，过度清理
}
}
```
#### 保留的核心配置
```json
{
"openrouter": {
"rate_limits": {
"concurrent_requests": 8 // 控制并发
}
},
"translation": {
"chunk_size": 5000, // 分块大小
"temperature": 0.2 // LLM参数
},
"processing": {
"min_paragraph_length": 30 // 段落过滤
}
}
```
---
## 📊 性能对比
### 翻译速度
| 场景 | 旧版（串行） | 新版（并发） | 提升 |
|------|-------------|-------------|------|
| 10个chunks | 10秒 | 1.3秒 | **7.7x** |
| 100个chunks | 100秒 | 13秒 | **7.7x** |
| 300页书籍 | 15分钟 | 2分钟 | **7.5x** |
### 代码质量
| 指标 | 旧版 | 新版 | 改善 |
|------|------|------|------|
| 代码行数 | ~1500 | ~900 | -40% |
| 核心文件 | 7个 | 6个 | -1个 |
| 配置参数 | 18个 | 8个 | -56% |
| 循环复杂度 | 高 | 低 | 显著降低 |
---
## 🧪 测试验证
### 新增测试脚本
1. **`test_global_id_system.py`**
- 测试全局编号系统
- 测试分块逻辑
- 测试翻译对应关系
2. **`test_concurrent.py`**
- 对比串行 vs 并发性能
- 验证RateLimiter工作
- 计算加速比
### 测试结果
```bash
$ python test_concurrent.py
📊 性能对比
串行耗时: 10.23 秒
并发耗时: 1.35 秒
加速比: 7.58x ✅
理论最大加速: 8x
```
---
## 🎯 技术要点
### 1. asyncio.gather并发
```python
# 创建所有任务
tasks = [translate_chunk(chunk) for chunk in chunks]
# 并发执行
results = await asyncio.gather(*tasks, return_exceptions=True)
# 优点：
# - 简洁高效
# - 自动并发
# - 异常隔离
```
### 2. Semaphore控制并发数
```python
class RateLimiter:
def __init__(self, concurrent_requests: int):
self.semaphore = asyncio.Semaphore(concurrent_requests)
async def acquire(self):
await self.semaphore.acquire() # 最多N个同时执行
```
### 3. 全局ID贯穿全流程
```python
# 提取
paragraph = {
'global_id': 'p_0001',
'text': '...'
}
# 翻译
translation_map = {
'p_0001': '翻译1',
'p_0002': '翻译2'
}
# 组装
for para in paragraphs:
translation = translation_map[para['global_id']]
insert_after(para, translation)
```
---
## 🔍 问题分析记录
### Token数量观察
**观察**：每个请求约1000+ tokens
**分析**：
```
chunk_size = 5000字符
计算：
- 5000字符 ÷ 5 = 1000单词
- 1000单词 × 1.3 = 1300 tokens（输入）
- + 系统提示 ≈ 200 tokens
- + 输出 ≈ 1500 tokens
= 总计约3000 tokens/请求
✅ 完全正常！
```
### 响应时间观察
**观察**：每个请求<1秒
**分析**：
- Gemini 2.5 Flash是超快模型
- 生成速度：100+ tokens/秒
- 1500 tokens输出约15秒
- 流式输出，首token<1秒
✅ 完全正常！
---
## 🚀 使用指南
### 快速开始
```bash
# 1. 测试API
python test_api.py
# 2. 测试并发
python test_concurrent.py
# 3. 测试全局ID
python test_global_id_system.py
# 4. 测试翻译
python main.py book.epub --test
# 5. 完整翻译
python main.py book.epub
```
### 性能调优
```json
// 追求速度
{
"concurrent_requests": 12,
"chunk_size": 8000
}
// 追求质量
{
"concurrent_requests": 4,
"chunk_size": 3000,
"temperature": 0.1
}
// 平衡模式（推荐）
{
"concurrent_requests": 8,
"chunk_size": 5000,
"temperature": 0.2
}
```
---
## 📋 文件清单
### 核心模块
- ✅ `src/epub_parser.py` - 简化的EPUB解析
- ✅ `src/text_processor.py` - 全局编号 + 智能分块
- ✅ `src/llm_client.py` - 编号翻译
- ✅ `src/translator.py` - **真并发翻译**
- ✅ `src/cache.py` - 基于ID的缓存
- ✅ `src/bilingual_builder.py` - 精确匹配组装
### 测试脚本
- ✅ `test_global_id_system.py` - 全局ID测试
- ✅ `test_concurrent.py` - 并发性能测试
### 配置文件
- ✅ `config/config.json` - 精简配置
- ✅ `README.md` - 完整文档
---
## ✅ 验证清单
- [x] 全局编号系统正常工作
- [x] 并发翻译速度提升7-8倍
- [x] 中英文精确对应，无错行
- [x] 缓存系统基于ID工作正常
- [x] 不切断段落，保持完整性
- [x] 配置精简，参数清晰
- [x] 代码简洁，易于维护
- [x] 测试脚本完整
- [x] 文档清晰详细
---
## 🎉 重构总结
### 成果
1. ✅ **根本性解决中英文错行问题**
2. ✅ **翻译速度提升7-8倍**
3. ✅ **代码精简40%**
4. ✅ **架构清晰，易维护**
### 关键技术
1. 全局唯一编号系统
2. asyncio.gather真并发
3. Semaphore并发控制
4. 基于ID的精确匹配
### 性能提升
- 串行 → 并发：**7.7x**
- 15分钟 → 2分钟
- 充分利用API能力
---
**重构完成日期**：2026-01-12
**版本**：v2.0.0
**状态**：✅ 生产就绪
archive/v0.09/analyze_manifest_errors.py
+110
View File
@@ -0,0 +1,110 @@
#!/usr/bin/env python3
"""
Manifest 错误分析工具
用于深入分析翻译结果中的占位符问题
"""
import sys
import re
import json
from pathlib import Path
sys.path.insert(0, '.')
from src.manifest_manager import ManifestManager
def analyze_item(item):
"""详细分析单个 Item 的占位符状态"""
issues = []
twp = item.text_with_placeholders or ""
trans = item.translation_with_placeholders or ""
pmap = item.placeholder_map or {}
ph_regex = r'φ(/?\d+)φ'
src_phs = set(re.findall(ph_regex, twp))
trans_phs = set(re.findall(ph_regex, trans))
map_ids = {k for k in pmap.keys() if not k.startswith('_')}
# 模拟 FormatRestorer 的宽松逻辑
unknowns = trans_phs - map_ids
real_unknowns = set()
for pid in unknowns:
# 如果是 /N，且 N 在 map 中，则认为是安全的冗余闭合
if pid.startswith('/') and pid[1:] in map_ids:
continue
real_unknowns.add(pid)
if real_unknowns:
issues.append(f"未知占位符(幻觉): {real_unknowns}")
missing = map_ids - trans_phs
if missing:
issues.append(f"丢失占位符: {missing}")
# Check Reordering
src_ph_list = re.findall(ph_regex, twp)
trans_ph_list = re.findall(ph_regex, trans)
common = [p for p in src_ph_list if p in trans_ph_list]
trans_common = [p for p in trans_ph_list if p in src_ph_list]
if common != trans_common:
issues.append(f"占位符乱序: 原文{common} -> 译文{trans_common}")
return issues
def main():
manifest_dir = Path("cache/manifests")
files = list(manifest_dir.glob("*_manifest.json"))
if not files:
print("未找到 manifest 文件")
return
target = next((f for f in files if "Karen Hao" in f.name), files[0])
print(f"Loading: {target}")
manifest = ManifestManager(str(target))
if not manifest.load():
print("加载失败")
return
items = manifest.get_items()
error_count = 0
total_analyzed = 0
print("\n" + "="*50)
print(" 异常项目分析报告")
print("="*50 + "\n")
for item in items:
# 只分析非成功状态或有 warning 的
if item.status == "completed":
continue
# 即使是 format_error, failed, translated (with issues)
issues = analyze_item(item)
has_error = (item.status in ["failed", "format_error"]) or bool(issues)
if has_error:
total_analyzed += 1
print(f"ID: {item.global_id} (Status: {item.status})")
if item.error_msg:
print(f" Error Msg: {item.error_msg}")
# 只有当有具体 issue 时才打印详细文本，避免刷屏
if issues or item.status == "format_error":
print(f" 原文: {item.text_with_placeholders[:100]}...")
print(f" 译文: {item.translation_with_placeholders[:100]}..." if item.translation_with_placeholders else " 译文: (None)")
print(f" Map: {list(item.placeholder_map.keys())}")
for issue in issues:
print(f" -> {issue}")
print("-" * 50)
error_count += 1
if error_count > 50:
print("... (Errors truncated) ...")
break
print(f"\n分析完成。共发现 {total_analyzed} 个异常项目。")
if __name__ == "__main__":
main()
archive/v0.09/config/config.example.json
+54
View File
@@ -0,0 +1,54 @@
{
"translation": {
"chunk_size": 5000,
"temperature": 0.3,
"glossary": {
"enabled": true,
"auto_generate": true,
"sample_size": 3000
}
},
"output": {
"output_dir": "output",
"filename_suffix": "_bilingual"
},
"logging": {
"level": "INFO",
"file": "logs/translator.log",
"rotation": "10 MB",
"retention": "7 days"
},
"providers": {
"openrouter": {
"base_url": "https://openrouter.ai/api/v1",
"api_key": "YOUR_OPENROUTER_API_KEY",
"models": {
"fast": "google/gemini-2.0-flash-001",
"smart": "google/gemini-2.0-flash-thinking-exp:free"
},
"extra_headers": {
"HTTP-Referer": "https://github.com/epub-translator",
"X-Title": "EPUB Translator"
},
"rate_limits": {
"requests_per_minute": 60,
"concurrent_requests": 32
}
},
"v3": {
"base_url": "https://api.gpt.ge/v1",
"api_key": "YOUR_V3_API_KEY",
"models": {
"fast": "gpt-4o-mini",
"smart": "gpt-4o"
},
"extra_headers": {
"x-foo": "true"
},
"rate_limits": {
"requests_per_minute": 500,
"concurrent_requests": 50
}
}
}
}
archive/v0.09/config/config.json
+69
View File
@@ -0,0 +1,69 @@
{
"translation": {
"chunk_size": 5000,
"temperature": 0.3,
"glossary": {
"enabled": true,
"auto_generate": true,
"sample_size": 3000
}
},
"output": {
"output_dir": "output",
"filename_suffix": "_bilingual"
},
"logging": {
"level": "INFO",
"file": "logs/translator.log",
"rotation": "10 MB",
"retention": "7 days"
},
"providers": {
"openrouter": {
"base_url": "https://openrouter.ai/api/v1",
"api_key": "YOUR_OPENROUTER_API_KEY",
"models": {
"fast": "google/gemini-2.0-flash-001",
"smart": "google/gemini-2.0-flash-thinking-exp:free"
},
"extra_headers": {
"HTTP-Referer": "https://github.com/epub-translator",
"X-Title": "EPUB Translator"
},
"rate_limits": {
"requests_per_minute": 60,
"concurrent_requests": 32
}
},
"v3": {
"base_url": "https://api.gpt.ge/v1",
"api_key": "YOUR_V3_API_KEY",
"models": {
"fast": "gemini-3-flash-preview",
"smart": "gemini-3-pro-preview"
},
"extra_headers": {
"x-foo": "true"
},
"rate_limits": {
"requests_per_minute": 500,
"concurrent_requests": 50
}
},
"openai": {
"base_url": "http://127.0.0.1:8045/v1",
"api_key": "YOUR_ANTIGRAVITY_API_KEY",
"models": {
"fast": "gemini-3-flash",
"smart": "gemini-3-pro-high"
},
"extra_headers": {
"x-foo": "true"
},
"rate_limits": {
"requests_per_minute": 500,
"concurrent_requests": 50
}
}
}
}
archive/v0.09/config/config_副本.json
+39
View File
@@ -0,0 +1,39 @@
{
"llm": {
"provider": "openrouter",
"base_url": "https://openrouter.ai/api/v1",
"api_key": "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",
"models": {
"fast": "google/gemini-3-flash-preview",
"smart": "google/gemini-3-pro-preview"
},
"arena_models": [
"google/gemini-3-flash-preview",
"openai/gpt-5-mini",
"anthropic/claude-haiku-4.5"
],
"judge_model": "openai/gpt-5.1",
"rate_limits": {
"requests_per_minute": 60,
"concurrent_requests": 32
}
},
"translation": {
"chunk_size": 5000,
"temperature": 0.3,
"strategy": "arena_winner"
},
"processing": {
"min_paragraph_length": 5
},
"output": {
"output_dir": "output",
"filename_suffix": "_bilingual"
},
"logging": {
"level": "INFO",
"file": "logs/translator.log",
"rotation": "10 MB",
"retention": "7 days"
}
}
archive/v0.09/config/prompts.json
+10
View File
@@ -0,0 +1,10 @@
{
"translation": {
"system": "你是一位精通中英文的专业翻译家。你的任务是翻译书籍内容。\n\n要求：\n1. 准确传达原文含义，语言流畅自然，符合中文阅读习惯。\n2. 严格保持【p_xxxxx】编号格式，不要遗漏，不要修改编号。\n3. **关键格式指令（CRITICAL）**：\n - 原文中包含特殊格式标记：【φ数字φ】（开始/单体）和【φ/数字φ】（结束）。\n - 示例：\"φ1φTable Talkφ/1φ\" 应翻译为 \"φ1φ桌谈φ/1φ\"。\n - 示例：\"φ2φ\" (单体) 表示公式或符号，必须保留在译文对应位置。\n - 规则：严禁删除任何标记！严禁修改数字编号！保持标记与文本的相对位置不变。\n4. **尾注锚点**：形如 φnφ 且中间没有文本的占位符（如 \"...他问道。φ3φ接着...\"）是尾注返回链接，必须保留在译文的对应位置，确保读者可以从尾注跳回正文。\n5. 不要添加任何解释、注释或无关内容，只返回【编号】+【译文】。\n\n{{glossary_instruction}}",
"user_template": "请翻译以下段落（务必原样保留所有 φnφ 格式标记）：\n\n{{content}}"
},
"glossary_extraction": {
"system": "你是一位资深的文学编辑和领域专家。你的任务是分析书籍样本，提取关键术语并制定统一的译名表。",
"user_template": "请阅读以下书籍片段（包含前言和正文采样）。\n\n任务：\n1. 识别文中出现的人名（如 'Masa', 'Steve Jobs'）、地名、机构名。\n2. 识别特定的行业术语或关键概念。\n3. 为上述词汇提供标准的中文译名。如果像 'Masa' 这样的昵称有对应的全名（如孙正义），请务必使用全名。\n\n请以 JSON 格式输出，格式如下：\n{\n \"Masa\": \"孙正义\",\n \"Apple\": \"苹果公司\",\n ...\n}\n\n书籍片段：\n\n{{content}}"
}
}
archive/v0.09/debug.epub
BIN
View File
Binary file not shown.
archive/v0.09/debug_condition.py
+16
View File
@@ -0,0 +1,16 @@
# 测试 Python 的条件判断
item_status = "translated"
translation_with_placeholders = "" # 空字符串
# 这是 translator.py 中的条件
if item_status != "translated" or not translation_with_placeholders:
print("SKIP: 条件成立，跳过此 item")
else:
print("PROCESS: 条件不成立，处理此 item")
# 现在测试有内容的情况
translation_with_placeholders = "你好"
if item_status != "translated" or not translation_with_placeholders:
print("SKIP: 条件成立，跳过此 item")
else:
print("PROCESS: 条件不成立，处理此 item")
archive/v0.09/debug_flow.py
+62
View File
@@ -0,0 +1,62 @@
"""模拟翻译流程，定位数据丢失问题"""
import asyncio
from src.manifest_manager import ManifestManager
async def simulate_worker(manifest, chunks):
"""模拟 worker 行为"""
for chunk in chunks:
for item in chunk:
# 模拟 LLM 返回
fake_translation = f"翻译_{item.global_id}"
# 模拟 worker 的 update_item 调用
manifest.update_item(
item.global_id,
fake_translation,
translation_with_placeholders=fake_translation,
status="translated"
)
print(f"Worker 完成，内存中 translated 数量: {len(manifest.get_items(status='translated'))}")
async def simulate_restoration(manifest):
"""模拟 process_format_restoration"""
items = manifest.get_items() # 不带参数，获取所有
print(f"Restoration 获取到 {len(items)} 个 items")
processed = 0
skipped = 0
for item in items:
# 这是关键的过滤条件
if item.status != "translated" or not item.translation_with_placeholders:
skipped += 1
continue
processed += 1
print(f"Restoration: 处理 {processed} 个，跳过 {skipped} 个")
async def main():
# 初始化
manifest = ManifestManager("test_flow.json")
manifest.init_manifest("test", {})
# 添加测试 items
for i in range(5):
manifest.add_item(f"test{i}.html", f"<p>Text {i}</p>", f"Text {i}", "p")
# 模拟 create_chunks_from_manifest
pending = manifest.get_items(status="pending")
chunks = [pending] # 一个 chunk 包含所有
print(f"Chunks 创建，pending 数量: {len(pending)}")
print(f"chunks[0][0] is manifest._items_by_id['p_00001']: {chunks[0][0] is manifest._items_by_id['p_00001']}")
# 模拟 worker
await simulate_worker(manifest, chunks)
# 模拟 restoration
await simulate_restoration(manifest)
# 清理
import os
if os.path.exists("test_flow.json"):
os.remove("test_flow.json")
asyncio.run(main())
archive/v0.09/debug_manifest.py
+31
View File
@@ -0,0 +1,31 @@
from src.manifest_manager import ManifestManager, ManifestItem
# 模拟 worker 更新流程
manifest = ManifestManager("test_manifest.json")
manifest.init_manifest("test", {})
# 添加一个 item
item = manifest.add_item("test.html", "<p>Hello</p>", "Hello", "p")
print(f"After add: item.status = {item.status}, item.translation = {item.translation}")
print(f"ID in manifest: {item.global_id}")
# 模拟 get_items 获取的是同一个对象吗？
pending = manifest.get_items(status="pending")
print(f"pending[0] is item: {pending[0] is item}")
# 模拟 worker 更新
manifest.update_item(item.global_id, "你好", translation_with_placeholders="你好", status="translated")
# 检查更新是否生效
print(f"After update: item.status = {item.status}, item.translation = {item.translation}")
print(f"After update: item.translation_with_placeholders = '{item.translation_with_placeholders}'")
# 验证 get_items 能获取到更新后的状态
translated = manifest.get_items(status="translated")
print(f"Translated items count: {len(translated)}")
if translated:
print(f"translated[0].translation_with_placeholders = '{translated[0].translation_with_placeholders}'")
import os
if os.path.exists("test_manifest.json"):
os.remove("test_manifest.json")
archive/v0.09/debug_prompt_construction.py
+120
View File
@@ -0,0 +1,120 @@
#!/usr/bin/env python3
"""
调试脚本：使用真实数据验证 Prompt 构建
完整展示发送给 LLM 的文本
"""
import sys
import json
from pathlib import Path
from loguru import logger
# 添加 src 到路径
sys.path.insert(0, '.')
from src.manifest_manager import ManifestManager
from src.text_processor import TextProcessor
from src.llm_client import LLMClient
from src.utils import load_config
# 配置日志到文件，避免控制台刷屏
logger.remove()
logger.add("debug_prompt.log", level="DEBUG")
logger.add(sys.stdout, level="INFO")
def debug_prompt():
print("=== 1. 加载配置 ===")
try:
config = load_config()
# 确保 LLM 配置存在 (Mock if needed for init)
if 'llm' not in config:
if 'v3' in config['providers']:
config['llm'] = config['providers']['v3']
else:
config['llm'] = {"api_key": "dummy", "models": {"fast": "dummy"}}
except Exception as e:
print(f"Config load failed: {e}")
return
print("=== 2. 加载真实 Manifest ===")
# 查找 cache/manifests 下的 json 文件
manifest_dir = Path("cache/manifests")
if not manifest_dir.exists():
print("Error: cache/manifests directory not found")
return
manifest_files = list(manifest_dir.glob("*_manifest.json"))
if not manifest_files:
print("Error: No manifest file found in cache/manifests")
return
manifest_path = manifest_files[0]
print(f"Using manifest: {manifest_path}")
manifest = ManifestManager(str(manifest_path))
if not manifest.load():
print("Failed to load manifest")
return
print(f"Loaded {len(manifest.get_items())} items")
# 获取 Pending items (模拟真实流程)
pending = manifest.get_items(status="pending")
if not pending:
print("No pending items found. Using ALL items for debug.")
items_to_process = manifest.get_items()
else:
items_to_process = pending
# 找到几个包含占位符的 item 用于验证
target_items = []
for item in items_to_process:
if item.placeholder_map and len(item.placeholder_map) > 0:
target_items.append(item)
if len(target_items) >= 5: # 取前5个
break
if not target_items:
print("No items with placeholders found!")
return
print(f"Selected {len(target_items)} items with placeholders for verification")
for item in target_items:
print(f" - {item.global_id}: twp length={len(item.text_with_placeholders or '')}")
print("\n=== 3. 生成 Prompt (Mode: Chinese) ===")
client = LLMClient(config)
# 只为这几个 item 生成 prompt
prompt = client._build_prompt(target_items, mode="chinese")
print("\n" + "="*40)
print("FULL PROMPT CONTENT (Snippet):")
print("="*40)
print(prompt)
print("="*40 + "\n")
print("\n=== 4. 关键验证 ===")
placeholders_found = prompt.count('φ')
print(f"Total 'φ' symbols in prompt: {placeholders_found}")
for item in target_items:
if item.text_with_placeholders and 'φ' in item.text_with_placeholders:
# 检查这个 item 的 ID 是否在 prompt 中
in_prompt = item.global_id in prompt
# 检查这个 item 的占位符是否在 prompt 中
# 注意：如果占位符是 φ1φ，我们检查 'φ1φ' 是否在 prompt 中
# 这里简单做，假设 text_with_placeholders 应该完整出现在 prompt 中 (忽略空白差异)
import re
normalized_twp = re.sub(r'\s+', '', item.text_with_placeholders)
normalized_prompt = re.sub(r'\s+', '', prompt)
content_in_prompt = normalized_twp in normalized_prompt
print(f"Item {item.global_id}:")
print(f" In prompt ID: {in_prompt}")
print(f" Original twp: {repr(item.text_with_placeholders)}")
print(f" Content match (ignoring whitespace): {content_in_prompt}")
if __name__ == "__main__":
debug_prompt()
archive/v0.09/debug_restore.py
+32
View File
@@ -0,0 +1,32 @@
import re
from src.format_restorer import FormatRestorer
restorer = FormatRestorer()
# 模拟一个真实场景
text_with_ph = '"But what is the goal?" φ1φAmodeiφ/1φ...'
placeholder_map = {
'1': '<em>',
'/1': '</em>'
}
print("Input text:", text_with_ph)
print("Placeholder map:", placeholder_map)
# 手动执行 restorer 的验证逻辑
inner_map = {k: v for k, v in placeholder_map.items() if not k.startswith("_")}
print("Inner map keys:", set(inner_map.keys()))
found_ids = set(re.findall(r'φ(/?\d+)φ', text_with_ph))
print("Found IDs in text:", found_ids)
expected_ids = set(inner_map.keys())
print("Expected IDs:", expected_ids)
missing_ids = expected_ids - found_ids
print("Missing IDs:", missing_ids)
# 调用 restore
html, success = restorer.restore(text_with_ph, placeholder_map)
print(f"\nResult: success={success}")
print(f"HTML: {html}")
archive/v0.09/install.sh
Executable
+50
View File
@@ -0,0 +1,50 @@
#!/bin/bash
# EPUB 双语翻译程序安装脚本
echo "正在安装 EPUB 双语翻译程序..."
# 检查 Python 版本
python_version=$(python3 --version 2>&1 | awk '{print $2}' | cut -d. -f1,2)
required_version="3.9"
if [ "$(printf '%s\n' "$required_version" "$python_version" | sort -V | head -n1)" != "$required_version" ]; then
echo "错误: 需要 Python 3.9 或更高版本，当前版本: $python_version"
exit 1
fi
# 检查 uv 是否安装
if ! command -v uv &> /dev/null; then
echo "正在安装 uv..."
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.cargo/env
fi
# 创建虚拟环境
echo "正在创建虚拟环境..."
uv venv
# 激活虚拟环境
source .venv/bin/activate
# 安装依赖
echo "正在安装依赖..."
uv pip install -r requirements.txt
# 创建必要的目录
mkdir -p output logs
# 复制配置文件示例
if [ ! -f ".env" ]; then
cp .env.example .env
echo "已创建 .env 文件，请编辑并添加你的 OpenRouter API Key"
fi
echo "安装完成！"
echo ""
echo "下一步："
echo "1. 编辑 .env 文件，添加你的 OpenRouter API Key"
echo "2. 激活虚拟环境: source .venv/bin/activate"
echo "3. 运行测试: python main.py your_book.epub --test"
echo ""
echo "使用帮助: python main.py --help"
archive/v0.09/main.py
+108
View File
@@ -0,0 +1,108 @@
#!/usr/bin/env python3
import asyncio
import sys
import argparse
from pathlib import Path
from loguru import logger
from src.translator import EPUBTranslator
from src.epub_parser import EPUBParser
from src.toc_parser import TOCParser
from src.utils import load_config, setup_logging
def parse_args():
parser = argparse.ArgumentParser(description="EPUB 双语翻译工具")
parser.add_argument("epub_path", help="输入 EPUB 文件路径")
parser.add_argument("--provider", "-p", default="openrouter", help="LLM 供应商 (config.json 中 providers 的 key)")
parser.add_argument("--mode", "-m", default="bilingual", choices=["bilingual", "chinese"],
help="输出模式: bilingual (双语对照) 或 chinese (纯中文，保留格式)")
parser.add_argument("--test", action="store_true", help="测试模式（仅翻译前几段）")
parser.add_argument("--output", "-o", help="输出目录")
parser.add_argument("--no-cache", action="store_true", help="禁用缓存（强制重新翻译）")
parser.add_argument("--clear-cache", action="store_true", help="清理所有缓存文件")
# TOC 章节选择
parser.add_argument("--show-toc", action="store_true", help="显示书籍目录结构")
parser.add_argument("--from", dest="from_chapter", help="起始章节标题")
parser.add_argument("--to", dest="to_chapter", help="结束章节标题")
return parser.parse_args()
def flatten_provider_config(config: dict, provider_name: str) -> dict:
"""
将选定的 provider 配置扁平化到 config['llm'] 中，
以便下游模块统一调用。
"""
providers = config.get('providers', {})
if provider_name not in providers:
available = list(providers.keys())
logger.error(f"未找到供应商 '{provider_name}'。可用供应商: {available}")
sys.exit(1)
selected_config = providers[provider_name]
logger.info(f"使用 LLM 供应商: {provider_name} ({selected_config.get('base_url')})")
# 注入到 config['llm']
config['llm'] = selected_config
return config
async def run_translation(args):
try:
# 1. 加载配置
config = load_config()
# 2. 处理 Provider 选择
config = flatten_provider_config(config, args.provider)
# 3. 设置日志
setup_logging(config)
logger.info("程序启动")
# 4. 初始化翻译器
translator = EPUBTranslator(config, use_cache=not args.no_cache)
# 5. 执行翻译 (传递章节范围参数)
await translator.translate_epub(
args.epub_path,
test_mode=args.test,
output_dir=args.output,
mode=args.mode,
from_chapter=args.from_chapter,
to_chapter=args.to_chapter
)
except Exception as e:
import traceback
traceback.print_exc()
logger.error(f"翻译失败: {e}")
sys.exit(1)
def show_toc(epub_path: str):
"""显示 EPUB 的目录结构"""
parser = EPUBParser(epub_path)
toc_parser = TOCParser(parser.book)
print(f"\n📖 {parser.metadata.get('title', 'Unknown')} - {parser.metadata.get('author', 'Unknown')}")
print(toc_parser.format_toc_table())
print("提示: 使用 --from \"章节名\" --to \"章节名\" 指定翻译范围")
print()
def main():
args = parse_args()
if args.clear_cache:
import shutil
cache_dir = Path("cache")
if cache_dir.exists():
shutil.rmtree(cache_dir)
print("缓存已清理")
# sys.exit(0) # 移除退出，允许继续执行
if args.show_toc:
show_toc(args.epub_path)
sys.exit(0)
asyncio.run(run_translation(args))
if __name__ == "__main__":
main()
archive/v0.09/pyproject.toml
+34
View File
@@ -0,0 +1,34 @@
[project]
name = "epub-translator"
version = "0.07"
description = "EPUB双语翻译程序"
authors = [
{name = "Kaitan", email = "your-email@example.com"}
]
dependencies = [
"ebooklib>=0.19",
"beautifulsoup4>=4.12.0",
"lxml>=4.9.0",
"openai>=1.0.0",
"aiohttp>=3.9.0",
"pydantic>=2.0.0",
"loguru>=0.7.0",
"rich>=13.0.0",
"asyncio-throttle>=1.0.2",
"tenacity>=8.0.0",
"python-dotenv>=1.0.0",
]
requires-python = ">=3.9"
[project.scripts]
epub-translator = "main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.uv]
dev-dependencies = [
"pytest>=7.0.0",
"pytest-asyncio>=0.21.0",
]
archive/v0.09/requirements.txt
+12
View File
@@ -0,0 +1,12 @@
ebooklib>=0.19
beautifulsoup4>=4.12.0
lxml>=4.9.0
openai>=1.0.0
aiohttp>=3.9.0
pydantic>=2.0.0
loguru>=0.7.0
rich>=13.0.0
asyncio-throttle>=1.0.2
tenacity>=8.0.0
python-dotenv>=1.0.0
socksio>=1.0.0
archive/v0.09/run.sh
Executable
+33
View File
@@ -0,0 +1,33 @@
#!/bin/bash
# 快速启动脚本
# 检查虚拟环境
if [ ! -d ".venv" ]; then
echo "虚拟环境不存在，正在创建..."
./install.sh
fi
# 激活虚拟环境
source .venv/bin/activate
# 检查依赖是否安装
if ! python -c "import ebooklib" 2>/dev/null; then
echo "依赖未安装，正在安装..."
uv pip install -r requirements.txt
fi
# 检查配置
if [ ! -f ".env" ] || ! grep -q "OPENROUTER_API_KEY=" .env || grep -q "your_openrouter_api_key_here" .env; then
echo "请先在 .env 文件中设置你的 OpenRouter API Key"
echo "示例: OPENROUTER_API_KEY=sk-or-v1-xxxxx"
exit 1
fi
# 加载环境变量
set -a
source .env
set +a
# 运行程序
python main.py "$@"
archive/v0.09/scripts/debug/debug.py
+216
View File
@@ -0,0 +1,216 @@
#!/usr/bin/env python3
"""
调试和测试脚本
用于诊断 EPUB 解析问题
"""
import sys
from pathlib import Path
# 添加 src 目录到路径
sys.path.insert(0, str(Path(__file__).parent / "src"))
from src.epub_parser import EPUBParser
from src.text_processor import TextProcessor
from src.utils import load_config
from rich.console import Console
from rich.table import Table
from rich.panel import Panel
from bs4 import BeautifulSoup
import ebooklib
def debug_epub_structure(epub_path: str):
"""调试 EPUB 结构"""
console = Console()
try:
# 加载配置
config = load_config('config/config.json')
# 初始化解析器
parser = EPUBParser(epub_path)
text_processor = TextProcessor(config)
console.print(f"[bold blue]调试 EPUB 文件: {epub_path}[/bold blue]\n")
# 显示基本信息
book_info = parser.get_book_info()
info_table = Table(title="书籍信息")
info_table.add_column("属性", style="cyan")
info_table.add_column("值", style="white")
for key, value in book_info.items():
info_table.add_row(key, str(value))
console.print(info_table)
# 显示目录结构
console.print("\n[bold green]目录结构分析:[/bold green]")
toc_table = Table(title="目录结构")
toc_table.add_column("类型", style="cyan")
toc_table.add_column("标题", style="white")
toc_table.add_column("状态", style="green")
# 检查各种内容类型
content_types = ['preface', 'introduction', 'prologue', 'abstract', 'epilogue', 'acknowledgments']
for content_type in content_types:
item = parser.toc_structure.get(content_type)
if item:
toc_table.add_row(content_type, item['title'], "✓ 找到")
else:
toc_table.add_row(content_type, "-", "✗ 未找到")
# 章节信息
chapters = parser.toc_structure['chapters']
toc_table.add_row("chapters", f"{len(chapters)} 个章节", "✓ 找到" if chapters else "✗ 未找到")
console.print(toc_table)
# 显示章节列表
if chapters:
console.print("\n[bold yellow]章节列表:[/bold yellow]")
chapter_table = Table()
chapter_table.add_column("序号", style="cyan")
chapter_table.add_column("标题", style="white")
chapter_table.add_column("内容长度", style="green")
for i, chapter in enumerate(chapters[:10], 1): # 只显示前10个
content = parser._extract_item_content(chapter)
content_length = len(content) if content else 0
chapter_table.add_row(str(i), chapter['title'], f"{content_length:,} 字符")
if len(chapters) > 10:
chapter_table.add_row("...", f"还有 {len(chapters) - 10} 个章节", "...")
console.print(chapter_table)
# 测试段落提取
console.print("\n[bold magenta]段落提取测试:[/bold magenta]")
# 选择第一个有内容的项目进行测试
test_content = None
test_title = ""
# 优先测试序言类内容
for content_type in ['prologue', 'preface', 'introduction', 'abstract']:
item = parser.toc_structure.get(content_type)
if item:
test_content = parser._extract_item_content(item)
test_title = f"{content_type}: {item['title']}"
break
# 如果没有序言，测试第一个章节
if not test_content and chapters:
test_content = parser._extract_item_content(chapters[0])
test_title = f"章节: {chapters[0]['title']}"
if test_content:
paragraphs = text_processor.extract_paragraphs(test_content)
console.print(f"测试内容: {test_title}")
console.print(f"原始内容长度: {len(test_content):,} 字符")
console.print(f"提取段落数: {len(paragraphs)}")
if paragraphs:
# 显示前几个段落
para_table = Table(title="段落示例")
para_table.add_column("序号", style="cyan")
para_table.add_column("类型", style="yellow")
para_table.add_column("内容预览", style="white")
para_table.add_column("长度", style="green")
for i, para in enumerate(paragraphs[:5], 1):
preview = para['text'][:100] + "..." if len(para['text']) > 100 else para['text']
para_table.add_row(
str(i),
para.get('type', 'unknown'),
preview,
str(len(para['text']))
)
console.print(para_table)
# 测试翻译块创建
chunks = text_processor.create_chunks(paragraphs, 3)
console.print(f"\n[cyan]翻译块信息:[/cyan] 创建了 {len(chunks)} 个翻译块")
if chunks:
chunk_table = Table(title="翻译块示例")
chunk_table.add_column("块号", style="cyan")
chunk_table.add_column("段落数", style="yellow")
chunk_table.add_column("总字符数", style="green")
for i, chunk in enumerate(chunks[:3], 1): # 显示前3个块
total_chars = sum(len(p['text']) for p in chunk)
chunk_table.add_row(str(i), str(len(chunk)), f"{total_chars:,}")
console.print(chunk_table)
else:
console.print("[red]未能提取到段落！[/red]")
# 显示原始内容的一部分用于调试
soup = BeautifulSoup(test_content, 'html.parser')
text_content = soup.get_text()[:500]
console.print(Panel(
text_content,
title="原始文本内容（前500字符）",
border_style="red"
))
else:
console.print("[red]未找到可测试的内容！[/red]")
# 显示所有 HTML 文件
console.print("\n[bold cyan]所有 HTML 文件:[/bold cyan]")
try:
html_items = list(parser.book.get_items_of_type(ebooklib.ITEM_DOCUMENT))
file_table = Table()
file_table.add_column("文件名", style="cyan")
file_table.add_column("大小", style="green")
file_table.add_column("内容预览", style="white")
for item in html_items[:10]: # 只显示前10个
try:
content = item.get_content().decode('utf-8', errors='ignore')
soup = BeautifulSoup(content, 'html.parser')
text_preview = soup.get_text()[:100].replace('\n', ' ')
file_table.add_row(
item.get_name(),
f"{len(content):,} 字符",
text_preview + "..." if len(text_preview) == 100 else text_preview
)
except Exception as e:
file_table.add_row(item.get_name(), "错误", f"读取失败: {e}")
if len(html_items) > 10:
file_table.add_row("...", f"还有 {len(html_items) - 10} 个文件", "...")
console.print(file_table)
except Exception as e:
console.print(f"[yellow]无法列出 HTML 文件: {e}[/yellow]")
# 总结
console.print(f"\n[bold green]✓ 调试完成[/bold green]")
console.print(f"[green]结论: EPUB 文件结构正常，可以进行翻译[/green]")
except Exception as e:
console.print(f"[red]调试失败: {e}[/red]")
import traceback
console.print(traceback.format_exc())
if __name__ == "__main__":
if len(sys.argv) != 2:
print("使用方法: python debug.py <epub_file>")
sys.exit(1)
epub_file = sys.argv[1]
debug_epub_structure(epub_file)
archive/v0.09/scripts/debug/debug_epub_comparison.py
+132
View File
@@ -0,0 +1,132 @@
import sys
import os
from pathlib import Path
from ebooklib import epub
import ebooklib
from loguru import logger
# Add project root to path
sys.path.insert(0, str(Path(__file__).parent))
from src.utils import load_config
def compare_epubs(original_path, new_path):
print(f"🔍 Comparing EPUBs:\n Original: {original_path}\n New: {new_path}")
print("=" * 60)
if not os.path.exists(new_path):
print(f"❌ New EPUB not found: {new_path}")
return
try:
orig_book = epub.read_epub(original_path)
new_book = epub.read_epub(new_path)
except Exception as e:
print(f"❌ Error reading EPUBs: {e}")
return
# 1. Metadata Comparison
print("\n[1] Metadata Comparison")
print("-" * 60)
namespaces = ['DC', 'OPF']
for ns in namespaces:
orig_meta = orig_book.metadata.get(ns, {})
new_meta = new_book.metadata.get(ns, {})
all_keys = set(orig_meta.keys()) | set(new_meta.keys())
for key in sorted(all_keys):
orig_vals = [v[0] for v in orig_meta.get(key, [])]
new_vals = [v[0] for v in new_meta.get(key, [])]
if orig_vals != new_vals:
print(f" ⚠️ {ns}:{key} Changed:")
print(f" Orig: {orig_vals}")
print(f" New: {new_vals}")
else:
# print(f" ✅ {ns}:{key} match")
pass
# Special check for Cover
print("\n[2] Cover Image Check")
print("-" * 60)
# Check via Metadata
orig_cover_meta = orig_book.get_metadata('OPF', 'cover')
new_cover_meta = new_book.get_metadata('OPF', 'cover')
print(f" Original Cover Meta (OPF): {orig_cover_meta}")
print(f" New Cover Meta (OPF): {new_cover_meta}")
# Check via Manifest Items
orig_cover_items = [i for i in orig_book.get_items() if 'cover' in i.get_name().lower() and i.media_type.startswith('image/')]
new_cover_items = [i for i in new_book.get_items() if 'cover' in i.get_name().lower() and i.media_type.startswith('image/')]
print(f" Original Cover Image Items: {[i.get_name() for i in orig_cover_items]}")
print(f" New Cover Image Items: {[i.get_name() for i in new_cover_items]}")
# 3. Spine Comparison (Reading Order)
print("\n[3] Spine (Reading Order) Comparison")
print("-" * 60)
orig_spine_ids = [item[0] for item in orig_book.spine]
new_spine_ids = [item[0] for item in new_book.spine]
print(f" Original Spine Length: {len(orig_spine_ids)}")
print(f" New Spine Length: {len(new_spine_ids)}")
# Map IDs to Filenames for better readability
def get_filename(book, item_id):
item = book.get_item_with_id(item_id)
return item.get_name() if item else "UNKNOWN"
# Compare first few and last few
limit = 5
print(f" First {limit} items:")
for i in range(min(len(orig_spine_ids), len(new_spine_ids), limit)):
f_orig = get_filename(orig_book, orig_spine_ids[i])
f_new = get_filename(new_book, new_spine_ids[i])
status = "✅" if f_orig == f_new else "❌"
print(f" {i+1}. {status} Orig: {f_orig} | New: {f_new}")
# Check for missing items in spine
orig_filenames = set(get_filename(orig_book, i) for i in orig_spine_ids)
new_filenames = set(get_filename(new_book, i) for i in new_spine_ids)
missing_in_new = orig_filenames - new_filenames
if missing_in_new:
print(f"\n ⚠️ Missing from New Spine ({len(missing_in_new)}):")
for f in list(missing_in_new)[:10]:
print(f" - {f}")
# 4. Manifest Comparison (All Resources)
print("\n[4] Manifest (All Resources) Comparison")
print("-" * 60)
orig_manifest = {i.get_name() for i in orig_book.get_items()}
new_manifest = {i.get_name() for i in new_book.get_items()}
missing_resources = orig_manifest - new_manifest
# Filter out NCX/Nav as they might be regenerated with different names
missing_resources = {f for f in missing_resources if not f.endswith('.ncx') and 'nav' not in f.lower()}
if missing_resources:
print(f" ⚠️ Resources Missing in New Book ({len(missing_resources)}):")
for f in sorted(list(missing_resources)):
print(f" - {f}")
else:
print(" ✅ All resources preserved.")
if __name__ == "__main__":
orig_path = "input/To Explain the World The Discovery of Modern Science (H) (Steven Weinberg [Weinberg, Steven]) (Z-Library).epub"
# Escaped path from user prompt: "input/To Explain the World The Discovery of Modern Science (H) (Steven Weinberg [Weinberg, Steven]) (Z-Library).epub"
# We generated this in the previous batch test
new_path = "test_output/To Explain the World The Discovery of Modern Science (H)_bilingual.epub"
if len(sys.argv) > 2:
orig_path = sys.argv[1]
new_path = sys.argv[2]
compare_epubs(orig_path, new_path)
archive/v0.09/scripts/debug/debug_structure.py
+74
View File
@@ -0,0 +1,74 @@
import sys
from pathlib import Path
from bs4 import BeautifulSoup
from loguru import logger
# Add project root to path
sys.path.insert(0, str(Path(__file__).parent))
from src.epub_parser import EPUBParser
from src.text_processor import TextProcessor
from src.utils import load_config
def debug_nested_structure(epub_path: str):
config = load_config()
parser = EPUBParser(epub_path)
content_items = parser.extract_all_content_items()
# Check just one chapter (e.g. Chapter 1)
target_item = None
for item in content_items:
if 'c01' in item['file_name']: # Chapter 1 usually
target_item = item
break
if not target_item:
target_item = content_items[2] # Fallback to 3rd item
print(f"Checking file: {target_item['file_name']}")
soup = BeautifulSoup(target_item['content'], 'html.parser')
# Simulate TextProcessor extraction logic
text_elements = soup.find_all(['p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
'blockquote', 'li', 'td'])
min_len = config['processing'].get('min_paragraph_length', 30)
extracted = []
for i, element in enumerate(text_elements):
# Clean text logic
clean_text = TextProcessor.clean_element_text(element)
is_valid = True
if len(clean_text) < min_len:
is_valid = False
if TextProcessor.is_navigation_element(element):
is_valid = False
if is_valid:
extracted.append((element, clean_text))
# Check for nesting
# If this element contains other valid extracted elements
for prev_el, prev_text in extracted[:-1]:
# Check if current element is inside previous element
if element in prev_el.descendants:
print(f"\n⚠️ NESTING DETECTED!")
print(f" Parent: <{prev_el.name}> {prev_text[:50]}...")
print(f" Child: <{element.name}> {clean_text[:50]}...")
# Check if previous element is inside current element
if prev_el in element.descendants:
print(f"\n⚠️ NESTING DETECTED!")
print(f" Parent: <{element.name}> {clean_text[:50]}...")
print(f" Child: <{prev_el.name}> {prev_text[:50]}...")
if __name__ == "__main__":
if len(sys.argv) < 2:
print("Usage: python debug_structure.py <epub_file>")
sys.exit(1)
epub_file = sys.argv[1]
debug_nested_structure(epub_file)
archive/v0.09/scripts/debug/quick_fix.py
+102
View File
@@ -0,0 +1,102 @@
#!/usr/bin/env python3
"""
快速修复和测试脚本
"""
import sys
import os
from pathlib import Path
# 添加 src 目录到路径
sys.path.insert(0, str(Path(__file__).parent / "src"))
from rich.console import Console
def check_api_key():
"""检查 API Key 设置"""
console = Console()
# 检查环境变量
env_key = os.environ.get('OPENROUTER_API_KEY')
if env_key and env_key != 'YOUR_OPENROUTER_API_KEY':
console.print(f"[green]✓ 环境变量中找到 API Key: {env_key[:10]}...[/green]")
return True
# 检查 .env 文件
env_file = Path('.env')
if env_file.exists():
with open(env_file, 'r') as f:
content = f.read()
if 'OPENROUTER_API_KEY=' in content and 'YOUR_OPENROUTER_API_KEY' not in content:
console.print("[green]✓ .env 文件中找到 API Key[/green]")
return True
# 检查配置文件
config_file = Path('config/config.json')
if config_file.exists():
import json
try:
with open(config_file, 'r') as f:
config = json.load(f)
api_key = config.get('openrouter', {}).get('api_key', '')
if api_key and api_key != 'YOUR_OPENROUTER_API_KEY':
console.print(f"[green]✓ 配置文件中找到 API Key: {api_key[:10]}...[/green]")
return True
except Exception as e:
console.print(f"[red]配置文件读取错误: {e}[/red]")
console.print("[red]✗ 未找到有效的 API Key[/red]")
console.print("\n请设置 OpenRouter API Key:")
console.print("1. 环境变量: export OPENROUTER_API_KEY='your_key'")
console.print("2. .env 文件: OPENROUTER_API_KEY=your_key")
console.print("3. 配置文件: 编辑 config/config.json")
return False
def quick_fix():
"""快速修复常见问题"""
console = Console()
console.print("[bold blue]EPUB 翻译器 - 快速修复[/bold blue]\n")
# 检查 API Key
if not check_api_key():
return False
# 检查依赖
console.print("\n[cyan]检查依赖...[/cyan]")
required_modules = [
'ebooklib', 'bs4', 'lxml', 'openai',
'aiohttp', 'pydantic', 'loguru', 'rich'
]
missing_modules = []
for module in required_modules:
try:
if module == 'bs4':
import bs4
else:
__import__(module)
console.print(f"[green]✓ {module}[/green]")
except ImportError:
console.print(f"[red]✗ {module}[/red]")
missing_modules.append(module)
if missing_modules:
console.print(f"\n[red]缺少依赖: {', '.join(missing_modules)}[/red]")
console.print("请运行: uv pip install -r requirements.txt")
return False
console.print("\n[green]✓ 所有检查通过[/green]")
return True
if __name__ == "__main__":
if quick_fix():
print("\n可以开始使用翻译器了！")
print("运行: python main.py your_book.epub --test")
else:
print("\n请先修复上述问题")
sys.exit(1)
archive/v0.09/scripts/example.py
+79
View File
@@ -0,0 +1,79 @@
#!/usr/bin/env python3
"""
EPUB 翻译器使用示例
演示如何使用程序进行翻译
"""
import asyncio
import sys
from pathlib import Path
# 添加 src 目录到路径
sys.path.insert(0, str(Path(__file__).parent / "src"))
from src.translator import EPUBTranslator
from src.utils import load_config, setup_logging
from rich.console import Console
async def example_usage():
"""使用示例"""
console = Console()
console.print("[bold blue]EPUB 翻译器使用示例[/bold blue]")
try:
# 加载配置
config = load_config('config/config.json')
setup_logging(config)
# 初始化翻译器
translator = EPUBTranslator(config)
# 示例 EPUB 文件路径（请替换为实际文件）
epub_file = "sample_book.epub"
if not Path(epub_file).exists():
console.print(f"[yellow]示例文件 {epub_file} 不存在[/yellow]")
console.print("请将你的 EPUB 文件放在当前目录并重命名为 sample_book.epub")
return
# 1. 估算翻译成本
console.print("\n[cyan]1. 估算翻译成本...[/cyan]")
estimate = await translator.get_translation_estimate(epub_file)
if estimate:
console.print(f"总段落数: {estimate['total_paragraphs']}")
console.print(f"估算时间: {estimate['estimated_time_minutes']:.1f} 分钟")
console.print(f"估算请求数: {estimate['estimated_requests']}")
# 2. 测试翻译
console.print("\n[cyan]2. 运行测试翻译...[/cyan]")
test_result = await translator.translate_epub(epub_file, test_mode=True)
if test_result.get('status') == 'success':
console.print("[green]测试翻译成功！[/green]")
else:
console.print("[red]测试翻译失败[/red]")
return
# 3. 询问是否继续完整翻译
console.print("\n[yellow]是否继续完整翻译？这可能需要一些时间和费用。[/yellow]")
response = input("输入 'yes' 继续，其他任意键退出: ")
if response.lower() == 'yes':
console.print("\n[cyan]3. 开始完整翻译...[/cyan]")
output_file = await translator.translate_epub(epub_file, test_mode=False)
console.print(f"[green]翻译完成！输出文件: {output_file}[/green]")
else:
console.print("[yellow]已取消完整翻译[/yellow]")
except Exception as e:
console.print(f"[red]示例运行失败: {e}[/red]")
finally:
await translator.llm_client.close()
if __name__ == "__main__":
asyncio.run(example_usage())
archive/v0.09/setup.py
+132
View File
@@ -0,0 +1,132 @@
#!/usr/bin/env python3
"""
快速安装和测试脚本
"""
import subprocess
import sys
import os
from pathlib import Path
def run_command(cmd, description):
"""运行命令并显示结果"""
print(f"\n🔄 {description}...")
try:
result = subprocess.run(cmd, shell=True, check=True, capture_output=True, text=True)
print(f"✅ {description}完成")
return True
except subprocess.CalledProcessError as e:
print(f"❌ {description}失败: {e}")
if e.stdout:
print(f"输出: {e.stdout}")
if e.stderr:
print(f"错误: {e.stderr}")
return False
def check_python_version():
"""检查 Python 版本"""
version = sys.version_info
if version.major < 3 or (version.major == 3 and version.minor < 9):
print(f"❌ Python 版本过低: {version.major}.{version.minor}")
print("需要 Python 3.9 或更高版本")
return False
print(f"✅ Python 版本: {version.major}.{version.minor}.{version.micro}")
return True
def setup_environment():
"""设置环境"""
print("🚀 EPUB 双语翻译程序 - 快速设置")
# 检查 Python 版本
if not check_python_version():
return False
# 检查 uv 是否安装
if not run_command("uv --version", "检查 uv"):
print("正在安装 uv...")
if not run_command("curl -LsSf https://astral.sh/uv/install.sh | sh", "安装 uv"):
print("❌ uv 安装失败，请手动安装")
return False
# 创建虚拟环境
if not Path(".venv").exists():
if not run_command("uv venv", "创建虚拟环境"):
return False
# 安装依赖
if not run_command("uv pip install -r requirements.txt", "安装依赖"):
return False
# 创建必要目录
for dir_name in ["output", "logs"]:
Path(dir_name).mkdir(exist_ok=True)
# 创建 .env 文件
if not Path(".env").exists():
with open(".env", "w") as f:
f.write("OPENROUTER_API_KEY=your_openrouter_api_key_here\n")
print("✅ 已创建 .env 文件")
return True
def test_installation():
"""测试安装"""
print("\n🧪 测试安装...")
# 测试导入
test_imports = [
"ebooklib",
"beautifulsoup4",
"lxml",
"openai",
"aiohttp",
"pydantic",
"loguru",
"rich"
]
for module in test_imports:
try:
if module == "beautifulsoup4":
import bs4
else:
__import__(module)
print(f"✅ {module}")
except ImportError:
print(f"❌ {module} 导入失败")
return False
return True
def main():
"""主函数"""
if not setup_environment():
print("\n❌ 环境设置失败")
return 1
if not test_installation():
print("\n❌ 安装测试失败")
return 1
print("\n🎉 安装完成！")
print("\n📋 下一步:")
print("1. 编辑 .env 文件，设置你的 OpenRouter API Key")
print("2. 运行测试: python main.py your_book.epub --test")
print("3. 查看帮助: python main.py --help")
# 检查是否有示例 EPUB 文件
epub_files = list(Path(".").glob("*.epub"))
if epub_files:
print(f"\n📚 发现 EPUB 文件: {epub_files[0].name}")
print(f"可以运行: python main.py '{epub_files[0].name}' --test")
return 0
if __name__ == "__main__":
sys.exit(main())
archive/v0.09/src/__init__.py
+32
View File
@@ -0,0 +1,32 @@
"""
EPUB 双语翻译程序
主要功能模块的初始化文件
"""
__version__ = "0.08"
__author__ = "Kaitan"
from .epub_parser import EPUBParser
from .translator import EPUBTranslator
from .llm_client import LLMClient as OpenRouterClient # Keep alias for compatibility
from .llm_client import LLMClient
from .text_processor import TextProcessor
from .bilingual_builder import BilingualEPUBBuilder
from .chinese_builder import ChineseEPUBBuilder
from .format_extractor import FormatExtractor
from .format_restorer import FormatRestorer
from .utils import load_config, setup_logging
__all__ = [
"EPUBParser",
"EPUBTranslator",
"LLMClient",
"OpenRouterClient",
"TextProcessor",
"BilingualEPUBBuilder",
"ChineseEPUBBuilder",
"FormatExtractor",
"FormatRestorer",
"load_config",
"setup_logging"
]
archive/v0.09/src/bilingual_builder.py
+235
View File
@@ -0,0 +1,235 @@
"""
双语 EPUB 构建器模块 (集成 V2)
基于 FineGrainedExtractor 的 DOM 回填机制，确保 100% 的内容对齐和格式保留。
"""
from ebooklib import epub
import ebooklib
from bs4 import BeautifulSoup
from typing import Dict, List
from pathlib import Path
from loguru import logger
import uuid
from .fine_grained_extractor import FineGrainedExtractor
class BilingualEPUBBuilder:
"""双语 EPUB 构建器"""
def __init__(self, original_book, config: Dict):
self.original_book = original_book
self.config = config
# 从配置中获取是否翻译目录
self.translate_toc = config['translation'].get('translate_toc', False)
def create_bilingual_epub_with_mapping(self, translation_map: Dict[str, str],
paragraph_map: Dict[str, Dict],
output_path: str) -> str:
"""
创建双语 EPUB。使用 ordered_ids 确保与 Manifest 严格一致。
Args:
translation_map: { global_item_id: translated_text_with_ph }
paragraph_map: { global_item_id: item_metadata_dict }
"""
try:
new_book = epub.EpubBook()
self._copy_metadata(new_book)
# 安全清理 TOC (虽然预处理已做，但构建新书对象时再次确保合规)
new_book.toc = self._sanitize_toc(self.original_book.toc)
# 1. 准备每个文件的有序ID列表
# 目的是将扁平的 map 重新按文件和顺序组织
file_ordered_ids = {}
# paragraph_map 的 key 是 global_id，通常包含顺序信息或我们依赖 items 的插入顺序
# 更好的方式是依赖 item ID 的数字部分排序，如果它们是 'id_0', 'id_1'...
# 假设 ID 包含顺序信息。
sorted_pids = sorted(paragraph_map.keys(), key=lambda x: self._extract_id_index(x))
for pid in sorted_pids:
info = paragraph_map[pid]
fname = info['file_name']
if fname not in file_ordered_ids:
file_ordered_ids[fname] = []
file_ordered_ids[fname].append(pid)
processed_item_ids = set()
item_map = {}
# 特殊处理：封面图片
self._handle_cover(new_book, processed_item_ids, item_map)
# 2. 复制所有非文档资源 (图片, CSS, 字体)
for item in self.original_book.get_items():
if item.get_type() != ebooklib.ITEM_DOCUMENT:
if item.id not in processed_item_ids:
new_book.add_item(item)
processed_item_ids.add(item.id)
item_map[item.id] = item
# 3. 处理并回填文档
new_spine = []
for spine_id, linear in self.original_book.spine:
item = self.original_book.get_item_with_id(spine_id)
if not item: continue
if item.get_type() == ebooklib.ITEM_DOCUMENT:
file_name = item.get_name()
new_item = item # 默认使用原 Item
# 如果该文件有翻译内容
if file_name in file_ordered_ids:
target_ids = file_ordered_ids[file_name]
# 执行回填
new_content = self._process_document_content(
item.get_content().decode('utf-8'),
file_name,
target_ids,
translation_map
)
# 创建新 item 避免污染原对象
new_item = epub.EpubHtml(
title=item.title,
file_name=file_name,
lang='zh-CN', # 双语版主要语言
uid=item.id
)
new_item.set_content(new_content.encode('utf-8'))
# 复制原 item 的其他属性如 style
for link in item.get_links():
# 这里不做深度复制，简单引用
pass
# 重新添加 links (特别是 CSS)
# 注意: EpubHtml 构造时不会自动带原来的 links，需要手动加
# 但我们在 dirty hack 里，直接 set_content 了 HTML。
# 如果 HTML head 里有 link, ebooklib 可能会解析并注册?
# Ebooklib 的行为是: 只有通过 add_link 加的才会出现在 opf manifest。
# 我们需要把原 item 的 links 复制过来
if hasattr(item, 'links'):
for link in item.links:
new_item.add_link(**link) # 不是很安全，视 ebooklib 版本而定
if new_item.id not in processed_item_ids:
new_book.add_item(new_item)
processed_item_ids.add(new_item.id)
new_spine.append(new_item)
else:
if item.id in item_map:
new_spine.append(item_map[item.id])
new_book.spine = new_spine
new_book.add_item(epub.EpubNcx())
new_book.add_item(epub.EpubNav())
# 生成输出文件名
output_file = self._generate_output_filename(output_path)
epub.write_epub(output_file, new_book, {})
logger.info(f"双语 EPUB 生成成功: {output_file}")
return output_file
except Exception as e:
logger.error(f"创建双语 EPUB 失败: {e}", exc_info=True)
raise
def _process_document_content(self, content: str, file_name: str,
target_ids: List[str], translation_map: Dict[str, str]) -> str:
"""
处理单个文档的内容：提取 -> 注入翻译 -> 回填
"""
try:
# 1. 再次提取，建立 DOM 映射
# 必须使用与 TextProcessing 阶段完全一致的参数
extractor = FineGrainedExtractor(translate_toc=self.translate_toc)
items = extractor.extract(content, file_name)
# 2. 筛选出应该翻译的项目
translatable_items = [i for i in items if i['should_translate']]
# 3. 一致性检查
if len(translatable_items) != len(target_ids):
logger.error(
f"严重错误 [{file_name}]: 提取项数 ({len(translatable_items)}) "
f"与 Manifest 记录数 ({len(target_ids)}) 不一致! "
"将跳过此文件的翻译回填以防错位。"
)
# Fallback: 返回原始内容
return content
# 4. 注入翻译
for extract_item, pid in zip(translatable_items, target_ids):
translation = translation_map.get(pid)
if translation:
extract_item['translation'] = translation
# 5. 回填
# 获取输出模式
output_mode = self.config.get('output', {}).get('mode', 'bilingual')
bilingual_mode = (output_mode == 'bilingual')
new_html = extractor.backfill(items, bilingual=bilingual_mode)
return new_html
except Exception as e:
logger.error(f"处理文档内容失败 {file_name}: {e}", exc_info=True)
return content
def _extract_id_index(self, pid: str) -> int:
"""从 ID 字符串中提取数字索引用于排序 (如 'p_10' -> 10)"""
try:
# 尝试常见格式 p_123, id_456
parts = pid.split('_')
if len(parts) > 1 and parts[-1].isdigit():
return int(parts[-1])
return 0
except:
return 0
def _sanitize_toc(self, toc):
"""确保 TOC 中的所有节点都有 ID"""
for item in toc:
if isinstance(item, (epub.Link, epub.Section)):
if not getattr(item, 'uid', None):
item.uid = f"navPoint-{uuid.uuid4().hex[:8]}"
elif isinstance(item, tuple) and len(item) == 2:
section, children = item
if isinstance(section, (epub.Link, epub.Section)):
if not getattr(section, 'uid', None):
section.uid = f"navPoint-{uuid.uuid4().hex[:8]}"
self._sanitize_toc(children)
return toc
def _handle_cover(self, new_book, processed_item_ids, item_map):
cover_id_meta = self.original_book.get_metadata('OPF', 'cover')
if cover_id_meta:
cover_item = self.original_book.get_item_with_id(cover_id_meta[0][0])
if cover_item:
new_book.add_item(cover_item)
processed_item_ids.add(cover_item.id)
item_map[cover_item.id] = cover_item
# 复制 cover metadata
new_book.add_metadata('OPF', 'cover', cover_item.id)
def _copy_metadata(self, new_book):
for namespace, meta_dict in self.original_book.metadata.items():
for name, values in meta_dict.items():
for value, other in values:
if name and hasattr(name, 'lower') and name.lower() == 'identifier': continue
new_book.add_metadata(namespace, name, value, other)
new_book.add_metadata('DC', 'language', 'zh-CN')
new_book.set_identifier(f"bilingual-{uuid.uuid4().hex[:12]}")
def _generate_output_filename(self, output_path: str) -> str:
# 根据模式生成不同的后缀
output_mode = self.config.get('output', {}).get('mode', 'bilingual')
suffix = "chinese" if output_mode == "chinese" else "bilingual"
title_meta = self.original_book.get_metadata('DC', 'title')
title = title_meta[0][0] if title_meta else "bilingual_book"
safe_title = "".join([c for c in title if c.isalnum() or c in (' ', '-', '_')]).strip()
Path(output_path).mkdir(parents=True, exist_ok=True)
return str(Path(output_path) / f"{safe_title}_{suffix}.epub")
archive/v0.09/src/book_profiler.py
+84
View File
@@ -0,0 +1,84 @@
'''Book Profiler Module
Features:
1. Automatically extract book samples to generate Book Profile (Genre, Style, Glossary).
'''
import json
import random
from pathlib import Path
from typing import Dict, List
from loguru import logger
from .manifest_manager import ManifestManager
from .llm_client import LLMClient
class BookProfiler:
def __init__(self, config: Dict, llm_client: LLMClient):
self.config = config
self.llm_client = llm_client
def extract_sample_text(self, manifest: ManifestManager, char_limit: int = 3000) -> str:
"""Extract sample text."""
items = manifest.get_items()
if not items: return ""
intro_text = []
for item in items[:50]:
if len(item.clean_text) > 50:
intro_text.append(item.clean_text)
body_text = []
body_items = [i for i in items[50:] if len(i.clean_text) > 80]
if body_items:
samples = random.sample(body_items, min(5, len(body_items)))
body_text = [i.clean_text for i in samples]
full_text = "\n\n".join(intro_text[:5] + body_text)
return full_text[:char_limit]
async def analyze_book(self, manifest: ManifestManager) -> Dict:
"""Generate Book Profile."""
existing_profile = manifest.data.get('metadata', {}).get('profile')
if existing_profile:
logger.info("Loaded existing Book Profile")
return existing_profile
sample = self.extract_sample_text(manifest)
if not sample: return {}
logger.info("Generating Book Profile...")
system_prompt = "You are a senior publishing editor. Analyze the text and output JSON."
user_prompt = f"""
Please analyze the following book excerpt.
Output JSON format:
{{
"genre": "Genre",
"style": "Style description",
"audience": "Target Audience",
"glossary": {{ "Term": "Chinese Translation" }},
"translation_instruction": "Specific instruction for translator"
}}
Excerpt:
{sample}
"""
try:
response = await self.llm_client.raw_chat_completion(system_prompt, user_prompt)
json_str = response.strip()
if "```json" in json_str:
json_str = json_str.split("```json")[1].split("```")[0].strip()
elif "```" in json_str:
json_str = json_str.split("```")[1].split("```")[0].strip()
profile = json.loads(json_str)
if 'metadata' not in manifest.data:
manifest.data['metadata'] = {}
manifest.data['metadata']['profile'] = profile
manifest.save()
return profile
except Exception as e:
logger.error(f"Profile generation failed: {e}")
return {}
archive/v0.09/src/cache.py
+226
View File
@@ -0,0 +1,226 @@
"""
翻译缓存管理模块 - 简化版
基于全局ID和chunk的缓存系统
"""
import json
import hashlib
from pathlib import Path
from datetime import datetime, timedelta
from typing import Dict, Optional, List
from loguru import logger
class TranslationCache:
"""翻译缓存管理器 - 简化版"""
def __init__(self, config: Dict):
"""初始化缓存管理器"""
self.config = config
cache_config = config.get('cache', {})
self.enabled = cache_config.get('enabled', True)
self.cache_dir = Path(cache_config.get('directory', 'cache'))
self.max_age_days = cache_config.get('max_age_days', 30)
if self.enabled:
self.cache_dir.mkdir(parents=True, exist_ok=True)
self.translations_dir = self.cache_dir / 'translations'
self.translations_dir.mkdir(parents=True, exist_ok=True)
logger.info(f"翻译缓存已启用: {self.cache_dir}")
def get_chunk_translation(self, chunk: List[Dict], model: str) -> Optional[Dict[str, str]]:
"""
获取chunk的缓存翻译
Args:
chunk: 段落列表（带global_id）
model: 模型名称
Returns:
{global_id: translation} 映射，如果不存在返回 None
"""
if not self.enabled:
return None
try:
cache_key = self._get_chunk_cache_key(chunk, model)
cache_file = self._get_cache_file_path(cache_key)
if not cache_file.exists():
return None
# 检查是否过期
file_age = datetime.now() - datetime.fromtimestamp(cache_file.stat().st_mtime)
if file_age > timedelta(days=self.max_age_days):
logger.debug(f"缓存已过期: {cache_key[:8]}...")
cache_file.unlink()
return None
# 读取缓存
with open(cache_file, 'r', encoding='utf-8') as f:
cache_data = json.load(f)
# 验证缓存
if (cache_data.get('success') and
cache_data.get('model') == model and
self._validate_cache_data(cache_data, chunk)):
logger.debug(f"缓存命中: {cache_key[:8]}... ({len(chunk)} 段落)")
return cache_data.get('translations', {})
return None
except Exception as e:
logger.warning(f"读取缓存失败: {e}")
return None
def save_chunk_translation(self, chunk: List[Dict], translations: Dict[str, str],
model: str, success: bool = True) -> None:
"""
保存chunk翻译到缓存
Args:
chunk: 段落列表（带global_id）
translations: {global_id: translation} 映射
model: 模型名称
success: 是否翻译成功
"""
if not self.enabled:
return
try:
cache_key = self._get_chunk_cache_key(chunk, model)
cache_file = self._get_cache_file_path(cache_key)
# 构建缓存数据
cache_data = {
'global_ids': [p['global_id'] for p in chunk],
'translations': translations,
'model': model,
'timestamp': datetime.now().isoformat(),
'success': success,
'paragraph_count': len(chunk),
'cache_version': '3.0'
}
with open(cache_file, 'w', encoding='utf-8') as f:
json.dump(cache_data, f, ensure_ascii=False, indent=2)
logger.debug(f"缓存已保存: {cache_key[:8]}... ({len(chunk)} 段落)")
except Exception as e:
logger.warning(f"保存缓存失败: {e}")
def _get_chunk_cache_key(self, chunk: List[Dict], model: str) -> str:
"""
生成chunk缓存键（基于全局ID序列）
Args:
chunk: 段落列表
model: 模型名称
Returns:
缓存键
"""
# 使用全局ID序列作为缓存键的一部分
id_sequence = ",".join(p['global_id'] for p in chunk)
combined = f"{id_sequence}|{model}"
return hashlib.md5(combined.encode('utf-8')).hexdigest()
def _get_cache_file_path(self, cache_key: str) -> Path:
"""获取缓存文件路径"""
# 使用 hash 前缀分目录，避免单目录文件过多
subdir = cache_key[:2]
cache_subdir = self.translations_dir / subdir
cache_subdir.mkdir(parents=True, exist_ok=True)
return cache_subdir / f"{cache_key}.json"
def _validate_cache_data(self, cache_data: Dict, chunk: List[Dict]) -> bool:
"""验证缓存数据的有效性"""
# 检查ID序列是否匹配
cached_ids = cache_data.get('global_ids', [])
chunk_ids = [p['global_id'] for p in chunk]
if cached_ids != chunk_ids:
logger.debug("缓存ID序列不匹配")
return False
# 只要有翻译结果就认为有效，不要求数量完全匹配
translations = cache_data.get('translations', {})
if not translations:
logger.debug("缓存无翻译结果")
return False
return True
def clear_cache(self, older_than_days: Optional[int] = None) -> int:
"""清理缓存"""
if not self.enabled or not self.translations_dir.exists():
return 0
cleared_count = 0
cutoff_time = None
if older_than_days is not None:
cutoff_time = datetime.now() - timedelta(days=older_than_days)
try:
for cache_file in self.translations_dir.rglob('*.json'):
should_delete = False
if cutoff_time is None:
should_delete = True
else:
file_time = datetime.fromtimestamp(cache_file.stat().st_mtime)
should_delete = file_time < cutoff_time
if should_delete:
cache_file.unlink()
cleared_count += 1
# 清理空目录
for date_dir in self.translations_dir.iterdir():
if date_dir.is_dir() and not any(date_dir.iterdir()):
date_dir.rmdir()
logger.info(f"清理了 {cleared_count} 个缓存文件")
return cleared_count
except Exception as e:
logger.error(f"清理缓存失败: {e}")
return 0
def get_cache_stats(self) -> Dict:
"""获取缓存统计信息"""
if not self.enabled or not self.translations_dir.exists():
return {'enabled': False}
try:
cache_files = list(self.translations_dir.rglob('*.json'))
total_files = len(cache_files)
total_size = sum(f.stat().st_size for f in cache_files)
# 统计段落数
total_paragraphs = 0
for cache_file in cache_files:
try:
with open(cache_file, 'r', encoding='utf-8') as f:
data = json.load(f)
total_paragraphs += data.get('paragraph_count', 0)
except:
continue
return {
'enabled': True,
'total_files': total_files,
'total_paragraphs': total_paragraphs,
'total_size_mb': round(total_size / 1024 / 1024, 2),
'cache_directory': str(self.cache_dir),
'max_age_days': self.max_age_days
}
except Exception as e:
logger.error(f"获取缓存统计失败: {e}")
return {'enabled': True, 'error': str(e)}
archive/v0.09/src/chinese_builder.py
+226
View File
@@ -0,0 +1,226 @@
"""
纯中文 EPUB 构建器模块
负责：
1. 复制原始 EPUB 结构
2. 使用译文替换原文
3. 调用 FormatRestorer 将译文占位符还原为 HTML 标签
"""
from ebooklib import epub
import ebooklib
from bs4 import BeautifulSoup
from typing import Dict, List, Any
from pathlib import Path
from loguru import logger
import uuid
from .fine_grained_extractor import FineGrainedExtractor
class ChineseEPUBBuilder:
"""纯中文 EPUB 构建器 (DOM Safe)"""
def __init__(self, original_book, config: Dict):
self.original_book = original_book
self.config = config
self.translate_toc = config['translation'].get('translate_toc', False)
def create_chinese_epub_with_mapping(self,
items: List[Any], # List[ManifestItem]
output_path: str) -> str:
"""
创建纯中文 EPUB。
"""
try:
new_book = epub.EpubBook()
self._copy_metadata(new_book)
# 安全清理 TOC
new_book.toc = self._sanitize_toc(self.original_book.toc)
# 1. 按文件分组 Manifest Items
# 假设 items 已经是按全局 ID 排序的 (ManifestManager.get_items 返回有序列表)
file_items_map = {}
for item in items:
fname = item.source_file
if fname not in file_items_map:
file_items_map[fname] = []
file_items_map[fname].append(item)
processed_item_ids = set()
item_map = {}
# 特殊处理：封面图片
# 复制资源
for item in self.original_book.get_items():
if item.get_type() != ebooklib.ITEM_DOCUMENT:
if item.id not in processed_item_ids:
new_book.add_item(item)
processed_item_ids.add(item.id)
item_map[item.id] = item
# 重建 Spine
new_spine = []
for spine_id, linear in self.original_book.spine:
item = self.original_book.get_item_with_id(spine_id)
if not item: continue
if item.get_type() == ebooklib.ITEM_DOCUMENT:
file_name = item.get_name()
if file_name in file_items:
new_item = self._create_chinese_document(
item, file_items[file_name]
)
new_item.id = item.id
else:
new_item = item
if new_item.id not in processed_item_ids:
new_book.add_item(new_item)
processed_item_ids.add(new_item.id)
new_spine.append(new_item)
else:
if item.id in item_map:
new_spine.append(item_map[item.id])
new_book.spine = new_spine
new_book.add_item(epub.EpubNcx())
new_book.add_item(epub.EpubNav())
output_file = self._generate_output_filename(output_path)
epub.write_epub(output_file, new_book, {})
return output_file
except Exception as e:
logger.error(f"创建纯中文 EPUB 失败: {e}", exc_info=True)
raise
def _sanitize_toc(self, toc):
"""确保 TOC 中的所有节点都有 ID"""
for item in toc:
if isinstance(item, (epub.Link, epub.Section)):
if not getattr(item, 'uid', None):
item.uid = f"navPoint-{uuid.uuid4().hex[:8]}"
elif isinstance(item, tuple) and len(item) == 2:
section, children = item
if isinstance(section, (epub.Link, epub.Section)):
if not getattr(section, 'uid', None):
section.uid = f"navPoint-{uuid.uuid4().hex[:8]}"
self._sanitize_toc(children)
return toc
def _copy_metadata(self, new_book):
try:
for namespace, meta_dict in self.original_book.metadata.items():
for name, values in meta_dict.items():
for value, other in values:
if name and hasattr(name, 'lower') and name.lower() == 'identifier': continue
new_book.add_metadata(namespace, name, value, other)
new_book.add_metadata('DC', 'language', 'zh-CN')
new_book.set_identifier(f"chinese-{uuid.uuid4().hex[:12]}")
except Exception as e:
logger.error(f"元数据复制出错: {e}")
def _create_chinese_document(self, original_item, manifest_items: list):
try:
from .text_processor import TextProcessor
soup = BeautifulSoup(original_item.get_content().decode('utf-8'), 'html.parser')
# 获取文本元素
text_elements = TextProcessor.get_valid_text_elements(soup)
current_idx = 0
for element in text_elements:
if not TextProcessor.clean_element_text(element): continue
if current_idx < len(manifest_items):
m_item = manifest_items[current_idx]
# 只有当非导航元素时才尝试替换内容
if not TextProcessor.is_navigation_element(element):
# 检查是否是嵌套容器（需要特殊处理）
is_nested = TextProcessor.is_nested_container(element)
# 优先使用带格式的翻译，降级到纯文本翻译
if m_item.translation_with_original_html:
if is_nested:
self._replace_direct_content(element, m_item.translation_with_original_html)
else:
self._replace_content(element, m_item.translation_with_original_html)
elif m_item.translation:
# 降级：使用纯文本翻译（无格式）
if is_nested:
self._replace_direct_content(element, m_item.translation)
else:
self._replace_content(element, m_item.translation)
current_idx += 1
new_item = epub.EpubHtml(title=original_item.title, file_name=original_item.get_name(), lang='zh-CN')
new_item.set_content(str(soup).encode('utf-8'))
return new_item
except Exception as e:
logger.error(f"创建中文文档失败 {original_item.get_name()}: {e}")
return original_item
def _replace_content(self, element, translated_html: str):
"""用译文替换元素的 inner_html"""
try:
# 将译文 HTML 字符串解析为 BeautifulSoup 对象
new_soup = BeautifulSoup(translated_html, 'html.parser')
# 清空原元素并填入新内容
element.clear()
# 重要：必须先转换为 list，否则 append 会修改 contents 导致跳过元素
for content in list(new_soup.contents):
element.append(content)
except Exception as e:
logger.error(f"替换内容失败: {e}")
def _replace_direct_content(self, element, translated_html: str):
"""
替换嵌套容器元素的直接文本内容（保留子块级元素）
策略：
1. 保存所有子块级元素
2. 清空元素内容
3. 填入译文
4. 在末尾追加保存的子块
"""
try:
from bs4 import NavigableString
block_tags = ['p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'blockquote', 'li', 'td']
# 1. 保存所有子块级元素
saved_blocks = []
for child in element.find_all(block_tags, recursive=False):
# 只保存直接子元素中的块
saved_blocks.append(child.extract())
# 2. 额外保存嵌套在 span 等内联元素中的块
for inline in element.find_all(['span', 'a', 'em', 'strong'], recursive=True):
for child in inline.find_all(block_tags, recursive=False):
saved_blocks.append(child.extract())
# 3. 清空并填入译文
new_soup = BeautifulSoup(translated_html, 'html.parser')
element.clear()
for content in list(new_soup.contents):
element.append(content)
# 4. 追加保存的子块
for block in saved_blocks:
element.append(block)
except Exception as e:
logger.error(f"替换嵌套内容失败: {e}")
# 降级到普通替换
self._replace_content(element, translated_html)
def _generate_output_filename(self, output_path: str) -> str:
from .utils import sanitize_filename
title = self.original_book.get_metadata('DC', 'title')
clean_title = sanitize_filename(title[0][0]) if title else "chinese_book"
Path(output_path).mkdir(parents=True, exist_ok=True)
return str(Path(output_path) / f"{clean_title}_chinese.epub")
archive/v0.09/src/epub_cleaner.py
+178
View File
@@ -0,0 +1,178 @@
"""
EPUB 清理器模块 (EpubCleaner)
负责在翻译前对 EPUB 进行标准化清洗，解决兼容性问题。
核心功能:
1. Flatten Structure: 将 div 转换为 p，简化结构
2. Fix TOC: 修复目录中的死链和缺失 UID
3. CSS Restoration: 找回丢失的样式表
"""
from bs4 import BeautifulSoup, Tag
from loguru import logger
from ebooklib import epub
import ebooklib
import zipfile
import uuid
from ebooklib.epub import Link
class EpubCleaner:
"""标准 EPUB 清理器"""
def clean_epub(self, input_path: str, output_path: str):
"""
清理 EPUB 文件并保存到新路径
"""
logger.info(f"开始清理: {input_path}")
# 1. 尝试打开 Zip 以读取原始内容 (Ebooklib 回退机制)
try:
input_zip = zipfile.ZipFile(input_path, 'r')
zip_files = set(input_zip.namelist())
except Exception as e:
logger.error(f"无法打开 Zip (样式回退功能将失效): {e}")
input_zip = None
zip_files = set()
# 2. 读取 EPUB
try:
book = epub.read_epub(input_path)
except Exception as e:
logger.error(f"Ebooklib 读取失败: {e}")
raise
# 3. 遍历并清理文档
count = 0
for item in book.get_items():
if item.get_type() == ebooklib.ITEM_DOCUMENT:
try:
file_name = item.get_name()
content = None
# 优先从 Zip 读取以保留 Head 信息 (CSS Links)
if input_zip and file_name in zip_files:
try:
content = input_zip.read(file_name).decode('utf-8')
except Exception:
pass
# 回退到 ebooklib
if content is None:
raw_content = item.get_content()
if raw_content:
content = raw_content.decode('utf-8')
if not content or not content.strip():
continue
# 执行清理
cleaned = self._clean_content(content, item)
# 安全检查
if not cleaned.strip():
logger.warning(f"警告: {file_name} 清理后为空，保留原始内容")
cleaned = content
item.set_content(cleaned.encode('utf-8'))
count += 1
except Exception as e:
logger.warning(f"清理文档失败 {item.get_name()}: {e}")
# 4. 修复 TOC (死链和 UID)
try:
book.toc = self._fix_and_clean_toc(book.toc, book)
except Exception as e:
logger.error(f"TOC 修复失败: {e}")
# 5. 保存
epub.write_epub(output_path, book)
logger.info(f"清理完成: {output_path} (处理了 {count} 个文档)")
def _clean_content(self, html_content: str, item=None) -> str:
"""
执行具体的 HTML 清理逻辑
"""
soup = BeautifulSoup(html_content, 'html.parser')
# 恢复 CSS 链接
if item:
self._restore_css_links(soup, item)
# div 转 p
stats = {'divs_to_p': 0}
inline_tags = {'span', 'em', 'strong', 'i', 'b', 'u', 'a', 'br', 'sub', 'sup'}
for div in list(soup.find_all('div')):
# 检查是否有块级子元素 (如果有，则保留 div 容器结构)
has_block = any(
isinstance(c, Tag) and c.name not in inline_tags
for c in div.children
)
if not has_block:
div.name = 'p'
stats['divs_to_p'] += 1
# 可以在这里添加更多清理逻辑 (如移除无用的空的 span 等)
return str(soup)
def _restore_css_links(self, soup, item):
"""从原始 HTML 中提取并恢复 CSS 链接到 item 对象"""
head = soup.find('head')
if head:
links = head.find_all('link', rel='stylesheet')
for link in links:
href = link.get('href')
if href:
existing_links = list(item.get_links())
exists = False
for l in existing_links:
l_href = getattr(l, 'href', None)
if l_href is None and isinstance(l, dict):
l_href = l.get('href')
if l_href == href:
exists = True
break
if not exists:
item.add_link(href=href, rel='stylesheet', type='text/css')
def _fix_and_clean_toc(self, toc, book):
"""修复 TOC：补全 UID 并移除指向不存在文件的死链"""
new_toc = []
for item in toc:
# Case 1: (Section, Children)
if isinstance(item, (tuple, list)):
section, children = item
cleaned_children = self._fix_and_clean_toc(children, book)
if isinstance(section, Link):
href = section.href.split('#')[0]
if book.get_item_with_href(href):
if section.uid is None:
section.uid = f'uuid-{uuid.uuid4()}'
new_toc.append((section, cleaned_children))
else:
logger.warning(f"移除无效 TOC 节点: {section.href}")
new_toc.extend(cleaned_children)
else:
new_toc.append((section, cleaned_children))
# Case 2: Link
elif isinstance(item, Link):
href = item.href.split('#')[0]
if book.get_item_with_href(href):
if item.uid is None:
item.uid = f'uuid-{uuid.uuid4()}'
new_toc.append(item)
else:
logger.warning(f"移除无效 TOC 节点: {item.href}")
# Case 3: Other
else:
new_toc.append(item)
return new_toc
archive/v0.09/src/epub_parser.py
+175
View File
@@ -0,0 +1,175 @@
"""
EPUB 解析器模块 (EPUB Parser Module)
该模块负责读取 EPUB 文件，提取元数据和内容项目。
它使用 ebooklib 库来处理 EPUB 格式的底层细节。
Classes:
EPUBParser: 负责 EPUB 文件的加载、元数据提取和内容项遍历。
"""
import ebooklib
from ebooklib import epub
from bs4 import BeautifulSoup
from typing import List, Dict, Any, Set, Optional
from pathlib import Path
from loguru import logger
class EPUBParser:
"""
EPUB 文件解析器。
负责加载 EPUB 文件，提取书籍元数据（如标题、作者），并提供方法来遍历和提取
书中的文档内容（HTML/XHTML）。
Attributes:
epub_path (Path): EPUB 文件的路径对象。
book (epub.EpubBook): ebooklib 加载的书籍对象。
metadata (Dict[str, str]): 提取的书籍元数据字典。
"""
def __init__(self, epub_path: str):
"""
初始化 EPUB 解析器。
Args:
epub_path (str): EPUB 文件的文件路径。
Raises:
FileNotFoundError: 如果指定的文件不存在。
Exception: 如果 EPUB 文件加载失败（格式错误等）。
"""
self.epub_path = Path(epub_path)
if not self.epub_path.exists():
raise FileNotFoundError(f"EPUB 文件不存在: {epub_path}")
try:
# ignore_ncx=True 是为了避免某些旧版 epub 的警告，但新版 ebooklib 可能行为不同
# 这里直接读取，让 ebooklib 处理
self.book = epub.read_epub(str(self.epub_path))
logger.info(f"成功加载 EPUB: {self.epub_path.name}")
except Exception as e:
logger.error(f"加载 EPUB 失败: {e}")
raise
self.metadata = self._extract_metadata()
def _extract_metadata(self) -> Dict[str, str]:
"""
从 EPUB 对象中提取标准元数据。
提取 Dublin Core (DC) 元数据，包括标题、作者和语言。
Returns:
Dict[str, str]: 包含 'title', 'author', 'language' 的字典。
如果提取失败，会使用默认值 ("Unknown", "en")。
"""
metadata = {}
try:
# get_metadata 返回的是 (value, dict) 的列表，我们取第一个结果
title_meta = self.book.get_metadata('DC', 'title')
metadata['title'] = title_meta[0][0] if title_meta else "Unknown"
author_meta = self.book.get_metadata('DC', 'creator')
metadata['author'] = author_meta[0][0] if author_meta else "Unknown"
lang_meta = self.book.get_metadata('DC', 'language')
metadata['language'] = lang_meta[0][0] if lang_meta else "en"
logger.info(f"书籍: {metadata['title']} - {metadata['author']}")
except Exception as e:
logger.warning(f"提取元数据时出错: {e}")
# 设置保底值
metadata.setdefault('title', 'Unknown')
metadata.setdefault('author', 'Unknown')
metadata.setdefault('language', 'en')
return metadata
def get_toc(self):
"""
获取书籍的目录结构 (Table of Contents)
Returns:
book.toc: ebooklib 的原始 TOC 结构
"""
return self.book.toc
def extract_all_content_items(self, include_files: Optional[Set[str]] = None) -> List[Dict[str, Any]]:
"""
提取所有可翻译的内容项目（文档）。
遍历 EPUB 中的所有 Item，筛选出类型为 ITEM_DOCUMENT 的项目。
同时会进行简单的过滤，跳过内容过短（<100字符）或看起来像非正文的文件（如 nav, toc, cover）。
Returns:
List[Dict[str, Any]]: 内容项目列表。每个字典包含：
- item (epub.EpubItem): 原始 Item 对象。
- file_name (str): 文件名。
- content (str): 解码后的 HTML 内容。
- text_length (int): 纯文本长度（用于统计）。
"""
content_items = []
# 获取所有文档类型的项目
for item in self.book.get_items():
if item.get_type() == ebooklib.ITEM_DOCUMENT:
# 如果指定了 include_files，则只处理其中的文件
if include_files is not None:
item_name = item.get_name()
if item_name not in include_files:
logger.debug(f"跳过未选中的文件: {item_name}")
continue
try:
# 获取内容 (bytes -> str)
content = item.get_content().decode('utf-8')
# 简单的内容验证：提取纯文本检查长度
soup = BeautifulSoup(content, 'html.parser')
text = soup.get_text().strip()
# 跳过太短的内容（可能是只有图片的页面、空页面）
if len(text) < 100:
logger.debug(f"跳过短内容: {item.get_name()} ({len(text)} 字符)")
continue
# 注意：文件名跳过逻辑已移至 TOCParser.get_skip_files()
# 通过 include_files 参数在调用前过滤
content_items.append({
'item': item,
'file_name': item.get_name(),
'content': content,
'text_length': len(text)
})
logger.debug(f"添加内容项: {item.get_name()} ({len(text)} 字符)")
except Exception as e:
logger.warning(f"处理项目失败 {item.get_name()}: {e}")
continue
logger.info(f"提取了 {len(content_items)} 个内容项目")
return content_items
def get_book_info(self) -> Dict[str, str]:
"""
获取书籍的摘要信息。
Returns:
Dict[str, str]: 包含文件名、标题、作者、语言和文档数量的字典。
"""
# 统计内容项
document_count = sum(1 for item in self.book.get_items()
if item.get_type() == ebooklib.ITEM_DOCUMENT)
return {
'filename': self.epub_path.name,
'title': self.metadata.get('title', 'Unknown'),
'author': self.metadata.get('author', 'Unknown'),
'language': self.metadata.get('language', 'en'),
'document_count': document_count
}
archive/v0.09/src/fine_grained_extractor.py
+222
View File
@@ -0,0 +1,222 @@
"""
Fine-Grained Extractor (集成格式保护版)
负责从 EPUB 中提取文本，进行精细化处理，并负责最终的回填工作。
集成 FormatExtractor 以实现行内格式的保护。
"""
from bs4 import BeautifulSoup, Tag, NavigableString
from typing import List, Dict, Any, Tuple
import re
from loguru import logger
from .format_extractor import FormatExtractor
from .format_restorer import FormatRestorer
class FineGrainedExtractor:
"""细粒度提取与回填器"""
SKIP_TRANSLATION_PATTERNS = [
r'index\.x?html',
r'bibliography\.x?html',
r'endnotes?\.x?html',
r'footnotes?\.x?html',
]
TOC_PATTERNS = [
r'nav\.x?html',
r'toc\.x?html',
]
def __init__(self, translate_toc: bool = False):
self.translate_toc = translate_toc
self.soup = None
# 初始化格式处理器
self.format_extractor = FormatExtractor()
self.format_restorer = FormatRestorer()
def extract(self, html_content: str, file_name: str = "") -> List[Dict[str, Any]]:
"""
提取所有正文元素 (p, h1-h6)，并进行格式分析
"""
self.soup = BeautifulSoup(html_content, 'html.parser')
# 不要移除 link/style/meta/script，否则回填时会丢失头部信息
# for element in self.soup(['script', 'style', 'meta', 'link']):
# element.decompose()
doc_type = self._classify_document(file_name)
items = []
# 目标: 所有段落和标题
target_tags = ['p', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6']
for element in self.soup.find_all(target_tags):
# 1. 初步文本提取 (用于过滤判断)
raw_text = element.get_text(separator=' ', strip=True)
if not raw_text.strip():
continue
# 2. 判断是否需要翻译
is_decorative = self._is_decorative(raw_text)
should_translate = self._should_translate(doc_type, is_decorative, raw_text)
item_data = {
'element': element, # 持有引用，用于回填
'tag': element.name,
'raw_text': raw_text,
'should_translate': should_translate,
'doc_type': doc_type,
'text_len': len(raw_text)
}
# 3. 如果需要翻译，执行通过 FormatExtractor 进行精细化格式提取
if should_translate:
# 必须传入 Outer HTML (str(element))，因为 FormatExtractor 内部会解析并剥离最外层标签
# 如果只传 inner_html，FormatExtractor 解析时只能拿到第一个子节点，导致内容丢失验证失败
outer_html = str(element)
# 提取 (clean_text, text_with_ph, map, type, endnote_anchors)
clean, text_ph, ph_map, _, endnote_anchors = self.format_extractor.extract(outer_html)
# 再次验证：如果提取后的 clean_text 为空 (比如全是公式)，则不翻译
if not clean.strip() or not text_ph.strip():
item_data['should_translate'] = False
else:
item_data['text'] = clean # 纯文本 (供人阅读/日志)
item_data['text_nodes'] = [] # 兼容旧字段 (空)
item_data['text_with_ph'] = text_ph # 发送给 LLM 的文本
item_data['placeholder_map'] = ph_map
item_data['endnote_anchors'] = endnote_anchors # 尾注锚点 ID 列表
else:
# 不需要翻译，仅保留基础信息
item_data['text'] = raw_text
item_data['text_with_ph'] = raw_text # Fallback
items.append(item_data)
logger.debug(f"[{doc_type}] {file_name}: 提取 {len(items)} 元素, 需翻译 {sum(1 for i in items if i['should_translate'])}")
return items
def backfill(self, items: List[Dict[str, Any]], bilingual: bool = True) -> str:
"""
回填翻译 (支持格式还原)
Args:
items: 提取的元素列表，且已注入 'translation' 字段 (带占位符的译文)
bilingual: 是否生成双语版本
"""
success_count = 0
for item in items:
if not item.get('should_translate'):
continue
# 使用预先注入的翻译 (解决了重复文本映射问题)
translated_ph = item.get('translation')
if not translated_ph:
continue
original_element = item['element']
# 4. 格式还原
ph_map = item.get('placeholder_map', {})
# 容错：如果 ph_map 为 None (未开启格式保护)，设为空字典
if ph_map is None: ph_map = {}
restored_html, _ = self.format_restorer.restore(translated_ph, ph_map)
# 5. 构建新 DOM 元素
if bilingual:
# 双语模式: Append
new_tag = self.soup.new_tag(original_element.name)
# 继承 class
classes = original_element.get('class', [])
new_tag['class'] = list(classes) + ['translation', 'chinese']
# 继承 style
style = original_element.get('style')
if style:
new_tag['style'] = style
# 设置内容 (解析 restored HTML)
inner_soup = BeautifulSoup(restored_html, 'html.parser')
if inner_soup.body:
for child in list(inner_soup.body.children):
new_tag.append(child)
else:
for child in list(inner_soup.children):
new_tag.append(child)
original_element.insert_after(new_tag)
else:
# 仅中文模式: Replace
# 直接修改 original_element 的内容
original_element.clear()
inner_soup = BeautifulSoup(restored_html, 'html.parser')
# 直接替换内容
if inner_soup.body:
for child in list(inner_soup.body.children):
original_element.append(child)
else:
for child in list(inner_soup.children):
original_element.append(child)
# 可以在这里移除 dropcap class?
# 但如果 dropcap 是内部 span，已经被还原回去了。
# 由于 Drop Cap 处理在 FormatExtractor 已经把首字母放入文本，Prefix 里的 dropcap 是空的
# 还原后的 HTML 大概是 <span class="dropcap"></span>这...
pass
success_count += 1
return str(self.soup)
# --- 以下是辅助判别逻辑 (同原版) ---
def _classify_document(self, file_name: str) -> str:
if not file_name: return 'core'
fname = file_name.lower()
if any(re.search(p, fname) for p in self.SKIP_TRANSLATION_PATTERNS): return 'skip'
if any(re.search(p, fname) for p in self.TOC_PATTERNS): return 'toc'
return 'core'
def _should_translate(self, doc_type: str, is_decorative: bool, text: str = "") -> bool:
if is_decorative: return False
if self._is_roman_numeral(text): return False
if doc_type == 'core': return True
if doc_type == 'toc': return self.translate_toc
return False
def _is_roman_numeral(self, text: str) -> bool:
text = text.strip().upper()
if not text: return False
# 简单宽松匹配: 纯字母且看起来像罗马数字 (I, V, X, L, C, M)
# 排除普通单词如 "I" (作为代词时应翻译，但作为单独段落通常是标题)
# 这是一个权衡。单独的 "I" 在小说里可能表示 "我"，但在章节标题里表示 "第一章"。
# 如果是正文中的 "I am...", 肯定会被提取。这里只有单独的 "I" 才会被这里匹配。
# 真正的问题是：单独一行 "I" 表示 "我" 的情况极少。
pattern = re.compile(r"^M{0,4}(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})$")
return bool(pattern.match(text))
def _is_decorative(self, text: str) -> bool:
s = text.strip()
if not s: return False
if not any(c.isalnum() for c in s): return True
if len(s) > 20: return False
patterns = [
r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾]+$',
r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾\s]+$',
]
for p in patterns:
if re.match(p, s): return True
# 字符种类很少且包含非字母 (e.g. "* * *")
unique = set(s.replace(' ', ''))
if len(unique) <= 3 and (unique & set('*-_~•◆◇■□▪▫●○◉◎⦿⦾')):
return True
return False
archive/v0.09/src/format_extractor.py
+580
View File
@@ -0,0 +1,580 @@
"""
格式提取模块 (优化版 v3)
核心优化：
1. 前缀/后缀标签分离：文本前后的标签不发送给 LLM，直接回填
2. 公式检测：将数学变量/公式作为整体占位符
3. 简化占位符：φ1φ 格式，每个段落独立编号
"""
import re
from bs4 import BeautifulSoup, Tag, NavigableString
from typing import Tuple, Dict, Any, List, Optional
class HeadingDetector:
"""标题与段落类型检测器"""
CHAPTER_PATTERNS = [
r'^(chapter|chap\.?|part)\s+([0-9]+|[ivxlc]+|[a-z])',
r'^(第\s*[0-9一二三四五六七八九十百]+\s*[章节部篇])',
r'^(\d+|[IVXLC]+|[A-Z])\.$'
]
EPIGRAPH_CLASSES = {
'epigraph', 'quote', 'blockquote', 'motto',
'dedication', 'verse', 'poetry', 'poem'
}
def detect(self, element: Tag, text: str) -> str:
if self._is_epigraph(element):
return "epigraph"
tag_name = element.name.lower()
if tag_name in ['h1', 'h2']:
return "chapter" if self._matches_chapter_pattern(text) else "section"
if tag_name == 'h3':
return "section"
if tag_name in ['h4', 'h5', 'h6']:
return "subsection"
if self._is_pseudo_heading(element, text):
return "subsection"
return "body"
def _is_epigraph(self, element: Tag) -> bool:
if element.name == 'blockquote':
return True
current = element
for _ in range(3):
if not current: break
classes = current.get('class', [])
if isinstance(classes, list):
classes = ' '.join(classes)
if any(k in classes.lower() for k in self.EPIGRAPH_CLASSES):
return True
current = current.parent
return False
def _matches_chapter_pattern(self, text: str) -> bool:
text = text.strip().lower()
for pattern in self.CHAPTER_PATTERNS:
if re.match(pattern, text, re.IGNORECASE):
return True
return False
def _is_pseudo_heading(self, element: Tag, text: str) -> bool:
if element.name != 'p':
return False
text = text.strip()
if not text or len(text) > 80:
return False
children = list(element.children)
if len(children) == 1 and isinstance(children[0], Tag):
if children[0].name in ['strong', 'b']:
return True
return False
class FormatExtractor:
"""
HTML 格式提取器 (优化版 v3)
核心改进：
1. 前缀/后缀标签分离 - 不发送给 LLM，自动回填
2. 公式元素整体替换
3. 简化占位符格式 φ1φ, φ2φ
"""
FORMULA_CHARS = re.compile(
r'^[\d\s\+\-\×\÷\=\(\)\[\]\{\}\<\>\^\*\/\.\,\;\:\'\"\`\~\@\#\$\%\&\|\\'
r'αβγδεζηθικλμνξοπρστυφχψωΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩ'
r'a-zA-Z]+$'
)
def __init__(self):
self.detector = HeadingDetector()
def extract(self, element_html: str) -> Tuple[str, str, Dict[str, str], str, List[str]]:
"""
提取格式信息
Returns:
clean_text: 纯文本
text_with_placeholders: 只包含内嵌占位符的文本（不含前缀/后缀标签）
placeholder_map: 占位符映射，包含特殊键 "_prefix" 和 "_suffix"
paragraph_type: 段落类型
endnote_anchors: 尾注锚点 ID 列表 (用于补救)
"""
soup = BeautifulSoup(element_html, 'html.parser')
root = list(soup.children)[0] if list(soup.children) else soup
# 获取纯文本和段落类型
clean_text = root.get_text().strip()
clean_text = re.sub(r'\s+', ' ', clean_text)
p_type = self.detector.detect(root, clean_text) if isinstance(root, Tag) else "body"
# 获取内部 HTML
inner_html = root.decode_contents() if isinstance(root, Tag) else str(root)
# 智能提取（分离前缀/后缀）
text_with_ph, local_map = self._smart_extract_v3(inner_html)
# 核心修复：清理 text_with_placeholders 中的换行符和多余空格
# 这一步至关重要，因为 inner_html 中的换行符会导致 LLM Prompt 格式混乱（多行）
# 从而导致 LLM 忽略不在同一行的占位符或内容
if text_with_ph:
text_with_ph = re.sub(r'\s+', ' ', text_with_ph).strip()
# === 验证完整性 ===
# 将 text_with_placeholders 去掉占位符后与 clean_text 比较
stripped_text = self._strip_placeholders(text_with_ph)
stripped_text = re.sub(r'\s+', ' ', stripped_text).strip()
if not self._verify_content_integrity(clean_text, stripped_text):
# 内容不完整，降级到简单模式
from loguru import logger
logger.warning(f"内容验证失败，降级处理: '{clean_text[:30]}...'")
# 降级：不使用前缀/后缀分离，只做简单占位符处理
text_with_ph, local_map = self._fallback_extract(inner_html, clean_text)
# === 识别尾注锚点 ===
# 尾注锚点特征：<span id="aXXX"></span> (短随机ID，通常 3-5 字符)
endnote_anchors = []
for pid, html in local_map.items():
if pid.startswith("_"):
continue # 跳过 _prefix, _suffix
# 匹配空锚点：<span id="aXXX"></span> 或 <a id="aXXX"></a>
if re.match(r'<(span|a)\s+id="[a-zA-Z][a-zA-Z0-9]{2,5}"\s*>\s*</\1>', html):
endnote_anchors.append(pid)
return clean_text, text_with_ph, local_map, p_type, endnote_anchors
def _strip_placeholders(self, text: str) -> str:
"""移除所有占位符（φXφ 和 φ/Xφ 格式）"""
return re.sub(r'φ/?[0-9]+φ', '', text)
def _verify_content_integrity(self, clean_text: str, stripped_text: str) -> bool:
"""
验证内容完整性：比较 clean_text 和 stripped_text
允许一定的容差（空格差异、标点差异）
"""
# 标准化：移除空格和常见标点进行比较
def normalize(s):
s = re.sub(r'\s+', '', s) # 移除空格
s = s.lower() # 忽略大小写
return s
norm_clean = normalize(clean_text)
norm_stripped = normalize(stripped_text)
# 完全匹配
if norm_clean == norm_stripped:
return True
# 检查是否只是缺少少量字符（<5%）
if len(norm_stripped) > 0:
coverage = len(norm_stripped) / len(norm_clean) if norm_clean else 0
if coverage >= 0.95:
return True
return False
def _fallback_extract(self, inner_html: str, clean_text: str) -> Tuple[str, Dict[str, str]]:
"""
降级提取：不分离前缀/后缀，直接返回纯文本
"""
return clean_text, {"_prefix": "", "_suffix": ""}
def _smart_extract_v3(self, inner_html: str) -> Tuple[str, Dict[str, str]]:
"""
智能提取 v3：分离前缀/后缀 + 合并内嵌公式块
核心逻辑：
1. 分离前缀（第一个可翻译文本之前的完整标签）和后缀（最后一个可翻译文本之后的完整标签）
2. 中间部分：检测"公式块"（连续标签+不可翻译文本），合并为单个占位符
3. 只有真正需要翻译的格式标签（如斜体包裹的长文本）才拆分
注意：前缀/后缀只包含不影响文本结构的完整标签，开始标签必须有匹配的结束标签
"""
# 使用正则分割标签和文本
parts = re.split(r'(<[^>]+>)', inner_html)
parts = [p for p in parts if p]
if not parts:
return "", {"_prefix": "", "_suffix": ""}
# 识别每个部分的类型
part_types = [] # 'tag', 'translatable', 'formula', 'whitespace'
for part in parts:
if part.startswith('<'):
part_types.append('tag')
elif not part.strip():
part_types.append('whitespace')
elif self._is_translatable_text(part):
part_types.append('translatable')
else:
part_types.append('formula')
# 找到第一个和最后一个可翻译文本的索引
first_trans_idx = None
last_trans_idx = None
for i, t in enumerate(part_types):
if t == 'translatable':
if first_trans_idx is None:
first_trans_idx = i
last_trans_idx = i
if first_trans_idx is None:
# 没有可翻译文本，全部作为前缀
return "", {"_prefix": inner_html, "_suffix": ""}
# === 安全前缀分离 ===
# 只将自闭合标签和空白作为前缀，一旦遇到开始标签就停止
# 因为开始标签可能包裹着后面的可翻译文本
safe_prefix_end = 0
for i in range(first_trans_idx):
if part_types[i] == 'tag':
tag = parts[i]
# 检查是否是自闭合标签或结束标签（不太可能在开头）
is_self_closing = tag.endswith('/>') or re.match(r'<(br|hr|img|meta|link)\b', tag, re.I)
is_closing = tag.startswith('</')
# 检查是否是空元素（如 <span id="xxx"></span>，紧跟着结束标签）
is_empty_element = False
if not is_self_closing and not is_closing and i + 1 < first_trans_idx:
# 查看下一个标签是否是对应的结束标签
next_idx = i + 1
while next_idx < first_trans_idx and part_types[next_idx] in ('whitespace',):
next_idx += 1
if next_idx < first_trans_idx and part_types[next_idx] == 'tag':
next_tag = parts[next_idx]
if next_tag.startswith('</'):
# 检查标签名是否匹配
open_name = re.match(r'<(\w+)', tag)
close_name = re.match(r'</(\w+)', next_tag)
if open_name and close_name and open_name.group(1) == close_name.group(1):
is_empty_element = True
# 跳过这对空元素
safe_prefix_end = next_idx + 1
continue
if is_self_closing or is_closing:
safe_prefix_end = i + 1
elif is_empty_element:
pass # 已在上面处理
else:
# 遇到普通开始标签，停止
break
elif part_types[i] == 'whitespace':
safe_prefix_end = i + 1
else:
# formula 类型，不应该出现在前缀中
break
# === 安全后缀分离 ===
# 从末尾开始向前，只剥离连续的完整结束标签/自闭合标签或空白
safe_suffix_start = len(parts)
for i in range(len(parts) - 1, last_trans_idx, -1):
if part_types[i] == 'tag':
tag = parts[i]
is_closing = tag.startswith('</')
is_self_closing = tag.endswith('/>') or re.match(r'<(br|hr|img|meta|link)\b', tag, re.I)
if is_closing or is_self_closing:
safe_suffix_start = i
else:
# 遇到开始标签，不能作为独立后缀剥离
break
elif part_types[i] == 'whitespace':
safe_suffix_start = i
else:
# 遇到普通文本或公式，停止剥离
break
# 重新分割
prefix_parts = parts[:safe_prefix_end]
middle_parts = parts[safe_prefix_end:safe_suffix_start]
middle_types = part_types[safe_prefix_end:safe_suffix_start]
suffix_parts = parts[safe_suffix_start:]
# === Drop Cap 检测 ===
# 英文书籍常用首字母放大样式，如 <span class="dropcap">T</span>his
# 检测：前缀末尾是格式化的单个字母，且中间部分第一个文本以小写字母开头
# 处理：移除格式标签，将纯字母加入中间部分
if prefix_parts and middle_parts:
prefix_parts, middle_parts, middle_types = self._handle_drop_cap(
prefix_parts, middle_parts, middle_types
)
# 构建映射
local_map = {}
# 前缀
prefix_html = "".join(prefix_parts)
if prefix_html:
local_map["_prefix"] = prefix_html
# 后缀
suffix_html = "".join(suffix_parts)
if suffix_html:
local_map["_suffix"] = suffix_html
# 中间部分处理：使用配对占位符格式
#
# 策略：
# 1. 连续的 (tag|formula|whitespace) 不包含可翻译文本 → 合并为单个占位符 φ1φ
# 2. 开始标签后接可翻译文本 → 配对格式 φ2φ文本φ/2φ
#
placeholder_counter = 1
result_parts = []
tag_stack = [] # 追踪开放标签 [(id, opening_tag), ...]
i = 0
while i < len(middle_parts):
part = middle_parts[i]
ptype = middle_types[i]
if ptype == 'translatable':
# 可翻译文本，直接保留
result_parts.append(part)
i += 1
elif ptype == 'tag':
# 检查是开始标签还是结束标签
is_closing = part.startswith('</')
if is_closing:
# 结束标签
if tag_stack:
# 匹配最近的开放标签
open_id, open_tag = tag_stack.pop()
local_map[f"/{open_id}"] = part
result_parts.append(f"φ/{open_id}φ")
else:
# 没有匹配的开放标签，作为单独占位符
pid = str(placeholder_counter)
placeholder_counter += 1
local_map[pid] = part
result_parts.append(f"φ{pid}φ")
i += 1
else:
# 开始标签，检查后面是否有可翻译文本
has_translatable_after = False
for j in range(i + 1, len(middle_parts)):
if middle_types[j] == 'translatable':
has_translatable_after = True
break
elif middle_types[j] == 'tag' and middle_parts[j].startswith('</'):
# 遇到结束标签但还没遇到可翻译文本
break
if has_translatable_after:
# 开始配对模式
pid = str(placeholder_counter)
placeholder_counter += 1
local_map[pid] = part
result_parts.append(f"φ{pid}φ")
tag_stack.append((pid, part))
i += 1
else:
# 后面没有可翻译文本，合并为公式块
block_parts = []
while i < len(middle_parts) and middle_types[i] != 'translatable':
block_parts.append(middle_parts[i])
i += 1
if block_parts:
block_html = "".join(block_parts)
pid = str(placeholder_counter)
placeholder_counter += 1
local_map[pid] = block_html
result_parts.append(f"φ{pid}φ")
elif ptype in ('formula', 'whitespace'):
# 简化处理：非可翻译文本直接保留
# 公式检测等复杂逻辑仅在增强模式下启用
result_parts.append(part)
i += 1
else:
i += 1
text_with_ph = "".join(result_parts)
text_with_ph = re.sub(r'\s+', ' ', text_with_ph).strip()
# === 连续占位符合并 ===
# 将 φ1φφ2φ 这样的连续占位符合并为一个
def merge_consecutive_placeholders(text: str, ph_map: dict) -> Tuple[str, dict]:
"""合并连续占位符"""
# 匹配连续的占位符（2个或更多）
pattern = r'(φ/?\d+φ)(φ/?\d+φ)+'
def merge_match(m):
full_match = m.group(0)
# 提取所有占位符ID
pids = re.findall(r'φ(/?\d+)φ', full_match)
if len(pids) <= 1:
return full_match
# 合并对应的 HTML
merged_html = ""
for pid in pids:
if pid in ph_map:
merged_html += ph_map[pid]
del ph_map[pid]
# 创建新的合并占位符
new_pid = pids[0] if pids[0].isdigit() else pids[0][1:] # 使用第一个数字
ph_map[new_pid] = merged_html
return f"φ{new_pid}φ"
merged_text = re.sub(pattern, merge_match, text)
return merged_text, ph_map
text_with_ph, local_map = merge_consecutive_placeholders(text_with_ph, local_map)
return text_with_ph, local_map
def _handle_drop_cap(self, prefix_parts: List[str], middle_parts: List[str],
middle_types: List[str]) -> Tuple[List[str], List[str], List[str]]:
"""
处理 Drop Cap（首字母放大）样式
典型模式：<span class="dropcap">T</span>his is...
问题：前缀会包含 <span>T</span>，但 T 是 This 的一部分
处理：
1. 检测前缀末尾是否为 "格式化的单个大写字母"
2. 检测中间部分首个文本是否以小写字母开头
3. 如果两者组合成一个单词，移除格式，将纯字母加入中间部分
"""
if not prefix_parts or not middle_parts:
return prefix_parts, middle_parts, middle_types
# 提取前缀中的文本内容
prefix_text = ""
last_text_idx = -1
for i, part in enumerate(prefix_parts):
if not part.startswith('<'):
prefix_text = part.strip()
last_text_idx = i
# 检测是否为单个大写字母
if not prefix_text or len(prefix_text) != 1 or not prefix_text.isupper():
return prefix_parts, middle_parts, middle_types
# 检测中间部分第一个可翻译文本
first_middle_text = ""
first_middle_idx = -1
for i, (part, ptype) in enumerate(zip(middle_parts, middle_types)):
if ptype == 'translatable':
first_middle_text = part.strip()
first_middle_idx = i
break
if not first_middle_text:
return prefix_parts, middle_parts, middle_types
# Drop Cap 检测条件（满足任一即可）:
# 1. 后续文本以小写字母开头: T + his = This
# 2. 后续文本以大写字母开头且紧连（无空格）: I + N OCTOBER = IN OCTOBER
is_drop_cap = False
first_char = first_middle_text[0] if first_middle_text else ''
if first_char.islower():
# 条件1: This 模式
is_drop_cap = True
elif first_char.isupper():
# 条件2: IN OCTOBER 模式 - 检查是否紧连（第一个字母后不应有空格）
# 原始 HTML 中 </span>N 表示 N 紧跟在 I 后面
is_drop_cap = True
if not is_drop_cap:
return prefix_parts, middle_parts, middle_types
# 组合检测：大写字母 + 后续文本的第一个单词
combined = prefix_text + first_middle_text.split()[0] if first_middle_text else ""
# 验证：组合后是否为合理的英文单词/大写序列（至少 2 个字母）
if len(combined) >= 2 and combined.isalpha():
# 确认是 Drop Cap，移除格式
# 从前缀中移除这个字母和其包裹的格式标签
new_prefix = []
skip_until_close = False
found_letter = False
for part in prefix_parts:
if part.startswith('<') and not part.startswith('</'):
# 开始标签，准备跳过
skip_until_close = True
elif part.startswith('</'):
# 结束标签
if skip_until_close:
skip_until_close = False
continue
new_prefix.append(part)
elif part.strip() == prefix_text:
# 这是那个大写字母，跳过
found_letter = True
continue
else:
if not skip_until_close:
new_prefix.append(part)
if found_letter:
# 将大写字母加到中间部分第一个可翻译文本的开头
middle_parts = middle_parts.copy()
middle_parts[first_middle_idx] = prefix_text + middle_parts[first_middle_idx]
prefix_parts = new_prefix
return prefix_parts, middle_parts, middle_types
def _is_translatable_text(self, text: str) -> bool:
"""
判断文本是否需要翻译（包含可翻译的单词）
条件（满足任一即可）：
1. 包含 3 个及以上连续字母（如 "and", "Art", "War"）
2. 包含空格分隔的多个单词（如 "and Sun Tzu's"）
3. 包含数字（如章节号 "10", "12"）
"""
text = text.strip()
if not text:
return False
# 条件1: 3 个及以上连续字母
if re.search(r'[a-zA-Z]{3,}', text):
return True
# 条件2: 包含空格的多单词文本（如 "a of"）
if ' ' in text and re.search(r'[a-zA-Z]', text):
return True
# 条件3: 包含数字（章节号等）
if re.search(r'\d', text):
return True
return False
def _is_formula_element(self, element: Tag, text_content: str) -> bool:
"""判断元素是否为公式元素（应整体保留不翻译）"""
if not text_content:
return True
if len(text_content) <= 3:
return True
if re.search(r'[a-zA-Z]{4,}', text_content):
return False
return bool(self.FORMULA_CHARS.match(text_content))
def _get_opening_tag(self, element: Tag) -> str:
"""获取元素的开始标签（含属性）"""
attrs_str = ""
for key, value in element.attrs.items():
if isinstance(value, list):
value = " ".join(value)
attrs_str += f' {key}="{value}"'
return f"<{element.name}{attrs_str}>"
def reset(self):
"""兼容旧接口"""
pass
def _is_pure_punctuation(self, text: str) -> bool:
"""判断文本是否仅包含标点符号和空格（不应该变成占位符）"""
# 常见标点符号集合（中英文混合）
punctuation_chars = ' ,.:;!?，。：；！？、""\'\'「」【】（）()[]{}—-–…·'
return all(c in punctuation_chars for c in text)
archive/v0.09/src/format_restorer.py
+99
View File
@@ -0,0 +1,99 @@
"""
格式恢复模块 (优化版 v4)
负责：
1. 解析译文中的配对占位符 (φ1φ...φ/1φ)
2. 还原前缀和后缀标签（_prefix, _suffix）
3. 从映射表中查找对应的 HTML 片段并替换
"""
import re
from typing import Dict, Tuple, List, Optional
from loguru import logger
class FormatRestorer:
"""
格式恢复器 (优化版 v4)
支持：
- 配对占位符 φ1φ...φ/1φ
- 前缀/后缀自动回填 (_prefix, _suffix)
"""
# 占位符正则: φ1φ, φ/1φ, φ12φ, φ/12φ (支持配对格式)
PLACEHOLDER_REGEX = re.compile(r'φ(/?\d+)φ')
def restore(self, text_with_placeholders: str, placeholder_map: Dict[str, str]) -> Tuple[str, bool]:
"""
将带占位符的文本还原为 HTML
自动处理 _prefix 和 _suffix 键，以及配对占位符 φ1φ...φ/1φ
Returns:
(html_string, success): 还原后的 HTML 和是否完全成功的标志
"""
if not placeholder_map:
return text_with_placeholders or "", True
if not text_with_placeholders:
# 没有文本，只有前缀/后缀（如空元素）
prefix = placeholder_map.get("_prefix", "")
suffix = placeholder_map.get("_suffix", "")
return prefix + suffix, True
# 提取前缀和后缀
prefix = placeholder_map.get("_prefix", "")
suffix = placeholder_map.get("_suffix", "")
# 创建只包含占位符键的映射（排除 _prefix, _suffix）
inner_map = {k: v for k, v in placeholder_map.items() if not k.startswith("_")}
# 校验占位符
found_ids = set(self.PLACEHOLDER_REGEX.findall(text_with_placeholders))
expected_ids = set(inner_map.keys())
success = True
missing_ids = expected_ids - found_ids
if missing_ids:
logger.warning(f"格式还原警告: 丢失占位符 {missing_ids}")
success = False
unknown_ids = found_ids - expected_ids
if unknown_ids:
# 过滤掉冗余的闭合标签（例如 map里有 "1"，但 LLM 输出了 "φ/1φ"）
real_unknowns = set()
for pid in unknown_ids:
# 如果是 /N，且 N 在 map 中，则认为是安全的冗余闭合
if pid.startswith('/') and pid[1:] in expected_ids:
continue
real_unknowns.add(pid)
if real_unknowns:
logger.warning(f"格式还原警告: 发现未知占位符 {real_unknowns}")
success = False
# 替换占位符
def replace_match(match):
pid = match.group(1) # 可能是 "1" 或 "/1"
if pid in inner_map:
return inner_map[pid]
else:
return "" # 删除未知占位符
try:
# 还原占位符
restored_inner = self.PLACEHOLDER_REGEX.sub(replace_match, text_with_placeholders)
# 添加前缀和后缀
restored_html = prefix + restored_inner + suffix
return restored_html, success
except Exception as e:
logger.error(f"格式还原失败: {e}")
return prefix + self._strip_placeholders(text_with_placeholders) + suffix, False
def _strip_placeholders(self, text: str) -> str:
"""移除所有 φ...φ 占位符"""
return self.PLACEHOLDER_REGEX.sub("", text)
archive/v0.09/src/llm_client.py
+290
View File
@@ -0,0 +1,290 @@
"""
LLM Client Module - Generic OpenAI Compatible
Features:
1. Fully configurable via config.json (base_url, headers).
2. Mode-aware prompt building (bilingual vs chinese).
3. Format repair capability for chinese mode.
"""
import asyncio
import json
import re
from openai import AsyncOpenAI
from typing import List, Dict, Optional, Any
from loguru import logger
import time
from tenacity import retry, stop_after_attempt, wait_exponential
from .manifest_manager import ManifestItem
class RateLimiter:
"""Rate limiter for concurrency and RPM."""
def __init__(self, requests_per_minute: int, concurrent_requests: int):
self.semaphore = asyncio.Semaphore(concurrent_requests)
self.min_interval = 60.0 / requests_per_minute if requests_per_minute > 0 else 0
self.last_request_time = 0
self._lock = asyncio.Lock()
async def acquire(self):
await self.semaphore.acquire()
async with self._lock:
current_time = time.time()
wait_time = self.min_interval - (current_time - self.last_request_time)
if wait_time > 0:
await asyncio.sleep(wait_time)
self.last_request_time = time.time()
def release(self):
self.semaphore.release()
class LLMClient:
"""Generic OpenAI-compatible API Client."""
def __init__(self, config: Dict):
self.config = config
llm_config = config["llm"]
api_key = llm_config.get("api_key")
base_url = llm_config.get("base_url")
extra_headers = llm_config.get("extra_headers", {})
if not api_key:
raise ValueError("API Key is missing in config")
self.client = AsyncOpenAI(
base_url=base_url,
api_key=api_key,
default_headers=extra_headers
)
self.models = llm_config.get("models", {"fast": "gpt-3.5-turbo", "smart": "gpt-4"})
self.rate_limiter = RateLimiter(
llm_config["rate_limits"]["requests_per_minute"],
llm_config["rate_limits"]["concurrent_requests"]
)
self.prompts = self._load_prompts()
def _load_prompts(self) -> Dict:
try:
with open("config/prompts.json", "r", encoding="utf-8") as f:
return json.load(f)
except Exception as e:
logger.error(f"严重错误：无法加载 config/prompts.json: {e}")
raise # 必须抛出异常，否则 System Prompt 会降级导致占位符指令丢失
async def translate_chunk(self, items: List[ManifestItem], glossary: Dict = None,
instruction: str = None, model_type: str = "fast",
mode: str = "bilingual") -> Dict[str, str]:
"""
Translate a chunk of items.
Args:
items: List of ManifestItem to translate
glossary: Term dictionary
instruction: Style guide
model_type: "fast" or "smart"
mode: "bilingual" or "chinese"
"""
if not items: return {}
model = self.models.get(model_type, self.models.get("fast"))
prompt = self._build_prompt(items, mode)
try:
# Build System Prompt
base_sys_prompt = self.prompts.get("translation", {}).get("system", "You are a professional translator.")
# 中文模式：Prompt 已在 config/prompts.json 中配置，无需额外硬编码
if mode == "chinese":
pass
if instruction:
base_sys_prompt += f"\n\nBook Style Guide:\n{instruction}"
if glossary:
glossary_text = "\n".join([f"{k} -> {v}" for k, v in glossary.items()])
base_sys_prompt += f"\n\nTerminology:\n{glossary_text}"
# Strict formatting instructions
base_sys_prompt += "\n\nRequirements:\n1. Each line MUST start with ID (p_xxxxx).\n2. DO NOT modify IDs.\n3. Return only translations."
# DEBUG: 打印发送给 LLM 的完整内容
logger.debug(f"=== LLM REQUEST DEBUG ===")
logger.debug(f"System Prompt:\n{base_sys_prompt[:500]}...")
logger.debug(f"User Prompt (first 1000 chars):\n{prompt[:1000]}")
logger.debug(f"=========================")
raw_response = await self._make_request(model, base_sys_prompt, prompt)
# DEBUG: 打印 LLM 返回的完整内容
logger.debug(f"=== LLM RESPONSE DEBUG ===")
logger.debug(f"Raw Response (first 1500 chars):\n{raw_response[:1500] if raw_response else 'EMPTY'}")
logger.debug(f"==========================")
if not raw_response:
return {item.global_id: f"[Error - Empty Response]" for item in items}
results = self._simple_parse(raw_response, items, mode)
return results
except Exception as e:
logger.error(f"Translation failed ({model}): {e}")
return {item.global_id: f"[Error - {str(e)}]" for item in items}
async def repair_format(self, original_text: str, broken_translation: str, missing_ids: set = None) -> str:
"""
修复翻译格式：将缺失的占位符正确插入到译文中。
Args:
original_text: 原文（带占位符）
broken_translation: 有占位符问题的译文
missing_ids: 缺失的占位符 ID 集合（可选，用于提示）
"""
model = self.models.get("fast")
system_prompt = """你是格式修复助手。你的任务是将缺失的占位符插入到译文中。
注意：
1. 不要重新翻译，保持中文译文内容完全不变
2. 只需要在正确位置插入缺失的占位符
3. 占位符格式：φ数字φ（如 φ1φ, φ/1φ）
4. 只输出修复后的译文，不要任何解释"""
missing_hint = ""
if missing_ids:
missing_list = ", ".join([f"φ{pid}φ" for pid in missing_ids])
missing_hint = f"\n缺失的占位符: {missing_list}"
user_prompt = f"""原文（带占位符）:
{original_text}
当前译文（占位符有误）:
{broken_translation}
{missing_hint}
请修复译文，在正确位置插入缺失的占位符。只输出修复后的译文："""
try:
return await self._make_request(model, system_prompt, user_prompt)
except Exception as e:
logger.error(f"Format repair failed: {e}")
return broken_translation
async def raw_chat_completion(self, system_prompt: str, user_prompt: str, model_type: str = "smart") -> str:
"""Generic chat completion (for Profiler)."""
model = self.models.get(model_type, self.models.get("smart"))
return await self._make_request(model, system_prompt, user_prompt)
def _build_prompt(self, items: List[ManifestItem], mode: str = "bilingual") -> str:
"""构建翻译提示词"""
lines = []
for item in items:
if mode == "chinese":
# 中文模式：使用带占位符的文本和段落类型
twp = item.text_with_placeholders
# DEBUG: 打印关键信息
logger.debug(f"BUILD_PROMPT {item.global_id}: twp='{twp[:50] if twp else 'EMPTY'}...', has_φ={'φ' in twp if twp else False}")
text = twp if twp else item.clean_text
# 防御性修复：强制清理换行符，兼容旧的脏 Manifest 数据
text = re.sub(r'\s+', ' ', text).strip()
p_type = getattr(item, 'paragraph_type', 'body').upper()
lines.append(f"{item.global_id} [{p_type}] {text}")
else:
# 双语模式：使用纯文本
lines.append(f"{item.global_id} {item.clean_text}")
return "\n".join(lines)
def _simple_parse(self, response: str, items: List[ManifestItem], mode: str = "bilingual") -> Dict[str, str]:
"""
解析 LLM 响应 - 位置切分版
策略：
1. 识别响应中所有出现的 p_xxxxx 及其位置
2. 按位置顺序将响应切分成每一段，消除对输入顺序的依赖
"""
results = {}
valid_ids = {item.global_id for item in items}
# 1. 查找所有可能的 ID 位置
# 模式匹配 p_ 后面跟着 5 位数字
matches = list(re.finditer(r'p_\d{5}', response))
if not matches:
# Fallback: 如果没有匹配到任何 ID，尝试按行扫描
for line in response.split("\n"):
line = line.strip()
for it in items:
if line.startswith(it.global_id):
content = line[len(it.global_id):].strip().lstrip(":： ")
if content: results[it.global_id] = content
return results
# 2. 按查找到的 ID 位置进行切割
for i, match in enumerate(matches):
current_id = match.group()
if current_id not in valid_ids:
continue
# 这一段内容的起始是当前 ID 之后，结束是下一个匹配的 ID 之前
start_pos = match.end()
end_pos = matches[i+1].start() if i + 1 < len(matches) else len(response)
content = response[start_pos:end_pos].strip()
# 3. 清理内容
content = content.lstrip(":： \t")
if mode == "chinese":
# 移除 [BODY] 等类型标记
content = re.sub(r'^\[[A-Z]+\]\s*', '', content)
content = content.strip()
if content:
results[current_id] = content
# 验证解析结果
parsed_count = len(results)
expected_count = len(items)
if parsed_count < expected_count:
missing_ids = [item.global_id for item in items if item.global_id not in results]
logger.warning(f"LLM 响应解析不完整: {parsed_count}/{expected_count} (缺失: {missing_ids[:3]}...)")
return results
@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
async def _make_request(self, model: str, system_prompt: str, user_prompt: str) -> str:
await self.rate_limiter.acquire()
try:
resp = await self.client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_prompt}
],
temperature=self.config['translation'].get('temperature', 0.2),
max_tokens=8000
)
return resp.choices[0].message.content.strip()
finally:
self.rate_limiter.release()
async def close(self):
await self.client.close()
# Alias for backward compatibility
OpenRouterClient = LLMClient
archive/v0.09/src/manifest_manager.py
+189
View File
@@ -0,0 +1,189 @@
"""
Manifest 管理器模块 (Manifest Manager Module)
该模块是系统的单一真理源 (SSOT)。
它记录了每一段文本的原始状态、清洗后的文本、哈希值以及翻译状态。
所有对翻译流程的操作（提取、翻译、回填）都必须通过修改此 Manifest 进行。
"""
import json
import os
import hashlib
from typing import List, Dict, Optional, Any
from pathlib import Path
from loguru import logger
from dataclasses import dataclass, asdict, field
@dataclass
class ManifestItem:
"""代表一个翻译单元（通常是一个段落）"""
global_id: str
source_file: str
original_html: str
clean_text: str
text_hash: str
tag: str
tag_attrs: Dict[str, Any] = field(default_factory=dict) # 外层标签的属性 (class, style...)
translation: Optional[str] = None
status: str = "pending" # pending, translated, ignored, failed
error_msg: Optional[str] = None
model_used: Optional[str] = None # 记录使用的模型
quality_score: Optional[int] = None # 记录质量评分
# === 中文模式专用字段 ===
text_with_placeholders: str = "" # 带占位符的文本
placeholder_map: Dict[str, str] = field(default_factory=dict) # 占位符映射表 {id: html_string}
paragraph_type: str = "body" # 段落类型：chapter/section/subsection/epigraph/body
translation_with_placeholders: str = "" # 带占位符的译文
translation_with_original_html: str = "" # 还原后的最终 HTML (中文模式)
endnote_anchors: List[str] = field(default_factory=list) # 尾注锚点 ID 列表 (用于补救)
metadata: Dict[str, Any] = field(default_factory=dict)
def to_dict(self):
return asdict(self)
class ManifestManager:
"""
负责 Manifest 的生命周期管理。
"""
def __init__(self, manifest_path: str):
self.manifest_path = Path(manifest_path)
self.data: Dict[str, Any] = {
"book_id": "",
"metadata": {},
"chapter_range": {
"start_title": None,
"end_title": None,
"included_files": []
},
"items": []
}
self._items_by_id: Dict[str, ManifestItem] = {}
def load(self) -> bool:
"""从文件加载 Manifest。如果文件不存在则返回 False。"""
if self.manifest_path.exists():
try:
with open(self.manifest_path, 'r', encoding='utf-8') as f:
self.data = json.load(f)
# 重建对象映射
self._items_by_id = {
item['global_id']: ManifestItem(**item)
for item in self.data["items"]
}
logger.info(f"成功从 {self.manifest_path} 加载 Manifest, 包含 {len(self._items_by_id)} 个项目")
return True
except Exception as e:
logger.error(f"加载 Manifest 失败: {e}")
return False
return False
def save(self):
"""将当前状态保存到 Manifest 文件。"""
# 确保目录存在
self.manifest_path.parent.mkdir(parents=True, exist_ok=True)
# 同步 items 到 data 字典
self.data["items"] = [item.to_dict() for item in self._items_by_id.values()]
with open(self.manifest_path, 'w', encoding='utf-8') as f:
json.dump(self.data, f, ensure_ascii=False, indent=2)
# logger.debug(f"Manifest 已保存到 {self.manifest_path}")
def init_manifest(self, book_id: str, metadata: Dict, chapter_range: Dict = None):
"""初始化一个新的 Manifest。"""
self.data = {
"book_id": book_id,
"metadata": metadata,
"chapter_range": chapter_range or {
"start_title": None,
"end_title": None,
"included_files": []
},
"items": []
}
self._items_by_id = {}
self.save()
def get_chapter_range(self) -> Dict:
"""获取记录的章节范围"""
return self.data.get("chapter_range", {})
def add_item(self, source_file: str, original_html: str, clean_text: str, tag: str, metadata: Dict = None) -> ManifestItem:
"""添加一个新的翻译项并分配 ID。"""
# 生成全局 ID
new_index = len(self._items_by_id) + 1
global_id = f"p_{new_index:05d}"
# 生成内容哈希 (用于排重和缓存)
text_hash = hashlib.sha256(clean_text.encode('utf-8')).hexdigest()
item = ManifestItem(
global_id=global_id,
source_file=source_file,
original_html=original_html,
clean_text=clean_text,
text_hash=text_hash,
tag=tag,
metadata=metadata or {}
)
self._items_by_id[global_id] = item
return item
def get_items(self, status: str = None, file_name: str = None) -> List[ManifestItem]:
"""按状态或文件名查询项目。"""
items = list(self._items_by_id.values())
if status:
items = [i for i in items if i.status == status]
if file_name:
items = [i for i in items if i.source_file == file_name]
# 必须按 ID 顺序返回以保证分块正确
return sorted(items, key=lambda x: x.global_id)
def update_item(self, global_id: str, translation: str, status: str = "translated",
error: str = None, model: str = None, score: int = None,
translation_with_placeholders: str = None,
translation_with_original_html: str = None):
"""更新翻译结果。"""
if global_id in self._items_by_id:
item = self._items_by_id[global_id]
if translation is not None:
item.translation = translation
item.status = status
if error:
item.error_msg = error
if model:
item.model_used = model
if score is not None:
item.quality_score = score
# 中文模式专用字段
if translation_with_placeholders is not None:
item.translation_with_placeholders = translation_with_placeholders
if translation_with_original_html is not None:
item.translation_with_original_html = translation_with_original_html
else:
logger.warning(f"尝试更新不存在的 ID: {global_id}")
@property
def stats(self) -> Dict:
"""获取翻译进度统计。"""
total = len(self._items_by_id)
if total == 0: return {"progress": "0%"}
translated = sum(1 for i in self._items_by_id.values() if i.status == "translated")
ignored = sum(1 for i in self._items_by_id.values() if i.status == "ignored")
failed = sum(1 for i in self._items_by_id.values() if i.status == "failed")
return {
"total": total,
"translated": translated,
"ignored": ignored,
"failed": failed,
"pending": total - translated - ignored - failed,
"progress_percent": round((translated + ignored) / total * 100, 1)
}
archive/v0.09/src/quality_manager.py
+87
View File
@@ -0,0 +1,87 @@
"""
Quality Manager Module
Responsible for evaluating translation quality and deciding on re-translation.
"""
import json
import random
from typing import List, Dict, Any, Tuple
from loguru import logger
from .manifest_manager import ManifestItem
from .llm_client import LLMClient
class QualityManager:
def __init__(self, config: Dict, llm_client: LLMClient):
self.config = config
self.llm_client = llm_client
self.qc_config = config['translation'].get('quality_control', {})
self.pass_score = self.qc_config.get('pass_score', 7)
self.sample_size = self.qc_config.get('sample_size', 2)
async def evaluate_chunk(self, chunk: List[ManifestItem]) -> Tuple[bool, int, str]:
"""
Evaluate a chunk of translations.
Returns:
(passed: bool, average_score: int, reason: str)
"""
if not self.qc_config.get('enabled', False):
return True, 10, "QC Disabled"
# 1. Sample items
# Filter for items that actually have content and translations
valid_items = [item for item in chunk if item.translation and len(item.clean_text) > 20]
if not valid_items:
return True, 10, "No valid items to sample"
sample_items = random.sample(valid_items, min(len(valid_items), self.sample_size))
# 2. Build Prompt
prompt = self._build_evaluation_prompt(sample_items)
# 3. Call LLM (Smart)
try:
response = await self.llm_client.raw_chat_completion(
system_prompt="You are a professional translation editor.",
user_prompt=prompt,
model_type="smart"
)
# 4. Parse JSON
# Clean potential markdown
json_str = response.strip()
if "```json" in json_str:
json_str = json_str.split("```json")[1].split("```")[0].strip()
elif "```" in json_str:
json_str = json_str.split("```")[1].split("```")[0].strip()
result = json.loads(json_str)
score = result.get('score', 0)
reason = result.get('reason', 'No reason provided')
passed = score >= self.pass_score
return passed, score, reason
except Exception as e:
logger.error(f"QC evaluation failed: {e}")
# If QC fails, we default to PASS to avoid blocking progress, but log it
return True, 0, f"QC Error: {e}"
def _build_evaluation_prompt(self, items: List[ManifestItem]) -> str:
content = ""
for i, item in enumerate(items, 1):
content += f"Item {i}:\nOriginal: {item.clean_text}\nTranslation: {item.translation}\n\n"
return f"""Please evaluate the following translations (English to Chinese).
Focus on accuracy, fluency, and terminology consistency.
Items to evaluate:
{content}
Return a JSON object with:
- \"score\": An integer from 1 to 10 (10 being perfect).
- \"reason\": A brief explanation of the score.
JSON Output:"""
archive/v0.09/src/text_processor.py
+117
View File
@@ -0,0 +1,117 @@
"""
文本处理器模块 (Text Processor Module) - Manifest 驱动版 (集成 V2)
该模块专注于 HTML 文档的遍历和段落提取。
已集成 FineGrainedExtractor，实现稳健的结构提取和格式保护。
"""
import re
from bs4 import BeautifulSoup
from typing import List, Dict, Any
from loguru import logger
from .manifest_manager import ManifestManager
from .fine_grained_extractor import FineGrainedExtractor
class TextProcessor:
"""
负责从 HTML 中识别有效段落并注册到 Manifest。
委托 FineGrainedExtractor 进行具体的提取工作。
"""
def __init__(self, config: Dict):
"""
Args:
config (Dict): 全局配置。
"""
self.config = config
self.chunk_size = config['translation'].get('chunk_size', 5000)
# 不再持有状态，每次调用实例化 Extractor 或复用
def extract_to_manifest(self, html_content: str, source_file: str, manifest: ManifestManager, mode: str = "bilingual"):
"""
解析 HTML 内容，并将识别出的段落注册到 Manifest 中。
Args:
html_content (str): HTML 源码。
source_file (str): 来源文件名。
manifest (ManifestManager): 清单管理器实例。
mode (str): 翻译模式 (保留参数)
"""
try:
# 实例化细粒度提取器 (集成格式保护)
# 是否翻译目录取决于文件名判断，这里交给 Extractor 内部逻辑
# 但 Extractor 构造函数需要参数，默认 False
translate_toc = self.config['translation'].get('translate_toc', False)
extractor = FineGrainedExtractor(translate_toc=translate_toc)
items = extractor.extract(html_content, source_file)
count = 0
for item in items:
# 只注册需要翻译的项
if not item['should_translate']:
continue
clean_text = item.get('text', '')
text_with_ph = item.get('text_with_ph', clean_text)
placeholder_map = item.get('placeholder_map')
# 注册到 Manifest
# original_html 用于记录，但实际回填依靠 FineGrainedExtractor 复原
manifest_item = manifest.add_item(
source_file=source_file,
original_html=str(item['element']),
clean_text=clean_text,
tag=item['tag'],
metadata={"status": "pending"}
)
# 显式设置格式保护字段
manifest_item.text_with_placeholders = text_with_ph
manifest_item.placeholder_map = placeholder_map
manifest_item.endnote_anchors = item.get('endnote_anchors', [])
# 记录段落类型 (从 FormatExtractor 获得的 p_type，目前 FineGrained 没返回，可以改进)
# FineGrained 可以把 FormatExtractor 返回的 p_type 也带出来
# 暂且设为 body，或根据 tag 判断
p_type = "header" if item['tag'].startswith('h') else "body"
manifest_item.paragraph_type = p_type
count += 1
logger.info(f"提取完成 {source_file}: 注册 {count} 个待翻译项")
except Exception as e:
logger.error(f"从 {source_file} 提取段落失败: {e}", exc_info=True)
def create_chunks_from_manifest(self, manifest: ManifestManager, mode: str = "bilingual") -> List[List[Any]]:
"""
从 Manifest 中筛选待翻译项目并分块。
(保留原有逻辑)
"""
pending_items = manifest.get_items(status="pending")
if not pending_items:
return []
chunks = []
current_chunk = []
current_size = 0
for item in pending_items:
# 优先使用带占位符的文本长度计算
text_len = len(item.text_with_placeholders) if item.text_with_placeholders else len(item.clean_text)
if current_size + text_len > self.chunk_size and current_chunk:
chunks.append(current_chunk)
current_chunk = []
current_size = 0
current_chunk.append(item)
current_size += text_len
if current_chunk:
chunks.append(current_chunk)
logger.info(f"分块完成: 共有 {len(pending_items)} 个待翻译项，分为 {len(chunks)} 个块")
return chunks
archive/v0.09/src/toc_parser.py
+411
View File
@@ -0,0 +1,411 @@
"""
TOC 解析器模块 (TOC Parser Module)
该模块负责从 EPUB 文件中提取目录结构，并提供章节范围选择功能。
支持嵌套的目录结构（如 Part -> Chapter）。
"""
from dataclasses import dataclass
from typing import List, Set, Optional, Tuple
from ebooklib import epub
from loguru import logger
from urllib.parse import urlparse
@dataclass
class TOCItem:
"""代表一个目录项"""
index: int # 序号 (1-based)
title: str # 章节标题
href: str # 文件路径 (如 index_split_003.html 或 e9781668053393/xhtml/ch01.xhtml)
file_name: str # 纯文件名 (不含锚点)
level: int # 层级 (0=顶级, 1=子章节, 2=子子章节)
skip_reason: str = "" # 跳过原因: 'front', 'back', 或空字符串表示不跳过
# 前置部分 - 通常不需要翻译
FRONT_MATTER_PATTERNS = [
'cover', 'title page', 'copyright', 'contents',
'table of contents', 'half title', 'halftitle',
'how to use this ebook', 'copyright page'
]
# 后置部分 - 通常不需要翻译
# 注意：使用精确匹配避免误伤，如 "notes" 会匹配 "Technical Notes"
BACK_MATTER_PATTERNS = [
'endnotes', 'footnotes', 'bibliography',
'references', 'index', 'about the author',
'about the publisher', 'credits', 'appendix',
'glossary', 'also by', 'resources for'
]
# 需要精确匹配的模式（标题必须完全等于这些值）
BACK_MATTER_EXACT = [
'notes' # 精确匹配，避免匹配 "Technical Notes"
]
class TOCParser:
"""
TOC 解析器
从 EPUB 提取扁平化的目录列表，并支持章节范围选择。
"""
def __init__(self, book: epub.EpubBook):
self.book = book
self._toc_items: List[TOCItem] = []
self._parse_toc()
self._classify_all_chapters()
def _parse_toc(self):
"""解析 book.toc，构建扁平化的目录列表"""
self._toc_items = []
index = [0] # 使用列表以便在嵌套函数中修改
def traverse(toc_list, level=0):
for item in toc_list:
if isinstance(item, tuple):
# 嵌套结构: (section, children)
section, children = item
index[0] += 1
href = section.href if hasattr(section, 'href') else ""
file_name = self._extract_file_name(href)
self._toc_items.append(TOCItem(
index=index[0],
title=section.title if hasattr(section, 'title') else str(section),
href=href,
file_name=file_name,
level=level
))
# 递归处理子节点
traverse(children, level + 1)
else:
# 叶子节点
index[0] += 1
href = item.href if hasattr(item, 'href') else ""
file_name = self._extract_file_name(href)
self._toc_items.append(TOCItem(
index=index[0],
title=item.title if hasattr(item, 'title') else str(item),
href=href,
file_name=file_name,
level=level
))
traverse(self.book.toc)
logger.debug(f"解析 TOC 完成，共 {len(self._toc_items)} 个章节")
def _classify_chapter(self, title: str) -> str:
"""
分类单个章节
Returns:
'front': 前置部分（跳过）
'back': 后置部分（跳过）
'': 正文内容（保留）
"""
title_lower = title.lower().strip()
# 检查前置部分（模糊匹配）
for pattern in FRONT_MATTER_PATTERNS:
if pattern in title_lower or title_lower == pattern:
return 'front'
# 检查后置部分（模糊匹配）
for pattern in BACK_MATTER_PATTERNS:
if pattern in title_lower or title_lower == pattern:
return 'back'
# 检查后置部分（精确匹配）
for pattern in BACK_MATTER_EXACT:
if title_lower == pattern:
return 'back'
return ''
def _classify_all_chapters(self):
"""对所有章节进行分类"""
for item in self._toc_items:
item.skip_reason = self._classify_chapter(item.title)
# 统计跳过数量
front_count = sum(1 for i in self._toc_items if i.skip_reason == 'front')
back_count = sum(1 for i in self._toc_items if i.skip_reason == 'back')
if front_count or back_count:
logger.debug(f"章节分类: 跳过前置 {front_count} 个，跳过后置 {back_count} 个")
def get_skip_files(self) -> Set[str]:
"""获取应该跳过的文件集合"""
return {item.file_name for item in self._toc_items
if item.skip_reason and item.file_name}
def get_content_files(self) -> Set[str]:
"""获取正文内容的文件集合（排除前置和后置）"""
return {item.file_name for item in self._toc_items
if not item.skip_reason and item.file_name}
def get_spine_files(self) -> List[str]:
"""获取 Spine 中的所有文件（按阅读顺序）"""
spine_files = []
for item_tuple in self.book.spine:
item_id = item_tuple[0]
item = self.book.get_item_with_id(item_id)
if item:
spine_files.append(item.get_name())
return spine_files
def get_content_files_from_spine(self) -> Set[str]:
"""
基于 Spine 获取正文内容文件（排除前置和后置）
核心逻辑：
1. 找到第一个正文章节在 Spine 中的位置
2. 找到最后一个正文章节在 Spine 中的位置
3. 返回这个范围内的所有 Spine 文件
"""
spine_files = self.get_spine_files()
if not spine_files:
return self.get_content_files() # 降级到 TOC 文件
# 获取正文和跳过的 TOC 文件
content_toc_files = self.get_content_files()
skip_toc_files = self.get_skip_files()
if not content_toc_files:
return set(spine_files) # 没有分类信息，返回所有
# 在 Spine 中找到正文内容的边界
first_content_idx = None
last_content_idx = None
for idx, spine_file in enumerate(spine_files):
if spine_file in content_toc_files:
if first_content_idx is None:
first_content_idx = idx
last_content_idx = idx
if first_content_idx is None:
return self.get_content_files() # 降级
# 收集边界内的所有 Spine 文件
result = set()
for idx in range(first_content_idx, last_content_idx + 1):
spine_file = spine_files[idx]
# 排除明确标记为跳过的文件
if spine_file not in skip_toc_files:
result.add(spine_file)
logger.debug(f"Spine 正文范围: {first_content_idx+1} ~ {last_content_idx+1}，共 {len(result)} 个文件")
return result
def get_spine_range(self, start_title: str = None, end_title: str = None) -> Tuple[Set[str], List[TOCItem]]:
"""
基于 Spine 和 TOC 边界获取文件范围
与 get_file_range 的区别：
- get_file_range: 只返回 TOC 中列出的文件
- get_spine_range: 返回 TOC 边界之间的所有 Spine 文件
"""
spine_files = self.get_spine_files()
# 确定 TOC 边界
start_item = self.find_by_title(start_title) if start_title else None
end_item = self.find_by_title(end_title) if end_title else None
start_idx = start_item.index if start_item else 1
end_idx = end_item.index if end_item else len(self._toc_items)
if start_idx > end_idx:
start_idx, end_idx = end_idx, start_idx
# 获取选中的 TOC 项
selected_items = [i for i in self._toc_items if start_idx <= i.index <= end_idx]
selected_toc_files = {i.file_name for i in selected_items if i.file_name}
# 在 Spine 中找到这些文件的边界
spine_start = None
spine_end = None
for idx, spine_file in enumerate(spine_files):
if spine_file in selected_toc_files:
if spine_start is None:
spine_start = idx
spine_end = idx
if spine_start is None:
# 降级到 TOC 文件
logger.warning("无法在 Spine 中定位章节边界，使用 TOC 文件")
return selected_toc_files, selected_items
# 扩展到下一个 TOC 章节之前
# 找到 end_idx 之后的下一个 TOC 章节在 Spine 中的位置
next_toc_file = None
if end_idx < len(self._toc_items):
next_toc_file = self._toc_items[end_idx].file_name # end_idx 是 1-based
if next_toc_file:
for idx, spine_file in enumerate(spine_files):
if spine_file == next_toc_file:
spine_end = idx - 1 # 到下一章之前
break
# 收集 Spine 范围内的所有文件
result = set()
for idx in range(spine_start, spine_end + 1):
if idx < len(spine_files):
result.add(spine_files[idx])
logger.info(f"Spine 范围: #{spine_start+1} ~ #{spine_end+1}，共 {len(result)} 个文件（TOC: {len(selected_toc_files)} 个）")
return result, selected_items
def _extract_file_name(self, href: str) -> str:
"""从 href 中提取纯文件名（去除锚点和路径前缀）"""
if not href:
return ""
# 去除锚点 (#section1)
path = href.split('#')[0]
# 返回完整路径（可能包含子目录）
return path
@property
def items(self) -> List[TOCItem]:
"""获取所有目录项"""
return self._toc_items
def find_by_title(self, title: str, fuzzy: bool = True) -> Optional[TOCItem]:
"""
根据标题查找目录项
Args:
title: 章节标题
fuzzy: 是否模糊匹配（包含即可）
Returns:
匹配的 TOCItem 或 None
"""
title_lower = title.lower().strip()
for item in self._toc_items:
item_title_lower = item.title.lower().strip()
if fuzzy:
# 模糊匹配：互相包含
if title_lower in item_title_lower or item_title_lower in title_lower:
return item
else:
# 精确匹配
if item_title_lower == title_lower:
return item
return None
def find_by_index(self, index: int) -> Optional[TOCItem]:
"""根据序号查找目录项 (1-based)"""
if 1 <= index <= len(self._toc_items):
return self._toc_items[index - 1]
return None
def get_file_range(self, start_title: str = None, end_title: str = None,
start_index: int = None, end_index: int = None) -> Tuple[Set[str], List[TOCItem]]:
"""
获取指定范围内的文件集合
支持两种方式指定范围：
1. 按标题: start_title ~ end_title
2. 按序号: start_index ~ end_index
Returns:
(文件名集合, 选中的目录项列表)
"""
# 确定起始位置
start_item = None
if start_title:
start_item = self.find_by_title(start_title)
if not start_item:
logger.warning(f"未找到起始章节: {start_title}")
elif start_index:
start_item = self.find_by_index(start_index)
# 确定结束位置
end_item = None
if end_title:
end_item = self.find_by_title(end_title)
if not end_item:
logger.warning(f"未找到结束章节: {end_title}")
elif end_index:
end_item = self.find_by_index(end_index)
# 默认值
start_idx = start_item.index if start_item else 1
end_idx = end_item.index if end_item else len(self._toc_items)
# 确保顺序正确
if start_idx > end_idx:
start_idx, end_idx = end_idx, start_idx
# 收集文件
selected_items = []
file_names = set()
for item in self._toc_items:
if start_idx <= item.index <= end_idx:
selected_items.append(item)
if item.file_name:
file_names.add(item.file_name)
logger.info(f"选择范围: #{start_idx} ~ #{end_idx}，共 {len(file_names)} 个文件")
return file_names, selected_items
def format_toc_table(self, selected_range: Tuple[int, int] = None, show_skip: bool = True) -> str:
"""
格式化 TOC 为表格形式，用于终端显示
Args:
selected_range: 可选的选中范围 (start_index, end_index)，用于高亮显示
show_skip: 是否显示跳过标记
Returns:
格式化的表格字符串
"""
if not self._toc_items:
return "目录为空"
lines = []
lines.append("")
lines.append("=" * 75)
lines.append(f"{'#':>4} {'状态':<6} {'章节名称':<35} {'文件'}")
lines.append("=" * 75)
for item in self._toc_items:
indent = " " * item.level
title_display = f"{indent}{item.title}"
if len(title_display) > 33:
title_display = title_display[:30] + "..."
# 跳过状态标记
status = ""
if show_skip and item.skip_reason:
status = "[SKIP]" if item.skip_reason else ""
# 如果在选中范围内，添加标记
marker = ""
if selected_range:
start_idx, end_idx = selected_range
if item.index == start_idx:
marker = " ▶"
elif item.index == end_idx:
marker = " ◀"
elif start_idx < item.index < end_idx:
marker = " │"
lines.append(f"{item.index:>4}{marker:2} {status:<6} {title_display:<35} {item.file_name}")
lines.append("=" * 75)
# 统计摘要
skip_count = sum(1 for i in self._toc_items if i.skip_reason)
content_count = len(self._toc_items) - skip_count
lines.append(f" 正文章节: {content_count} | 跳过章节: {skip_count}")
lines.append("")
return "\n".join(lines)
archive/v0.09/src/translator.py
+348
View File
@@ -0,0 +1,348 @@
"""
EPUB Translator Core Module - v0.09 (TOC Selection Support)
"""
import asyncio
import traceback
from typing import List, Dict, Any
from pathlib import Path
from loguru import logger
from rich.console import Console
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TimeElapsedColumn
from .epub_parser import EPUBParser
from .toc_parser import TOCParser
from .llm_client import LLMClient
from .text_processor import TextProcessor
from .bilingual_builder import BilingualEPUBBuilder
from .chinese_builder import ChineseEPUBBuilder
from .manifest_manager import ManifestManager
from .book_profiler import BookProfiler
from .cache import TranslationCache
from .format_restorer import FormatRestorer
from .utils import add_spacing_between_cn_and_en_num
class EPUBTranslator:
def __init__(self, config: Dict, use_cache: bool = True):
self.config = config
self.console = Console()
self.use_cache = use_cache
self.parser = None
self.llm_client = LLMClient(config)
self.text_processor = TextProcessor(config)
self.profiler = BookProfiler(config, self.llm_client)
self.cache = TranslationCache(config) if use_cache else None
self.restorer = FormatRestorer()
self.manifest_dir = Path("cache/manifests")
self.manifest_dir.mkdir(parents=True, exist_ok=True)
async def translate_epub(self, epub_path: str, test_mode: bool = False,
output_dir: str = None, mode: str = "bilingual",
from_chapter: str = None, to_chapter: str = None) -> str:
"""
翻译 EPUB 文件
"""
try:
epub_path = Path(epub_path)
# --- 0. Preprocessing (集成的 EpubCleaner) ---
# 创建临时清理文件
from .epub_cleaner import EpubCleaner
cleaner = EpubCleaner()
processed_dir = self.manifest_dir / "processed_epubs"
processed_dir.mkdir(parents=True, exist_ok=True)
cleaned_epub_path = processed_dir / f"{epub_path.stem}_cleaned.epub"
self.console.print(f"[yellow]Preprocessing: Cleaning EPUB structures...[/yellow]")
cleaner.clean_epub(str(epub_path), str(cleaned_epub_path))
# 关键：后续操作全都基于清理后的 EPUB
# 注意：这会改变 source_file 的上下文吗？只要 parser 读的是 cleaned_epub，manifest 记录的就是 cleaned_epub 里的文件名。
# 而 Builder 用 cleaned_epub 初始化的，所以也是匹配的。
actual_epub_path = cleaned_epub_path
self.parser = EPUBParser(str(actual_epub_path))
# 0.5 解析 TOC 并处理章节范围
toc_parser = TOCParser(self.parser.book)
include_files = None
chapter_range_info = None
if from_chapter or to_chapter:
include_files, selected_items = toc_parser.get_spine_range(
start_title=from_chapter, end_title=to_chapter
)
if selected_items:
start_title = selected_items[0].title
end_title = selected_items[-1].title
self.console.print(f"[cyan]📚 Range: {start_title} ~ {end_title} ({len(include_files)} files)[/cyan]")
chapter_range_info = {"start_title": start_title, "end_title": end_title, "included_files": list(include_files)}
else:
skip_files = toc_parser.get_skip_files()
if skip_files:
include_files = toc_parser.get_content_files_from_spine()
self.console.print(f"[cyan]📚 Smart Skip: {len(skip_files)} non-content files[/cyan]")
# 1. Manifest
manifest_suffix = "_chinese" if mode == "chinese" else ""
manifest_path = self.manifest_dir / f"{epub_path.stem}{manifest_suffix}_manifest.json"
manifest = ManifestManager(str(manifest_path))
if not manifest.load() or not self.use_cache:
self.console.print(f"[yellow]Initializing Manifest...[/yellow]")
manifest.init_manifest(book_id=epub_path.name, metadata=self.parser.get_book_info(), chapter_range=chapter_range_info)
content_items = self.parser.extract_all_content_items(include_files=include_files)
for item in content_items:
self.text_processor.extract_to_manifest(item['content'], item['file_name'], manifest, mode=mode)
manifest.save()
# (Stats logic...)
stats = manifest.stats
self.console.print(f"[green]Manifest: {stats['total']} items ({stats['pending']} pending)[/green]")
# 2. Profile
profile = {}
if not test_mode and stats['pending'] > 0:
self.console.print("[yellow]Profiling Book...[/yellow]")
profile = await self.profiler.analyze_book(manifest)
# 3. Translate
chunks = self.text_processor.create_chunks_from_manifest(manifest, mode=mode)
if test_mode: chunks = chunks[:10]
if chunks:
await self._translate_concurrently(chunks, manifest, profile, mode=mode)
# 4. Build
self.console.print(f"\n[yellow]Building {mode} EPUB...[/yellow]")
output_path = output_dir or self.config['output']['output_dir']
# 统一使用 BilingualEPUBBuilder (集成 V2)
# 因为它已经支持了 FineGrained backfill，可以处理 bilingual 参数
# 但目前 builder 还没暴露 bilingual 参数给 create 方法？
# 我们可以简单地在 builder.create... 里改一下，或者总是用 BilingualBuilder。
# 用户想要 "chinese" mode (replace).
# BilingualBuilder.create... 目前 hardcode 了 bilingual=True (TODO comment in previous step).
# 我们应该让 BilingualBuilder 支持 mode 参数。
# 为了简单，我假设 builder 内部会处理，或者我之后微调 builder。
# 修改: BilingualEPUBBuilder 是通用的 backfiller。
# 构建 Mapping
if mode == "chinese":
# 中文模式：使用还原后的 HTML (保留格式)
translation_map = {
item.global_id: (item.translation_with_original_html or item.translation)
for item in manifest.get_items()
if item.translation_with_original_html or item.translation
}
else:
# 双语模式：使用纯文本
translation_map = {item.global_id: item.translation for item in manifest.get_items() if item.translation}
paragraph_map = {item.global_id: {
"file_name": item.source_file,
# 其他 metadata 其实不需要了，builder 会重新提取
} for item in manifest.get_items()}
builder = BilingualEPUBBuilder(self.parser.book, self.config)
# 临时 Hack: 如果是 chinese 模式，修改 builder 的逻辑 (或者 builder 自动读取 config)
# Builder 构造函数读了 config。
# 我们需要在 config 里设置 mode 吗？或者 Builder 可以加个 set_mode?
# 这里的 config 是全局 config。main.py 里并没有把 args.mode 写入 config['output']。
# 我们可以在这里 patch 一下 config。
self.config['output']['mode'] = mode # 确保 Builder 知道模式
# 注意: 之前的 BilingualBuilder._process_document_content 里写死 bilingual_mode = True
# 我需要去修一下 BilingualBuilder，让它读 self.config['output']['mode']
result_file = builder.create_bilingual_epub_with_mapping(translation_map, paragraph_map, output_path)
self.console.print(f"[green]Done: {result_file}[/green]")
return result_file
except Exception as e:
traceback.print_exc()
logger.error(f"Translation failed: {e}")
raise
async def _translate_concurrently(self, chunks: List[List[Any]], manifest: ManifestManager,
profile: Dict, mode: str = "bilingual"):
"""并发翻译核心逻辑 (统一单双语)"""
total_chunks = len(chunks)
glossary = profile.get('glossary', {})
instruction = profile.get('translation_instruction', "")
with Progress(
SpinnerColumn(), TextColumn("[progress.description]{task.description}"),
BarColumn(), TextColumn("{task.percentage:>3.0f}%"), TimeElapsedColumn(),
console=self.console
) as progress:
task_id = progress.add_task(f"[cyan]Translating...", total=total_chunks)
async def worker(chunk, idx):
try:
chunk_dicts = [item.to_dict() for item in chunk]
results = None
model_name = self.llm_client.models.get('fast', 'unknown')
if self.cache:
results = self.cache.get_chunk_translation(chunk_dicts, model=model_name)
if not results:
# FIX: Use keyword arguments to avoid positional mismatch (model_type vs mode)
results = await self.llm_client.translate_chunk(
items=chunk,
glossary=glossary,
instruction=instruction,
mode=mode
)
if self.cache and results:
self.cache.save_chunk_translation(chunk_dicts, results, model=model_name)
for item in chunk:
if item.global_id in results:
raw_trans = results[item.global_id]
if "[Error" in raw_trans:
manifest.update_item(item.global_id, None, status="failed", error=raw_trans)
continue
processed_trans = add_spacing_between_cn_and_en_num(raw_trans)
# 仅保存译文，还原逻辑外移至所有翻译完成后执行
manifest.update_item(
item.global_id,
processed_trans,
translation_with_placeholders=processed_trans,
status="translated"
)
else:
manifest.update_item(item.global_id, None, status="failed", error="Translate failed: ID not found in response")
except Exception as e:
logger.error(f"Worker {idx} error: {e}")
finally:
progress.update(task_id, advance=1)
# 并发执行翻译任务
tasks = [worker(chunk, i) for i, chunk in enumerate(chunks)]
await asyncio.gather(*tasks)
# 翻译完成后立即保存，确保数据持久化
manifest.save()
logger.info("翻译阶段完成，manifest 已保存")
# --- 第二阶段：统一进行格式还原与修复 ---
logger.info("开始进行格式还原与占位符校验...")
await self.process_format_restoration(manifest, mode)
async def process_format_restoration(self, manifest, mode):
"""统一处理所有段落的格式还原和修复"""
items = manifest.get_items()
success_count = 0
failed_count = 0
repaired_count = 0
for item in items:
# 仅处理已翻译或之前格式还原失败的项目，或者已完成但缺少还原HTML的项目
should_process = (
item.status == "translated" or
item.status == "format_error" or
(item.status == "completed" and not item.translation_with_original_html)
)
if not should_process or not item.translation_with_placeholders:
continue
processed_trans = item.translation_with_placeholders
translation_with_ph = processed_trans
restored_html = ""
success = False
if item.placeholder_map:
# 过滤内嵌占位符（排除 _prefix, _suffix）
inner_placeholders = {k: v for k, v in item.placeholder_map.items()
if not k.startswith("_")}
if not inner_placeholders:
# 没有内嵌占位符，清除可能多出的占位符
clean_translation = self.restorer._strip_placeholders(processed_trans)
translation_with_ph = clean_translation
restored_html, success = self.restorer.restore(clean_translation, item.placeholder_map)
else:
# 有内嵌占位符，尝试直接还原
restored_html, success = self.restorer.restore(processed_trans, item.placeholder_map)
if not success:
# 尝试修复逻辑
import re
found_ids = set(re.findall(r'φ(/?\d+)φ', processed_trans))
expected_ids = set(inner_placeholders.keys())
missing_ids = expected_ids - found_ids
if missing_ids:
logger.warning(f"占位符缺失 (ID: {item.global_id}), 尝试修复: {missing_ids}")
try:
fixed_trans = await self.llm_client.repair_format(
item.text_with_placeholders,
processed_trans,
missing_ids=missing_ids
)
restored_html_2, success_2 = self.restorer.restore(fixed_trans, item.placeholder_map)
if success_2:
repaired_count += 1
translation_with_ph = fixed_trans
restored_html = restored_html_2
success = True
else:
# 尾注补救
if hasattr(item, 'endnote_anchors') and item.endnote_anchors:
still_missing = [a for a in item.endnote_anchors
if f"φ{a}φ" not in fixed_trans]
if still_missing:
for anchor_id in still_missing:
fixed_trans += f"φ{anchor_id}φ"
restored_html_3, success_3 = self.restorer.restore(fixed_trans, item.placeholder_map)
if success_3:
translation_with_ph = fixed_trans
restored_html = restored_html_3
success = True
except Exception as e:
logger.error(f"修复失败 (ID: {item.global_id}): {e}")
else:
# 无需还原
restored_html = processed_trans
success = True
# 更新 Manifest
final_translation = self.restorer._strip_placeholders(translation_with_ph)
manifest.update_item(
item.global_id,
final_translation,
translation_with_placeholders=translation_with_ph,
translation_with_original_html=restored_html,
status="completed" if success else "format_error"
)
if success: success_count += 1
else: failed_count += 1
# 保存还原结果
manifest.save()
# 统计错误比例
total_processed = success_count + failed_count
if total_processed > 0:
error_rate = failed_count / total_processed
logger.info(f"占位符还原完成: 成功 {success_count}, 失败 {failed_count}, 修复 {repaired_count}, 错误率 {error_rate*100:.2f}%")
# 超过 1% 错误率，判定为严重问题，中止操作
if error_rate > 0.01:
error_msg = f"占位符错误率过高 ({error_rate*100:.2f}% > 1%)，检测到严重不可修复问题，中止操作"
logger.error(error_msg)
raise RuntimeError(error_msg)
else:
logger.warning("没有处理任何段落")
archive/v0.09/src/utils.py
+208
View File
@@ -0,0 +1,208 @@
"""
工具函数模块
提供配置加载、日志设置等通用功能
"""
import json
import os
from pathlib import Path
from typing import Dict, Any
from loguru import logger
import sys
from dotenv import load_dotenv
def load_config(config_path: str = "config/config.json") -> Dict[str, Any]:
"""
加载配置文件
Args:
config_path: 配置文件路径
Returns:
配置字典
"""
# 加载 .env 文件
load_dotenv()
try:
with open(config_path, 'r', encoding='utf-8') as f:
config = json.load(f)
# 从环境变量获取 API Key
for provider_name, provider_config in config.get('providers', {}).items():
env_key = f"{provider_name.upper()}_API_KEY"
if env_key in os.environ:
provider_config['api_key'] = os.environ[env_key]
return config
except FileNotFoundError:
raise FileNotFoundError(f"配置文件未找到: {config_path}")
except json.JSONDecodeError as e:
raise ValueError(f"配置文件格式错误: {e}")
def load_prompts(prompts_path: str = "config/prompts.json") -> Dict[str, str]:
"""
加载提示词模板
Args:
prompts_path: 提示词文件路径
Returns:
提示词字典
"""
try:
with open(prompts_path, 'r', encoding='utf-8') as f:
return json.load(f)
except FileNotFoundError:
raise FileNotFoundError(f"提示词文件未找到: {prompts_path}")
def setup_logging(config: Dict[str, Any]) -> None:
"""
设置日志配置
Args:
config: 配置字典
"""
log_config = config.get('logging', {})
# 移除默认处理器
logger.remove()
# 添加控制台输出
logger.add(
sys.stdout,
level=log_config.get('level', 'INFO'),
format="<green>{time:YYYY-MM-DD HH:mm:ss}</green> | <level>{level: <8}</level> | <cyan>{name}</cyan>:<cyan>{function}</cyan>:<cyan>{line}</cyan> - <level>{message}</level>"
)
# 添加文件输出
if 'file' in log_config:
log_file = log_config['file']
# 确保日志目录存在
Path(log_file).parent.mkdir(parents=True, exist_ok=True)
logger.add(
log_file,
level=log_config.get('level', 'INFO'),
rotation=log_config.get('rotation', '10 MB'),
retention=log_config.get('retention', '7 days'),
encoding='utf-8',
format="{time:YYYY-MM-DD HH:mm:ss} | {level: <8} | {name}:{function}:{line} - {message}"
)
def ensure_output_dir(output_dir: str) -> Path:
"""
确保输出目录存在
Args:
output_dir: 输出目录路径
Returns:
输出目录的 Path 对象
"""
output_path = Path(output_dir)
output_path.mkdir(parents=True, exist_ok=True)
return output_path
def sanitize_filename(filename: str) -> str:
"""
清理文件名，移除非法字符
Args:
filename: 原始文件名
Returns:
清理后的文件名
"""
import re
# 移除或替换非法字符
filename = re.sub(r'[<>:"/\\|?*]', '_', filename)
# 移除多余的空格和点
filename = re.sub(r'\s+', ' ', filename).strip('. ')
return filename
def format_file_size(size_bytes: int) -> str:
"""
格式化文件大小显示
Args:
size_bytes: 字节数
Returns:
格式化的大小字符串
"""
if size_bytes == 0:
return "0B"
size_names = ["B", "KB", "MB", "GB"]
import math
i = int(math.floor(math.log(size_bytes, 1024)))
p = math.pow(1024, i)
s = round(size_bytes / p, 2)
return f"{s} {size_names[i]}"
def estimate_tokens(text: str) -> int:
"""
估算文本的 token 数量
Args:
text: 输入文本
Returns:
估算的 token 数量
"""
# 简单估算：英文约 4 字符/token，中文约 1.5 字符/token
import re
# 分离中英文
chinese_chars = len(re.findall(r'[\u4e00-\u9fff]', text))
other_chars = len(text) - chinese_chars
# 估算 tokens
estimated_tokens = chinese_chars / 1.5 + other_chars / 4
return int(estimated_tokens)
def truncate_text(text: str, max_length: int = 100) -> str:
"""
截断文本用于显示
Args:
text: 原始文本
max_length: 最大长度
Returns:
截断后的文本
"""
if len(text) <= max_length:
return text
return text[:max_length-3] + "..."
def add_spacing_between_cn_and_en_num(text: str) -> str:
"""
在中文和英文/数字之间添加空格（盘古之白）
Args:
text: 原始文本
Returns:
处理后的文本
"""
import re
if not text:
return text
# 中文-英文/数字
text = re.sub(r'([\u4e00-\u9fff])([a-zA-Z0-9])', r'\1 \2', text)
# 英文/数字-中文
text = re.sub(r'([a-zA-Z0-9])([\u4e00-\u9fff])', r'\1 \2', text)
return text
archive/v0.09/test_build_prompt.py
+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/env python3
"""模拟翻译流程，检查 _build_prompt 使用的数据"""
import sys
sys.path.insert(0, '.')
from src.manifest_manager import ManifestManager
from src.text_processor import TextProcessor
# 加载 manifest
manifest = ManifestManager("cache/manifests/Empire of AI Dreams and Nightmares in Sam Altmans OpenAI (Karen Hao)_chinese_manifest.json")
manifest.load()
# 创建 chunks (模拟 create_chunks_from_manifest)
processor = TextProcessor({})
chunks = processor.create_chunks_from_manifest(manifest, mode="chinese")
print(f"Total chunks: {len(chunks)}")
if chunks:
# 取第一个 chunk 的前几个 item
first_chunk = chunks[0]
print(f"First chunk has {len(first_chunk)} items")
for item in first_chunk[:5]:
print(f"\n=== ITEM {item.global_id} ===")
print(f" type(item): {type(item)}")
print(f" clean_text: '{item.clean_text[:50]}...'")
print(f" text_with_placeholders: '{item.text_with_placeholders[:50] if item.text_with_placeholders else 'EMPTY'}...'")
print(f" has φ in twp: {'φ' in (item.text_with_placeholders or '')}")
print(f" placeholder_map: {item.placeholder_map}")
# 模拟 _build_prompt 的逻辑
if item.text_with_placeholders:
text = item.text_with_placeholders
else:
text = item.clean_text
prompt_line = f"{item.global_id} [BODY] {text}"
print(f" -> Will send to LLM: '{prompt_line[:80]}...'")
print(f" -> Prompt contains φ: {'φ' in prompt_line}")
archive/v0.09/test_build_prompt2.py
+33
View File
@@ -0,0 +1,33 @@
#!/usr/bin/env python3
"""直接检查 manifest 中的 pending items"""
import sys
sys.path.insert(0, '.')
from src.manifest_manager import ManifestManager
# 加载 manifest
manifest = ManifestManager("cache/manifests/Empire of AI Dreams and Nightmares in Sam Altmans OpenAI (Karen Hao)_chinese_manifest.json")
manifest.load()
# 获取 pending items（这是 create_chunks_from_manifest 内部调用的）
pending_items = manifest.get_items(status="pending")
print(f"Pending items: {len(pending_items)}")
# 如果没有 pending，获取所有
if not pending_items:
print("No pending items, getting all...")
pending_items = manifest.get_items()
print(f"All items: {len(pending_items)}")
# 检查前 10 个 item
for item in pending_items[:10]:
twp = item.text_with_placeholders
has_phi = 'φ' in twp if twp else False
inner_ph = {k: v for k, v in item.placeholder_map.items() if not k.startswith('_')} if item.placeholder_map else {}
print(f"\n{item.global_id}:")
print(f" status: {item.status}")
print(f" clean_text: '{item.clean_text[:40]}...'")
print(f" text_with_placeholders: '{twp[:40] if twp else 'EMPTY'}...'")
print(f" has φ: {has_phi}")
print(f" inner_placeholders: {list(inner_ph.keys())}")
archive/v0.09/test_cache_check.py
+37
View File
@@ -0,0 +1,37 @@
#!/usr/bin/env python3
"""检查翻译缓存中的结果是否包含占位符"""
import json
import os
translations_dir = "cache/translations"
all_translations = {}
# 遍历缓存文件
for root, dirs, files in os.walk(translations_dir):
for f in files:
if f.endswith('.json'):
path = os.path.join(root, f)
with open(path, 'r') as fp:
cache = json.load(fp)
if 'translations' in cache:
for k, v in cache['translations'].items():
all_translations[k] = {'trans': v, 'file': path}
# 检查 p_00006 (已知有占位符的 item)
target_id = 'p_00006'
if target_id in all_translations:
data = all_translations[target_id]
print(f"=== CACHED TRANSLATION FOR {target_id} ===")
print(f"Cache file: {data['file']}")
print(f"Translation: {data['trans']}")
print(f"Contains φ: {'φ' in data['trans']}")
else:
print(f"{target_id} not found in cache")
# 统计有多少缓存翻译包含 φ
with_phi = sum(1 for v in all_translations.values() if 'φ' in v['trans'])
total = len(all_translations)
print(f"\n=== CACHE STATS ===")
print(f"Total cached translations: {total}")
print(f"Translations with φ: {with_phi}")
print(f"Translations without φ: {total - with_phi}")
archive/v0.09/test_extractor.py
+31
View File
@@ -0,0 +1,31 @@
import re
from src.format_extractor import FormatExtractor
extractor = FormatExtractor()
cases = [
("<p>“But what is the goal?” <em>Amodei</em>...</p>", "Quoted text with em"),
("<p>Q. What is artificial intelligence?</p>", "Simple Q&A"),
("<p>Text <i>italic</i> followed by dots...</p>", "Italic with trailing dots"),
]
for html, desc in cases:
print(f"--- Testing: {desc} ---")
print(f"HTML: {html}")
clean, text_ph, ph_map, p_type, anchors = extractor.extract(html)
# Validation logic from FormatExtractor.extract
stripped_text = re.sub(r'φ/?[0-9]+φ', '', text_ph)
stripped_text = re.sub(r'\s+', ' ', stripped_text).strip()
clean_normalized = re.sub(r'\s+', ' ', clean).strip()
print(f"Clean: '{clean_normalized}'")
print(f"Stripped: '{stripped_text}'")
print(f"Text ph: '{text_ph}'")
print(f"Map: {ph_map}")
if clean_normalized == stripped_text:
print("✅ SUCCESS")
else:
print("❌ FAILED")
print()
archive/v0.09/test_extractor_v3.py
+38
View File
@@ -0,0 +1,38 @@
import re
from src.format_extractor import FormatExtractor
extractor = FormatExtractor()
cases = [
("<p>“But what is the goal?” <em>Amodei</em>...</p>", "Quoted text with em"),
("<p>Q. What is artificial intelligence?</p>", "Simple Q&A"),
("<p>Text <i>italic</i> followed by dots...</p>", "Italic with trailing dots"),
]
for html, desc in cases:
print(f"--- Testing: {desc} ---")
# Simulate how extract() identifies inner_html
from bs4 import BeautifulSoup, Tag
soup = BeautifulSoup(html, 'html.parser')
root = list(soup.children)[0] if list(soup.children) else soup
clean_text = root.get_text().strip()
clean_text = re.sub(r'\s+', ' ', clean_text).strip()
inner_html = root.decode_contents() if isinstance(root, Tag) else str(root)
# Call v3 directly
text_with_ph, local_map = extractor._smart_extract_v3(inner_html)
stripped_text = re.sub(r'φ/?[0-9]+φ', '', text_with_ph)
stripped_text = re.sub(r'\s+', ' ', stripped_text).strip()
print(f"Clean: '{clean_text}'")
print(f"Stripped: '{stripped_text}'")
print(f"Text ph: '{text_with_ph}'")
print(f"Map: {local_map}")
if clean_text == stripped_text:
print("✅ SUCCESS")
else:
print("❌ FAILED")
print()
archive/v0.09/test_llm_client_io.py
+62
View File
@@ -0,0 +1,62 @@
#!/usr/bin/env python3
"""
最小化测试脚本：测试 LLMClient 的输入输出
完全模拟真实调用路径，排除 Translator/TextProcessor 的干扰
"""
import sys
import asyncio
import json
import logging
from loguru import logger
from src.llm_client import LLMClient
from src.manifest_manager import ManifestItem
from src.utils import load_config
# 配置日志输出到控制台
logger.remove()
logger.add(sys.stdout, level="DEBUG")
async def test_io():
print("=== 初始化 LLMClient ===")
config = load_config()
# 使用 v3 provider
if 'v3' in config['providers']:
config['llm'] = config['providers']['v3']
print(f"Using provider: v3 (model: {config['llm']['models']['fast']})")
client = LLMClient(config)
# 构造测试 Item (模拟真实数据)
item = ManifestItem(
global_id="p_00006",
source_file="test.html",
original_html="<p>in the name of <span id='page_vi'></span> abundance...</p>",
clean_text="in the name of abundance...",
text_hash="dummy_hash",
tag="p",
# 关键：设置 text_with_placeholders
text_with_placeholders="in the name of φ1φabundance...", # 故意不加空格，模拟原始数据
placeholder_map={"1": "<span id='page_vi'></span>"},
paragraph_type="BODY"
)
items = [item]
mode = "chinese"
print("\n=== 1. 测试 _build_prompt 输出 ===")
# 直接调用私有方法查看生成的 prompt
prompt = client._build_prompt(items, mode=mode)
print(f"Generated Prompt:\n{prompt}")
print(f"Contains φ1φ: {'φ1φ' in prompt}")
print("\n=== 2. 测试 translate_chunk 完整调用 ===")
# 这会触发我们之前添加的 ERROR/DEBUG 日志
results = await client.translate_chunk(items, mode=mode)
print("\n=== 3. 检查结果 ===")
trans = results.get("p_00006", "MISSING")
print(f"Translation: {trans}")
print(f"Contains φ: {'φ' in trans}")
if __name__ == "__main__":
asyncio.run(test_io())
archive/v0.09/test_llm_placeholder.py
+81
View File
@@ -0,0 +1,81 @@
#!/usr/bin/env python3
"""
测试 LLM 是否正确保留占位符
"""
import asyncio
import json
from openai import AsyncOpenAI
# 加载配置
with open("config/config.json", "r") as f:
config = json.load(f)
v3_config = config["providers"]["v3"]
# 从 .env 加载 API Key
import os
from dotenv import load_dotenv
load_dotenv()
api_key = os.getenv("V3_API_KEY") or v3_config.get("api_key", "")
# 加载 prompts
with open("config/prompts.json", "r") as f:
prompts = json.load(f)
system_prompt = prompts["translation"]["system"]
# 测试文本 - 包含占位符
test_text = """p_00058 [BODY] Also that night, the board and the remaining leadership at the company were holding a series of increasingly hostile meetings. After the all-φ1φhands, the false projection of unity between Sutskever and the other leaders had collapsed. Many of the executives who had sat next to Sutskever during the livestream had been nearly as blindsided as the rest of the staff, having learned of Altman's dismissal moments before it was announced. φ2φRiled up by Sutskever's poor performance, they had demanded to meet with the rest of the board. Roughly a dozen executives, including Murati and Lightcap, had gathered in a conference room at the office."""
async def test_translation():
client = AsyncOpenAI(
base_url=v3_config["base_url"],
api_key=api_key,
default_headers=v3_config.get("extra_headers", {})
)
model = v3_config["models"]["fast"]
print("=" * 60)
print("SYSTEM PROMPT:")
print("=" * 60)
print(system_prompt)
print()
print("=" * 60)
print("USER PROMPT:")
print("=" * 60)
print(test_text)
print()
print("=" * 60)
print(f"Calling LLM ({model})...")
print("=" * 60)
response = await client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": test_text}
],
temperature=0.3
)
result = response.choices[0].message.content
print()
print("=" * 60)
print("LLM RESPONSE:")
print("=" * 60)
print(result)
print()
# 检查占位符
has_phi1 = "φ1φ" in result
has_phi2 = "φ2φ" in result
print("=" * 60)
print("PLACEHOLDER CHECK:")
print(f" φ1φ present: {has_phi1}")
print(f" φ2φ present: {has_phi2}")
print("=" * 60)
if __name__ == "__main__":
asyncio.run(test_translation())
archive/v0.09/test_manifest_data.py
+44
View File
@@ -0,0 +1,44 @@
#!/usr/bin/env python3
"""验证 manifest 数据和 _build_prompt 输出"""
import json
manifest_file = "cache/manifests/Empire of AI Dreams and Nightmares in Sam Altmans OpenAI (Karen Hao)_chinese_manifest.json"
with open(manifest_file, 'r') as f:
data = json.load(f)
items = data.get('items', [])
# 找一个有占位符的 item
target = None
for item in items:
if item.get('placeholder_map') and len(item['placeholder_map']) > 0:
inner = {k: v for k, v in item['placeholder_map'].items() if not k.startswith('_')}
if inner:
target = item
break
if target:
print("=== TARGET ITEM ===")
print(f"global_id: {target['global_id']}")
print(f"status: {target['status']}")
print(f"clean_text: {target['clean_text'][:80]}...")
print(f"text_with_placeholders: '{target['text_with_placeholders'][:80]}...'")
print(f"placeholder_map: {target['placeholder_map']}")
print()
# 模拟 _build_prompt 的行为
text_with_ph = target['text_with_placeholders']
if text_with_ph:
prompt_line = f"{target['global_id']} [BODY] {text_with_ph}"
else:
prompt_line = f"{target['global_id']} [BODY] {target['clean_text']}"
print("=== SIMULATED PROMPT LINE ===")
print(prompt_line[:150])
print()
# 检查 text_with_placeholders 是否包含 φ
has_phi = 'φ' in (text_with_ph or '')
print(f"text_with_placeholders contains φ: {has_phi}")
else:
print("No item with placeholders found")
archive/v0.09/test_manifest_load.py
+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/env python3
"""验证 manifest 加载后 ManifestItem 对象的 text_with_placeholders"""
import sys
sys.path.insert(0, '.')
from src.manifest_manager import ManifestManager
manifest = ManifestManager("cache/manifests/Empire of AI Dreams and Nightmares in Sam Altmans OpenAI (Karen Hao)_chinese_manifest.json")
manifest.load()
# 获取有占位符的 item
items = manifest.get_items(status="pending")
if not items:
items = manifest.get_items() # 任意状态
target = None
for item in items:
if item.placeholder_map:
inner = {k: v for k, v in item.placeholder_map.items() if not k.startswith('_')}
if inner:
target = item
break
if target:
print("=== ManifestItem OBJECT ===")
print(f"global_id: {target.global_id}")
print(f"status: {target.status}")
print(f"clean_text: '{target.clean_text[:80]}...'")
print(f"text_with_placeholders: '{target.text_with_placeholders[:80] if target.text_with_placeholders else 'EMPTY'}...'")
print(f"placeholder_map: {target.placeholder_map}")
print()
# 检查是否包含 φ
twp = target.text_with_placeholders
has_phi = 'φ' in twp if twp else False
print(f"text_with_placeholders contains φ: {has_phi}")
print(f"text_with_placeholders length: {len(twp) if twp else 0}")
else:
print("No target item found with placeholders")
archive/v0.09/test_manifest_to_prompt.py
+125
View File
@@ -0,0 +1,125 @@
#!/usr/bin/env python3
"""
最小测试脚本：测试 Manifest -> TextProcessor -> LLMClient 的数据流
验证 text_with_placeholders 是否在传递过程中丢失或损坏
"""
import sys
import json
import asyncio
from pathlib import Path
from loguru import logger
# 添加 src 到路径
sys.path.insert(0, '.')
from src.manifest_manager import ManifestManager, ManifestItem
from src.text_processor import TextProcessor
from src.llm_client import LLMClient
# Mock config
from src.utils import load_config
# 配置日志
logger.remove()
logger.add(sys.stdout, level="DEBUG")
# 1. 创建临时的 Manifest 文件，模拟包含问题的真实数据
# 模拟一个带有换行符的占位符文本，这是我们怀疑的根源
mock_manifest_data = {
"metadata": {"book_id": "test_book"},
"items": [
{
"global_id": "p_00006",
"source_file": "test.html",
"original_html": "<p>in the name of <span id='page_vi'></span> abundance</p>",
"clean_text": "in the name of abundance",
"text_hash": "hash1",
"tag": "p",
# 模拟包含换行符的情况 (FormatExtractor 之前的问题)
"text_with_placeholders": "in the name of \nφ1φ\n abundance",
"placeholder_map": {"1": "<span id='page_vi'></span>"},
"paragraph_type": "BODY",
"status": "pending"
},
{
"global_id": "p_00058",
"source_file": "test.html",
"original_html": "<p>all-<span id='a536'></span>hands</p>",
"clean_text": "all-hands",
"text_hash": "hash2",
"tag": "p",
# 正常情况
"text_with_placeholders": "all-φ1φhands",
"placeholder_map": {"1": "<span id='a536'></span>"},
"paragraph_type": "BODY",
"status": "pending"
}
]
}
manifest_path = Path("cache/test_manifest.json")
manifest_path.parent.mkdir(parents=True, exist_ok=True)
with open(manifest_path, 'w') as f:
json.dump(mock_manifest_data, f)
print(f"=== Created Mock Manifest at {manifest_path} ===")
async def run_test():
# 2. 加载 Manifest
manifest = ManifestManager(str(manifest_path))
manifest.load()
print(f"Loaded {len(manifest.get_items())} items")
# 3. 创建 TextProcessor 和 Chunks
# Mock config for processor
processor = TextProcessor({"translation": {"chunk_size": 1000}})
# 这一步会从 manifest 读取 item
chunks = processor.create_chunks_from_manifest(manifest, mode="chinese")
print(f"Created {len(chunks)} chunks")
chunk = chunks[0]
print(f"Chunk 0 has {len(chunk)} items")
# 4. 模拟 LLMClient 构建 Prompt
# 不需要真正的 API key，只需要测试 _build_prompt
# 添加 dummy key 和 rate_limits 防止初始化报错
client = LLMClient({
"providers": {},
"llm": {
"api_key": "dummy_key",
"models": {"fast": "dummy_model", "smart": "dummy_model"},
"rate_limits": {"requests_per_minute": 60, "concurrent_requests": 2}
}
})
# 强制重新加载 prompts (确保我们使用最新的代码逻辑)
# 注意：我们之前修了 _load_prompts，如果 prompts.json 不存在会报错
# 这里我们假设 config/prompts.json 存在
print("\n=== 构建 Prompt (Mode: Chinese) ===")
prompt = client._build_prompt(chunk, mode="chinese")
print("-" * 40)
print(prompt)
print("-" * 40)
# 验证关键点
print("\n=== 验证结果 ===")
# 检查 p_00006
# 注意：我们之前修了 FormatExtractor，但那是针对**新提取**的内容。
# 这里我们测试的是**从旧 Manifest 读取**的内容。
# ManifestManager 读取时并不会自动清理换行符！
# 所以如果旧 manifest 里有换行，这里应该能复现出带换行的 prompt。
has_p00006 = "p_00006 [BODY] in the name of \nφ1φ\n abundance" in prompt
print(f"p_00006 has newlines (bad): {has_p00006}")
has_p00006_clean = "p_00006 [BODY] in the name of φ1φ abundance" in prompt
print(f"p_00006 is clean (good): {has_p00006_clean}")
has_p00058 = "p_00058 [BODY] all-φ1φhands" in prompt
print(f"p_00058 is correct: {has_p00058}")
if __name__ == "__main__":
asyncio.run(run_test())
archive/v0.09/test_newline_effect.py
+23
View File
@@ -0,0 +1,23 @@
import re
from src.format_extractor import FormatExtractor
# 模拟带换行的 HTML
html_with_newlines = """
in the name of
<span id="page_vi"></span>
abundance...
"""
extractor = FormatExtractor()
clean_text, text_with_ph, _, _, _ = extractor.extract(f"<p>{html_with_newlines}</p>")
print(f"Original HTML: {repr(html_with_newlines)}")
print(f"Clean Text: {repr(clean_text)}")
print(f"Text with PH: {repr(text_with_ph)}")
print(f"Has Newline: {'\\n' in text_with_ph}")
print("-" * 20)
# 模拟 Prompt 构建
prompt_line = f"p_00006 [BODY] {text_with_ph}"
print("Prompt Line Preview:")
print(prompt_line)
archive/v0.09/test_spacing.py
+16
View File
@@ -0,0 +1,16 @@
from src.utils import add_spacing_between_cn_and_en_num
cases = [
"在全φ1φ员会议",
"全φ1φ员",
"φ1φTable Talkφ/1φ",
"测试φ12φ测试",
"测试φ/12φ测试"
]
for text in cases:
processed = add_spacing_between_cn_and_en_num(text)
print(f"Original: '{text}'")
print(f"Processed: '{processed}'")
print(f"Changed: {text != processed}")
print()
archive/v0.09/test_specific_failure_cases.py
+124
View File
@@ -0,0 +1,124 @@
#!/usr/bin/env python3
"""
针对性测试脚本：复现用户报告的占位符丢失问题
直接读取 Manifest 中的特定失败 Item (p_00006, p_00011 等)
调用真实 LLM 进行翻译，并打印完整的 Prompt 和 Response
"""
import sys
import asyncio
import json
from pathlib import Path
from loguru import logger
# 添加 src 到路径
sys.path.insert(0, '.')
from src.manifest_manager import ManifestManager
from src.llm_client import LLMClient
from src.utils import load_config
# 配置日志
logger.remove()
logger.add(sys.stdout, level="DEBUG")
async def run_test():
print("=== 1. 加载配置 ===")
try:
config = load_config()
# 自动选择配置好的 provider
if 'v3' in config['providers'] and config['providers']['v3'].get('api_key'):
config['llm'] = config['providers']['v3']
print("Using Provider: v3")
elif 'openrouter' in config['providers']:
config['llm'] = config['providers']['openrouter']
print("Using Provider: openrouter")
else:
print("No valid provider found with API key in config!")
return
except Exception as e:
print(f"Config load failed: {e}")
return
print("\n=== 2. 加载真实 Manifest ===")
manifest_dir = Path("cache/manifests")
manifest_files = list(manifest_dir.glob("*.json"))
if not manifest_files:
print("Error: No manifest file found")
return
# 优先选择包含 "OpenAI" 的那个文件（用户截图）
target_manifest = next((f for f in manifest_files if "OpenAI" in f.name), manifest_files[0])
print(f"Loading: {target_manifest}")
manifest = ManifestManager(str(target_manifest))
if not manifest.load():
print("Failed to load manifest")
return
# 提取目标失败案例
target_ids = ["p_00006", "p_00009", "p_00011", "p_00013"]
# 也包括上下文以免错位 (p_00003 - p_00006)
context_ids = ["p_00003", "p_00004", "p_00005", "p_00006"]
items_to_test = []
# 测试组 1: 上下文错位测试
print("\n=== 准备测试组 1: 上下文错位及占位符 (p_00003-00006) ===")
group1 = []
for uid in context_ids:
item = manifest._items_by_id.get(uid)
if item:
# 强制清空旧翻译，模拟重新翻译
item.translation = None
item.translation_with_placeholders = None
group1.append(item)
print(f"Added {uid}: {item.text_with_placeholders}")
# 测试组 2: 独立行占位符丢失测试 (p_00009, p_00011)
print("\n=== 准备测试组 2: 独立行占位符 (p_00009, p_00011) ===")
group2 = []
for uid in ["p_00009", "p_00011"]:
item = manifest._items_by_id.get(uid)
if item:
item.translation = None
group2.append(item)
print(f"Added {uid}: {item.text_with_placeholders}")
client = LLMClient(config)
# 执行测试 1
if group1:
print("\n\n>>> 执行 Group 1 测试 (Context Alignment) <<<")
# 打印 Prompt 预览
prompt = client._build_prompt(group1, mode="chinese")
print("\n[PROMPT PREVIEW]:")
print("-" * 20)
print(prompt)
print("-" * 20)
# 调用 LLM
print("\n[CALLING LLM]...")
results = await client.translate_chunk(group1, mode="chinese")
print("\n[RESULTS Group 1]:")
for uid, trans in results.items():
print(f"{uid}: {trans}")
if uid == "p_00006":
print(f" > Contains φ1φ? {'φ1φ' in trans}")
# 执行测试 2
if group2:
print("\n\n>>> 执行 Group 2 测试 (Isolated Placeholders) <<<")
prompt = client._build_prompt(group2, mode="chinese")
print("\n[PROMPT PREVIEW]:")
print(prompt)
print("\n[CALLING LLM]...")
results = await client.translate_chunk(group2, mode="chinese")
print("\n[RESULTS Group 2]:")
for uid, trans in results.items():
print(f"{uid}: {trans}")
if __name__ == "__main__":
asyncio.run(run_test())
archive/v0.09/tests/__init__.py
+9
View File
@@ -0,0 +1,9 @@
"""
测试模块初始化文件
"""
import sys
from pathlib import Path
# 添加 src 目录到路径
sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
archive/v0.09/tests/analyze_placeholder_errors.py
+257
View File
@@ -0,0 +1,257 @@
#!/usr/bin/env python3
"""
占位符错误分析测试脚本
功能：
1. 使用 v3 provider (config 中配置) 翻译指定章节
2. 收集所有占位符错误
3. 输出分析报告
"""
import asyncio
import json
import re
from pathlib import Path
from collections import defaultdict
# 添加项目路径
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent))
from src.epub_parser import EPUBParser
from src.epub_cleaner import EpubCleaner
from src.text_processor import TextProcessor
from src.llm_client import LLMClient
from src.format_restorer import FormatRestorer
from src.manifest_manager import ManifestManager
from src.utils import add_spacing_between_cn_and_en_num
from loguru import logger
class PlaceholderAnalyzer:
"""占位符错误分析器"""
def __init__(self, config_path: str = "config/config.json", provider: str = "openrouter"):
with open(config_path, 'r') as f:
self.config = json.load(f)
# 扁平化 provider 配置到 config['llm']
providers = self.config.get('providers', {})
if provider not in providers:
raise ValueError(f"Provider '{provider}' not found. Available: {list(providers.keys())}")
self.config['llm'] = providers[provider]
print(f"使用 LLM 供应商: {provider} ({self.config['llm'].get('base_url')})")
self.llm_client = LLMClient(self.config)
self.text_processor = TextProcessor(self.config)
self.restorer = FormatRestorer()
# 错误收集
self.errors = []
self.success_count = 0
self.total_with_placeholders = 0
async def analyze_chapter(self, epub_path: str, chapter_file: str = None):
"""
分析单个章节的占位符处理情况
Args:
epub_path: EPUB 文件路径
chapter_file: 指定章节文件名 (如 'OEBPS/c3Z.xhtml'), 不指定则使用第一个内容章节
"""
# 1. 清理 EPUB
cleaner = EpubCleaner()
cleaned_path = "cache/manifests/processed_epubs/test_cleaned.epub"
Path(cleaned_path).parent.mkdir(parents=True, exist_ok=True)
cleaner.clean_epub(epub_path, cleaned_path)
# 2. 解析
parser = EPUBParser(cleaned_path)
content_items = parser.extract_all_content_items()
# 3. 选择章节
if chapter_file:
target_items = [i for i in content_items if i['file_name'] == chapter_file]
else:
# 默认选择第一个有较多内容的章节
target_items = [i for i in content_items if len(i['content']) > 5000][:1]
if not target_items:
print("未找到目标章节")
return
target = target_items[0]
print(f"\n分析章节: {target['file_name']}")
print("=" * 60)
# 4. 提取文本
manifest = ManifestManager("cache/manifests/test_analysis_manifest.json")
manifest.init_manifest(book_id="test", metadata={})
self.text_processor.extract_to_manifest(target['content'], target['file_name'], manifest, mode="chinese")
manifest.save()
# 5. 获取待翻译项
items = manifest.get_items(status="pending")
print(f"待翻译项: {len(items)}")
# 6. 筛选有占位符的项目
items_with_ph = [i for i in items if i.placeholder_map and
any(k for k in i.placeholder_map.keys() if not k.startswith("_"))]
self.total_with_placeholders = len(items_with_ph)
print(f"含内嵌占位符的项: {self.total_with_placeholders}")
# 7. 翻译并分析
print("\n开始翻译...")
# 分块翻译
chunks = self.text_processor.create_chunks_from_manifest(manifest, mode="chinese")
for i, chunk in enumerate(chunks):
print(f" 处理块 {i+1}/{len(chunks)}...")
await self._process_chunk(chunk)
# 8. 输出分析报告
self._print_report()
async def _process_chunk(self, chunk):
"""处理单个翻译块"""
try:
results = await self.llm_client.translate_chunk(
chunk,
glossary={},
instruction="",
mode="chinese"
)
for item in chunk:
if item.global_id not in results:
continue
raw_trans = results[item.global_id]
if "[Error" in raw_trans:
continue
processed_trans = add_spacing_between_cn_and_en_num(raw_trans)
# 检查是否有内嵌占位符
inner_ph = {k: v for k, v in (item.placeholder_map or {}).items()
if not k.startswith("_")}
if inner_ph:
# 验证还原
restored, success = self.restorer.restore(processed_trans, item.placeholder_map)
if not success:
# 记录错误
expected = set(inner_ph.keys())
found = set(re.findall(r'φ(/?\\d+)φ', processed_trans))
missing = expected - found
extra = found - expected
self.errors.append({
'id': item.global_id,
'text_with_ph': item.text_with_placeholders,
'translation_with_ph': processed_trans,
'placeholder_map': inner_ph,
'missing': list(missing),
'extra': list(extra),
'expected': list(expected),
'found': list(found)
})
else:
self.success_count += 1
except Exception as e:
logger.error(f"处理块失败: {e}")
def _print_report(self):
"""输出分析报告"""
print("\n" + "=" * 80)
print("占位符错误分析报告")
print("=" * 80)
print(f"\n总计含占位符项: {self.total_with_placeholders}")
print(f"成功还原: {self.success_count}")
print(f"失败: {len(self.errors)}")
if self.total_with_placeholders > 0:
success_rate = (self.success_count / self.total_with_placeholders) * 100
print(f"成功率: {success_rate:.1f}%")
if not self.errors:
print("\n🎉 没有占位符错误!")
return
print("\n" + "-" * 80)
print("错误详情")
print("-" * 80)
# 按错误类型分组
missing_only = [e for e in self.errors if e['missing'] and not e['extra']]
extra_only = [e for e in self.errors if e['extra'] and not e['missing']]
both = [e for e in self.errors if e['missing'] and e['extra']]
print(f"\n丢失占位符: {len(missing_only)} 个")
print(f"多余占位符: {len(extra_only)} 个")
print(f"两者都有: {len(both)} 个")
# 详细错误列表
print("\n" + "-" * 80)
print("详细错误列表 (最多显示 10 个)")
print("-" * 80)
for i, err in enumerate(self.errors[:10]):
print(f"\n[{i+1}] ID: {err['id']}")
print(f" 原文 (带占位符): {err['text_with_ph'][:100]}...")
print(f" 译文 (带占位符): {err['translation_with_ph'][:100]}...")
print(f" 期望占位符: {err['expected']}")
print(f" 找到占位符: {err['found']}")
print(f" 丢失: {err['missing']}")
print(f" 多余: {err['extra']}")
# 模式分析
print("\n" + "-" * 80)
print("错误模式分析")
print("-" * 80)
# 分析常见的丢失模式
all_missing = []
for e in self.errors:
all_missing.extend(e['missing'])
from collections import Counter
missing_counter = Counter(all_missing)
print("\n最常丢失的占位符:")
for ph, count in missing_counter.most_common(5):
print(f" φ{ph}φ: {count} 次")
# 保存完整报告到文件
report_path = "cache/placeholder_error_report.json"
with open(report_path, 'w', encoding='utf-8') as f:
json.dump({
'summary': {
'total_with_placeholders': self.total_with_placeholders,
'success_count': self.success_count,
'error_count': len(self.errors),
'success_rate': (self.success_count / self.total_with_placeholders * 100) if self.total_with_placeholders > 0 else 0
},
'errors': self.errors
}, f, ensure_ascii=False, indent=2)
print(f"\n完整报告已保存到: {report_path}")
async def main():
import argparse
parser = argparse.ArgumentParser(description="占位符错误分析")
parser.add_argument("epub", help="EPUB 文件路径")
parser.add_argument("--chapter", help="指定章节文件名")
args = parser.parse_args()
analyzer = PlaceholderAnalyzer()
await analyzer.analyze_chapter(args.epub, args.chapter)
await analyzer.llm_client.close()
if __name__ == "__main__":
asyncio.run(main())
archive/v0.09/tests/extraction_experiment/__init__.py
+5
View File
@@ -0,0 +1,5 @@
"""
文本提取实验模块
"""
__version__ = "0.1.0"
archive/v0.09/tests/extraction_experiment/analyze_missing.py
+233
View File
@@ -0,0 +1,233 @@
"""
缺失文本分析工具
详细分析提取器缺失的文本片段,找出根本原因
"""
import sys
from pathlib import Path
from loguru import logger
import re
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
sys.path.insert(0, str(Path(__file__).parent))
from extractors.enhanced_bs4 import EnhancedBS4Extractor
from extractors.baseline_pandoc import PandocBaseline
def analyze_missing_text(epub_path: Path, max_missing_samples: int = 20):
"""
分析缺失的文本片段
Args:
epub_path: ePub 文件路径
max_missing_samples: 最多显示的缺失样本数
"""
print(f"\n{'='*80}")
print(f"分析文件: {epub_path.name}")
print(f"{'='*80}\n")
# 1. 获取 Pandoc 基准
pandoc = PandocBaseline()
baseline_text = pandoc.extract_from_epub(str(epub_path))
if not baseline_text:
print("❌ Pandoc 提取失败")
return
print(f"Pandoc 基准长度: {len(baseline_text):,} 字符\n")
# 2. 提取器提取
book = epub.read_epub(str(epub_path))
html_docs = []
for item in book.get_items():
if item.get_type() == 9:
try:
content = item.get_content().decode('utf-8')
html_docs.append(content)
except:
continue
combined_html = "\n\n".join(html_docs)
extractor = EnhancedBS4Extractor(min_text_length=1, preserve_decorative=True) # 不过滤短文本
items = extractor.extract(combined_html)
# 分离内容和装饰性元素
content_items = [i for i in items if not i.get('is_decorative') and not i.get('is_navigation')]
decorative_items = [i for i in items if i.get('is_decorative')]
nav_items = [i for i in items if i.get('is_navigation')]
extracted_text = " ".join([item['text'] for item in content_items])
print(f"提取器统计:")
print(f" - 内容元素: {len(content_items)}")
print(f" - 装饰性元素: {len(decorative_items)}")
print(f" - 导航元素: {len(nav_items)}")
print(f" - 提取文本长度: {len(extracted_text):,} 字符\n")
# 3. 标准化文本
def normalize(text):
text = text.lower()
text = re.sub(r'[^\w\s]', ' ', text)
text = re.sub(r'\s+', ' ', text)
return text.strip()
baseline_norm = normalize(baseline_text)
extracted_norm = normalize(extracted_text)
# 4. 分词对比
baseline_words = baseline_norm.split()
extracted_words = set(extracted_norm.split())
print(f"词级别对比:")
print(f" - Pandoc 词数: {len(baseline_words):,}")
print(f" - 提取器词数: {len(extracted_words):,}")
# 5. 找出缺失的句子
print(f"\n{'='*80}")
print("分析缺失的文本片段")
print(f"{'='*80}\n")
# 将 Pandoc 文本分成句子
baseline_sentences = re.split(r'[.!?\n]+', baseline_text)
baseline_sentences = [s.strip() for s in baseline_sentences if len(s.strip()) > 10]
missing_sentences = []
for sentence in baseline_sentences:
sentence_norm = normalize(sentence)
if sentence_norm and sentence_norm not in extracted_norm:
# 检查是否有部分匹配
words = sentence_norm.split()
if len(words) > 3:
matched_words = sum(1 for w in words if w in extracted_words)
match_ratio = matched_words / len(words)
if match_ratio < 0.5: # 少于50%的词匹配,认为缺失
missing_sentences.append({
'text': sentence[:200], # 只取前200字符
'length': len(sentence),
'match_ratio': match_ratio
})
print(f"发现 {len(missing_sentences)} 个可能缺失的文本片段\n")
# 6. 分类缺失原因
print(f"{'='*80}")
print("缺失片段分类分析")
print(f"{'='*80}\n")
# 显示样本
for i, missing in enumerate(missing_sentences[:max_missing_samples], 1):
print(f"--- 缺失片段 {i} ---")
print(f"长度: {missing['length']} 字符")
print(f"匹配率: {missing['match_ratio']:.1%}")
print(f"内容: {missing['text']}")
# 尝试分析原因
text = missing['text'].lower()
reasons = []
if any(kw in text for kw in ['copyright', '©', 'isbn', 'publisher', 'published']):
reasons.append("📚 可能是版权/出版信息")
if any(kw in text for kw in ['table of contents', 'chapter', 'part', 'section']):
reasons.append("📑 可能是目录信息")
if any(kw in text for kw in ['page', 'pg', 'p.']):
reasons.append("📄 可能是页码")
if len(missing['text']) < 30:
reasons.append("📏 文本过短")
if re.match(r'^[0-9\s\-\.]+$', missing['text'].strip()):
reasons.append("🔢 纯数字")
if not reasons:
reasons.append("❓ 未知原因 - 需要进一步分析")
print(f"可能原因: {', '.join(reasons)}")
print()
if len(missing_sentences) > max_missing_samples:
print(f"... 还有 {len(missing_sentences) - max_missing_samples} 个缺失片段\n")
# 7. 统计缺失原因
print(f"{'='*80}")
print("缺失原因统计")
print(f"{'='*80}\n")
reason_counts = {
'版权/出版信息': 0,
'目录信息': 0,
'页码': 0,
'文本过短': 0,
'纯数字': 0,
'未知原因': 0
}
for missing in missing_sentences:
text = missing['text'].lower()
if any(kw in text for kw in ['copyright', '©', 'isbn', 'publisher', 'published']):
reason_counts['版权/出版信息'] += 1
elif any(kw in text for kw in ['table of contents', 'chapter', 'part', 'section']):
reason_counts['目录信息'] += 1
elif any(kw in text for kw in ['page', 'pg', 'p.']):
reason_counts['页码'] += 1
elif len(missing['text']) < 30:
reason_counts['文本过短'] += 1
elif re.match(r'^[0-9\s\-\.]+$', missing['text'].strip()):
reason_counts['纯数字'] += 1
else:
reason_counts['未知原因'] += 1
for reason, count in reason_counts.items():
if count > 0:
percentage = count / len(missing_sentences) * 100
print(f"{reason}: {count} 个 ({percentage:.1f}%)")
# 8. 建议
print(f"\n{'='*80}")
print("改进建议")
print(f"{'='*80}\n")
if reason_counts['文本过短'] > 0:
print(f"⚠️ 发现 {reason_counts['文本过短']} 个过短文本被过滤")
print(" 建议: 移除 min_text_length 限制,提取所有文本\n")
if reason_counts['版权/出版信息'] > 0:
print(f"📚 发现 {reason_counts['版权/出版信息']} 个版权/出版信息")
print(" 建议: 这些通常不需要翻译,可以保持过滤\n")
if reason_counts['目录信息'] > 0:
print(f"📑 发现 {reason_counts['目录信息']} 个目录信息")
print(" 建议: 目录通常需要翻译,检查是否被错误过滤\n")
if reason_counts['未知原因'] > 0:
print(f"❓ 发现 {reason_counts['未知原因']} 个未知原因的缺失")
print(" 建议: 需要详细分析这些片段\n")
def main():
"""主函数"""
logger.remove()
logger.add(sys.stderr, level="WARNING") # 只显示警告和错误
# 测试一本书
test_file = "Gambling Man.epub"
epub_path = project_root / "input" / test_file
if not epub_path.exists():
print(f"文件不存在: {test_file}")
return
analyze_missing_text(epub_path, max_missing_samples=30)
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/analyze_precise.py
+198
View File
@@ -0,0 +1,198 @@
"""
精准缺失文本分析 - 直接对比原始 HTML
不使用 Pandoc,直接分析原始 HTML 中的文本
"""
import sys
from pathlib import Path
from loguru import logger
from bs4 import BeautifulSoup
import re
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
sys.path.insert(0, str(Path(__file__).parent))
from extractors.enhanced_bs4 import EnhancedBS4Extractor
def extract_all_text_from_html(html_content: str) -> str:
"""
从 HTML 中提取所有可见文本(包括所有元素)
这是"真正的100%"基准
"""
soup = BeautifulSoup(html_content, 'html.parser')
# 移除不可见元素
for element in soup(['script', 'style', 'meta', 'link']):
element.decompose()
# 获取所有文本
text = soup.get_text(separator=' ', strip=True)
# 清理空白
text = re.sub(r'\s+', ' ', text)
return text.strip()
def compare_extraction(epub_path: Path):
"""对比提取器与真实 HTML 文本"""
print(f"\n{'='*80}")
print(f"精准文本覆盖率分析: {epub_path.name}")
print(f"{'='*80}\n")
# 加载 ePub
book = epub.read_epub(str(epub_path))
# 提取所有 HTML 文档
html_docs = []
for item in book.get_items():
if item.get_type() == 9:
try:
content = item.get_content().decode('utf-8')
html_docs.append({
'name': item.get_name(),
'content': content
})
except:
continue
print(f"找到 {len(html_docs)} 个 HTML 文档\n")
# 逐个文档分析
total_baseline_length = 0
total_extracted_length = 0
total_missing_length = 0
missing_samples = []
for doc in html_docs:
# 基准: 所有文本
baseline_text = extract_all_text_from_html(doc['content'])
# 提取器提取
extractor = EnhancedBS4Extractor(min_text_length=1, preserve_decorative=True)
items = extractor.extract(doc['content'])
# 只统计内容元素(不包括装饰性和导航)
content_items = [i for i in items if not i.get('is_decorative') and not i.get('is_navigation')]
extracted_text = " ".join([item['text'] for item in content_items])
# 统计
baseline_len = len(baseline_text)
extracted_len = len(extracted_text)
total_baseline_length += baseline_len
total_extracted_length += extracted_len
# 找出缺失的文本
if baseline_len > 0:
coverage = extracted_len / baseline_len
if coverage < 0.99: # 覆盖率 < 99%
missing_len = baseline_len - extracted_len
total_missing_length += missing_len
# 找出具体缺失的片段
baseline_words = set(baseline_text.lower().split())
extracted_words = set(extracted_text.lower().split())
missing_words = baseline_words - extracted_words
if missing_words:
missing_samples.append({
'file': doc['name'],
'baseline_length': baseline_len,
'extracted_length': extracted_len,
'coverage': coverage,
'missing_words_count': len(missing_words),
'missing_words_sample': list(missing_words)[:20]
})
# 总体统计
overall_coverage = total_extracted_length / total_baseline_length if total_baseline_length > 0 else 0
print(f"{'='*80}")
print("总体统计")
print(f"{'='*80}\n")
print(f"基准文本总长度: {total_baseline_length:,} 字符")
print(f"提取文本总长度: {total_extracted_length:,} 字符")
print(f"缺失文本长度: {total_missing_length:,} 字符")
print(f"**覆盖率: {overall_coverage:.2%}**\n")
# 显示缺失样本
if missing_samples:
print(f"{'='*80}")
print(f"发现 {len(missing_samples)} 个文档存在缺失")
print(f"{'='*80}\n")
for i, sample in enumerate(missing_samples[:10], 1):
print(f"--- 文档 {i}: {sample['file']} ---")
print(f"基准长度: {sample['baseline_length']:,} 字符")
print(f"提取长度: {sample['extracted_length']:,} 字符")
print(f"覆盖率: {sample['coverage']:.2%}")
print(f"缺失词数: {sample['missing_words_count']}")
print(f"缺失词样本: {', '.join(sample['missing_words_sample'][:10])}")
print()
if len(missing_samples) > 10:
print(f"... 还有 {len(missing_samples) - 10} 个文档\n")
else:
print("✅ 所有文档覆盖率 ≥ 99%\n")
# 详细分析第一个缺失文档
if missing_samples:
print(f"{'='*80}")
print("详细分析第一个缺失文档")
print(f"{'='*80}\n")
first_missing = missing_samples[0]
doc_content = next(d['content'] for d in html_docs if d['name'] == first_missing['file'])
# 重新提取
baseline_text = extract_all_text_from_html(doc_content)
extractor = EnhancedBS4Extractor(min_text_length=1, preserve_decorative=True)
items = extractor.extract(doc_content)
print(f"文件: {first_missing['file']}\n")
print(f"提取了 {len(items)} 个元素:")
for item in items[:20]:
item_type = ""
if item.get('is_decorative'):
item_type = " [装饰性]"
elif item.get('is_navigation'):
item_type = " [导航]"
print(f" - [{item['tag']}] {item['text'][:60]}{item_type}")
if len(items) > 20:
print(f" ... 还有 {len(items) - 20} 个元素\n")
# 显示原始 HTML 的所有文本
print(f"\n原始 HTML 的所有文本 (前 500 字符):")
print(baseline_text[:500])
print("...\n")
def main():
"""主函数"""
logger.remove()
logger.add(sys.stderr, level="ERROR")
test_file = "Gambling Man.epub"
epub_path = project_root / "input" / test_file
if not epub_path.exists():
print(f"文件不存在: {test_file}")
return
compare_extraction(epub_path)
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/calibre_cleaner.py
+268
View File
@@ -0,0 +1,268 @@
"""
Calibre ePub 清理器
清理 Calibre 生成的冗余 HTML 结构:
1. 将嵌套的 <div> 转为 <p>
2. 简化只有单一格式的 <span> (bold, italic)
3. 移除冗余的 calibre* 类
4. 合并相同的样式
"""
from bs4 import BeautifulSoup, Tag, NavigableString
from typing import Dict, Set
import re
from loguru import logger
class CalibreHTMLCleaner:
"""Calibre HTML 清理器"""
# 简单格式映射
SIMPLE_FORMAT_MAP = {
'bold': 'strong',
'italic': 'em',
'underline': 'u',
}
def __init__(self):
self.stats = {
'divs_to_p': 0,
'spans_simplified': 0,
'classes_removed': 0,
}
def clean(self, html_content: str) -> str:
"""
清理 HTML
Args:
html_content: 原始 HTML
Returns:
清理后的 HTML
"""
soup = BeautifulSoup(html_content, 'html.parser')
# 1. 将嵌套的 div 转为 p
self._convert_nested_divs_to_p(soup)
# 2. 简化格式 span
self._simplify_format_spans(soup)
# 3. 移除冗余的 span
self._remove_redundant_spans(soup)
# 4. 清理冗余的 calibre 类
self._clean_calibre_classes(soup)
logger.info(
f"清理完成: div→p {self.stats['divs_to_p']}, "
f"span简化 {self.stats['spans_simplified']}, "
f"类移除 {self.stats['classes_removed']}"
)
return str(soup)
def _convert_nested_divs_to_p(self, soup: BeautifulSoup):
"""
将嵌套的 div 转为 p
策略:
- 如果 div 只包含内联元素(span, em, strong等),转为 p
- 保留包含块级元素的 div
"""
inline_tags = {'span', 'em', 'strong', 'i', 'b', 'u', 'a', 'br'}
for div in soup.find_all('div'):
# 检查是否只包含内联元素
has_block_children = False
for child in div.children:
if isinstance(child, Tag):
if child.name not in inline_tags:
has_block_children = True
break
# 如果只包含内联元素,转为 p
if not has_block_children:
div.name = 'p'
self.stats['divs_to_p'] += 1
def _simplify_format_spans(self, soup: BeautifulSoup):
"""
简化只有单一格式的 span
例如:
<span class="calibre9"><span class="italic">Text</span></span>
→ <em>Text</em>
"""
for span in soup.find_all('span'):
# 检查 class 属性
classes = span.get('class', [])
if not classes:
continue
# 检查是否是简单格式
simple_format = None
for cls in classes:
for format_name, tag_name in self.SIMPLE_FORMAT_MAP.items():
if format_name in cls.lower():
simple_format = tag_name
break
if simple_format:
break
if simple_format:
# 替换为语义化标签
new_tag = soup.new_tag(simple_format)
# 复制内容
for child in list(span.children):
new_tag.append(child)
# 替换
span.replace_with(new_tag)
self.stats['spans_simplified'] += 1
def _remove_redundant_spans(self, soup: BeautifulSoup):
"""
移除冗余的 span
策略:
- 只移除完全没有属性的 span
- 保留有 class 的 span(即使是 calibre*)
- 确保不丢失任何文本
"""
removed_count = 0
# 只遍历一次,更保守
for span in soup.find_all('span'):
# 只移除完全没有属性的 span
if not span.attrs:
# 检查是否有文本内容
if span.get_text(strip=True):
# 有文本,安全地展开
span.unwrap()
removed_count += 1
self.stats['spans_removed'] = removed_count
def _remove_empty_elements(self, soup: BeautifulSoup):
"""
移除空元素
更谨慎的策略:
- 只删除完全没有内容的元素
- 保留有文本或图片的元素
"""
removed_count = 0
# 只遍历一次
for element in soup.find_all():
if isinstance(element, Tag):
# 检查是否完全为空
text = element.get_text(strip=True)
has_img = element.find('img') is not None
# 只删除既没有文本也没有图片的元素
if not text and not has_img:
element.decompose()
removed_count += 1
self.stats['empty_removed'] = removed_count
def _clean_calibre_classes(self, soup: BeautifulSoup):
"""
清理冗余的 calibre 类
策略:
- 保留有实际样式的类
- 移除纯数字的 calibre 类(如 calibre1, calibre2)
"""
for element in soup.find_all(class_=True):
classes = element.get('class', [])
if not classes:
continue
# 过滤掉纯数字的 calibre 类
new_classes = []
for cls in classes:
# 保留非 calibre 类
if not cls.startswith('calibre'):
new_classes.append(cls)
# 保留有语义的 calibre 类
elif any(keyword in cls.lower() for keyword in ['title', 'chapter', 'quote', 'note']):
new_classes.append(cls)
else:
self.stats['classes_removed'] += 1
if new_classes:
element['class'] = new_classes
else:
# 移除整个 class 属性
del element['class']
class CalibreEPUBCleaner:
"""Calibre ePub 清理器"""
def __init__(self):
self.html_cleaner = CalibreHTMLCleaner()
def clean_epub(self, epub_path: str, output_path: str):
"""
清理整个 ePub
Args:
epub_path: 输入 ePub 路径
output_path: 输出 ePub 路径
"""
from ebooklib import epub
logger.info(f"开始清理 ePub: {epub_path}")
# 加载 ePub
book = epub.read_epub(epub_path)
# 清理每个 HTML 文档
cleaned_count = 0
for item in book.get_items():
if item.get_type() != 9: # 不是 HTML
continue
try:
content = item.get_content().decode('utf-8')
except:
continue
# 清理 HTML
cleaned_html = self.html_cleaner.clean(content)
# 更新内容
item.set_content(cleaned_html.encode('utf-8'))
cleaned_count += 1
# 保存
epub.write_epub(output_path, book, {
'epub2_guide': False,
'epub3_landmark': False,
'epub3_pages': False,
'spine_direction': True,
})
logger.info(f"清理完成: 处理了 {cleaned_count} 个文档")
logger.info(f"输出: {output_path}")
if __name__ == "__main__":
import sys
from pathlib import Path
if len(sys.argv) < 3:
print("用法: python calibre_cleaner.py <input.epub> <output.epub>")
sys.exit(1)
input_path = sys.argv[1]
output_path = sys.argv[2]
cleaner = CalibreEPUBCleaner()
cleaner.clean_epub(input_path, output_path)
archive/v0.09/tests/extraction_experiment/check_css.py
+78
View File
@@ -0,0 +1,78 @@
"""
检查 CSS 链接保留情况
对比原始 EPUB 和清理后 EPUB 的 Head 部分
"""
import sys
from pathlib import Path
from bs4 import BeautifulSoup
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
def check_css_links(epub_path: Path):
"""检查 CSS 链接"""
print(f"\n{'='*80}")
print(f"检查 CSS 链接: {epub_path.name}")
print(f"{'='*80}\n")
book = epub.read_epub(str(epub_path))
count = 0
css_count = 0
for item in book.get_items():
if item.get_type() == 9 and 'dummy_split_002' in item.get_name():
content = item.get_content().decode('utf-8')
soup = BeautifulSoup(content, 'html.parser')
print(f"文档: {item.get_name()}\n")
# 检查 head
head = soup.find('head')
if head:
print("Head 内容:")
print(head.prettify())
links = head.find_all('link', rel='stylesheet')
if links:
print(f"\n✅ 找到 {len(links)} 个 CSS 链接")
for link in links:
print(f" - {link}")
else:
print("\n❌ 未找到 CSS 链接")
styles = head.find_all('style')
if styles:
print(f"\n✅ 找到 {len(styles)} 个 Style 标签")
for style in styles:
print(f" - {style.get_text()[:50]}...")
else:
print("\n❌ 未找到 Style 标签")
else:
print("❌ 未找到 Head 标签")
break
def main():
original_path = project_root / "input" / "On_China_Henry_Kissinger.epub"
cleaned_path = project_root / "test_output" / "On_China_cleaned.epub"
bilingual_path = project_root / "test_output" / "On_China_bilingual_test.epub"
if original_path.exists():
check_css_links(original_path)
if cleaned_path.exists():
check_css_links(cleaned_path)
if bilingual_path.exists():
check_css_links(bilingual_path)
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/create_bilingual_final.py
+124
View File
@@ -0,0 +1,124 @@
"""
创建完整的双语测试版本
使用 BS4 骨架保留方案,生成保留所有样式的双语 ePub
"""
import sys
from pathlib import Path
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
from loguru import logger
sys.path.insert(0, str(Path(__file__).parent))
from extractors.bs4_skeleton import BS4SkeletonExtractor
def create_bilingual_epub(epub_path: Path, output_path: Path, translate_toc: bool = False):
"""创建双语测试版本"""
print(f"\n{'='*80}")
print(f"创建双语测试版本: {epub_path.name}")
print(f"目录翻译: {'是' if translate_toc else '否'}")
print(f"{'='*80}\n")
# 加载 ePub
book = epub.read_epub(str(epub_path))
# 提取器
extractor = BS4SkeletonExtractor(translate_toc=translate_toc)
# 统计
total_items = 0
total_translate = 0
total_skip = 0
# 处理每个 HTML 文档
for item in book.get_items():
if item.get_type() != 9:
continue
try:
content = item.get_content().decode('utf-8')
except:
continue
file_name = item.get_name()
# 提取
items = extractor.extract(content, file_name)
if not items:
continue
# 统计
translate_items = [i for i in items if i['should_translate']]
skip_items = [i for i in items if not i['should_translate']]
total_items += len(items)
total_translate += len(translate_items)
total_skip += len(skip_items)
# 创建翻译映射
translation_map = {}
for i in items:
if i['should_translate']:
translation_map[i['text']] = f"{i['text']} [翻译]"
elif i['is_decorative']:
translation_map[i['text']] = f"{i['text']} [装饰]"
else:
translation_map[i['text']] = f"{i['text']} [跳过]"
# 回填
new_content = extractor.backfill(items, translation_map)
# 更新 item
item.set_content(new_content.encode('utf-8'))
# 关键修复: 使用 epub.write_epub 的选项参数
# 确保所有资源文件都被保存
epub.write_epub(str(output_path), book, {
'epub2_guide': False, # 不生成 guide
'epub3_landmark': False, # 不生成 landmark
'epub3_pages': False, # 不生成 pages
'spine_direction': True, # 保留 spine 方向
})
# 显示统计
print(f"处理统计:")
print(f" - 总元素: {total_items}")
print(f" - 翻译: {total_translate} ({total_translate/total_items*100:.1f}%)")
print(f" - 跳过: {total_skip} ({total_skip/total_items*100:.1f}%)")
print(f"\n✅ 双语测试版本已保存: {output_path}")
print(f"\n请在 ePub 阅读器中打开检查:")
print(f" 1. 样式是否完整保留 (居中、缩进、字体等)")
print(f" 2. 翻译是否正确回填")
print(f" 3. 是否有遗漏或错位")
def main():
"""主函数"""
logger.remove()
logger.add(sys.stderr, level="INFO")
# 测试文件
epub_path = project_root / "input" / "On_China_Henry_Kissinger.epub"
if not epub_path.exists():
print(f"❌ 文件不存在: {epub_path}")
return
# 输出目录
output_dir = project_root / "test_output"
output_dir.mkdir(exist_ok=True)
# 生成双语版本 (不翻译目录)
output_path = output_dir / "On_China_bilingual_skeleton.epub"
create_bilingual_epub(epub_path, output_path, translate_toc=False)
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/debug_cleaner.py
+53
View File
@@ -0,0 +1,53 @@
"""
调试 SimpleCleaner
"""
from bs4 import BeautifulSoup, Tag
import sys
content = """<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops" epub:prefix="z3998: http://www.daisy.org/z3998/2012/vocab/structure/#" lang="en" xml:lang="en">
<head/>
<body><div>
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="100%" height="100%" viewbox="0 0 486 751" preserveaspectratio="none">
<image width="486" height="751" xlink:href="cover.jpeg"/>
</svg>
</div>
</body>
</html>"""
print(f"Input length: {len(content)}")
try:
soup = BeautifulSoup(content, 'html.parser')
# SimpleCleaner 逻辑复现
inline_tags = {'span', 'em', 'strong', 'i', 'b', 'u', 'a', 'br'}
divs = list(soup.find_all('div'))
print(f"Found {len(divs)} divs")
for div in divs:
has_block = any(
isinstance(c, Tag) and c.name not in inline_tags
for c in div.children
)
print(f"Div content: {div}")
print(f"Has block: {has_block}")
if not has_block:
print("Converting div to p")
div.name = 'p'
# 清理 calibre 类
elements = list(soup.find_all(class_=True))
print(f"Found {len(elements)} elements with class")
result = str(soup)
print(f"Result length: {len(result)}")
print("Result preview:")
print(result[:200])
except Exception as e:
print(f"Error: {e}")
archive/v0.09/tests/extraction_experiment/debug_parser.py
+34
View File
@@ -0,0 +1,34 @@
"""
对比解析器
"""
from bs4 import BeautifulSoup
import sys
content = """<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops" epub:prefix="z3998: http://www.daisy.org/z3998/2012/vocab/structure/#" lang="en" xml:lang="en">
<head/>
<body><p>Test</p></body>
</html>"""
print("--- html.parser ---")
soup = BeautifulSoup(content, 'html.parser')
print(soup.prettify())
print("\nHead:", soup.head)
try:
print("\n--- lxml-xml ---")
soup = BeautifulSoup(content, 'lxml-xml')
print(soup.prettify())
print("\nHead:", soup.head)
except Exception as e:
print(f"\nlxml-xml error: {e}")
try:
print("\n--- lxml ---")
soup = BeautifulSoup(content, 'lxml')
print(soup.prettify())
print("\nHead:", soup.head)
except Exception as e:
print(f"\nlxml error: {e}")
archive/v0.09/tests/extraction_experiment/debug_poem.py
+45
View File
@@ -0,0 +1,45 @@
"""
测试诗歌格式问题
分析为什么诗歌格式会丢失
"""
from bs4 import BeautifulSoup
# 模拟诗歌 HTML
html = """
<div class="poem">
<div class="line1">War is</div>
<div class="line2">A grave affair of the state;</div>
<div class="line3">It is a place</div>
</div>
"""
print("原始 HTML:")
print(html)
print("\n" + "="*80 + "\n")
# 使用当前的提取器
soup = BeautifulSoup(html, 'html.parser')
# 查找所有 div
divs = soup.find_all('div')
print(f"找到 {len(divs)} 个 div:")
for i, div in enumerate(divs, 1):
print(f"{i}. <{div.name} class='{div.get('class')}'> {div.get_text()}")
print("\n" + "="*80 + "\n")
# 问题: 如果我们提取每个 div 的文本
texts = []
for div in divs:
if div.get('class') and 'line' in str(div.get('class')):
texts.append(div.get_text())
print(f"提取的文本: {texts}")
# 如果我们回填时只替换第一个文本节点...
print("\n问题演示:")
print("如果把所有文本替换成第一个元素的翻译,会导致:")
print(" - line1, line2, line3 都变成 'War is [翻译]'")
print(" - 其他内容丢失!")
archive/v0.09/tests/extraction_experiment/debug_read_diff.py
+49
View File
@@ -0,0 +1,49 @@
"""
对比 ebooklib 读取内容与 zipfile 直接读取内容
"""
import sys
from pathlib import Path
from ebooklib import epub
import zipfile
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
def compare_reads(epub_path: Path):
print(f"\n对比读取: {epub_path.name}\n")
# 1. ZipFile 读取
zip_content = {}
with zipfile.ZipFile(epub_path, 'r') as zf:
for name in zf.namelist():
if 'dummy_split_002' in name:
print(f"Zip 文件名: {name}")
content = zf.read(name).decode('utf-8')
zip_content[name] = content
print(f"Zip 内容 Head 预览:\n{content[:300]}")
break
# 2. EbookLib 读取
book = epub.read_epub(str(epub_path))
for item in book.get_items():
if 'dummy_split_002' in item.get_name():
print(f"\nItem 文件名: {item.get_name()}")
content = item.get_content().decode('utf-8')
print(f"EbookLib 内容 Head 预览:\n{content[:300]}")
# 对比
if zip_content:
zip_head = zip_content[list(zip_content.keys())[0]][:300]
if zip_head != content[:300]:
print("\n⚠️ 内容不一致!")
else:
print("\n✅ 内容一致")
break
if __name__ == "__main__":
epub_path = project_root / "input" / "On_China_Henry_Kissinger.epub"
if epub_path.exists():
compare_reads(epub_path)
archive/v0.09/tests/extraction_experiment/diagnose_epub.py
+113
View File
@@ -0,0 +1,113 @@
"""
诊断 EPUB Manifest 问题
"""
import sys
from pathlib import Path
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
def diagnose_epub(epub_path: Path):
"""诊断 EPUB 结构"""
print(f"\n{'='*80}")
print(f"诊断 EPUB: {epub_path.name}")
print(f"{'='*80}\n")
try:
book = epub.read_epub(str(epub_path))
except Exception as e:
print(f"❌ 无法读取 EPUB: {e}")
return
# 获取所有 items
all_items = list(book.get_items())
print(f"总 items 数: {len(all_items)}\n")
# 按类型分组
by_type = {}
for item in all_items:
item_type = item.get_type()
if item_type not in by_type:
by_type[item_type] = []
by_type[item_type].append(item)
print("按类型统计:")
for item_type, items in sorted(by_type.items()):
type_name = {
0: 'UNKNOWN',
1: 'IMAGE',
2: 'STYLE',
3: 'SCRIPT',
4: 'NAVIGATION',
5: 'VECTOR',
6: 'FONT',
7: 'VIDEO',
8: 'AUDIO',
9: 'DOCUMENT',
10: 'COVER'
}.get(item_type, f'TYPE_{item_type}')
print(f" {type_name}: {len(items)}")
print()
# 检查 spine
spine = book.spine
print(f"Spine 项数: {len(spine)}\n")
# 检查 titlepage
print("检查 titlepage 相关项:")
titlepage_items = [item for item in all_items if 'titlepage' in item.get_name().lower()]
if titlepage_items:
print(f" 找到 {len(titlepage_items)} 个 titlepage 项:")
for item in titlepage_items:
print(f" - {item.get_name()} (type: {item.get_type()})")
else:
print(" ❌ 未找到 titlepage 项")
print()
# 检查 spine 中的引用
print("检查 spine 引用:")
spine_refs = [ref for ref, _ in spine]
for ref in spine_refs[:10]:
# 查找对应的 item
found = False
for item in all_items:
if item.get_id() == ref:
print(f" ✓ {ref} -> {item.get_name()}")
found = True
break
if not found:
print(f" ❌ {ref} -> 未找到对应 item")
if len(spine_refs) > 10:
print(f" ... 还有 {len(spine_refs) - 10} 个引用")
print()
def main():
"""主函数"""
# 检查原始清理后的 EPUB
cleaned_path = project_root / "test_output" / "On_China_cleaned.epub"
if cleaned_path.exists():
diagnose_epub(cleaned_path)
# 检查生成的双语 EPUB
bilingual_path = project_root / "test_output" / "On_China_bilingual_test.epub"
if bilingual_path.exists():
diagnose_epub(bilingual_path)
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/dom_path_utils.py
+130
View File
@@ -0,0 +1,130 @@
"""
DOM 路径工具模块
提供 DOM 路径的生成和查找功能,用于精准定位 HTML 元素
"""
from bs4 import BeautifulSoup, Tag
from typing import Optional
from loguru import logger
class DOMPathUtils:
"""DOM 路径工具类"""
@staticmethod
def get_dom_path(element: Tag) -> str:
"""
生成元素的唯一 DOM 路径
格式: "html>body>div[0]>p[2]"
Args:
element: BeautifulSoup Tag 对象
Returns:
DOM 路径字符串
"""
if not isinstance(element, Tag):
raise ValueError("element 必须是 BeautifulSoup Tag 对象")
path_parts = []
current = element
while current and current.name:
# 跳过非标准标签(如 BeautifulSoup 的 [document])
if current.name in ['[document]', 'html']:
current = current.parent
continue
# 获取同名兄弟元素
parent = current.parent
if parent:
siblings = [
sibling for sibling in parent.children
if isinstance(sibling, Tag) and sibling.name == current.name
]
# 找到当前元素在同名兄弟中的索引
try:
index = siblings.index(current)
except ValueError:
# 如果找不到,使用 0
index = 0
path_parts.append(f"{current.name}[{index}]")
else:
# 根元素
if current.name not in ['[document]', 'html']:
path_parts.append(current.name)
current = parent
# 反转路径(从根到叶)
return ">".join(reversed(path_parts))
@staticmethod
def find_by_path(soup: BeautifulSoup, path: str) -> Optional[Tag]:
"""
通过 DOM 路径查找元素
Args:
soup: BeautifulSoup 对象
path: DOM 路径字符串
Returns:
找到的元素,如果未找到则返回 None
"""
try:
parts = path.split(">")
current = soup
for part in parts:
# 解析标签名和索引
if "[" in part:
tag_name, index_str = part.split("[")
index = int(index_str.rstrip("]"))
else:
# 根元素可能没有索引
tag_name = part
index = 0
# 查找所有同名标签
if isinstance(current, BeautifulSoup):
# 从根开始
candidates = [current.find(tag_name)]
else:
candidates = current.find_all(tag_name, recursive=False)
if not candidates or index >= len(candidates):
logger.warning(f"路径查找失败: {path} (在 {part} 处)")
return None
current = candidates[index]
return current if isinstance(current, Tag) else None
except Exception as e:
logger.error(f"路径解析错误 {path}: {e}")
return None
@staticmethod
def validate_path(soup: BeautifulSoup, path: str, original_element: Tag) -> bool:
"""
验证路径是否能正确定位到原始元素
Args:
soup: BeautifulSoup 对象
path: DOM 路径
original_element: 原始元素
Returns:
是否验证成功
"""
found = DOMPathUtils.find_by_path(soup, path)
if found is None:
return False
# 比较元素的文本内容和标签名
return (found.name == original_element.name and
found.get_text(strip=True) == original_element.get_text(strip=True))
archive/v0.09/tests/extraction_experiment/extractors/__init__.py
+1
View File
@@ -0,0 +1 @@
"""提取器模块"""
archive/v0.09/tests/extraction_experiment/extractors/baseline_pandoc.py
+157
View File
@@ -0,0 +1,157 @@
"""
Pandoc 基准提取器
使用 pandoc 将 ePub 转换为 Markdown,作为文本提取的参考基准
"""
import subprocess
import tempfile
from pathlib import Path
from typing import Optional
from loguru import logger
class PandocBaseline:
"""Pandoc 基准提取器"""
def __init__(self):
"""初始化,检查 pandoc 是否可用"""
self.pandoc_available = self._check_pandoc()
def _check_pandoc(self) -> bool:
"""检查 pandoc 是否安装"""
try:
result = subprocess.run(
['pandoc', '--version'],
capture_output=True,
text=True,
timeout=5
)
if result.returncode == 0:
logger.info(f"Pandoc 可用: {result.stdout.split()[1]}")
return True
except Exception as e:
logger.warning(f"Pandoc 不可用: {e}")
return False
def extract_from_epub(self, epub_path: str) -> Optional[str]:
"""
使用 pandoc 从 ePub 提取文本
Args:
epub_path: ePub 文件路径
Returns:
提取的 Markdown 文本,如果失败返回 None
"""
if not self.pandoc_available:
logger.error("Pandoc 不可用,无法提取基准文本")
return None
epub_path = Path(epub_path)
if not epub_path.exists():
logger.error(f"ePub 文件不存在: {epub_path}")
return None
try:
# 创建临时输出文件
with tempfile.NamedTemporaryFile(mode='w', suffix='.md', delete=False) as tmp:
tmp_path = tmp.name
# 运行 pandoc
cmd = [
'pandoc',
str(epub_path),
'-t', 'markdown',
'-o', tmp_path,
'--wrap=none' # 不自动换行
]
logger.info(f"运行 pandoc: {' '.join(cmd)}")
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=60
)
if result.returncode != 0:
logger.error(f"Pandoc 执行失败: {result.stderr}")
return None
# 读取结果
with open(tmp_path, 'r', encoding='utf-8') as f:
markdown_text = f.read()
# 清理临时文件
Path(tmp_path).unlink()
logger.info(f"Pandoc 提取成功: {len(markdown_text)} 字符")
return markdown_text
except subprocess.TimeoutExpired:
logger.error("Pandoc 执行超时")
return None
except Exception as e:
logger.error(f"Pandoc 提取失败: {e}")
return None
def extract_from_html(self, html_content: str) -> Optional[str]:
"""
使用 pandoc 从 HTML 提取文本
Args:
html_content: HTML 字符串
Returns:
提取的 Markdown 文本
"""
if not self.pandoc_available:
return None
try:
# 创建临时 HTML 文件
with tempfile.NamedTemporaryFile(mode='w', suffix='.html', delete=False, encoding='utf-8') as tmp_html:
tmp_html.write(html_content)
tmp_html_path = tmp_html.name
# 创建临时输出文件
with tempfile.NamedTemporaryFile(mode='w', suffix='.md', delete=False) as tmp_md:
tmp_md_path = tmp_md.name
# 运行 pandoc
cmd = [
'pandoc',
tmp_html_path,
'-f', 'html',
'-t', 'markdown',
'-o', tmp_md_path,
'--wrap=none'
]
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=30
)
if result.returncode != 0:
logger.error(f"Pandoc HTML 转换失败: {result.stderr}")
return None
# 读取结果
with open(tmp_md_path, 'r', encoding='utf-8') as f:
markdown_text = f.read()
# 清理临时文件
Path(tmp_html_path).unlink()
Path(tmp_md_path).unlink()
return markdown_text
except Exception as e:
logger.error(f"Pandoc HTML 提取失败: {e}")
return None
archive/v0.09/tests/extraction_experiment/extractors/bs4_optimized.py
+207
View File
@@ -0,0 +1,207 @@
"""
优化的 BeautifulSoup 提取器
使用 DOM 路径标识系统,实现完整的文本提取和精准回填
"""
from bs4 import BeautifulSoup, Tag, NavigableString
from typing import List, Dict, Any, Set
import re
from loguru import logger
import sys
from pathlib import Path
# 添加父目录到路径
sys.path.insert(0, str(Path(__file__).parent.parent))
from dom_path_utils import DOMPathUtils
class BS4OptimizedExtractor:
"""优化的 BS4 提取器"""
# 标准块级标签
BLOCK_TAGS = [
'p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
'blockquote', 'li', 'td', 'dd', 'dt', 'figcaption',
'section', 'article', 'aside', 'header', 'footer', 'main'
]
# 导航相关的 class 关键词
NAV_KEYWORDS = [
'nav', 'navigation', 'toc', 'menu', 'header', 'footer',
'page-number', 'page-num', 'sidebar'
]
def __init__(self, min_text_length: int = 10):
"""
初始化提取器
Args:
min_text_length: 最小文本长度,过滤过短的文本
"""
self.min_text_length = min_text_length
self.path_utils = DOMPathUtils()
def extract(self, html_content: str) -> List[Dict[str, Any]]:
"""
从 HTML 中提取所有文本元素
Args:
html_content: HTML 字符串
Returns:
提取的元素列表,每个元素包含:
- path: DOM 路径
- element: BeautifulSoup Tag 对象
- text: 清理后的文本
- html: 原始 HTML
- tag: 标签名
- is_navigation: 是否是导航元素
"""
soup = BeautifulSoup(html_content, 'html.parser')
# 移除不需要的元素
for element in soup(['script', 'style', 'meta', 'link']):
element.decompose()
items = []
processed_ids = set()
# 遍历所有块级元素
for element in soup.find_all(self.BLOCK_TAGS):
elem_id = id(element)
# 避免重复处理
if elem_id in processed_ids:
continue
# 检查是否被已处理的父元素包含
if self._is_contained_in_processed(element, processed_ids):
continue
# 提取文本
text = self._clean_text(element)
# 过滤过短的文本
if len(text.strip()) < self.min_text_length:
continue
# 生成 DOM 路径
path = self.path_utils.get_dom_path(element)
# 判断是否是导航元素
is_nav = self._is_navigation_element(element)
items.append({
'path': path,
'element': element,
'text': text,
'html': str(element),
'tag': element.name,
'is_navigation': is_nav
})
processed_ids.add(elem_id)
logger.info(f"提取了 {len(items)} 个文本元素")
return items
def _is_contained_in_processed(self, element: Tag, processed_ids: Set[int]) -> bool:
"""检查元素是否被已处理的父元素包含"""
for parent in element.parents:
if isinstance(parent, Tag) and id(parent) in processed_ids:
return True
return False
def _clean_text(self, element: Tag) -> str:
"""
清理元素文本
移除:
- 脚注引用
- 仅包含数字的 span
- 多余空白
"""
# 创建副本避免修改原始元素
element_copy = BeautifulSoup(str(element), 'html.parser').find(element.name)
if not element_copy:
return ""
# 移除脚注引用
for tag in element_copy.find_all(['sup', 'sub']):
tag.decompose()
footnote_patterns = re.compile(r'footnote|endnote|reference|note|super|sub', re.I)
for tag in element_copy.find_all(['a', 'span', 'div'], class_=footnote_patterns):
tag.decompose()
# 移除仅包含数字的 span
for tag in element_copy.find_all('span'):
if re.match(r'^(\[\d+\]|\(\d+\)|\d+)$', tag.get_text().strip()):
tag.decompose()
text = element_copy.get_text().strip()
# 清理残留引用标识
text = re.sub(r'(\.|。|,|，)\s*(\[\d+\]|\d+)(?=\s|$)', r'\1', text)
text = re.sub(r'\s+', ' ', text)
return text
def _is_navigation_element(self, element: Tag) -> bool:
"""判断是否是导航元素"""
# 检查元素自身的 class
classes = element.get('class', [])
class_str = ' '.join(classes).lower() if isinstance(classes, list) else str(classes).lower()
if any(keyword in class_str for keyword in self.NAV_KEYWORDS):
return True
# 检查父元素的 class
parent = element.parent
if parent and isinstance(parent, Tag):
p_classes = parent.get('class', [])
p_class_str = ' '.join(p_classes).lower() if isinstance(p_classes, list) else str(p_classes).lower()
if any(keyword in p_class_str for keyword in self.NAV_KEYWORDS):
return True
return False
def backfill(self, html_content: str, translation_map: Dict[str, str]) -> str:
"""
使用 DOM 路径精准回填翻译
Args:
html_content: 原始 HTML
translation_map: {dom_path: translation} 映射
Returns:
回填后的 HTML 字符串
"""
soup = BeautifulSoup(html_content, 'html.parser')
success_count = 0
fail_count = 0
for path, translation in translation_map.items():
element = self.path_utils.find_by_path(soup, path)
if element is None:
logger.warning(f"回填失败: 未找到路径 {path}")
fail_count += 1
continue
# 创建新元素(这里简化处理,实际应根据模式创建)
new_tag = soup.new_tag(element.name)
new_tag.string = translation
# 复制属性
for attr, value in element.attrs.items():
new_tag[attr] = value
# 替换
element.replace_with(new_tag)
success_count += 1
logger.info(f"回填完成: 成功 {success_count}, 失败 {fail_count}")
return str(soup)
archive/v0.09/tests/extraction_experiment/extractors/bs4_skeleton.py
+256
View File
@@ -0,0 +1,256 @@
"""
BS4 骨架保留提取器
核心思想:
1. 提取: 保留元素引用,提取纯文本
2. 回填: 只替换文本节点,保留所有 HTML 结构和属性
"""
from bs4 import BeautifulSoup, Tag, NavigableString
from typing import List, Dict, Any
import re
from loguru import logger
class BS4SkeletonExtractor:
"""BS4 骨架保留提取器"""
SKIP_TRANSLATION_PATTERNS = [
r'index\.x?html',
r'bibliography\.x?html',
r'endnotes?\.x?html',
r'footnotes?\.x?html',
]
TOC_PATTERNS = [
r'nav\.x?html',
r'toc\.x?html',
]
OTHER_NON_CORE_PATTERNS = [
r'copyright\.x?html',
r'title\.x?html',
r'cover\.x?html',
]
BLOCK_TAGS = [
'p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
'blockquote', 'li', 'td', 'dd', 'dt', 'figcaption',
'section', 'article', 'aside', 'header', 'footer', 'main'
]
DECORATIVE_PATTERNS = [
r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾]+$',
r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾\s]+$',
]
def __init__(self, translate_toc: bool = False):
self.translate_toc = translate_toc
self.soup = None # 保存 soup 引用
def extract(self, html_content: str, file_name: str = "") -> List[Dict[str, Any]]:
"""
提取文本,保留元素引用
关键: 返回的 items 中包含对原始元素的引用
"""
self.soup = BeautifulSoup(html_content, 'html.parser')
# 移除不需要的元素
for element in self.soup(['script', 'style', 'meta', 'link']):
element.decompose()
doc_type = self._classify_document(file_name)
items = []
processed_ids = set()
for element in self.soup.find_all(self.BLOCK_TAGS):
elem_id = id(element)
if elem_id in processed_ids:
continue
if self._is_contained_in_processed(element, processed_ids):
continue
# 提取纯文本
text = element.get_text(separator=' ', strip=True)
if not text.strip():
continue
is_decorative = self._is_decorative(text)
should_translate = self._should_translate(doc_type, is_decorative)
# 关键: 保存元素引用,不是字符串!
items.append({
'element': element, # 保存元素引用
'text': text,
'should_translate': should_translate,
'doc_type': doc_type,
'is_decorative': is_decorative,
'tag': element.name
})
processed_ids.add(elem_id)
logger.info(
f"[{doc_type}] 提取 {len(items)} 个元素: "
f"翻译 {sum(1 for i in items if i['should_translate'])}"
)
return items
def backfill(self, items: List[Dict[str, Any]], translation_map: Dict[str, str]) -> str:
"""
回填翻译,保留完整的 HTML 结构
Args:
items: extract() 返回的元素列表
translation_map: {original_text: translated_text}
Returns:
回填后的完整 HTML
"""
success_count = 0
for item in items:
element = item['element']
original_text = item['text']
# 查找翻译
translation = translation_map.get(original_text)
if translation is None:
continue
# 关键: 只替换文本节点,保留所有子元素和属性
self._replace_text_only(element, translation)
success_count += 1
logger.info(f"回填完成: 成功 {success_count}/{len(translation_map)}")
# 返回完整的 HTML
return str(self.soup)
def _replace_text_only(self, element: Tag, new_text: str):
"""
只替换元素的文本内容,完全保留 HTML 结构
关键策略:
1. 只处理当前元素,不影响其他元素
2. 保留所有子元素(span, em, strong等)
3. 只替换直接的文本节点
示例:
原始: <div><span class="bold">Text</span></div>
翻译: "Text [翻译]"
结果: <div><span class="bold">Text [翻译]</span></div>
"""
from bs4 import NavigableString, Comment
# 检查元素是否有子标签
child_tags = [child for child in element.children if isinstance(child, Tag)]
if not child_tags:
# 情况1: 元素只包含文本,没有子标签
# 例如: <div>Simple text</div>
element.clear()
element.string = new_text
else:
# 情况2: 元素包含子标签
# 例如: <div><span class="bold">Text</span> more text</div>
# 策略: 找到最深层的文本节点,替换它
# 这样可以保留所有格式标签
# 递归查找最深的包含文本的元素
deepest = self._find_deepest_text_element(element)
if deepest and deepest != element:
# 在最深的元素中替换文本
deepest.clear()
deepest.string = new_text
else:
# 没有更深的元素,直接替换当前元素的所有内容
element.clear()
element.string = new_text
def _find_deepest_text_element(self, element: Tag) -> Tag:
"""
递归查找最深的包含文本的元素
返回包含实际文本内容的最深层元素
"""
from bs4 import NavigableString
# 查找所有子标签
child_tags = [child for child in element.children if isinstance(child, Tag)]
if not child_tags:
# 没有子标签,这就是最深的元素
return element
# 有子标签,递归查找
# 优先查找第一个包含文本的子标签
for child in child_tags:
if child.get_text().strip():
return self._find_deepest_text_element(child)
# 所有子标签都没有文本,返回当前元素
return element
def _classify_document(self, file_name: str) -> str:
if not file_name:
return 'core'
file_name_lower = file_name.lower()
for pattern in self.SKIP_TRANSLATION_PATTERNS:
if re.search(pattern, file_name_lower):
return 'skip'
for pattern in self.TOC_PATTERNS:
if re.search(pattern, file_name_lower):
return 'toc'
for pattern in self.OTHER_NON_CORE_PATTERNS:
if re.search(pattern, file_name_lower):
return 'other'
return 'core'
def _should_translate(self, doc_type: str, is_decorative: bool) -> bool:
if is_decorative:
return False
if doc_type == 'core':
return True
if doc_type == 'toc':
return self.translate_toc
return False
def _is_contained_in_processed(self, element: Tag, processed_ids: set) -> bool:
for parent in element.parents:
if isinstance(parent, Tag) and id(parent) in processed_ids:
return True
return False
def _is_decorative(self, text: str) -> bool:
text_stripped = text.strip()
if not text_stripped or len(text_stripped) > 20:
return False
for pattern in self.DECORATIVE_PATTERNS:
if re.match(pattern, text_stripped):
return True
unique_chars = set(text_stripped.replace(' ', ''))
if len(unique_chars) <= 3:
decorative_chars = set('*-_~•◆◇■□▪▫●○◉◎⦿⦾')
if unique_chars & decorative_chars:
return True
return False
archive/v0.09/tests/extraction_experiment/extractors/enhanced_bs4.py
+276
View File
@@ -0,0 +1,276 @@
"""
增强的提取器 - 保留装饰性元素
在原有 BS4 优化方案基础上,增强对装饰性符号和特殊元素的提取
"""
from bs4 import BeautifulSoup, Tag, NavigableString
from typing import List, Dict, Any, Set
import re
from loguru import logger
import sys
from pathlib import Path
# 添加父目录到路径
sys.path.insert(0, str(Path(__file__).parent.parent))
from dom_path_utils import DOMPathUtils
class EnhancedBS4Extractor:
"""增强的 BS4 提取器 - 保留装饰性元素"""
# 标准块级标签
BLOCK_TAGS = [
'p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
'blockquote', 'li', 'td', 'dd', 'dt', 'figcaption',
'section', 'article', 'aside', 'header', 'footer', 'main'
]
# 可能包含装饰性符号的标签
DECORATIVE_TAGS = [
'hr', # 水平线
'div', # 可能包含装饰性符号的 div
'p', # 可能只包含符号的段落
'span' # 装饰性 span
]
# 导航相关的 class 关键词
NAV_KEYWORDS = [
'nav', 'navigation', 'toc', 'menu', 'header', 'footer',
'page-number', 'page-num', 'sidebar'
]
# 装饰性符号的正则模式
DECORATIVE_PATTERNS = [
r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾]+$', # 纯符号
r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾\s]+$', # 符号+空白
r'^[\u2022-\u2027\u2030-\u205E]+$', # Unicode 装饰符号
]
def __init__(self, min_text_length: int = 10, preserve_decorative: bool = True):
"""
初始化提取器
Args:
min_text_length: 最小文本长度(装饰性元素不受此限制)
preserve_decorative: 是否保留装饰性元素
"""
self.min_text_length = min_text_length
self.preserve_decorative = preserve_decorative
self.path_utils = DOMPathUtils()
def extract(self, html_content: str) -> List[Dict[str, Any]]:
"""
从 HTML 中提取所有文本元素,包括装饰性元素
Args:
html_content: HTML 字符串
Returns:
提取的元素列表
"""
soup = BeautifulSoup(html_content, 'html.parser')
# 移除不需要的元素
for element in soup(['script', 'style', 'meta', 'link']):
element.decompose()
items = []
processed_ids = set()
# 1. 提取标准块级元素
for element in soup.find_all(self.BLOCK_TAGS):
elem_id = id(element)
if elem_id in processed_ids:
continue
if self._is_contained_in_processed(element, processed_ids):
continue
text = self._clean_text(element)
# 检查是否是装饰性元素
is_decorative = self._is_decorative_element(element, text)
# 过滤逻辑
if not is_decorative and len(text.strip()) < self.min_text_length:
continue
# 如果是装饰性元素但不保留,跳过
if is_decorative and not self.preserve_decorative:
continue
path = self.path_utils.get_dom_path(element)
is_nav = self._is_navigation_element(element)
items.append({
'path': path,
'element': element,
'text': text,
'html': str(element),
'tag': element.name,
'is_navigation': is_nav,
'is_decorative': is_decorative
})
processed_ids.add(elem_id)
# 2. 提取 <hr> 等纯装饰性标签
if self.preserve_decorative:
for hr in soup.find_all('hr'):
elem_id = id(hr)
if elem_id not in processed_ids:
path = self.path_utils.get_dom_path(hr)
items.append({
'path': path,
'element': hr,
'text': '---', # 用文本表示水平线
'html': str(hr),
'tag': 'hr',
'is_navigation': False,
'is_decorative': True
})
processed_ids.add(elem_id)
logger.info(f"提取了 {len(items)} 个文本元素 (包含 {sum(1 for i in items if i.get('is_decorative'))} 个装饰性元素)")
return items
def _is_decorative_element(self, element: Tag, text: str) -> bool:
"""
判断元素是否是装饰性元素
装饰性元素的特征:
1. 只包含符号(如 ***, ---, •••)
2. 文本很短但包含特殊 Unicode 符号
3. 有特定的 class (如 'separator', 'divider')
"""
# 检查 class
classes = element.get('class', [])
class_str = ' '.join(classes).lower() if isinstance(classes, list) else str(classes).lower()
decorative_classes = ['separator', 'divider', 'ornament', 'decoration', 'break']
if any(dc in class_str for dc in decorative_classes):
return True
# 检查文本是否匹配装饰性模式
text_stripped = text.strip()
if not text_stripped:
return False
for pattern in self.DECORATIVE_PATTERNS:
if re.match(pattern, text_stripped):
return True
# 检查是否只包含少量重复字符
if len(text_stripped) <= 20:
unique_chars = set(text_stripped.replace(' ', ''))
if len(unique_chars) <= 3: # 只有1-3种不同字符
# 检查是否是常见装饰符号
decorative_chars = set('*-_~•◆◇■□▪▫●○◉◎⦿⦾')
if unique_chars & decorative_chars:
return True
return False
def _is_contained_in_processed(self, element: Tag, processed_ids: Set[int]) -> bool:
"""检查元素是否被已处理的父元素包含"""
for parent in element.parents:
if isinstance(parent, Tag) and id(parent) in processed_ids:
return True
return False
def _clean_text(self, element: Tag) -> str:
"""
清理元素文本
注意: 对于装饰性元素,保留原始符号
"""
# 创建副本
element_copy = BeautifulSoup(str(element), 'html.parser').find(element.name)
if not element_copy:
return ""
# 移除脚注引用(但保留装饰性符号)
for tag in element_copy.find_all(['sup', 'sub']):
tag.decompose()
footnote_patterns = re.compile(r'footnote|endnote|reference|note|super|sub', re.I)
for tag in element_copy.find_all(['a', 'span', 'div'], class_=footnote_patterns):
tag.decompose()
# 移除仅包含数字的 span
for tag in element_copy.find_all('span'):
if re.match(r'^(\[\d+\]|\(\d+\)|\d+)$', tag.get_text().strip()):
tag.decompose()
text = element_copy.get_text().strip()
# 清理残留引用标识
text = re.sub(r'(\.|。|,|，)\s*(\[\d+\]|\d+)(?=\s|$)', r'\1', text)
# 对于非装饰性文本,压缩空白
# 对于装饰性文本,保留原样
if not self._is_decorative_element(element, text):
text = re.sub(r'\s+', ' ', text)
return text
def _is_navigation_element(self, element: Tag) -> bool:
"""判断是否是导航元素"""
classes = element.get('class', [])
class_str = ' '.join(classes).lower() if isinstance(classes, list) else str(classes).lower()
if any(keyword in class_str for keyword in self.NAV_KEYWORDS):
return True
parent = element.parent
if parent and isinstance(parent, Tag):
p_classes = parent.get('class', [])
p_class_str = ' '.join(p_classes).lower() if isinstance(p_classes, list) else str(p_classes).lower()
if any(keyword in p_class_str for keyword in self.NAV_KEYWORDS):
return True
return False
def backfill(self, html_content: str, translation_map: Dict[str, str]) -> str:
"""
使用 DOM 路径精准回填翻译
对于装饰性元素,保持原样不翻译
"""
soup = BeautifulSoup(html_content, 'html.parser')
success_count = 0
fail_count = 0
decorative_kept = 0
for path, translation in translation_map.items():
element = self.path_utils.find_by_path(soup, path)
if element is None:
logger.warning(f"回填失败: 未找到路径 {path}")
fail_count += 1
continue
# 检查是否是装饰性元素
original_text = element.get_text().strip()
if self._is_decorative_element(element, original_text):
# 装饰性元素保持原样
decorative_kept += 1
continue
# 创建新元素
new_tag = soup.new_tag(element.name)
new_tag.string = translation
# 复制属性
for attr, value in element.attrs.items():
new_tag[attr] = value
# 替换
element.replace_with(new_tag)
success_count += 1
logger.info(f"回填完成: 成功 {success_count}, 失败 {fail_count}, 装饰性元素保留 {decorative_kept}")
return str(soup)
archive/v0.09/tests/extraction_experiment/extractors/final_extractor.py
+319
View File
@@ -0,0 +1,319 @@
"""
最终版智能提取器
明确的翻译策略:
1. 正文: 100% 翻译
2. 目录: 全翻译或全不翻译 (根据配置)
3. 索引/参考文献/尾注: 明确不翻译
4. 装饰性元素: 不翻译
"""
from bs4 import BeautifulSoup, Tag
from typing import List, Dict, Any, Set
import re
from loguru import logger
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent))
from dom_path_utils import DOMPathUtils
class FinalExtractor:
"""最终版智能提取器"""
# 明确不翻译的文档
SKIP_TRANSLATION_PATTERNS = [
r'index\.x?html', # 索引
r'bibliography\.x?html', # 参考文献
r'endnotes?\.x?html', # 尾注
r'footnotes?\.x?html', # 脚注
]
# 目录文档 (可配置是否翻译)
TOC_PATTERNS = [
r'nav\.x?html',
r'toc\.x?html',
]
# 其他非核心文档 (通常不翻译)
OTHER_NON_CORE_PATTERNS = [
r'copyright\.x?html',
r'title\.x?html',
r'cover\.x?html',
]
BLOCK_TAGS = [
'p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
'blockquote', 'li', 'td', 'dd', 'dt', 'figcaption',
'section', 'article', 'aside', 'header', 'footer', 'main'
]
DECORATIVE_PATTERNS = [
r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾]+$',
r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾\s]+$',
r'^[\u2022-\u2027\u2030-\u205E]+$',
]
def __init__(self, translate_toc: bool = False, preserve_decorative: bool = True):
"""
初始化提取器
Args:
translate_toc: 是否翻译目录 (默认不翻译)
preserve_decorative: 是否保留装饰性元素
"""
self.translate_toc = translate_toc
self.preserve_decorative = preserve_decorative
self.path_utils = DOMPathUtils()
def extract(self, html_content: str, file_name: str = "") -> List[Dict[str, Any]]:
"""
提取文本元素
每个 DOM 节点作为一个独立单位,不合并
"""
soup = BeautifulSoup(html_content, 'html.parser')
# 移除不需要的元素
for element in soup(['script', 'style', 'meta', 'link']):
element.decompose()
# 判断文档类型
doc_type = self._classify_document(file_name)
items = []
processed_ids = set()
# 遍历所有块级元素
for element in soup.find_all(self.BLOCK_TAGS):
elem_id = id(element)
if elem_id in processed_ids:
continue
if self._is_contained_in_processed(element, processed_ids):
continue
# 提取文本 (100% 完整)
text = element.get_text(separator=' ', strip=True)
if not text.strip():
continue
# 判断是否装饰性
is_decorative = self._is_decorative_element(element, text)
# 生成路径
path = self.path_utils.get_dom_path(element)
# 决定是否翻译
should_translate = self._should_translate(doc_type, is_decorative)
items.append({
'path': path,
'element': element,
'text': text,
'html': str(element),
'tag': element.name,
'is_decorative': is_decorative,
'doc_type': doc_type,
'should_translate': should_translate,
'file_name': file_name
})
processed_ids.add(elem_id)
# 添加 <hr>
if self.preserve_decorative:
for hr in soup.find_all('hr'):
elem_id = id(hr)
if elem_id not in processed_ids:
path = self.path_utils.get_dom_path(hr)
items.append({
'path': path,
'element': hr,
'text': '---',
'html': str(hr),
'tag': 'hr',
'is_decorative': True,
'doc_type': doc_type,
'should_translate': False,
'file_name': file_name
})
processed_ids.add(elem_id)
# 统计
translate_count = sum(1 for i in items if i['should_translate'])
skip_count = sum(1 for i in items if not i['should_translate'] and not i['is_decorative'])
decorative_count = sum(1 for i in items if i['is_decorative'])
logger.info(
f"[{doc_type}] 提取 {len(items)} 个元素: "
f"翻译 {translate_count}, 跳过 {skip_count}, 装饰 {decorative_count}"
)
return items
def _classify_document(self, file_name: str) -> str:
"""
分类文档类型
Returns:
'core' - 核心正文
'toc' - 目录
'skip' - 明确跳过 (索引/参考文献/尾注)
'other' - 其他非核心
"""
if not file_name:
return 'core'
file_name_lower = file_name.lower()
# 检查是否是明确跳过的
for pattern in self.SKIP_TRANSLATION_PATTERNS:
if re.search(pattern, file_name_lower):
return 'skip'
# 检查是否是目录
for pattern in self.TOC_PATTERNS:
if re.search(pattern, file_name_lower):
return 'toc'
# 检查其他非核心
for pattern in self.OTHER_NON_CORE_PATTERNS:
if re.search(pattern, file_name_lower):
return 'other'
return 'core'
def _should_translate(self, doc_type: str, is_decorative: bool) -> bool:
"""
决定是否翻译
规则:
1. 装饰性: 不翻译
2. core: 翻译
3. toc: 根据配置
4. skip: 不翻译
5. other: 不翻译
"""
if is_decorative:
return False
if doc_type == 'core':
return True
if doc_type == 'toc':
return self.translate_toc
# skip 和 other 都不翻译
return False
def _is_contained_in_processed(self, element: Tag, processed_ids: Set[int]) -> bool:
"""检查元素是否被已处理的父元素包含"""
for parent in element.parents:
if isinstance(parent, Tag) and id(parent) in processed_ids:
return True
return False
def _is_decorative_element(self, element: Tag, text: str) -> bool:
"""判断是否是装饰性元素"""
classes = element.get('class', [])
class_str = ' '.join(classes).lower() if isinstance(classes, list) else str(classes).lower()
decorative_classes = ['separator', 'divider', 'ornament', 'decoration', 'break']
if any(dc in class_str for dc in decorative_classes):
return True
text_stripped = text.strip()
if not text_stripped:
return False
for pattern in self.DECORATIVE_PATTERNS:
if re.match(pattern, text_stripped):
return True
if len(text_stripped) <= 20:
unique_chars = set(text_stripped.replace(' ', ''))
if len(unique_chars) <= 3:
decorative_chars = set('*-_~•◆◇■□▪▫●○◉◎⦿⦾')
if unique_chars & decorative_chars:
return True
return False
def backfill(self, html_content: str, translation_map: Dict[str, str]) -> str:
"""
精准回填翻译
保留原始 HTML 结构和样式,只替换文本内容
"""
soup = BeautifulSoup(html_content, 'html.parser')
success_count = 0
fail_count = 0
for path, translation in translation_map.items():
element = self.path_utils.find_by_path(soup, path)
if element is None:
logger.warning(f"回填失败: 未找到路径 {path}")
fail_count += 1
continue
# 保留原始元素结构,只替换文本节点
self._replace_text_nodes(element, translation)
success_count += 1
logger.info(f"回填完成: 成功 {success_count}, 失败 {fail_count}")
return str(soup)
def _replace_text_nodes(self, element: Tag, new_text: str):
"""
智能替换元素中的文本节点,完全保留 HTML 结构
策略:
1. 如果元素只包含纯文本(无子标签),直接替换
2. 如果元素包含子标签,递归查找并替换所有文本节点
3. 保留所有属性、class、style 等
"""
from bs4 import NavigableString
# 检查是否有子标签
child_tags = [child for child in element.children if isinstance(child, Tag)]
if not child_tags:
# 只有文本节点,直接替换
element.clear()
element.string = new_text
else:
# 有子标签,需要智能处理
# 策略: 找到所有文本节点,用新文本替换
self._replace_all_text_nodes(element, new_text)
def _replace_all_text_nodes(self, element: Tag, new_text: str):
"""
递归替换元素中的所有文本节点
保留所有子元素和属性,只替换文本内容
"""
from bs4 import NavigableString
# 收集所有文本节点
text_nodes = []
for child in element.descendants:
if isinstance(child, NavigableString) and not isinstance(child, (type(None),)):
# 跳过空白文本
if child.strip():
text_nodes.append(child)
if not text_nodes:
# 没有文本节点,直接设置
element.string = new_text
return
# 简化策略: 清空所有内容,保留结构,设置新文本
# 这会丢失内部格式,但保留外层容器的所有属性
element.clear()
element.string = new_text
archive/v0.09/tests/extraction_experiment/extractors/fine_grained.py
+232
View File
@@ -0,0 +1,232 @@
"""
细粒度提取器
策略: 提取所有 <p> 元素,每个独立处理
"""
from bs4 import BeautifulSoup, Tag, NavigableString
from typing import List, Dict, Any, Tuple
import re
from loguru import logger
class FineGrainedExtractor:
"""细粒度提取器 - 每个 p 元素独立提取"""
SKIP_TRANSLATION_PATTERNS = [
r'index\.x?html',
r'bibliography\.x?html',
r'endnotes?\.x?html',
r'footnotes?\.x?html',
]
TOC_PATTERNS = [
r'nav\.x?html',
r'toc\.x?html',
]
def __init__(self, translate_toc: bool = False):
self.translate_toc = translate_toc
self.soup = None
def extract(self, html_content: str, file_name: str = "") -> List[Dict[str, Any]]:
"""
细粒度提取: 每个 <p> 和标题元素独立提取
关键: 不管嵌套,所有 <p>, h1-h6 都提取
"""
self.soup = BeautifulSoup(html_content, 'html.parser')
# 移除不需要的元素
for element in self.soup(['script', 'style', 'meta', 'link']):
element.decompose()
doc_type = self._classify_document(file_name)
items = []
# 提取所有文本块元素
target_tags = ['p', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6']
for p in self.soup.find_all(target_tags):
# 提取文本
text = p.get_text(separator=' ', strip=True)
if not text.strip():
continue
# 收集所有文本节点
text_nodes = self._collect_text_nodes(p)
if not text_nodes:
continue
is_decorative = self._is_decorative(text)
should_translate = self._should_translate(doc_type, is_decorative, text)
items.append({
'element': p,
'text': text,
'text_nodes': text_nodes,
'should_translate': should_translate,
'doc_type': doc_type,
'is_decorative': is_decorative,
'tag': p.name
})
logger.info(
f"[{doc_type}] 提取 {len(items)} 个元素 (p, h1-h6): "
f"翻译 {sum(1 for i in items if i['should_translate'])}"
)
return items
def _collect_text_nodes(self, element: Tag) -> List[Tuple[NavigableString, str]]:
"""
收集元素中的所有文本节点
"""
text_nodes = []
for descendant in element.descendants:
if isinstance(descendant, NavigableString):
# 跳过注释
if isinstance(descendant, type(element)):
continue
text = str(descendant).strip()
if text:
text_nodes.append((descendant, text))
return text_nodes
def backfill(self, items: List[Dict[str, Any]], translation_map: Dict[str, str], bilingual: bool = True) -> str:
"""
回填翻译
Args:
items: 提取的元素列表
translation_map: 翻译映射 {原文: 译文}
bilingual: 是否生成双语版本 (True: 保留原文+译文, False: 只保留译文)
"""
success_count = 0
for item in items:
original_text = item['text']
element = item['element']
# 查找翻译
translation = translation_map.get(original_text)
if translation is None:
continue
if bilingual:
# 双语模式: 在元素末尾添加译文
# 创建一个新的标签用于译文 (使用相同的标签名, 如 p, h1, h2...)
translation_p = self.soup.new_tag(element.name)
# 1. 继承 class
original_classes = element.get('class', [])
if original_classes:
# 复制列表以防引用修改
translation_p['class'] = list(original_classes) + ['translation']
else:
translation_p['class'] = ['translation']
# 2. 继承 style (如果有)
original_style = element.get('style')
if original_style:
translation_p['style'] = original_style
# 3. 设置内容
translation_p.string = translation
# 在原始元素后插入译文
element.insert_after(translation_p)
else:
# 纯译文模式: 替换所有文本节点
text_nodes = item['text_nodes']
if text_nodes:
text_nodes[0][0].replace_with(translation)
for node, _ in text_nodes[1:]:
try:
node.replace_with('')
except:
pass # 节点可能已被移除
success_count += 1
logger.info(f"回填完成: 成功 {success_count}/{len(translation_map)}")
return str(self.soup)
def _classify_document(self, file_name: str) -> str:
if not file_name:
return 'core'
file_name_lower = file_name.lower()
for pattern in self.SKIP_TRANSLATION_PATTERNS:
if re.search(pattern, file_name_lower):
return 'skip'
for pattern in self.TOC_PATTERNS:
if re.search(pattern, file_name_lower):
return 'toc'
return 'core'
def _should_translate(self, doc_type: str, is_decorative: bool, text: str = "") -> bool:
if is_decorative:
return False
# 检查是否为罗马数字 (通常是章节号: I, II, III...)
if self._is_roman_numeral(text):
return False
if doc_type == 'core':
return True
if doc_type == 'toc':
return self.translate_toc
return False
def _is_roman_numeral(self, text: str) -> bool:
"""检查是否为罗马数字 (I, II, III, IV, V...)"""
text = text.strip().upper()
if not text:
return False
# 简单正则，覆盖常见直到 3999
# ^M{0,4}(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})$
# 注意: 避免匹配空字符串 (已经由 if not text 处理)
pattern = re.compile(r"^M{0,4}(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})$")
return bool(pattern.match(text))
def _is_decorative(self, text: str) -> bool:
text_stripped = text.strip()
if not text_stripped:
return False
# 增强: 如果只包含非字母数字字符 (标点, 符号, 分隔线等), 视为装饰性
# 这将覆盖 ***, ---, ..., _____, —— 等
if not any(c.isalnum() for c in text_stripped):
return True
if len(text_stripped) > 20:
return False
decorative_patterns = [
r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾]+$',
r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾\s]+$',
]
for pattern in decorative_patterns:
if re.match(pattern, text_stripped):
return True
unique_chars = set(text_stripped.replace(' ', ''))
if len(unique_chars) <= 3:
decorative_chars = set('*-_~•◆◇■□▪▫●○◉◎⦿⦾')
if unique_chars & decorative_chars:
return True
return False
archive/v0.09/tests/extraction_experiment/extractors/lxml_xpath.py
+188
View File
@@ -0,0 +1,188 @@
"""
lxml XPath 提取器
使用 lxml 的 XPath 功能实现精准的文本提取和定位
"""
from lxml import etree, html
from typing import List, Dict, Any
from loguru import logger
import re
class LxmlXPathExtractor:
"""基于 lxml 和 XPath 的提取器"""
# 块级元素的 XPath 表达式
BLOCK_XPATH = ' | '.join([
f'//{tag}' for tag in [
'p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
'blockquote', 'li', 'td', 'dd', 'dt', 'figcaption',
'section', 'article', 'aside', 'header', 'footer', 'main'
]
])
NAV_KEYWORDS = [
'nav', 'navigation', 'toc', 'menu', 'header', 'footer',
'page-number', 'page-num', 'sidebar'
]
def __init__(self, min_text_length: int = 10):
"""
初始化提取器
Args:
min_text_length: 最小文本长度
"""
self.min_text_length = min_text_length
def extract(self, html_content: str) -> List[Dict[str, Any]]:
"""
从 HTML 中提取所有文本元素
Args:
html_content: HTML 字符串
Returns:
提取的元素列表
"""
try:
# 移除 XML 声明(如果存在)
import re
html_content = re.sub(r'<\?xml[^?]*\?>', '', html_content)
# 解析 HTML
tree = html.fromstring(html_content)
except Exception as e:
logger.error(f"lxml 解析失败: {e}")
return []
items = []
processed_xpaths = set()
# 使用 XPath 查找所有块级元素
try:
elements = tree.xpath(self.BLOCK_XPATH)
except Exception as e:
logger.error(f"XPath 查询失败: {e}")
return []
for element in elements:
# 获取 XPath (需要通过 ElementTree 包装)
try:
xpath = tree.getroottree().getpath(element)
except:
# 备用方案:生成简单路径
xpath = f"//{element.tag}[{elements.index(element)}]"
# 避免重复
if xpath in processed_xpaths:
continue
# 提取文本
text = self._clean_text(element)
# 过滤过短文本
if len(text.strip()) < self.min_text_length:
continue
# 判断是否是导航元素
is_nav = self._is_navigation_element(element)
# 获取 HTML
try:
element_html = etree.tostring(element, encoding='unicode')
except:
element_html = ""
items.append({
'xpath': xpath,
'text': text,
'html': element_html,
'tag': element.tag,
'is_navigation': is_nav
})
processed_xpaths.add(xpath)
logger.info(f"lxml 提取了 {len(items)} 个文本元素")
return items
def _clean_text(self, element) -> str:
"""清理元素文本"""
# lxml 的 text_content() 方法
text = element.text_content().strip()
# 清理多余空白
text = re.sub(r'\s+', ' ', text)
return text
def _is_navigation_element(self, element) -> bool:
"""判断是否是导航元素"""
# 检查 class 属性
classes = element.get('class', '')
class_str = classes.lower() if isinstance(classes, str) else ''
if any(keyword in class_str for keyword in self.NAV_KEYWORDS):
return True
# 检查父元素
parent = element.getparent()
if parent is not None:
p_classes = parent.get('class', '')
p_class_str = p_classes.lower() if isinstance(p_classes, str) else ''
if any(keyword in p_class_str for keyword in self.NAV_KEYWORDS):
return True
return False
def backfill(self, html_content: str, translation_map: Dict[str, str]) -> str:
"""
使用 XPath 精准回填翻译
Args:
html_content: 原始 HTML
translation_map: {xpath: translation} 映射
Returns:
回填后的 HTML 字符串
"""
try:
# 移除 XML 声明
import re
html_content = re.sub(r'<\?xml[^?]*\?>', '', html_content)
tree = html.fromstring(html_content)
except Exception as e:
logger.error(f"lxml 解析失败: {e}")
return html_content
success_count = 0
fail_count = 0
for xpath, translation in translation_map.items():
try:
elements = tree.xpath(xpath)
if not elements:
logger.warning(f"回填失败: 未找到 XPath {xpath}")
fail_count += 1
continue
element = elements[0]
# 清空元素内容并设置新文本
element.clear()
element.text = translation
success_count += 1
except Exception as e:
logger.error(f"回填错误 {xpath}: {e}")
fail_count += 1
logger.info(f"lxml 回填完成: 成功 {success_count}, 失败 {fail_count}")
try:
return etree.tostring(tree, encoding='unicode', method='html')
except:
return html_content
archive/v0.09/tests/extraction_experiment/extractors/one_to_one.py
+267
View File
@@ -0,0 +1,267 @@
"""
真正的一比一对应提取器
核心原则:
1. 提取时: 记录每个元素的所有文本节点位置
2. 回填时: 精确替换这些文本节点,不改变任何结构
"""
from bs4 import BeautifulSoup, Tag, NavigableString
from typing import List, Dict, Any, Tuple
import re
from loguru import logger
class OneToOneExtractor:
"""一比一对应提取器"""
SKIP_TRANSLATION_PATTERNS = [
r'index\.x?html',
r'bibliography\.x?html',
r'endnotes?\.x?html',
r'footnotes?\.x?html',
]
TOC_PATTERNS = [
r'nav\.x?html',
r'toc\.x?html',
]
OTHER_NON_CORE_PATTERNS = [
r'copyright\.x?html',
r'title\.x?html',
r'cover\.x?html',
]
BLOCK_TAGS = [
'p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
'blockquote', 'li', 'td', 'dd', 'dt', 'figcaption',
'section', 'article', 'aside', 'header', 'footer', 'main'
]
def __init__(self, translate_toc: bool = False):
self.translate_toc = translate_toc
self.soup = None
def extract(self, html_content: str, file_name: str = "") -> List[Dict[str, Any]]:
"""
提取文本,记录精确的文本节点位置
关键改进: 不跳过嵌套元素,每个块级元素都独立提取
返回:
{
'element': 元素引用,
'text': 完整文本,
'text_nodes': [(node, text), ...], # 只包含直接子节点的文本
'should_translate': bool
}
"""
self.soup = BeautifulSoup(html_content, 'html.parser')
# 移除不需要的元素
for element in self.soup(['script', 'style', 'meta', 'link']):
element.decompose()
doc_type = self._classify_document(file_name)
items = []
# 关键: 不使用 processed_ids,每个元素都独立提取
all_elements = []
for element in self.soup.find_all(self.BLOCK_TAGS):
# 提取完整文本
full_text = element.get_text(separator=' ', strip=True)
if not full_text.strip():
continue
# 关键: 只收集当前元素的直接文本节点
# 不包括子元素中的文本节点
text_nodes = self._collect_direct_text_nodes(element)
# 如果没有直接文本节点,说明所有文本都在子元素中
# 这种情况下跳过,让子元素自己处理
if not text_nodes:
continue
all_elements.append({
'element': element,
'text': full_text,
'text_nodes': text_nodes,
'doc_type': doc_type,
})
# 过滤: 只保留叶子节点 (没有被其他提取元素包含的元素)
for elem_data in all_elements:
element = elem_data['element']
# 检查是否被其他提取元素包含
is_contained = False
for other_data in all_elements:
if other_data is elem_data:
continue
other_element = other_data['element']
# 检查 element 是否是 other_element 的子孙
if element in other_element.descendants:
is_contained = True
break
if is_contained:
continue
# 这是叶子节点,添加到结果
is_decorative = self._is_decorative(elem_data['text'])
should_translate = self._should_translate(doc_type, is_decorative)
items.append({
'element': element,
'text': elem_data['text'],
'text_nodes': elem_data['text_nodes'],
'should_translate': should_translate,
'doc_type': doc_type,
'is_decorative': is_decorative,
'tag': element.name
})
logger.info(
f"[{doc_type}] 提取 {len(items)} 个元素: "
f"翻译 {sum(1 for i in items if i['should_translate'])}"
)
return items
def _collect_direct_text_nodes(self, element: Tag) -> List[Tuple[NavigableString, str]]:
"""
收集元素的文本节点
策略:
1. 优先收集直接文本节点
2. 如果没有直接文本节点,收集所有子孙文本节点
例如:
<div>Text1 <span>Text2</span></div> → 收集 Text1 (直接)
<div><span>Text2</span></div> → 收集 Text2 (子孙)
"""
text_nodes = []
# 先尝试收集直接子节点的文本
for child in element.children:
if isinstance(child, NavigableString):
if isinstance(child, type(element)): # 跳过注释
continue
text = str(child).strip()
if text:
text_nodes.append((child, text))
# 如果有直接文本节点,返回
if text_nodes:
return text_nodes
# 否则,收集所有子孙文本节点
for descendant in element.descendants:
if isinstance(descendant, NavigableString):
if isinstance(descendant, type(element)): # 跳过注释
continue
text = str(descendant).strip()
if text:
text_nodes.append((descendant, text))
return text_nodes
def backfill(self, items: List[Dict[str, Any]], translation_map: Dict[str, str]) -> str:
"""
一比一精确回填
策略:
1. 对于每个元素,找到对应的翻译
2. 将翻译分配给所有文本节点
3. 精确替换每个文本节点
"""
success_count = 0
for item in items:
original_text = item['text']
text_nodes = item['text_nodes']
# 查找翻译
translation = translation_map.get(original_text)
if translation is None:
continue
# 一比一替换: 将翻译替换到第一个文本节点,清空其他
if text_nodes:
# 第一个文本节点替换为完整翻译
text_nodes[0][0].replace_with(translation)
# 其他文本节点清空(保留结构)
for node, _ in text_nodes[1:]:
node.replace_with('')
success_count += 1
logger.info(f"回填完成: 成功 {success_count}/{len(translation_map)}")
return str(self.soup)
def _classify_document(self, file_name: str) -> str:
if not file_name:
return 'core'
file_name_lower = file_name.lower()
for pattern in self.SKIP_TRANSLATION_PATTERNS:
if re.search(pattern, file_name_lower):
return 'skip'
for pattern in self.TOC_PATTERNS:
if re.search(pattern, file_name_lower):
return 'toc'
for pattern in self.OTHER_NON_CORE_PATTERNS:
if re.search(pattern, file_name_lower):
return 'other'
return 'core'
def _should_translate(self, doc_type: str, is_decorative: bool) -> bool:
if is_decorative:
return False
if doc_type == 'core':
return True
if doc_type == 'toc':
return self.translate_toc
return False
def _is_contained_in_processed(self, element: Tag, processed_ids: set) -> bool:
for parent in element.parents:
if isinstance(parent, Tag) and id(parent) in processed_ids:
return True
return False
def _is_decorative(self, text: str) -> bool:
text_stripped = text.strip()
if not text_stripped or len(text_stripped) > 20:
return False
decorative_patterns = [
r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾]+$',
r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾\s]+$',
]
for pattern in decorative_patterns:
if re.match(pattern, text_stripped):
return True
unique_chars = set(text_stripped.replace(' ', ''))
if len(unique_chars) <= 3:
decorative_chars = set('*-_~•◆◇■□▪▫●○◉◎⦿⦾')
if unique_chars & decorative_chars:
return True
return False
archive/v0.09/tests/extraction_experiment/extractors/smart_extractor.py
+325
View File
@@ -0,0 +1,325 @@
"""
智能分类提取器
根据文档类型和复杂度,智能决定提取策略:
- 正文: 100% 提取,必须翻译
- 非核心部分: 如果复杂度高,标记为跳过翻译
"""
from bs4 import BeautifulSoup, Tag
from typing import List, Dict, Any, Set
import re
from loguru import logger
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent))
from dom_path_utils import DOMPathUtils
class SmartExtractor:
"""智能分类提取器"""
# 非核心文档的文件名模式
NON_CORE_PATTERNS = [
r'nav\.x?html', # 目录
r'toc\.x?html', # 目录
r'index\.x?html', # 索引
r'bibliography\.x?html', # 参考文献
r'endnotes?\.x?html', # 尾注
r'footnotes?\.x?html', # 脚注
r'copyright\.x?html', # 版权页
r'title\.x?html', # 标题页
r'cover\.x?html', # 封面
]
# 块级标签
BLOCK_TAGS = [
'p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
'blockquote', 'li', 'td', 'dd', 'dt', 'figcaption',
'section', 'article', 'aside', 'header', 'footer', 'main'
]
# 装饰性符号模式
DECORATIVE_PATTERNS = [
r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾]+$',
r'^[*\-_~•◆◇■□▪▫●○◉◎⦿⦾\s]+$',
r'^[\u2022-\u2027\u2030-\u205E]+$',
]
def __init__(self, preserve_decorative: bool = True):
"""
初始化提取器
Args:
preserve_decorative: 是否保留装饰性元素
"""
self.preserve_decorative = preserve_decorative
self.path_utils = DOMPathUtils()
def extract(self, html_content: str, file_name: str = "") -> List[Dict[str, Any]]:
"""
智能提取文本元素
Args:
html_content: HTML 字符串
file_name: 文件名(用于判断文档类型)
Returns:
提取的元素列表,每个元素包含:
- path: DOM 路径
- text: 文本内容
- tag: 标签名
- is_decorative: 是否装饰性
- is_core: 是否核心内容(正文)
- should_translate: 是否应该翻译
"""
soup = BeautifulSoup(html_content, 'html.parser')
# 移除不需要的元素
for element in soup(['script', 'style', 'meta', 'link']):
element.decompose()
# 判断文档类型
is_core_document = self._is_core_document(file_name)
items = []
processed_ids = set()
# 遍历所有块级元素
for element in soup.find_all(self.BLOCK_TAGS):
elem_id = id(element)
if elem_id in processed_ids:
continue
if self._is_contained_in_processed(element, processed_ids):
continue
# 提取文本(不过滤任何内容)
text = self._extract_text(element)
if not text.strip():
continue
# 判断元素类型
is_decorative = self._is_decorative_element(element, text)
# 生成路径
path = self.path_utils.get_dom_path(element)
# 决定是否翻译
should_translate = self._should_translate(
element, text, is_core_document, is_decorative
)
items.append({
'path': path,
'element': element,
'text': text,
'html': str(element),
'tag': element.name,
'is_decorative': is_decorative,
'is_core': is_core_document,
'should_translate': should_translate,
'file_name': file_name
})
processed_ids.add(elem_id)
# 添加 <hr> 等装饰性标签
if self.preserve_decorative:
for hr in soup.find_all('hr'):
elem_id = id(hr)
if elem_id not in processed_ids:
path = self.path_utils.get_dom_path(hr)
items.append({
'path': path,
'element': hr,
'text': '---',
'html': str(hr),
'tag': 'hr',
'is_decorative': True,
'is_core': is_core_document,
'should_translate': False,
'file_name': file_name
})
processed_ids.add(elem_id)
# 统计
core_count = sum(1 for i in items if i['is_core'])
translate_count = sum(1 for i in items if i['should_translate'])
decorative_count = sum(1 for i in items if i['is_decorative'])
logger.info(
f"提取了 {len(items)} 个元素 "
f"(核心: {core_count}, 需翻译: {translate_count}, 装饰性: {decorative_count})"
)
return items
def _is_core_document(self, file_name: str) -> bool:
"""
判断是否是核心文档(正文)
非核心文档包括: 目录、索引、参考文献、版权页等
"""
if not file_name:
return True # 默认认为是核心文档
file_name_lower = file_name.lower()
for pattern in self.NON_CORE_PATTERNS:
if re.search(pattern, file_name_lower):
return False
return True
def _should_translate(self, element: Tag, text: str,
is_core_document: bool, is_decorative: bool) -> bool:
"""
决定元素是否应该翻译
规则:
1. 装饰性元素: 不翻译
2. 核心文档: 全部翻译
3. 非核心文档: 根据复杂度决定
"""
# 装饰性元素不翻译
if is_decorative:
return False
# 核心文档全部翻译
if is_core_document:
return True
# 非核心文档: 检查复杂度
complexity = self._calculate_complexity(element, text)
# 复杂度阈值: 如果太复杂,不翻译
if complexity > 0.5:
logger.debug(f"非核心元素复杂度过高 ({complexity:.2f}), 跳过翻译: {text[:50]}")
return False
return True
def _calculate_complexity(self, element: Tag, text: str) -> float:
"""
计算元素的复杂度
复杂度指标:
- 嵌套深度
- 链接数量
- 数字比例
- 特殊字符比例
Returns:
0.0 - 1.0, 越高越复杂
"""
complexity_score = 0.0
# 1. 嵌套深度 (最大贡献 0.3)
depth = len(list(element.parents))
complexity_score += min(depth / 20, 0.3)
# 2. 链接数量 (最大贡献 0.3)
links = element.find_all('a')
if links:
link_ratio = len(links) / max(len(text.split()), 1)
complexity_score += min(link_ratio, 0.3)
# 3. 数字比例 (最大贡献 0.2)
digits = sum(c.isdigit() for c in text)
if text:
digit_ratio = digits / len(text)
complexity_score += min(digit_ratio * 2, 0.2)
# 4. 特殊字符比例 (最大贡献 0.2)
special_chars = sum(1 for c in text if not c.isalnum() and not c.isspace())
if text:
special_ratio = special_chars / len(text)
complexity_score += min(special_ratio * 2, 0.2)
return min(complexity_score, 1.0)
def _is_contained_in_processed(self, element: Tag, processed_ids: Set[int]) -> bool:
"""检查元素是否被已处理的父元素包含"""
for parent in element.parents:
if isinstance(parent, Tag) and id(parent) in processed_ids:
return True
return False
def _extract_text(self, element: Tag) -> str:
"""
提取元素文本 - 100% 完整提取,不过滤任何内容
注意: 这里不做任何清理,保证 100% 提取
"""
return element.get_text(separator=' ', strip=True)
def _is_decorative_element(self, element: Tag, text: str) -> bool:
"""判断是否是装饰性元素"""
# 检查 class
classes = element.get('class', [])
class_str = ' '.join(classes).lower() if isinstance(classes, list) else str(classes).lower()
decorative_classes = ['separator', 'divider', 'ornament', 'decoration', 'break']
if any(dc in class_str for dc in decorative_classes):
return True
# 检查文本模式
text_stripped = text.strip()
if not text_stripped:
return False
for pattern in self.DECORATIVE_PATTERNS:
if re.match(pattern, text_stripped):
return True
# 检查重复字符
if len(text_stripped) <= 20:
unique_chars = set(text_stripped.replace(' ', ''))
if len(unique_chars) <= 3:
decorative_chars = set('*-_~•◆◇■□▪▫●○◉◎⦿⦾')
if unique_chars & decorative_chars:
return True
return False
def backfill(self, html_content: str, translation_map: Dict[str, str]) -> str:
"""
精准回填翻译
只回填 should_translate=True 的元素
"""
soup = BeautifulSoup(html_content, 'html.parser')
success_count = 0
skip_count = 0
fail_count = 0
for path, translation in translation_map.items():
element = self.path_utils.find_by_path(soup, path)
if element is None:
logger.warning(f"回填失败: 未找到路径 {path}")
fail_count += 1
continue
# 检查是否应该翻译
# (这个信息应该在 translation_map 的构建阶段就过滤了)
# 创建新元素
new_tag = soup.new_tag(element.name)
new_tag.string = translation
# 复制属性
for attr, value in element.attrs.items():
new_tag[attr] = value
# 替换
element.replace_with(new_tag)
success_count += 1
logger.info(f"回填完成: 成功 {success_count}, 跳过 {skip_count}, 失败 {fail_count}")
return str(soup)
archive/v0.09/tests/extraction_experiment/reports/multi_epub_test_report.md
+90
View File
@@ -0,0 +1,90 @@
# 多 ePub 提取完整性测试报告
**测试时间**: 2026-01-19 12:26:59
**测试文件数**: 5
**成功**: 5/5
## 测试结果汇总
| 文件名 | 文件大小 | 提取元素 | 装饰性 | 文本长度 | 覆盖率 |
|--------|---------|---------|--------|---------|--------|
| Gambling Man.epub | 3428.3KB | 60 | 0 | 723,609 | 77.7% |
| On_China_Henry_Kissinger.epub | 924.5KB | 3602 | 32 | 1,141,726 | 87.6% |
| The World Atlas of Coffee - Fr | 20406.0KB | 1621 | 340 | 353,627 | 74.6% |
| The_Philosopher_in_the_Valley. | 4687.7KB | 54 | 15 | 524,578 | 93.9% |
| To_Explain_the_World.epub | 1756.4KB | 3564 | 105 | 782,608 | 87.9% |
## 详细分析
### Gambling Man.epub
- **HTML 文档数**: 47
- **提取元素总数**: 60
- 内容元素: 51
- 装饰性元素: 0
- 导航元素: 9
- **提取文本长度**: 723,609 字符
- **提取词数**: 118,115
- **Pandoc 基准长度**: 889,384 字符
- **覆盖率**: 77.70%
- **共同词数**: 11,607
### On_China_Henry_Kissinger.epub
- **HTML 文档数**: 144
- **提取元素总数**: 3602
- 内容元素: 3570
- 装饰性元素: 32
- 导航元素: 0
- **提取文本长度**: 1,141,726 字符
- **提取词数**: 181,503
- **Pandoc 基准长度**: 1,509,848 字符
- **覆盖率**: 87.57%
- **共同词数**: 12,834
### The World Atlas of Coffee - From Beans to Brewing - Coffees Explored, Explained and Enjoyed (James Hoffmann) (Z-Library).epub
- **HTML 文档数**: 98
- **提取元素总数**: 1621
- 内容元素: 1254
- 装饰性元素: 340
- 导航元素: 27
- **提取文本长度**: 353,627 字符
- **提取词数**: 59,885
- **Pandoc 基准长度**: 560,850 字符
- **覆盖率**: 74.62%
- **共同词数**: 5,885
### The_Philosopher_in_the_Valley.epub
- **HTML 文档数**: 22
- **提取元素总数**: 54
- 内容元素: 24
- 装饰性元素: 15
- 导航元素: 15
- **提取文本长度**: 524,578 字符
- **提取词数**: 86,566
- **Pandoc 基准长度**: 550,137 字符
- **覆盖率**: 93.95%
- **共同词数**: 9,760
### To_Explain_the_World.epub
- **HTML 文档数**: 105
- **提取元素总数**: 3564
- 内容元素: 3459
- 装饰性元素: 105
- 导航元素: 0
- **提取文本长度**: 782,608 字符
- **提取词数**: 133,415
- **Pandoc 基准长度**: 950,720 字符
- **覆盖率**: 87.90%
- **共同词数**: 9,237
## 总结
- **平均覆盖率**: 84.35%
- **总装饰性元素**: 492 个
- **提取器状态**: ⚠️ 需要优化
archive/v0.09/tests/extraction_experiment/run_batch.py
+71
View File
@@ -0,0 +1,71 @@
"""
批量运行测试: 对 input 目录下的所有 EPUB 执行清理和生成双语版本
"""
import sys
from pathlib import Path
import os
import time
from loguru import logger
# 配置路径
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
sys.path.insert(0, str(Path(__file__).parent))
# 导入功能模块
from simple_cleaner import clean_epub
from test_end_to_end import create_bilingual_epub
def run_batch():
input_dir = project_root / "input"
output_dir = project_root / "test_output"
output_dir.mkdir(exist_ok=True)
epubs = list(input_dir.glob("*.epub"))
epubs.sort() # 按文件名排序
print(f"\n{'='*80}")
print(f"批量测试开始: 共 {len(epubs)} 个文件")
print(f"{'='*80}\n")
success_count = 0
for i, epub_path in enumerate(epubs, 1):
print(f"[{i}/{len(epubs)}] 📖 处理: {epub_path.name}")
cleaned_path = output_dir / f"{epub_path.stem}_cleaned.epub"
bilingual_path = output_dir / f"{epub_path.stem}_bilingual.epub"
try:
# 1. 清理
print(" ➤ 正在清理...")
start = time.time()
# 捕获日志或只允许 ERROR? 暂时保持默认
clean_epub(str(epub_path), str(cleaned_path))
print(f" ✓ 清理完成用时: {time.time() - start:.2f}s")
# 2. 生成双语
print(" ➤ 正在生成双语版本...")
start = time.time()
create_bilingual_epub(cleaned_path, bilingual_path)
print(f" ✓ 生成完成用时: {time.time() - start:.2f}s")
print(f" ✅ 成功! 输出: {bilingual_path.name}\n")
success_count += 1
except Exception as e:
print(f" ❌ 处理失败: {e}\n")
# 不中断后续任务
continue
print(f"{'='*80}")
print(f"批量测试结束: 成功 {success_count}/{len(epubs)}")
print(f"{'='*80}\n")
if __name__ == "__main__":
# 配置 logger 只显示 WARNING 以上，以免刷屏
logger.remove()
logger.add(sys.stderr, level="WARNING")
run_batch()
archive/v0.09/tests/extraction_experiment/simple_cleaner.py
+205
View File
@@ -0,0 +1,205 @@
"""
简化版 Calibre 清理器 - 避免复杂操作
只做最基本的清理:
1. div 转 p
2. 移除 calibre 类
"""
from bs4 import BeautifulSoup, Tag
from loguru import logger
class SimpleCleaner:
"""简化版清理器"""
def clean(self, html_content: str, item=None) -> str:
"""
清理 HTML
Args:
html_content: HTML 内容
item: EpubItem 对象(可选), 用于注册 links
"""
soup = BeautifulSoup(html_content, 'html.parser')
# 0. 提取并保留 CSS 链接 (解决 ebooklib 丢失 link 的问题)
if item:
head = soup.find('head')
if head:
# 提取 link
links = head.find_all('link', rel='stylesheet')
for link in links:
href = link.get('href')
if href:
# 检查是否已存在(避免重复)
existing_links = list(item.get_links())
exists = False
for l in existing_links:
l_href = getattr(l, 'href', None)
if l_href is None and isinstance(l, dict):
l_href = l.get('href')
if l_href == href:
exists = True
break
if not exists:
logger.debug(f"恢复 CSS 链接: {href}")
item.add_link(href=href, rel='stylesheet', type='text/css')
stats = {'divs_to_p': 0, 'classes_removed': 0}
# 1. div 转 p (只转换没有块级子元素的)
inline_tags = {'span', 'em', 'strong', 'i', 'b', 'u', 'a', 'br'}
divs = list(soup.find_all('div')) # 先收集所有div
for div in divs:
has_block = any(
isinstance(c, Tag) and c.name not in inline_tags
for c in div.children
)
if not has_block:
div.name = 'p'
stats['divs_to_p'] += 1
# 2. 清理 calibre 类 - 暂时禁用，以保留样式
# elements = list(soup.find_all(class_=True)) # 先收集
# ... (保留原注释代码)
logger.info(f"清理完成: div→p {stats['divs_to_p']}, 类移除 {stats['classes_removed']}")
return str(soup)
def clean_epub(input_path: str, output_path: str):
"""清理 ePub"""
from ebooklib import epub
import zipfile
logger.info(f"开始清理: {input_path}")
# 打开 zip 以读取原始内容
try:
input_zip = zipfile.ZipFile(input_path, 'r')
zip_files = set(input_zip.namelist())
except Exception as e:
logger.error(f"无法打开 Zip: {e}")
input_zip = None
zip_files = set()
book = epub.read_epub(input_path)
cleaner = SimpleCleaner()
count = 0
for item in book.get_items():
if item.get_type() == 9:
try:
file_name = item.get_name()
content = None
# 优先从 Zip 读取以保留 Head 信息
if input_zip and file_name in zip_files:
try:
content = input_zip.read(file_name).decode('utf-8')
except Exception as e:
logger.warning(f"Zip 读取失败 {file_name}: {e}")
# 回退到 ebooklib
if content is None:
raw_content = item.get_content()
if raw_content:
content = raw_content.decode('utf-8')
# 检查原始内容
if not content or not content.strip():
logger.warning(f"跳过空文档: {item.get_name()}")
continue
# 清理并提取信息
# 注意: 我们需要传入 item 以便 cleaner 可以注册 links
cleaned = cleaner.clean(content, item)
# 检查清理后内容
if not cleaned.strip():
logger.error(f"⚠️ 清理后内容为空: {item.get_name()} (原始长度: {len(content)})")
# 如果清理变为空，保留原始内容
cleaned = content
item.set_content(cleaned.encode('utf-8'))
count += 1
if 'titlepage' in item.get_name():
logger.info(f"Titlepage 处理完成: {len(cleaned)} chars")
except Exception as e:
logger.warning(f"清理失败 {item.get_name()}: {e}")
# 修复 TOC：补全 UID 并移除指向不存在文件的死链
def fix_and_clean_toc(toc, book):
new_toc = []
import uuid
from ebooklib.epub import Link
for item in toc:
# Case 1: (Section, Children) 元组
if isinstance(item, (tuple, list)):
section, children = item
# 递归清理子节点
cleaned_children = fix_and_clean_toc(children, book)
# 检查 Section 节点
if isinstance(section, Link):
href = section.href.split('#')[0]
# 有效性检查：目标文件必须在 manifest 中存在
if book.get_item_with_href(href):
if section.uid is None:
section.uid = f'uuid-{uuid.uuid4()}'
new_toc.append((section, cleaned_children))
else:
logger.warning(f"移除无效 TOC 节点 (目标缺失): {section.title} -> {section.href}")
# 如果父节点无效，这里选择提升子节点，还是丢弃？
# 策略：如果父节点都无效了，就把子节点提升上来（如果子节点有效）
new_toc.extend(cleaned_children)
else:
# 如果 Section 不是 Link (罕见)，保留
new_toc.append((section, cleaned_children))
# Case 2: 单个 Link 节点
elif isinstance(item, Link):
href = item.href.split('#')[0]
if book.get_item_with_href(href):
if item.uid is None:
item.uid = f'uuid-{uuid.uuid4()}'
new_toc.append(item)
else:
logger.warning(f"移除无效 TOC 节点 (目标缺失): {item.title} -> {item.href}")
# Case 3: 其他 (如自定义 dict 等? 一般不会)
else:
new_toc.append(item)
return new_toc
try:
book.toc = fix_and_clean_toc(book.toc, book)
except Exception as e:
logger.warning(f"修复 TOC 失败: {e}")
import traceback
logger.warning(traceback.format_exc())
epub.write_epub(output_path, book)
logger.info(f"完成: 处理了 {count} 个文档")
if __name__ == "__main__":
import sys
if len(sys.argv) < 3:
print("用法: python simple_cleaner.py <input.epub> <output.epub>")
sys.exit(1)
clean_epub(sys.argv[1], sys.argv[2])
archive/v0.09/tests/extraction_experiment/test_backfill.py
+195
View File
@@ -0,0 +1,195 @@
"""
完整回填测试
使用 On_China 书籍,模拟翻译并回填,生成双语版本供检查
"""
import sys
from pathlib import Path
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
from loguru import logger
import shutil
sys.path.insert(0, str(Path(__file__).parent))
from extractors.final_extractor import FinalExtractor
def create_bilingual_test_epub(epub_path: Path, output_path: Path, translate_toc: bool = False):
"""
创建双语测试版本
将每个元素的翻译替换为: 原文 + 标记
- 翻译元素: "原文 [翻译]"
- 跳过元素: "原文 [跳过]"
- 装饰性: "原文 [装饰]"
"""
print(f"\n{'='*80}")
print(f"创建双语测试版本: {epub_path.name}")
print(f"目录翻译: {'是' if translate_toc else '否'}")
print(f"{'='*80}\n")
# 加载 ePub
book = epub.read_epub(str(epub_path))
# 提取器
extractor = FinalExtractor(translate_toc=translate_toc, preserve_decorative=True)
# 统计
total_items = 0
total_translate = 0
total_skip = 0
total_decorative = 0
doc_stats = []
# 处理每个 HTML 文档
for item in book.get_items():
if item.get_type() != 9: # 只处理 ITEM_DOCUMENT
continue
try:
content = item.get_content().decode('utf-8')
except:
continue
file_name = item.get_name()
# 提取
items = extractor.extract(content, file_name)
if not items:
continue
# 统计
translate_items = [i for i in items if i['should_translate']]
skip_items = [i for i in items if not i['should_translate'] and not i['is_decorative']]
decorative_items = [i for i in items if i['is_decorative']]
total_items += len(items)
total_translate += len(translate_items)
total_skip += len(skip_items)
total_decorative += len(decorative_items)
doc_stats.append({
'file': file_name,
'doc_type': items[0]['doc_type'],
'total': len(items),
'translate': len(translate_items),
'skip': len(skip_items),
'decorative': len(decorative_items)
})
# 创建翻译映射
translation_map = {}
for i in items:
if i['should_translate']:
translation_map[i['path']] = f"{i['text']} [翻译]"
elif i['is_decorative']:
translation_map[i['path']] = f"{i['text']} [装饰]"
else:
translation_map[i['path']] = f"{i['text']} [跳过]"
# 回填
new_content = extractor.backfill(content, translation_map)
# 更新 item
item.set_content(new_content.encode('utf-8'))
# 保存新 ePub
epub.write_epub(str(output_path), book)
# 显示统计
print(f"{'='*80}")
print("处理统计")
print(f"{'='*80}\n")
print(f"总元素数: {total_items}")
print(f" - 翻译: {total_translate} ({total_translate/total_items*100:.1f}%)")
print(f" - 跳过: {total_skip} ({total_skip/total_items*100:.1f}%)")
print(f" - 装饰: {total_decorative} ({total_decorative/total_items*100:.1f}%)\n")
# 按文档类型分组统计
print(f"{'='*80}")
print("按文档类型统计")
print(f"{'='*80}\n")
doc_type_stats = {}
for stat in doc_stats:
doc_type = stat['doc_type']
if doc_type not in doc_type_stats:
doc_type_stats[doc_type] = {
'count': 0,
'total': 0,
'translate': 0,
'skip': 0,
'decorative': 0
}
doc_type_stats[doc_type]['count'] += 1
doc_type_stats[doc_type]['total'] += stat['total']
doc_type_stats[doc_type]['translate'] += stat['translate']
doc_type_stats[doc_type]['skip'] += stat['skip']
doc_type_stats[doc_type]['decorative'] += stat['decorative']
for doc_type, stats in sorted(doc_type_stats.items()):
print(f"📄 {doc_type.upper()} ({stats['count']} 个文档)")
print(f" 总元素: {stats['total']}")
print(f" 翻译: {stats['translate']} ({stats['translate']/stats['total']*100:.1f}%)")
print(f" 跳过: {stats['skip']} ({stats['skip']/stats['total']*100:.1f}%)")
print(f" 装饰: {stats['decorative']} ({stats['decorative']/stats['total']*100:.1f}%)")
print()
# 显示详细文档列表
print(f"{'='*80}")
print("详细文档列表")
print(f"{'='*80}\n")
for stat in doc_stats[:20]:
doc_type_label = stat['doc_type'].upper()
print(f"[{doc_type_label:6}] {stat['file']}")
print(f" 元素: {stat['total']:4} | 翻译: {stat['translate']:4} | 跳过: {stat['skip']:4} | 装饰: {stat['decorative']:2}")
if len(doc_stats) > 20:
print(f"\n... 还有 {len(doc_stats) - 20} 个文档\n")
print(f"\n✅ 双语测试版本已保存: {output_path}")
print(f"\n请在 ePub 阅读器中打开检查:")
print(f" - 正文应该显示: '原文 [翻译]'")
print(f" - 索引/参考文献/尾注应该显示: '原文 [跳过]'")
print(f" - 目录应该显示: '原文 [{'翻译' if translate_toc else '跳过'}]'")
print(f" - 装饰性符号应该显示: '原文 [装饰]'")
def main():
"""主函数"""
logger.remove()
logger.add(sys.stderr, level="INFO")
# 测试文件
epub_path = project_root / "input" / "On_China_Henry_Kissinger.epub"
if not epub_path.exists():
print(f"❌ 文件不存在: {epub_path}")
return
# 输出目录
output_dir = project_root / "test_output"
output_dir.mkdir(exist_ok=True)
# 测试1: 不翻译目录
output_path_1 = output_dir / "On_China_bilingual_no_toc.epub"
create_bilingual_test_epub(epub_path, output_path_1, translate_toc=False)
print(f"\n{'='*80}\n")
# 测试2: 翻译目录
output_path_2 = output_dir / "On_China_bilingual_with_toc.epub"
create_bilingual_test_epub(epub_path, output_path_2, translate_toc=True)
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/test_bs4_skeleton.py
+195
View File
@@ -0,0 +1,195 @@
"""
测试 BS4 骨架保留
验证是否完整保留所有 HTML 结构、CSS 样式和属性
"""
import sys
from pathlib import Path
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
from loguru import logger
import re
sys.path.insert(0, str(Path(__file__).parent))
from extractors.bs4_skeleton import BS4SkeletonExtractor
def test_simple_html():
"""测试简单 HTML"""
html = """<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE html>
<html>
<head/>
<body>
<div class="calibre3"><span class="calibre6"><span class="bold">Table of Contents</span></span></div>
<p class="text-center" style="font-size: 18px; color: blue;">This is a centered paragraph.</p>
<blockquote class="quote" style="margin-left: 40px;">A famous quote here.</blockquote>
<h1 id="chapter1" class="chapter-title">Chapter One</h1>
</body>
</html>"""
print("\n" + "="*80)
print("简单 HTML 骨架保留测试")
print("="*80 + "\n")
# 提取
extractor = BS4SkeletonExtractor()
items = extractor.extract(html)
print(f"提取了 {len(items)} 个元素:\n")
for i, item in enumerate(items, 1):
print(f"{i}. [{item['tag']}] {item['text'][:50]}")
# 模拟翻译
translation_map = {}
for item in items:
if item['should_translate']:
translation_map[item['text']] = f"{item['text']} [翻译]"
print(f"\n待翻译: {len(translation_map)} 个元素\n")
# 回填
result_html = extractor.backfill(items, translation_map)
print("="*80)
print("回填后的 HTML:")
print("="*80 + "\n")
print(result_html)
# 验证
print("\n" + "="*80)
print("验证结果:")
print("="*80 + "\n")
checks = [
('class="calibre3"', 'class 属性'),
('class="text-center"', 'class 属性'),
('style="font-size: 18px; color: blue;"', 'style 属性'),
('style="margin-left: 40px;"', 'style 属性'),
('id="chapter1"', 'id 属性'),
('<span class="bold">', '内部格式标签'),
('<span class="calibre6">', '嵌套标签'),
]
for pattern, name in checks:
if pattern in result_html:
print(f"✅ {name} 保留: {pattern}")
else:
print(f"❌ {name} 丢失: {pattern}")
def test_real_epub():
"""测试真实 ePub"""
epub_path = project_root / "input" / "On_China_Henry_Kissinger.epub"
if not epub_path.exists():
print(f"\n跳过真实 ePub 测试: 文件不存在")
return
print("\n" + "="*80)
print("真实 ePub 骨架保留测试")
print("="*80 + "\n")
book = epub.read_epub(str(epub_path))
# 找第一个内容文档
for item in book.get_items():
if item.get_type() == 9 and 'dummy_split_002' in item.get_name():
content = item.get_content().decode('utf-8')
print(f"测试文件: {item.get_name()}\n")
# 统计原始 HTML 的属性
original_classes = len(re.findall(r'class="[^"]*"', content))
original_styles = len(re.findall(r'style="[^"]*"', content))
original_ids = len(re.findall(r'id="[^"]*"', content))
print(f"原始 HTML 统计:")
print(f" - class 属性: {original_classes} 个")
print(f" - style 属性: {original_styles} 个")
print(f" - id 属性: {original_ids} 个\n")
# 提取
extractor = BS4SkeletonExtractor()
items = extractor.extract(content, item.get_name())
print(f"提取了 {len(items)} 个元素\n")
# 显示前 3 个
for i, elem in enumerate(items[:3], 1):
print(f"元素 {i}:")
print(f" 标签: <{elem['tag']}>")
print(f" 文本: {elem['text'][:60]}...")
print()
# 模拟翻译
translation_map = {}
for elem in items:
if elem['should_translate']:
translation_map[elem['text']] = f"{elem['text']} [翻译]"
print(f"待翻译: {len(translation_map)} 个元素\n")
# 回填
result_html = extractor.backfill(items, translation_map)
# 统计回填后的属性
result_classes = len(re.findall(r'class="[^"]*"', result_html))
result_styles = len(re.findall(r'style="[^"]*"', result_html))
result_ids = len(re.findall(r'id="[^"]*"', result_html))
print("="*80)
print("回填后 HTML 统计:")
print("="*80 + "\n")
print(f" - class 属性: {result_classes} 个")
print(f" - style 属性: {result_styles} 个")
print(f" - id 属性: {result_ids} 个\n")
# 验证
print("="*80)
print("验证结果:")
print("="*80 + "\n")
if original_classes == result_classes:
print(f"✅ 所有 class 属性保留 ({original_classes} 个)")
else:
print(f"❌ class 属性丢失: {original_classes} → {result_classes}")
if original_styles == result_styles:
print(f"✅ 所有 style 属性保留 ({original_styles} 个)")
else:
print(f"❌ style 属性丢失: {original_styles} → {result_styles}")
if original_ids == result_ids:
print(f"✅ 所有 id 属性保留 ({original_ids} 个)")
else:
print(f"❌ id 属性丢失: {original_ids} → {result_ids}")
# 检查翻译是否成功
if '[翻译]' in result_html:
print(f"✅ 翻译成功回填")
else:
print(f"❌ 翻译未回填")
break
def main():
"""主函数"""
logger.remove()
logger.add(sys.stderr, level="INFO")
# 测试 1: 简单 HTML
test_simple_html()
# 测试 2: 真实 ePub
test_real_epub()
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/test_calibre_cleaner.py
+89
View File
@@ -0,0 +1,89 @@
"""
测试 Calibre 清理器
"""
import sys
from pathlib import Path
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from loguru import logger
from calibre_cleaner import CalibreHTMLCleaner
def test_html_cleaning():
"""测试 HTML 清理"""
# 测试用例: Calibre 生成的屎山代码
html = """
<div class="calibre16">
<span class="calibre9">
<div class="calibre16">
<span class="calibre9">
<span class="italic">A ruler</span>
</span>
</div>
<div class="calibre11">
<span class="calibre9">
<span class="italic">Must never</span>
</span>
</div>
<div class="calibre11">
<span class="calibre9">
<span class="italic">Mobilize his men</span>
</span>
</div>
</span>
</div>
"""
print("\n" + "="*80)
print("Calibre HTML 清理测试")
print("="*80 + "\n")
print("原始 HTML:")
print(html)
print()
# 清理
cleaner = CalibreHTMLCleaner()
cleaned = cleaner.clean(html)
print("="*80)
print("清理后的 HTML:")
print("="*80 + "\n")
print(cleaned)
print()
# 验证
print("="*80)
print("验证:")
print("="*80 + "\n")
if '<p>' in cleaned:
print(f"✅ div 转为 p: {cleaner.stats['divs_to_p']} 个")
else:
print("❌ div 未转为 p")
if '<em>' in cleaned:
print(f"✅ span 简化为 em: {cleaner.stats['spans_simplified']} 个")
else:
print("❌ span 未简化")
if 'calibre' not in cleaned or cleaner.stats['classes_removed'] > 0:
print(f"✅ 移除 calibre 类: {cleaner.stats['classes_removed']} 个")
else:
print("❌ calibre 类未移除")
def main():
"""主函数"""
logger.remove()
logger.add(sys.stderr, level="INFO")
test_html_cleaning()
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/test_cleaned_epub.py
+79
View File
@@ -0,0 +1,79 @@
"""
测试清理后的 ePub 提取效果
"""
import sys
from pathlib import Path
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
from loguru import logger
sys.path.insert(0, str(Path(__file__).parent))
from extractors.one_to_one import OneToOneExtractor
def test_cleaned_epub():
"""测试清理后的 ePub"""
cleaned_path = project_root / "test_output" / "On_China_cleaned.epub"
if not cleaned_path.exists():
print(f"❌ 清理后的 ePub 不存在: {cleaned_path}")
return
print("\n" + "="*80)
print("测试清理后的 ePub 提取效果")
print("="*80 + "\n")
# 加载 ePub
book = epub.read_epub(str(cleaned_path))
# 找诗歌部分
for item in book.get_items():
if item.get_type() == 9 and 'dummy_split_010' in item.get_name():
content = item.get_content().decode('utf-8')
print(f"测试文件: {item.get_name()}\n")
# 提取
extractor = OneToOneExtractor()
items = extractor.extract(content, item.get_name())
print(f"提取了 {len(items)} 个元素\n")
# 显示前10个
for i, elem in enumerate(items[:10], 1):
print(f"{i}. <{elem['tag']}> {elem['text'][:60]}...")
print(f"\n... (共 {len(items)} 个元素)")
# 检查诗歌部分
print("\n" + "="*80)
print("检查诗歌部分:")
print("="*80 + "\n")
poem_lines = [item for item in items if 'ruler' in item['text'].lower() or 'mobilize' in item['text'].lower()]
if poem_lines:
print(f"找到 {len(poem_lines)} 行诗歌:")
for i, line in enumerate(poem_lines[:5], 1):
print(f" {i}. {line['text'][:50]}")
else:
print("❌ 未找到诗歌部分")
break
def main():
"""主函数"""
logger.remove()
logger.add(sys.stderr, level="INFO")
test_cleaned_epub()
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/test_decorative.py
+200
View File
@@ -0,0 +1,200 @@
"""
装饰性元素提取测试
验证增强提取器对装饰性符号的识别和保留
"""
import sys
from pathlib import Path
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
from loguru import logger
sys.path.insert(0, str(Path(__file__).parent))
from extractors.enhanced_bs4 import EnhancedBS4Extractor
from extractors.bs4_optimized import BS4OptimizedExtractor
def test_decorative_elements():
"""测试装饰性元素的识别"""
html = """
<html>
<body>
<h1>Chapter 1</h1>
<p>This is a normal paragraph.</p>
<!-- 装饰性分隔符 -->
<p class="separator">***</p>
<p>• • •</p>
<div class="divider">———</div>
<hr/>
<h2>Section 1.1</h2>
<p>Another paragraph here.</p>
<!-- 装饰性符号 -->
<p>◆◇◆</p>
<p>Final paragraph.</p>
</body>
</html>
"""
print("\n" + "="*60)
print("装饰性元素识别测试")
print("="*60)
# 标准提取器
print("\n--- 标准 BS4 提取器 ---")
standard_extractor = BS4OptimizedExtractor(min_text_length=3)
standard_items = standard_extractor.extract(html)
print(f"提取元素数: {len(standard_items)}")
for i, item in enumerate(standard_items, 1):
print(f"{i}. [{item['tag']}] {item['text'][:50]}")
# 增强提取器
print("\n--- 增强 BS4 提取器 (保留装饰性元素) ---")
enhanced_extractor = EnhancedBS4Extractor(min_text_length=10, preserve_decorative=True)
enhanced_items = enhanced_extractor.extract(html)
print(f"提取元素数: {len(enhanced_items)}")
decorative_count = 0
for i, item in enumerate(enhanced_items, 1):
decorative_flag = " [装饰性]" if item.get('is_decorative') else ""
print(f"{i}. [{item['tag']}] {item['text'][:50]}{decorative_flag}")
if item.get('is_decorative'):
decorative_count += 1
print(f"\n装饰性元素数: {decorative_count}")
# 对比
print("\n" + "-"*60)
print(f"标准提取器: {len(standard_items)} 个元素")
print(f"增强提取器: {len(enhanced_items)} 个元素 (含 {decorative_count} 个装饰性)")
print(f"差异: +{len(enhanced_items) - len(standard_items)} 个元素")
def test_real_epub_decorative():
"""测试真实 ePub 中的装饰性元素"""
epub_path = project_root / "input" / "Gambling Man.epub"
if not epub_path.exists():
print(f"\n跳过真实 ePub 测试: 文件不存在")
return
print("\n" + "="*60)
print(f"真实 ePub 装饰性元素测试")
print("="*60)
book = epub.read_epub(str(epub_path))
# 提取前几个 HTML 文档
html_docs = []
for item in book.get_items():
if item.get_type() == 9: # ITEM_DOCUMENT
try:
content = item.get_content().decode('utf-8')
html_docs.append((item.get_name(), content))
if len(html_docs) >= 5:
break
except:
continue
total_decorative = 0
for filename, html_content in html_docs:
print(f"\n--- 文件: {filename} ---")
# 标准提取
standard_extractor = BS4OptimizedExtractor()
standard_items = standard_extractor.extract(html_content)
# 增强提取
enhanced_extractor = EnhancedBS4Extractor(preserve_decorative=True)
enhanced_items = enhanced_extractor.extract(html_content)
decorative_items = [item for item in enhanced_items if item.get('is_decorative')]
total_decorative += len(decorative_items)
print(f"标准提取: {len(standard_items)} 个元素")
print(f"增强提取: {len(enhanced_items)} 个元素")
print(f"装饰性元素: {len(decorative_items)} 个")
if decorative_items:
print("\n装饰性元素示例:")
for item in decorative_items[:3]:
print(f" - [{item['tag']}] {item['text'][:30]}")
print("\n" + "="*60)
print(f"总计发现 {total_decorative} 个装饰性元素")
def test_decorative_preservation():
"""测试装饰性元素在回填时的保留"""
html = """
<html>
<body>
<p>First paragraph.</p>
<p>***</p>
<p>Second paragraph.</p>
</body>
</html>
"""
print("\n" + "="*60)
print("装饰性元素回填保留测试")
print("="*60)
extractor = EnhancedBS4Extractor(min_text_length=5, preserve_decorative=True)
items = extractor.extract(html)
print(f"\n提取了 {len(items)} 个元素:")
for i, item in enumerate(items, 1):
decorative_flag = " [装饰性]" if item.get('is_decorative') else ""
print(f"{i}. {item['text']}{decorative_flag}")
# 创建翻译映射(只翻译非装饰性元素)
translation_map = {}
for i, item in enumerate(items):
if not item.get('is_decorative'):
translation_map[item['path']] = f"TRANSLATED_{i}"
print(f"\n待翻译: {len(translation_map)} 个元素")
# 回填
backfilled_html = extractor.backfill(html, translation_map)
print("\n回填后的 HTML:")
from bs4 import BeautifulSoup
soup = BeautifulSoup(backfilled_html, 'html.parser')
for p in soup.find_all('p'):
print(f" <p>{p.get_text()}</p>")
# 验证装饰性元素是否保留
if '***' in backfilled_html:
print("\n✅ 装饰性符号 '***' 已保留")
else:
print("\n❌ 装饰性符号 '***' 丢失")
def main():
"""主函数"""
logger.remove()
logger.add(sys.stderr, level="INFO")
# 测试 1: 装饰性元素识别
test_decorative_elements()
# 测试 2: 真实 ePub
test_real_epub_decorative()
# 测试 3: 回填保留
test_decorative_preservation()
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/test_end_to_end.py
+310
View File
@@ -0,0 +1,310 @@
"""
端到端测试: 清理 → 提取 → 模拟翻译 → 回填 → 生成双语 EPUB
完整流程验证
"""
import sys
from pathlib import Path
import hashlib
import os
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
from loguru import logger
sys.path.insert(0, str(Path(__file__).parent))
from extractors.fine_grained import FineGrainedExtractor
# Added MockTranslator class
class MockTranslator:
def translate(self, text: str) -> str:
"""
模拟翻译: 在文本前添加 [中文] 标记
这样可以清楚地看到哪些文本被翻译了
"""
return f"[中文] {text}"
def create_bilingual_epub(epub_path: Path, output_path: Path):
"""创建双语 EPUB"""
print(f"\n{'='*80}")
print(f"端到端测试: 生成双语 EPUB")
print(f"{'='*80}\n")
print(f"输入: {os.path.basename(epub_path)}")
print(f"输出: {os.path.basename(output_path)}")
# 1. 读取 EPUB
book = epub.read_epub(str(epub_path))
# 准备 Zip 读取以修复 CSS 链接
import zipfile
try:
input_zip = zipfile.ZipFile(epub_path, 'r')
zip_files = set(input_zip.namelist())
except Exception as e:
print(f"无法打开 Zip: {e}")
input_zip = None
zip_files = set()
extractor = FineGrainedExtractor()
translator = MockTranslator() # Using the new MockTranslator class
# Statistics variables
total_docs = 0
total_elements = 0
total_translated = 0
print("\n" + "="*80)
print("处理统计")
print("="*80 + "\n")
# 逐个文档处理
for item in book.get_items():
if item.get_type() == 9: # ITEM_DOCUMENT
try:
# 尝试从 Zip 读取原始内容
file_name = item.get_name()
content = None
if input_zip:
# 尝试精确匹配
if file_name in zip_files:
try:
content = input_zip.read(file_name).decode('utf-8')
except:
pass
else:
# 尝试模糊匹配 (处理路径前缀问题)
# 例如 item name 是 'dummy.html', zip 是 'EPUB/dummy.html'
for z_name in zip_files:
if z_name.endswith(file_name) or file_name.endswith(z_name):
try:
content = input_zip.read(z_name).decode('utf-8')
# print(f"Zip 模糊匹配: {file_name} -> {z_name}")
break
except:
pass
if content is None:
content = item.get_content().decode('utf-8')
if not content.strip():
continue
# 修复 item 的 links (如果从 Zip 读到了 link)
from bs4 import BeautifulSoup
if input_zip: # Only attempt if zipfile was successfully opened
soup = BeautifulSoup(content, 'html.parser')
head = soup.find('head')
if head:
links = head.find_all('link', rel='stylesheet')
for link in links:
href = link.get('href')
if href:
# 检查是否已存在
existing_links = list(item.get_links())
exists = False
for l in existing_links:
l_href = getattr(l, 'href', None) or (l.get('href') if isinstance(l, dict) else None)
if l_href == href:
exists = True
break
if not exists:
item.add_link(href=href, rel='stylesheet', type='text/css')
# 提取
items = extractor.extract(content, file_name)
if not items:
continue
total_docs += 1
total_elements += len(items)
# 构建翻译映射
translation_map = {}
for elem in items:
if elem['should_translate']:
original_text = elem['text']
translated_text = translator.translate(original_text)
translation_map[original_text] = translated_text
total_translated += 1
# 回填
if translation_map:
modified_html = extractor.backfill(items, translation_map)
item.set_content(modified_html.encode('utf-8'))
except Exception as e:
logger.error(f"处理失败 {item.get_name()}: {e}")
# 修复 TOC：补全 UID 并移除指向不存在文件的死链
def fix_and_clean_toc(toc, book):
new_toc = []
import uuid
from ebooklib.epub import Link
for item in toc:
if isinstance(item, (tuple, list)):
section, children = item
cleaned_children = fix_and_clean_toc(children, book)
if isinstance(section, Link):
href = section.href.split('#')[0]
if book.get_item_with_href(href):
if section.uid is None:
section.uid = f'uuid-{uuid.uuid4()}'
new_toc.append((section, cleaned_children))
else:
print(f"移除无效 TOC 节点: {section.href}")
new_toc.extend(cleaned_children)
else:
new_toc.append((section, cleaned_children))
elif isinstance(item, Link):
href = item.href.split('#')[0]
if book.get_item_with_href(href):
if item.uid is None:
item.uid = f'uuid-{uuid.uuid4()}'
new_toc.append(item)
else:
print(f"移除无效 TOC 节点: {item.href}")
else:
new_toc.append(item)
return new_toc
try:
book.toc = fix_and_clean_toc(book.toc, book)
except Exception as e:
print(f"修复 TOC 失败: {e}")
# 保存
epub.write_epub(str(output_path), book)
# 统计
print(f"{'='*80}")
print("处理统计")
print(f"{'='*80}\n")
print(f"处理文档数: {total_docs}")
print(f"提取元素数: {total_elements:,}")
print(f"翻译元素数: {total_translated:,}")
print(f"\n✅ 双语 EPUB 已生成: {output_path}\n")
def verify_bilingual_epub(epub_path: Path):
"""验证双语 EPUB"""
print(f"{'='*80}")
print(f"验证双语 EPUB")
print(f"{'='*80}\n")
book = epub.read_epub(str(epub_path))
# 检查第一个有内容的核心文档
for item in book.get_items():
if item.get_type() == 9:
content = item.get_content().decode('utf-8')
# 跳过空文档
if len(content) < 100:
continue
# 检查是否包含 [中文] 标记
if '[中文]' in content:
count = content.count('[中文]')
print(f"✅ 发现 {count} 个翻译标记\n")
# 显示部分内容
from bs4 import BeautifulSoup
soup = BeautifulSoup(content, 'html.parser')
paragraphs = soup.find_all('p')
print(f"段落总数: {len(paragraphs)}\n")
print("前10个段落:\n")
for i, p in enumerate(paragraphs[:10], 1):
text = p.get_text(strip=True)
preview = text[:80]
if len(text) > 80:
preview += "..."
# 标记译文段落
is_translation = 'translation' in p.get('class', [])
marker = " [译文]" if is_translation else " [原文]"
print(f"{i}. {preview}{marker}")
print()
# 找到一个有效的验证文件后退出循环
break
else:
print("ℹ️ 该文档无翻译标记 (可能无翻译内容)，继续查找下一个...\n")
continue
else:
# 如果循环结束还没找到
print("❌ 在所有文档中均未发现翻译标记！\n")
def main():
"""主函数"""
import sys
logger.remove()
logger.add(sys.stderr, level="ERROR")
from simple_cleaner import clean_epub
input_file = "On_China_Henry_Kissinger.epub"
if len(sys.argv) > 1:
input_file = sys.argv[1]
# 推断路径
epub_name = Path(input_file).name
epub_stem = Path(input_file).stem
# 查找输入文件
input_path = Path(input_file)
if not input_path.exists():
input_path = project_root / "input" / epub_name
if not input_path.exists():
print(f"❌ 输入文件不存在: {input_path}")
# 尝试看看是不是已经在 test_output 下的 cleaned 文件
cleaned_path = project_root / "test_output" / input_file
if cleaned_path.exists() and "cleaned" in str(cleaned_path):
print(f"⚠️ 检测到已清理文件，跳过清理步骤: {cleaned_path}")
else:
return
else:
# 执行清理
cleaned_file = f"{epub_stem}_cleaned.epub"
cleaned_path = project_root / "test_output" / cleaned_file
print(f"正在清理: {input_path.name} -> {cleaned_path.name}")
try:
clean_epub(str(input_path), str(cleaned_path))
except Exception as e:
print(f"❌ 清理失败: {e}")
return
# 生成双语
bilingual_file = f"{epub_stem}_bilingual.epub"
if "cleaned" in epub_stem:
bilingual_file = epub_stem.replace("_cleaned", "_bilingual") + ".epub"
bilingual_path = project_root / "test_output" / bilingual_file
# 生成双语 EPUB
create_bilingual_epub(cleaned_path, bilingual_path)
# 验证
verify_bilingual_epub(bilingual_path)
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/test_extraction.py
+368
View File
@@ -0,0 +1,368 @@
"""
文本提取实验主测试脚本
对比不同提取方案的效果,生成详细报告
"""
import sys
from pathlib import Path
from datetime import datetime
from loguru import logger
# 添加项目根目录到路径
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
from extractors.bs4_optimized import BS4OptimizedExtractor
from extractors.lxml_xpath import LxmlXPathExtractor
from extractors.baseline_pandoc import PandocBaseline
from validators.completeness_check import CompletenessValidator
from validators.backfill_check import BackfillValidator
class ExtractionExperiment:
"""文本提取实验"""
def __init__(self, epub_path: str):
"""
初始化实验
Args:
epub_path: ePub 文件路径
"""
self.epub_path = Path(epub_path)
if not self.epub_path.exists():
raise FileNotFoundError(f"ePub 文件不存在: {epub_path}")
# 初始化提取器
self.bs4_extractor = BS4OptimizedExtractor()
self.lxml_extractor = LxmlXPathExtractor()
self.pandoc_baseline = PandocBaseline()
# 初始化验证器
self.completeness_validator = CompletenessValidator()
self.backfill_validator = BackfillValidator()
# 加载 ePub
self.book = epub.read_epub(str(self.epub_path))
logger.info(f"加载 ePub: {self.epub_path.name}")
def run_experiment(self) -> dict:
"""
运行完整实验
Returns:
实验结果字典
"""
results = {
'file_name': self.epub_path.name,
'timestamp': datetime.now().isoformat(),
'methods': {}
}
# 1. 获取 Pandoc 基准
logger.info("步骤 1: 提取 Pandoc 基准")
baseline_text = self.pandoc_baseline.extract_from_epub(str(self.epub_path))
if baseline_text:
results['baseline_length'] = len(baseline_text)
logger.info(f"Pandoc 基准: {len(baseline_text)} 字符")
else:
logger.warning("Pandoc 基准提取失败,将跳过覆盖率对比")
results['baseline_length'] = 0
# 2. 获取测试 HTML 内容
logger.info("步骤 2: 提取 ePub 中的 HTML 内容")
html_contents = self._extract_html_from_epub()
logger.info(f"提取了 {len(html_contents)} 个 HTML 文档")
if not html_contents:
logger.error("未找到 HTML 内容")
return results
# 合并所有 HTML(用于整体测试)
combined_html = "\n\n".join(html_contents)
# 3. 测试 BS4 优化方案
logger.info("步骤 3: 测试 BS4 优化方案")
bs4_results = self._test_extractor(
"BS4 优化方案",
self.bs4_extractor,
combined_html,
baseline_text
)
results['methods']['BS4 优化方案'] = bs4_results
# 4. 测试 lxml 方案
logger.info("步骤 4: 测试 lxml 方案")
lxml_results = self._test_extractor(
"lxml XPath 方案",
self.lxml_extractor,
combined_html,
baseline_text
)
results['methods']['lxml XPath 方案'] = lxml_results
return results
def _extract_html_from_epub(self) -> list:
"""从 ePub 中提取所有 HTML 文档"""
html_contents = []
for item in self.book.get_items():
if item.get_type() == 9: # ITEM_DOCUMENT
try:
content = item.get_content().decode('utf-8')
html_contents.append(content)
except Exception as e:
logger.warning(f"解码失败 {item.get_name()}: {e}")
return html_contents
def _test_extractor(self, method_name: str, extractor, html_content: str,
baseline_text: str = None) -> dict:
"""
测试单个提取器
Args:
method_name: 方案名称
extractor: 提取器实例
html_content: HTML 内容
baseline_text: Pandoc 基准文本
Returns:
测试结果字典
"""
results = {}
try:
# 1. 提取文本
items = extractor.extract(html_content)
results['element_count'] = len(items)
# 合并提取的文本
extracted_text = " ".join([item['text'] for item in items])
results['text_length'] = len(extracted_text)
logger.info(f"{method_name}: 提取了 {len(items)} 个元素, {len(extracted_text)} 字符")
# 2. 完整性验证
if baseline_text:
coverage = self.completeness_validator.calculate_coverage(
extracted_text, baseline_text
)
similarity = self.completeness_validator.calculate_similarity(
extracted_text, baseline_text
)
missing_segments = self.completeness_validator.find_missing_segments(
extracted_text, baseline_text
)
results['coverage'] = coverage
results['similarity'] = similarity
results['missing_segments'] = missing_segments
logger.info(f"{method_name}: 覆盖率 {coverage:.2%}, 相似度 {similarity:.2%}")
# 3. 回填验证
logger.info(f"{method_name}: 测试回填准确性")
# 位置准确性验证
success, failed = self.backfill_validator.validate_position_accuracy(
html_content, items, extractor
)
results['position_accuracy'] = success / (success + failed) if (success + failed) > 0 else 0
results['position_success'] = success
results['position_failed'] = failed
logger.info(f"{method_name}: 位置准确性 {results['position_accuracy']:.2%}")
# 模拟翻译回填
backfilled_html, backfill_results = self.backfill_validator.simulate_translation_backfill(
html_content, items, extractor
)
results['backfill_accuracy'] = backfill_results['accuracy']
results['backfill_success'] = backfill_results['success']
results['backfill_failed'] = backfill_results['failed']
logger.info(f"{method_name}: 回填准确性 {results['backfill_accuracy']:.2%}")
except Exception as e:
logger.error(f"{method_name} 测试失败: {e}")
results['error'] = str(e)
return results
def generate_report(self, results: dict) -> str:
"""
生成实验报告
Args:
results: 实验结果
Returns:
Markdown 格式的报告
"""
report = ["# 文本提取实验报告\n"]
# 基本信息
report.append("## 基本信息\n")
report.append(f"- **测试文件**: {results['file_name']}")
report.append(f"- **测试时间**: {results['timestamp']}")
report.append(f"- **Pandoc 基准长度**: {results.get('baseline_length', 0):,} 字符\n")
# 方案对比表
report.append("## 方案对比\n")
report.append("### 提取完整性\n")
report.append("| 方案 | 提取元素数 | 文本长度 | 覆盖率 | 相似度 |")
report.append("|------|-----------|---------|--------|--------|")
for method_name, method_results in results.get('methods', {}).items():
if 'error' in method_results:
report.append(f"| {method_name} | ❌ 错误 | - | - | - |")
else:
report.append(
f"| {method_name} | "
f"{method_results.get('element_count', 0):,} | "
f"{method_results.get('text_length', 0):,} | "
f"{method_results.get('coverage', 0):.2%} | "
f"{method_results.get('similarity', 0):.2%} |"
)
report.append("")
# 回填准确性
report.append("### 回填准确性\n")
report.append("| 方案 | 位置准确性 | 回填准确性 | 成功/失败 |")
report.append("|------|-----------|-----------|----------|")
for method_name, method_results in results.get('methods', {}).items():
if 'error' not in method_results:
report.append(
f"| {method_name} | "
f"{method_results.get('position_accuracy', 0):.2%} | "
f"{method_results.get('backfill_accuracy', 0):.2%} | "
f"{method_results.get('backfill_success', 0)}/{method_results.get('backfill_failed', 0)} |"
)
report.append("")
# 详细分析
report.append("## 详细分析\n")
for method_name, method_results in results.get('methods', {}).items():
report.append(f"### {method_name}\n")
if 'error' in method_results:
report.append(f"**错误**: {method_results['error']}\n")
continue
# 统计信息
report.append(f"- 提取元素数: {method_results.get('element_count', 0):,}")
report.append(f"- 文本总长度: {method_results.get('text_length', 0):,} 字符")
if 'coverage' in method_results:
report.append(f"- 覆盖率: {method_results['coverage']:.2%}")
report.append(f"- 相似度: {method_results['similarity']:.2%}")
report.append(f"- 位置准确性: {method_results.get('position_accuracy', 0):.2%}")
report.append(f"- 回填准确性: {method_results.get('backfill_accuracy', 0):.2%}")
# 缺失片段
missing = method_results.get('missing_segments', [])
if missing:
report.append(f"\n**缺失片段** ({len(missing)} 个):\n")
for i, segment in enumerate(missing[:3], 1):
report.append(f"{i}. {segment[:80]}...")
if len(missing) > 3:
report.append(f"\n... 还有 {len(missing) - 3} 个片段")
report.append("")
# 结论
report.append("## 结论\n")
# 找出最佳方案
best_method = None
best_score = 0
for method_name, method_results in results.get('methods', {}).items():
if 'error' in method_results:
continue
# 综合评分: 覆盖率 40% + 回填准确性 60%
score = (
method_results.get('coverage', 0) * 0.4 +
method_results.get('backfill_accuracy', 0) * 0.6
)
if score > best_score:
best_score = score
best_method = method_name
if best_method:
report.append(f"**推荐方案**: {best_method} (综合评分: {best_score:.2%})\n")
report.append("评分标准: 覆盖率 40% + 回填准确性 60%")
return "\n".join(report)
def main():
"""主函数"""
# 配置日志
logger.remove()
logger.add(sys.stderr, level="INFO")
# 测试文件
test_files = [
"input/Gambling Man.epub",
"input/On_China_Henry_Kissinger.epub",
"input/The World Atlas of Coffee - From Beans to Brewing - Coffees Explored, Explained and Enjoyed (James Hoffmann) (Z-Library).epub"
]
project_root = Path(__file__).parent.parent.parent
for test_file in test_files:
epub_path = project_root / test_file
if not epub_path.exists():
logger.warning(f"跳过不存在的文件: {test_file}")
continue
logger.info(f"\n{'='*60}")
logger.info(f"测试文件: {test_file}")
logger.info(f"{'='*60}\n")
try:
# 运行实验
experiment = ExtractionExperiment(str(epub_path))
results = experiment.run_experiment()
# 生成报告
report = experiment.generate_report(results)
# 保存报告
report_dir = project_root / "tests" / "extraction_experiment" / "reports"
report_dir.mkdir(parents=True, exist_ok=True)
report_file = report_dir / f"{epub_path.stem}_report.md"
with open(report_file, 'w', encoding='utf-8') as f:
f.write(report)
logger.info(f"报告已保存: {report_file}")
# 打印摘要
print("\n" + "="*60)
print(report)
print("="*60 + "\n")
except Exception as e:
logger.error(f"实验失败: {e}")
import traceback
traceback.print_exc()
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/test_fine_grained.py
+95
View File
@@ -0,0 +1,95 @@
"""
测试细粒度提取器
"""
import sys
from pathlib import Path
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
from loguru import logger
sys.path.insert(0, str(Path(__file__).parent))
from extractors.fine_grained import FineGrainedExtractor
def test_fine_grained():
"""测试细粒度提取"""
cleaned_path = project_root / "test_output" / "On_China_cleaned_v2.epub"
if not cleaned_path.exists():
print(f"❌ 清理后的 ePub 不存在: {cleaned_path}")
print("请先运行清理器生成 cleaned_v2.epub")
return
print("\n" + "="*80)
print("细粒度提取测试")
print("="*80 + "\n")
# 加载 ePub
book = epub.read_epub(str(cleaned_path))
# 测试第一个文档
for item in book.get_items():
if item.get_type() == 9 and 'dummy_split_010' in item.get_name():
content = item.get_content().decode('utf-8')
print(f"测试文件: {item.get_name()}\n")
# 提取
extractor = FineGrainedExtractor()
items = extractor.extract(content, item.get_name())
print(f"✅ 提取了 {len(items)} 个 <p> 元素\n")
# 显示前10个
print("前10个元素:")
for i, elem in enumerate(items[:10], 1):
print(f" {i}. {elem['text'][:60]}...")
print(f"\n... (共 {len(items)} 个)")
# 模拟翻译
translation_map = {}
for elem in items:
if elem['should_translate']:
translation_map[elem['text']] = f"{elem['text']} [翻译]"
print(f"\n待翻译: {len(translation_map)} 个元素")
# 回填
result_html = extractor.backfill(items, translation_map)
# 验证
print("\n" + "="*80)
print("验证:")
print("="*80 + "\n")
if '[翻译]' in result_html:
print("✅ 翻译成功回填")
else:
print("❌ 翻译未回填")
# 检查 <p> 数量
from bs4 import BeautifulSoup
result_soup = BeautifulSoup(result_html, 'html.parser')
result_p_count = len(result_soup.find_all('p'))
print(f"✅ 回填后 <p> 元素: {result_p_count} 个")
break
def main():
"""主函数"""
logger.remove()
logger.add(sys.stderr, level="INFO")
test_fine_grained()
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/test_fine_grained_extractor.py
+177
View File
@@ -0,0 +1,177 @@
"""
测试细粒度提取器
验证 fine_grained.py 的提取效果
"""
import sys
from pathlib import Path
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
from loguru import logger
sys.path.insert(0, str(Path(__file__).parent))
from extractors.fine_grained import FineGrainedExtractor
def test_fine_grained_extraction(epub_path: Path):
"""测试细粒度提取器"""
print(f"\n{'='*80}")
print(f"细粒度提取器测试: {epub_path.name}")
print(f"{'='*80}\n")
# 加载 EPUB
book = epub.read_epub(str(epub_path))
# 统计信息
total_docs = 0
total_elements = 0
total_to_translate = 0
total_decorative = 0
doc_stats = []
# 逐个文档提取
for item in book.get_items():
if item.get_type() == 9:
try:
content = item.get_content().decode('utf-8')
file_name = item.get_name()
# 提取
extractor = FineGrainedExtractor()
items = extractor.extract(content, file_name)
if items:
total_docs += 1
total_elements += len(items)
to_translate = sum(1 for i in items if i['should_translate'])
decorative = sum(1 for i in items if i.get('is_decorative'))
total_to_translate += to_translate
total_decorative += decorative
doc_stats.append({
'name': file_name,
'total': len(items),
'to_translate': to_translate,
'decorative': decorative,
'doc_type': items[0]['doc_type'] if items else 'unknown'
})
except Exception as e:
logger.error(f"处理失败 {item.get_name()}: {e}")
# 总体统计
print(f"{'='*80}")
print("总体统计")
print(f"{'='*80}\n")
print(f"处理文档数: {total_docs}")
print(f"提取元素总数: {total_elements:,}")
print(f"需要翻译: {total_to_translate:,} ({total_to_translate/total_elements*100:.1f}%)")
print(f"装饰性元素: {total_decorative:,} ({total_decorative/total_elements*100:.1f}%)")
print()
# 按文档类型分组
core_docs = [d for d in doc_stats if d['doc_type'] == 'core']
toc_docs = [d for d in doc_stats if d['doc_type'] == 'toc']
skip_docs = [d for d in doc_stats if d['doc_type'] == 'skip']
print(f"{'='*80}")
print("按文档类型统计")
print(f"{'='*80}\n")
if core_docs:
core_elements = sum(d['total'] for d in core_docs)
core_translate = sum(d['to_translate'] for d in core_docs)
print(f"核心文档 (core): {len(core_docs)} 个")
print(f" - 元素数: {core_elements:,}")
print(f" - 需翻译: {core_translate:,}")
print()
if toc_docs:
toc_elements = sum(d['total'] for d in toc_docs)
toc_translate = sum(d['to_translate'] for d in toc_docs)
print(f"目录文档 (toc): {len(toc_docs)} 个")
print(f" - 元素数: {toc_elements:,}")
print(f" - 需翻译: {toc_translate:,}")
print()
if skip_docs:
skip_elements = sum(d['total'] for d in skip_docs)
print(f"跳过文档 (skip): {len(skip_docs)} 个")
print(f" - 元素数: {skip_elements:,}")
print()
# 显示部分文档详情
print(f"{'='*80}")
print("核心文档详情 (前10个)")
print(f"{'='*80}\n")
for i, doc in enumerate(core_docs[:10], 1):
print(f"{i}. {doc['name']}")
print(f" 元素: {doc['total']}, 翻译: {doc['to_translate']}, 装饰: {doc['decorative']}")
if len(core_docs) > 10:
print(f"\n... 还有 {len(core_docs) - 10} 个核心文档\n")
# 抽样显示提取内容
print(f"\n{'='*80}")
print("提取内容抽样 (第一个核心文档的前10个元素)")
print(f"{'='*80}\n")
if core_docs:
first_doc_name = core_docs[0]['name']
# 重新提取第一个文档
for item in book.get_items():
if item.get_type() == 9 and item.get_name() == first_doc_name:
content = item.get_content().decode('utf-8')
extractor = FineGrainedExtractor()
items = extractor.extract(content, first_doc_name)
print(f"文档: {first_doc_name}\n")
for i, elem in enumerate(items[:10], 1):
translate_flag = "✓" if elem['should_translate'] else "✗"
decorative_flag = " [装饰]" if elem.get('is_decorative') else ""
text_preview = elem['text'][:60]
if len(elem['text']) > 60:
text_preview += "..."
print(f"{i}. [{translate_flag}] <{elem['tag']}> {text_preview}{decorative_flag}")
if len(items) > 10:
print(f"\n... 还有 {len(items) - 10} 个元素")
break
print()
def main():
"""主函数"""
logger.remove()
logger.add(sys.stderr, level="ERROR")
# 测试清理后的 EPUB
cleaned_file = "On_China_cleaned.epub"
cleaned_path = project_root / "test_output" / cleaned_file
if not cleaned_path.exists():
print(f"❌ 清理文件不存在: {cleaned_path}")
print(f"\n提示: 请先运行清理器:")
print(f" python tests/extraction_experiment/simple_cleaner.py \\")
print(f" 'input/On_China_Henry_Kissinger.epub' \\")
print(f" 'test_output/{cleaned_file}'")
return
test_fine_grained_extraction(cleaned_path)
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/test_multi_epub.py
+272
View File
@@ -0,0 +1,272 @@
"""
优化的完整性测试脚本
快速对比多个 ePub 的提取完整性,与 Pandoc 基准对比
"""
import sys
from pathlib import Path
from datetime import datetime
from loguru import logger
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
sys.path.insert(0, str(Path(__file__).parent))
from extractors.enhanced_bs4 import EnhancedBS4Extractor
from extractors.baseline_pandoc import PandocBaseline
def quick_coverage_check(extracted_text: str, baseline_text: str) -> dict:
"""
快速覆盖率检查(优化版)
使用简化的词级别对比,避免复杂的相似度计算
"""
import re
# 标准化
def normalize(text):
text = text.lower()
text = re.sub(r'[^\w\s]', ' ', text)
text = re.sub(r'\s+', ' ', text)
return text.strip()
extracted_norm = normalize(extracted_text)
baseline_norm = normalize(baseline_text)
# 分词
extracted_words = set(extracted_norm.split())
baseline_words = set(baseline_norm.split())
if not baseline_words:
return {'coverage': 0.0, 'common_words': 0, 'baseline_words': 0}
common = extracted_words & baseline_words
coverage = len(common) / len(baseline_words)
return {
'coverage': coverage,
'common_words': len(common),
'baseline_words': len(baseline_words),
'extracted_words': len(extracted_words)
}
def test_single_epub(epub_path: Path, use_pandoc: bool = True) -> dict:
"""
测试单个 ePub 文件
Args:
epub_path: ePub 文件路径
use_pandoc: 是否使用 Pandoc 基准
Returns:
测试结果字典
"""
result = {
'file_name': epub_path.name,
'file_size': epub_path.stat().st_size,
'timestamp': datetime.now().isoformat()
}
try:
# 1. Pandoc 基准(可选)
baseline_text = None
if use_pandoc:
logger.info(f"提取 Pandoc 基准: {epub_path.name}")
pandoc = PandocBaseline()
baseline_text = pandoc.extract_from_epub(str(epub_path))
if baseline_text:
result['baseline_length'] = len(baseline_text)
result['baseline_words'] = len(baseline_text.split())
# 2. 加载 ePub
logger.info(f"加载 ePub: {epub_path.name}")
book = epub.read_epub(str(epub_path))
# 3. 提取 HTML 内容
html_docs = []
for item in book.get_items():
if item.get_type() == 9: # ITEM_DOCUMENT
try:
content = item.get_content().decode('utf-8')
html_docs.append(content)
except:
continue
result['html_doc_count'] = len(html_docs)
# 合并 HTML
combined_html = "\n\n".join(html_docs)
# 4. 增强提取器测试
logger.info(f"测试增强提取器")
extractor = EnhancedBS4Extractor(preserve_decorative=True)
items = extractor.extract(combined_html)
# 统计
decorative_items = [i for i in items if i.get('is_decorative')]
nav_items = [i for i in items if i.get('is_navigation')]
content_items = [i for i in items if not i.get('is_navigation') and not i.get('is_decorative')]
result['total_elements'] = len(items)
result['content_elements'] = len(content_items)
result['decorative_elements'] = len(decorative_items)
result['navigation_elements'] = len(nav_items)
# 提取的文本
extracted_text = " ".join([item['text'] for item in content_items])
result['extracted_length'] = len(extracted_text)
result['extracted_words'] = len(extracted_text.split())
# 5. 与 Pandoc 对比
if baseline_text:
coverage_result = quick_coverage_check(extracted_text, baseline_text)
result['coverage'] = coverage_result['coverage']
result['common_words'] = coverage_result['common_words']
result['status'] = 'success'
except Exception as e:
logger.error(f"测试失败 {epub_path.name}: {e}")
result['status'] = 'failed'
result['error'] = str(e)
return result
def generate_summary_report(results: list) -> str:
"""生成汇总报告"""
report = ["# 多 ePub 提取完整性测试报告\n"]
report.append(f"**测试时间**: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n")
report.append(f"**测试文件数**: {len(results)}\n")
# 成功/失败统计
success_count = sum(1 for r in results if r.get('status') == 'success')
report.append(f"**成功**: {success_count}/{len(results)}\n")
# 汇总表
report.append("## 测试结果汇总\n")
report.append("| 文件名 | 文件大小 | 提取元素 | 装饰性 | 文本长度 | 覆盖率 |")
report.append("|--------|---------|---------|--------|---------|--------|")
for r in results:
if r.get('status') != 'success':
report.append(f"| {r['file_name'][:30]} | - | ❌ 失败 | - | - | - |")
continue
file_size = f"{r.get('file_size', 0) / 1024:.1f}KB"
total_elem = r.get('total_elements', 0)
decorative = r.get('decorative_elements', 0)
text_len = f"{r.get('extracted_length', 0):,}"
coverage = r.get('coverage', 0)
coverage_str = f"{coverage:.1%}" if coverage > 0 else "N/A"
report.append(
f"| {r['file_name'][:30]} | {file_size} | {total_elem} | {decorative} | {text_len} | {coverage_str} |"
)
report.append("")
# 详细分析
report.append("## 详细分析\n")
for r in results:
if r.get('status') != 'success':
continue
report.append(f"### {r['file_name']}\n")
report.append(f"- **HTML 文档数**: {r.get('html_doc_count', 0)}")
report.append(f"- **提取元素总数**: {r.get('total_elements', 0)}")
report.append(f" - 内容元素: {r.get('content_elements', 0)}")
report.append(f" - 装饰性元素: {r.get('decorative_elements', 0)}")
report.append(f" - 导航元素: {r.get('navigation_elements', 0)}")
report.append(f"- **提取文本长度**: {r.get('extracted_length', 0):,} 字符")
report.append(f"- **提取词数**: {r.get('extracted_words', 0):,}")
if 'baseline_length' in r:
report.append(f"- **Pandoc 基准长度**: {r.get('baseline_length', 0):,} 字符")
report.append(f"- **覆盖率**: {r.get('coverage', 0):.2%}")
report.append(f"- **共同词数**: {r.get('common_words', 0):,}")
report.append("")
# 总结
report.append("## 总结\n")
if success_count > 0:
avg_coverage = sum(r.get('coverage', 0) for r in results if r.get('status') == 'success') / success_count
total_decorative = sum(r.get('decorative_elements', 0) for r in results if r.get('status') == 'success')
report.append(f"- **平均覆盖率**: {avg_coverage:.2%}")
report.append(f"- **总装饰性元素**: {total_decorative} 个")
report.append(f"- **提取器状态**: {'✅ 正常' if avg_coverage > 0.9 else '⚠️ 需要优化'}")
return "\n".join(report)
def main():
"""主函数"""
logger.remove()
logger.add(sys.stderr, level="INFO")
# 测试文件列表
test_files = [
"Gambling Man.epub",
"On_China_Henry_Kissinger.epub",
"The World Atlas of Coffee - From Beans to Brewing - Coffees Explored, Explained and Enjoyed (James Hoffmann) (Z-Library).epub",
"The_Philosopher_in_the_Valley.epub",
"To_Explain_the_World.epub"
]
input_dir = project_root / "input"
results = []
for filename in test_files:
epub_path = input_dir / filename
if not epub_path.exists():
logger.warning(f"跳过不存在的文件: {filename}")
continue
logger.info(f"\n{'='*60}")
logger.info(f"测试: {filename}")
logger.info(f"{'='*60}")
result = test_single_epub(epub_path, use_pandoc=True)
results.append(result)
# 打印简要结果
if result.get('status') == 'success':
logger.info(f"✅ 成功: {result.get('total_elements')} 个元素, "
f"{result.get('decorative_elements')} 个装饰性, "
f"覆盖率 {result.get('coverage', 0):.1%}")
else:
logger.error(f"❌ 失败: {result.get('error')}")
# 生成报告
report = generate_summary_report(results)
# 保存报告
report_dir = project_root / "tests" / "extraction_experiment" / "reports"
report_dir.mkdir(parents=True, exist_ok=True)
report_file = report_dir / "multi_epub_test_report.md"
with open(report_file, 'w', encoding='utf-8') as f:
f.write(report)
logger.info(f"\n报告已保存: {report_file}")
# 打印报告
print("\n" + "="*60)
print(report)
print("="*60)
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/test_one_to_one.py
+103
View File
@@ -0,0 +1,103 @@
"""
测试一比一对应提取器
"""
import sys
from pathlib import Path
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from loguru import logger
sys.path.insert(0, str(Path(__file__).parent))
from extractors.one_to_one import OneToOneExtractor
def test_one_to_one():
"""测试一比一对应"""
# 测试 HTML
html = """
<div class="poem">
<div class="line"><span class="italic">War is</span></div>
<div class="line"><span class="italic">A grave affair of the state;</span></div>
<div class="line"><span class="italic">It is a place</span></div>
</div>
"""
print("\n" + "="*80)
print("一比一对应测试")
print("="*80 + "\n")
print("原始 HTML:")
print(html)
print()
# 提取
extractor = OneToOneExtractor()
items = extractor.extract(html)
print(f"提取了 {len(items)} 个元素:\n")
for i, item in enumerate(items, 1):
print(f"{i}. <{item['tag']}> {item['text']}")
print(f" 文本节点数: {len(item['text_nodes'])}")
for j, (node, text) in enumerate(item['text_nodes'], 1):
print(f" 节点 {j}: '{text}'")
print()
# 创建翻译映射
translation_map = {}
for item in items:
if item['should_translate']:
translation_map[item['text']] = f"{item['text']} [翻译]"
print(f"待翻译: {len(translation_map)} 个元素\n")
# 回填
result_html = extractor.backfill(items, translation_map)
print("="*80)
print("回填后的 HTML:")
print("="*80 + "\n")
print(result_html)
# 验证
print("\n" + "="*80)
print("验证:")
print("="*80 + "\n")
if '<span class="italic">War is [翻译]</span>' in result_html:
print("✅ 第1行格式保留")
else:
print("❌ 第1行格式丢失")
if '<span class="italic">A grave affair of the state; [翻译]</span>' in result_html:
print("✅ 第2行格式保留")
else:
print("❌ 第2行格式丢失")
if '<span class="italic">It is a place [翻译]</span>' in result_html:
print("✅ 第3行格式保留")
else:
print("❌ 第3行格式丢失")
div_count = result_html.count('<div class="line">')
if div_count == 3:
print("✅ 所有3个 div 都保留")
else:
print(f"❌ div 数量错误: {div_count}")
def main():
"""主函数"""
logger.remove()
logger.add(sys.stderr, level="INFO")
test_one_to_one()
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/test_simple.py
+202
View File
@@ -0,0 +1,202 @@
"""
简化的提取测试脚本
快速验证提取器的基本功能
"""
import sys
from pathlib import Path
# 添加项目根目录到路径
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
from bs4 import BeautifulSoup
from loguru import logger
# 导入提取器
sys.path.insert(0, str(Path(__file__).parent))
from extractors.bs4_optimized import BS4OptimizedExtractor
from extractors.lxml_xpath import LxmlXPathExtractor
def test_simple_html():
"""测试简单的 HTML 提取"""
html = """
<html>
<body>
<h1>Chapter 1</h1>
<p>This is the first paragraph.</p>
<div class="nav">Navigation</div>
<p>This is the second paragraph.</p>
<blockquote>A quote here.</blockquote>
</body>
</html>
"""
print("\n" + "="*60)
print("测试 1: 简单 HTML 提取")
print("="*60)
# BS4 提取器
print("\n--- BS4 优化方案 ---")
bs4_extractor = BS4OptimizedExtractor(min_text_length=5)
bs4_items = bs4_extractor.extract(html)
print(f"提取元素数: {len(bs4_items)}")
for i, item in enumerate(bs4_items, 1):
nav_flag = " [导航]" if item['is_navigation'] else ""
print(f"{i}. [{item['tag']}] {item['text'][:50]}{nav_flag}")
print(f" 路径: {item['path']}")
# lxml 提取器
print("\n--- lxml XPath 方案 ---")
lxml_extractor = LxmlXPathExtractor(min_text_length=5)
lxml_items = lxml_extractor.extract(html)
print(f"提取元素数: {len(lxml_items)}")
for i, item in enumerate(lxml_items, 1):
nav_flag = " [导航]" if item['is_navigation'] else ""
print(f"{i}. [{item['tag']}] {item['text'][:50]}{nav_flag}")
print(f" XPath: {item['xpath']}")
return bs4_items, lxml_items
def test_backfill(html, items, extractor, method_name):
"""测试回填功能"""
print(f"\n--- {method_name} 回填测试 ---")
# 创建模拟翻译
translation_map = {}
for i, item in enumerate(items):
if not item.get('is_navigation', False):
path = item.get('path') or item.get('xpath')
translation_map[path] = f"TRANSLATED_{i:02d}"
print(f"待回填: {len(translation_map)} 个元素")
# 执行回填
backfilled_html = extractor.backfill(html, translation_map)
# 验证
soup = BeautifulSoup(backfilled_html, 'html.parser')
found_count = 0
for path, translation in translation_map.items():
if translation in soup.get_text():
found_count += 1
accuracy = found_count / len(translation_map) if translation_map else 0
print(f"回填准确性: {accuracy:.2%} ({found_count}/{len(translation_map)})")
return accuracy
def test_epub_extraction(epub_path):
"""测试真实 ePub 文件的提取"""
print("\n" + "="*60)
print(f"测试 2: ePub 文件提取 - {Path(epub_path).name}")
print("="*60)
# 加载 ePub
book = epub.read_epub(epub_path)
# 提取第一个 HTML 文档
html_content = None
for item in book.get_items():
if item.get_type() == 9: # ITEM_DOCUMENT
try:
html_content = item.get_content().decode('utf-8')
print(f"\n测试文件: {item.get_name()}")
break
except:
continue
if not html_content:
print("未找到 HTML 内容")
return
# BS4 提取
print("\n--- BS4 优化方案 ---")
bs4_extractor = BS4OptimizedExtractor()
bs4_items = bs4_extractor.extract(html_content)
print(f"提取元素数: {len(bs4_items)}")
print(f"总文本长度: {sum(len(item['text']) for item in bs4_items):,} 字符")
# 显示前5个元素
print("\n前 5 个元素:")
for i, item in enumerate(bs4_items[:5], 1):
print(f"{i}. [{item['tag']}] {item['text'][:80]}...")
# lxml 提取
print("\n--- lxml XPath 方案 ---")
lxml_extractor = LxmlXPathExtractor()
lxml_items = lxml_extractor.extract(html_content)
print(f"提取元素数: {len(lxml_items)}")
print(f"总文本长度: {sum(len(item['text']) for item in lxml_items):,} 字符")
# 显示前5个元素
print("\n前 5 个元素:")
for i, item in enumerate(lxml_items[:5], 1):
print(f"{i}. [{item['tag']}] {item['text'][:80]}...")
# 回填测试
print("\n" + "-"*60)
print("回填测试")
print("-"*60)
bs4_accuracy = test_backfill(html_content, bs4_items, bs4_extractor, "BS4")
lxml_accuracy = test_backfill(html_content, lxml_items, lxml_extractor, "lxml")
# 对比
print("\n" + "="*60)
print("对比总结")
print("="*60)
print(f"{'方案':<15} {'元素数':<10} {'回填准确性':<15}")
print("-"*60)
print(f"{'BS4 优化':<15} {len(bs4_items):<10} {bs4_accuracy:>13.2%}")
print(f"{'lxml XPath':<15} {len(lxml_items):<10} {lxml_accuracy:>13.2%}")
def main():
"""主函数"""
# 配置日志
logger.remove()
logger.add(sys.stderr, level="INFO")
# 测试 1: 简单 HTML
bs4_items, lxml_items = test_simple_html()
# 简单 HTML 回填测试
html = """
<html>
<body>
<h1>Chapter 1</h1>
<p>This is the first paragraph.</p>
<div class="nav">Navigation</div>
<p>This is the second paragraph.</p>
</body>
</html>
"""
bs4_extractor = BS4OptimizedExtractor(min_text_length=5)
lxml_extractor = LxmlXPathExtractor(min_text_length=5)
print("\n" + "="*60)
print("简单 HTML 回填测试")
print("="*60)
test_backfill(html, bs4_items, bs4_extractor, "BS4")
test_backfill(html, lxml_items, lxml_extractor, "lxml")
# 测试 2: 真实 ePub
epub_path = project_root / "input" / "Gambling Man.epub"
if epub_path.exists():
test_epub_extraction(str(epub_path))
else:
print(f"\n跳过 ePub 测试: 文件不存在 {epub_path}")
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/test_smart.py
+186
View File
@@ -0,0 +1,186 @@
"""
测试智能提取器
验证正文 100% 提取,非核心部分智能跳过
"""
import sys
from pathlib import Path
project_root = Path(__file__).parent.parent.parent
sys.path.insert(0, str(project_root))
from ebooklib import epub
from loguru import logger
sys.path.insert(0, str(Path(__file__).parent))
from extractors.smart_extractor import SmartExtractor
def extract_all_text_from_html(html_content: str) -> str:
"""提取 HTML 中的所有文本(基准)"""
from bs4 import BeautifulSoup
import re
soup = BeautifulSoup(html_content, 'html.parser')
for element in soup(['script', 'style', 'meta', 'link']):
element.decompose()
text = soup.get_text(separator=' ', strip=True)
text = re.sub(r'\s+', ' ', text)
return text.strip()
def test_smart_extraction(epub_path: Path):
"""测试智能提取器"""
print(f"\n{'='*80}")
print(f"智能提取器测试: {epub_path.name}")
print(f"{'='*80}\n")
# 加载 ePub
book = epub.read_epub(str(epub_path))
# 提取所有 HTML 文档
html_docs = []
for item in book.get_items():
if item.get_type() == 9:
try:
content = item.get_content().decode('utf-8')
html_docs.append({
'name': item.get_name(),
'content': content
})
except:
continue
print(f"找到 {len(html_docs)} 个 HTML 文档\n")
# 智能提取器
extractor = SmartExtractor(preserve_decorative=True)
# 分类统计
core_docs = []
non_core_docs = []
total_core_baseline = 0
total_core_extracted = 0
total_non_core_baseline = 0
total_non_core_translated = 0
total_non_core_skipped = 0
for doc in html_docs:
# 基准文本
baseline_text = extract_all_text_from_html(doc['content'])
baseline_len = len(baseline_text)
# 智能提取
items = extractor.extract(doc['content'], doc['name'])
# 分类
is_core = items[0]['is_core'] if items else True
translate_items = [i for i in items if i['should_translate']]
skip_items = [i for i in items if not i['should_translate'] and not i['is_decorative']]
extracted_text = " ".join([i['text'] for i in translate_items])
extracted_len = len(extracted_text)
if is_core:
core_docs.append({
'name': doc['name'],
'baseline_len': baseline_len,
'extracted_len': extracted_len,
'coverage': extracted_len / baseline_len if baseline_len > 0 else 0
})
total_core_baseline += baseline_len
total_core_extracted += extracted_len
else:
non_core_docs.append({
'name': doc['name'],
'baseline_len': baseline_len,
'translate_len': extracted_len,
'skip_len': sum(len(i['text']) for i in skip_items),
'translate_count': len(translate_items),
'skip_count': len(skip_items)
})
total_non_core_baseline += baseline_len
total_non_core_translated += extracted_len
total_non_core_skipped += sum(len(i['text']) for i in skip_items)
# 显示结果
print(f"{'='*80}")
print("核心文档 (正文) - 必须 100%")
print(f"{'='*80}\n")
for doc in core_docs[:10]:
coverage = doc['coverage']
status = "✅" if coverage >= 0.99 else "❌"
print(f"{status} {doc['name']}")
print(f" 基准: {doc['baseline_len']:,} | 提取: {doc['extracted_len']:,} | 覆盖率: {coverage:.2%}")
if len(core_docs) > 10:
print(f"\n... 还有 {len(core_docs) - 10} 个核心文档\n")
core_coverage = total_core_extracted / total_core_baseline if total_core_baseline > 0 else 0
print(f"\n**核心文档总体覆盖率: {core_coverage:.2%}**")
print(f"基准: {total_core_baseline:,} | 提取: {total_core_extracted:,}\n")
# 非核心文档
print(f"{'='*80}")
print("非核心文档 (目录/索引/参考文献) - 智能跳过")
print(f"{'='*80}\n")
for doc in non_core_docs:
print(f"📄 {doc['name']}")
print(f" 基准: {doc['baseline_len']:,} 字符")
print(f" 翻译: {doc['translate_count']} 个元素 ({doc['translate_len']:,} 字符)")
print(f" 跳过: {doc['skip_count']} 个元素 ({doc['skip_len']:,} 字符)")
if doc['baseline_len'] > 0:
translate_ratio = doc['translate_len'] / doc['baseline_len']
skip_ratio = doc['skip_len'] / doc['baseline_len']
print(f" 翻译比例: {translate_ratio:.1%} | 跳过比例: {skip_ratio:.1%}")
print()
print(f"非核心文档统计:")
print(f" - 总基准: {total_non_core_baseline:,} 字符")
print(f" - 翻译: {total_non_core_translated:,} 字符")
print(f" - 跳过: {total_non_core_skipped:,} 字符")
# 总体统计
print(f"\n{'='*80}")
print("总体统计")
print(f"{'='*80}\n")
total_baseline = total_core_baseline + total_non_core_baseline
total_extracted = total_core_extracted + total_non_core_translated
overall_coverage = total_extracted / total_baseline if total_baseline > 0 else 0
print(f"总基准: {total_baseline:,} 字符")
print(f"总提取: {total_extracted:,} 字符")
print(f"**总体覆盖率: {overall_coverage:.2%}**\n")
print(f"✅ 核心文档覆盖率: {core_coverage:.2%} (目标: 100%)")
if core_coverage >= 0.995:
print(" 状态: 达标 ✅")
else:
print(f" 状态: 需要改进 ⚠️ (差距: {(1.0 - core_coverage) * 100:.2f}%)")
def main():
"""主函数"""
logger.remove()
logger.add(sys.stderr, level="INFO")
test_file = "Gambling Man.epub"
epub_path = project_root / "input" / test_file
if not epub_path.exists():
print(f"文件不存在: {test_file}")
return
test_smart_extraction(epub_path)
if __name__ == "__main__":
main()
archive/v0.09/tests/extraction_experiment/validators/__init__.py
+1
View File
@@ -0,0 +1 @@
"""验证器模块"""
archive/v0.09/tests/extraction_experiment/validators/backfill_check.py
+212
View File
@@ -0,0 +1,212 @@
"""
回填验证器
验证翻译回填的准确性,确保每个翻译都回填到正确位置
"""
from bs4 import BeautifulSoup
from typing import List, Dict, Any, Tuple
from loguru import logger
class BackfillValidator:
"""回填验证器"""
def __init__(self):
"""初始化验证器"""
pass
def validate_backfill(self, original_items: List[Dict[str, Any]],
backfilled_html: str,
translation_map: Dict[str, str]) -> Dict[str, Any]:
"""
验证回填的准确性
Args:
original_items: 原始提取的元素列表
backfilled_html: 回填后的 HTML
translation_map: {path/xpath: translation} 映射
Returns:
验证结果字典
"""
soup = BeautifulSoup(backfilled_html, 'html.parser')
results = {
'total': len(translation_map),
'success': 0,
'failed': 0,
'errors': []
}
for path, expected_translation in translation_map.items():
# 尝试在回填后的 HTML 中查找翻译
found = self._find_translation_in_html(soup, expected_translation)
if found:
results['success'] += 1
else:
results['failed'] += 1
results['errors'].append({
'path': path,
'expected': expected_translation,
'reason': '未在回填后的 HTML 中找到翻译'
})
results['accuracy'] = results['success'] / results['total'] if results['total'] > 0 else 0
return results
def validate_position_accuracy(self, original_html: str,
extraction_items: List[Dict[str, Any]],
extractor) -> Tuple[int, int]:
"""
验证位置定位的准确性
测试方法:
1. 从原始 HTML 提取元素
2. 为每个元素生成路径
3. 使用路径重新定位元素
4. 对比定位到的元素是否与原始元素一致
Args:
original_html: 原始 HTML
extraction_items: 提取的元素列表
extractor: 提取器实例(需要有 find_by_path 或类似方法)
Returns:
(成功数, 失败数)
"""
soup = BeautifulSoup(original_html, 'html.parser')
success = 0
failed = 0
for item in extraction_items:
path = item.get('path') or item.get('xpath')
if not path:
continue
original_text = item['text']
# 尝试通过路径重新定位
try:
if hasattr(extractor, 'path_utils'):
# BS4 提取器
found_element = extractor.path_utils.find_by_path(soup, path)
if found_element:
found_text = found_element.get_text().strip()
else:
found_text = None
else:
# lxml 提取器
from lxml import html as lxml_html
tree = lxml_html.fromstring(original_html)
elements = tree.xpath(path)
if elements:
found_text = elements[0].text_content().strip()
else:
found_text = None
# 对比文本
if found_text and self._texts_match(original_text, found_text):
success += 1
else:
failed += 1
logger.debug(f"位置验证失败: {path}")
except Exception as e:
failed += 1
logger.error(f"位置验证错误 {path}: {e}")
return success, failed
def simulate_translation_backfill(self, original_html: str,
extraction_items: List[Dict[str, Any]],
extractor) -> Tuple[str, Dict[str, Any]]:
"""
模拟翻译回填过程
为每个提取的元素生成模拟翻译,然后回填,验证是否能正确回填
Args:
original_html: 原始 HTML
extraction_items: 提取的元素列表
extractor: 提取器实例
Returns:
(回填后的 HTML, 验证结果)
"""
# 生成模拟翻译
translation_map = {}
for i, item in enumerate(extraction_items):
path = item.get('path') or item.get('xpath')
if path:
# 使用简单的标记作为"翻译"
translation_map[path] = f"TRANSLATED_{i:04d}"
# 执行回填
backfilled_html = extractor.backfill(original_html, translation_map)
# 验证回填结果
validation_results = self.validate_backfill(
extraction_items,
backfilled_html,
translation_map
)
return backfilled_html, validation_results
def _find_translation_in_html(self, soup: BeautifulSoup, translation: str) -> bool:
"""在 HTML 中查找翻译文本"""
# 简单的文本搜索
html_text = soup.get_text()
return translation in html_text
def _texts_match(self, text1: str, text2: str) -> bool:
"""
判断两段文本是否匹配
允许一定的空白差异
"""
import re
# 标准化空白
normalized1 = re.sub(r'\s+', ' ', text1.strip())
normalized2 = re.sub(r'\s+', ' ', text2.strip())
return normalized1 == normalized2
def generate_report(self, results: Dict[str, Any]) -> str:
"""
生成回填验证报告
Args:
results: 验证结果
Returns:
Markdown 格式的报告
"""
report = ["# 回填准确性验证报告\n"]
# 总体统计
report.append("## 总体统计\n")
report.append(f"- 总计: {results.get('total', 0)} 个元素")
report.append(f"- 成功: {results.get('success', 0)} 个")
report.append(f"- 失败: {results.get('failed', 0)} 个")
report.append(f"- 准确率: {results.get('accuracy', 0):.2%}\n")
# 错误详情
errors = results.get('errors', [])
if errors:
report.append("## 错误详情\n")
for i, error in enumerate(errors[:10], 1): # 只显示前10个
report.append(f"### 错误 {i}\n")
report.append(f"- 路径: `{error.get('path', 'N/A')}`")
report.append(f"- 预期翻译: {error.get('expected', 'N/A')[:100]}")
report.append(f"- 原因: {error.get('reason', 'N/A')}\n")
if len(errors) > 10:
report.append(f"\n... 还有 {len(errors) - 10} 个错误\n")
return "\n".join(report)

Some files were not shown because too many files have changed in this diff Show More