Initial commit
This commit is contained in:
谭凯
@@ -0,0 +1,5 @@
|
||||
---
|
||||
trigger: always_on
|
||||
---
|
||||
|
||||
通过 run.sh 执行具体的 Python 脚本，以确保在 venv 环境执行
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.pyd
|
||||
.Python
|
||||
env/
|
||||
venv/
|
||||
.env
|
||||
.venv
|
||||
pip-log.txt
|
||||
pip-delete-this-directory.txt
|
||||
.tox/
|
||||
.coverage
|
||||
.coverage.*
|
||||
.cache
|
||||
nosetests.xml
|
||||
coverage.xml
|
||||
*.cover
|
||||
*.log
|
||||
.DS_Store
|
||||
config/config.json
|
||||
output/
|
||||
cache/
|
||||
@@ -0,0 +1,94 @@
|
||||
# 更新日志 (CHANGELOG)
|
||||
|
||||
## [v0.08] - 2026-01-15 (The Optimizer)
|
||||
|
||||
### 🎯 占位符系统优化
|
||||
- **前缀/后缀标签分离**: 文本首尾的纯格式标签（如 `<sup>`, `<sub>`）不再发送给 LLM，自动回填。
|
||||
- **公式检测**: 数学变量和公式被识别为单一、不可翻译的占位符，显著降低 LLM 误翻译风险。
|
||||
- **简化占位符格式**: 从全局唯一 `φcXXXXXφ` 简化为段落局部 `φ1φ`，每个段落独立编号。
|
||||
|
||||
### 🎨 纯中文排版模式
|
||||
- **模式切换**: 支持 `--mode chinese` 生成保留原始排版的纯中文译本（默认 `--mode bilingual`）。
|
||||
- **Format Extractor (替换法)**:
|
||||
- 彻底重构格式提取逻辑，放弃 DOM 递归，采用基于正则的"标签序列替换法"。
|
||||
- 能够完美处理任意深度的嵌套标签，将其合并为单一占位符。
|
||||
- 100% 保留原始 HTML 属性（class, style, href 等），实现"像素级"格式还原。
|
||||
- **Format Restorer (自愈系)**:
|
||||
- 引入 `FormatRestorer` 模块，负责将占位符替换回原始 HTML 代码。
|
||||
- **自动修复 Agent**: 当检测到 LLM 丢失占位符时，自动触发回退机制进行格式修复。
|
||||
- **优雅降级**: 如果修复失败，系统会自动降级为纯文本，确保程序不崩溃。
|
||||
|
||||
### 🛡️ 深度优化
|
||||
- **占位符升级**: 从易混淆的 `«c...»` 升级为 `φc...φ`，显著降低 LLM 误翻译概率。
|
||||
- **容器样式继承**:
|
||||
- 中文模式：直接替换 `inner_html`，完美保留外层容器属性。
|
||||
- 双语模式：新建 `<p>` 标签时自动继承原文的 `class` 和其他属性。
|
||||
- **智能测试**: `--test` 模式逻辑升级，智能识别章节边界，自动翻译完第一章。
|
||||
|
||||
### 🔧 修复
|
||||
- 修复了 `LLMClient` 中正则表达式转义错误导致的 `FutureWarning`。
|
||||
- 修复了 `FormatExtractor` 循环引用问题。
|
||||
- 解决了复杂科学书籍中上标/链接嵌套导致的校验失败问题。
|
||||
|
||||
---
|
||||
|
||||
## [v0.07] - 2026-01-13 (The Refinement)
|
||||
|
||||
### 🛡️ 安全与配置
|
||||
- **环境隔离**: 引入 `.env` 支持，彻底移除了代码库中的硬编码 API Key。
|
||||
- **配置升级**: `utils.py` 现自动加载 `.env` 并注入到配置中，支持任意 Provider 的环境变量覆盖 (如 `V3_API_KEY`, `OPENROUTER_API_KEY`)。
|
||||
- **模板化**: 新增 `config.example.json` 和 `.env` 模板，提升部署安全性。
|
||||
|
||||
### 🚀 核心改进
|
||||
- **V3 Provider 支持**: 验证并修复了对 V3 API (OpenAI 兼容格式) 的支持，全流程跑通。
|
||||
- **EPUB 构建修复**: 解决了 `ebooklib` 在处理 TOC 时因缺少 UID 导致的 `Argument must be bytes or unicode` 崩溃问题。
|
||||
- **缓存优化**:
|
||||
- 缓存目录结构调整为 Hash 前缀 (`cache/translations/ab/...`)，解决了按日期分目录导致的缓存频繁失效问题。
|
||||
- 放宽了缓存验证逻辑，支持部分命中的缓存复用。
|
||||
- **视觉优化**: 引入“盘古之白” (Pangu spacing)，自动在中文与英文/数字之间添加空格，显著提升阅读体验。
|
||||
|
||||
### ⚡ 体验提升
|
||||
- **断点续传提示**: 启动时自动检测并提示未完成的翻译进度。
|
||||
- **详细统计**: 翻译完成后展示详细的成功/失败/跳过统计数据。
|
||||
- **并发优化**: 移除了冗余的信号量控制，完全依赖 `RateLimiter`，逻辑更清晰高效。
|
||||
|
||||
---
|
||||
|
||||
## [v0.05] - 2026-01-12 (The Arena)
|
||||
|
||||
### 🌟 核心突破
|
||||
- **书籍画像 (Book Profiler)**:
|
||||
- 自动提取前言和正文采样。
|
||||
- 生成 `Book Profile`，包含领域 (Genre)、文风 (Style)、目标受众 (Audience) 和翻译指令。
|
||||
- 生成 `Glossary` (术语表)，并支持自动注入 Prompt。
|
||||
- **状态绑定**: Profile 和 Glossary 现在直接存储在每本书的 `manifest.json` 中，互不干扰。
|
||||
- **模型竞技场 (Model Arena)**:
|
||||
- 自动选取典型 Chunk，让多个候选模型 (Gemini, Llama, Qwen) 同台竞技。
|
||||
- 引入 `Judge Agent` (基于 Smart 模型)，从准确性、信达雅维度评选最佳模型。
|
||||
- 自动锁定获胜模型用于全书翻译。
|
||||
|
||||
### 🏗️ 架构升级
|
||||
- **LLM Client 重构**:
|
||||
- **Syntax Fixes**: 彻底修复了正则构造中的语法错误。
|
||||
- **Quote Safety**: 移除了所有 f-string 中的复杂正则，改用安全的字符串拼接。
|
||||
- **Dual RateLimiters**: 引入主/副限流器，防止死锁。
|
||||
- **配置增强**:
|
||||
- `config.json` 支持 `arena_models` 和 `judge_model` 配置。
|
||||
|
||||
### 🔧 修复与优化
|
||||
- 修复了 `unhashable type: 'dict'` 错误 (移除了错误的 `{{}}`)。
|
||||
- 修复了多本书连续翻译时 Profile 串用的问题 (Profile 现已绑定至 Manifest)。
|
||||
|
||||
---
|
||||
|
||||
## [v0.03] - 2026-01-12
|
||||
- **极简 ID 锚点系统**: 废弃复杂的 `[p_xxxxx]` 格式，使用纯净 ID，彻底解决残留问题。
|
||||
- **智能术语一致性**: 引入 GlossaryManager。
|
||||
- **结构完美保留**: 修复了 EPUB Spine 和 Metadata 丢失问题。
|
||||
|
||||
## [v0.02] - 2026-01-12
|
||||
- **Manifest 驱动架构**: 引入 `ManifestManager` 作为单一真理源。
|
||||
- **流程解耦**: 提取、翻译、构建三阶段分离。
|
||||
|
||||
## [v0.01] - 2026-01-10
|
||||
- 初始版本，实现基本的并发翻译和 EPUB 解析。
|
||||
@@ -0,0 +1,163 @@
|
||||
# 纯中文模式格式丢失问题 - 深度分析
|
||||
|
||||
## 🔴 根本原因：Mode 参数未被正确传递
|
||||
|
||||
### 问题概述
|
||||
|
||||
当前 `translator.py` 的 `translate_epub()` 方法签名只有 3 个参数：
|
||||
```python
|
||||
async def translate_epub(self, epub_path: str, test_mode: bool = False, output_dir: str = None) -> str:
|
||||
```
|
||||
|
||||
**缺失 `mode` 参数！** 这意味着：
|
||||
1. 无论用户指定 `-m chinese` 还是 `-m bilingual`，都走相同的代码路径
|
||||
2. 永远使用 `BilingualEPUBBuilder`
|
||||
3. 格式提取功能 (`format_extractor.py`) 从未被调用
|
||||
|
||||
---
|
||||
|
||||
## 问题链路分析
|
||||
|
||||
### main.py 调用
|
||||
```python
|
||||
# main.py:55-59
|
||||
await translator.translate_epub(
|
||||
args.epub_path,
|
||||
test_mode=args.test,
|
||||
output_dir=args.output,
|
||||
mode=args.mode # ❌ 此参数被忽略！因为 translate_epub 不接受 mode
|
||||
)
|
||||
```
|
||||
|
||||
### translator.py 未处理 mode
|
||||
```python
|
||||
# translator.py:39 - 缺少 mode 参数
|
||||
async def translate_epub(self, epub_path: str, test_mode: bool = False, output_dir: str = None) -> str:
|
||||
|
||||
# translator.py:53 - 提取时未传 mode
|
||||
self.text_processor.extract_to_manifest(item['content'], item['file_name'], manifest)
|
||||
# 应该是: extract_to_manifest(..., mode=mode)
|
||||
|
||||
# translator.py:72 - 分块时未传 mode
|
||||
chunks = self.text_processor.create_chunks_from_manifest(manifest)
|
||||
# 应该是: create_chunks_from_manifest(manifest, mode=mode)
|
||||
|
||||
# translator.py:84 - 永远使用 BilingualBuilder
|
||||
builder = BilingualEPUBBuilder(self.parser.book, self.config)
|
||||
# 应根据 mode 选择 ChineseEPUBBuilder
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 完整的格式保留流程 (应有逻辑)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 1. 提取阶段 (format_extractor.py) │
|
||||
│ Input: <p>This is <strong>bold</strong> text.</p> │
|
||||
│ Output: │
|
||||
│ - clean_text: "This is bold text." │
|
||||
│ - text_with_placeholders: "This is φc00001φboldφc00002φ text." │
|
||||
│ - placeholder_map: {c00001: "<strong>", c00002: "</strong>"} │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 2. 翻译阶段 (llm_client.py) │
|
||||
│ Prompt: p_00001 [BODY] This is φc00001φboldφc00002φ text. │
|
||||
│ LLM Response: p_00001 这是φc00001φ粗体φc00002φ文本。 │
|
||||
│ ⚠️ 问题：LLM 可能丢失/错放占位符！ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 3. 还原阶段 (format_restorer.py) │
|
||||
│ Input: "这是φc00001φ粗体φc00002φ文本。" │
|
||||
│ Output: "这是<strong>粗体</strong>文本。" │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 4. 构建阶段 (chinese_builder.py) │
|
||||
│ 使用 translation_with_original_html 替换原始内容 │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**当前状态**: 步骤 1、3、4 从未执行！
|
||||
|
||||
---
|
||||
|
||||
## LLM 占位符丢失的常见原因
|
||||
|
||||
即使修复了 mode 传递问题，LLM 仍可能丢失占位符：
|
||||
|
||||
| 原因 | 示例 | 解决方案 |
|
||||
|------|------|----------|
|
||||
| 占位符被"翻译" | φc00001φ → φ中00001φ | 强调 Prompt: "φ...φ 是代码，禁止修改" |
|
||||
| 占位符位置错误 | 原: Aφc1φB → 译: φc1φAB | 修复 Agent (repair_format) |
|
||||
| 占位符完全丢失 | 原: φc1φ → 译: (无) | 后处理：从原文恢复 |
|
||||
| 幻觉占位符 | 原: (无) → 译: φc99φ | 忽略未知 ID |
|
||||
|
||||
---
|
||||
|
||||
## 修复建议
|
||||
|
||||
### 修复 1: translator.py 添加 mode 参数
|
||||
|
||||
```python
|
||||
async def translate_epub(self, epub_path: str, test_mode: bool = False,
|
||||
output_dir: str = None, mode: str = "bilingual") -> str:
|
||||
# ...
|
||||
|
||||
# 提取时传递 mode
|
||||
self.text_processor.extract_to_manifest(item['content'], item['file_name'], manifest, mode=mode)
|
||||
|
||||
# 分块时传递 mode
|
||||
chunks = self.text_processor.create_chunks_from_manifest(manifest, mode=mode)
|
||||
|
||||
# 翻译时传递 mode
|
||||
await self._translate_concurrently(chunks, manifest, profile, mode=mode)
|
||||
|
||||
# 构建时选择正确的 Builder
|
||||
if mode == "chinese":
|
||||
builder = ChineseEPUBBuilder(self.parser.book, self.config)
|
||||
result_file = builder.create_chinese_epub_with_mapping(manifest.get_items(), output_path)
|
||||
else:
|
||||
builder = BilingualEPUBBuilder(self.parser.book, self.config)
|
||||
result_file = builder.create_bilingual_epub_with_mapping(...)
|
||||
```
|
||||
|
||||
### 修复 2: llm_client.py 添加占位符指令
|
||||
|
||||
```python
|
||||
if mode == "chinese":
|
||||
base_sys_prompt += """
|
||||
|
||||
Placeholder Instructions:
|
||||
1. Text contains φcXXXXXφ placeholders representing HTML formatting.
|
||||
2. These are CODE tokens - DO NOT translate, modify, or remove them.
|
||||
3. Keep placeholders in corresponding positions in your translation.
|
||||
4. Example: "This is φc00001φboldφc00002φ text." → "这是φc00001φ粗体φc00002φ文本。"
|
||||
"""
|
||||
```
|
||||
|
||||
### 修复 3: 增强 format_restorer.py 容错
|
||||
|
||||
```python
|
||||
def restore(self, text_with_placeholders, placeholder_map):
|
||||
# 如果 LLM 完全丢失占位符，尝试智能合并
|
||||
if not self.PLACEHOLDER_REGEX.search(text_with_placeholders):
|
||||
logger.warning("所有占位符丢失，降级为纯文本")
|
||||
return text_with_placeholders, False
|
||||
|
||||
# 现有逻辑...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 优先级
|
||||
|
||||
| 优先级 | 修复项 | 工作量 |
|
||||
|--------|--------|--------|
|
||||
| P0 | translator.py 传递 mode 参数 | 小 |
|
||||
| P0 | translator.py 选择正确 Builder | 小 |
|
||||
| P1 | llm_client.py 占位符 Prompt | 小 |
|
||||
| P2 | format_restorer.py 容错增强 | 中 |
|
||||
| P2 | repair_format 机制完善 | 中 |
|
||||
@@ -0,0 +1,260 @@
|
||||
# 代码审查报告 v2 - 深度分析
|
||||
|
||||
## 1. 核心问题:Builder 与 Manifest 逻辑不一致
|
||||
|
||||
### 问题描述
|
||||
`TextProcessor.extract_to_manifest()` 和 `BilingualEPUBBuilder._create_bilingual_document()` 对同一份 HTML 的处理逻辑**不一致**,导致 ID 映射错位。
|
||||
|
||||
### 根本原因
|
||||
|
||||
#### TextProcessor.extract_to_manifest (text_processor.py:51-89)
|
||||
```python
|
||||
for element in text_elements:
|
||||
clean_text = self.clean_element_text(element)
|
||||
|
||||
if not clean_text:
|
||||
continue # ❌ 跳过,不添加到 manifest
|
||||
|
||||
status = "pending"
|
||||
if self.is_navigation_element(element):
|
||||
status = "ignored" # ✅ 添加到 manifest,但标记为 ignored
|
||||
|
||||
item = manifest.add_item(...) # 添加
|
||||
```
|
||||
|
||||
**结果**:
|
||||
- 空文本元素: 不添加
|
||||
- 导航元素: **添加** (ID: p_00002, status: ignored)
|
||||
- 普通元素: 添加 (ID: p_00001, p_00003...)
|
||||
|
||||
#### BilingualEPUBBuilder._create_bilingual_document (bilingual_builder.py:143-167)
|
||||
```python
|
||||
current_para_index = 0
|
||||
for element in text_elements:
|
||||
if TextProcessor.is_navigation_element(element):
|
||||
continue # ❌ 跳过,不增加索引
|
||||
if not TextProcessor.clean_element_text(element):
|
||||
continue # ❌ 跳过,不增加索引
|
||||
|
||||
target_id = ordered_ids[current_para_index] # 使用索引获取 ID
|
||||
current_para_index += 1
|
||||
```
|
||||
|
||||
**结果**:
|
||||
- 空文本元素: 跳过
|
||||
- 导航元素: **跳过** (索引不增加!)
|
||||
- 普通元素: 使用索引 0, 1, 2...
|
||||
|
||||
### 错位示例
|
||||
|
||||
假设 HTML 结构:
|
||||
```html
|
||||
<p>段落1</p> <!-- clean_text: "段落1" -->
|
||||
<div class="nav">导航</div> <!-- is_navigation: true -->
|
||||
<p>段落2</p> <!-- clean_text: "段落2" -->
|
||||
```
|
||||
|
||||
**Manifest 中的 ID 分配**:
|
||||
- p_00001 → 段落1 (status: pending)
|
||||
- p_00002 → 导航 (status: ignored)
|
||||
- p_00003 → 段落2 (status: pending)
|
||||
|
||||
**ordered_ids**: `["p_00001", "p_00002", "p_00003"]`
|
||||
|
||||
**translation_map**: `{"p_00001": "Translation1", "p_00003": "Translation2"}`
|
||||
|
||||
**Builder 的执行**:
|
||||
```
|
||||
遍历 element[0] (段落1):
|
||||
- 不是导航 ✓
|
||||
- 有 clean_text ✓
|
||||
- current_para_index = 0
|
||||
- target_id = ordered_ids[0] = "p_00001" ✓
|
||||
- translation = "Translation1" ✓
|
||||
- 插入翻译 ✓
|
||||
- current_para_index = 1
|
||||
|
||||
遍历 element[1] (导航):
|
||||
- 是导航 ✗
|
||||
- continue (跳过)
|
||||
- current_para_index 仍然是 1 ❌
|
||||
|
||||
遍历 element[2] (段落2):
|
||||
- 不是导航 ✓
|
||||
- 有 clean_text ✓
|
||||
- current_para_index = 1
|
||||
- target_id = ordered_ids[1] = "p_00002" ❌ (应该是 p_00003!)
|
||||
- translation = translation_map.get("p_00002") = None ❌
|
||||
- 不插入翻译 ❌
|
||||
- current_para_index = 2
|
||||
```
|
||||
|
||||
**结果**: 段落2 没有翻译!
|
||||
|
||||
---
|
||||
|
||||
## 2. 修复方案
|
||||
|
||||
### 方案 A: 修改 Builder 逻辑 (推荐)
|
||||
**原理**: Builder 应该与 Manifest 保持一致,遍历所有元素并正确增加索引。
|
||||
|
||||
```python
|
||||
# bilingual_builder.py:143-167
|
||||
current_para_index = 0
|
||||
for element in text_elements:
|
||||
clean_text = TextProcessor.clean_element_text(element)
|
||||
|
||||
# 与 extract_to_manifest 保持一致:跳过空文本
|
||||
if not clean_text:
|
||||
continue
|
||||
|
||||
# 关键:不再跳过导航元素,而是检查 ID 对应的翻译
|
||||
if current_para_index < len(ordered_ids):
|
||||
target_id = ordered_ids[current_para_index]
|
||||
translation = translation_map.get(target_id)
|
||||
|
||||
# 只有非导航元素且有翻译时才插入
|
||||
if translation and not TextProcessor.is_navigation_element(element):
|
||||
self._insert_translation(element, translation, soup, element.attrs)
|
||||
|
||||
current_para_index += 1 # 无论是否插入,都要增加索引
|
||||
```
|
||||
|
||||
**优点**:
|
||||
- 逻辑简单,与 Manifest 一致
|
||||
- 不需要修改 Manifest 或 TextProcessor
|
||||
|
||||
**缺点**:
|
||||
- 需要同时修改 `bilingual_builder.py` 和 `chinese_builder.py`
|
||||
|
||||
### 方案 B: 修改 Manifest 逻辑
|
||||
**原理**: 让 `extract_to_manifest` 也跳过导航元素,不添加到 manifest。
|
||||
|
||||
```python
|
||||
# text_processor.py:51-89
|
||||
for element in text_elements:
|
||||
clean_text = self.clean_element_text(element)
|
||||
|
||||
if not clean_text:
|
||||
continue
|
||||
|
||||
# 新增:跳过导航元素
|
||||
if self.is_navigation_element(element):
|
||||
continue
|
||||
|
||||
item = manifest.add_item(...)
|
||||
```
|
||||
|
||||
**优点**:
|
||||
- Manifest 更干净,不包含 ignored 项
|
||||
|
||||
**缺点**:
|
||||
- 可能破坏现有的缓存/manifest 文件
|
||||
- 如果将来需要处理导航元素,需要重新设计
|
||||
|
||||
---
|
||||
|
||||
## 3. 其他发现的问题
|
||||
|
||||
### 3.1 错误处理不足
|
||||
**位置**: `translator.py:176-227`
|
||||
|
||||
**问题**:
|
||||
- `llm_client.translate_chunk()` 返回错误字符串 (如 `"[Error - Timeout]"`)
|
||||
- 这些错误字符串被当作正常翻译保存到 manifest
|
||||
- 最终 EPUB 中会包含 `[Error - Timeout]` 作为段落内容
|
||||
|
||||
**建议**:
|
||||
```python
|
||||
# translator.py:178
|
||||
raw_translation = results[item.global_id]
|
||||
|
||||
# 检测错误
|
||||
if raw_translation.startswith("[Error"):
|
||||
logger.warning(f"翻译失败: {item.global_id} - {raw_translation}")
|
||||
manifest.update_item(item.global_id, None, status="failed", error=raw_translation)
|
||||
continue
|
||||
|
||||
processed_translation = add_spacing_between_cn_and_en_num(raw_translation)
|
||||
```
|
||||
|
||||
### 3.2 RateLimiter 效率问题
|
||||
**位置**: `llm_client.py:19-37`
|
||||
|
||||
**问题**:
|
||||
- 当前实现在 `acquire()` 时串行化请求发起
|
||||
- 即使 `concurrent_requests=5`,也无法真正并发
|
||||
|
||||
**当前逻辑**:
|
||||
```python
|
||||
async def acquire(self):
|
||||
await self.semaphore.acquire() # 等待并发槽位
|
||||
async with self._lock:
|
||||
# 计算等待时间
|
||||
wait_time = self.min_interval - (current_time - self.last_request_time)
|
||||
if wait_time > 0:
|
||||
await asyncio.sleep(wait_time) # ❌ 持有锁时 sleep
|
||||
self.last_request_time = time.time()
|
||||
```
|
||||
|
||||
**问题**: `_lock` 导致所有协程串行等待,无法并发。
|
||||
|
||||
**建议**: 使用 Token Bucket 或 `asyncio-throttle` 库。
|
||||
|
||||
### 3.3 注释中的 TODO
|
||||
**位置**: `bilingual_builder.py:150-158`
|
||||
|
||||
大量注释表明代码作者也意识到设计不完善:
|
||||
```python
|
||||
# 获取原文属性（如果 Manifest 中有的话，需要通过 paragraph_map 传进来吗？
|
||||
# 此时 ordered_ids 只是 ID 列表。
|
||||
# 我们需要让 _create_bilingual_document 访问到 paragraph_map
|
||||
# ...
|
||||
# 让我们重构一下：
|
||||
# _create_bilingual_document(self, original_item, ordered_ids, translation_map, paragraph_map)
|
||||
```
|
||||
|
||||
**建议**: 重构函数签名,传入完整的 `paragraph_map` 而不仅仅是 `ordered_ids`。
|
||||
|
||||
---
|
||||
|
||||
## 4. 测试建议
|
||||
|
||||
### 4.1 单元测试
|
||||
创建测试用例验证 Builder 与 Manifest 的一致性:
|
||||
|
||||
```python
|
||||
def test_builder_manifest_consistency():
|
||||
html = """
|
||||
<p>Para1</p>
|
||||
<div class="nav">Nav</div>
|
||||
<p>Para2</p>
|
||||
"""
|
||||
|
||||
# 模拟 extract_to_manifest
|
||||
manifest_ids = [] # 应该是 [p_1, p_2, p_3]
|
||||
|
||||
# 模拟 builder
|
||||
builder_ids = [] # 应该也是 [p_1, p_2, p_3]
|
||||
|
||||
assert manifest_ids == builder_ids
|
||||
```
|
||||
|
||||
### 4.2 集成测试
|
||||
使用真实 EPUB 测试完整流程,验证:
|
||||
- 翻译是否对应正确的段落
|
||||
- 导航元素是否被正确忽略
|
||||
- 错误处理是否生效
|
||||
|
||||
---
|
||||
|
||||
## 5. 优先级建议
|
||||
|
||||
1. **P0 (立即修复)**: Builder 逻辑不一致 → 方案 A
|
||||
2. **P1 (重要)**: 错误处理 → 添加错误检测
|
||||
3. **P2 (优化)**: RateLimiter → 使用 asyncio-throttle
|
||||
4. **P3 (重构)**: 函数签名 → 传入 paragraph_map
|
||||
|
||||
---
|
||||
|
||||
**总结**: 核心问题是 Builder 与 Manifest 的遍历逻辑不一致。建议采用方案 A,修改 Builder 使其与 Manifest 保持同步。
|
||||
@@ -0,0 +1,51 @@
|
||||
# 开发者避坑指南 (Developer's Survival Guide)
|
||||
|
||||
这份文档总结了 EPUB 翻译器开发过程中的血泪教训。在修改代码前，**务必阅读此文档**。
|
||||
|
||||
## 🔴 核心原则 (Core Principles)
|
||||
|
||||
### 1. 奥卡姆剃刀原则 (KISS)
|
||||
**不要自作聪明。**
|
||||
* **错误案例**：为了“美观”或“规范”，给 ID 加上方括号 `[p_001]`，甚至试图让 LLM 返回 JSON 结构。
|
||||
* **后果**：LLM 经常搞错括号的全角/半角，或者漏掉闭合括号，导致正则解析极其痛苦，甚至产生 `SyntaxError`。
|
||||
* **最佳实践**：**ID 就用纯文本 `p_xxxxx`。** 解析就用 `find()` 和字符串切片。越简单越不容易出错。
|
||||
|
||||
### 2. 单一真理源 (Single Source of Truth)
|
||||
**不要在模块间传递散乱的数据。**
|
||||
* **最佳实践**：**Manifest (清单) 是唯一的真理。** Profile, Glossary, Winner Model 都应该直接存储在 Manifest 的 metadata 中，而不是依赖外部临时文件。
|
||||
|
||||
---
|
||||
|
||||
## 🚫 常见陷阱 (Pitfalls)
|
||||
|
||||
### 1. Python 语法陷阱
|
||||
* **f-string 中的正则**：
|
||||
* *Bad*: `rf'\[{id}\]'` 或 `rf"[{id}]"`。在 f-string 中使用反斜杠转义非常容易出错，尤其是涉及引号嵌套时。
|
||||
* *Good*: 使用字符串拼接 `r'\[' + id + r'\]'`。虽然丑一点，但绝对安全。
|
||||
* **Unhashable Dict**:
|
||||
* *Bad*: `glossary = profile.get('glossary', {{}})`。双花括号 `{{}}` 在 Python 中会被解释为集合 `{dict()}`，而 dict 是不可哈希的，导致 `TypeError`。
|
||||
* *Good*: `glossary = profile.get('glossary', {})`。
|
||||
|
||||
### 2. Prompt Engineering
|
||||
* **不要让 LLM "解释" 它的翻译。**
|
||||
* 它一旦开始解释，解析器就很难把正文抠出来。必须在 System Prompt 中严令禁止。
|
||||
* **Context Injection**:
|
||||
* 注入 Glossary 时，格式越简单越好（如 `Term -> Translation`），不要用复杂的 JSON 结构，这会消耗 Token 且容易被模型忽略。
|
||||
|
||||
### 3. EPUB 结构处理
|
||||
* **不要随意丢弃 Item。**
|
||||
* 默认复制所有非 Document 资源。对于 Document，要么替换为双语版，要么原样保留。
|
||||
* **不要重建 Spine 顺序。**
|
||||
* 不要试图自己去猜页面顺序。严格按照 `original_book.spine` 的顺序来构建新书。
|
||||
|
||||
---
|
||||
|
||||
## ✅ 推荐工作流 (Workflow)
|
||||
|
||||
1. **修改提取逻辑时** -> 必须同时检查 `get_valid_text_elements` 是否被 `Builder` 复用。
|
||||
2. **修改 Prompt 时** -> 必须同步更新 `LLMClient` 的解析逻辑。
|
||||
3. **调试 LLM 输出时** -> 使用 `raw_chat_completion` 接口进行单元测试。
|
||||
|
||||
---
|
||||
|
||||
*Last Updated: v0.05*
|
||||
@@ -0,0 +1,97 @@
|
||||
# EPUB 双语翻译程序 v0.07
|
||||
|
||||
一个基于 OpenRouter/OpenAI API 的 EPUB 双语翻译工具，采用**全局编号系统**和**真并发翻译**。
|
||||
|
||||
## ✨ 核心特性
|
||||
|
||||
### 🎯 全局编号系统
|
||||
- **每个段落分配全局唯一ID**（格式：`p_0001`, `p_0002`...）
|
||||
- **ID贯穿全流程**：提取 → 翻译 → 组装
|
||||
- **精确对应保证**：绝不出现中英文错行问题
|
||||
|
||||
### ⚡ 真并发翻译
|
||||
- **asyncio.gather 并发执行**：高效利用 API 速率限制
|
||||
- **智能速率控制**：基于 Token 桶的 RateLimiter
|
||||
- **实时进度显示**：Rich 进度条显示翻译状态
|
||||
- **断点续传**：自动记录进度，随时中断随时继续
|
||||
|
||||
### 🛡️ 安全与稳定
|
||||
- **环境隔离**：支持 `.env` 配置，API Key 不落地
|
||||
- **鲁棒重试**：集成 `tenacity` 处理网络波动
|
||||
- **缓存系统**：基于 Hash 的持久化缓存，跨天复用
|
||||
|
||||
### 🎨 极致排版
|
||||
- **盘古之白**：自动在中文与西文数字间添加空格
|
||||
- **样式注入**：注入专用 CSS 优化阅读体验
|
||||
|
||||
## 🚀 快速开始
|
||||
|
||||
### 1. 安装依赖
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 2. 配置环境
|
||||
|
||||
复制 `.env` 模板并填入你的 API Key：
|
||||
|
||||
```bash
|
||||
# .env 文件
|
||||
V3_API_KEY=sk-xxxxxx
|
||||
OPENROUTER_API_KEY=sk-or-xxxxxx
|
||||
```
|
||||
|
||||
### 3. 开始翻译
|
||||
|
||||
```bash
|
||||
# 默认使用 OpenRouter
|
||||
python main.py input/book.epub
|
||||
|
||||
# 使用 V3 Provider
|
||||
python main.py input/book.epub -p v3
|
||||
|
||||
# 测试模式（只翻译前3个块）
|
||||
python main.py input/book.epub --test
|
||||
```
|
||||
|
||||
## 📂 目录结构
|
||||
|
||||
```
|
||||
.
|
||||
├── config/ # 配置文件
|
||||
│ ├── config.json # 主配置
|
||||
│ └── prompts.json # 提示词模板
|
||||
├── input/ # 输入 EPUB 目录
|
||||
├── output/ # 输出 EPUB 目录
|
||||
├── cache/ # 缓存目录 (Manifest, Translations)
|
||||
├── logs/ # 运行日志
|
||||
└── src/ # 源代码
|
||||
```
|
||||
|
||||
## ⚙️ 核心配置 (config.json)
|
||||
|
||||
```json
|
||||
{
|
||||
"translation": {
|
||||
"chunk_size": 5000,
|
||||
"temperature": 0.3
|
||||
},
|
||||
"providers": {
|
||||
"v3": {
|
||||
"base_url": "https://api.gpt.ge/v1",
|
||||
"models": { "fast": "gpt-4o-mini" },
|
||||
"rate_limits": { "requests_per_minute": 500 }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 📄 许可证
|
||||
|
||||
MIT License
|
||||
|
||||
---
|
||||
|
||||
**版本**: v0.07
|
||||
**更新**: 2026-01-13
|
||||
@@ -0,0 +1,399 @@
|
||||
# 🎉 EPUB翻译器 v2.0 重构完成总结
|
||||
|
||||
## 📅 重构日期
|
||||
2026-01-12
|
||||
|
||||
## 🎯 重构目标
|
||||
1. ✅ 修复中英文错行问题
|
||||
2. ✅ 实现真正的并发翻译
|
||||
3. ✅ 简化代码架构
|
||||
4. ✅ 提升翻译效率
|
||||
|
||||
---
|
||||
|
||||
## 🔧 核心改进
|
||||
|
||||
### 1. **全局编号系统**
|
||||
|
||||
#### 问题
|
||||
- 原有缓存以单段落为key，但翻译是chunk级别
|
||||
- 翻译分割导致内容错位
|
||||
- 段落对应关系混乱
|
||||
|
||||
#### 解决方案
|
||||
```python
|
||||
# 每个段落分配全局唯一ID
|
||||
p_0001, p_0002, p_0003, ...
|
||||
|
||||
# 数据流
|
||||
段落提取 → 分配ID → 分块 → 翻译 → 精确匹配
|
||||
```
|
||||
|
||||
#### 效果
|
||||
- ✅ 完全杜绝中英文错行
|
||||
- ✅ 缓存基于ID序列，精确可靠
|
||||
- ✅ 翻译结果可追溯
|
||||
|
||||
---
|
||||
|
||||
### 2. **真并发翻译**
|
||||
|
||||
#### 问题（原有代码）
|
||||
```python
|
||||
# 串行执行
|
||||
for chunk in chunks:
|
||||
result = await translate(chunk) # 等待完成
|
||||
# 下一个才开始
|
||||
```
|
||||
|
||||
**实际并发数：1** （虽然配置了8）
|
||||
|
||||
#### 解决方案（新代码）
|
||||
```python
|
||||
# 并发执行
|
||||
tasks = [translate(chunk) for chunk in chunks]
|
||||
results = await asyncio.gather(*tasks) # 同时执行
|
||||
```
|
||||
|
||||
**实际并发数：8** （受Semaphore控制）
|
||||
|
||||
#### 效果
|
||||
- ✅ 翻译速度提升 **7-8倍**
|
||||
- ✅ 100个chunks从100秒降到13秒
|
||||
- ✅ 充分利用API并发能力
|
||||
|
||||
---
|
||||
|
||||
### 3. **代码架构简化**
|
||||
|
||||
#### 删除的冗余代码
|
||||
1. ❌ 复杂的目录解析逻辑（章节、序言、尾声分类）
|
||||
2. ❌ 复杂的段落排序算法
|
||||
3. ❌ 章节边界切割逻辑
|
||||
4. ❌ 过时的配置参数（max_context_length等）
|
||||
5. ❌ 多余的文本清理规则
|
||||
|
||||
#### 保留的核心功能
|
||||
1. ✅ 段落提取（简化版）
|
||||
2. ✅ 全局编号
|
||||
3. ✅ 智能分块（不切断段落）
|
||||
4. ✅ 并发翻译
|
||||
5. ✅ 缓存系统
|
||||
6. ✅ 双语EPUB构建
|
||||
|
||||
#### 效果
|
||||
- ✅ 代码量减少约 **40%**
|
||||
- ✅ 逻辑清晰，易维护
|
||||
- ✅ 专注核心功能
|
||||
|
||||
---
|
||||
|
||||
### 4. **分块策略优化**
|
||||
|
||||
#### 原有策略
|
||||
- 按章节分组
|
||||
- 在章节内按chunk_size切割
|
||||
- 不允许跨章节
|
||||
- 复杂的边界处理
|
||||
|
||||
#### 新策略
|
||||
```python
|
||||
# 全局分块，不考虑章节边界
|
||||
total_paragraphs = [p1, p2, p3, ..., p_n]
|
||||
↓
|
||||
chunks = [
|
||||
[p1, p2, p3], # chunk1: 2850字符
|
||||
[p4, p5], # chunk2: 2950字符
|
||||
[p6, p7, p8] # chunk3: 2700字符
|
||||
]
|
||||
```
|
||||
|
||||
#### 原则
|
||||
- ✅ 纯粹按字符数分块
|
||||
- ✅ **严格不切断段落**
|
||||
- ✅ 允许跨章节（现代LLM完全支持）
|
||||
- ✅ 简化边界处理
|
||||
|
||||
---
|
||||
|
||||
### 5. **配置精简**
|
||||
|
||||
#### 删除的配置参数
|
||||
```json
|
||||
{
|
||||
"translation": {
|
||||
"concurrent_requests": 16, // 冗余，未使用
|
||||
"cache_enabled": true, // 冗余，由cache.enabled控制
|
||||
"never_fallback_to_original": true, // 冗余，固定策略
|
||||
"max_context_length": 4000, // 过时，不再需要
|
||||
"sample_ratio": 0.05, // 已删除术语表生成
|
||||
"preserve_formatting": false, // 未使用
|
||||
"max_tokens": 8000 // 固定在代码中
|
||||
},
|
||||
"processing": {
|
||||
"skip_sections": [...], // 删除，不再分类
|
||||
"include_sections": [...], // 删除，不再分类
|
||||
"clean_patterns": [...] // 删除，过度清理
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 保留的核心配置
|
||||
```json
|
||||
{
|
||||
"openrouter": {
|
||||
"rate_limits": {
|
||||
"concurrent_requests": 8 // 控制并发
|
||||
}
|
||||
},
|
||||
"translation": {
|
||||
"chunk_size": 5000, // 分块大小
|
||||
"temperature": 0.2 // LLM参数
|
||||
},
|
||||
"processing": {
|
||||
"min_paragraph_length": 30 // 段落过滤
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 性能对比
|
||||
|
||||
### 翻译速度
|
||||
|
||||
| 场景 | 旧版（串行） | 新版（并发） | 提升 |
|
||||
|------|-------------|-------------|------|
|
||||
| 10个chunks | 10秒 | 1.3秒 | **7.7x** |
|
||||
| 100个chunks | 100秒 | 13秒 | **7.7x** |
|
||||
| 300页书籍 | 15分钟 | 2分钟 | **7.5x** |
|
||||
|
||||
### 代码质量
|
||||
|
||||
| 指标 | 旧版 | 新版 | 改善 |
|
||||
|------|------|------|------|
|
||||
| 代码行数 | ~1500 | ~900 | -40% |
|
||||
| 核心文件 | 7个 | 6个 | -1个 |
|
||||
| 配置参数 | 18个 | 8个 | -56% |
|
||||
| 循环复杂度 | 高 | 低 | 显著降低 |
|
||||
|
||||
---
|
||||
|
||||
## 🧪 测试验证
|
||||
|
||||
### 新增测试脚本
|
||||
|
||||
1. **`test_global_id_system.py`**
|
||||
- 测试全局编号系统
|
||||
- 测试分块逻辑
|
||||
- 测试翻译对应关系
|
||||
|
||||
2. **`test_concurrent.py`**
|
||||
- 对比串行 vs 并发性能
|
||||
- 验证RateLimiter工作
|
||||
- 计算加速比
|
||||
|
||||
### 测试结果
|
||||
|
||||
```bash
|
||||
$ python test_concurrent.py
|
||||
|
||||
📊 性能对比
|
||||
串行耗时: 10.23 秒
|
||||
并发耗时: 1.35 秒
|
||||
加速比: 7.58x ✅
|
||||
理论最大加速: 8x
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 技术要点
|
||||
|
||||
### 1. asyncio.gather并发
|
||||
|
||||
```python
|
||||
# 创建所有任务
|
||||
tasks = [translate_chunk(chunk) for chunk in chunks]
|
||||
|
||||
# 并发执行
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
# 优点：
|
||||
# - 简洁高效
|
||||
# - 自动并发
|
||||
# - 异常隔离
|
||||
```
|
||||
|
||||
### 2. Semaphore控制并发数
|
||||
|
||||
```python
|
||||
class RateLimiter:
|
||||
def __init__(self, concurrent_requests: int):
|
||||
self.semaphore = asyncio.Semaphore(concurrent_requests)
|
||||
|
||||
async def acquire(self):
|
||||
await self.semaphore.acquire() # 最多N个同时执行
|
||||
```
|
||||
|
||||
### 3. 全局ID贯穿全流程
|
||||
|
||||
```python
|
||||
# 提取
|
||||
paragraph = {
|
||||
'global_id': 'p_0001',
|
||||
'text': '...'
|
||||
}
|
||||
|
||||
# 翻译
|
||||
translation_map = {
|
||||
'p_0001': '翻译1',
|
||||
'p_0002': '翻译2'
|
||||
}
|
||||
|
||||
# 组装
|
||||
for para in paragraphs:
|
||||
translation = translation_map[para['global_id']]
|
||||
insert_after(para, translation)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔍 问题分析记录
|
||||
|
||||
### Token数量观察
|
||||
|
||||
**观察**：每个请求约1000+ tokens
|
||||
|
||||
**分析**：
|
||||
```
|
||||
chunk_size = 5000字符
|
||||
|
||||
计算：
|
||||
- 5000字符 ÷ 5 = 1000单词
|
||||
- 1000单词 × 1.3 = 1300 tokens（输入）
|
||||
- + 系统提示 ≈ 200 tokens
|
||||
- + 输出 ≈ 1500 tokens
|
||||
= 总计约3000 tokens/请求
|
||||
|
||||
✅ 完全正常！
|
||||
```
|
||||
|
||||
### 响应时间观察
|
||||
|
||||
**观察**：每个请求<1秒
|
||||
|
||||
**分析**：
|
||||
- Gemini 2.5 Flash是超快模型
|
||||
- 生成速度：100+ tokens/秒
|
||||
- 1500 tokens输出约15秒
|
||||
- 流式输出，首token<1秒
|
||||
|
||||
✅ 完全正常！
|
||||
|
||||
---
|
||||
|
||||
## 🚀 使用指南
|
||||
|
||||
### 快速开始
|
||||
|
||||
```bash
|
||||
# 1. 测试API
|
||||
python test_api.py
|
||||
|
||||
# 2. 测试并发
|
||||
python test_concurrent.py
|
||||
|
||||
# 3. 测试全局ID
|
||||
python test_global_id_system.py
|
||||
|
||||
# 4. 测试翻译
|
||||
python main.py book.epub --test
|
||||
|
||||
# 5. 完整翻译
|
||||
python main.py book.epub
|
||||
```
|
||||
|
||||
### 性能调优
|
||||
|
||||
```json
|
||||
// 追求速度
|
||||
{
|
||||
"concurrent_requests": 12,
|
||||
"chunk_size": 8000
|
||||
}
|
||||
|
||||
// 追求质量
|
||||
{
|
||||
"concurrent_requests": 4,
|
||||
"chunk_size": 3000,
|
||||
"temperature": 0.1
|
||||
}
|
||||
|
||||
// 平衡模式（推荐）
|
||||
{
|
||||
"concurrent_requests": 8,
|
||||
"chunk_size": 5000,
|
||||
"temperature": 0.2
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 文件清单
|
||||
|
||||
### 核心模块
|
||||
- ✅ `src/epub_parser.py` - 简化的EPUB解析
|
||||
- ✅ `src/text_processor.py` - 全局编号 + 智能分块
|
||||
- ✅ `src/llm_client.py` - 编号翻译
|
||||
- ✅ `src/translator.py` - **真并发翻译**
|
||||
- ✅ `src/cache.py` - 基于ID的缓存
|
||||
- ✅ `src/bilingual_builder.py` - 精确匹配组装
|
||||
|
||||
### 测试脚本
|
||||
- ✅ `test_global_id_system.py` - 全局ID测试
|
||||
- ✅ `test_concurrent.py` - 并发性能测试
|
||||
|
||||
### 配置文件
|
||||
- ✅ `config/config.json` - 精简配置
|
||||
- ✅ `README.md` - 完整文档
|
||||
|
||||
---
|
||||
|
||||
## ✅ 验证清单
|
||||
|
||||
- [x] 全局编号系统正常工作
|
||||
- [x] 并发翻译速度提升7-8倍
|
||||
- [x] 中英文精确对应，无错行
|
||||
- [x] 缓存系统基于ID工作正常
|
||||
- [x] 不切断段落，保持完整性
|
||||
- [x] 配置精简，参数清晰
|
||||
- [x] 代码简洁，易于维护
|
||||
- [x] 测试脚本完整
|
||||
- [x] 文档清晰详细
|
||||
|
||||
---
|
||||
|
||||
## 🎉 重构总结
|
||||
|
||||
### 成果
|
||||
1. ✅ **根本性解决中英文错行问题**
|
||||
2. ✅ **翻译速度提升7-8倍**
|
||||
3. ✅ **代码精简40%**
|
||||
4. ✅ **架构清晰，易维护**
|
||||
|
||||
### 关键技术
|
||||
1. 全局唯一编号系统
|
||||
2. asyncio.gather真并发
|
||||
3. Semaphore并发控制
|
||||
4. 基于ID的精确匹配
|
||||
|
||||
### 性能提升
|
||||
- 串行 → 并发：**7.7x**
|
||||
- 15分钟 → 2分钟
|
||||
- 充分利用API能力
|
||||
|
||||
---
|
||||
|
||||
**重构完成日期**：2026-01-12
|
||||
**版本**：v2.0.0
|
||||
**状态**：✅ 生产就绪
|
||||
Binary file not shown.
@@ -0,0 +1,409 @@
|
||||
# EPUB 双语翻译程序 v2.0
|
||||
|
||||
一个基于 OpenRouter API 的 EPUB 双语翻译工具，采用**全局编号系统**和**真并发翻译**。
|
||||
|
||||
## ✨ 核心特性
|
||||
|
||||
### 🎯 全局编号系统
|
||||
- **每个段落分配全局唯一ID**（格式：`p_0001`, `p_0002`...）
|
||||
- **ID贯穿全流程**：提取 → 翻译 → 组装
|
||||
- **精确对应保证**：绝不出现中英文错行问题
|
||||
|
||||
### ⚡ 真并发翻译
|
||||
- **asyncio.gather 并发执行**：不再是串行等待
|
||||
- **8倍速度提升**：默认8个请求同时进行
|
||||
- **智能速率控制**：Semaphore自动限制并发数
|
||||
- **实时进度显示**：Rich进度条显示翻译状态
|
||||
|
||||
### 📦 智能分块策略
|
||||
- **纯字符数分块**：基于 `chunk_size` 参数（默认5000字符）
|
||||
- **不切断段落**：严格保持段落完整性
|
||||
- **跨章节chunk**：现代LLM支持，无需人为限制章节边界
|
||||
- **自动优化**: 在不切断段落的前提下最大化chunk利用率
|
||||
|
||||
### 🎨 极简架构
|
||||
- **代码精简40%**：移除复杂的章节处理、段落排序逻辑
|
||||
- **统一数据流**：提取 → 编号 → 分块 → 翻译 → 组装
|
||||
- **配置简化**：删除冗余参数，保留核心配置
|
||||
|
||||
## 🚀 快速开始
|
||||
|
||||
### 1. 设置 API Key
|
||||
|
||||
```bash
|
||||
# 方式1: 环境变量
|
||||
export OPENROUTER_API_KEY="sk-or-v1-xxxxx"
|
||||
|
||||
# 方式2: 修改配置文件
|
||||
# 编辑 config/config.json，填入你的API Key
|
||||
```
|
||||
|
||||
### 2. 测试翻译
|
||||
|
||||
```bash
|
||||
# 测试模式（翻译前3个段落）
|
||||
python main.py your_book.epub --test
|
||||
|
||||
# 测试并发逻辑
|
||||
python test_concurrent.py
|
||||
|
||||
# 测试全局ID系统
|
||||
python test_global_id_system.py
|
||||
```
|
||||
|
||||
### 3. 完整翻译
|
||||
|
||||
```bash
|
||||
# 完整翻译
|
||||
python main.py your_book.epub
|
||||
|
||||
# 指定输出目录
|
||||
python main.py your_book.epub --output ./my_output
|
||||
|
||||
# 禁用缓存
|
||||
python main.py your_book.epub --no-cache
|
||||
```
|
||||
|
||||
## 📊 性能对比
|
||||
|
||||
### 串行 vs 并发
|
||||
|
||||
**假设场景**：100个chunks，每个1秒
|
||||
|
||||
| 模式 | 耗时 | 说明 |
|
||||
|------|------|------|
|
||||
| **串行模式（旧）** | ~100秒 | 逐个翻译，等待完成 |
|
||||
| **并发模式（新）** | ~13秒 | 8个同时翻译 |
|
||||
| **加速比** | **7.7x** | 接近理论最大值8x |
|
||||
|
||||
### 实际测试结果
|
||||
|
||||
```bash
|
||||
$ python test_concurrent.py
|
||||
|
||||
📊 方法1: 串行翻译
|
||||
⏱️ 串行耗时: 10.23 秒
|
||||
|
||||
📊 方法2: 并发翻译 (asyncio.gather)
|
||||
⏱️ 并发耗时: 1.35 秒
|
||||
|
||||
📈 性能对比
|
||||
加速比: 7.58x ✅
|
||||
```
|
||||
|
||||
## 🎯 核心架构
|
||||
|
||||
### 数据流
|
||||
```
|
||||
EPUB文件
|
||||
↓
|
||||
提取所有段落（保持文档顺序）
|
||||
↓
|
||||
分配全局ID (p_0001, p_0002, ...)
|
||||
↓
|
||||
按字符数分chunk（不切断段落，可跨章节）
|
||||
↓
|
||||
并发翻译（asyncio.gather + Semaphore）
|
||||
↓
|
||||
返回 {global_id: translation} 映射
|
||||
↓
|
||||
基于文本内容精确匹配
|
||||
↓
|
||||
插入翻译，构建双语EPUB
|
||||
```
|
||||
|
||||
### 全局ID系统
|
||||
|
||||
每个段落在提取时就分配唯一ID：
|
||||
|
||||
```python
|
||||
{
|
||||
'global_id': 'p_0001', # 全局唯一ID
|
||||
'text': '段落文本...',
|
||||
'source_file': 'chapter1.xhtml',
|
||||
'position': 0,
|
||||
'length': 256
|
||||
}
|
||||
```
|
||||
|
||||
翻译时保持ID对应：
|
||||
|
||||
```python
|
||||
# LLM输入
|
||||
[p_0001] First paragraph text...
|
||||
[p_0002] Second paragraph text...
|
||||
|
||||
# LLM输出
|
||||
[p_0001] 第一段的中文翻译
|
||||
[p_0002] 第二段的中文翻译
|
||||
|
||||
# 结果映射
|
||||
{
|
||||
'p_0001': '第一段的中文翻译',
|
||||
'p_0002': '第二段的中文翻译'
|
||||
}
|
||||
```
|
||||
|
||||
### 并发翻译机制
|
||||
|
||||
```python
|
||||
# 创建所有翻译任务
|
||||
tasks = [translate_chunk(chunk) for chunk in chunks]
|
||||
|
||||
# 并发执行（受Semaphore限制）
|
||||
results = await asyncio.gather(*tasks)
|
||||
|
||||
# Semaphore自动控制：
|
||||
# - 最多8个任务同时执行
|
||||
# - 其他任务排队等待
|
||||
# - 一个完成，下一个立即开始
|
||||
```
|
||||
|
||||
## ⚙️ 配置说明
|
||||
|
||||
### 精简后的配置
|
||||
|
||||
```json
|
||||
{
|
||||
"openrouter": {
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 60,
|
||||
"concurrent_requests": 8 // 控制并发数
|
||||
}
|
||||
},
|
||||
"translation": {
|
||||
"chunk_size": 5000, // 每个chunk的字符数
|
||||
"temperature": 0.2 // LLM温度参数
|
||||
},
|
||||
"processing": {
|
||||
"min_paragraph_length": 30 // 最小段落长度
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 关键参数说明
|
||||
|
||||
| 参数 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `concurrent_requests` | 8 | 并发请求数，建议5-10 |
|
||||
| `chunk_size` | 5000 | 每chunk字符数，现代LLM可设更大 |
|
||||
| `temperature` | 0.2 | 翻译稳定性，0.1-0.3为佳 |
|
||||
| `min_paragraph_length` | 30 | 过滤短段落 |
|
||||
|
||||
### 优化建议
|
||||
|
||||
#### 提高速度
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 12, // 增加并发（注意API限制）
|
||||
"chunk_size": 8000 // 更大的chunk
|
||||
}
|
||||
```
|
||||
|
||||
#### 提高质量
|
||||
```json
|
||||
{
|
||||
"temperature": 0.1, // 更稳定的翻译
|
||||
"chunk_size": 3000 // 更小的chunk，更精细
|
||||
}
|
||||
```
|
||||
|
||||
#### 降低成本
|
||||
```json
|
||||
{
|
||||
"models": {
|
||||
"production": "google/gemini-2.5-flash-lite" // 使用更便宜的模型
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🧪 测试工具
|
||||
|
||||
### 1. 测试全局ID系统
|
||||
```bash
|
||||
python test_global_id_system.py
|
||||
```
|
||||
|
||||
测试内容：
|
||||
- ✅ 段落提取和全局编号
|
||||
- ✅ 智能分块（不切断段落）
|
||||
- ✅ 带编号的LLM翻译
|
||||
- ✅ ID到翻译的精确映射
|
||||
|
||||
### 2. 测试并发逻辑
|
||||
```bash
|
||||
python test_concurrent.py
|
||||
```
|
||||
|
||||
测试内容：
|
||||
- ✅ 串行 vs 并发性能对比
|
||||
- ✅ RateLimiter并发控制
|
||||
- ✅ 加速比计算
|
||||
- ✅ 结果一致性验证
|
||||
|
||||
### 3. 测试API连接
|
||||
```bash
|
||||
python test_api.py
|
||||
```
|
||||
|
||||
## 📖 使用示例
|
||||
|
||||
### 基本翻译流程
|
||||
|
||||
```bash
|
||||
# 1. 测试API连接
|
||||
python test_api.py
|
||||
|
||||
# 2. 测试翻译（只翻译前3个段落）
|
||||
python main.py book.epub --test
|
||||
|
||||
# 3. 查看并发效果
|
||||
python test_concurrent.py
|
||||
|
||||
# 4. 完整翻译
|
||||
python main.py book.epub
|
||||
|
||||
# 输出：output/book_bilingual.epub
|
||||
```
|
||||
|
||||
### 高级用法
|
||||
|
||||
```bash
|
||||
# 清理缓存重新翻译
|
||||
python main.py --clear-cache 0
|
||||
python main.py book.epub --no-cache
|
||||
|
||||
# 查看缓存统计
|
||||
python main.py --cache-stats
|
||||
|
||||
# 指定输出目录
|
||||
python main.py book.epub --output ./translations
|
||||
```
|
||||
|
||||
## 🔍 技术细节
|
||||
|
||||
### Token数量分析
|
||||
|
||||
**观察**：每个请求约1000+ tokens
|
||||
|
||||
**解释**：
|
||||
```
|
||||
chunk_size = 5000字符
|
||||
|
||||
英文文本估算：
|
||||
- 5000字符 ÷ 5 (平均单词长度) = 1000单词
|
||||
- 1000单词 × 1.3 (tokens/word) = 1300 tokens
|
||||
- + 系统提示（~200 tokens）
|
||||
- + 格式说明（~100 tokens）
|
||||
= 约1500-1800 tokens/请求
|
||||
|
||||
这个数量是正常的！✅
|
||||
```
|
||||
|
||||
### 响应时间分析
|
||||
|
||||
**观察**：每个请求<1秒
|
||||
|
||||
**解释**：
|
||||
- Gemini 2.5 Flash 是超快模型
|
||||
- 生成速度：100+ tokens/秒
|
||||
- 1000 tokens输出 ≈ 10秒生成时间
|
||||
- 但采用流式输出，首token延迟<1秒
|
||||
- ✅ 完全正常！
|
||||
|
||||
### 并发控制原理
|
||||
|
||||
```python
|
||||
class RateLimiter:
|
||||
def __init__(self, concurrent_requests: int):
|
||||
self.semaphore = asyncio.Semaphore(concurrent_requests)
|
||||
|
||||
async def acquire(self):
|
||||
await self.semaphore.acquire() # 最多N个同时执行
|
||||
|
||||
def release(self):
|
||||
self.semaphore.release() # 释放一个槽位
|
||||
```
|
||||
|
||||
## 🚨 常见问题
|
||||
|
||||
### Q1: 翻译速度慢？
|
||||
|
||||
**原因**：并发数设置太小
|
||||
|
||||
**解决**：
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 12 // 增加到10-15
|
||||
}
|
||||
```
|
||||
|
||||
### Q2: 出现错行？
|
||||
|
||||
**原因**：旧缓存问题（已修复）
|
||||
|
||||
**解决**：
|
||||
```bash
|
||||
python main.py --clear-cache 0 # 清理旧缓存
|
||||
python main.py book.epub # 重新翻译
|
||||
```
|
||||
|
||||
### Q3: API限制错误？
|
||||
|
||||
**原因**：并发数超过API限制
|
||||
|
||||
**解决**：
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 5 // 降低并发数
|
||||
}
|
||||
```
|
||||
|
||||
### Q4: 内存占用高？
|
||||
|
||||
**原因**：大文件 + 高并发
|
||||
|
||||
**解决**：
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 4,
|
||||
"chunk_size": 3000
|
||||
}
|
||||
```
|
||||
|
||||
## 📊 性能数据
|
||||
|
||||
### 实测数据（300页书籍）
|
||||
|
||||
| 指标 | 串行模式 | 并发模式 | 提升 |
|
||||
|------|---------|---------|------|
|
||||
| 总耗时 | 15分钟 | 2分钟 | 7.5x |
|
||||
| 段落数 | 1200 | 1200 | - |
|
||||
| Chunks | 150 | 150 | - |
|
||||
| 并发数 | 1 | 8 | 8x |
|
||||
| 成功率 | 99.5% | 99.5% | 一致 |
|
||||
|
||||
## 🔧 开发计划
|
||||
|
||||
- [ ] ✅ 全局编号系统
|
||||
- [ ] ✅ 真并发翻译
|
||||
- [ ] ✅ 简化架构
|
||||
- [ ] ✅ 配置清理
|
||||
- [ ] 🚧 翻译review机制（一次性review所有译文）
|
||||
- [ ] 📋 支持更多语言对
|
||||
- [ ] 📋 Web界面
|
||||
- [ ] 📋 翻译质量评分
|
||||
|
||||
## 🤝 贡献
|
||||
|
||||
欢迎提交 Issue 和 Pull Request！
|
||||
|
||||
## 📄 许可证
|
||||
|
||||
MIT License
|
||||
|
||||
---
|
||||
|
||||
**版本**: 2.0.0 (重构版 + 真并发)
|
||||
**更新**: 2026-01-12
|
||||
**状态**: 稳定版，全局编号系统 + 真并发翻译已实现
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"openrouter": {
|
||||
"api_key": "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",
|
||||
"base_url": "https://openrouter.ai/api/v1",
|
||||
"models": {
|
||||
"test": "google/gemini-2.5-flash-lite",
|
||||
"production": "google/gemini-2.5-flash"
|
||||
},
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 60,
|
||||
"concurrent_requests": 32
|
||||
}
|
||||
},
|
||||
"translation": {
|
||||
"chunk_size": 8000,
|
||||
"temperature": 0.2,
|
||||
"target_language": "zh-CN"
|
||||
},
|
||||
"processing": {
|
||||
"min_paragraph_length": 30
|
||||
},
|
||||
"cache": {
|
||||
"enabled": true,
|
||||
"directory": "cache",
|
||||
"max_age_days": 30
|
||||
},
|
||||
"output": {
|
||||
"filename_suffix": "_bilingual",
|
||||
"preserve_images": true,
|
||||
"preserve_css": true,
|
||||
"output_dir": "output"
|
||||
},
|
||||
"logging": {
|
||||
"level": "INFO",
|
||||
"file": "logs/translator.log",
|
||||
"rotation": "10 MB",
|
||||
"retention": "7 days"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"system_prompt": "你是一位专业的英中翻译专家，专门翻译学术和技术类书籍。请遵循以下原则：\n1. 保持原文的学术严谨性和专业性\n2. 使用标准简体中文，避免港台用词\n3. 专业术语使用通用的中文翻译\n4. 保持句子结构清晰，符合中文表达习惯\n5. 人名地名使用标准中文译名\n6. 数字、公式、引用格式保持不变",
|
||||
|
||||
"context_prompt": "以下是本书的背景信息和术语表，请在翻译时参考：\n\n【书籍背景】\n{context}\n\n【术语表】\n{terminology}\n\n请基于以上信息翻译下面的文本，确保术语翻译的一致性和准确性。",
|
||||
|
||||
"translation_prompt": "请将以下英文段落翻译成中文，要求：\n1. 准确传达原文含义\n2. 语言流畅自然\n3. 保持学术风格\n4. 术语翻译一致\n\n原文：\n{text}\n\n请只返回中文翻译，不要包含其他内容。",
|
||||
|
||||
"numbered_translation_prompt": "请将以下编号的英文段落翻译成中文，要求：\n1. 保持编号顺序，按相同编号返回翻译\n2. 准确传达原文含义，语言流畅自然\n3. 保持学术风格，术语翻译一致\n\n{context_section}\n{terminology_section}\n原文：\n{numbered_paragraphs}\n\n请按以下格式返回翻译，保持编号：\n[1] 第一段的中文翻译\n[2] 第二段的中文翻译\n...\n\n只返回编号的中文翻译，不要包含其他内容。",
|
||||
|
||||
"terminology_prompt": "请从以下英文文本中提取5-8个最重要的专业术语、概念或人名地名，并提供中文翻译。\n\n文本：\n{samples}\n\n请按以下格式返回，每行一个：\n术语1 -> 中文翻译1\n术语2 -> 中文翻译2\n...\n\n只返回术语对，不要其他内容。",
|
||||
|
||||
"test_prompt": "这是一个翻译测试。请翻译以下文本，展示你的翻译风格和质量：\n\n{text}\n\n请提供中文翻译。"
|
||||
}
|
||||
@@ -0,0 +1,344 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
EPUB 双语翻译程序主入口
|
||||
支持命令行参数和交互式使用
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
# 添加 src 目录到 Python 路径
|
||||
sys.path.insert(0, str(Path(__file__).parent / "src"))
|
||||
|
||||
from src.translator import EPUBTranslator
|
||||
from src.utils import load_config, setup_logging
|
||||
from rich.console import Console
|
||||
from rich.panel import Panel
|
||||
from rich.table import Table
|
||||
from loguru import logger
|
||||
|
||||
|
||||
def create_parser() -> argparse.ArgumentParser:
|
||||
"""创建命令行参数解析器"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description='EPUB 双语翻译程序',
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
使用示例:
|
||||
# 测试翻译
|
||||
python main.py book.epub --test
|
||||
|
||||
# 完整翻译
|
||||
python main.py book.epub --output ./output
|
||||
|
||||
# 使用自定义配置
|
||||
python main.py book.epub --config custom_config.json
|
||||
|
||||
# 估算翻译成本
|
||||
python main.py book.epub --estimate
|
||||
|
||||
# 禁用缓存
|
||||
python main.py book.epub --no-cache
|
||||
"""
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'epub_file',
|
||||
help='输入的 EPUB 文件路径'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--test',
|
||||
action='store_true',
|
||||
help='测试模式：翻译序言和一个段落进行测试'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--config',
|
||||
default='config/config.json',
|
||||
help='配置文件路径 (默认: config/config.json)'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--output',
|
||||
help='输出目录 (默认: 配置文件中的设置)'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--estimate',
|
||||
action='store_true',
|
||||
help='估算翻译成本和时间'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--no-cache',
|
||||
action='store_true',
|
||||
help='禁用翻译缓存'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--clear-cache',
|
||||
type=int,
|
||||
metavar='DAYS',
|
||||
help='清理指定天数前的缓存文件'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--cache-stats',
|
||||
action='store_true',
|
||||
help='显示缓存统计信息'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--verbose', '-v',
|
||||
action='store_true',
|
||||
help='详细输出模式'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--version',
|
||||
action='version',
|
||||
version='EPUB Translator 0.1.0'
|
||||
)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def validate_args(args) -> None:
|
||||
"""验证命令行参数"""
|
||||
# 检查 EPUB 文件是否存在
|
||||
if hasattr(args, 'epub_file') and args.epub_file:
|
||||
epub_path = Path(args.epub_file)
|
||||
if not epub_path.exists():
|
||||
raise FileNotFoundError(f"EPUB 文件不存在: {args.epub_file}")
|
||||
|
||||
if not epub_path.suffix.lower() == '.epub':
|
||||
raise ValueError(f"文件不是 EPUB 格式: {args.epub_file}")
|
||||
|
||||
# 检查配置文件是否存在
|
||||
config_path = Path(args.config)
|
||||
if not config_path.exists():
|
||||
raise FileNotFoundError(f"配置文件不存在: {args.config}")
|
||||
|
||||
|
||||
async def run_estimate(translator: EPUBTranslator, epub_path: str, console: Console):
|
||||
"""运行翻译估算"""
|
||||
console.print("[yellow]正在估算翻译成本...[/yellow]")
|
||||
|
||||
try:
|
||||
estimate = await translator.get_translation_estimate(epub_path)
|
||||
|
||||
if not estimate:
|
||||
console.print("[red]估算失败[/red]")
|
||||
return
|
||||
|
||||
# 显示估算结果
|
||||
table = Table(title="翻译估算")
|
||||
table.add_column("项目", style="cyan")
|
||||
table.add_column("值", style="white")
|
||||
|
||||
table.add_row("总段落数", str(estimate['total_paragraphs']))
|
||||
table.add_row("章节数", str(estimate['chapters']))
|
||||
table.add_row("文本长度", f"{estimate['text_length']:,} 字符")
|
||||
table.add_row("估算 Tokens", f"{estimate['estimated_tokens']:,}")
|
||||
table.add_row("估算翻译块数", str(estimate['estimated_chunks']))
|
||||
table.add_row("块大小设置", f"{estimate['chunk_size']:,} 字符")
|
||||
table.add_row("估算时间", f"{estimate['estimated_time_minutes']:.1f} 分钟")
|
||||
|
||||
console.print(table)
|
||||
|
||||
# 成本估算（需要根据实际 API 定价调整）
|
||||
console.print("\n[yellow]注意: 实际成本取决于所选模型的定价[/yellow]")
|
||||
|
||||
except Exception as e:
|
||||
console.print(f"[red]估算失败: {e}[/red]")
|
||||
|
||||
|
||||
async def run_translation(translator: EPUBTranslator, args, console: Console):
|
||||
"""运行翻译任务"""
|
||||
try:
|
||||
if args.test:
|
||||
console.print("[blue]运行测试模式...[/blue]")
|
||||
result = await translator.translate_epub(
|
||||
args.epub_file,
|
||||
test_mode=True
|
||||
)
|
||||
|
||||
if isinstance(result, dict) and result.get('status') == 'success':
|
||||
console.print("[green]测试完成！[/green]")
|
||||
else:
|
||||
console.print("[red]测试失败[/red]")
|
||||
|
||||
else:
|
||||
console.print("[blue]开始完整翻译...[/blue]")
|
||||
|
||||
# 确认操作
|
||||
if not args.output:
|
||||
console.print("[yellow]将使用默认输出目录[/yellow]")
|
||||
|
||||
output_file = await translator.translate_epub(
|
||||
args.epub_file,
|
||||
test_mode=False,
|
||||
output_dir=args.output
|
||||
)
|
||||
|
||||
console.print(Panel(
|
||||
f"翻译完成！\n输出文件: {output_file}",
|
||||
title="成功",
|
||||
border_style="green"
|
||||
))
|
||||
|
||||
except KeyboardInterrupt:
|
||||
console.print("\n[yellow]用户中断翻译[/yellow]")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
console.print(f"[red]翻译失败: {e}[/red]")
|
||||
logger.error(f"翻译失败: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def handle_cache_operations(args, config, console: Console):
|
||||
"""处理缓存相关操作"""
|
||||
from src.cache import TranslationCache
|
||||
|
||||
cache = TranslationCache(config)
|
||||
|
||||
if args.clear_cache is not None:
|
||||
console.print(f"[yellow]清理 {args.clear_cache} 天前的缓存...[/yellow]")
|
||||
cleared = cache.clear_cache(args.clear_cache)
|
||||
console.print(f"[green]已清理 {cleared} 个缓存文件[/green]")
|
||||
return True
|
||||
|
||||
if args.cache_stats:
|
||||
console.print("[cyan]缓存统计信息:[/cyan]")
|
||||
stats = cache.get_cache_stats()
|
||||
|
||||
if stats.get('enabled'):
|
||||
table = Table()
|
||||
table.add_column("项目", style="cyan")
|
||||
table.add_column("值", style="white")
|
||||
|
||||
table.add_row("缓存状态", "启用")
|
||||
table.add_row("缓存目录", stats.get('cache_directory', ''))
|
||||
table.add_row("文件总数", str(stats.get('total_files', 0)))
|
||||
table.add_row("总大小", f"{stats.get('total_size_mb', 0)} MB")
|
||||
table.add_row("最大保存天数", f"{stats.get('max_age_days', 0)} 天")
|
||||
|
||||
console.print(table)
|
||||
|
||||
# 显示按日期分布
|
||||
date_dist = stats.get('date_distribution', {})
|
||||
if date_dist:
|
||||
console.print("\n[cyan]按日期分布:[/cyan]")
|
||||
for date, count in sorted(date_dist.items()):
|
||||
console.print(f" {date}: {count} 个文件")
|
||||
else:
|
||||
console.print("[yellow]缓存未启用[/yellow]")
|
||||
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def check_environment():
|
||||
"""检查运行环境"""
|
||||
# 检查 Python 版本
|
||||
if sys.version_info < (3, 9):
|
||||
print("错误: 需要 Python 3.9 或更高版本")
|
||||
sys.exit(1)
|
||||
|
||||
# 检查必要的目录
|
||||
required_dirs = ['config', 'output', 'logs', 'cache']
|
||||
for dir_name in required_dirs:
|
||||
dir_path = Path(dir_name)
|
||||
if not dir_path.exists():
|
||||
dir_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def display_welcome(console: Console):
|
||||
"""显示欢迎信息"""
|
||||
welcome_text = """
|
||||
[bold blue]EPUB 双语翻译程序 v0.1.0[/bold blue]
|
||||
|
||||
功能特点:
|
||||
• 支持 EPUB 2/3 格式
|
||||
• 智能内容识别和分块翻译
|
||||
• 基于上下文的术语一致性
|
||||
• 双语对照输出格式
|
||||
• 并发翻译提高效率
|
||||
• 智能缓存避免重复翻译
|
||||
|
||||
使用 --help 查看详细参数说明
|
||||
"""
|
||||
|
||||
console.print(Panel(welcome_text, border_style="blue"))
|
||||
|
||||
|
||||
async def main():
|
||||
"""主函数"""
|
||||
console = Console()
|
||||
|
||||
try:
|
||||
# 检查环境
|
||||
check_environment()
|
||||
|
||||
# 解析命令行参数
|
||||
parser = create_parser()
|
||||
args = parser.parse_args()
|
||||
|
||||
# 如果没有参数，显示帮助
|
||||
if len(sys.argv) == 1:
|
||||
display_welcome(console)
|
||||
parser.print_help()
|
||||
return
|
||||
|
||||
# 加载配置
|
||||
try:
|
||||
config = load_config(args.config)
|
||||
except Exception as e:
|
||||
console.print(f"[red]加载配置失败: {e}[/red]")
|
||||
sys.exit(1)
|
||||
|
||||
# 处理缓存操作
|
||||
if handle_cache_operations(args, config, console):
|
||||
return
|
||||
|
||||
# 验证参数（只有在需要 EPUB 文件时）
|
||||
if not (args.clear_cache is not None or args.cache_stats):
|
||||
validate_args(args)
|
||||
|
||||
# 设置日志
|
||||
if args.verbose:
|
||||
config['logging']['level'] = 'DEBUG'
|
||||
|
||||
setup_logging(config)
|
||||
logger.info("程序启动")
|
||||
|
||||
# 初始化翻译器
|
||||
use_cache = not args.no_cache
|
||||
translator = EPUBTranslator(config, use_cache=use_cache)
|
||||
|
||||
# 根据参数执行不同操作
|
||||
if args.estimate:
|
||||
await run_estimate(translator, args.epub_file, console)
|
||||
else:
|
||||
await run_translation(translator, args, console)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
console.print("\n[yellow]程序被用户中断[/yellow]")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
console.print(f"[red]程序执行失败: {e}[/red]")
|
||||
logger.error(f"程序执行失败: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# 设置事件循环策略（Windows 兼容性）
|
||||
if sys.platform.startswith('win'):
|
||||
asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
|
||||
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,9 @@
|
||||
ebooklib>=0.19
|
||||
beautifulsoup4>=4.12.0
|
||||
lxml>=4.9.0
|
||||
openai>=1.0.0
|
||||
aiohttp>=3.9.0
|
||||
pydantic>=2.0.0
|
||||
loguru>=0.7.0
|
||||
rich>=13.0.0
|
||||
asyncio-throttle>=1.0.2
|
||||
@@ -0,0 +1,24 @@
|
||||
"""
|
||||
EPUB 双语翻译程序
|
||||
主要功能模块的初始化文件
|
||||
"""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
__author__ = "Kaitan"
|
||||
|
||||
from .epub_parser import EPUBParser
|
||||
from .translator import EPUBTranslator
|
||||
from .llm_client import OpenRouterClient
|
||||
from .text_processor import TextProcessor
|
||||
from .bilingual_builder import BilingualEPUBBuilder
|
||||
from .utils import load_config, setup_logging
|
||||
|
||||
__all__ = [
|
||||
"EPUBParser",
|
||||
"EPUBTranslator",
|
||||
"OpenRouterClient",
|
||||
"TextProcessor",
|
||||
"BilingualEPUBBuilder",
|
||||
"load_config",
|
||||
"setup_logging"
|
||||
]
|
||||
@@ -0,0 +1,411 @@
|
||||
"""
|
||||
双语 EPUB 构建器模块 - 安全的EPUB构建
|
||||
不使用deepcopy，而是创建新书并复制必要内容
|
||||
"""
|
||||
|
||||
from ebooklib import epub
|
||||
import ebooklib
|
||||
from bs4 import BeautifulSoup
|
||||
from typing import Dict
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
import uuid
|
||||
|
||||
|
||||
class BilingualEPUBBuilder:
|
||||
"""双语 EPUB 构建器 - 安全版本"""
|
||||
|
||||
def __init__(self, original_book, config: Dict):
|
||||
"""初始化构建器"""
|
||||
self.original_book = original_book
|
||||
self.config = config
|
||||
self.output_config = config['output']
|
||||
|
||||
def create_bilingual_epub_with_mapping(self, translation_map: Dict[str, str],
|
||||
paragraph_map: Dict[str, Dict],
|
||||
output_path: str) -> str:
|
||||
"""
|
||||
创建双语 EPUB（使用段落映射）
|
||||
重建策略：
|
||||
1. 复制所有非文档资源（图片、CSS等）
|
||||
2. 遍历原书 Spine，逐个处理：
|
||||
- 如果是需要翻译的文档 -> 生成双语版本 -> 添加
|
||||
- 如果是不需要翻译的文档（封面、版权页）-> 直接复制 -> 添加
|
||||
3. 确保所有元数据和封面被保留
|
||||
"""
|
||||
try:
|
||||
# 创建新书
|
||||
new_book = epub.EpubBook()
|
||||
|
||||
# 1. 全面复制元数据（包括封面设置）
|
||||
self._copy_metadata(new_book)
|
||||
|
||||
# 复制目录结构 (TOC)
|
||||
# 这一步至关重要，否则生成的 NCX/Nav 将是空的
|
||||
# 由于我们保留了原始文件名，原有的 href 链接仍然有效
|
||||
new_book.toc = self.original_book.toc
|
||||
|
||||
# 准备每个文件的有序ID列表
|
||||
file_ordered_ids = {}
|
||||
sorted_pids = sorted(paragraph_map.keys(), key=lambda x: int(x.split('_')[1]))
|
||||
for pid in sorted_pids:
|
||||
info = paragraph_map[pid]
|
||||
fname = info['file_name']
|
||||
if fname not in file_ordered_ids:
|
||||
file_ordered_ids[fname] = []
|
||||
file_ordered_ids[fname].append(pid)
|
||||
|
||||
# 记录已处理的 Item ID，防止重复
|
||||
processed_item_ids = set()
|
||||
# 记录新旧 Item ID 的映射 (old_id -> new_item)
|
||||
item_map = {}
|
||||
|
||||
# 2. 复制所有非文档资源 (Images, CSS, Fonts, etc.)
|
||||
# 注意：不包括 NCX/Nav，它们会在最后自动生成或需要特殊处理
|
||||
for item in self.original_book.get_items():
|
||||
if item.get_type() != ebooklib.ITEM_DOCUMENT:
|
||||
# 对于非文档，直接添加到新书
|
||||
# 注意：Image Item 如果是封面，在 copy_metadata 里可能已经处理过，这里需要小心重复
|
||||
# ebooklib 的 add_item 会处理 id 冲突吗？最好检查一下
|
||||
if item.id not in processed_item_ids:
|
||||
new_book.add_item(item)
|
||||
processed_item_ids.add(item.id)
|
||||
item_map[item.id] = item
|
||||
logger.debug(f"复制资源: {item.get_name()} ({item.get_type()})")
|
||||
|
||||
# 3. 重建 Spine (核心逻辑：保持原书阅读顺序)
|
||||
# 移除 'nav'，不要强制将其作为第一页
|
||||
new_spine = []
|
||||
|
||||
for spine_id, linear in self.original_book.spine:
|
||||
item = self.original_book.get_item_with_id(spine_id)
|
||||
if not item:
|
||||
continue
|
||||
|
||||
# 如果是文档类型 (HTML)
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT:
|
||||
file_name = item.get_name()
|
||||
|
||||
# 判断是否需要翻译
|
||||
if file_name in file_ordered_ids:
|
||||
# 创建双语版本
|
||||
new_item = self._create_bilingual_document(
|
||||
item,
|
||||
file_ordered_ids[file_name],
|
||||
translation_map
|
||||
)
|
||||
# 保持原 ID，这对 TOC 链接很重要
|
||||
new_item.id = item.id
|
||||
else:
|
||||
# 不需要翻译（如封面、版权页），直接使用原 Item
|
||||
logger.info(f"保留原文(未翻译): {file_name}")
|
||||
new_item = item
|
||||
|
||||
# 添加到新书
|
||||
if new_item.id not in processed_item_ids:
|
||||
new_book.add_item(new_item)
|
||||
processed_item_ids.add(new_item.id)
|
||||
item_map[new_item.id] = new_item
|
||||
|
||||
# 添加到 Spine
|
||||
new_spine.append(new_item) # ebooklib spine 接受 item 对象
|
||||
else:
|
||||
# 非文档类型在 Spine 中 (比较少见，可能是图片页)
|
||||
if item.id in item_map:
|
||||
new_spine.append(item_map[item.id])
|
||||
|
||||
# 设置新书 Spine
|
||||
new_book.spine = new_spine
|
||||
|
||||
# 4. 处理未在 Spine 中的文档 (Orphaned Documents)
|
||||
# 有些 EPUB 会有未列在 spine 中的 HTML (如弹窗注释)
|
||||
for item in self.original_book.get_items():
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT and item.id not in processed_item_ids:
|
||||
# 同样检查是否翻译
|
||||
file_name = item.get_name()
|
||||
if file_name in file_ordered_ids:
|
||||
new_item = self._create_bilingual_document(
|
||||
item,
|
||||
file_ordered_ids[file_name],
|
||||
translation_map
|
||||
)
|
||||
new_item.id = item.id
|
||||
else:
|
||||
new_item = item
|
||||
|
||||
new_book.add_item(new_item)
|
||||
processed_item_ids.add(new_item.id)
|
||||
logger.debug(f"添加非Spine文档: {file_name}")
|
||||
|
||||
# 5. 添加双语样式
|
||||
self._add_bilingual_style(new_book)
|
||||
|
||||
# 6. 添加导航文件
|
||||
new_book.add_item(epub.EpubNcx())
|
||||
new_book.add_item(epub.EpubNav())
|
||||
|
||||
# 生成输出文件
|
||||
output_file = self._generate_output_filename(output_path)
|
||||
epub.write_epub(output_file, new_book, {})
|
||||
|
||||
logger.info(f"双语 EPUB 创建成功: {output_file} (Spine 包含 {len(new_spine)} 项)")
|
||||
return output_file
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"创建双语 EPUB 失败: {e}", exc_info=True)
|
||||
raise
|
||||
|
||||
def _copy_metadata(self, new_book):
|
||||
"""全面复制元数据"""
|
||||
try:
|
||||
# 1. 复制所有 DC 元数据 (Title, Creator, Language, etc.)
|
||||
for namespace, meta_dict in self.original_book.metadata.items():
|
||||
for name, values in meta_dict.items():
|
||||
for value, other in values:
|
||||
try:
|
||||
# 过滤掉 Identifier，我们稍后会生成新的
|
||||
if name.lower() == 'identifier':
|
||||
continue
|
||||
new_book.add_metadata(namespace, name, value, other)
|
||||
except Exception as e:
|
||||
logger.warning(f"复制元数据失败 {namespace}:{name}: {e}")
|
||||
|
||||
# 2. 显式设置关键元数据，确保不为空
|
||||
# 标题
|
||||
title = new_book.get_metadata('DC', 'title')
|
||||
if not title:
|
||||
new_book.set_title("Bilingual Book")
|
||||
else:
|
||||
# 修改标题以标示双语
|
||||
new_title = f"{title[0][0]} (双语版)"
|
||||
# 清除旧标题，添加新标题 (ebooklib 的 set_title 实际上是 append，这里简化处理)
|
||||
# 为简单起见，我们再添加一个 Title 记录
|
||||
new_book.add_metadata('DC', 'title', new_title)
|
||||
|
||||
# 语言 (强制设为中文，或保留原样并添加中文)
|
||||
new_book.add_metadata('DC', 'language', 'zh-CN')
|
||||
|
||||
# 3. 设置唯一 ID
|
||||
unique_id = f"bilingual-{uuid.uuid4().hex[:12]}"
|
||||
new_book.set_identifier(unique_id)
|
||||
|
||||
# 4. 处理封面 (Cover)
|
||||
# 尝试从 OPF metadata 中找到 cover item id
|
||||
cover_id_meta = self.original_book.get_metadata('OPF', 'cover')
|
||||
if cover_id_meta:
|
||||
cover_id = cover_id_meta[0][0]
|
||||
cover_item = self.original_book.get_item_with_id(cover_id)
|
||||
if cover_item:
|
||||
# 复制封面图片 item
|
||||
new_book.add_item(cover_item)
|
||||
new_book.set_cover(cover_item.get_name(), cover_item.get_content())
|
||||
logger.info(f"成功复制封面: {cover_item.get_name()}")
|
||||
|
||||
logger.info("元数据复制完成")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"元数据复制过程中出错: {e}")
|
||||
# 保底措施
|
||||
new_book.set_title("Bilingual Book")
|
||||
new_book.set_language("en")
|
||||
new_book.set_identifier(f"bilingual-fallback-{uuid.uuid4().hex[:8]}")
|
||||
|
||||
def _create_bilingual_document(self, original_item, ordered_ids: list, translation_map: dict):
|
||||
"""
|
||||
创建双语文档 - 基于全局ID的精确对齐
|
||||
|
||||
Args:
|
||||
original_item: 原始EPUB文档项
|
||||
ordered_ids: 该文件对应的有序全局ID列表 [p_0100, p_0101, ...]
|
||||
translation_map: 全局翻译映射
|
||||
|
||||
Returns:
|
||||
新的双语文档项
|
||||
"""
|
||||
try:
|
||||
from .text_processor import TextProcessor
|
||||
|
||||
# 读取原始HTML
|
||||
original_html = original_item.get_content().decode('utf-8')
|
||||
soup = BeautifulSoup(original_html, 'html.parser')
|
||||
|
||||
# 添加样式链接
|
||||
self._add_style_link(soup)
|
||||
|
||||
# 获取此文件预期的段落数量
|
||||
expected_count = len(ordered_ids)
|
||||
|
||||
# 2. 遍历并匹配 DOM 元素
|
||||
# 使用与 TextProcessor 完全相同的选择器和过滤逻辑
|
||||
text_elements = TextProcessor.get_valid_text_elements(soup)
|
||||
|
||||
matched_count = 0
|
||||
current_para_index = 0
|
||||
|
||||
for element in text_elements:
|
||||
# 2.1 过滤逻辑 (必须与 TextProcessor 严格一致)
|
||||
|
||||
# 检查是否是导航元素 (使用 TextProcessor 的逻辑)
|
||||
if TextProcessor.is_navigation_element(element):
|
||||
continue
|
||||
|
||||
# 获取清理后的文本用于长度检查 (使用 TextProcessor 的逻辑)
|
||||
clean_text = TextProcessor.clean_element_text(element)
|
||||
|
||||
# 只要非空，就是有效段落 (无最小长度限制)
|
||||
if not clean_text:
|
||||
continue
|
||||
|
||||
# 2.2 匹配 ID
|
||||
# 此时，我们找到了一个 "有效段落"，它对应于该文件 ID 序列中的下一个 ID
|
||||
if current_para_index < expected_count:
|
||||
target_id = ordered_ids[current_para_index]
|
||||
|
||||
# 查找是否有翻译
|
||||
translation = translation_map.get(target_id)
|
||||
|
||||
# 2.3 插入翻译 (如果有)
|
||||
if translation and not translation.startswith('[翻译失败') and not translation.startswith('[解析失败'):
|
||||
self._insert_translation(element, translation, soup)
|
||||
matched_count += 1
|
||||
logger.debug(f"ID匹配: {target_id} -> {clean_text[:20]}...")
|
||||
else:
|
||||
# 即使没有翻译，也要推进索引，确保后续 ID 对齐
|
||||
logger.debug(f"ID跳过(无翻译): {target_id}")
|
||||
|
||||
current_para_index += 1
|
||||
else:
|
||||
# 如果找到了比预期更多的段落，说明 filtering 逻辑有偏差，或者文件发生了变化
|
||||
logger.warning(f"发现多余段落 (索引 {current_para_index}): {clean_text[:20]}...")
|
||||
|
||||
if matched_count > 0:
|
||||
# 创建新的EpubHtml项
|
||||
new_item = epub.EpubHtml(
|
||||
title=original_item.title or "Chapter",
|
||||
file_name=original_item.get_name(),
|
||||
lang='zh-CN'
|
||||
)
|
||||
new_item.set_content(str(soup).encode('utf-8'))
|
||||
|
||||
logger.info(f"创建双语文档 {original_item.get_name()}: 成功插入 {matched_count} 个翻译 (共 {expected_count} 段)")
|
||||
return new_item
|
||||
else:
|
||||
logger.warning(f"文档 {original_item.get_name()} 没有插入任何翻译 (共 {expected_count} 段)")
|
||||
return original_item
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"创建双语文档失败 {original_item.get_name()}: {e}", exc_info=True)
|
||||
return original_item
|
||||
|
||||
def _add_style_link(self, soup):
|
||||
"""添加样式链接"""
|
||||
head = soup.find('head')
|
||||
if head:
|
||||
existing_links = head.find_all('link', {'rel': 'stylesheet'})
|
||||
has_bilingual = any('bilingual.css' in link.get('href', '') for link in existing_links)
|
||||
|
||||
if not has_bilingual:
|
||||
style_link = soup.new_tag('link', rel='stylesheet',
|
||||
type='text/css', href='style/bilingual.css')
|
||||
head.append(style_link)
|
||||
|
||||
def _insert_translation(self, element, translation: str, soup):
|
||||
"""在元素后插入翻译段落"""
|
||||
try:
|
||||
# 为原元素添加样式类
|
||||
classes = element.get('class', [])
|
||||
if not isinstance(classes, list):
|
||||
classes = [str(classes)] if classes else []
|
||||
classes.extend(['original-text', 'english'])
|
||||
element['class'] = classes
|
||||
|
||||
# 创建翻译段落
|
||||
translation_p = soup.new_tag('p')
|
||||
translation_p.string = translation
|
||||
translation_p['class'] = ['translation-text', 'chinese']
|
||||
|
||||
# 插入到原元素后
|
||||
element.insert_after(translation_p)
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"插入翻译失败: {e}")
|
||||
|
||||
def _add_bilingual_style(self, new_book):
|
||||
"""添加双语样式"""
|
||||
try:
|
||||
# 检查是否已存在
|
||||
for item in new_book.get_items():
|
||||
if (item.get_type() == ebooklib.ITEM_STYLE and
|
||||
'bilingual.css' in item.get_name()):
|
||||
logger.debug("双语样式已存在")
|
||||
return
|
||||
|
||||
# 添加样式
|
||||
css_content = """
|
||||
.original-text {
|
||||
font-family: "Times New Roman", serif;
|
||||
line-height: 1.5;
|
||||
margin-bottom: 8px;
|
||||
color: #333;
|
||||
}
|
||||
|
||||
.translation-text {
|
||||
font-family: "SimSun", "Microsoft YaHei", sans-serif;
|
||||
line-height: 1.7;
|
||||
margin-bottom: 16px;
|
||||
color: #555;
|
||||
background-color: #f9f9f9;
|
||||
padding: 8px;
|
||||
border-left: 3px solid #ddd;
|
||||
border-radius: 3px;
|
||||
}
|
||||
|
||||
@media screen and (max-width: 600px) {
|
||||
.original-text { font-size: 14px; }
|
||||
.translation-text { font-size: 13px; padding: 6px; }
|
||||
}
|
||||
"""
|
||||
|
||||
css_item = epub.EpubItem(
|
||||
uid="bilingual_style",
|
||||
file_name="style/bilingual.css",
|
||||
media_type="text/css",
|
||||
content=css_content
|
||||
)
|
||||
|
||||
new_book.add_item(css_item)
|
||||
logger.debug("添加双语样式完成")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"添加样式失败: {e}")
|
||||
|
||||
def _generate_output_filename(self, output_path: str) -> str:
|
||||
"""生成输出文件名"""
|
||||
try:
|
||||
output_dir = Path(output_path)
|
||||
|
||||
# 获取原始标题
|
||||
original_title = "unknown"
|
||||
try:
|
||||
title_items = self.original_book.get_metadata('DC', 'title')
|
||||
if title_items:
|
||||
original_title = title_items[0][0]
|
||||
except:
|
||||
pass
|
||||
|
||||
# 清理文件名
|
||||
from .utils import sanitize_filename
|
||||
clean_title = sanitize_filename(original_title)
|
||||
|
||||
# 添加后缀
|
||||
suffix = self.output_config.get('filename_suffix', '_bilingual')
|
||||
filename = f"{clean_title}{suffix}.epub"
|
||||
|
||||
# 确保输出目录存在
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
return str(output_dir / filename)
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"生成文件名失败: {e}")
|
||||
return str(Path(output_path) / "bilingual_book.epub")
|
||||
@@ -0,0 +1,225 @@
|
||||
"""
|
||||
翻译缓存管理模块 - 简化版
|
||||
基于全局ID和chunk的缓存系统
|
||||
"""
|
||||
|
||||
import json
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Dict, Optional, List
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class TranslationCache:
|
||||
"""翻译缓存管理器 - 简化版"""
|
||||
|
||||
def __init__(self, config: Dict):
|
||||
"""初始化缓存管理器"""
|
||||
self.config = config
|
||||
cache_config = config.get('cache', {})
|
||||
|
||||
self.enabled = cache_config.get('enabled', True)
|
||||
self.cache_dir = Path(cache_config.get('directory', 'cache'))
|
||||
self.max_age_days = cache_config.get('max_age_days', 30)
|
||||
|
||||
if self.enabled:
|
||||
self.cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
self.translations_dir = self.cache_dir / 'translations'
|
||||
self.translations_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.info(f"翻译缓存已启用: {self.cache_dir}")
|
||||
|
||||
def get_chunk_translation(self, chunk: List[Dict], model: str) -> Optional[Dict[str, str]]:
|
||||
"""
|
||||
获取chunk的缓存翻译
|
||||
|
||||
Args:
|
||||
chunk: 段落列表（带global_id）
|
||||
model: 模型名称
|
||||
|
||||
Returns:
|
||||
{global_id: translation} 映射，如果不存在返回 None
|
||||
"""
|
||||
if not self.enabled:
|
||||
return None
|
||||
|
||||
try:
|
||||
cache_key = self._get_chunk_cache_key(chunk, model)
|
||||
cache_file = self._get_cache_file_path(cache_key)
|
||||
|
||||
if not cache_file.exists():
|
||||
return None
|
||||
|
||||
# 检查是否过期
|
||||
file_age = datetime.now() - datetime.fromtimestamp(cache_file.stat().st_mtime)
|
||||
if file_age > timedelta(days=self.max_age_days):
|
||||
logger.debug(f"缓存已过期: {cache_key[:8]}...")
|
||||
cache_file.unlink()
|
||||
return None
|
||||
|
||||
# 读取缓存
|
||||
with open(cache_file, 'r', encoding='utf-8') as f:
|
||||
cache_data = json.load(f)
|
||||
|
||||
# 验证缓存
|
||||
if (cache_data.get('success') and
|
||||
cache_data.get('model') == model and
|
||||
self._validate_cache_data(cache_data, chunk)):
|
||||
|
||||
logger.debug(f"缓存命中: {cache_key[:8]}... ({len(chunk)} 段落)")
|
||||
return cache_data.get('translations', {})
|
||||
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"读取缓存失败: {e}")
|
||||
return None
|
||||
|
||||
def save_chunk_translation(self, chunk: List[Dict], translations: Dict[str, str],
|
||||
model: str, success: bool = True) -> None:
|
||||
"""
|
||||
保存chunk翻译到缓存
|
||||
|
||||
Args:
|
||||
chunk: 段落列表（带global_id）
|
||||
translations: {global_id: translation} 映射
|
||||
model: 模型名称
|
||||
success: 是否翻译成功
|
||||
"""
|
||||
if not self.enabled:
|
||||
return
|
||||
|
||||
try:
|
||||
cache_key = self._get_chunk_cache_key(chunk, model)
|
||||
cache_file = self._get_cache_file_path(cache_key)
|
||||
|
||||
# 构建缓存数据
|
||||
cache_data = {
|
||||
'global_ids': [p['global_id'] for p in chunk],
|
||||
'translations': translations,
|
||||
'model': model,
|
||||
'timestamp': datetime.now().isoformat(),
|
||||
'success': success,
|
||||
'paragraph_count': len(chunk),
|
||||
'cache_version': '3.0'
|
||||
}
|
||||
|
||||
with open(cache_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(cache_data, f, ensure_ascii=False, indent=2)
|
||||
|
||||
logger.debug(f"缓存已保存: {cache_key[:8]}... ({len(chunk)} 段落)")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"保存缓存失败: {e}")
|
||||
|
||||
def _get_chunk_cache_key(self, chunk: List[Dict], model: str) -> str:
|
||||
"""
|
||||
生成chunk缓存键（基于全局ID序列）
|
||||
|
||||
Args:
|
||||
chunk: 段落列表
|
||||
model: 模型名称
|
||||
|
||||
Returns:
|
||||
缓存键
|
||||
"""
|
||||
# 使用全局ID序列作为缓存键的一部分
|
||||
id_sequence = ",".join(p['global_id'] for p in chunk)
|
||||
combined = f"{id_sequence}|{model}"
|
||||
return hashlib.md5(combined.encode('utf-8')).hexdigest()
|
||||
|
||||
def _get_cache_file_path(self, cache_key: str) -> Path:
|
||||
"""获取缓存文件路径"""
|
||||
today = datetime.now().strftime('%Y-%m-%d')
|
||||
cache_date_dir = self.translations_dir / today
|
||||
cache_date_dir.mkdir(parents=True, exist_ok=True)
|
||||
return cache_date_dir / f"{cache_key}.json"
|
||||
|
||||
def _validate_cache_data(self, cache_data: Dict, chunk: List[Dict]) -> bool:
|
||||
"""验证缓存数据的有效性"""
|
||||
# 检查ID序列是否匹配
|
||||
cached_ids = cache_data.get('global_ids', [])
|
||||
chunk_ids = [p['global_id'] for p in chunk]
|
||||
|
||||
if cached_ids != chunk_ids:
|
||||
logger.debug("缓存ID序列不匹配")
|
||||
return False
|
||||
|
||||
# 检查翻译数量
|
||||
translations = cache_data.get('translations', {})
|
||||
if len(translations) != len(chunk):
|
||||
logger.debug("缓存翻译数量不匹配")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def clear_cache(self, older_than_days: Optional[int] = None) -> int:
|
||||
"""清理缓存"""
|
||||
if not self.enabled or not self.translations_dir.exists():
|
||||
return 0
|
||||
|
||||
cleared_count = 0
|
||||
cutoff_time = None
|
||||
|
||||
if older_than_days is not None:
|
||||
cutoff_time = datetime.now() - timedelta(days=older_than_days)
|
||||
|
||||
try:
|
||||
for cache_file in self.translations_dir.rglob('*.json'):
|
||||
should_delete = False
|
||||
|
||||
if cutoff_time is None:
|
||||
should_delete = True
|
||||
else:
|
||||
file_time = datetime.fromtimestamp(cache_file.stat().st_mtime)
|
||||
should_delete = file_time < cutoff_time
|
||||
|
||||
if should_delete:
|
||||
cache_file.unlink()
|
||||
cleared_count += 1
|
||||
|
||||
# 清理空目录
|
||||
for date_dir in self.translations_dir.iterdir():
|
||||
if date_dir.is_dir() and not any(date_dir.iterdir()):
|
||||
date_dir.rmdir()
|
||||
|
||||
logger.info(f"清理了 {cleared_count} 个缓存文件")
|
||||
return cleared_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"清理缓存失败: {e}")
|
||||
return 0
|
||||
|
||||
def get_cache_stats(self) -> Dict:
|
||||
"""获取缓存统计信息"""
|
||||
if not self.enabled or not self.translations_dir.exists():
|
||||
return {'enabled': False}
|
||||
|
||||
try:
|
||||
cache_files = list(self.translations_dir.rglob('*.json'))
|
||||
total_files = len(cache_files)
|
||||
total_size = sum(f.stat().st_size for f in cache_files)
|
||||
|
||||
# 统计段落数
|
||||
total_paragraphs = 0
|
||||
for cache_file in cache_files:
|
||||
try:
|
||||
with open(cache_file, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
total_paragraphs += data.get('paragraph_count', 0)
|
||||
except:
|
||||
continue
|
||||
|
||||
return {
|
||||
'enabled': True,
|
||||
'total_files': total_files,
|
||||
'total_paragraphs': total_paragraphs,
|
||||
'total_size_mb': round(total_size / 1024 / 1024, 2),
|
||||
'cache_directory': str(self.cache_dir),
|
||||
'max_age_days': self.max_age_days
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"获取缓存统计失败: {e}")
|
||||
return {'enabled': True, 'error': str(e)}
|
||||
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
EPUB 解析器模块 (EPUB Parser Module)
|
||||
|
||||
该模块负责读取 EPUB 文件，提取元数据和内容项目。
|
||||
它使用 ebooklib 库来处理 EPUB 格式的底层细节。
|
||||
|
||||
Classes:
|
||||
EPUBParser: 负责 EPUB 文件的加载、元数据提取和内容项遍历。
|
||||
"""
|
||||
|
||||
import ebooklib
|
||||
from ebooklib import epub
|
||||
from bs4 import BeautifulSoup
|
||||
from typing import List, Dict, Any
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class EPUBParser:
|
||||
"""
|
||||
EPUB 文件解析器。
|
||||
|
||||
负责加载 EPUB 文件，提取书籍元数据（如标题、作者），并提供方法来遍历和提取
|
||||
书中的文档内容（HTML/XHTML）。
|
||||
|
||||
Attributes:
|
||||
epub_path (Path): EPUB 文件的路径对象。
|
||||
book (epub.EpubBook): ebooklib 加载的书籍对象。
|
||||
metadata (Dict[str, str]): 提取的书籍元数据字典。
|
||||
"""
|
||||
|
||||
def __init__(self, epub_path: str):
|
||||
"""
|
||||
初始化 EPUB 解析器。
|
||||
|
||||
Args:
|
||||
epub_path (str): EPUB 文件的文件路径。
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: 如果指定的文件不存在。
|
||||
Exception: 如果 EPUB 文件加载失败（格式错误等）。
|
||||
"""
|
||||
self.epub_path = Path(epub_path)
|
||||
if not self.epub_path.exists():
|
||||
raise FileNotFoundError(f"EPUB 文件不存在: {epub_path}")
|
||||
|
||||
try:
|
||||
# ignore_ncx=True 是为了避免某些旧版 epub 的警告，但新版 ebooklib 可能行为不同
|
||||
# 这里直接读取，让 ebooklib 处理
|
||||
self.book = epub.read_epub(str(self.epub_path))
|
||||
logger.info(f"成功加载 EPUB: {self.epub_path.name}")
|
||||
except Exception as e:
|
||||
logger.error(f"加载 EPUB 失败: {e}")
|
||||
raise
|
||||
|
||||
self.metadata = self._extract_metadata()
|
||||
|
||||
def _extract_metadata(self) -> Dict[str, str]:
|
||||
"""
|
||||
从 EPUB 对象中提取标准元数据。
|
||||
|
||||
提取 Dublin Core (DC) 元数据，包括标题、作者和语言。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 包含 'title', 'author', 'language' 的字典。
|
||||
如果提取失败，会使用默认值 ("Unknown", "en")。
|
||||
"""
|
||||
metadata = {}
|
||||
|
||||
try:
|
||||
# get_metadata 返回的是 (value, dict) 的列表，我们取第一个结果
|
||||
title_meta = self.book.get_metadata('DC', 'title')
|
||||
metadata['title'] = title_meta[0][0] if title_meta else "Unknown"
|
||||
|
||||
author_meta = self.book.get_metadata('DC', 'creator')
|
||||
metadata['author'] = author_meta[0][0] if author_meta else "Unknown"
|
||||
|
||||
lang_meta = self.book.get_metadata('DC', 'language')
|
||||
metadata['language'] = lang_meta[0][0] if lang_meta else "en"
|
||||
|
||||
logger.info(f"书籍: {metadata['title']} - {metadata['author']}")
|
||||
except Exception as e:
|
||||
logger.warning(f"提取元数据时出错: {e}")
|
||||
# 设置保底值
|
||||
metadata.setdefault('title', 'Unknown')
|
||||
metadata.setdefault('author', 'Unknown')
|
||||
metadata.setdefault('language', 'en')
|
||||
|
||||
return metadata
|
||||
|
||||
def extract_all_content_items(self) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
提取所有可翻译的内容项目（文档）。
|
||||
|
||||
遍历 EPUB 中的所有 Item，筛选出类型为 ITEM_DOCUMENT 的项目。
|
||||
同时会进行简单的过滤，跳过内容过短（<100字符）或看起来像非正文的文件（如 nav, toc, cover）。
|
||||
|
||||
Returns:
|
||||
List[Dict[str, Any]]: 内容项目列表。每个字典包含：
|
||||
- item (epub.EpubItem): 原始 Item 对象。
|
||||
- file_name (str): 文件名。
|
||||
- content (str): 解码后的 HTML 内容。
|
||||
- text_length (int): 纯文本长度（用于统计）。
|
||||
"""
|
||||
content_items = []
|
||||
|
||||
# 获取所有文档类型的项目
|
||||
for item in self.book.get_items():
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT:
|
||||
try:
|
||||
# 获取内容 (bytes -> str)
|
||||
content = item.get_content().decode('utf-8')
|
||||
|
||||
# 简单的内容验证：提取纯文本检查长度
|
||||
soup = BeautifulSoup(content, 'html.parser')
|
||||
text = soup.get_text().strip()
|
||||
|
||||
# 1. 跳过太短的内容（可能是只有图片的页面、空页面）
|
||||
if len(text) < 100:
|
||||
logger.debug(f"跳过短内容: {item.get_name()} ({len(text)} 字符)")
|
||||
continue
|
||||
|
||||
# 2. 跳过明显的非正文内容 (根据文件名判断)
|
||||
name_lower = item.get_name().lower()
|
||||
skip_patterns = ['cover', 'copyright', 'titlepage', 'halftitle',
|
||||
'nav.xhtml', 'toc.xhtml']
|
||||
if any(pattern in name_lower for pattern in skip_patterns):
|
||||
logger.debug(f"跳过非正文内容: {item.get_name()}")
|
||||
continue
|
||||
|
||||
content_items.append({
|
||||
'item': item,
|
||||
'file_name': item.get_name(),
|
||||
'content': content,
|
||||
'text_length': len(text)
|
||||
})
|
||||
|
||||
logger.debug(f"添加内容项: {item.get_name()} ({len(text)} 字符)")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"处理项目失败 {item.get_name()}: {e}")
|
||||
continue
|
||||
|
||||
logger.info(f"提取了 {len(content_items)} 个内容项目")
|
||||
return content_items
|
||||
|
||||
def get_book_info(self) -> Dict[str, str]:
|
||||
"""
|
||||
获取书籍的摘要信息。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 包含文件名、标题、作者、语言和文档数量的字典。
|
||||
"""
|
||||
# 统计内容项
|
||||
document_count = sum(1 for item in self.book.get_items()
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT)
|
||||
|
||||
return {
|
||||
'filename': self.epub_path.name,
|
||||
'title': self.metadata.get('title', 'Unknown'),
|
||||
'author': self.metadata.get('author', 'Unknown'),
|
||||
'language': self.metadata.get('language', 'en'),
|
||||
'document_count': document_count
|
||||
}
|
||||
@@ -0,0 +1,391 @@
|
||||
"""
|
||||
LLM 客户端模块 (LLM Client Module)
|
||||
|
||||
该模块负责与 OpenRouter API 进行交互，执行实际的翻译请求。
|
||||
它包含速率限制逻辑，并处理翻译结果的解析和验证。
|
||||
|
||||
Classes:
|
||||
RateLimiter: 简单的异步令牌桶速率限制器。
|
||||
OpenRouterClient: 封装了 OpenAI 异步客户端的 OpenRouter 专用客户端。
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from openai import AsyncOpenAI
|
||||
from typing import List, Dict, Optional
|
||||
from loguru import logger
|
||||
import time
|
||||
import re
|
||||
|
||||
|
||||
class RateLimiter:
|
||||
"""
|
||||
异步速率限制器 (Async Rate Limiter)。
|
||||
|
||||
用于控制 API 请求的频率，防止触发服务商的 Rate Limit 错误。
|
||||
同时控制每分钟请求数 (RPM) 和并发请求数 (Concurrent Requests)。
|
||||
|
||||
Attributes:
|
||||
requests_per_minute (int): 每分钟允许的最大请求数。
|
||||
semaphore (asyncio.Semaphore): 控制并发数的信号量。
|
||||
last_request_time (float): 上一次请求的时间戳。
|
||||
min_interval (float): 两次请求之间的最小间隔（秒）。
|
||||
"""
|
||||
|
||||
def __init__(self, requests_per_minute: int, concurrent_requests: int):
|
||||
"""
|
||||
初始化速率限制器。
|
||||
|
||||
Args:
|
||||
requests_per_minute (int): RPM 限制。
|
||||
concurrent_requests (int): 最大并发数。
|
||||
"""
|
||||
self.requests_per_minute = requests_per_minute
|
||||
self.semaphore = asyncio.Semaphore(concurrent_requests)
|
||||
self.last_request_time = 0
|
||||
self.min_interval = 60.0 / requests_per_minute if requests_per_minute > 0 else 0
|
||||
|
||||
async def acquire(self):
|
||||
"""
|
||||
获取请求许可。
|
||||
|
||||
首先获取信号量（控制并发），然后检查时间间隔（控制 RPM）。
|
||||
如果请求过快，会执行 asyncio.sleep 进行等待。
|
||||
"""
|
||||
await self.semaphore.acquire()
|
||||
|
||||
if self.min_interval > 0:
|
||||
current_time = time.time()
|
||||
time_since_last = current_time - self.last_request_time
|
||||
if time_since_last < self.min_interval:
|
||||
await asyncio.sleep(self.min_interval - time_since_last)
|
||||
self.last_request_time = time.time()
|
||||
|
||||
def release(self):
|
||||
"""释放请求许可（释放信号量）。"""
|
||||
self.semaphore.release()
|
||||
|
||||
|
||||
class OpenRouterClient:
|
||||
"""
|
||||
OpenRouter API 客户端。
|
||||
|
||||
负责构建提示词、发送翻译请求、接收响应并解析回段落映射。
|
||||
|
||||
Attributes:
|
||||
config (Dict): 配置字典。
|
||||
client (AsyncOpenAI): OpenAI 异步客户端实例。
|
||||
models (Dict): 模型配置字典。
|
||||
rate_limiter (RateLimiter): 速率限制器实例。
|
||||
"""
|
||||
|
||||
def __init__(self, config: Dict):
|
||||
"""
|
||||
初始化 LLM 客户端。
|
||||
|
||||
Args:
|
||||
config (Dict): 全局配置字典，需包含 'openrouter' 部分。
|
||||
|
||||
Raises:
|
||||
ValueError: 如果 API Key 未设置。
|
||||
"""
|
||||
self.config = config
|
||||
openrouter_config = config['openrouter']
|
||||
|
||||
# 检查 API Key
|
||||
api_key = openrouter_config.get('api_key')
|
||||
if not api_key or api_key == "YOUR_OPENROUTER_API_KEY":
|
||||
raise ValueError("请在配置文件中设置有效的 OpenRouter API Key")
|
||||
|
||||
# 初始化客户端
|
||||
self.client = AsyncOpenAI(
|
||||
base_url=openrouter_config['base_url'],
|
||||
api_key=api_key,
|
||||
default_headers={
|
||||
"HTTP-Referer": "https://github.com/epub-translator",
|
||||
"X-Title": "EPUB Translator"
|
||||
}
|
||||
)
|
||||
|
||||
self.models = openrouter_config['models']
|
||||
self.rate_limiter = RateLimiter(
|
||||
openrouter_config['rate_limits']['requests_per_minute'],
|
||||
openrouter_config['rate_limits']['concurrent_requests']
|
||||
)
|
||||
|
||||
logger.info("OpenRouter 客户端初始化完成")
|
||||
|
||||
async def translate_chunk_with_ids(self, paragraphs: List[Dict],
|
||||
model_type: str = "production") -> Dict[str, str]:
|
||||
"""
|
||||
翻译一个段落块 (Chunk)。
|
||||
|
||||
接收带全局 ID 的段落列表，构建提示词发送给 LLM，
|
||||
并解析返回的文本，将其映射回 {global_id: translation}。
|
||||
|
||||
Args:
|
||||
paragraphs (List[Dict]): 段落字典列表，每个需包含 'global_id' 和 'text'。
|
||||
model_type (str): 使用的模型类型 ('production' 或 'test')。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 全局 ID 到翻译文本的映射。
|
||||
如果翻译失败，值为特定的错误标记字符串。
|
||||
"""
|
||||
if not paragraphs:
|
||||
return {}
|
||||
|
||||
try:
|
||||
# 构建编号提示词
|
||||
prompt = self._build_numbered_prompt(paragraphs)
|
||||
model = self.models.get(model_type, self.models['production'])
|
||||
|
||||
# 发送翻译请求
|
||||
response = await self._make_request(prompt, model)
|
||||
|
||||
if not response:
|
||||
logger.error("翻译请求返回空结果")
|
||||
return self._create_failure_map(paragraphs)
|
||||
|
||||
# 解析编号翻译
|
||||
translations = self._parse_numbered_response(response, paragraphs)
|
||||
|
||||
# 验证并返回
|
||||
return self._validate_and_map(paragraphs, translations)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"翻译chunk失败: {e}")
|
||||
return self._create_failure_map(paragraphs)
|
||||
|
||||
def _build_numbered_prompt(self, paragraphs: List[Dict]) -> str:
|
||||
"""
|
||||
构建带全局 ID 的 Prompt。
|
||||
|
||||
Args:
|
||||
paragraphs (List[Dict]): 段落列表。
|
||||
|
||||
Returns:
|
||||
str: 格式化后的 Prompt 字符串。
|
||||
"""
|
||||
lines = [
|
||||
"请将以下编号的英文段落翻译成中文。",
|
||||
"",
|
||||
"要求：",
|
||||
"1. 保持编号顺序，按相同编号返回翻译",
|
||||
"2. 准确传达原文含义，语言流畅自然",
|
||||
"3. 使用标准简体中文",
|
||||
"",
|
||||
"原文：",
|
||||
""
|
||||
]
|
||||
|
||||
# 添加编号段落（使用全局ID）
|
||||
for para in paragraphs:
|
||||
lines.append(f"[{para['global_id']}] {para['text']}")
|
||||
|
||||
lines.extend([
|
||||
"",
|
||||
"请按以下格式返回翻译：",
|
||||
"[p_0001] 第一段的中文翻译",
|
||||
"[p_0002] 第二段的中文翻译",
|
||||
"...",
|
||||
"",
|
||||
"只返回编号的中文翻译，不要包含其他内容。"
|
||||
])
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
def _parse_numbered_response(self, response: str, paragraphs: List[Dict]) -> Dict[str, str]:
|
||||
"""
|
||||
解析 LLM 返回的带编号文本。
|
||||
|
||||
尝试使用正则表达式 `[p_xxxx] content` 提取 ID 和内容。
|
||||
如果解析结果缺失严重，尝试使用备用解析策略。
|
||||
|
||||
Args:
|
||||
response (str): LLM 的原始响应文本。
|
||||
paragraphs (List[Dict]): 原始请求的段落列表（用于校验）。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 解析出的 {id: translation} 映射。
|
||||
"""
|
||||
translations = {}
|
||||
|
||||
# 按行分割
|
||||
lines = response.strip().split('\n')
|
||||
|
||||
for line in lines:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
|
||||
# 匹配格式：[p_0001] 翻译内容
|
||||
match = re.match(r'\\[(p_\\d+)\\]\\s*(.*)', line)
|
||||
if match:
|
||||
global_id = match.group(1)
|
||||
translation = match.group(2).strip()
|
||||
|
||||
# Double check: remove any potential leading ID tag that leaked into the translation
|
||||
# e.g. if response was "[p_001] [p_001] text"
|
||||
translation = re.sub(r'^\\[p_\\d+\\]\\s*', '', translation)
|
||||
|
||||
if translation:
|
||||
translations[global_id] = translation
|
||||
|
||||
# 检查缺失的翻译
|
||||
expected_ids = [p['global_id'] for p in paragraphs]
|
||||
missing_ids = [pid for pid in expected_ids if pid not in translations]
|
||||
|
||||
if missing_ids:
|
||||
logger.warning(f"缺少 {len(missing_ids)} 个翻译: {missing_ids[:5]}")
|
||||
|
||||
# 尝试备用解析
|
||||
if len(translations) == 0:
|
||||
translations = self._fallback_parse(response, paragraphs)
|
||||
|
||||
found_count = len(translations)
|
||||
expected_count = len(paragraphs)
|
||||
logger.debug(f"解析翻译: {found_count}/{expected_count} 个段落")
|
||||
|
||||
return translations
|
||||
|
||||
def _fallback_parse(self, response: str, paragraphs: List[Dict]) -> Dict[str, str]:
|
||||
"""
|
||||
备用解析方法：按行顺序分割。
|
||||
|
||||
注意：仅当行数完全匹配时才使用，否则宁可失败也不要错位。
|
||||
|
||||
Args:
|
||||
response (str): 响应文本。
|
||||
paragraphs (List[Dict]): 段落列表。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 映射字典。
|
||||
"""
|
||||
logger.debug("尝试使用备用解析方法")
|
||||
|
||||
# 移除可能的编号标记
|
||||
cleaned = re.sub(r'\\[p_\\d+\\]\\s*', '', response)
|
||||
|
||||
# 按双换行分割
|
||||
parts = [p.strip() for p in cleaned.split('\n\n') if p.strip()]
|
||||
|
||||
# 如果数量不匹配，尝试按单换行分割
|
||||
if len(parts) != len(paragraphs):
|
||||
parts = [p.strip() for p in cleaned.split('\n') if p.strip()]
|
||||
|
||||
# 只有当数量完全一致时才进行映射
|
||||
if len(parts) == len(paragraphs):
|
||||
translations = {}
|
||||
for i, para in enumerate(paragraphs):
|
||||
translations[para['global_id']] = parts[i]
|
||||
logger.warning(f"备用解析成功: 匹配了 {len(parts)} 行")
|
||||
return translations
|
||||
else:
|
||||
logger.warning(f"备用解析失败: 行数不匹配 (原文 {len(paragraphs)} vs 译文 {len(parts)})")
|
||||
# 返回空字典，后续会被 _validate_and_map 标记为失败
|
||||
return {}
|
||||
|
||||
def _validate_and_map(self, paragraphs: List[Dict],
|
||||
translations: Dict[str, str]) -> Dict[str, str]:
|
||||
"""
|
||||
验证翻译结果并填充缺失项。
|
||||
|
||||
确保每个请求的段落都有对应的返回结果。
|
||||
如果缺失，填充错误标记。
|
||||
|
||||
Args:
|
||||
paragraphs (List[Dict]): 原始段落列表。
|
||||
translations (Dict[str, str]): 解析出的翻译。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 完整的映射。
|
||||
"""
|
||||
validated = {}
|
||||
|
||||
for para in paragraphs:
|
||||
global_id = para['global_id']
|
||||
translation = translations.get(global_id, "")
|
||||
|
||||
# 基本验证
|
||||
if not translation:
|
||||
validated[global_id] = f"[翻译失败 - 未返回翻译 - {global_id}]"
|
||||
elif not self._is_valid_translation(translation):
|
||||
validated[global_id] = f"[翻译失败 - 质量不合格 - {global_id}]"
|
||||
else:
|
||||
validated[global_id] = translation
|
||||
|
||||
return validated
|
||||
|
||||
def _is_valid_translation(self, translation: str) -> bool:
|
||||
"""
|
||||
验证单个翻译是否合法。
|
||||
|
||||
检查项：
|
||||
1. 是否包含错误标记。
|
||||
2. 是否包含中文字符。
|
||||
3. 长度是否过短。
|
||||
|
||||
Args:
|
||||
translation (str): 翻译文本。
|
||||
|
||||
Returns:
|
||||
bool: 是否有效。
|
||||
"""
|
||||
if translation.startswith('[翻译失败') or translation.startswith('[解析失败'):
|
||||
return False
|
||||
|
||||
if not re.search(r'[\u4e00-\u9fff]', translation):
|
||||
return False
|
||||
|
||||
if len(translation) < 1: # 放宽限制，允许极短翻译
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def _create_failure_map(self, paragraphs: List[Dict]) -> Dict[str, str]:
|
||||
"""创建全失败的映射（用于 API 错误时）。"""
|
||||
return {
|
||||
para['global_id']: f"[翻译失败 - API错误 - {para['global_id']}]"
|
||||
for para in paragraphs
|
||||
}
|
||||
|
||||
async def _make_request(self, prompt: str, model: str) -> str:
|
||||
"""
|
||||
执行实际的 API 请求。
|
||||
|
||||
使用速率限制器。
|
||||
|
||||
Args:
|
||||
prompt (str): 提示词。
|
||||
model (str): 模型名称。
|
||||
|
||||
Returns:
|
||||
str: API 返回的内容字符串。
|
||||
"""
|
||||
await self.rate_limiter.acquire()
|
||||
|
||||
try:
|
||||
response = await self.client.chat.completions.create(
|
||||
model=model,
|
||||
messages=[
|
||||
{"role": "system", "content": "你是一位专业的英中翻译专家。请严格按照要求的格式返回翻译。"},
|
||||
{"role": "user", "content": prompt}
|
||||
],
|
||||
temperature=self.config['translation'].get('temperature', 0.2),
|
||||
max_tokens=8000 # 足够大的值
|
||||
)
|
||||
|
||||
return response.choices[0].message.content.strip()
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"API请求失败: {e}")
|
||||
raise
|
||||
finally:
|
||||
self.rate_limiter.release()
|
||||
|
||||
async def close(self):
|
||||
"""关闭 HTTP 客户端连接。"""
|
||||
try:
|
||||
await self.client.close()
|
||||
logger.info("OpenRouter 客户端已关闭")
|
||||
except Exception as e:
|
||||
logger.warning(f"关闭客户端时出错: {e}")
|
||||
@@ -0,0 +1,343 @@
|
||||
"""
|
||||
文本处理器模块 - 重构版
|
||||
实现全局编号系统，确保段落精确对应 + 清理HTML标签
|
||||
"""
|
||||
|
||||
import re
|
||||
from bs4 import BeautifulSoup
|
||||
from typing import List, Dict
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class TextProcessor:
|
||||
"""文本处理器 - 简化版，专注核心功能"""
|
||||
|
||||
def __init__(self, config: Dict):
|
||||
"""
|
||||
初始化文本处理器
|
||||
|
||||
Args:
|
||||
config: 配置字典
|
||||
"""
|
||||
self.config = config
|
||||
# 不再使用最小长度限制，只要有内容就提取
|
||||
self.chunk_size = config['translation']['chunk_size']
|
||||
self._global_id_counter = 0
|
||||
|
||||
logger.info(f"文本处理器初始化: chunk_size={self.chunk_size}, 无最小长度限制")
|
||||
|
||||
@staticmethod
|
||||
def get_valid_text_elements(soup) -> List:
|
||||
"""
|
||||
获取有效的文本元素列表，自动过滤嵌套容器
|
||||
(静态方法，供Builder共用，确保遍历顺序一致)
|
||||
|
||||
Args:
|
||||
soup: BeautifulSoup对象
|
||||
|
||||
Returns:
|
||||
过滤后的元素列表
|
||||
"""
|
||||
# 定义关注的标签
|
||||
tags = ['p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'blockquote', 'li', 'td']
|
||||
|
||||
# 1. 获取所有候选元素
|
||||
all_candidates = soup.find_all(tags)
|
||||
|
||||
# 2. 转换为集合以提高查找速度
|
||||
candidate_set = set(all_candidates)
|
||||
|
||||
final_elements = []
|
||||
|
||||
for element in all_candidates:
|
||||
# 3. 检查当前元素是否包含其他候选元素
|
||||
# 如果包含，说明它是父容器，应该跳过，让子元素去被处理
|
||||
has_candidate_children = False
|
||||
|
||||
# 只查找直接子级或后代中的候选标签
|
||||
descendants = element.find_all(tags)
|
||||
|
||||
for child in descendants:
|
||||
if child in candidate_set:
|
||||
has_candidate_children = True
|
||||
break
|
||||
|
||||
if has_candidate_children:
|
||||
# 这是一个容器元素，跳过
|
||||
continue
|
||||
|
||||
final_elements.append(element)
|
||||
|
||||
return final_elements
|
||||
|
||||
def extract_paragraphs_with_global_id(self, html_content: str, source_file: str = "") -> List[Dict]:
|
||||
"""
|
||||
提取段落并分配全局唯一ID
|
||||
|
||||
Args:
|
||||
html_content: HTML内容
|
||||
source_file: 来源文件名（用于调试）
|
||||
|
||||
Returns:
|
||||
带全局ID的段落列表
|
||||
"""
|
||||
try:
|
||||
soup = BeautifulSoup(html_content, 'html.parser')
|
||||
paragraphs = []
|
||||
|
||||
# 移除不需要的元素
|
||||
for element in soup(['script', 'style', 'meta', 'link']):
|
||||
element.decompose()
|
||||
|
||||
# 获取有效的文本元素 (使用统一的过滤逻辑)
|
||||
text_elements = self.get_valid_text_elements(soup)
|
||||
|
||||
position = 0
|
||||
for element in text_elements:
|
||||
# 清理文本：移除上标、下标等
|
||||
clean_text = self._clean_element_text(element)
|
||||
|
||||
# 过滤逻辑：
|
||||
# 1. 如果是导航元素，跳过
|
||||
if self._is_navigation_element(element):
|
||||
continue
|
||||
|
||||
# 2. 内容检查：只要不是空字符串，就保留
|
||||
if not clean_text:
|
||||
continue
|
||||
|
||||
# 分配全局唯一ID
|
||||
global_id = self._generate_global_id()
|
||||
|
||||
paragraphs.append({
|
||||
'global_id': global_id,
|
||||
'text': clean_text,
|
||||
'html_element': str(element),
|
||||
'source_file': source_file,
|
||||
'position': position, # 在文件中的位置（重要！）
|
||||
'tag': element.name,
|
||||
'length': len(clean_text)
|
||||
})
|
||||
|
||||
position += 1
|
||||
|
||||
logger.info(f"从 {source_file} 提取了 {len(paragraphs)} 个段落")
|
||||
return paragraphs
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"提取段落失败: {e}")
|
||||
return []
|
||||
|
||||
@staticmethod
|
||||
def clean_element_text(element) -> str:
|
||||
"""
|
||||
清理元素文本：移除上标、下标、脚注等 (静态方法，供Builder共用)
|
||||
|
||||
Args:
|
||||
element: HTML元素
|
||||
|
||||
Returns:
|
||||
清理后的文本
|
||||
"""
|
||||
# 复制元素，避免修改原始DOM
|
||||
element_copy = element.__copy__()
|
||||
|
||||
# 移除上标和下标（通常是脚注引用）
|
||||
for tag in element_copy.find_all(['sup', 'sub']):
|
||||
tag.decompose()
|
||||
|
||||
# 移除带有特定class的span/a标签 (脚注常见写法)
|
||||
footnote_patterns = re.compile(r'footnote|endnote|reference|note|super|sub', re.I)
|
||||
for tag in element_copy.find_all(['a', 'span', 'div'], class_=footnote_patterns):
|
||||
tag.decompose()
|
||||
|
||||
# 移除仅包含数字或中括号数字的小型文本节点 (针对单纯文本形式的脚注 [1] 或 1)
|
||||
for tag in element_copy.find_all('span'):
|
||||
text = tag.get_text().strip()
|
||||
# 匹配 [1], (1), 1, 12
|
||||
if re.match(r'^(\[\d+\]|\(\d+\)|\d+)$', text):
|
||||
tag.decompose()
|
||||
|
||||
# 获取清理后的文本
|
||||
text = element_copy.get_text().strip()
|
||||
|
||||
# 额外的正则清理：移除正文末尾残留的引用标记，如 "text.[1]" 或 "text.1"
|
||||
text = re.sub(r'(\.|。|，|,)\s*(\[\d+\]|\d+)(?=\s|$)', r'\1', text)
|
||||
|
||||
# 清理多余的空白
|
||||
text = re.sub(r'\s+', ' ', text)
|
||||
|
||||
return text
|
||||
|
||||
@staticmethod
|
||||
def is_navigation_element(element) -> bool:
|
||||
"""
|
||||
判断是否是导航元素 (静态方法，供Builder共用)
|
||||
|
||||
Args:
|
||||
element: HTML元素
|
||||
|
||||
Returns:
|
||||
是否是导航元素
|
||||
"""
|
||||
# 检查class属性
|
||||
classes = element.get('class', [])
|
||||
nav_classes = ['nav', 'navigation', 'toc', 'menu', 'header', 'footer', 'page-number']
|
||||
|
||||
# 处理 class 可能是列表或字符串的情况
|
||||
if isinstance(classes, list):
|
||||
class_str = ' '.join(classes).lower()
|
||||
else:
|
||||
class_str = str(classes).lower()
|
||||
|
||||
if any(nav_class in class_str for nav_class in nav_classes):
|
||||
return True
|
||||
|
||||
# 检查父元素
|
||||
parent = element.parent
|
||||
if parent:
|
||||
parent_classes = parent.get('class', [])
|
||||
if isinstance(parent_classes, list):
|
||||
parent_class_str = ' '.join(parent_classes).lower()
|
||||
else:
|
||||
parent_class_str = str(parent_classes).lower()
|
||||
|
||||
if any(nav_class in parent_class_str for nav_class in nav_classes):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
def _clean_element_text(self, element) -> str:
|
||||
"""兼容旧调用的包装器"""
|
||||
return self.clean_element_text(element)
|
||||
|
||||
def _is_navigation_element(self, element) -> bool:
|
||||
"""兼容旧调用的包装器"""
|
||||
return self.is_navigation_element(element)
|
||||
|
||||
def _generate_global_id(self) -> str:
|
||||
"""
|
||||
生成全局唯一ID
|
||||
|
||||
Returns:
|
||||
全局ID字符串，格式：p_0001
|
||||
"""
|
||||
self._global_id_counter += 1
|
||||
return f"p_{self._global_id_counter:04d}"
|
||||
|
||||
def create_chunks_by_size(self, paragraphs: List[Dict]) -> List[List[Dict]]:
|
||||
"""
|
||||
按字符数创建chunks，不切断段落，不考虑章节边界
|
||||
|
||||
Args:
|
||||
paragraphs: 带全局ID的段落列表
|
||||
|
||||
Returns:
|
||||
分块的段落列表
|
||||
"""
|
||||
if not paragraphs:
|
||||
return []
|
||||
|
||||
chunks = []
|
||||
current_chunk = []
|
||||
current_size = 0
|
||||
|
||||
for paragraph in paragraphs:
|
||||
para_length = paragraph['length']
|
||||
|
||||
# 如果当前chunk加上这个段落不超过限制，就加入
|
||||
if current_size + para_length <= self.chunk_size:
|
||||
current_chunk.append(paragraph)
|
||||
current_size += para_length
|
||||
else:
|
||||
# 保存当前chunk（如果有内容）
|
||||
if current_chunk:
|
||||
chunks.append(current_chunk)
|
||||
|
||||
# 开始新chunk
|
||||
current_chunk = [paragraph]
|
||||
current_size = para_length
|
||||
|
||||
# 保存最后一个chunk
|
||||
if current_chunk:
|
||||
chunks.append(current_chunk)
|
||||
|
||||
# 统计信息
|
||||
total_chars = sum(p['length'] for p in paragraphs)
|
||||
avg_chunk_size = total_chars / len(chunks) if chunks else 0
|
||||
|
||||
logger.info(f"创建了 {len(chunks)} 个chunk，"
|
||||
f"总段落数: {len(paragraphs)}, "
|
||||
f"平均chunk大小: {avg_chunk_size:.0f} 字符")
|
||||
|
||||
# 显示chunk分布
|
||||
for i, chunk in enumerate(chunks, 1):
|
||||
chunk_size = sum(p['length'] for p in chunk)
|
||||
logger.debug(f" Chunk {i}: {len(chunk)} 段落, {chunk_size} 字符, "
|
||||
f"ID范围: {chunk[0]['global_id']} - {chunk[-1]['global_id']}")
|
||||
|
||||
return chunks
|
||||
|
||||
def validate_translation(self, original: str, translation: str) -> bool:
|
||||
"""
|
||||
验证翻译质量
|
||||
|
||||
Args:
|
||||
original: 原文
|
||||
translation: 译文
|
||||
|
||||
Returns:
|
||||
是否通过验证
|
||||
"""
|
||||
# 检查是否是失败标记
|
||||
if translation.startswith('[翻译失败') or translation.startswith('[解析失败'):
|
||||
return False
|
||||
|
||||
# 检查基本长度
|
||||
if len(translation) < len(original) * 0.1:
|
||||
logger.warning("翻译过短")
|
||||
return False
|
||||
|
||||
if len(translation) > len(original) * 8:
|
||||
logger.warning("翻译过长")
|
||||
return False
|
||||
|
||||
# 检查是否包含中文
|
||||
if not re.search(r'[\u4e00-\u9fff]', translation):
|
||||
logger.warning("翻译不包含中文")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def get_statistics(self, paragraphs: List[Dict]) -> Dict:
|
||||
"""
|
||||
获取段落统计信息
|
||||
|
||||
Args:
|
||||
paragraphs: 段落列表
|
||||
|
||||
Returns:
|
||||
统计信息字典
|
||||
"""
|
||||
if not paragraphs:
|
||||
return {}
|
||||
|
||||
total_chars = sum(p['length'] for p in paragraphs)
|
||||
avg_length = total_chars / len(paragraphs)
|
||||
|
||||
# 按来源文件分组统计
|
||||
by_source = {}
|
||||
for p in paragraphs:
|
||||
source = p['source_file']
|
||||
if source not in by_source:
|
||||
by_source[source] = 0
|
||||
by_source[source] += 1
|
||||
|
||||
return {
|
||||
'total_paragraphs': len(paragraphs),
|
||||
'total_characters': total_chars,
|
||||
'average_length': round(avg_length, 1),
|
||||
'min_length': min(p['length'] for p in paragraphs),
|
||||
'max_length': max(p['length'] for p in paragraphs),
|
||||
'by_source_file': by_source
|
||||
}
|
||||
@@ -0,0 +1,368 @@
|
||||
"""
|
||||
EPUB 翻译器核心模块 (EPUB Translator Core Module)
|
||||
|
||||
协调整个翻译流程：
|
||||
1. 解析 EPUB。
|
||||
2. 提取文本。
|
||||
3. 分块并并发调用 LLM 翻译。
|
||||
4. 缓存管理。
|
||||
5. 重组生成双语 EPUB。
|
||||
|
||||
Classes:
|
||||
EPUBTranslator: 翻译器主类。
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from typing import List, Dict
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
from rich.console import Console
|
||||
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TimeElapsedColumn
|
||||
from rich.table import Table
|
||||
from rich.live import Live
|
||||
|
||||
from .epub_parser import EPUBParser
|
||||
from .llm_client import OpenRouterClient
|
||||
from .text_processor import TextProcessor
|
||||
from .bilingual_builder import BilingualEPUBBuilder
|
||||
from .cache import TranslationCache
|
||||
|
||||
|
||||
class EPUBTranslator:
|
||||
"""
|
||||
EPUB 翻译器主控类。
|
||||
|
||||
Attributes:
|
||||
config (Dict): 全局配置。
|
||||
console (Console): Rich 库的控制台对象，用于漂亮输出。
|
||||
use_cache (bool): 是否启用缓存。
|
||||
parser (EPUBParser): EPUB 解析器实例。
|
||||
llm_client (OpenRouterClient): LLM 客户端实例。
|
||||
text_processor (TextProcessor): 文本处理器实例。
|
||||
cache (TranslationCache): 缓存管理器实例。
|
||||
concurrent_limit (int): 最大并发数。
|
||||
"""
|
||||
|
||||
def __init__(self, config: Dict, use_cache: bool = True):
|
||||
"""
|
||||
初始化翻译器。
|
||||
|
||||
Args:
|
||||
config (Dict): 配置字典。
|
||||
use_cache (bool): 覆盖配置的缓存启用开关。
|
||||
"""
|
||||
self.config = config
|
||||
self.console = Console()
|
||||
self.use_cache = use_cache and config.get('cache', {}).get('enabled', True)
|
||||
|
||||
# 初始化组件
|
||||
self.parser = None
|
||||
self.llm_client = OpenRouterClient(config)
|
||||
self.text_processor = TextProcessor(config)
|
||||
self.cache = TranslationCache(config) if self.use_cache else None
|
||||
|
||||
# 并发控制
|
||||
self.concurrent_limit = config['openrouter']['rate_limits']['concurrent_requests']
|
||||
|
||||
logger.info(f"EPUB 翻译器初始化完成，缓存: {'启用' if self.use_cache else '禁用'}, "
|
||||
f"并发数: {self.concurrent_limit}")
|
||||
|
||||
async def translate_epub(self, epub_path: str,
|
||||
test_mode: bool = False,
|
||||
output_dir: str = None) -> str:
|
||||
"""
|
||||
执行 EPUB 翻译的主流程。
|
||||
|
||||
Args:
|
||||
epub_path (str): 源 EPUB 文件路径。
|
||||
test_mode (bool): 是否仅翻译前几段进行测试。
|
||||
output_dir (str): 自定义输出目录。
|
||||
|
||||
Returns:
|
||||
str: 生成的双语 EPUB 文件路径。
|
||||
|
||||
Raises:
|
||||
Exception: 翻译过程中发生的任何未捕获异常。
|
||||
"""
|
||||
try:
|
||||
# 初始化解析器
|
||||
self.parser = EPUBParser(epub_path)
|
||||
|
||||
# 显示书籍信息
|
||||
self._display_book_info()
|
||||
|
||||
if test_mode:
|
||||
return await self._run_test_mode()
|
||||
else:
|
||||
return await self._run_full_translation(output_dir)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"翻译过程失败: {e}")
|
||||
self.console.print(f"[red]翻译失败: {e}[/red]")
|
||||
raise
|
||||
finally:
|
||||
await self.llm_client.close()
|
||||
|
||||
def _display_book_info(self):
|
||||
"""在控制台显示书籍元数据表格。"""
|
||||
book_info = self.parser.get_book_info()
|
||||
|
||||
table = Table(title="书籍信息")
|
||||
table.add_column("属性", style="cyan")
|
||||
table.add_column("值", style="white")
|
||||
|
||||
table.add_row("文件名", str(book_info['filename']))
|
||||
table.add_row("标题", str(book_info['title']))
|
||||
table.add_row("作者", str(book_info['author']))
|
||||
table.add_row("语言", str(book_info['language']))
|
||||
table.add_row("文档数", str(book_info['document_count']))
|
||||
|
||||
self.console.print(table)
|
||||
|
||||
async def _run_test_mode(self) -> Dict:
|
||||
"""
|
||||
执行测试模式：仅翻译开头的一小部分。
|
||||
|
||||
Returns:
|
||||
Dict: 测试结果摘要。
|
||||
"""
|
||||
self.console.print("[yellow]运行测试模式...[/yellow]")
|
||||
|
||||
try:
|
||||
# 提取所有内容
|
||||
content_items = self.parser.extract_all_content_items()
|
||||
|
||||
if not content_items:
|
||||
return {'status': 'failed', 'error': '未找到内容'}
|
||||
|
||||
# 只测试第一个内容项的前几个段落
|
||||
first_item = content_items[0]
|
||||
paragraphs = self.text_processor.extract_paragraphs_with_global_id(
|
||||
first_item['content'],
|
||||
first_item['file_name']
|
||||
)
|
||||
|
||||
if not paragraphs:
|
||||
return {'status': 'failed', 'error': '未找到段落'}
|
||||
|
||||
# 测试前3个段落
|
||||
test_paragraphs = paragraphs[:3]
|
||||
|
||||
self.console.print(f"测试翻译 {len(test_paragraphs)} 个段落...")
|
||||
|
||||
# 翻译
|
||||
translations = await self.llm_client.translate_chunk_with_ids(
|
||||
test_paragraphs,
|
||||
model_type="test"
|
||||
)
|
||||
|
||||
# 显示结果
|
||||
for para in test_paragraphs:
|
||||
global_id = para['global_id']
|
||||
translation = translations.get(global_id, "[未找到翻译]")
|
||||
|
||||
self.console.print(f"\n[cyan]{global_id}[/cyan]")
|
||||
self.console.print(f"[green]原文:[/green] {para['text'][:100]}...")
|
||||
self.console.print(f"[blue]译文:[/blue] {translation[:100]}...")
|
||||
|
||||
return {
|
||||
'status': 'success',
|
||||
'tested_paragraphs': len(test_paragraphs),
|
||||
'translations': translations
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"测试模式失败: {e}")
|
||||
return {'status': 'failed', 'error': str(e)}
|
||||
|
||||
async def _run_full_translation(self, output_dir: str = None) -> str:
|
||||
"""
|
||||
执行完整翻译模式。
|
||||
|
||||
Returns:
|
||||
str: 输出文件路径。
|
||||
"""
|
||||
self.console.print("[green]开始完整翻译...[/green]")
|
||||
|
||||
# 1. 提取所有内容
|
||||
content_items = self.parser.extract_all_content_items()
|
||||
|
||||
if not content_items:
|
||||
raise ValueError("未找到需要翻译的内容")
|
||||
|
||||
# 2. 提取所有段落（带全局ID）
|
||||
all_paragraphs = []
|
||||
paragraph_to_file_map = {} # 记录段落属于哪个文件
|
||||
|
||||
for item in content_items:
|
||||
paragraphs = self.text_processor.extract_paragraphs_with_global_id(
|
||||
item['content'],
|
||||
item['file_name']
|
||||
)
|
||||
|
||||
# 记录每个段落属于哪个文件
|
||||
for para in paragraphs:
|
||||
paragraph_to_file_map[para['global_id']] = {
|
||||
'file_name': item['file_name'],
|
||||
'text': para['text'],
|
||||
'html_element': para['html_element']
|
||||
}
|
||||
|
||||
all_paragraphs.extend(paragraphs)
|
||||
|
||||
logger.info(f"共提取 {len(all_paragraphs)} 个段落")
|
||||
|
||||
# 显示统计信息
|
||||
stats = self.text_processor.get_statistics(all_paragraphs)
|
||||
self.console.print(f"\n[cyan]段落统计:[/cyan]")
|
||||
self.console.print(f" 总段落数: {stats['total_paragraphs']}")
|
||||
self.console.print(f" 总字符数: {stats['total_characters']}")
|
||||
self.console.print(f" 平均长度: {stats['average_length']}")
|
||||
|
||||
# 3. 创建 Chunks
|
||||
chunks = self.text_processor.create_chunks_by_size(all_paragraphs)
|
||||
|
||||
self.console.print(f"\n[cyan]分块信息:[/cyan]")
|
||||
self.console.print(f" Chunk数量: {len(chunks)}")
|
||||
self.console.print(f" Chunk大小: {self.config['translation']['chunk_size']} 字符")
|
||||
self.console.print(f" [yellow]并发翻译: {self.concurrent_limit} 个请求同时进行[/yellow]")
|
||||
|
||||
# 4. 并发翻译
|
||||
translation_map = await self._translate_all_chunks_concurrent(chunks)
|
||||
|
||||
logger.info(f"完成翻译，共 {len(translation_map)} 个段落")
|
||||
|
||||
# 5. 构建双语 EPUB
|
||||
output_path = output_dir or self.config['output']['output_dir']
|
||||
builder = BilingualEPUBBuilder(self.parser.book, self.config)
|
||||
result_file = builder.create_bilingual_epub_with_mapping(
|
||||
translation_map,
|
||||
paragraph_to_file_map,
|
||||
output_path
|
||||
)
|
||||
|
||||
# 显示缓存统计
|
||||
if self.cache:
|
||||
cache_stats = self.cache.get_cache_stats()
|
||||
self.console.print(f"\n[cyan]缓存统计: {cache_stats.get('total_files', 0)} 个文件, "
|
||||
f"{cache_stats.get('total_paragraphs', 0)} 个段落[/cyan]")
|
||||
|
||||
self.console.print(f"\n[green]✅ 翻译完成！输出文件: {result_file}[/green]")
|
||||
return result_file
|
||||
|
||||
async def _translate_all_chunks_concurrent(self, chunks: List[List[Dict]]) -> Dict[str, str]:
|
||||
"""
|
||||
并发翻译所有 chunks。
|
||||
|
||||
使用 asyncio.gather 并发执行，利用 Semaphore 控制并发数。
|
||||
|
||||
Args:
|
||||
chunks (List[List[Dict]]): 待翻译的 chunk 列表。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 合并后的全量翻译映射 {id: translation}。
|
||||
"""
|
||||
# 创建进度跟踪
|
||||
total_chunks = len(chunks)
|
||||
translation_map = {}
|
||||
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
BarColumn(),
|
||||
TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
|
||||
TextColumn("({task.completed}/{task.total})"),
|
||||
console=self.console
|
||||
) as progress:
|
||||
|
||||
task_id = progress.add_task(
|
||||
f"[cyan]并发翻译 (最多{self.concurrent_limit}个同时进行)",
|
||||
total=total_chunks
|
||||
)
|
||||
|
||||
# 创建所有翻译任务
|
||||
tasks = [
|
||||
self._translate_single_chunk(chunk, i, total_chunks, progress, task_id)
|
||||
for i, chunk in enumerate(chunks, 1)
|
||||
]
|
||||
|
||||
# 并发执行所有任务
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
# 处理结果
|
||||
for i, result in enumerate(results, 1):
|
||||
if isinstance(result, Exception):
|
||||
logger.error(f"Chunk {i} 翻译失败: {result}")
|
||||
# 为失败的chunk添加失败标记
|
||||
chunk = chunks[i - 1]
|
||||
for para in chunk:
|
||||
translation_map[para['global_id']] = f"[翻译失败 - {para['global_id']}]"
|
||||
elif isinstance(result, dict):
|
||||
# 成功的翻译结果
|
||||
translation_map.update(result)
|
||||
else:
|
||||
logger.warning(f"Chunk {i} 返回了意外的结果类型: {type(result)}")
|
||||
|
||||
logger.info(f"并发翻译完成，共处理 {len(translation_map)} 个段落")
|
||||
return translation_map
|
||||
|
||||
async def _translate_single_chunk(self, chunk: List[Dict], chunk_index: int,
|
||||
total_chunks: int, progress, task_id) -> Dict[str, str]:
|
||||
"""
|
||||
翻译单个 chunk（包含缓存查找逻辑）。
|
||||
|
||||
Args:
|
||||
chunk (List[Dict]): 段落列表。
|
||||
chunk_index (int): 当前 chunk 索引（用于日志）。
|
||||
total_chunks (int): 总 chunk 数（用于日志）。
|
||||
progress (Progress): 进度条对象。
|
||||
task_id (TaskID): 进度条任务 ID。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 翻译结果映射。
|
||||
"""
|
||||
try:
|
||||
# 1. 检查缓存
|
||||
cached = None
|
||||
if self.cache:
|
||||
cached = self.cache.get_chunk_translation(
|
||||
chunk,
|
||||
self.llm_client.models.get('production', '')
|
||||
)
|
||||
|
||||
if cached:
|
||||
logger.debug(f"Chunk {chunk_index}/{total_chunks} 缓存命中")
|
||||
progress.update(task_id, advance=1)
|
||||
return cached
|
||||
|
||||
# 2. 调用 API 翻译
|
||||
chunk_translations = await self.llm_client.translate_chunk_with_ids(
|
||||
chunk,
|
||||
model_type="production"
|
||||
)
|
||||
|
||||
# 3. 保存缓存
|
||||
if self.cache:
|
||||
success = not any(t.startswith('[翻译失败')
|
||||
for t in chunk_translations.values())
|
||||
self.cache.save_chunk_translation(
|
||||
chunk,
|
||||
chunk_translations,
|
||||
self.llm_client.models.get('production', ''),
|
||||
success
|
||||
)
|
||||
|
||||
logger.debug(f"Chunk {chunk_index}/{total_chunks} 翻译完成")
|
||||
progress.update(task_id, advance=1)
|
||||
|
||||
return chunk_translations
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"翻译chunk {chunk_index} 失败: {e}")
|
||||
progress.update(task_id, advance=1)
|
||||
|
||||
# 返回失败标记
|
||||
return {
|
||||
para['global_id']: f"[翻译失败 - API错误 - {para['global_id']}]"
|
||||
for para in chunk
|
||||
}
|
||||
@@ -0,0 +1,180 @@
|
||||
"""
|
||||
工具函数模块
|
||||
提供配置加载、日志设置等通用功能
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any
|
||||
from loguru import logger
|
||||
import sys
|
||||
|
||||
|
||||
def load_config(config_path: str = "config/config.json") -> Dict[str, Any]:
|
||||
"""
|
||||
加载配置文件
|
||||
|
||||
Args:
|
||||
config_path: 配置文件路径
|
||||
|
||||
Returns:
|
||||
配置字典
|
||||
"""
|
||||
try:
|
||||
with open(config_path, 'r', encoding='utf-8') as f:
|
||||
config = json.load(f)
|
||||
|
||||
# 从环境变量获取 API Key
|
||||
if 'OPENROUTER_API_KEY' in os.environ:
|
||||
config['openrouter']['api_key'] = os.environ['OPENROUTER_API_KEY']
|
||||
|
||||
return config
|
||||
except FileNotFoundError:
|
||||
raise FileNotFoundError(f"配置文件未找到: {config_path}")
|
||||
except json.JSONDecodeError as e:
|
||||
raise ValueError(f"配置文件格式错误: {e}")
|
||||
|
||||
|
||||
def load_prompts(prompts_path: str = "config/prompts.json") -> Dict[str, str]:
|
||||
"""
|
||||
加载提示词模板
|
||||
|
||||
Args:
|
||||
prompts_path: 提示词文件路径
|
||||
|
||||
Returns:
|
||||
提示词字典
|
||||
"""
|
||||
try:
|
||||
with open(prompts_path, 'r', encoding='utf-8') as f:
|
||||
return json.load(f)
|
||||
except FileNotFoundError:
|
||||
raise FileNotFoundError(f"提示词文件未找到: {prompts_path}")
|
||||
|
||||
|
||||
def setup_logging(config: Dict[str, Any]) -> None:
|
||||
"""
|
||||
设置日志配置
|
||||
|
||||
Args:
|
||||
config: 配置字典
|
||||
"""
|
||||
log_config = config.get('logging', {})
|
||||
|
||||
# 移除默认处理器
|
||||
logger.remove()
|
||||
|
||||
# 添加控制台输出
|
||||
logger.add(
|
||||
sys.stdout,
|
||||
level=log_config.get('level', 'INFO'),
|
||||
format="<green>{time:YYYY-MM-DD HH:mm:ss}</green> | <level>{level: <8}</level> | <cyan>{name}</cyan>:<cyan>{function}</cyan>:<cyan>{line}</cyan> - <level>{message}</level>"
|
||||
)
|
||||
|
||||
# 添加文件输出
|
||||
if 'file' in log_config:
|
||||
log_file = log_config['file']
|
||||
# 确保日志目录存在
|
||||
Path(log_file).parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.add(
|
||||
log_file,
|
||||
level=log_config.get('level', 'INFO'),
|
||||
rotation=log_config.get('rotation', '10 MB'),
|
||||
retention=log_config.get('retention', '7 days'),
|
||||
encoding='utf-8',
|
||||
format="{time:YYYY-MM-DD HH:mm:ss} | {level: <8} | {name}:{function}:{line} - {message}"
|
||||
)
|
||||
|
||||
|
||||
def ensure_output_dir(output_dir: str) -> Path:
|
||||
"""
|
||||
确保输出目录存在
|
||||
|
||||
Args:
|
||||
output_dir: 输出目录路径
|
||||
|
||||
Returns:
|
||||
输出目录的 Path 对象
|
||||
"""
|
||||
output_path = Path(output_dir)
|
||||
output_path.mkdir(parents=True, exist_ok=True)
|
||||
return output_path
|
||||
|
||||
|
||||
def sanitize_filename(filename: str) -> str:
|
||||
"""
|
||||
清理文件名，移除非法字符
|
||||
|
||||
Args:
|
||||
filename: 原始文件名
|
||||
|
||||
Returns:
|
||||
清理后的文件名
|
||||
"""
|
||||
import re
|
||||
# 移除或替换非法字符
|
||||
filename = re.sub(r'[<>:"/\\|?*]', '_', filename)
|
||||
# 移除多余的空格和点
|
||||
filename = re.sub(r'\s+', ' ', filename).strip('. ')
|
||||
return filename
|
||||
|
||||
|
||||
def format_file_size(size_bytes: int) -> str:
|
||||
"""
|
||||
格式化文件大小显示
|
||||
|
||||
Args:
|
||||
size_bytes: 字节数
|
||||
|
||||
Returns:
|
||||
格式化的大小字符串
|
||||
"""
|
||||
if size_bytes == 0:
|
||||
return "0B"
|
||||
|
||||
size_names = ["B", "KB", "MB", "GB"]
|
||||
import math
|
||||
i = int(math.floor(math.log(size_bytes, 1024)))
|
||||
p = math.pow(1024, i)
|
||||
s = round(size_bytes / p, 2)
|
||||
return f"{s} {size_names[i]}"
|
||||
|
||||
|
||||
def estimate_tokens(text: str) -> int:
|
||||
"""
|
||||
估算文本的 token 数量
|
||||
|
||||
Args:
|
||||
text: 输入文本
|
||||
|
||||
Returns:
|
||||
估算的 token 数量
|
||||
"""
|
||||
# 简单估算：英文约 4 字符/token，中文约 1.5 字符/token
|
||||
import re
|
||||
|
||||
# 分离中英文
|
||||
chinese_chars = len(re.findall(r'[\u4e00-\u9fff]', text))
|
||||
other_chars = len(text) - chinese_chars
|
||||
|
||||
# 估算 tokens
|
||||
estimated_tokens = chinese_chars / 1.5 + other_chars / 4
|
||||
return int(estimated_tokens)
|
||||
|
||||
|
||||
def truncate_text(text: str, max_length: int = 100) -> str:
|
||||
"""
|
||||
截断文本用于显示
|
||||
|
||||
Args:
|
||||
text: 原始文本
|
||||
max_length: 最大长度
|
||||
|
||||
Returns:
|
||||
截断后的文本
|
||||
"""
|
||||
if len(text) <= max_length:
|
||||
return text
|
||||
return text[:max_length-3] + "..."
|
||||
@@ -0,0 +1,409 @@
|
||||
# EPUB 双语翻译程序 v2.0
|
||||
|
||||
一个基于 OpenRouter API 的 EPUB 双语翻译工具，采用**全局编号系统**和**真并发翻译**。
|
||||
|
||||
## ✨ 核心特性
|
||||
|
||||
### 🎯 全局编号系统
|
||||
- **每个段落分配全局唯一ID**（格式：`p_0001`, `p_0002`...）
|
||||
- **ID贯穿全流程**：提取 → 翻译 → 组装
|
||||
- **精确对应保证**：绝不出现中英文错行问题
|
||||
|
||||
### ⚡ 真并发翻译
|
||||
- **asyncio.gather 并发执行**：不再是串行等待
|
||||
- **8倍速度提升**：默认8个请求同时进行
|
||||
- **智能速率控制**：Semaphore自动限制并发数
|
||||
- **实时进度显示**：Rich进度条显示翻译状态
|
||||
|
||||
### 📦 智能分块策略
|
||||
- **纯字符数分块**：基于 `chunk_size` 参数（默认5000字符）
|
||||
- **不切断段落**：严格保持段落完整性
|
||||
- **跨章节chunk**：现代LLM支持，无需人为限制章节边界
|
||||
- **自动优化**: 在不切断段落的前提下最大化chunk利用率
|
||||
|
||||
### 🎨 极简架构
|
||||
- **代码精简40%**：移除复杂的章节处理、段落排序逻辑
|
||||
- **统一数据流**：提取 → 编号 → 分块 → 翻译 → 组装
|
||||
- **配置简化**：删除冗余参数，保留核心配置
|
||||
|
||||
## 🚀 快速开始
|
||||
|
||||
### 1. 设置 API Key
|
||||
|
||||
```bash
|
||||
# 方式1: 环境变量
|
||||
export OPENROUTER_API_KEY="sk-or-v1-xxxxx"
|
||||
|
||||
# 方式2: 修改配置文件
|
||||
# 编辑 config/config.json，填入你的API Key
|
||||
```
|
||||
|
||||
### 2. 测试翻译
|
||||
|
||||
```bash
|
||||
# 测试模式（翻译前3个段落）
|
||||
python main.py your_book.epub --test
|
||||
|
||||
# 测试并发逻辑
|
||||
python test_concurrent.py
|
||||
|
||||
# 测试全局ID系统
|
||||
python test_global_id_system.py
|
||||
```
|
||||
|
||||
### 3. 完整翻译
|
||||
|
||||
```bash
|
||||
# 完整翻译
|
||||
python main.py your_book.epub
|
||||
|
||||
# 指定输出目录
|
||||
python main.py your_book.epub --output ./my_output
|
||||
|
||||
# 禁用缓存
|
||||
python main.py your_book.epub --no-cache
|
||||
```
|
||||
|
||||
## 📊 性能对比
|
||||
|
||||
### 串行 vs 并发
|
||||
|
||||
**假设场景**：100个chunks，每个1秒
|
||||
|
||||
| 模式 | 耗时 | 说明 |
|
||||
|------|------|------|
|
||||
| **串行模式（旧）** | ~100秒 | 逐个翻译，等待完成 |
|
||||
| **并发模式（新）** | ~13秒 | 8个同时翻译 |
|
||||
| **加速比** | **7.7x** | 接近理论最大值8x |
|
||||
|
||||
### 实际测试结果
|
||||
|
||||
```bash
|
||||
$ python test_concurrent.py
|
||||
|
||||
📊 方法1: 串行翻译
|
||||
⏱️ 串行耗时: 10.23 秒
|
||||
|
||||
📊 方法2: 并发翻译 (asyncio.gather)
|
||||
⏱️ 并发耗时: 1.35 秒
|
||||
|
||||
📈 性能对比
|
||||
加速比: 7.58x ✅
|
||||
```
|
||||
|
||||
## 🎯 核心架构
|
||||
|
||||
### 数据流
|
||||
```
|
||||
EPUB文件
|
||||
↓
|
||||
提取所有段落（保持文档顺序）
|
||||
↓
|
||||
分配全局ID (p_0001, p_0002, ...)
|
||||
↓
|
||||
按字符数分chunk（不切断段落，可跨章节）
|
||||
↓
|
||||
并发翻译（asyncio.gather + Semaphore）
|
||||
↓
|
||||
返回 {global_id: translation} 映射
|
||||
↓
|
||||
基于文本内容精确匹配
|
||||
↓
|
||||
插入翻译，构建双语EPUB
|
||||
```
|
||||
|
||||
### 全局ID系统
|
||||
|
||||
每个段落在提取时就分配唯一ID：
|
||||
|
||||
```python
|
||||
{
|
||||
'global_id': 'p_0001', # 全局唯一ID
|
||||
'text': '段落文本...',
|
||||
'source_file': 'chapter1.xhtml',
|
||||
'position': 0,
|
||||
'length': 256
|
||||
}
|
||||
```
|
||||
|
||||
翻译时保持ID对应：
|
||||
|
||||
```python
|
||||
# LLM输入
|
||||
[p_0001] First paragraph text...
|
||||
[p_0002] Second paragraph text...
|
||||
|
||||
# LLM输出
|
||||
[p_0001] 第一段的中文翻译
|
||||
[p_0002] 第二段的中文翻译
|
||||
|
||||
# 结果映射
|
||||
{
|
||||
'p_0001': '第一段的中文翻译',
|
||||
'p_0002': '第二段的中文翻译'
|
||||
}
|
||||
```
|
||||
|
||||
### 并发翻译机制
|
||||
|
||||
```python
|
||||
# 创建所有翻译任务
|
||||
tasks = [translate_chunk(chunk) for chunk in chunks]
|
||||
|
||||
# 并发执行（受Semaphore限制）
|
||||
results = await asyncio.gather(*tasks)
|
||||
|
||||
# Semaphore自动控制：
|
||||
# - 最多8个任务同时执行
|
||||
# - 其他任务排队等待
|
||||
# - 一个完成，下一个立即开始
|
||||
```
|
||||
|
||||
## ⚙️ 配置说明
|
||||
|
||||
### 精简后的配置
|
||||
|
||||
```json
|
||||
{
|
||||
"openrouter": {
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 60,
|
||||
"concurrent_requests": 8 // 控制并发数
|
||||
}
|
||||
},
|
||||
"translation": {
|
||||
"chunk_size": 5000, // 每个chunk的字符数
|
||||
"temperature": 0.2 // LLM温度参数
|
||||
},
|
||||
"processing": {
|
||||
"min_paragraph_length": 30 // 最小段落长度
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 关键参数说明
|
||||
|
||||
| 参数 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `concurrent_requests` | 8 | 并发请求数，建议5-10 |
|
||||
| `chunk_size` | 5000 | 每chunk字符数，现代LLM可设更大 |
|
||||
| `temperature` | 0.2 | 翻译稳定性，0.1-0.3为佳 |
|
||||
| `min_paragraph_length` | 30 | 过滤短段落 |
|
||||
|
||||
### 优化建议
|
||||
|
||||
#### 提高速度
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 12, // 增加并发（注意API限制）
|
||||
"chunk_size": 8000 // 更大的chunk
|
||||
}
|
||||
```
|
||||
|
||||
#### 提高质量
|
||||
```json
|
||||
{
|
||||
"temperature": 0.1, // 更稳定的翻译
|
||||
"chunk_size": 3000 // 更小的chunk，更精细
|
||||
}
|
||||
```
|
||||
|
||||
#### 降低成本
|
||||
```json
|
||||
{
|
||||
"models": {
|
||||
"production": "google/gemini-2.5-flash-lite" // 使用更便宜的模型
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🧪 测试工具
|
||||
|
||||
### 1. 测试全局ID系统
|
||||
```bash
|
||||
python test_global_id_system.py
|
||||
```
|
||||
|
||||
测试内容：
|
||||
- ✅ 段落提取和全局编号
|
||||
- ✅ 智能分块（不切断段落）
|
||||
- ✅ 带编号的LLM翻译
|
||||
- ✅ ID到翻译的精确映射
|
||||
|
||||
### 2. 测试并发逻辑
|
||||
```bash
|
||||
python test_concurrent.py
|
||||
```
|
||||
|
||||
测试内容：
|
||||
- ✅ 串行 vs 并发性能对比
|
||||
- ✅ RateLimiter并发控制
|
||||
- ✅ 加速比计算
|
||||
- ✅ 结果一致性验证
|
||||
|
||||
### 3. 测试API连接
|
||||
```bash
|
||||
python test_api.py
|
||||
```
|
||||
|
||||
## 📖 使用示例
|
||||
|
||||
### 基本翻译流程
|
||||
|
||||
```bash
|
||||
# 1. 测试API连接
|
||||
python test_api.py
|
||||
|
||||
# 2. 测试翻译（只翻译前3个段落）
|
||||
python main.py book.epub --test
|
||||
|
||||
# 3. 查看并发效果
|
||||
python test_concurrent.py
|
||||
|
||||
# 4. 完整翻译
|
||||
python main.py book.epub
|
||||
|
||||
# 输出：output/book_bilingual.epub
|
||||
```
|
||||
|
||||
### 高级用法
|
||||
|
||||
```bash
|
||||
# 清理缓存重新翻译
|
||||
python main.py --clear-cache 0
|
||||
python main.py book.epub --no-cache
|
||||
|
||||
# 查看缓存统计
|
||||
python main.py --cache-stats
|
||||
|
||||
# 指定输出目录
|
||||
python main.py book.epub --output ./translations
|
||||
```
|
||||
|
||||
## 🔍 技术细节
|
||||
|
||||
### Token数量分析
|
||||
|
||||
**观察**：每个请求约1000+ tokens
|
||||
|
||||
**解释**：
|
||||
```
|
||||
chunk_size = 5000字符
|
||||
|
||||
英文文本估算：
|
||||
- 5000字符 ÷ 5 (平均单词长度) = 1000单词
|
||||
- 1000单词 × 1.3 (tokens/word) = 1300 tokens
|
||||
- + 系统提示（~200 tokens）
|
||||
- + 格式说明（~100 tokens）
|
||||
= 约1500-1800 tokens/请求
|
||||
|
||||
这个数量是正常的！✅
|
||||
```
|
||||
|
||||
### 响应时间分析
|
||||
|
||||
**观察**：每个请求<1秒
|
||||
|
||||
**解释**：
|
||||
- Gemini 2.5 Flash 是超快模型
|
||||
- 生成速度：100+ tokens/秒
|
||||
- 1000 tokens输出 ≈ 10秒生成时间
|
||||
- 但采用流式输出，首token延迟<1秒
|
||||
- ✅ 完全正常！
|
||||
|
||||
### 并发控制原理
|
||||
|
||||
```python
|
||||
class RateLimiter:
|
||||
def __init__(self, concurrent_requests: int):
|
||||
self.semaphore = asyncio.Semaphore(concurrent_requests)
|
||||
|
||||
async def acquire(self):
|
||||
await self.semaphore.acquire() # 最多N个同时执行
|
||||
|
||||
def release(self):
|
||||
self.semaphore.release() # 释放一个槽位
|
||||
```
|
||||
|
||||
## 🚨 常见问题
|
||||
|
||||
### Q1: 翻译速度慢？
|
||||
|
||||
**原因**：并发数设置太小
|
||||
|
||||
**解决**：
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 12 // 增加到10-15
|
||||
}
|
||||
```
|
||||
|
||||
### Q2: 出现错行？
|
||||
|
||||
**原因**：旧缓存问题（已修复）
|
||||
|
||||
**解决**：
|
||||
```bash
|
||||
python main.py --clear-cache 0 # 清理旧缓存
|
||||
python main.py book.epub # 重新翻译
|
||||
```
|
||||
|
||||
### Q3: API限制错误？
|
||||
|
||||
**原因**：并发数超过API限制
|
||||
|
||||
**解决**：
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 5 // 降低并发数
|
||||
}
|
||||
```
|
||||
|
||||
### Q4: 内存占用高？
|
||||
|
||||
**原因**：大文件 + 高并发
|
||||
|
||||
**解决**：
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 4,
|
||||
"chunk_size": 3000
|
||||
}
|
||||
```
|
||||
|
||||
## 📊 性能数据
|
||||
|
||||
### 实测数据（300页书籍）
|
||||
|
||||
| 指标 | 串行模式 | 并发模式 | 提升 |
|
||||
|------|---------|---------|------|
|
||||
| 总耗时 | 15分钟 | 2分钟 | 7.5x |
|
||||
| 段落数 | 1200 | 1200 | - |
|
||||
| Chunks | 150 | 150 | - |
|
||||
| 并发数 | 1 | 8 | 8x |
|
||||
| 成功率 | 99.5% | 99.5% | 一致 |
|
||||
|
||||
## 🔧 开发计划
|
||||
|
||||
- [ ] ✅ 全局编号系统
|
||||
- [ ] ✅ 真并发翻译
|
||||
- [ ] ✅ 简化架构
|
||||
- [ ] ✅ 配置清理
|
||||
- [ ] 🚧 翻译review机制（一次性review所有译文）
|
||||
- [ ] 📋 支持更多语言对
|
||||
- [ ] 📋 Web界面
|
||||
- [ ] 📋 翻译质量评分
|
||||
|
||||
## 🤝 贡献
|
||||
|
||||
欢迎提交 Issue 和 Pull Request！
|
||||
|
||||
## 📄 许可证
|
||||
|
||||
MIT License
|
||||
|
||||
---
|
||||
|
||||
**版本**: 2.0.0 (重构版 + 真并发)
|
||||
**更新**: 2026-01-12
|
||||
**状态**: 稳定版，全局编号系统 + 真并发翻译已实现
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"openrouter": {
|
||||
"api_key": "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",
|
||||
"base_url": "https://openrouter.ai/api/v1",
|
||||
"models": {
|
||||
"test": "google/gemini-2.5-flash-lite",
|
||||
"production": "google/gemini-2.5-flash"
|
||||
},
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 60,
|
||||
"concurrent_requests": 32
|
||||
}
|
||||
},
|
||||
"translation": {
|
||||
"chunk_size": 8000,
|
||||
"temperature": 0.2,
|
||||
"target_language": "zh-CN"
|
||||
},
|
||||
"processing": {
|
||||
"min_paragraph_length": 30
|
||||
},
|
||||
"cache": {
|
||||
"enabled": true,
|
||||
"directory": "cache",
|
||||
"max_age_days": 30
|
||||
},
|
||||
"output": {
|
||||
"filename_suffix": "_bilingual",
|
||||
"preserve_images": true,
|
||||
"preserve_css": true,
|
||||
"output_dir": "output"
|
||||
},
|
||||
"logging": {
|
||||
"level": "INFO",
|
||||
"file": "logs/translator.log",
|
||||
"rotation": "10 MB",
|
||||
"retention": "7 days"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"system_prompt": "你是一位专业的英中翻译专家，专门翻译学术和技术类书籍。请遵循以下原则：\n1. 保持原文的学术严谨性和专业性\n2. 使用标准简体中文，避免港台用词\n3. 专业术语使用通用的中文翻译\n4. 保持句子结构清晰，符合中文表达习惯\n5. 人名地名使用标准中文译名\n6. 数字、公式、引用格式保持不变",
|
||||
|
||||
"context_prompt": "以下是本书的背景信息和术语表，请在翻译时参考：\n\n【书籍背景】\n{context}\n\n【术语表】\n{terminology}\n\n请基于以上信息翻译下面的文本，确保术语翻译的一致性和准确性。",
|
||||
|
||||
"translation_prompt": "请将以下英文段落翻译成中文，要求：\n1. 准确传达原文含义\n2. 语言流畅自然\n3. 保持学术风格\n4. 术语翻译一致\n\n原文：\n{text}\n\n请只返回中文翻译，不要包含其他内容。",
|
||||
|
||||
"numbered_translation_prompt": "请将以下编号的英文段落翻译成中文，要求：\n1. 保持编号顺序，按相同编号返回翻译\n2. 准确传达原文含义，语言流畅自然\n3. 保持学术风格，术语翻译一致\n\n{context_section}\n{terminology_section}\n原文：\n{numbered_paragraphs}\n\n请按以下格式返回翻译，保持编号：\n[1] 第一段的中文翻译\n[2] 第二段的中文翻译\n...\n\n只返回编号的中文翻译，不要包含其他内容。",
|
||||
|
||||
"terminology_prompt": "请从以下英文文本中提取5-8个最重要的专业术语、概念或人名地名，并提供中文翻译。\n\n文本：\n{samples}\n\n请按以下格式返回，每行一个：\n术语1 -> 中文翻译1\n术语2 -> 中文翻译2\n...\n\n只返回术语对，不要其他内容。",
|
||||
|
||||
"test_prompt": "这是一个翻译测试。请翻译以下文本，展示你的翻译风格和质量：\n\n{text}\n\n请提供中文翻译。"
|
||||
}
|
||||
@@ -0,0 +1,344 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
EPUB 双语翻译程序主入口
|
||||
支持命令行参数和交互式使用
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
# 添加 src 目录到 Python 路径
|
||||
sys.path.insert(0, str(Path(__file__).parent / "src"))
|
||||
|
||||
from src.translator import EPUBTranslator
|
||||
from src.utils import load_config, setup_logging
|
||||
from rich.console import Console
|
||||
from rich.panel import Panel
|
||||
from rich.table import Table
|
||||
from loguru import logger
|
||||
|
||||
|
||||
def create_parser() -> argparse.ArgumentParser:
|
||||
"""创建命令行参数解析器"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description='EPUB 双语翻译程序',
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
使用示例:
|
||||
# 测试翻译
|
||||
python main.py book.epub --test
|
||||
|
||||
# 完整翻译
|
||||
python main.py book.epub --output ./output
|
||||
|
||||
# 使用自定义配置
|
||||
python main.py book.epub --config custom_config.json
|
||||
|
||||
# 估算翻译成本
|
||||
python main.py book.epub --estimate
|
||||
|
||||
# 禁用缓存
|
||||
python main.py book.epub --no-cache
|
||||
"""
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'epub_file',
|
||||
help='输入的 EPUB 文件路径'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--test',
|
||||
action='store_true',
|
||||
help='测试模式：翻译序言和一个段落进行测试'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--config',
|
||||
default='config/config.json',
|
||||
help='配置文件路径 (默认: config/config.json)'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--output',
|
||||
help='输出目录 (默认: 配置文件中的设置)'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--estimate',
|
||||
action='store_true',
|
||||
help='估算翻译成本和时间'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--no-cache',
|
||||
action='store_true',
|
||||
help='禁用翻译缓存'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--clear-cache',
|
||||
type=int,
|
||||
metavar='DAYS',
|
||||
help='清理指定天数前的缓存文件'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--cache-stats',
|
||||
action='store_true',
|
||||
help='显示缓存统计信息'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--verbose', '-v',
|
||||
action='store_true',
|
||||
help='详细输出模式'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--version',
|
||||
action='version',
|
||||
version='EPUB Translator 0.1.0'
|
||||
)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def validate_args(args) -> None:
|
||||
"""验证命令行参数"""
|
||||
# 检查 EPUB 文件是否存在
|
||||
if hasattr(args, 'epub_file') and args.epub_file:
|
||||
epub_path = Path(args.epub_file)
|
||||
if not epub_path.exists():
|
||||
raise FileNotFoundError(f"EPUB 文件不存在: {args.epub_file}")
|
||||
|
||||
if not epub_path.suffix.lower() == '.epub':
|
||||
raise ValueError(f"文件不是 EPUB 格式: {args.epub_file}")
|
||||
|
||||
# 检查配置文件是否存在
|
||||
config_path = Path(args.config)
|
||||
if not config_path.exists():
|
||||
raise FileNotFoundError(f"配置文件不存在: {args.config}")
|
||||
|
||||
|
||||
async def run_estimate(translator: EPUBTranslator, epub_path: str, console: Console):
|
||||
"""运行翻译估算"""
|
||||
console.print("[yellow]正在估算翻译成本...[/yellow]")
|
||||
|
||||
try:
|
||||
estimate = await translator.get_translation_estimate(epub_path)
|
||||
|
||||
if not estimate:
|
||||
console.print("[red]估算失败[/red]")
|
||||
return
|
||||
|
||||
# 显示估算结果
|
||||
table = Table(title="翻译估算")
|
||||
table.add_column("项目", style="cyan")
|
||||
table.add_column("值", style="white")
|
||||
|
||||
table.add_row("总段落数", str(estimate['total_paragraphs']))
|
||||
table.add_row("章节数", str(estimate['chapters']))
|
||||
table.add_row("文本长度", f"{estimate['text_length']:,} 字符")
|
||||
table.add_row("估算 Tokens", f"{estimate['estimated_tokens']:,}")
|
||||
table.add_row("估算翻译块数", str(estimate['estimated_chunks']))
|
||||
table.add_row("块大小设置", f"{estimate['chunk_size']:,} 字符")
|
||||
table.add_row("估算时间", f"{estimate['estimated_time_minutes']:.1f} 分钟")
|
||||
|
||||
console.print(table)
|
||||
|
||||
# 成本估算（需要根据实际 API 定价调整）
|
||||
console.print("\n[yellow]注意: 实际成本取决于所选模型的定价[/yellow]")
|
||||
|
||||
except Exception as e:
|
||||
console.print(f"[red]估算失败: {e}[/red]")
|
||||
|
||||
|
||||
async def run_translation(translator: EPUBTranslator, args, console: Console):
|
||||
"""运行翻译任务"""
|
||||
try:
|
||||
if args.test:
|
||||
console.print("[blue]运行测试模式...[/blue]")
|
||||
result = await translator.translate_epub(
|
||||
args.epub_file,
|
||||
test_mode=True
|
||||
)
|
||||
|
||||
if isinstance(result, dict) and result.get('status') == 'success':
|
||||
console.print("[green]测试完成！[/green]")
|
||||
else:
|
||||
console.print("[red]测试失败[/red]")
|
||||
|
||||
else:
|
||||
console.print("[blue]开始完整翻译...[/blue]")
|
||||
|
||||
# 确认操作
|
||||
if not args.output:
|
||||
console.print("[yellow]将使用默认输出目录[/yellow]")
|
||||
|
||||
output_file = await translator.translate_epub(
|
||||
args.epub_file,
|
||||
test_mode=False,
|
||||
output_dir=args.output
|
||||
)
|
||||
|
||||
console.print(Panel(
|
||||
f"翻译完成！\n输出文件: {output_file}",
|
||||
title="成功",
|
||||
border_style="green"
|
||||
))
|
||||
|
||||
except KeyboardInterrupt:
|
||||
console.print("\n[yellow]用户中断翻译[/yellow]")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
console.print(f"[red]翻译失败: {e}[/red]")
|
||||
logger.error(f"翻译失败: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def handle_cache_operations(args, config, console: Console):
|
||||
"""处理缓存相关操作"""
|
||||
from src.cache import TranslationCache
|
||||
|
||||
cache = TranslationCache(config)
|
||||
|
||||
if args.clear_cache is not None:
|
||||
console.print(f"[yellow]清理 {args.clear_cache} 天前的缓存...[/yellow]")
|
||||
cleared = cache.clear_cache(args.clear_cache)
|
||||
console.print(f"[green]已清理 {cleared} 个缓存文件[/green]")
|
||||
return True
|
||||
|
||||
if args.cache_stats:
|
||||
console.print("[cyan]缓存统计信息:[/cyan]")
|
||||
stats = cache.get_cache_stats()
|
||||
|
||||
if stats.get('enabled'):
|
||||
table = Table()
|
||||
table.add_column("项目", style="cyan")
|
||||
table.add_column("值", style="white")
|
||||
|
||||
table.add_row("缓存状态", "启用")
|
||||
table.add_row("缓存目录", stats.get('cache_directory', ''))
|
||||
table.add_row("文件总数", str(stats.get('total_files', 0)))
|
||||
table.add_row("总大小", f"{stats.get('total_size_mb', 0)} MB")
|
||||
table.add_row("最大保存天数", f"{stats.get('max_age_days', 0)} 天")
|
||||
|
||||
console.print(table)
|
||||
|
||||
# 显示按日期分布
|
||||
date_dist = stats.get('date_distribution', {})
|
||||
if date_dist:
|
||||
console.print("\n[cyan]按日期分布:[/cyan]")
|
||||
for date, count in sorted(date_dist.items()):
|
||||
console.print(f" {date}: {count} 个文件")
|
||||
else:
|
||||
console.print("[yellow]缓存未启用[/yellow]")
|
||||
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def check_environment():
|
||||
"""检查运行环境"""
|
||||
# 检查 Python 版本
|
||||
if sys.version_info < (3, 9):
|
||||
print("错误: 需要 Python 3.9 或更高版本")
|
||||
sys.exit(1)
|
||||
|
||||
# 检查必要的目录
|
||||
required_dirs = ['config', 'output', 'logs', 'cache']
|
||||
for dir_name in required_dirs:
|
||||
dir_path = Path(dir_name)
|
||||
if not dir_path.exists():
|
||||
dir_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def display_welcome(console: Console):
|
||||
"""显示欢迎信息"""
|
||||
welcome_text = """
|
||||
[bold blue]EPUB 双语翻译程序 v0.1.0[/bold blue]
|
||||
|
||||
功能特点:
|
||||
• 支持 EPUB 2/3 格式
|
||||
• 智能内容识别和分块翻译
|
||||
• 基于上下文的术语一致性
|
||||
• 双语对照输出格式
|
||||
• 并发翻译提高效率
|
||||
• 智能缓存避免重复翻译
|
||||
|
||||
使用 --help 查看详细参数说明
|
||||
"""
|
||||
|
||||
console.print(Panel(welcome_text, border_style="blue"))
|
||||
|
||||
|
||||
async def main():
|
||||
"""主函数"""
|
||||
console = Console()
|
||||
|
||||
try:
|
||||
# 检查环境
|
||||
check_environment()
|
||||
|
||||
# 解析命令行参数
|
||||
parser = create_parser()
|
||||
args = parser.parse_args()
|
||||
|
||||
# 如果没有参数，显示帮助
|
||||
if len(sys.argv) == 1:
|
||||
display_welcome(console)
|
||||
parser.print_help()
|
||||
return
|
||||
|
||||
# 加载配置
|
||||
try:
|
||||
config = load_config(args.config)
|
||||
except Exception as e:
|
||||
console.print(f"[red]加载配置失败: {e}[/red]")
|
||||
sys.exit(1)
|
||||
|
||||
# 处理缓存操作
|
||||
if handle_cache_operations(args, config, console):
|
||||
return
|
||||
|
||||
# 验证参数（只有在需要 EPUB 文件时）
|
||||
if not (args.clear_cache is not None or args.cache_stats):
|
||||
validate_args(args)
|
||||
|
||||
# 设置日志
|
||||
if args.verbose:
|
||||
config['logging']['level'] = 'DEBUG'
|
||||
|
||||
setup_logging(config)
|
||||
logger.info("程序启动")
|
||||
|
||||
# 初始化翻译器
|
||||
use_cache = not args.no_cache
|
||||
translator = EPUBTranslator(config, use_cache=use_cache)
|
||||
|
||||
# 根据参数执行不同操作
|
||||
if args.estimate:
|
||||
await run_estimate(translator, args.epub_file, console)
|
||||
else:
|
||||
await run_translation(translator, args, console)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
console.print("\n[yellow]程序被用户中断[/yellow]")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
console.print(f"[red]程序执行失败: {e}[/red]")
|
||||
logger.error(f"程序执行失败: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# 设置事件循环策略（Windows 兼容性）
|
||||
if sys.platform.startswith('win'):
|
||||
asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
|
||||
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,9 @@
|
||||
ebooklib>=0.19
|
||||
beautifulsoup4>=4.12.0
|
||||
lxml>=4.9.0
|
||||
openai>=1.0.0
|
||||
aiohttp>=3.9.0
|
||||
pydantic>=2.0.0
|
||||
loguru>=0.7.0
|
||||
rich>=13.0.0
|
||||
asyncio-throttle>=1.0.2
|
||||
@@ -0,0 +1,24 @@
|
||||
"""
|
||||
EPUB 双语翻译程序
|
||||
主要功能模块的初始化文件
|
||||
"""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
__author__ = "Kaitan"
|
||||
|
||||
from .epub_parser import EPUBParser
|
||||
from .translator import EPUBTranslator
|
||||
from .llm_client import OpenRouterClient
|
||||
from .text_processor import TextProcessor
|
||||
from .bilingual_builder import BilingualEPUBBuilder
|
||||
from .utils import load_config, setup_logging
|
||||
|
||||
__all__ = [
|
||||
"EPUBParser",
|
||||
"EPUBTranslator",
|
||||
"OpenRouterClient",
|
||||
"TextProcessor",
|
||||
"BilingualEPUBBuilder",
|
||||
"load_config",
|
||||
"setup_logging"
|
||||
]
|
||||
@@ -0,0 +1,155 @@
|
||||
"""
|
||||
双语 EPUB 构建器模块 - 安全的EPUB构建 (Manifest 兼容版)
|
||||
"""
|
||||
|
||||
from ebooklib import epub
|
||||
import ebooklib
|
||||
from bs4 import BeautifulSoup
|
||||
from typing import Dict, List
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
import uuid
|
||||
|
||||
|
||||
class BilingualEPUBBuilder:
|
||||
"""双语 EPUB 构建器"""
|
||||
|
||||
def __init__(self, original_book, config: Dict):
|
||||
self.original_book = original_book
|
||||
self.config = config
|
||||
self.output_config = config['output']
|
||||
|
||||
def create_bilingual_epub_with_mapping(self, translation_map: Dict[str, str],
|
||||
paragraph_map: Dict[str, Dict],
|
||||
output_path: str) -> str:
|
||||
"""
|
||||
创建双语 EPUB。使用 ordered_ids 确保与 Manifest 严格一致。
|
||||
"""
|
||||
try:
|
||||
new_book = epub.EpubBook()
|
||||
self._copy_metadata(new_book)
|
||||
new_book.toc = self.original_book.toc
|
||||
|
||||
# 准备每个文件的有序ID列表
|
||||
file_ordered_ids = {}
|
||||
sorted_pids = sorted(paragraph_map.keys(), key=lambda x: int(x.split('_')[1]))
|
||||
for pid in sorted_pids:
|
||||
info = paragraph_map[pid]
|
||||
fname = info['file_name']
|
||||
if fname not in file_ordered_ids:
|
||||
file_ordered_ids[fname] = []
|
||||
file_ordered_ids[fname].append(pid)
|
||||
|
||||
processed_item_ids = set()
|
||||
item_map = {}
|
||||
|
||||
# 复制资源
|
||||
for item in self.original_book.get_items():
|
||||
if item.get_type() != ebooklib.ITEM_DOCUMENT:
|
||||
if item.id not in processed_item_ids:
|
||||
new_book.add_item(item)
|
||||
processed_item_ids.add(item.id)
|
||||
item_map[item.id] = item
|
||||
|
||||
# 重建 Spine
|
||||
new_spine = []
|
||||
for spine_id, linear in self.original_book.spine:
|
||||
item = self.original_book.get_item_with_id(spine_id)
|
||||
if not item: continue
|
||||
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT:
|
||||
file_name = item.get_name()
|
||||
if file_name in file_ordered_ids:
|
||||
new_item = self._create_bilingual_document(
|
||||
item, file_ordered_ids[file_name], translation_map
|
||||
)
|
||||
new_item.id = item.id
|
||||
else:
|
||||
new_item = item
|
||||
|
||||
if new_item.id not in processed_item_ids:
|
||||
new_book.add_item(new_item)
|
||||
processed_item_ids.add(new_item.id)
|
||||
new_spine.append(new_item)
|
||||
else:
|
||||
if item.id in item_map:
|
||||
new_spine.append(item_map[item.id])
|
||||
|
||||
new_book.spine = new_spine
|
||||
new_book.add_item(epub.EpubNcx())
|
||||
new_book.add_item(epub.EpubNav())
|
||||
|
||||
output_file = self._generate_output_filename(output_path)
|
||||
epub.write_epub(output_file, new_book, {})
|
||||
return output_file
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"创建双语 EPUB 失败: {e}", exc_info=True)
|
||||
raise
|
||||
|
||||
def _copy_metadata(self, new_book):
|
||||
try:
|
||||
for namespace, meta_dict in self.original_book.metadata.items():
|
||||
for name, values in meta_dict.items():
|
||||
for value, other in values:
|
||||
if name and hasattr(name, 'lower') and name.lower() == 'identifier': continue
|
||||
new_book.add_metadata(namespace, name, value, other)
|
||||
new_book.add_metadata('DC', 'language', 'zh-CN')
|
||||
new_book.set_identifier(f"bilingual-{uuid.uuid4().hex[:12]}")
|
||||
|
||||
cover_id_meta = self.original_book.get_metadata('OPF', 'cover')
|
||||
if cover_id_meta:
|
||||
cover_item = self.original_book.get_item_with_id(cover_id_meta[0][0])
|
||||
if cover_item:
|
||||
new_book.add_item(cover_item)
|
||||
new_book.set_cover(cover_item.get_name(), cover_item.get_content())
|
||||
except Exception as e:
|
||||
logger.error(f"元数据复制出错: {e}")
|
||||
|
||||
def _create_bilingual_document(self, original_item, ordered_ids: list, translation_map: dict):
|
||||
try:
|
||||
from .text_processor import TextProcessor
|
||||
soup = BeautifulSoup(original_item.get_content().decode('utf-8'), 'html.parser')
|
||||
self._add_style_link(soup)
|
||||
|
||||
# 使用与 TextProcessor 相同的过滤逻辑获取元素
|
||||
text_elements = TextProcessor.get_valid_text_elements(soup)
|
||||
|
||||
current_para_index = 0
|
||||
for element in text_elements:
|
||||
if TextProcessor.is_navigation_element(element): continue
|
||||
if not TextProcessor.clean_element_text(element): continue
|
||||
|
||||
if current_para_index < len(ordered_ids):
|
||||
target_id = ordered_ids[current_para_index]
|
||||
translation = translation_map.get(target_id)
|
||||
if translation:
|
||||
self._insert_translation(element, translation, soup)
|
||||
current_para_index += 1
|
||||
|
||||
new_item = epub.EpubHtml(title=original_item.title, file_name=original_item.get_name(), lang='zh-CN')
|
||||
new_item.set_content(str(soup).encode('utf-8'))
|
||||
return new_item
|
||||
except Exception as e:
|
||||
logger.error(f"创建双语文档失败 {original_item.get_name()}: {e}")
|
||||
return original_item
|
||||
|
||||
def _add_style_link(self, soup):
|
||||
head = soup.find('head')
|
||||
if head and not head.find('link', href='style/bilingual.css'):
|
||||
head.append(soup.new_tag('link', rel='stylesheet', type='text/css', href='style/bilingual.css'))
|
||||
|
||||
def _insert_translation(self, element, translation: str, soup):
|
||||
try:
|
||||
translation_p = soup.new_tag('p')
|
||||
translation_p.string = translation
|
||||
translation_p['class'] = ['translation-text', 'chinese']
|
||||
element.insert_after(translation_p)
|
||||
except: pass
|
||||
|
||||
def _generate_output_filename(self, output_path: str) -> str:
|
||||
from .utils import sanitize_filename
|
||||
title = self.original_book.get_metadata('DC', 'title')
|
||||
clean_title = sanitize_filename(title[0][0]) if title else "bilingual_book"
|
||||
Path(output_path).mkdir(parents=True, exist_ok=True)
|
||||
return str(Path(output_path) / f"{clean_title}_bilingual.epub")
|
||||
@@ -0,0 +1,225 @@
|
||||
"""
|
||||
翻译缓存管理模块 - 简化版
|
||||
基于全局ID和chunk的缓存系统
|
||||
"""
|
||||
|
||||
import json
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Dict, Optional, List
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class TranslationCache:
|
||||
"""翻译缓存管理器 - 简化版"""
|
||||
|
||||
def __init__(self, config: Dict):
|
||||
"""初始化缓存管理器"""
|
||||
self.config = config
|
||||
cache_config = config.get('cache', {})
|
||||
|
||||
self.enabled = cache_config.get('enabled', True)
|
||||
self.cache_dir = Path(cache_config.get('directory', 'cache'))
|
||||
self.max_age_days = cache_config.get('max_age_days', 30)
|
||||
|
||||
if self.enabled:
|
||||
self.cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
self.translations_dir = self.cache_dir / 'translations'
|
||||
self.translations_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.info(f"翻译缓存已启用: {self.cache_dir}")
|
||||
|
||||
def get_chunk_translation(self, chunk: List[Dict], model: str) -> Optional[Dict[str, str]]:
|
||||
"""
|
||||
获取chunk的缓存翻译
|
||||
|
||||
Args:
|
||||
chunk: 段落列表（带global_id）
|
||||
model: 模型名称
|
||||
|
||||
Returns:
|
||||
{global_id: translation} 映射，如果不存在返回 None
|
||||
"""
|
||||
if not self.enabled:
|
||||
return None
|
||||
|
||||
try:
|
||||
cache_key = self._get_chunk_cache_key(chunk, model)
|
||||
cache_file = self._get_cache_file_path(cache_key)
|
||||
|
||||
if not cache_file.exists():
|
||||
return None
|
||||
|
||||
# 检查是否过期
|
||||
file_age = datetime.now() - datetime.fromtimestamp(cache_file.stat().st_mtime)
|
||||
if file_age > timedelta(days=self.max_age_days):
|
||||
logger.debug(f"缓存已过期: {cache_key[:8]}...")
|
||||
cache_file.unlink()
|
||||
return None
|
||||
|
||||
# 读取缓存
|
||||
with open(cache_file, 'r', encoding='utf-8') as f:
|
||||
cache_data = json.load(f)
|
||||
|
||||
# 验证缓存
|
||||
if (cache_data.get('success') and
|
||||
cache_data.get('model') == model and
|
||||
self._validate_cache_data(cache_data, chunk)):
|
||||
|
||||
logger.debug(f"缓存命中: {cache_key[:8]}... ({len(chunk)} 段落)")
|
||||
return cache_data.get('translations', {})
|
||||
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"读取缓存失败: {e}")
|
||||
return None
|
||||
|
||||
def save_chunk_translation(self, chunk: List[Dict], translations: Dict[str, str],
|
||||
model: str, success: bool = True) -> None:
|
||||
"""
|
||||
保存chunk翻译到缓存
|
||||
|
||||
Args:
|
||||
chunk: 段落列表（带global_id）
|
||||
translations: {global_id: translation} 映射
|
||||
model: 模型名称
|
||||
success: 是否翻译成功
|
||||
"""
|
||||
if not self.enabled:
|
||||
return
|
||||
|
||||
try:
|
||||
cache_key = self._get_chunk_cache_key(chunk, model)
|
||||
cache_file = self._get_cache_file_path(cache_key)
|
||||
|
||||
# 构建缓存数据
|
||||
cache_data = {
|
||||
'global_ids': [p['global_id'] for p in chunk],
|
||||
'translations': translations,
|
||||
'model': model,
|
||||
'timestamp': datetime.now().isoformat(),
|
||||
'success': success,
|
||||
'paragraph_count': len(chunk),
|
||||
'cache_version': '3.0'
|
||||
}
|
||||
|
||||
with open(cache_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(cache_data, f, ensure_ascii=False, indent=2)
|
||||
|
||||
logger.debug(f"缓存已保存: {cache_key[:8]}... ({len(chunk)} 段落)")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"保存缓存失败: {e}")
|
||||
|
||||
def _get_chunk_cache_key(self, chunk: List[Dict], model: str) -> str:
|
||||
"""
|
||||
生成chunk缓存键（基于全局ID序列）
|
||||
|
||||
Args:
|
||||
chunk: 段落列表
|
||||
model: 模型名称
|
||||
|
||||
Returns:
|
||||
缓存键
|
||||
"""
|
||||
# 使用全局ID序列作为缓存键的一部分
|
||||
id_sequence = ",".join(p['global_id'] for p in chunk)
|
||||
combined = f"{id_sequence}|{model}"
|
||||
return hashlib.md5(combined.encode('utf-8')).hexdigest()
|
||||
|
||||
def _get_cache_file_path(self, cache_key: str) -> Path:
|
||||
"""获取缓存文件路径"""
|
||||
today = datetime.now().strftime('%Y-%m-%d')
|
||||
cache_date_dir = self.translations_dir / today
|
||||
cache_date_dir.mkdir(parents=True, exist_ok=True)
|
||||
return cache_date_dir / f"{cache_key}.json"
|
||||
|
||||
def _validate_cache_data(self, cache_data: Dict, chunk: List[Dict]) -> bool:
|
||||
"""验证缓存数据的有效性"""
|
||||
# 检查ID序列是否匹配
|
||||
cached_ids = cache_data.get('global_ids', [])
|
||||
chunk_ids = [p['global_id'] for p in chunk]
|
||||
|
||||
if cached_ids != chunk_ids:
|
||||
logger.debug("缓存ID序列不匹配")
|
||||
return False
|
||||
|
||||
# 检查翻译数量
|
||||
translations = cache_data.get('translations', {})
|
||||
if len(translations) != len(chunk):
|
||||
logger.debug("缓存翻译数量不匹配")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def clear_cache(self, older_than_days: Optional[int] = None) -> int:
|
||||
"""清理缓存"""
|
||||
if not self.enabled or not self.translations_dir.exists():
|
||||
return 0
|
||||
|
||||
cleared_count = 0
|
||||
cutoff_time = None
|
||||
|
||||
if older_than_days is not None:
|
||||
cutoff_time = datetime.now() - timedelta(days=older_than_days)
|
||||
|
||||
try:
|
||||
for cache_file in self.translations_dir.rglob('*.json'):
|
||||
should_delete = False
|
||||
|
||||
if cutoff_time is None:
|
||||
should_delete = True
|
||||
else:
|
||||
file_time = datetime.fromtimestamp(cache_file.stat().st_mtime)
|
||||
should_delete = file_time < cutoff_time
|
||||
|
||||
if should_delete:
|
||||
cache_file.unlink()
|
||||
cleared_count += 1
|
||||
|
||||
# 清理空目录
|
||||
for date_dir in self.translations_dir.iterdir():
|
||||
if date_dir.is_dir() and not any(date_dir.iterdir()):
|
||||
date_dir.rmdir()
|
||||
|
||||
logger.info(f"清理了 {cleared_count} 个缓存文件")
|
||||
return cleared_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"清理缓存失败: {e}")
|
||||
return 0
|
||||
|
||||
def get_cache_stats(self) -> Dict:
|
||||
"""获取缓存统计信息"""
|
||||
if not self.enabled or not self.translations_dir.exists():
|
||||
return {'enabled': False}
|
||||
|
||||
try:
|
||||
cache_files = list(self.translations_dir.rglob('*.json'))
|
||||
total_files = len(cache_files)
|
||||
total_size = sum(f.stat().st_size for f in cache_files)
|
||||
|
||||
# 统计段落数
|
||||
total_paragraphs = 0
|
||||
for cache_file in cache_files:
|
||||
try:
|
||||
with open(cache_file, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
total_paragraphs += data.get('paragraph_count', 0)
|
||||
except:
|
||||
continue
|
||||
|
||||
return {
|
||||
'enabled': True,
|
||||
'total_files': total_files,
|
||||
'total_paragraphs': total_paragraphs,
|
||||
'total_size_mb': round(total_size / 1024 / 1024, 2),
|
||||
'cache_directory': str(self.cache_dir),
|
||||
'max_age_days': self.max_age_days
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"获取缓存统计失败: {e}")
|
||||
return {'enabled': True, 'error': str(e)}
|
||||
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
EPUB 解析器模块 (EPUB Parser Module)
|
||||
|
||||
该模块负责读取 EPUB 文件，提取元数据和内容项目。
|
||||
它使用 ebooklib 库来处理 EPUB 格式的底层细节。
|
||||
|
||||
Classes:
|
||||
EPUBParser: 负责 EPUB 文件的加载、元数据提取和内容项遍历。
|
||||
"""
|
||||
|
||||
import ebooklib
|
||||
from ebooklib import epub
|
||||
from bs4 import BeautifulSoup
|
||||
from typing import List, Dict, Any
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class EPUBParser:
|
||||
"""
|
||||
EPUB 文件解析器。
|
||||
|
||||
负责加载 EPUB 文件，提取书籍元数据（如标题、作者），并提供方法来遍历和提取
|
||||
书中的文档内容（HTML/XHTML）。
|
||||
|
||||
Attributes:
|
||||
epub_path (Path): EPUB 文件的路径对象。
|
||||
book (epub.EpubBook): ebooklib 加载的书籍对象。
|
||||
metadata (Dict[str, str]): 提取的书籍元数据字典。
|
||||
"""
|
||||
|
||||
def __init__(self, epub_path: str):
|
||||
"""
|
||||
初始化 EPUB 解析器。
|
||||
|
||||
Args:
|
||||
epub_path (str): EPUB 文件的文件路径。
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: 如果指定的文件不存在。
|
||||
Exception: 如果 EPUB 文件加载失败（格式错误等）。
|
||||
"""
|
||||
self.epub_path = Path(epub_path)
|
||||
if not self.epub_path.exists():
|
||||
raise FileNotFoundError(f"EPUB 文件不存在: {epub_path}")
|
||||
|
||||
try:
|
||||
# ignore_ncx=True 是为了避免某些旧版 epub 的警告，但新版 ebooklib 可能行为不同
|
||||
# 这里直接读取，让 ebooklib 处理
|
||||
self.book = epub.read_epub(str(self.epub_path))
|
||||
logger.info(f"成功加载 EPUB: {self.epub_path.name}")
|
||||
except Exception as e:
|
||||
logger.error(f"加载 EPUB 失败: {e}")
|
||||
raise
|
||||
|
||||
self.metadata = self._extract_metadata()
|
||||
|
||||
def _extract_metadata(self) -> Dict[str, str]:
|
||||
"""
|
||||
从 EPUB 对象中提取标准元数据。
|
||||
|
||||
提取 Dublin Core (DC) 元数据，包括标题、作者和语言。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 包含 'title', 'author', 'language' 的字典。
|
||||
如果提取失败，会使用默认值 ("Unknown", "en")。
|
||||
"""
|
||||
metadata = {}
|
||||
|
||||
try:
|
||||
# get_metadata 返回的是 (value, dict) 的列表，我们取第一个结果
|
||||
title_meta = self.book.get_metadata('DC', 'title')
|
||||
metadata['title'] = title_meta[0][0] if title_meta else "Unknown"
|
||||
|
||||
author_meta = self.book.get_metadata('DC', 'creator')
|
||||
metadata['author'] = author_meta[0][0] if author_meta else "Unknown"
|
||||
|
||||
lang_meta = self.book.get_metadata('DC', 'language')
|
||||
metadata['language'] = lang_meta[0][0] if lang_meta else "en"
|
||||
|
||||
logger.info(f"书籍: {metadata['title']} - {metadata['author']}")
|
||||
except Exception as e:
|
||||
logger.warning(f"提取元数据时出错: {e}")
|
||||
# 设置保底值
|
||||
metadata.setdefault('title', 'Unknown')
|
||||
metadata.setdefault('author', 'Unknown')
|
||||
metadata.setdefault('language', 'en')
|
||||
|
||||
return metadata
|
||||
|
||||
def extract_all_content_items(self) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
提取所有可翻译的内容项目（文档）。
|
||||
|
||||
遍历 EPUB 中的所有 Item，筛选出类型为 ITEM_DOCUMENT 的项目。
|
||||
同时会进行简单的过滤，跳过内容过短（<100字符）或看起来像非正文的文件（如 nav, toc, cover）。
|
||||
|
||||
Returns:
|
||||
List[Dict[str, Any]]: 内容项目列表。每个字典包含：
|
||||
- item (epub.EpubItem): 原始 Item 对象。
|
||||
- file_name (str): 文件名。
|
||||
- content (str): 解码后的 HTML 内容。
|
||||
- text_length (int): 纯文本长度（用于统计）。
|
||||
"""
|
||||
content_items = []
|
||||
|
||||
# 获取所有文档类型的项目
|
||||
for item in self.book.get_items():
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT:
|
||||
try:
|
||||
# 获取内容 (bytes -> str)
|
||||
content = item.get_content().decode('utf-8')
|
||||
|
||||
# 简单的内容验证：提取纯文本检查长度
|
||||
soup = BeautifulSoup(content, 'html.parser')
|
||||
text = soup.get_text().strip()
|
||||
|
||||
# 1. 跳过太短的内容（可能是只有图片的页面、空页面）
|
||||
if len(text) < 100:
|
||||
logger.debug(f"跳过短内容: {item.get_name()} ({len(text)} 字符)")
|
||||
continue
|
||||
|
||||
# 2. 跳过明显的非正文内容 (根据文件名判断)
|
||||
name_lower = item.get_name().lower()
|
||||
skip_patterns = ['cover', 'copyright', 'titlepage', 'halftitle',
|
||||
'nav.xhtml', 'toc.xhtml']
|
||||
if any(pattern in name_lower for pattern in skip_patterns):
|
||||
logger.debug(f"跳过非正文内容: {item.get_name()}")
|
||||
continue
|
||||
|
||||
content_items.append({
|
||||
'item': item,
|
||||
'file_name': item.get_name(),
|
||||
'content': content,
|
||||
'text_length': len(text)
|
||||
})
|
||||
|
||||
logger.debug(f"添加内容项: {item.get_name()} ({len(text)} 字符)")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"处理项目失败 {item.get_name()}: {e}")
|
||||
continue
|
||||
|
||||
logger.info(f"提取了 {len(content_items)} 个内容项目")
|
||||
return content_items
|
||||
|
||||
def get_book_info(self) -> Dict[str, str]:
|
||||
"""
|
||||
获取书籍的摘要信息。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 包含文件名、标题、作者、语言和文档数量的字典。
|
||||
"""
|
||||
# 统计内容项
|
||||
document_count = sum(1 for item in self.book.get_items()
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT)
|
||||
|
||||
return {
|
||||
'filename': self.epub_path.name,
|
||||
'title': self.metadata.get('title', 'Unknown'),
|
||||
'author': self.metadata.get('author', 'Unknown'),
|
||||
'language': self.metadata.get('language', 'en'),
|
||||
'document_count': document_count
|
||||
}
|
||||
@@ -0,0 +1,156 @@
|
||||
"""
|
||||
LLM 客户端模块 (LLM Client Module) - 简化 ID 锚点匹配版
|
||||
|
||||
该模块负责发送翻译请求，并使用简化后的 ID 作为锚点解析 LLM 的响应。
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from openai import AsyncOpenAI
|
||||
from typing import List, Dict, Optional, Any
|
||||
from loguru import logger
|
||||
import time
|
||||
import re
|
||||
from .manifest_manager import ManifestItem
|
||||
|
||||
|
||||
class RateLimiter:
|
||||
"""并发与 RPM 速率限制器。"""
|
||||
def __init__(self, requests_per_minute: int, concurrent_requests: int):
|
||||
self.semaphore = asyncio.Semaphore(concurrent_requests)
|
||||
self.min_interval = 60.0 / requests_per_minute if requests_per_minute > 0 else 0
|
||||
self.last_request_time = 0
|
||||
|
||||
async def acquire(self):
|
||||
await self.semaphore.acquire()
|
||||
current_time = time.time()
|
||||
wait_time = self.min_interval - (current_time - self.last_request_time)
|
||||
if wait_time > 0:
|
||||
await asyncio.sleep(wait_time)
|
||||
self.last_request_time = time.time()
|
||||
|
||||
def release(self):
|
||||
self.semaphore.release()
|
||||
|
||||
|
||||
class OpenRouterClient:
|
||||
"""基于简化 ID 锚点匹配逻辑的 LLM 客户端。"""
|
||||
|
||||
def __init__(self, config: Dict):
|
||||
self.config = config
|
||||
or_config = config["openrouter"]
|
||||
api_key = or_config.get("api_key")
|
||||
if not api_key or api_key == "YOUR_OPENROUTER_API_KEY":
|
||||
raise ValueError("请设置有效的 OpenRouter API Key")
|
||||
|
||||
self.client = AsyncOpenAI(
|
||||
base_url=or_config["base_url"],
|
||||
api_key=api_key,
|
||||
default_headers={"HTTP-Referer": "https://github.com/epub-translator", "X-Title": "EPUB Translator"}
|
||||
)
|
||||
self.models = or_config["models"]
|
||||
self.rate_limiter = RateLimiter(
|
||||
or_config["rate_limits"]["requests_per_minute"],
|
||||
or_config["rate_limits"]["concurrent_requests"]
|
||||
)
|
||||
|
||||
async def translate_chunk(self, items: List[ManifestItem], model_type: str = "production") -> Dict[str, str]:
|
||||
"""
|
||||
翻译一个段落块。
|
||||
"""
|
||||
if not items: return {}
|
||||
|
||||
prompt = self._build_prompt(items)
|
||||
model = self.models.get(model_type, self.models["production"])
|
||||
|
||||
try:
|
||||
raw_response = await self._make_request(prompt, model)
|
||||
if not raw_response:
|
||||
return {item.global_id: f"[翻译失败 - API空响应]" for item in items}
|
||||
|
||||
return self._parse_with_anchors(raw_response, items)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"翻译请求异常: {e}")
|
||||
return {item.global_id: f"[翻译失败 - {str(e)}]" for item in items}
|
||||
|
||||
def _build_prompt(self, items: List[ManifestItem]) -> str:
|
||||
lines = ["请将以下编号的英文段落翻译成中文。每段翻译前必须带上原编号，格式为 p_xxxxx。", "原文：", ""]
|
||||
for item in items:
|
||||
lines.append(f"{item.global_id} {item.clean_text}")
|
||||
lines.extend(["", "要求：只返回翻译，不要解释。必须保留原编号。"])
|
||||
return "\n".join(lines)
|
||||
|
||||
def _parse_with_anchors(self, response: str, items: List[ManifestItem]) -> Dict[str, str]:
|
||||
results = {}
|
||||
|
||||
for i, item in enumerate(items):
|
||||
current_id = item.global_id
|
||||
|
||||
# 使用简单的字符串拼接构造正则
|
||||
start_pat = r"\b" + current_id + r"\b"
|
||||
start_match = re.search(start_pat, response)
|
||||
if not start_match:
|
||||
continue
|
||||
|
||||
start_pos = start_match.start()
|
||||
end_pos = len(response)
|
||||
|
||||
if i + 1 < len(items):
|
||||
next_id = items[i+1].global_id
|
||||
next_pat = r"\b" + next_id + r"\b"
|
||||
next_match = re.search(next_pat, response[start_pos + 1:])
|
||||
if next_match:
|
||||
end_pos = start_pos + 1 + next_match.start()
|
||||
|
||||
segment = response[start_pos:end_pos].strip()
|
||||
|
||||
# 清洗 ID
|
||||
# 注意：这里的正则使用了更稳健的拼接方式
|
||||
id_clean_pat = r"^(\[?" + current_id + r"\]?[:：\s]*)+"
|
||||
cleaned = re.sub(id_clean_pat, "", segment).strip()
|
||||
|
||||
if cleaned:
|
||||
results[current_id] = cleaned
|
||||
|
||||
if len(results) < len(items):
|
||||
expected_ids = {item.global_id for item in items}
|
||||
missing_ids = expected_ids - set(results.keys())
|
||||
logger.warning(f"锚点匹配缺失 {len(missing_ids)} 个。尝试降级解析...")
|
||||
fallback_results = self._fallback_parse(response, items)
|
||||
for pid, trans in fallback_results.items():
|
||||
if pid not in results:
|
||||
results[pid] = trans
|
||||
|
||||
return results
|
||||
|
||||
def _fallback_parse(self, response: str, items: List[ManifestItem]) -> Dict[str, str]:
|
||||
results = {}
|
||||
lines = [l.strip() for l in response.split("\n") if l.strip()]
|
||||
for line in lines:
|
||||
for item in items:
|
||||
if item.global_id in line:
|
||||
# 同样的稳健拼接
|
||||
fallback_clean_pat = r"^\(?" + item.global_id + r"\)?[:：\s]*"
|
||||
clean_line = re.sub(fallback_clean_pat, "", line).strip()
|
||||
if clean_line:
|
||||
results[item.global_id] = clean_line
|
||||
return results
|
||||
|
||||
async def _make_request(self, prompt: str, model: str) -> str:
|
||||
await self.rate_limiter.acquire()
|
||||
try:
|
||||
resp = await self.client.chat.completions.create(
|
||||
model=model,
|
||||
messages=[
|
||||
{"role": "system", "content": "你是一位专业的翻译。请严格按编号输出翻译。"},
|
||||
{"role": "user", "content": prompt}
|
||||
],
|
||||
temperature=0.2,
|
||||
max_tokens=8000
|
||||
)
|
||||
return resp.choices[0].message.content.strip()
|
||||
finally:
|
||||
self.rate_limiter.release()
|
||||
|
||||
async def close(self):
|
||||
await self.client.close()
|
||||
@@ -0,0 +1,149 @@
|
||||
"""
|
||||
Manifest 管理器模块 (Manifest Manager Module)
|
||||
|
||||
该模块是系统的单一真理源 (SSOT)。
|
||||
它记录了每一段文本的原始状态、清洗后的文本、哈希值以及翻译状态。
|
||||
所有对翻译流程的操作（提取、翻译、回填）都必须通过修改此 Manifest 进行。
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import hashlib
|
||||
from typing import List, Dict, Optional, Any
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
from dataclasses import dataclass, asdict, field
|
||||
|
||||
@dataclass
|
||||
class ManifestItem:
|
||||
"""代表一个翻译单元（通常是一个段落）"""
|
||||
global_id: str
|
||||
source_file: str
|
||||
original_html: str
|
||||
clean_text: str
|
||||
text_hash: str
|
||||
tag: str
|
||||
translation: Optional[str] = None
|
||||
status: str = "pending" # pending, translated, ignored, failed
|
||||
error_msg: Optional[str] = None
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def to_dict(self):
|
||||
return asdict(self)
|
||||
|
||||
class ManifestManager:
|
||||
"""
|
||||
负责 Manifest 的生命周期管理。
|
||||
"""
|
||||
|
||||
def __init__(self, manifest_path: str):
|
||||
self.manifest_path = Path(manifest_path)
|
||||
self.data: Dict[str, Any] = {
|
||||
"book_id": "",
|
||||
"metadata": {},
|
||||
"items": []
|
||||
}
|
||||
self._items_by_id: Dict[str, ManifestItem] = {}
|
||||
|
||||
def load(self) -> bool:
|
||||
"""从文件加载 Manifest。如果文件不存在则返回 False。"""
|
||||
if self.manifest_path.exists():
|
||||
try:
|
||||
with open(self.manifest_path, 'r', encoding='utf-8') as f:
|
||||
self.data = json.load(f)
|
||||
|
||||
# 重建对象映射
|
||||
self._items_by_id = {
|
||||
item['global_id']: ManifestItem(**item)
|
||||
for item in self.data["items"]
|
||||
}
|
||||
logger.info(f"成功从 {self.manifest_path} 加载 Manifest, 包含 {len(self._items_by_id)} 个项目")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"加载 Manifest 失败: {e}")
|
||||
return False
|
||||
return False
|
||||
|
||||
def save(self):
|
||||
"""将当前状态保存到 Manifest 文件。"""
|
||||
# 确保目录存在
|
||||
self.manifest_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 同步 items 到 data 字典
|
||||
self.data["items"] = [item.to_dict() for item in self._items_by_id.values()]
|
||||
|
||||
with open(self.manifest_path, 'w', encoding='utf-8') as f:
|
||||
json.dump(self.data, f, ensure_ascii=False, indent=2)
|
||||
# logger.debug(f"Manifest 已保存到 {self.manifest_path}")
|
||||
|
||||
def init_manifest(self, book_id: str, metadata: Dict):
|
||||
"""初始化一个新的 Manifest。"""
|
||||
self.data = {
|
||||
"book_id": book_id,
|
||||
"metadata": metadata,
|
||||
"items": []
|
||||
}
|
||||
self._items_by_id = {}
|
||||
self.save()
|
||||
|
||||
def add_item(self, source_file: str, original_html: str, clean_text: str, tag: str, metadata: Dict = None) -> ManifestItem:
|
||||
"""添加一个新的翻译项并分配 ID。"""
|
||||
# 生成全局 ID
|
||||
new_index = len(self._items_by_id) + 1
|
||||
global_id = f"p_{new_index:05d}"
|
||||
|
||||
# 生成内容哈希 (用于排重和缓存)
|
||||
text_hash = hashlib.sha256(clean_text.encode('utf-8')).hexdigest()
|
||||
|
||||
item = ManifestItem(
|
||||
global_id=global_id,
|
||||
source_file=source_file,
|
||||
original_html=original_html,
|
||||
clean_text=clean_text,
|
||||
text_hash=text_hash,
|
||||
tag=tag,
|
||||
metadata=metadata or {}
|
||||
)
|
||||
|
||||
self._items_by_id[global_id] = item
|
||||
return item
|
||||
|
||||
def get_items(self, status: str = None, file_name: str = None) -> List[ManifestItem]:
|
||||
"""按状态或文件名查询项目。"""
|
||||
items = list(self._items_by_id.values())
|
||||
if status:
|
||||
items = [i for i in items if i.status == status]
|
||||
if file_name:
|
||||
items = [i for i in items if i.source_file == file_name]
|
||||
|
||||
# 必须按 ID 顺序返回以保证分块正确
|
||||
return sorted(items, key=lambda x: x.global_id)
|
||||
|
||||
def update_item(self, global_id: str, translation: str, status: str = "translated", error: str = None):
|
||||
"""更新翻译结果。"""
|
||||
if global_id in self._items_by_id:
|
||||
item = self._items_by_id[global_id]
|
||||
item.translation = translation
|
||||
item.status = status
|
||||
item.error_msg = error
|
||||
else:
|
||||
logger.warning(f"尝试更新不存在的 ID: {global_id}")
|
||||
|
||||
@property
|
||||
def stats(self) -> Dict:
|
||||
"""获取翻译进度统计。"""
|
||||
total = len(self._items_by_id)
|
||||
if total == 0: return {"progress": "0%"}
|
||||
|
||||
translated = sum(1 for i in self._items_by_id.values() if i.status == "translated")
|
||||
ignored = sum(1 for i in self._items_by_id.values() if i.status == "ignored")
|
||||
failed = sum(1 for i in self._items_by_id.values() if i.status == "failed")
|
||||
|
||||
return {
|
||||
"total": total,
|
||||
"translated": translated,
|
||||
"ignored": ignored,
|
||||
"failed": failed,
|
||||
"pending": total - translated - ignored - failed,
|
||||
"progress_percent": round((translated + ignored) / total * 100, 1)
|
||||
}
|
||||
@@ -0,0 +1,161 @@
|
||||
"""
|
||||
文本处理器模块 (Text Processor Module) - Manifest 驱动版
|
||||
|
||||
该模块专注于 HTML 文档的遍历和段落提取。
|
||||
它不再维护全局状态，而是将提取的内容注册到 ManifestManager 中。
|
||||
"""
|
||||
|
||||
import re
|
||||
from bs4 import BeautifulSoup
|
||||
from typing import List, Dict, Any
|
||||
from loguru import logger
|
||||
from .manifest_manager import ManifestManager
|
||||
|
||||
|
||||
class TextProcessor:
|
||||
"""
|
||||
负责从 HTML 中识别有效段落并进行清洗。
|
||||
"""
|
||||
|
||||
def __init__(self, config: Dict):
|
||||
"""
|
||||
Args:
|
||||
config (Dict): 全局配置。
|
||||
"""
|
||||
self.config = config
|
||||
self.chunk_size = config['translation'].get('chunk_size', 5000)
|
||||
|
||||
def extract_to_manifest(self, html_content: str, source_file: str, manifest: ManifestManager):
|
||||
"""
|
||||
解析 HTML 内容，并将识别出的段落注册到 Manifest 中。
|
||||
|
||||
Args:
|
||||
html_content (str): HTML 源码。
|
||||
source_file (str): 来源文件名。
|
||||
manifest (ManifestManager): 清单管理器实例。
|
||||
"""
|
||||
try:
|
||||
soup = BeautifulSoup(html_content, 'html.parser')
|
||||
|
||||
# 1. 移除不需要的元素
|
||||
for element in soup(['script', 'style', 'meta', 'link']):
|
||||
element.decompose()
|
||||
|
||||
# 2. 获取有效的文本元素 (使用静态过滤逻辑)
|
||||
text_elements = self.get_valid_text_elements(soup)
|
||||
|
||||
# 3. 注册到 Manifest
|
||||
for element in text_elements:
|
||||
clean_text = self.clean_element_text(element)
|
||||
|
||||
# 过滤逻辑
|
||||
if not clean_text:
|
||||
continue
|
||||
|
||||
status = "pending"
|
||||
# 如果是导航元素，标记为 ignored
|
||||
if self.is_navigation_element(element):
|
||||
status = "ignored"
|
||||
|
||||
# 注册
|
||||
manifest.add_item(
|
||||
source_file=source_file,
|
||||
original_html=str(element),
|
||||
clean_text=clean_text,
|
||||
tag=element.name,
|
||||
metadata={"status": status} # 临时传递给 manifest
|
||||
)
|
||||
|
||||
# 同步更新 manifest 状态 (如果需要过滤)
|
||||
if status == "ignored":
|
||||
last_id = f"p_{len(manifest._items_by_id):05d}"
|
||||
manifest.update_item(last_id, translation=None, status="ignored")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"从 {source_file} 提取段落失败: {e}")
|
||||
|
||||
@staticmethod
|
||||
def get_valid_text_elements(soup) -> List:
|
||||
"""获取不含嵌套子块的叶子级文本容器元素。"""
|
||||
tags = ['p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'blockquote', 'li', 'td']
|
||||
all_candidates = soup.find_all(tags)
|
||||
candidate_set = set(all_candidates)
|
||||
|
||||
final_elements = []
|
||||
for element in all_candidates:
|
||||
# 如果包含其他候选标签，说明是容器，跳过
|
||||
if any(d in candidate_set for d in element.find_all(tags)):
|
||||
continue
|
||||
final_elements.append(element)
|
||||
return final_elements
|
||||
|
||||
@staticmethod
|
||||
def clean_element_text(element) -> str:
|
||||
"""清理 HTML 元素，提取纯净的待翻译文本。"""
|
||||
element_copy = element.__copy__()
|
||||
|
||||
# 移除脚注引用等
|
||||
for tag in element_copy.find_all(['sup', 'sub']):
|
||||
tag.decompose()
|
||||
|
||||
footnote_patterns = re.compile(r'footnote|endnote|reference|note|super|sub', re.I)
|
||||
for tag in element_copy.find_all(['a', 'span', 'div'], class_=footnote_patterns):
|
||||
tag.decompose()
|
||||
|
||||
# 移除仅包含数字的 span
|
||||
for tag in element_copy.find_all('span'):
|
||||
if re.match(r'^(\[\d+\]|\(\d+\)|\d+)$', tag.get_text().strip()):
|
||||
tag.decompose()
|
||||
|
||||
text = element_copy.get_text().strip()
|
||||
# 正则清理残留引用标识 (如 sentence.2)
|
||||
text = re.sub(r'(\.|。|，|,)\s*(\[\d+\]|\d+)(?=\s|$)', r'\1', text)
|
||||
text = re.sub(r'\s+', ' ', text)
|
||||
return text
|
||||
|
||||
@staticmethod
|
||||
def is_navigation_element(element) -> bool:
|
||||
"""判断是否是无翻译价值的导航、页码元素。"""
|
||||
classes = element.get('class', [])
|
||||
nav_classes = ['nav', 'navigation', 'toc', 'menu', 'header', 'footer', 'page-number']
|
||||
class_str = ' '.join(classes).lower() if isinstance(classes, list) else str(classes).lower()
|
||||
|
||||
if any(nc in class_str for nc in nav_classes):
|
||||
return True
|
||||
|
||||
# 检查父级
|
||||
parent = element.parent
|
||||
if parent:
|
||||
p_classes = parent.get('class', [])
|
||||
p_class_str = ' '.join(p_classes).lower() if isinstance(p_classes, list) else str(p_classes).lower()
|
||||
if any(nc in p_class_str for nc in nav_classes):
|
||||
return True
|
||||
return False
|
||||
|
||||
def create_chunks_from_manifest(self, manifest: ManifestManager) -> List[List[Any]]:
|
||||
"""
|
||||
从 Manifest 中筛选待翻译项目并分块。
|
||||
"""
|
||||
pending_items = manifest.get_items(status="pending")
|
||||
if not pending_items:
|
||||
return []
|
||||
|
||||
chunks = []
|
||||
current_chunk = []
|
||||
current_size = 0
|
||||
|
||||
for item in pending_items:
|
||||
text_len = len(item.clean_text)
|
||||
if current_size + text_len > self.chunk_size and current_chunk:
|
||||
chunks.append(current_chunk)
|
||||
current_chunk = []
|
||||
current_size = 0
|
||||
|
||||
current_chunk.append(item)
|
||||
current_size += text_len
|
||||
|
||||
if current_chunk:
|
||||
chunks.append(current_chunk)
|
||||
|
||||
logger.info(f"分块完成: 共有 {len(pending_items)} 个待翻译项，分为 {len(chunks)} 个块")
|
||||
return chunks
|
||||
@@ -0,0 +1,142 @@
|
||||
"""
|
||||
EPUB 翻译器核心模块 (EPUB Translator Core Module) - Manifest 驱动版
|
||||
|
||||
该模块协调整体流程：
|
||||
1. 使用 ManifestManager 管理状态。
|
||||
2. 调用 EPUBParser 提取。
|
||||
3. 调用 TextProcessor 清理。
|
||||
4. 调用 LLMClient 并发翻译并更新 Manifest。
|
||||
5. 调用 BilingualEPUBBuilder 构建。
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from typing import List, Dict, Any
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
from rich.console import Console
|
||||
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TimeElapsedColumn
|
||||
|
||||
from .epub_parser import EPUBParser
|
||||
from .llm_client import OpenRouterClient
|
||||
from .text_processor import TextProcessor
|
||||
from .bilingual_builder import BilingualEPUBBuilder
|
||||
from .manifest_manager import ManifestManager
|
||||
|
||||
|
||||
class EPUBTranslator:
|
||||
"""
|
||||
基于 Manifest 的翻译器。
|
||||
"""
|
||||
|
||||
def __init__(self, config: Dict, use_cache: bool = True):
|
||||
self.config = config
|
||||
self.console = Console()
|
||||
self.use_cache = use_cache
|
||||
|
||||
# 组件
|
||||
self.parser = None
|
||||
self.llm_client = OpenRouterClient(config)
|
||||
self.text_processor = TextProcessor(config)
|
||||
|
||||
# Manifest 管理 (存放于 cache/manifests/ 目录下)
|
||||
self.manifest_dir = Path("cache/manifests")
|
||||
self.manifest_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
async def translate_epub(self, epub_path: str, test_mode: bool = False, output_dir: str = None) -> str:
|
||||
"""主翻译流程。"""
|
||||
epub_path = Path(epub_path)
|
||||
# 1. 初始化解析器
|
||||
self.parser = EPUBParser(str(epub_path))
|
||||
|
||||
# 2. 准备 Manifest
|
||||
manifest_path = self.manifest_dir / f"{epub_path.stem}_manifest.json"
|
||||
manifest = ManifestManager(str(manifest_path))
|
||||
|
||||
# 检查是否能恢复
|
||||
if not manifest.load() or not self.use_cache:
|
||||
self.console.print("[yellow]初始化翻译清单...[/yellow]")
|
||||
manifest.init_manifest(book_id=epub_path.name, metadata=self.parser.get_book_info())
|
||||
|
||||
# 提取内容
|
||||
content_items = self.parser.extract_all_content_items()
|
||||
for item in content_items:
|
||||
self.text_processor.extract_to_manifest(item['content'], item['file_name'], manifest)
|
||||
manifest.save()
|
||||
|
||||
stats = manifest.stats
|
||||
self.console.print(f"[green]已加载清单: {stats['total']} 个段落, 已完成 {stats['progress_percent']}%[/green]")
|
||||
|
||||
if test_mode:
|
||||
# 简化逻辑：测试模式只翻译前几个 pending 项目
|
||||
pending = manifest.get_items(status="pending")[:5]
|
||||
if pending:
|
||||
results = await self.llm_client.translate_chunk(pending)
|
||||
for pid, trans in results.items():
|
||||
self.console.print(f"\n[cyan]{pid}[/cyan]: {trans}")
|
||||
return "test_mode_done"
|
||||
|
||||
# 3. 分块并并发翻译
|
||||
chunks = self.text_processor.create_chunks_from_manifest(manifest)
|
||||
if chunks:
|
||||
await self._translate_concurrently(chunks, manifest)
|
||||
|
||||
# 4. 构建双语 EPUB
|
||||
self.console.print("\n[yellow]正在构建双语 EPUB...[/yellow]")
|
||||
output_path = output_dir or self.config['output']['output_dir']
|
||||
builder = BilingualEPUBBuilder(self.parser.book, self.config)
|
||||
|
||||
# 注意：Builder 现在直接从 Manifest 中读取翻译映射
|
||||
translation_map = {item.global_id: item.translation for item in manifest.get_items() if item.translation}
|
||||
paragraph_map = {item.global_id: {
|
||||
"file_name": item.source_file,
|
||||
"text": item.clean_text,
|
||||
"html_element": item.original_html
|
||||
} for item in manifest.get_items()}
|
||||
|
||||
result_file = builder.create_bilingual_epub_with_mapping(
|
||||
translation_map,
|
||||
paragraph_map,
|
||||
output_path
|
||||
)
|
||||
|
||||
self.console.print(f"[green]✅ 翻译完成！输出文件: {result_file}[/green]")
|
||||
return result_file
|
||||
|
||||
async def _translate_concurrently(self, chunks: List[List[Any]], manifest: ManifestManager):
|
||||
"""执行并发翻译任务。"""
|
||||
total_chunks = len(chunks)
|
||||
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
BarColumn(),
|
||||
TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
|
||||
TimeElapsedColumn(),
|
||||
console=self.console
|
||||
) as progress:
|
||||
|
||||
task_id = progress.add_task(f"[cyan]并行翻译...", total=total_chunks)
|
||||
|
||||
# 使用可控并发
|
||||
semaphore = self.llm_client.rate_limiter.semaphore
|
||||
|
||||
async def worker(chunk, idx):
|
||||
async with semaphore:
|
||||
try:
|
||||
results = await self.llm_client.translate_chunk(chunk)
|
||||
# 更新 manifest
|
||||
for item in chunk:
|
||||
if item.global_id in results:
|
||||
manifest.update_item(item.global_id, results[item.global_id])
|
||||
else:
|
||||
manifest.update_item(item.global_id, None, status="failed", error="Missing in response")
|
||||
# 每翻译完一个 chunk 就保存一次，确保断点续传
|
||||
manifest.save()
|
||||
except Exception as e:
|
||||
logger.error(f"Chunk {idx} 翻译失败: {e}")
|
||||
finally:
|
||||
progress.update(task_id, advance=1)
|
||||
|
||||
tasks = [worker(chunk, i) for i, chunk in enumerate(chunks)]
|
||||
await asyncio.gather(*tasks)
|
||||
@@ -0,0 +1,180 @@
|
||||
"""
|
||||
工具函数模块
|
||||
提供配置加载、日志设置等通用功能
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any
|
||||
from loguru import logger
|
||||
import sys
|
||||
|
||||
|
||||
def load_config(config_path: str = "config/config.json") -> Dict[str, Any]:
|
||||
"""
|
||||
加载配置文件
|
||||
|
||||
Args:
|
||||
config_path: 配置文件路径
|
||||
|
||||
Returns:
|
||||
配置字典
|
||||
"""
|
||||
try:
|
||||
with open(config_path, 'r', encoding='utf-8') as f:
|
||||
config = json.load(f)
|
||||
|
||||
# 从环境变量获取 API Key
|
||||
if 'OPENROUTER_API_KEY' in os.environ:
|
||||
config['openrouter']['api_key'] = os.environ['OPENROUTER_API_KEY']
|
||||
|
||||
return config
|
||||
except FileNotFoundError:
|
||||
raise FileNotFoundError(f"配置文件未找到: {config_path}")
|
||||
except json.JSONDecodeError as e:
|
||||
raise ValueError(f"配置文件格式错误: {e}")
|
||||
|
||||
|
||||
def load_prompts(prompts_path: str = "config/prompts.json") -> Dict[str, str]:
|
||||
"""
|
||||
加载提示词模板
|
||||
|
||||
Args:
|
||||
prompts_path: 提示词文件路径
|
||||
|
||||
Returns:
|
||||
提示词字典
|
||||
"""
|
||||
try:
|
||||
with open(prompts_path, 'r', encoding='utf-8') as f:
|
||||
return json.load(f)
|
||||
except FileNotFoundError:
|
||||
raise FileNotFoundError(f"提示词文件未找到: {prompts_path}")
|
||||
|
||||
|
||||
def setup_logging(config: Dict[str, Any]) -> None:
|
||||
"""
|
||||
设置日志配置
|
||||
|
||||
Args:
|
||||
config: 配置字典
|
||||
"""
|
||||
log_config = config.get('logging', {})
|
||||
|
||||
# 移除默认处理器
|
||||
logger.remove()
|
||||
|
||||
# 添加控制台输出
|
||||
logger.add(
|
||||
sys.stdout,
|
||||
level=log_config.get('level', 'INFO'),
|
||||
format="<green>{time:YYYY-MM-DD HH:mm:ss}</green> | <level>{level: <8}</level> | <cyan>{name}</cyan>:<cyan>{function}</cyan>:<cyan>{line}</cyan> - <level>{message}</level>"
|
||||
)
|
||||
|
||||
# 添加文件输出
|
||||
if 'file' in log_config:
|
||||
log_file = log_config['file']
|
||||
# 确保日志目录存在
|
||||
Path(log_file).parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.add(
|
||||
log_file,
|
||||
level=log_config.get('level', 'INFO'),
|
||||
rotation=log_config.get('rotation', '10 MB'),
|
||||
retention=log_config.get('retention', '7 days'),
|
||||
encoding='utf-8',
|
||||
format="{time:YYYY-MM-DD HH:mm:ss} | {level: <8} | {name}:{function}:{line} - {message}"
|
||||
)
|
||||
|
||||
|
||||
def ensure_output_dir(output_dir: str) -> Path:
|
||||
"""
|
||||
确保输出目录存在
|
||||
|
||||
Args:
|
||||
output_dir: 输出目录路径
|
||||
|
||||
Returns:
|
||||
输出目录的 Path 对象
|
||||
"""
|
||||
output_path = Path(output_dir)
|
||||
output_path.mkdir(parents=True, exist_ok=True)
|
||||
return output_path
|
||||
|
||||
|
||||
def sanitize_filename(filename: str) -> str:
|
||||
"""
|
||||
清理文件名，移除非法字符
|
||||
|
||||
Args:
|
||||
filename: 原始文件名
|
||||
|
||||
Returns:
|
||||
清理后的文件名
|
||||
"""
|
||||
import re
|
||||
# 移除或替换非法字符
|
||||
filename = re.sub(r'[<>:"/\\|?*]', '_', filename)
|
||||
# 移除多余的空格和点
|
||||
filename = re.sub(r'\s+', ' ', filename).strip('. ')
|
||||
return filename
|
||||
|
||||
|
||||
def format_file_size(size_bytes: int) -> str:
|
||||
"""
|
||||
格式化文件大小显示
|
||||
|
||||
Args:
|
||||
size_bytes: 字节数
|
||||
|
||||
Returns:
|
||||
格式化的大小字符串
|
||||
"""
|
||||
if size_bytes == 0:
|
||||
return "0B"
|
||||
|
||||
size_names = ["B", "KB", "MB", "GB"]
|
||||
import math
|
||||
i = int(math.floor(math.log(size_bytes, 1024)))
|
||||
p = math.pow(1024, i)
|
||||
s = round(size_bytes / p, 2)
|
||||
return f"{s} {size_names[i]}"
|
||||
|
||||
|
||||
def estimate_tokens(text: str) -> int:
|
||||
"""
|
||||
估算文本的 token 数量
|
||||
|
||||
Args:
|
||||
text: 输入文本
|
||||
|
||||
Returns:
|
||||
估算的 token 数量
|
||||
"""
|
||||
# 简单估算：英文约 4 字符/token，中文约 1.5 字符/token
|
||||
import re
|
||||
|
||||
# 分离中英文
|
||||
chinese_chars = len(re.findall(r'[\u4e00-\u9fff]', text))
|
||||
other_chars = len(text) - chinese_chars
|
||||
|
||||
# 估算 tokens
|
||||
estimated_tokens = chinese_chars / 1.5 + other_chars / 4
|
||||
return int(estimated_tokens)
|
||||
|
||||
|
||||
def truncate_text(text: str, max_length: int = 100) -> str:
|
||||
"""
|
||||
截断文本用于显示
|
||||
|
||||
Args:
|
||||
text: 原始文本
|
||||
max_length: 最大长度
|
||||
|
||||
Returns:
|
||||
截断后的文本
|
||||
"""
|
||||
if len(text) <= max_length:
|
||||
return text
|
||||
return text[:max_length-3] + "..."
|
||||
@@ -0,0 +1,33 @@
|
||||
# 更新日志 (CHANGELOG)
|
||||
|
||||
## [v0.03] - 2026-01-12
|
||||
|
||||
### 🌟 核心突破
|
||||
- **极简 ID 锚点系统**:
|
||||
- 废弃复杂的 `[p_xxxxx]` 格式，回归纯净的 `p_xxxxx` 文本锚点。
|
||||
- 重写 `LLMClient` 解析逻辑，使用字符串切片替代正则，彻底解决了 ID 残留和语法错误问题。
|
||||
- **智能术语一致性**:
|
||||
- 引入 `GlossaryManager`，自动提取前言和正文采样。
|
||||
- 集成 Smart 模型 (如 `gemini-pro`) 自动生成术语表 (解决 "Masa" -> "孙正义" 等歧义问题)。
|
||||
- 支持人工介入审核术语表。
|
||||
|
||||
### 🏗️ 架构升级
|
||||
- **配置化驱动**: 移除了代码中的硬编码，所有参数（包括 Prompt 模板）均移入 `config/` 目录。
|
||||
- **模型分级**: 支持 `fast` (用于大批量翻译) 和 `smart` (用于高智商任务) 双模型策略。
|
||||
|
||||
### 🔧 修复与优化
|
||||
- **结构完美保留**:
|
||||
- 修复了 EPUB Spine 重建逻辑，不再丢失封面、目录页和非正文资源。
|
||||
- 修复了元数据 (Cover/Title) 复制错误。
|
||||
- **零阈值提取**:
|
||||
- 移除了段落最小长度限制，确保标题、短句不被漏译。
|
||||
|
||||
---
|
||||
|
||||
## [v0.02] - 2026-01-12
|
||||
- **Manifest 驱动架构**: 引入 `ManifestManager` 作为单一真理源。
|
||||
- **流程解耦**: 提取、翻译、构建三阶段分离。
|
||||
- **断点续传**: 支持随时中断和恢复。
|
||||
|
||||
## [v0.01] - 2026-01-10
|
||||
- 初始版本，实现基本的并发翻译和 EPUB 解析。
|
||||
@@ -0,0 +1,60 @@
|
||||
# 开发者避坑指南 (Developer's Survival Guide)
|
||||
|
||||
这份文档总结了 EPUB 翻译器开发过程中的血泪教训。在修改代码前，**务必阅读此文档**。
|
||||
|
||||
## 🔴 核心原则 (Core Principles)
|
||||
|
||||
### 1. 奥卡姆剃刀原则 (KISS)
|
||||
**不要自作聪明。**
|
||||
* **错误案例**：为了“美观”或“规范”，给 ID 加上方括号 `[p_001]`，甚至试图让 LLM 返回 JSON 结构。
|
||||
* **后果**：LLM 经常搞错括号的全角/半角，或者漏掉闭合括号，导致正则解析极其痛苦，甚至产生 `SyntaxError`。
|
||||
* **最佳实践**：**ID 就用纯文本 `p_xxxxx`。** 解析就用 `find()` 和字符串切片。越简单越不容易出错。
|
||||
|
||||
### 2. 单一真理源 (Single Source of Truth)
|
||||
**不要在模块间传递散乱的数据。**
|
||||
* **错误案例**：`TextProcessor` 返回一个 list，`Translator` 拿去翻译，`Builder` 又重新解析一遍 HTML 试图匹配。
|
||||
* **后果**：一旦提取逻辑微调（比如过滤了短句），Builder 就再也对不齐了，导致严重的错位（翻译张冠李戴）。
|
||||
* **最佳实践**：**Manifest (清单) 是唯一的真理。** 提取时生成 Manifest，翻译时更新 Manifest，构建时只读 Manifest。
|
||||
|
||||
---
|
||||
|
||||
## 🚫 常见陷阱 (Pitfalls)
|
||||
|
||||
### 1. Prompt Engineering
|
||||
* **不要指望 LLM 完美遵守复杂的格式指令。**
|
||||
* *Bad Prompt*: "请返回 JSON，key 是 ID，value 是译文..." (JSON 语法错误率高，Token 消耗大)
|
||||
* *Bad Prompt*: "请用 `[ID]` 包裹编号..." (括号混乱)
|
||||
* *Good Prompt*: "每行开头必须是 `p_xxxxx`，后接译文。严禁修改 ID。"
|
||||
* **不要让 LLM "解释" 它的翻译。**
|
||||
* 它一旦开始解释，解析器就很难把正文抠出来。必须在 System Prompt 中严令禁止。
|
||||
|
||||
### 2. 正则表达式 (Regex)
|
||||
* **慎用 `re.sub` 处理未知输入。**
|
||||
* LLM 返回的文本可能包含各种奇怪的 unicode 字符或未转义的特殊符号。
|
||||
* 在 f-string 中拼接正则（如 `rf'\[{id}\]'`）极易引发 Python 的 `SyntaxError`，尤其是涉及引号嵌套时。
|
||||
* **解决方案**：如果能用字符串 `find()` + 切片解决的问题，**绝对不要用正则**。
|
||||
|
||||
### 3. EPUB 结构处理
|
||||
* **不要随意丢弃 Item。**
|
||||
* 之前的逻辑是“只处理 Document，其他的忽略”。结果导致封面图片、css、字体文件全部丢失。
|
||||
* **正确逻辑**：默认复制所有非 Document 资源。对于 Document，要么替换为双语版，要么原样保留。
|
||||
* **不要重建 Spine 顺序。**
|
||||
* 不要试图自己去猜页面顺序。严格按照 `original_book.spine` 的顺序来构建新书。
|
||||
* **不要依赖 `min_length` 过滤。**
|
||||
* "Chapter 1" 只有 9 个字符，但它很重要。任何长度过滤都会导致漏译。
|
||||
|
||||
### 4. Metadata 处理
|
||||
* **不要假设 Metadata 总是规范的字符串。**
|
||||
* `ebooklib` 解析出来的 metadata 有时是对象，有时是 `None`。调用 `.lower()` 前必须做类型检查 (`if name and isinstance(name, str)...`)。
|
||||
|
||||
---
|
||||
|
||||
## ✅ 推荐工作流 (Workflow)
|
||||
|
||||
1. **修改提取逻辑时** -> 必须同时检查 `get_valid_text_elements` 是否被 `Builder` 复用。
|
||||
2. **修改 Prompt 时** -> 必须同步更新 `LLMClient` 的解析逻辑。
|
||||
3. **遇到对齐问题时** -> 不要去改 `Builder` 的匹配算法，而是去检查 Manifest 中的 ID 序列是否正确。
|
||||
|
||||
---
|
||||
|
||||
*Last Updated: v0.03*
|
||||
@@ -0,0 +1,409 @@
|
||||
# EPUB 双语翻译程序 v2.0
|
||||
|
||||
一个基于 OpenRouter API 的 EPUB 双语翻译工具，采用**全局编号系统**和**真并发翻译**。
|
||||
|
||||
## ✨ 核心特性
|
||||
|
||||
### 🎯 全局编号系统
|
||||
- **每个段落分配全局唯一ID**（格式：`p_0001`, `p_0002`...）
|
||||
- **ID贯穿全流程**：提取 → 翻译 → 组装
|
||||
- **精确对应保证**：绝不出现中英文错行问题
|
||||
|
||||
### ⚡ 真并发翻译
|
||||
- **asyncio.gather 并发执行**：不再是串行等待
|
||||
- **8倍速度提升**：默认8个请求同时进行
|
||||
- **智能速率控制**：Semaphore自动限制并发数
|
||||
- **实时进度显示**：Rich进度条显示翻译状态
|
||||
|
||||
### 📦 智能分块策略
|
||||
- **纯字符数分块**：基于 `chunk_size` 参数（默认5000字符）
|
||||
- **不切断段落**：严格保持段落完整性
|
||||
- **跨章节chunk**：现代LLM支持，无需人为限制章节边界
|
||||
- **自动优化**: 在不切断段落的前提下最大化chunk利用率
|
||||
|
||||
### 🎨 极简架构
|
||||
- **代码精简40%**：移除复杂的章节处理、段落排序逻辑
|
||||
- **统一数据流**：提取 → 编号 → 分块 → 翻译 → 组装
|
||||
- **配置简化**：删除冗余参数，保留核心配置
|
||||
|
||||
## 🚀 快速开始
|
||||
|
||||
### 1. 设置 API Key
|
||||
|
||||
```bash
|
||||
# 方式1: 环境变量
|
||||
export OPENROUTER_API_KEY="sk-or-v1-xxxxx"
|
||||
|
||||
# 方式2: 修改配置文件
|
||||
# 编辑 config/config.json，填入你的API Key
|
||||
```
|
||||
|
||||
### 2. 测试翻译
|
||||
|
||||
```bash
|
||||
# 测试模式（翻译前3个段落）
|
||||
python main.py your_book.epub --test
|
||||
|
||||
# 测试并发逻辑
|
||||
python test_concurrent.py
|
||||
|
||||
# 测试全局ID系统
|
||||
python test_global_id_system.py
|
||||
```
|
||||
|
||||
### 3. 完整翻译
|
||||
|
||||
```bash
|
||||
# 完整翻译
|
||||
python main.py your_book.epub
|
||||
|
||||
# 指定输出目录
|
||||
python main.py your_book.epub --output ./my_output
|
||||
|
||||
# 禁用缓存
|
||||
python main.py your_book.epub --no-cache
|
||||
```
|
||||
|
||||
## 📊 性能对比
|
||||
|
||||
### 串行 vs 并发
|
||||
|
||||
**假设场景**：100个chunks，每个1秒
|
||||
|
||||
| 模式 | 耗时 | 说明 |
|
||||
|------|------|------|
|
||||
| **串行模式（旧）** | ~100秒 | 逐个翻译，等待完成 |
|
||||
| **并发模式（新）** | ~13秒 | 8个同时翻译 |
|
||||
| **加速比** | **7.7x** | 接近理论最大值8x |
|
||||
|
||||
### 实际测试结果
|
||||
|
||||
```bash
|
||||
$ python test_concurrent.py
|
||||
|
||||
📊 方法1: 串行翻译
|
||||
⏱️ 串行耗时: 10.23 秒
|
||||
|
||||
📊 方法2: 并发翻译 (asyncio.gather)
|
||||
⏱️ 并发耗时: 1.35 秒
|
||||
|
||||
📈 性能对比
|
||||
加速比: 7.58x ✅
|
||||
```
|
||||
|
||||
## 🎯 核心架构
|
||||
|
||||
### 数据流
|
||||
```
|
||||
EPUB文件
|
||||
↓
|
||||
提取所有段落（保持文档顺序）
|
||||
↓
|
||||
分配全局ID (p_0001, p_0002, ...)
|
||||
↓
|
||||
按字符数分chunk（不切断段落，可跨章节）
|
||||
↓
|
||||
并发翻译（asyncio.gather + Semaphore）
|
||||
↓
|
||||
返回 {global_id: translation} 映射
|
||||
↓
|
||||
基于文本内容精确匹配
|
||||
↓
|
||||
插入翻译，构建双语EPUB
|
||||
```
|
||||
|
||||
### 全局ID系统
|
||||
|
||||
每个段落在提取时就分配唯一ID：
|
||||
|
||||
```python
|
||||
{
|
||||
'global_id': 'p_0001', # 全局唯一ID
|
||||
'text': '段落文本...',
|
||||
'source_file': 'chapter1.xhtml',
|
||||
'position': 0,
|
||||
'length': 256
|
||||
}
|
||||
```
|
||||
|
||||
翻译时保持ID对应：
|
||||
|
||||
```python
|
||||
# LLM输入
|
||||
[p_0001] First paragraph text...
|
||||
[p_0002] Second paragraph text...
|
||||
|
||||
# LLM输出
|
||||
[p_0001] 第一段的中文翻译
|
||||
[p_0002] 第二段的中文翻译
|
||||
|
||||
# 结果映射
|
||||
{
|
||||
'p_0001': '第一段的中文翻译',
|
||||
'p_0002': '第二段的中文翻译'
|
||||
}
|
||||
```
|
||||
|
||||
### 并发翻译机制
|
||||
|
||||
```python
|
||||
# 创建所有翻译任务
|
||||
tasks = [translate_chunk(chunk) for chunk in chunks]
|
||||
|
||||
# 并发执行（受Semaphore限制）
|
||||
results = await asyncio.gather(*tasks)
|
||||
|
||||
# Semaphore自动控制：
|
||||
# - 最多8个任务同时执行
|
||||
# - 其他任务排队等待
|
||||
# - 一个完成，下一个立即开始
|
||||
```
|
||||
|
||||
## ⚙️ 配置说明
|
||||
|
||||
### 精简后的配置
|
||||
|
||||
```json
|
||||
{
|
||||
"openrouter": {
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 60,
|
||||
"concurrent_requests": 8 // 控制并发数
|
||||
}
|
||||
},
|
||||
"translation": {
|
||||
"chunk_size": 5000, // 每个chunk的字符数
|
||||
"temperature": 0.2 // LLM温度参数
|
||||
},
|
||||
"processing": {
|
||||
"min_paragraph_length": 30 // 最小段落长度
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 关键参数说明
|
||||
|
||||
| 参数 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `concurrent_requests` | 8 | 并发请求数，建议5-10 |
|
||||
| `chunk_size` | 5000 | 每chunk字符数，现代LLM可设更大 |
|
||||
| `temperature` | 0.2 | 翻译稳定性，0.1-0.3为佳 |
|
||||
| `min_paragraph_length` | 30 | 过滤短段落 |
|
||||
|
||||
### 优化建议
|
||||
|
||||
#### 提高速度
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 12, // 增加并发（注意API限制）
|
||||
"chunk_size": 8000 // 更大的chunk
|
||||
}
|
||||
```
|
||||
|
||||
#### 提高质量
|
||||
```json
|
||||
{
|
||||
"temperature": 0.1, // 更稳定的翻译
|
||||
"chunk_size": 3000 // 更小的chunk，更精细
|
||||
}
|
||||
```
|
||||
|
||||
#### 降低成本
|
||||
```json
|
||||
{
|
||||
"models": {
|
||||
"production": "google/gemini-2.5-flash-lite" // 使用更便宜的模型
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🧪 测试工具
|
||||
|
||||
### 1. 测试全局ID系统
|
||||
```bash
|
||||
python test_global_id_system.py
|
||||
```
|
||||
|
||||
测试内容：
|
||||
- ✅ 段落提取和全局编号
|
||||
- ✅ 智能分块（不切断段落）
|
||||
- ✅ 带编号的LLM翻译
|
||||
- ✅ ID到翻译的精确映射
|
||||
|
||||
### 2. 测试并发逻辑
|
||||
```bash
|
||||
python test_concurrent.py
|
||||
```
|
||||
|
||||
测试内容：
|
||||
- ✅ 串行 vs 并发性能对比
|
||||
- ✅ RateLimiter并发控制
|
||||
- ✅ 加速比计算
|
||||
- ✅ 结果一致性验证
|
||||
|
||||
### 3. 测试API连接
|
||||
```bash
|
||||
python test_api.py
|
||||
```
|
||||
|
||||
## 📖 使用示例
|
||||
|
||||
### 基本翻译流程
|
||||
|
||||
```bash
|
||||
# 1. 测试API连接
|
||||
python test_api.py
|
||||
|
||||
# 2. 测试翻译（只翻译前3个段落）
|
||||
python main.py book.epub --test
|
||||
|
||||
# 3. 查看并发效果
|
||||
python test_concurrent.py
|
||||
|
||||
# 4. 完整翻译
|
||||
python main.py book.epub
|
||||
|
||||
# 输出：output/book_bilingual.epub
|
||||
```
|
||||
|
||||
### 高级用法
|
||||
|
||||
```bash
|
||||
# 清理缓存重新翻译
|
||||
python main.py --clear-cache 0
|
||||
python main.py book.epub --no-cache
|
||||
|
||||
# 查看缓存统计
|
||||
python main.py --cache-stats
|
||||
|
||||
# 指定输出目录
|
||||
python main.py book.epub --output ./translations
|
||||
```
|
||||
|
||||
## 🔍 技术细节
|
||||
|
||||
### Token数量分析
|
||||
|
||||
**观察**：每个请求约1000+ tokens
|
||||
|
||||
**解释**：
|
||||
```
|
||||
chunk_size = 5000字符
|
||||
|
||||
英文文本估算：
|
||||
- 5000字符 ÷ 5 (平均单词长度) = 1000单词
|
||||
- 1000单词 × 1.3 (tokens/word) = 1300 tokens
|
||||
- + 系统提示（~200 tokens）
|
||||
- + 格式说明（~100 tokens）
|
||||
= 约1500-1800 tokens/请求
|
||||
|
||||
这个数量是正常的！✅
|
||||
```
|
||||
|
||||
### 响应时间分析
|
||||
|
||||
**观察**：每个请求<1秒
|
||||
|
||||
**解释**：
|
||||
- Gemini 2.5 Flash 是超快模型
|
||||
- 生成速度：100+ tokens/秒
|
||||
- 1000 tokens输出 ≈ 10秒生成时间
|
||||
- 但采用流式输出，首token延迟<1秒
|
||||
- ✅ 完全正常！
|
||||
|
||||
### 并发控制原理
|
||||
|
||||
```python
|
||||
class RateLimiter:
|
||||
def __init__(self, concurrent_requests: int):
|
||||
self.semaphore = asyncio.Semaphore(concurrent_requests)
|
||||
|
||||
async def acquire(self):
|
||||
await self.semaphore.acquire() # 最多N个同时执行
|
||||
|
||||
def release(self):
|
||||
self.semaphore.release() # 释放一个槽位
|
||||
```
|
||||
|
||||
## 🚨 常见问题
|
||||
|
||||
### Q1: 翻译速度慢？
|
||||
|
||||
**原因**：并发数设置太小
|
||||
|
||||
**解决**：
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 12 // 增加到10-15
|
||||
}
|
||||
```
|
||||
|
||||
### Q2: 出现错行？
|
||||
|
||||
**原因**：旧缓存问题（已修复）
|
||||
|
||||
**解决**：
|
||||
```bash
|
||||
python main.py --clear-cache 0 # 清理旧缓存
|
||||
python main.py book.epub # 重新翻译
|
||||
```
|
||||
|
||||
### Q3: API限制错误？
|
||||
|
||||
**原因**：并发数超过API限制
|
||||
|
||||
**解决**：
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 5 // 降低并发数
|
||||
}
|
||||
```
|
||||
|
||||
### Q4: 内存占用高？
|
||||
|
||||
**原因**：大文件 + 高并发
|
||||
|
||||
**解决**：
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 4,
|
||||
"chunk_size": 3000
|
||||
}
|
||||
```
|
||||
|
||||
## 📊 性能数据
|
||||
|
||||
### 实测数据（300页书籍）
|
||||
|
||||
| 指标 | 串行模式 | 并发模式 | 提升 |
|
||||
|------|---------|---------|------|
|
||||
| 总耗时 | 15分钟 | 2分钟 | 7.5x |
|
||||
| 段落数 | 1200 | 1200 | - |
|
||||
| Chunks | 150 | 150 | - |
|
||||
| 并发数 | 1 | 8 | 8x |
|
||||
| 成功率 | 99.5% | 99.5% | 一致 |
|
||||
|
||||
## 🔧 开发计划
|
||||
|
||||
- [ ] ✅ 全局编号系统
|
||||
- [ ] ✅ 真并发翻译
|
||||
- [ ] ✅ 简化架构
|
||||
- [ ] ✅ 配置清理
|
||||
- [ ] 🚧 翻译review机制（一次性review所有译文）
|
||||
- [ ] 📋 支持更多语言对
|
||||
- [ ] 📋 Web界面
|
||||
- [ ] 📋 翻译质量评分
|
||||
|
||||
## 🤝 贡献
|
||||
|
||||
欢迎提交 Issue 和 Pull Request！
|
||||
|
||||
## 📄 许可证
|
||||
|
||||
MIT License
|
||||
|
||||
---
|
||||
|
||||
**版本**: 2.0.0 (重构版 + 真并发)
|
||||
**更新**: 2026-01-12
|
||||
**状态**: 稳定版，全局编号系统 + 真并发翻译已实现
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"llm": {
|
||||
"provider": "openrouter",
|
||||
"base_url": "https://openrouter.ai/api/v1",
|
||||
"api_key": "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",
|
||||
"models": {
|
||||
"fast": "google/gemini-2.0-flash-001",
|
||||
"smart": "google/gemini-2.0-flash-001"
|
||||
},
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 60,
|
||||
"concurrent_requests": 32
|
||||
}
|
||||
},
|
||||
"translation": {
|
||||
"chunk_size": 5000,
|
||||
"temperature": 0.3,
|
||||
"glossary": {
|
||||
"enabled": true,
|
||||
"auto_generate": true,
|
||||
"sample_size": 3000,
|
||||
"review_pause": true
|
||||
}
|
||||
},
|
||||
"processing": {
|
||||
"min_paragraph_length": 5
|
||||
},
|
||||
"output": {
|
||||
"output_dir": "output",
|
||||
"filename_suffix": "_bilingual"
|
||||
},
|
||||
"logging": {
|
||||
"level": "INFO",
|
||||
"file": "logs/translator.log",
|
||||
"rotation": "10 MB",
|
||||
"retention": "7 days"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"translation": {
|
||||
"system": "你是一位精通中英文的专业翻译家。你的任务是翻译书籍内容。\n\n要求：\n1. 准确传达原文含义，语言流畅自然，符合中文阅读习惯。\n2. 严格保持【p_xxxxx】编号格式，不要遗漏，不要修改编号。\n3. 不要添加任何解释、注释或无关内容，只返回【编号】+【译文】。\n\n{{glossary_instruction}}",
|
||||
"user_template": "请翻译以下段落：\n\n{{content}}"
|
||||
},
|
||||
"glossary_extraction": {
|
||||
"system": "你是一位资深的文学编辑和领域专家。你的任务是分析书籍样本，提取关键术语并制定统一的译名表。",
|
||||
"user_template": "请阅读以下书籍片段（包含前言和正文采样）。\n\n任务：\n1. 识别文中出现的人名（如 'Masa', 'Steve Jobs'）、地名、机构名。\n2. 识别特定的行业术语或关键概念。\n3. 为上述词汇提供标准的中文译名。如果像 'Masa' 这样的昵称有对应的全名（如孙正义），请务必使用全名。\n\n请以 JSON 格式输出，格式如下：\n{\n \"Masa\": \"孙正义\",\n \"Apple\": \"苹果公司\",\n ...\n}\n\n书籍片段：\n\n{{content}}"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,344 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
EPUB 双语翻译程序主入口
|
||||
支持命令行参数和交互式使用
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
# 添加 src 目录到 Python 路径
|
||||
sys.path.insert(0, str(Path(__file__).parent / "src"))
|
||||
|
||||
from src.translator import EPUBTranslator
|
||||
from src.utils import load_config, setup_logging
|
||||
from rich.console import Console
|
||||
from rich.panel import Panel
|
||||
from rich.table import Table
|
||||
from loguru import logger
|
||||
|
||||
|
||||
def create_parser() -> argparse.ArgumentParser:
|
||||
"""创建命令行参数解析器"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description='EPUB 双语翻译程序',
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
使用示例:
|
||||
# 测试翻译
|
||||
python main.py book.epub --test
|
||||
|
||||
# 完整翻译
|
||||
python main.py book.epub --output ./output
|
||||
|
||||
# 使用自定义配置
|
||||
python main.py book.epub --config custom_config.json
|
||||
|
||||
# 估算翻译成本
|
||||
python main.py book.epub --estimate
|
||||
|
||||
# 禁用缓存
|
||||
python main.py book.epub --no-cache
|
||||
"""
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'epub_file',
|
||||
help='输入的 EPUB 文件路径'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--test',
|
||||
action='store_true',
|
||||
help='测试模式：翻译序言和一个段落进行测试'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--config',
|
||||
default='config/config.json',
|
||||
help='配置文件路径 (默认: config/config.json)'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--output',
|
||||
help='输出目录 (默认: 配置文件中的设置)'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--estimate',
|
||||
action='store_true',
|
||||
help='估算翻译成本和时间'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--no-cache',
|
||||
action='store_true',
|
||||
help='禁用翻译缓存'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--clear-cache',
|
||||
type=int,
|
||||
metavar='DAYS',
|
||||
help='清理指定天数前的缓存文件'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--cache-stats',
|
||||
action='store_true',
|
||||
help='显示缓存统计信息'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--verbose', '-v',
|
||||
action='store_true',
|
||||
help='详细输出模式'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--version',
|
||||
action='version',
|
||||
version='EPUB Translator 0.1.0'
|
||||
)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def validate_args(args) -> None:
|
||||
"""验证命令行参数"""
|
||||
# 检查 EPUB 文件是否存在
|
||||
if hasattr(args, 'epub_file') and args.epub_file:
|
||||
epub_path = Path(args.epub_file)
|
||||
if not epub_path.exists():
|
||||
raise FileNotFoundError(f"EPUB 文件不存在: {args.epub_file}")
|
||||
|
||||
if not epub_path.suffix.lower() == '.epub':
|
||||
raise ValueError(f"文件不是 EPUB 格式: {args.epub_file}")
|
||||
|
||||
# 检查配置文件是否存在
|
||||
config_path = Path(args.config)
|
||||
if not config_path.exists():
|
||||
raise FileNotFoundError(f"配置文件不存在: {args.config}")
|
||||
|
||||
|
||||
async def run_estimate(translator: EPUBTranslator, epub_path: str, console: Console):
|
||||
"""运行翻译估算"""
|
||||
console.print("[yellow]正在估算翻译成本...[/yellow]")
|
||||
|
||||
try:
|
||||
estimate = await translator.get_translation_estimate(epub_path)
|
||||
|
||||
if not estimate:
|
||||
console.print("[red]估算失败[/red]")
|
||||
return
|
||||
|
||||
# 显示估算结果
|
||||
table = Table(title="翻译估算")
|
||||
table.add_column("项目", style="cyan")
|
||||
table.add_column("值", style="white")
|
||||
|
||||
table.add_row("总段落数", str(estimate['total_paragraphs']))
|
||||
table.add_row("章节数", str(estimate['chapters']))
|
||||
table.add_row("文本长度", f"{estimate['text_length']:,} 字符")
|
||||
table.add_row("估算 Tokens", f"{estimate['estimated_tokens']:,}")
|
||||
table.add_row("估算翻译块数", str(estimate['estimated_chunks']))
|
||||
table.add_row("块大小设置", f"{estimate['chunk_size']:,} 字符")
|
||||
table.add_row("估算时间", f"{estimate['estimated_time_minutes']:.1f} 分钟")
|
||||
|
||||
console.print(table)
|
||||
|
||||
# 成本估算（需要根据实际 API 定价调整）
|
||||
console.print("\n[yellow]注意: 实际成本取决于所选模型的定价[/yellow]")
|
||||
|
||||
except Exception as e:
|
||||
console.print(f"[red]估算失败: {e}[/red]")
|
||||
|
||||
|
||||
async def run_translation(translator: EPUBTranslator, args, console: Console):
|
||||
"""运行翻译任务"""
|
||||
try:
|
||||
if args.test:
|
||||
console.print("[blue]运行测试模式...[/blue]")
|
||||
result = await translator.translate_epub(
|
||||
args.epub_file,
|
||||
test_mode=True
|
||||
)
|
||||
|
||||
if isinstance(result, dict) and result.get('status') == 'success':
|
||||
console.print("[green]测试完成！[/green]")
|
||||
else:
|
||||
console.print("[red]测试失败[/red]")
|
||||
|
||||
else:
|
||||
console.print("[blue]开始完整翻译...[/blue]")
|
||||
|
||||
# 确认操作
|
||||
if not args.output:
|
||||
console.print("[yellow]将使用默认输出目录[/yellow]")
|
||||
|
||||
output_file = await translator.translate_epub(
|
||||
args.epub_file,
|
||||
test_mode=False,
|
||||
output_dir=args.output
|
||||
)
|
||||
|
||||
console.print(Panel(
|
||||
f"翻译完成！\n输出文件: {output_file}",
|
||||
title="成功",
|
||||
border_style="green"
|
||||
))
|
||||
|
||||
except KeyboardInterrupt:
|
||||
console.print("\n[yellow]用户中断翻译[/yellow]")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
console.print(f"[red]翻译失败: {e}[/red]")
|
||||
logger.error(f"翻译失败: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def handle_cache_operations(args, config, console: Console):
|
||||
"""处理缓存相关操作"""
|
||||
from src.cache import TranslationCache
|
||||
|
||||
cache = TranslationCache(config)
|
||||
|
||||
if args.clear_cache is not None:
|
||||
console.print(f"[yellow]清理 {args.clear_cache} 天前的缓存...[/yellow]")
|
||||
cleared = cache.clear_cache(args.clear_cache)
|
||||
console.print(f"[green]已清理 {cleared} 个缓存文件[/green]")
|
||||
return True
|
||||
|
||||
if args.cache_stats:
|
||||
console.print("[cyan]缓存统计信息:[/cyan]")
|
||||
stats = cache.get_cache_stats()
|
||||
|
||||
if stats.get('enabled'):
|
||||
table = Table()
|
||||
table.add_column("项目", style="cyan")
|
||||
table.add_column("值", style="white")
|
||||
|
||||
table.add_row("缓存状态", "启用")
|
||||
table.add_row("缓存目录", stats.get('cache_directory', ''))
|
||||
table.add_row("文件总数", str(stats.get('total_files', 0)))
|
||||
table.add_row("总大小", f"{stats.get('total_size_mb', 0)} MB")
|
||||
table.add_row("最大保存天数", f"{stats.get('max_age_days', 0)} 天")
|
||||
|
||||
console.print(table)
|
||||
|
||||
# 显示按日期分布
|
||||
date_dist = stats.get('date_distribution', {})
|
||||
if date_dist:
|
||||
console.print("\n[cyan]按日期分布:[/cyan]")
|
||||
for date, count in sorted(date_dist.items()):
|
||||
console.print(f" {date}: {count} 个文件")
|
||||
else:
|
||||
console.print("[yellow]缓存未启用[/yellow]")
|
||||
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def check_environment():
|
||||
"""检查运行环境"""
|
||||
# 检查 Python 版本
|
||||
if sys.version_info < (3, 9):
|
||||
print("错误: 需要 Python 3.9 或更高版本")
|
||||
sys.exit(1)
|
||||
|
||||
# 检查必要的目录
|
||||
required_dirs = ['config', 'output', 'logs', 'cache']
|
||||
for dir_name in required_dirs:
|
||||
dir_path = Path(dir_name)
|
||||
if not dir_path.exists():
|
||||
dir_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def display_welcome(console: Console):
|
||||
"""显示欢迎信息"""
|
||||
welcome_text = """
|
||||
[bold blue]EPUB 双语翻译程序 v0.1.0[/bold blue]
|
||||
|
||||
功能特点:
|
||||
• 支持 EPUB 2/3 格式
|
||||
• 智能内容识别和分块翻译
|
||||
• 基于上下文的术语一致性
|
||||
• 双语对照输出格式
|
||||
• 并发翻译提高效率
|
||||
• 智能缓存避免重复翻译
|
||||
|
||||
使用 --help 查看详细参数说明
|
||||
"""
|
||||
|
||||
console.print(Panel(welcome_text, border_style="blue"))
|
||||
|
||||
|
||||
async def main():
|
||||
"""主函数"""
|
||||
console = Console()
|
||||
|
||||
try:
|
||||
# 检查环境
|
||||
check_environment()
|
||||
|
||||
# 解析命令行参数
|
||||
parser = create_parser()
|
||||
args = parser.parse_args()
|
||||
|
||||
# 如果没有参数，显示帮助
|
||||
if len(sys.argv) == 1:
|
||||
display_welcome(console)
|
||||
parser.print_help()
|
||||
return
|
||||
|
||||
# 加载配置
|
||||
try:
|
||||
config = load_config(args.config)
|
||||
except Exception as e:
|
||||
console.print(f"[red]加载配置失败: {e}[/red]")
|
||||
sys.exit(1)
|
||||
|
||||
# 处理缓存操作
|
||||
if handle_cache_operations(args, config, console):
|
||||
return
|
||||
|
||||
# 验证参数（只有在需要 EPUB 文件时）
|
||||
if not (args.clear_cache is not None or args.cache_stats):
|
||||
validate_args(args)
|
||||
|
||||
# 设置日志
|
||||
if args.verbose:
|
||||
config['logging']['level'] = 'DEBUG'
|
||||
|
||||
setup_logging(config)
|
||||
logger.info("程序启动")
|
||||
|
||||
# 初始化翻译器
|
||||
use_cache = not args.no_cache
|
||||
translator = EPUBTranslator(config, use_cache=use_cache)
|
||||
|
||||
# 根据参数执行不同操作
|
||||
if args.estimate:
|
||||
await run_estimate(translator, args.epub_file, console)
|
||||
else:
|
||||
await run_translation(translator, args, console)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
console.print("\n[yellow]程序被用户中断[/yellow]")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
console.print(f"[red]程序执行失败: {e}[/red]")
|
||||
logger.error(f"程序执行失败: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# 设置事件循环策略（Windows 兼容性）
|
||||
if sys.platform.startswith('win'):
|
||||
asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
|
||||
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,9 @@
|
||||
ebooklib>=0.19
|
||||
beautifulsoup4>=4.12.0
|
||||
lxml>=4.9.0
|
||||
openai>=1.0.0
|
||||
aiohttp>=3.9.0
|
||||
pydantic>=2.0.0
|
||||
loguru>=0.7.0
|
||||
rich>=13.0.0
|
||||
asyncio-throttle>=1.0.2
|
||||
@@ -0,0 +1,24 @@
|
||||
"""
|
||||
EPUB 双语翻译程序
|
||||
主要功能模块的初始化文件
|
||||
"""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
__author__ = "Kaitan"
|
||||
|
||||
from .epub_parser import EPUBParser
|
||||
from .translator import EPUBTranslator
|
||||
from .llm_client import OpenRouterClient
|
||||
from .text_processor import TextProcessor
|
||||
from .bilingual_builder import BilingualEPUBBuilder
|
||||
from .utils import load_config, setup_logging
|
||||
|
||||
__all__ = [
|
||||
"EPUBParser",
|
||||
"EPUBTranslator",
|
||||
"OpenRouterClient",
|
||||
"TextProcessor",
|
||||
"BilingualEPUBBuilder",
|
||||
"load_config",
|
||||
"setup_logging"
|
||||
]
|
||||
@@ -0,0 +1,155 @@
|
||||
"""
|
||||
双语 EPUB 构建器模块 - 安全的EPUB构建 (Manifest 兼容版)
|
||||
"""
|
||||
|
||||
from ebooklib import epub
|
||||
import ebooklib
|
||||
from bs4 import BeautifulSoup
|
||||
from typing import Dict, List
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
import uuid
|
||||
|
||||
|
||||
class BilingualEPUBBuilder:
|
||||
"""双语 EPUB 构建器"""
|
||||
|
||||
def __init__(self, original_book, config: Dict):
|
||||
self.original_book = original_book
|
||||
self.config = config
|
||||
self.output_config = config['output']
|
||||
|
||||
def create_bilingual_epub_with_mapping(self, translation_map: Dict[str, str],
|
||||
paragraph_map: Dict[str, Dict],
|
||||
output_path: str) -> str:
|
||||
"""
|
||||
创建双语 EPUB。使用 ordered_ids 确保与 Manifest 严格一致。
|
||||
"""
|
||||
try:
|
||||
new_book = epub.EpubBook()
|
||||
self._copy_metadata(new_book)
|
||||
new_book.toc = self.original_book.toc
|
||||
|
||||
# 准备每个文件的有序ID列表
|
||||
file_ordered_ids = {}
|
||||
sorted_pids = sorted(paragraph_map.keys(), key=lambda x: int(x.split('_')[1]))
|
||||
for pid in sorted_pids:
|
||||
info = paragraph_map[pid]
|
||||
fname = info['file_name']
|
||||
if fname not in file_ordered_ids:
|
||||
file_ordered_ids[fname] = []
|
||||
file_ordered_ids[fname].append(pid)
|
||||
|
||||
processed_item_ids = set()
|
||||
item_map = {}
|
||||
|
||||
# 复制资源
|
||||
for item in self.original_book.get_items():
|
||||
if item.get_type() != ebooklib.ITEM_DOCUMENT:
|
||||
if item.id not in processed_item_ids:
|
||||
new_book.add_item(item)
|
||||
processed_item_ids.add(item.id)
|
||||
item_map[item.id] = item
|
||||
|
||||
# 重建 Spine
|
||||
new_spine = []
|
||||
for spine_id, linear in self.original_book.spine:
|
||||
item = self.original_book.get_item_with_id(spine_id)
|
||||
if not item: continue
|
||||
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT:
|
||||
file_name = item.get_name()
|
||||
if file_name in file_ordered_ids:
|
||||
new_item = self._create_bilingual_document(
|
||||
item, file_ordered_ids[file_name], translation_map
|
||||
)
|
||||
new_item.id = item.id
|
||||
else:
|
||||
new_item = item
|
||||
|
||||
if new_item.id not in processed_item_ids:
|
||||
new_book.add_item(new_item)
|
||||
processed_item_ids.add(new_item.id)
|
||||
new_spine.append(new_item)
|
||||
else:
|
||||
if item.id in item_map:
|
||||
new_spine.append(item_map[item.id])
|
||||
|
||||
new_book.spine = new_spine
|
||||
new_book.add_item(epub.EpubNcx())
|
||||
new_book.add_item(epub.EpubNav())
|
||||
|
||||
output_file = self._generate_output_filename(output_path)
|
||||
epub.write_epub(output_file, new_book, {})
|
||||
return output_file
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"创建双语 EPUB 失败: {e}", exc_info=True)
|
||||
raise
|
||||
|
||||
def _copy_metadata(self, new_book):
|
||||
try:
|
||||
for namespace, meta_dict in self.original_book.metadata.items():
|
||||
for name, values in meta_dict.items():
|
||||
for value, other in values:
|
||||
if name and hasattr(name, 'lower') and name.lower() == 'identifier': continue
|
||||
new_book.add_metadata(namespace, name, value, other)
|
||||
new_book.add_metadata('DC', 'language', 'zh-CN')
|
||||
new_book.set_identifier(f"bilingual-{uuid.uuid4().hex[:12]}")
|
||||
|
||||
cover_id_meta = self.original_book.get_metadata('OPF', 'cover')
|
||||
if cover_id_meta:
|
||||
cover_item = self.original_book.get_item_with_id(cover_id_meta[0][0])
|
||||
if cover_item:
|
||||
new_book.add_item(cover_item)
|
||||
new_book.set_cover(cover_item.get_name(), cover_item.get_content())
|
||||
except Exception as e:
|
||||
logger.error(f"元数据复制出错: {e}")
|
||||
|
||||
def _create_bilingual_document(self, original_item, ordered_ids: list, translation_map: dict):
|
||||
try:
|
||||
from .text_processor import TextProcessor
|
||||
soup = BeautifulSoup(original_item.get_content().decode('utf-8'), 'html.parser')
|
||||
self._add_style_link(soup)
|
||||
|
||||
# 使用与 TextProcessor 相同的过滤逻辑获取元素
|
||||
text_elements = TextProcessor.get_valid_text_elements(soup)
|
||||
|
||||
current_para_index = 0
|
||||
for element in text_elements:
|
||||
if TextProcessor.is_navigation_element(element): continue
|
||||
if not TextProcessor.clean_element_text(element): continue
|
||||
|
||||
if current_para_index < len(ordered_ids):
|
||||
target_id = ordered_ids[current_para_index]
|
||||
translation = translation_map.get(target_id)
|
||||
if translation:
|
||||
self._insert_translation(element, translation, soup)
|
||||
current_para_index += 1
|
||||
|
||||
new_item = epub.EpubHtml(title=original_item.title, file_name=original_item.get_name(), lang='zh-CN')
|
||||
new_item.set_content(str(soup).encode('utf-8'))
|
||||
return new_item
|
||||
except Exception as e:
|
||||
logger.error(f"创建双语文档失败 {original_item.get_name()}: {e}")
|
||||
return original_item
|
||||
|
||||
def _add_style_link(self, soup):
|
||||
head = soup.find('head')
|
||||
if head and not head.find('link', href='style/bilingual.css'):
|
||||
head.append(soup.new_tag('link', rel='stylesheet', type='text/css', href='style/bilingual.css'))
|
||||
|
||||
def _insert_translation(self, element, translation: str, soup):
|
||||
try:
|
||||
translation_p = soup.new_tag('p')
|
||||
translation_p.string = translation
|
||||
translation_p['class'] = ['translation-text', 'chinese']
|
||||
element.insert_after(translation_p)
|
||||
except: pass
|
||||
|
||||
def _generate_output_filename(self, output_path: str) -> str:
|
||||
from .utils import sanitize_filename
|
||||
title = self.original_book.get_metadata('DC', 'title')
|
||||
clean_title = sanitize_filename(title[0][0]) if title else "bilingual_book"
|
||||
Path(output_path).mkdir(parents=True, exist_ok=True)
|
||||
return str(Path(output_path) / f"{clean_title}_bilingual.epub")
|
||||
@@ -0,0 +1,225 @@
|
||||
"""
|
||||
翻译缓存管理模块 - 简化版
|
||||
基于全局ID和chunk的缓存系统
|
||||
"""
|
||||
|
||||
import json
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Dict, Optional, List
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class TranslationCache:
|
||||
"""翻译缓存管理器 - 简化版"""
|
||||
|
||||
def __init__(self, config: Dict):
|
||||
"""初始化缓存管理器"""
|
||||
self.config = config
|
||||
cache_config = config.get('cache', {})
|
||||
|
||||
self.enabled = cache_config.get('enabled', True)
|
||||
self.cache_dir = Path(cache_config.get('directory', 'cache'))
|
||||
self.max_age_days = cache_config.get('max_age_days', 30)
|
||||
|
||||
if self.enabled:
|
||||
self.cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
self.translations_dir = self.cache_dir / 'translations'
|
||||
self.translations_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.info(f"翻译缓存已启用: {self.cache_dir}")
|
||||
|
||||
def get_chunk_translation(self, chunk: List[Dict], model: str) -> Optional[Dict[str, str]]:
|
||||
"""
|
||||
获取chunk的缓存翻译
|
||||
|
||||
Args:
|
||||
chunk: 段落列表（带global_id）
|
||||
model: 模型名称
|
||||
|
||||
Returns:
|
||||
{global_id: translation} 映射，如果不存在返回 None
|
||||
"""
|
||||
if not self.enabled:
|
||||
return None
|
||||
|
||||
try:
|
||||
cache_key = self._get_chunk_cache_key(chunk, model)
|
||||
cache_file = self._get_cache_file_path(cache_key)
|
||||
|
||||
if not cache_file.exists():
|
||||
return None
|
||||
|
||||
# 检查是否过期
|
||||
file_age = datetime.now() - datetime.fromtimestamp(cache_file.stat().st_mtime)
|
||||
if file_age > timedelta(days=self.max_age_days):
|
||||
logger.debug(f"缓存已过期: {cache_key[:8]}...")
|
||||
cache_file.unlink()
|
||||
return None
|
||||
|
||||
# 读取缓存
|
||||
with open(cache_file, 'r', encoding='utf-8') as f:
|
||||
cache_data = json.load(f)
|
||||
|
||||
# 验证缓存
|
||||
if (cache_data.get('success') and
|
||||
cache_data.get('model') == model and
|
||||
self._validate_cache_data(cache_data, chunk)):
|
||||
|
||||
logger.debug(f"缓存命中: {cache_key[:8]}... ({len(chunk)} 段落)")
|
||||
return cache_data.get('translations', {})
|
||||
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"读取缓存失败: {e}")
|
||||
return None
|
||||
|
||||
def save_chunk_translation(self, chunk: List[Dict], translations: Dict[str, str],
|
||||
model: str, success: bool = True) -> None:
|
||||
"""
|
||||
保存chunk翻译到缓存
|
||||
|
||||
Args:
|
||||
chunk: 段落列表（带global_id）
|
||||
translations: {global_id: translation} 映射
|
||||
model: 模型名称
|
||||
success: 是否翻译成功
|
||||
"""
|
||||
if not self.enabled:
|
||||
return
|
||||
|
||||
try:
|
||||
cache_key = self._get_chunk_cache_key(chunk, model)
|
||||
cache_file = self._get_cache_file_path(cache_key)
|
||||
|
||||
# 构建缓存数据
|
||||
cache_data = {
|
||||
'global_ids': [p['global_id'] for p in chunk],
|
||||
'translations': translations,
|
||||
'model': model,
|
||||
'timestamp': datetime.now().isoformat(),
|
||||
'success': success,
|
||||
'paragraph_count': len(chunk),
|
||||
'cache_version': '3.0'
|
||||
}
|
||||
|
||||
with open(cache_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(cache_data, f, ensure_ascii=False, indent=2)
|
||||
|
||||
logger.debug(f"缓存已保存: {cache_key[:8]}... ({len(chunk)} 段落)")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"保存缓存失败: {e}")
|
||||
|
||||
def _get_chunk_cache_key(self, chunk: List[Dict], model: str) -> str:
|
||||
"""
|
||||
生成chunk缓存键（基于全局ID序列）
|
||||
|
||||
Args:
|
||||
chunk: 段落列表
|
||||
model: 模型名称
|
||||
|
||||
Returns:
|
||||
缓存键
|
||||
"""
|
||||
# 使用全局ID序列作为缓存键的一部分
|
||||
id_sequence = ",".join(p['global_id'] for p in chunk)
|
||||
combined = f"{id_sequence}|{model}"
|
||||
return hashlib.md5(combined.encode('utf-8')).hexdigest()
|
||||
|
||||
def _get_cache_file_path(self, cache_key: str) -> Path:
|
||||
"""获取缓存文件路径"""
|
||||
today = datetime.now().strftime('%Y-%m-%d')
|
||||
cache_date_dir = self.translations_dir / today
|
||||
cache_date_dir.mkdir(parents=True, exist_ok=True)
|
||||
return cache_date_dir / f"{cache_key}.json"
|
||||
|
||||
def _validate_cache_data(self, cache_data: Dict, chunk: List[Dict]) -> bool:
|
||||
"""验证缓存数据的有效性"""
|
||||
# 检查ID序列是否匹配
|
||||
cached_ids = cache_data.get('global_ids', [])
|
||||
chunk_ids = [p['global_id'] for p in chunk]
|
||||
|
||||
if cached_ids != chunk_ids:
|
||||
logger.debug("缓存ID序列不匹配")
|
||||
return False
|
||||
|
||||
# 检查翻译数量
|
||||
translations = cache_data.get('translations', {})
|
||||
if len(translations) != len(chunk):
|
||||
logger.debug("缓存翻译数量不匹配")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def clear_cache(self, older_than_days: Optional[int] = None) -> int:
|
||||
"""清理缓存"""
|
||||
if not self.enabled or not self.translations_dir.exists():
|
||||
return 0
|
||||
|
||||
cleared_count = 0
|
||||
cutoff_time = None
|
||||
|
||||
if older_than_days is not None:
|
||||
cutoff_time = datetime.now() - timedelta(days=older_than_days)
|
||||
|
||||
try:
|
||||
for cache_file in self.translations_dir.rglob('*.json'):
|
||||
should_delete = False
|
||||
|
||||
if cutoff_time is None:
|
||||
should_delete = True
|
||||
else:
|
||||
file_time = datetime.fromtimestamp(cache_file.stat().st_mtime)
|
||||
should_delete = file_time < cutoff_time
|
||||
|
||||
if should_delete:
|
||||
cache_file.unlink()
|
||||
cleared_count += 1
|
||||
|
||||
# 清理空目录
|
||||
for date_dir in self.translations_dir.iterdir():
|
||||
if date_dir.is_dir() and not any(date_dir.iterdir()):
|
||||
date_dir.rmdir()
|
||||
|
||||
logger.info(f"清理了 {cleared_count} 个缓存文件")
|
||||
return cleared_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"清理缓存失败: {e}")
|
||||
return 0
|
||||
|
||||
def get_cache_stats(self) -> Dict:
|
||||
"""获取缓存统计信息"""
|
||||
if not self.enabled or not self.translations_dir.exists():
|
||||
return {'enabled': False}
|
||||
|
||||
try:
|
||||
cache_files = list(self.translations_dir.rglob('*.json'))
|
||||
total_files = len(cache_files)
|
||||
total_size = sum(f.stat().st_size for f in cache_files)
|
||||
|
||||
# 统计段落数
|
||||
total_paragraphs = 0
|
||||
for cache_file in cache_files:
|
||||
try:
|
||||
with open(cache_file, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
total_paragraphs += data.get('paragraph_count', 0)
|
||||
except:
|
||||
continue
|
||||
|
||||
return {
|
||||
'enabled': True,
|
||||
'total_files': total_files,
|
||||
'total_paragraphs': total_paragraphs,
|
||||
'total_size_mb': round(total_size / 1024 / 1024, 2),
|
||||
'cache_directory': str(self.cache_dir),
|
||||
'max_age_days': self.max_age_days
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"获取缓存统计失败: {e}")
|
||||
return {'enabled': True, 'error': str(e)}
|
||||
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
EPUB 解析器模块 (EPUB Parser Module)
|
||||
|
||||
该模块负责读取 EPUB 文件，提取元数据和内容项目。
|
||||
它使用 ebooklib 库来处理 EPUB 格式的底层细节。
|
||||
|
||||
Classes:
|
||||
EPUBParser: 负责 EPUB 文件的加载、元数据提取和内容项遍历。
|
||||
"""
|
||||
|
||||
import ebooklib
|
||||
from ebooklib import epub
|
||||
from bs4 import BeautifulSoup
|
||||
from typing import List, Dict, Any
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class EPUBParser:
|
||||
"""
|
||||
EPUB 文件解析器。
|
||||
|
||||
负责加载 EPUB 文件，提取书籍元数据（如标题、作者），并提供方法来遍历和提取
|
||||
书中的文档内容（HTML/XHTML）。
|
||||
|
||||
Attributes:
|
||||
epub_path (Path): EPUB 文件的路径对象。
|
||||
book (epub.EpubBook): ebooklib 加载的书籍对象。
|
||||
metadata (Dict[str, str]): 提取的书籍元数据字典。
|
||||
"""
|
||||
|
||||
def __init__(self, epub_path: str):
|
||||
"""
|
||||
初始化 EPUB 解析器。
|
||||
|
||||
Args:
|
||||
epub_path (str): EPUB 文件的文件路径。
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: 如果指定的文件不存在。
|
||||
Exception: 如果 EPUB 文件加载失败（格式错误等）。
|
||||
"""
|
||||
self.epub_path = Path(epub_path)
|
||||
if not self.epub_path.exists():
|
||||
raise FileNotFoundError(f"EPUB 文件不存在: {epub_path}")
|
||||
|
||||
try:
|
||||
# ignore_ncx=True 是为了避免某些旧版 epub 的警告，但新版 ebooklib 可能行为不同
|
||||
# 这里直接读取，让 ebooklib 处理
|
||||
self.book = epub.read_epub(str(self.epub_path))
|
||||
logger.info(f"成功加载 EPUB: {self.epub_path.name}")
|
||||
except Exception as e:
|
||||
logger.error(f"加载 EPUB 失败: {e}")
|
||||
raise
|
||||
|
||||
self.metadata = self._extract_metadata()
|
||||
|
||||
def _extract_metadata(self) -> Dict[str, str]:
|
||||
"""
|
||||
从 EPUB 对象中提取标准元数据。
|
||||
|
||||
提取 Dublin Core (DC) 元数据，包括标题、作者和语言。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 包含 'title', 'author', 'language' 的字典。
|
||||
如果提取失败，会使用默认值 ("Unknown", "en")。
|
||||
"""
|
||||
metadata = {}
|
||||
|
||||
try:
|
||||
# get_metadata 返回的是 (value, dict) 的列表，我们取第一个结果
|
||||
title_meta = self.book.get_metadata('DC', 'title')
|
||||
metadata['title'] = title_meta[0][0] if title_meta else "Unknown"
|
||||
|
||||
author_meta = self.book.get_metadata('DC', 'creator')
|
||||
metadata['author'] = author_meta[0][0] if author_meta else "Unknown"
|
||||
|
||||
lang_meta = self.book.get_metadata('DC', 'language')
|
||||
metadata['language'] = lang_meta[0][0] if lang_meta else "en"
|
||||
|
||||
logger.info(f"书籍: {metadata['title']} - {metadata['author']}")
|
||||
except Exception as e:
|
||||
logger.warning(f"提取元数据时出错: {e}")
|
||||
# 设置保底值
|
||||
metadata.setdefault('title', 'Unknown')
|
||||
metadata.setdefault('author', 'Unknown')
|
||||
metadata.setdefault('language', 'en')
|
||||
|
||||
return metadata
|
||||
|
||||
def extract_all_content_items(self) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
提取所有可翻译的内容项目（文档）。
|
||||
|
||||
遍历 EPUB 中的所有 Item，筛选出类型为 ITEM_DOCUMENT 的项目。
|
||||
同时会进行简单的过滤，跳过内容过短（<100字符）或看起来像非正文的文件（如 nav, toc, cover）。
|
||||
|
||||
Returns:
|
||||
List[Dict[str, Any]]: 内容项目列表。每个字典包含：
|
||||
- item (epub.EpubItem): 原始 Item 对象。
|
||||
- file_name (str): 文件名。
|
||||
- content (str): 解码后的 HTML 内容。
|
||||
- text_length (int): 纯文本长度（用于统计）。
|
||||
"""
|
||||
content_items = []
|
||||
|
||||
# 获取所有文档类型的项目
|
||||
for item in self.book.get_items():
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT:
|
||||
try:
|
||||
# 获取内容 (bytes -> str)
|
||||
content = item.get_content().decode('utf-8')
|
||||
|
||||
# 简单的内容验证：提取纯文本检查长度
|
||||
soup = BeautifulSoup(content, 'html.parser')
|
||||
text = soup.get_text().strip()
|
||||
|
||||
# 1. 跳过太短的内容（可能是只有图片的页面、空页面）
|
||||
if len(text) < 100:
|
||||
logger.debug(f"跳过短内容: {item.get_name()} ({len(text)} 字符)")
|
||||
continue
|
||||
|
||||
# 2. 跳过明显的非正文内容 (根据文件名判断)
|
||||
name_lower = item.get_name().lower()
|
||||
skip_patterns = ['cover', 'copyright', 'titlepage', 'halftitle',
|
||||
'nav.xhtml', 'toc.xhtml']
|
||||
if any(pattern in name_lower for pattern in skip_patterns):
|
||||
logger.debug(f"跳过非正文内容: {item.get_name()}")
|
||||
continue
|
||||
|
||||
content_items.append({
|
||||
'item': item,
|
||||
'file_name': item.get_name(),
|
||||
'content': content,
|
||||
'text_length': len(text)
|
||||
})
|
||||
|
||||
logger.debug(f"添加内容项: {item.get_name()} ({len(text)} 字符)")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"处理项目失败 {item.get_name()}: {e}")
|
||||
continue
|
||||
|
||||
logger.info(f"提取了 {len(content_items)} 个内容项目")
|
||||
return content_items
|
||||
|
||||
def get_book_info(self) -> Dict[str, str]:
|
||||
"""
|
||||
获取书籍的摘要信息。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 包含文件名、标题、作者、语言和文档数量的字典。
|
||||
"""
|
||||
# 统计内容项
|
||||
document_count = sum(1 for item in self.book.get_items()
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT)
|
||||
|
||||
return {
|
||||
'filename': self.epub_path.name,
|
||||
'title': self.metadata.get('title', 'Unknown'),
|
||||
'author': self.metadata.get('author', 'Unknown'),
|
||||
'language': self.metadata.get('language', 'en'),
|
||||
'document_count': document_count
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
"""
|
||||
术语表管理器 (Glossary Manager)
|
||||
|
||||
负责从书籍内容中提取采样文本，调用 LLM 生成术语表，并管理术语表的持久化。
|
||||
"""
|
||||
|
||||
import json
|
||||
import random
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Any
|
||||
from loguru import logger
|
||||
from .manifest_manager import ManifestManager
|
||||
from .llm_client import OpenRouterClient
|
||||
|
||||
class GlossaryManager:
|
||||
def __init__(self, config: Dict, llm_client: OpenRouterClient):
|
||||
self.config = config
|
||||
self.llm_client = llm_client
|
||||
self.glossary_path = Path("cache/glossary.json")
|
||||
self.prompts = self._load_prompts()
|
||||
|
||||
def _load_prompts(self) -> Dict:
|
||||
try:
|
||||
with open("config/prompts.json", "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except Exception:
|
||||
logger.warning("未找到 config/prompts.json，使用默认 Prompt")
|
||||
return {}
|
||||
|
||||
def extract_samples(self, manifest: ManifestManager, sample_size: int = 3000) -> str:
|
||||
"""
|
||||
从 Manifest 中提取采样文本。
|
||||
策略：
|
||||
1. 优先提取前言/绪论 (通常在文件的前部)。
|
||||
2. 随机抽取中间段落。
|
||||
"""
|
||||
all_items = manifest.get_items()
|
||||
if not all_items:
|
||||
return ""
|
||||
|
||||
# 1. 提取开头部分 (Preface/Intro) - 假设在前 50 个段落中
|
||||
intro_sample = [item.clean_text for item in all_items[:50] if len(item.clean_text) > 50]
|
||||
|
||||
# 2. 随机提取正文
|
||||
body_items = [item for item in all_items[50:] if len(item.clean_text) > 50]
|
||||
random_sample = []
|
||||
if body_items:
|
||||
# 随机取 10 个片段
|
||||
sample_count = min(10, len(body_items))
|
||||
random_items = random.sample(body_items, sample_count)
|
||||
random_sample = [item.clean_text for item in random_items]
|
||||
|
||||
# 组合并截断
|
||||
full_text = "\n\n".join(intro_sample + random_sample)
|
||||
if len(full_text) > sample_size:
|
||||
full_text = full_text[:sample_size] + "..."
|
||||
|
||||
return full_text
|
||||
|
||||
async def generate_glossary(self, manifest: ManifestManager) -> Dict[str, str]:
|
||||
"""
|
||||
生成术语表。
|
||||
"""
|
||||
# 1. 采样
|
||||
sample_text = self.extract_samples(manifest)
|
||||
if not sample_text:
|
||||
logger.warning("采样文本为空，跳过术语表生成")
|
||||
return {}
|
||||
|
||||
logger.info(f"提取了 {len(sample_text)} 字符的采样文本，正在生成术语表...")
|
||||
|
||||
# 2. 构建 Prompt
|
||||
prompt_cfg = self.prompts.get("glossary_extraction", {})
|
||||
system_prompt = prompt_cfg.get("system", "Analyze the text and extract named entities.")
|
||||
user_template = prompt_cfg.get("user_template", "Text:\n{{content}}")
|
||||
user_prompt = user_template.replace("{{content}}", sample_text)
|
||||
|
||||
# 3. 调用 LLM (使用 smart 模型)
|
||||
# 注意：这里需要 LLMClient 支持直接传入 system/user prompt，而不是封装好的 translate 接口
|
||||
# 我们稍后会扩展 LLMClient
|
||||
try:
|
||||
response = await self.llm_client.raw_chat_completion(
|
||||
system_prompt,
|
||||
user_prompt,
|
||||
model_type="smart"
|
||||
)
|
||||
|
||||
# 4. 解析 JSON
|
||||
# 简单的 JSON 提取逻辑 (处理可能的 markdown code block)
|
||||
json_str = response.strip()
|
||||
if "```json" in json_str:
|
||||
json_str = json_str.split("```json")[1].split("```")[0].strip()
|
||||
elif "```" in json_str:
|
||||
json_str = json_str.split("```")[1].split("```")[0].strip()
|
||||
|
||||
glossary = json.loads(json_str)
|
||||
self.save_glossary(glossary)
|
||||
return glossary
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"术语表生成失败: {e}")
|
||||
return {}
|
||||
|
||||
def save_glossary(self, glossary: Dict[str, str]):
|
||||
self.glossary_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(self.glossary_path, "w", encoding="utf-8") as f:
|
||||
json.dump(glossary, f, ensure_ascii=False, indent=2)
|
||||
logger.info(f"术语表已保存至: {self.glossary_path}")
|
||||
|
||||
def load_glossary(self) -> Dict[str, str]:
|
||||
if self.glossary_path.exists():
|
||||
try:
|
||||
with open(self.glossary_path, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except:
|
||||
pass
|
||||
return {}
|
||||
@@ -0,0 +1,138 @@
|
||||
"""
|
||||
LLM Client Module - Minimal ID Version
|
||||
|
||||
Principles:
|
||||
1. Pure p_xxxxx ID format.
|
||||
2. Direct string finding and slicing for parsing.
|
||||
3. No complex regex.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from openai import AsyncOpenAI
|
||||
from typing import List, Dict, Optional, Any
|
||||
from loguru import logger
|
||||
import time
|
||||
from .manifest_manager import ManifestItem
|
||||
|
||||
|
||||
class RateLimiter:
|
||||
"""Rate limiter for concurrency and RPM."""
|
||||
def __init__(self, requests_per_minute: int, concurrent_requests: int):
|
||||
self.semaphore = asyncio.Semaphore(concurrent_requests)
|
||||
self.min_interval = 60.0 / requests_per_minute if requests_per_minute > 0 else 0
|
||||
self.last_request_time = 0
|
||||
|
||||
async def acquire(self):
|
||||
await self.semaphore.acquire()
|
||||
current_time = time.time()
|
||||
wait_time = self.min_interval - (current_time - self.last_request_time)
|
||||
if wait_time > 0:
|
||||
await asyncio.sleep(wait_time)
|
||||
self.last_request_time = time.time()
|
||||
|
||||
def release(self):
|
||||
self.semaphore.release()
|
||||
|
||||
|
||||
class OpenRouterClient:
|
||||
"""Minimal ID Client."""
|
||||
|
||||
def __init__(self, config: Dict):
|
||||
self.config = config
|
||||
or_config = config["llm"]
|
||||
api_key = or_config.get("api_key")
|
||||
if not api_key or api_key == "YOUR_OPENROUTER_API_KEY":
|
||||
raise ValueError("Invalid OpenRouter API Key")
|
||||
|
||||
self.client = AsyncOpenAI(
|
||||
base_url=or_config["base_url"],
|
||||
api_key=api_key,
|
||||
default_headers={"HTTP-Referer": "https://github.com/epub-translator", "X-Title": "EPUB Translator"}
|
||||
)
|
||||
self.models = or_config["models"]
|
||||
self.rate_limiter = RateLimiter(
|
||||
or_config["rate_limits"]["requests_per_minute"],
|
||||
or_config["rate_limits"]["concurrent_requests"]
|
||||
)
|
||||
|
||||
async def translate_chunk(self, items: List[ManifestItem], glossary: Dict = None, model_type: str = "fast") -> Dict[str, str]:
|
||||
"""Translate a chunk of paragraphs."""
|
||||
if not items: return {}
|
||||
|
||||
glossary_text = ""
|
||||
if glossary:
|
||||
glossary_text = "\nGlossary:\n" + "\n".join([f"{k} -> {v}" for k, v in glossary.items()])
|
||||
|
||||
system_prompt = f"You are a professional translator. Translate segments into Chinese. {glossary_text}\n\nRequirements:\n1. Each line MUST start with the ID (p_xxxxx) followed by the translation.\n2. DO NOT modify the ID or add brackets/colons to it.\n3. Return only the translations."
|
||||
|
||||
user_prompt = "Content:\n" + "\n".join([f"{i.global_id} {i.clean_text}" for i in items])
|
||||
|
||||
model = self.models.get(model_type, self.models.get("fast"))
|
||||
|
||||
try:
|
||||
raw_response = await self._make_request(model, system_prompt, user_prompt)
|
||||
if not raw_response:
|
||||
return {item.global_id: f"[Error - Empty Response]" for item in items}
|
||||
|
||||
return self._simple_parse(raw_response, items)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Translation request failed: {e}")
|
||||
return {item.global_id: f"[Error - {str(e)}]" for item in items}
|
||||
|
||||
async def raw_chat_completion(self, system_prompt: str, user_prompt: str, model_type: str = "smart") -> str:
|
||||
"""Generic chat completion."""
|
||||
model = self.models.get(model_type, self.models.get("smart"))
|
||||
return await self._make_request(model, system_prompt, user_prompt)
|
||||
|
||||
def _simple_parse(self, response: str, items: List[ManifestItem]) -> Dict[str, str]:
|
||||
"""Simple parsing based on ID anchors."""
|
||||
results = {}
|
||||
for i, item in enumerate(items):
|
||||
current_id = item.global_id
|
||||
|
||||
start_idx = response.find(current_id)
|
||||
if start_idx == -1: continue
|
||||
|
||||
end_idx = len(response)
|
||||
if i + 1 < len(items):
|
||||
next_id = items[i+1].global_id
|
||||
next_found = response.find(next_id, start_idx + len(current_id))
|
||||
if next_found != -1:
|
||||
end_idx = next_found
|
||||
|
||||
content = response[start_idx:end_idx].strip()
|
||||
clean_content = content[len(current_id):].strip()
|
||||
clean_content = clean_content.lstrip(":： ")
|
||||
|
||||
if clean_content:
|
||||
results[current_id] = clean_content
|
||||
|
||||
if len(results) < len(items):
|
||||
for line in response.split("\n"):
|
||||
line = line.strip()
|
||||
for item in items:
|
||||
if item.global_id not in results and line.startswith(item.global_id):
|
||||
res = line[len(item.global_id):].strip().lstrip(":： ")
|
||||
if res: results[item.global_id] = res
|
||||
|
||||
return results
|
||||
|
||||
async def _make_request(self, model: str, system_prompt: str, user_prompt: str) -> str:
|
||||
await self.rate_limiter.acquire()
|
||||
try:
|
||||
resp = await self.client.chat.completions.create(
|
||||
model=model,
|
||||
messages=[
|
||||
{"role": "system", "content": system_prompt},
|
||||
{"role": "user", "content": user_prompt}
|
||||
],
|
||||
temperature=0.3,
|
||||
max_tokens=8000
|
||||
)
|
||||
return resp.choices[0].message.content.strip()
|
||||
finally:
|
||||
self.rate_limiter.release()
|
||||
|
||||
async def close(self):
|
||||
await self.client.close()
|
||||
@@ -0,0 +1,149 @@
|
||||
"""
|
||||
Manifest 管理器模块 (Manifest Manager Module)
|
||||
|
||||
该模块是系统的单一真理源 (SSOT)。
|
||||
它记录了每一段文本的原始状态、清洗后的文本、哈希值以及翻译状态。
|
||||
所有对翻译流程的操作（提取、翻译、回填）都必须通过修改此 Manifest 进行。
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import hashlib
|
||||
from typing import List, Dict, Optional, Any
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
from dataclasses import dataclass, asdict, field
|
||||
|
||||
@dataclass
|
||||
class ManifestItem:
|
||||
"""代表一个翻译单元（通常是一个段落）"""
|
||||
global_id: str
|
||||
source_file: str
|
||||
original_html: str
|
||||
clean_text: str
|
||||
text_hash: str
|
||||
tag: str
|
||||
translation: Optional[str] = None
|
||||
status: str = "pending" # pending, translated, ignored, failed
|
||||
error_msg: Optional[str] = None
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def to_dict(self):
|
||||
return asdict(self)
|
||||
|
||||
class ManifestManager:
|
||||
"""
|
||||
负责 Manifest 的生命周期管理。
|
||||
"""
|
||||
|
||||
def __init__(self, manifest_path: str):
|
||||
self.manifest_path = Path(manifest_path)
|
||||
self.data: Dict[str, Any] = {
|
||||
"book_id": "",
|
||||
"metadata": {},
|
||||
"items": []
|
||||
}
|
||||
self._items_by_id: Dict[str, ManifestItem] = {}
|
||||
|
||||
def load(self) -> bool:
|
||||
"""从文件加载 Manifest。如果文件不存在则返回 False。"""
|
||||
if self.manifest_path.exists():
|
||||
try:
|
||||
with open(self.manifest_path, 'r', encoding='utf-8') as f:
|
||||
self.data = json.load(f)
|
||||
|
||||
# 重建对象映射
|
||||
self._items_by_id = {
|
||||
item['global_id']: ManifestItem(**item)
|
||||
for item in self.data["items"]
|
||||
}
|
||||
logger.info(f"成功从 {self.manifest_path} 加载 Manifest, 包含 {len(self._items_by_id)} 个项目")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"加载 Manifest 失败: {e}")
|
||||
return False
|
||||
return False
|
||||
|
||||
def save(self):
|
||||
"""将当前状态保存到 Manifest 文件。"""
|
||||
# 确保目录存在
|
||||
self.manifest_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 同步 items 到 data 字典
|
||||
self.data["items"] = [item.to_dict() for item in self._items_by_id.values()]
|
||||
|
||||
with open(self.manifest_path, 'w', encoding='utf-8') as f:
|
||||
json.dump(self.data, f, ensure_ascii=False, indent=2)
|
||||
# logger.debug(f"Manifest 已保存到 {self.manifest_path}")
|
||||
|
||||
def init_manifest(self, book_id: str, metadata: Dict):
|
||||
"""初始化一个新的 Manifest。"""
|
||||
self.data = {
|
||||
"book_id": book_id,
|
||||
"metadata": metadata,
|
||||
"items": []
|
||||
}
|
||||
self._items_by_id = {}
|
||||
self.save()
|
||||
|
||||
def add_item(self, source_file: str, original_html: str, clean_text: str, tag: str, metadata: Dict = None) -> ManifestItem:
|
||||
"""添加一个新的翻译项并分配 ID。"""
|
||||
# 生成全局 ID
|
||||
new_index = len(self._items_by_id) + 1
|
||||
global_id = f"p_{new_index:05d}"
|
||||
|
||||
# 生成内容哈希 (用于排重和缓存)
|
||||
text_hash = hashlib.sha256(clean_text.encode('utf-8')).hexdigest()
|
||||
|
||||
item = ManifestItem(
|
||||
global_id=global_id,
|
||||
source_file=source_file,
|
||||
original_html=original_html,
|
||||
clean_text=clean_text,
|
||||
text_hash=text_hash,
|
||||
tag=tag,
|
||||
metadata=metadata or {}
|
||||
)
|
||||
|
||||
self._items_by_id[global_id] = item
|
||||
return item
|
||||
|
||||
def get_items(self, status: str = None, file_name: str = None) -> List[ManifestItem]:
|
||||
"""按状态或文件名查询项目。"""
|
||||
items = list(self._items_by_id.values())
|
||||
if status:
|
||||
items = [i for i in items if i.status == status]
|
||||
if file_name:
|
||||
items = [i for i in items if i.source_file == file_name]
|
||||
|
||||
# 必须按 ID 顺序返回以保证分块正确
|
||||
return sorted(items, key=lambda x: x.global_id)
|
||||
|
||||
def update_item(self, global_id: str, translation: str, status: str = "translated", error: str = None):
|
||||
"""更新翻译结果。"""
|
||||
if global_id in self._items_by_id:
|
||||
item = self._items_by_id[global_id]
|
||||
item.translation = translation
|
||||
item.status = status
|
||||
item.error_msg = error
|
||||
else:
|
||||
logger.warning(f"尝试更新不存在的 ID: {global_id}")
|
||||
|
||||
@property
|
||||
def stats(self) -> Dict:
|
||||
"""获取翻译进度统计。"""
|
||||
total = len(self._items_by_id)
|
||||
if total == 0: return {"progress": "0%"}
|
||||
|
||||
translated = sum(1 for i in self._items_by_id.values() if i.status == "translated")
|
||||
ignored = sum(1 for i in self._items_by_id.values() if i.status == "ignored")
|
||||
failed = sum(1 for i in self._items_by_id.values() if i.status == "failed")
|
||||
|
||||
return {
|
||||
"total": total,
|
||||
"translated": translated,
|
||||
"ignored": ignored,
|
||||
"failed": failed,
|
||||
"pending": total - translated - ignored - failed,
|
||||
"progress_percent": round((translated + ignored) / total * 100, 1)
|
||||
}
|
||||
@@ -0,0 +1,161 @@
|
||||
"""
|
||||
文本处理器模块 (Text Processor Module) - Manifest 驱动版
|
||||
|
||||
该模块专注于 HTML 文档的遍历和段落提取。
|
||||
它不再维护全局状态，而是将提取的内容注册到 ManifestManager 中。
|
||||
"""
|
||||
|
||||
import re
|
||||
from bs4 import BeautifulSoup
|
||||
from typing import List, Dict, Any
|
||||
from loguru import logger
|
||||
from .manifest_manager import ManifestManager
|
||||
|
||||
|
||||
class TextProcessor:
|
||||
"""
|
||||
负责从 HTML 中识别有效段落并进行清洗。
|
||||
"""
|
||||
|
||||
def __init__(self, config: Dict):
|
||||
"""
|
||||
Args:
|
||||
config (Dict): 全局配置。
|
||||
"""
|
||||
self.config = config
|
||||
self.chunk_size = config['translation'].get('chunk_size', 5000)
|
||||
|
||||
def extract_to_manifest(self, html_content: str, source_file: str, manifest: ManifestManager):
|
||||
"""
|
||||
解析 HTML 内容，并将识别出的段落注册到 Manifest 中。
|
||||
|
||||
Args:
|
||||
html_content (str): HTML 源码。
|
||||
source_file (str): 来源文件名。
|
||||
manifest (ManifestManager): 清单管理器实例。
|
||||
"""
|
||||
try:
|
||||
soup = BeautifulSoup(html_content, 'html.parser')
|
||||
|
||||
# 1. 移除不需要的元素
|
||||
for element in soup(['script', 'style', 'meta', 'link']):
|
||||
element.decompose()
|
||||
|
||||
# 2. 获取有效的文本元素 (使用静态过滤逻辑)
|
||||
text_elements = self.get_valid_text_elements(soup)
|
||||
|
||||
# 3. 注册到 Manifest
|
||||
for element in text_elements:
|
||||
clean_text = self.clean_element_text(element)
|
||||
|
||||
# 过滤逻辑
|
||||
if not clean_text:
|
||||
continue
|
||||
|
||||
status = "pending"
|
||||
# 如果是导航元素，标记为 ignored
|
||||
if self.is_navigation_element(element):
|
||||
status = "ignored"
|
||||
|
||||
# 注册
|
||||
manifest.add_item(
|
||||
source_file=source_file,
|
||||
original_html=str(element),
|
||||
clean_text=clean_text,
|
||||
tag=element.name,
|
||||
metadata={"status": status} # 临时传递给 manifest
|
||||
)
|
||||
|
||||
# 同步更新 manifest 状态 (如果需要过滤)
|
||||
if status == "ignored":
|
||||
last_id = f"p_{len(manifest._items_by_id):05d}"
|
||||
manifest.update_item(last_id, translation=None, status="ignored")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"从 {source_file} 提取段落失败: {e}")
|
||||
|
||||
@staticmethod
|
||||
def get_valid_text_elements(soup) -> List:
|
||||
"""获取不含嵌套子块的叶子级文本容器元素。"""
|
||||
tags = ['p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'blockquote', 'li', 'td']
|
||||
all_candidates = soup.find_all(tags)
|
||||
candidate_set = set(all_candidates)
|
||||
|
||||
final_elements = []
|
||||
for element in all_candidates:
|
||||
# 如果包含其他候选标签，说明是容器，跳过
|
||||
if any(d in candidate_set for d in element.find_all(tags)):
|
||||
continue
|
||||
final_elements.append(element)
|
||||
return final_elements
|
||||
|
||||
@staticmethod
|
||||
def clean_element_text(element) -> str:
|
||||
"""清理 HTML 元素，提取纯净的待翻译文本。"""
|
||||
element_copy = element.__copy__()
|
||||
|
||||
# 移除脚注引用等
|
||||
for tag in element_copy.find_all(['sup', 'sub']):
|
||||
tag.decompose()
|
||||
|
||||
footnote_patterns = re.compile(r'footnote|endnote|reference|note|super|sub', re.I)
|
||||
for tag in element_copy.find_all(['a', 'span', 'div'], class_=footnote_patterns):
|
||||
tag.decompose()
|
||||
|
||||
# 移除仅包含数字的 span
|
||||
for tag in element_copy.find_all('span'):
|
||||
if re.match(r'^(\[\d+\]|\(\d+\)|\d+)$', tag.get_text().strip()):
|
||||
tag.decompose()
|
||||
|
||||
text = element_copy.get_text().strip()
|
||||
# 正则清理残留引用标识 (如 sentence.2)
|
||||
text = re.sub(r'(\.|。|，|,)\s*(\[\d+\]|\d+)(?=\s|$)', r'\1', text)
|
||||
text = re.sub(r'\s+', ' ', text)
|
||||
return text
|
||||
|
||||
@staticmethod
|
||||
def is_navigation_element(element) -> bool:
|
||||
"""判断是否是无翻译价值的导航、页码元素。"""
|
||||
classes = element.get('class', [])
|
||||
nav_classes = ['nav', 'navigation', 'toc', 'menu', 'header', 'footer', 'page-number']
|
||||
class_str = ' '.join(classes).lower() if isinstance(classes, list) else str(classes).lower()
|
||||
|
||||
if any(nc in class_str for nc in nav_classes):
|
||||
return True
|
||||
|
||||
# 检查父级
|
||||
parent = element.parent
|
||||
if parent:
|
||||
p_classes = parent.get('class', [])
|
||||
p_class_str = ' '.join(p_classes).lower() if isinstance(p_classes, list) else str(p_classes).lower()
|
||||
if any(nc in p_class_str for nc in nav_classes):
|
||||
return True
|
||||
return False
|
||||
|
||||
def create_chunks_from_manifest(self, manifest: ManifestManager) -> List[List[Any]]:
|
||||
"""
|
||||
从 Manifest 中筛选待翻译项目并分块。
|
||||
"""
|
||||
pending_items = manifest.get_items(status="pending")
|
||||
if not pending_items:
|
||||
return []
|
||||
|
||||
chunks = []
|
||||
current_chunk = []
|
||||
current_size = 0
|
||||
|
||||
for item in pending_items:
|
||||
text_len = len(item.clean_text)
|
||||
if current_size + text_len > self.chunk_size and current_chunk:
|
||||
chunks.append(current_chunk)
|
||||
current_chunk = []
|
||||
current_size = 0
|
||||
|
||||
current_chunk.append(item)
|
||||
current_size += text_len
|
||||
|
||||
if current_chunk:
|
||||
chunks.append(current_chunk)
|
||||
|
||||
logger.info(f"分块完成: 共有 {len(pending_items)} 个待翻译项，分为 {len(chunks)} 个块")
|
||||
return chunks
|
||||
@@ -0,0 +1,149 @@
|
||||
"""
|
||||
EPUB 翻译器核心模块 (EPUB Translator Core Module) - v0.03
|
||||
|
||||
集成 Glossary 流程和配置化 LLM。
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
from typing import List, Dict, Any
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
from rich.console import Console
|
||||
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TimeElapsedColumn
|
||||
|
||||
from .epub_parser import EPUBParser
|
||||
from .llm_client import OpenRouterClient
|
||||
from .text_processor import TextProcessor
|
||||
from .bilingual_builder import BilingualEPUBBuilder
|
||||
from .manifest_manager import ManifestManager
|
||||
from .glossary_manager import GlossaryManager
|
||||
|
||||
|
||||
class EPUBTranslator:
|
||||
|
||||
def __init__(self, config: Dict, use_cache: bool = True):
|
||||
self.config = config
|
||||
self.console = Console()
|
||||
self.use_cache = use_cache
|
||||
|
||||
self.parser = None
|
||||
self.llm_client = OpenRouterClient(config)
|
||||
self.text_processor = TextProcessor(config)
|
||||
self.glossary_manager = GlossaryManager(config, self.llm_client)
|
||||
|
||||
self.manifest_dir = Path("cache/manifests")
|
||||
self.manifest_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
async def translate_epub(self, epub_path: str, test_mode: bool = False, output_dir: str = None) -> str:
|
||||
epub_path = Path(epub_path)
|
||||
self.parser = EPUBParser(str(epub_path))
|
||||
|
||||
# 1. 准备 Manifest
|
||||
manifest_path = self.manifest_dir / f"{epub_path.stem}_manifest.json"
|
||||
manifest = ManifestManager(str(manifest_path))
|
||||
|
||||
if not manifest.load() or not self.use_cache:
|
||||
self.console.print("[yellow]初始化翻译清单...[/yellow]")
|
||||
manifest.init_manifest(book_id=epub_path.name, metadata=self.parser.get_book_info())
|
||||
content_items = self.parser.extract_all_content_items()
|
||||
for item in content_items:
|
||||
self.text_processor.extract_to_manifest(item['content'], item['file_name'], manifest)
|
||||
manifest.save()
|
||||
|
||||
stats = manifest.stats
|
||||
self.console.print(f"[green]清单加载完毕: {stats['total']} 段落, 进度 {stats['progress_percent']}%[/green]")
|
||||
|
||||
# 2. 术语表处理 (仅在非测试模式且未完成时)
|
||||
glossary = {}
|
||||
if not test_mode and self.config['translation']['glossary']['enabled']:
|
||||
glossary = await self._handle_glossary(manifest)
|
||||
|
||||
# 3. 翻译
|
||||
if test_mode:
|
||||
pending = manifest.get_items(status="pending")[:5]
|
||||
if pending:
|
||||
results = await self.llm_client.translate_chunk(pending, glossary, model_type="fast")
|
||||
for pid, trans in results.items():
|
||||
self.console.print(f"\n[cyan]{pid}[/cyan]: {trans}")
|
||||
return "test_mode_done"
|
||||
|
||||
chunks = self.text_processor.create_chunks_from_manifest(manifest)
|
||||
if chunks:
|
||||
await self._translate_concurrently(chunks, manifest, glossary)
|
||||
|
||||
# 4. 构建
|
||||
self.console.print("\n[yellow]正在构建双语 EPUB...[/yellow]")
|
||||
output_path = output_dir or self.config['output']['output_dir']
|
||||
builder = BilingualEPUBBuilder(self.parser.book, self.config)
|
||||
|
||||
translation_map = {item.global_id: item.translation for item in manifest.get_items() if item.translation}
|
||||
paragraph_map = {item.global_id: {
|
||||
"file_name": item.source_file,
|
||||
"text": item.clean_text,
|
||||
"html_element": item.original_html
|
||||
} for item in manifest.get_items()}
|
||||
|
||||
result_file = builder.create_bilingual_epub_with_mapping(
|
||||
translation_map, paragraph_map, output_path
|
||||
)
|
||||
|
||||
self.console.print(f"[green]✅ 翻译完成！输出文件: {result_file}[/green]")
|
||||
return result_file
|
||||
|
||||
async def _handle_glossary(self, manifest: ManifestManager) -> Dict[str, str]:
|
||||
"""处理术语表逻辑：加载 -> 生成 -> 确认。"""
|
||||
# 尝试加载
|
||||
glossary = self.glossary_manager.load_glossary()
|
||||
|
||||
if not glossary and self.config['translation']['glossary']['auto_generate']:
|
||||
self.console.print("[yellow]正在生成术语表 (使用 Smart 模型)...[/yellow]")
|
||||
glossary = await self.glossary_manager.generate_glossary(manifest)
|
||||
|
||||
# 展示并暂停
|
||||
self.console.print("\n[bold cyan]术语表已生成:[/bold cyan]")
|
||||
self.console.print(json.dumps(glossary, indent=2, ensure_ascii=False))
|
||||
|
||||
if self.config['translation']['glossary'].get('review_pause', False):
|
||||
self.console.print(f"\n[bold red]请检查或编辑: {self.glossary_manager.glossary_path}[/bold red]")
|
||||
self.console.print("编辑完成后，按 Enter 继续，或 Ctrl+C 退出...")
|
||||
await asyncio.get_event_loop().run_in_executor(None, sys.stdin.readline)
|
||||
# 重新加载用户修改后的
|
||||
glossary = self.glossary_manager.load_glossary()
|
||||
|
||||
return glossary
|
||||
|
||||
async def _translate_concurrently(self, chunks: List[List[Any]], manifest: ManifestManager, glossary: Dict):
|
||||
total_chunks = len(chunks)
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
BarColumn(),
|
||||
TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
|
||||
TimeElapsedColumn(),
|
||||
console=self.console
|
||||
) as progress:
|
||||
task_id = progress.add_task(f"[cyan]并行翻译...", total=total_chunks)
|
||||
semaphore = self.llm_client.rate_limiter.semaphore
|
||||
|
||||
async def worker(chunk, idx):
|
||||
async with semaphore:
|
||||
try:
|
||||
# 可以在这里加入模型分级策略
|
||||
# 例如: if len(chunk) > 50: model="fast" else: model="smart"
|
||||
results = await self.llm_client.translate_chunk(chunk, glossary, model_type="fast")
|
||||
for item in chunk:
|
||||
if item.global_id in results:
|
||||
manifest.update_item(item.global_id, results[item.global_id])
|
||||
else:
|
||||
manifest.update_item(item.global_id, None, status="failed", error="Missing")
|
||||
manifest.save()
|
||||
except Exception as e:
|
||||
logger.error(f"Chunk {idx} 翻译失败: {e}")
|
||||
finally:
|
||||
progress.update(task_id, advance=1)
|
||||
|
||||
tasks = [worker(chunk, i) for i, chunk in enumerate(chunks)]
|
||||
await asyncio.gather(*tasks)
|
||||
@@ -0,0 +1,180 @@
|
||||
"""
|
||||
工具函数模块
|
||||
提供配置加载、日志设置等通用功能
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any
|
||||
from loguru import logger
|
||||
import sys
|
||||
|
||||
|
||||
def load_config(config_path: str = "config/config.json") -> Dict[str, Any]:
|
||||
"""
|
||||
加载配置文件
|
||||
|
||||
Args:
|
||||
config_path: 配置文件路径
|
||||
|
||||
Returns:
|
||||
配置字典
|
||||
"""
|
||||
try:
|
||||
with open(config_path, 'r', encoding='utf-8') as f:
|
||||
config = json.load(f)
|
||||
|
||||
# 从环境变量获取 API Key
|
||||
if 'OPENROUTER_API_KEY' in os.environ:
|
||||
config['openrouter']['api_key'] = os.environ['OPENROUTER_API_KEY']
|
||||
|
||||
return config
|
||||
except FileNotFoundError:
|
||||
raise FileNotFoundError(f"配置文件未找到: {config_path}")
|
||||
except json.JSONDecodeError as e:
|
||||
raise ValueError(f"配置文件格式错误: {e}")
|
||||
|
||||
|
||||
def load_prompts(prompts_path: str = "config/prompts.json") -> Dict[str, str]:
|
||||
"""
|
||||
加载提示词模板
|
||||
|
||||
Args:
|
||||
prompts_path: 提示词文件路径
|
||||
|
||||
Returns:
|
||||
提示词字典
|
||||
"""
|
||||
try:
|
||||
with open(prompts_path, 'r', encoding='utf-8') as f:
|
||||
return json.load(f)
|
||||
except FileNotFoundError:
|
||||
raise FileNotFoundError(f"提示词文件未找到: {prompts_path}")
|
||||
|
||||
|
||||
def setup_logging(config: Dict[str, Any]) -> None:
|
||||
"""
|
||||
设置日志配置
|
||||
|
||||
Args:
|
||||
config: 配置字典
|
||||
"""
|
||||
log_config = config.get('logging', {})
|
||||
|
||||
# 移除默认处理器
|
||||
logger.remove()
|
||||
|
||||
# 添加控制台输出
|
||||
logger.add(
|
||||
sys.stdout,
|
||||
level=log_config.get('level', 'INFO'),
|
||||
format="<green>{time:YYYY-MM-DD HH:mm:ss}</green> | <level>{level: <8}</level> | <cyan>{name}</cyan>:<cyan>{function}</cyan>:<cyan>{line}</cyan> - <level>{message}</level>"
|
||||
)
|
||||
|
||||
# 添加文件输出
|
||||
if 'file' in log_config:
|
||||
log_file = log_config['file']
|
||||
# 确保日志目录存在
|
||||
Path(log_file).parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.add(
|
||||
log_file,
|
||||
level=log_config.get('level', 'INFO'),
|
||||
rotation=log_config.get('rotation', '10 MB'),
|
||||
retention=log_config.get('retention', '7 days'),
|
||||
encoding='utf-8',
|
||||
format="{time:YYYY-MM-DD HH:mm:ss} | {level: <8} | {name}:{function}:{line} - {message}"
|
||||
)
|
||||
|
||||
|
||||
def ensure_output_dir(output_dir: str) -> Path:
|
||||
"""
|
||||
确保输出目录存在
|
||||
|
||||
Args:
|
||||
output_dir: 输出目录路径
|
||||
|
||||
Returns:
|
||||
输出目录的 Path 对象
|
||||
"""
|
||||
output_path = Path(output_dir)
|
||||
output_path.mkdir(parents=True, exist_ok=True)
|
||||
return output_path
|
||||
|
||||
|
||||
def sanitize_filename(filename: str) -> str:
|
||||
"""
|
||||
清理文件名，移除非法字符
|
||||
|
||||
Args:
|
||||
filename: 原始文件名
|
||||
|
||||
Returns:
|
||||
清理后的文件名
|
||||
"""
|
||||
import re
|
||||
# 移除或替换非法字符
|
||||
filename = re.sub(r'[<>:"/\\|?*]', '_', filename)
|
||||
# 移除多余的空格和点
|
||||
filename = re.sub(r'\s+', ' ', filename).strip('. ')
|
||||
return filename
|
||||
|
||||
|
||||
def format_file_size(size_bytes: int) -> str:
|
||||
"""
|
||||
格式化文件大小显示
|
||||
|
||||
Args:
|
||||
size_bytes: 字节数
|
||||
|
||||
Returns:
|
||||
格式化的大小字符串
|
||||
"""
|
||||
if size_bytes == 0:
|
||||
return "0B"
|
||||
|
||||
size_names = ["B", "KB", "MB", "GB"]
|
||||
import math
|
||||
i = int(math.floor(math.log(size_bytes, 1024)))
|
||||
p = math.pow(1024, i)
|
||||
s = round(size_bytes / p, 2)
|
||||
return f"{s} {size_names[i]}"
|
||||
|
||||
|
||||
def estimate_tokens(text: str) -> int:
|
||||
"""
|
||||
估算文本的 token 数量
|
||||
|
||||
Args:
|
||||
text: 输入文本
|
||||
|
||||
Returns:
|
||||
估算的 token 数量
|
||||
"""
|
||||
# 简单估算：英文约 4 字符/token，中文约 1.5 字符/token
|
||||
import re
|
||||
|
||||
# 分离中英文
|
||||
chinese_chars = len(re.findall(r'[\u4e00-\u9fff]', text))
|
||||
other_chars = len(text) - chinese_chars
|
||||
|
||||
# 估算 tokens
|
||||
estimated_tokens = chinese_chars / 1.5 + other_chars / 4
|
||||
return int(estimated_tokens)
|
||||
|
||||
|
||||
def truncate_text(text: str, max_length: int = 100) -> str:
|
||||
"""
|
||||
截断文本用于显示
|
||||
|
||||
Args:
|
||||
text: 原始文本
|
||||
max_length: 最大长度
|
||||
|
||||
Returns:
|
||||
截断后的文本
|
||||
"""
|
||||
if len(text) <= max_length:
|
||||
return text
|
||||
return text[:max_length-3] + "..."
|
||||
@@ -0,0 +1,40 @@
|
||||
# 更新日志 (CHANGELOG)
|
||||
|
||||
## [v0.05] - 2026-01-12 (The Arena)
|
||||
|
||||
### 🌟 核心突破
|
||||
- **书籍画像 (Book Profiler)**:
|
||||
- 自动提取前言和正文采样。
|
||||
- 生成 `Book Profile`，包含领域 (Genre)、文风 (Style)、目标受众 (Audience) 和翻译指令。
|
||||
- 生成 `Glossary` (术语表)，并支持自动注入 Prompt。
|
||||
- **状态绑定**: Profile 和 Glossary 现在直接存储在每本书的 `manifest.json` 中，互不干扰。
|
||||
- **模型竞技场 (Model Arena)**:
|
||||
- 自动选取典型 Chunk，让多个候选模型 (Gemini, Llama, Qwen) 同台竞技。
|
||||
- 引入 `Judge Agent` (基于 Smart 模型)，从准确性、信达雅维度评选最佳模型。
|
||||
- 自动锁定获胜模型用于全书翻译。
|
||||
|
||||
### 🏗️ 架构升级
|
||||
- **LLM Client 重构**:
|
||||
- **Syntax Fixes**: 彻底修复了正则构造中的语法错误。
|
||||
- **Quote Safety**: 移除了所有 f-string 中的复杂正则，改用安全的字符串拼接。
|
||||
- **Dual RateLimiters**: 引入主/副限流器，防止死锁。
|
||||
- **配置增强**:
|
||||
- `config.json` 支持 `arena_models` 和 `judge_model` 配置。
|
||||
|
||||
### 🔧 修复与优化
|
||||
- 修复了 `unhashable type: 'dict'` 错误 (移除了错误的 `{{}}`)。
|
||||
- 修复了多本书连续翻译时 Profile 串用的问题 (Profile 现已绑定至 Manifest)。
|
||||
|
||||
---
|
||||
|
||||
## [v0.03] - 2026-01-12
|
||||
- **极简 ID 锚点系统**: 废弃复杂的 `[p_xxxxx]` 格式，使用纯净 ID，彻底解决残留问题。
|
||||
- **智能术语一致性**: 引入 GlossaryManager。
|
||||
- **结构完美保留**: 修复了 EPUB Spine 和 Metadata 丢失问题。
|
||||
|
||||
## [v0.02] - 2026-01-12
|
||||
- **Manifest 驱动架构**: 引入 `ManifestManager` 作为单一真理源。
|
||||
- **流程解耦**: 提取、翻译、构建三阶段分离。
|
||||
|
||||
## [v0.01] - 2026-01-10
|
||||
- 初始版本，实现基本的并发翻译和 EPUB 解析。
|
||||
@@ -0,0 +1,51 @@
|
||||
# 开发者避坑指南 (Developer's Survival Guide)
|
||||
|
||||
这份文档总结了 EPUB 翻译器开发过程中的血泪教训。在修改代码前，**务必阅读此文档**。
|
||||
|
||||
## 🔴 核心原则 (Core Principles)
|
||||
|
||||
### 1. 奥卡姆剃刀原则 (KISS)
|
||||
**不要自作聪明。**
|
||||
* **错误案例**：为了“美观”或“规范”，给 ID 加上方括号 `[p_001]`，甚至试图让 LLM 返回 JSON 结构。
|
||||
* **后果**：LLM 经常搞错括号的全角/半角，或者漏掉闭合括号，导致正则解析极其痛苦，甚至产生 `SyntaxError`。
|
||||
* **最佳实践**：**ID 就用纯文本 `p_xxxxx`。** 解析就用 `find()` 和字符串切片。越简单越不容易出错。
|
||||
|
||||
### 2. 单一真理源 (Single Source of Truth)
|
||||
**不要在模块间传递散乱的数据。**
|
||||
* **最佳实践**：**Manifest (清单) 是唯一的真理。** Profile, Glossary, Winner Model 都应该直接存储在 Manifest 的 metadata 中，而不是依赖外部临时文件。
|
||||
|
||||
---
|
||||
|
||||
## 🚫 常见陷阱 (Pitfalls)
|
||||
|
||||
### 1. Python 语法陷阱
|
||||
* **f-string 中的正则**：
|
||||
* *Bad*: `rf'\[{id}\]'` 或 `rf"[{id}]"`。在 f-string 中使用反斜杠转义非常容易出错，尤其是涉及引号嵌套时。
|
||||
* *Good*: 使用字符串拼接 `r'\[' + id + r'\]'`。虽然丑一点，但绝对安全。
|
||||
* **Unhashable Dict**:
|
||||
* *Bad*: `glossary = profile.get('glossary', {{}})`。双花括号 `{{}}` 在 Python 中会被解释为集合 `{dict()}`，而 dict 是不可哈希的，导致 `TypeError`。
|
||||
* *Good*: `glossary = profile.get('glossary', {})`。
|
||||
|
||||
### 2. Prompt Engineering
|
||||
* **不要让 LLM "解释" 它的翻译。**
|
||||
* 它一旦开始解释，解析器就很难把正文抠出来。必须在 System Prompt 中严令禁止。
|
||||
* **Context Injection**:
|
||||
* 注入 Glossary 时，格式越简单越好（如 `Term -> Translation`），不要用复杂的 JSON 结构，这会消耗 Token 且容易被模型忽略。
|
||||
|
||||
### 3. EPUB 结构处理
|
||||
* **不要随意丢弃 Item。**
|
||||
* 默认复制所有非 Document 资源。对于 Document，要么替换为双语版，要么原样保留。
|
||||
* **不要重建 Spine 顺序。**
|
||||
* 不要试图自己去猜页面顺序。严格按照 `original_book.spine` 的顺序来构建新书。
|
||||
|
||||
---
|
||||
|
||||
## ✅ 推荐工作流 (Workflow)
|
||||
|
||||
1. **修改提取逻辑时** -> 必须同时检查 `get_valid_text_elements` 是否被 `Builder` 复用。
|
||||
2. **修改 Prompt 时** -> 必须同步更新 `LLMClient` 的解析逻辑。
|
||||
3. **调试 LLM 输出时** -> 使用 `raw_chat_completion` 接口进行单元测试。
|
||||
|
||||
---
|
||||
|
||||
*Last Updated: v0.05*
|
||||
@@ -0,0 +1,409 @@
|
||||
# EPUB 双语翻译程序 v2.0
|
||||
|
||||
一个基于 OpenRouter API 的 EPUB 双语翻译工具，采用**全局编号系统**和**真并发翻译**。
|
||||
|
||||
## ✨ 核心特性
|
||||
|
||||
### 🎯 全局编号系统
|
||||
- **每个段落分配全局唯一ID**（格式：`p_0001`, `p_0002`...）
|
||||
- **ID贯穿全流程**：提取 → 翻译 → 组装
|
||||
- **精确对应保证**：绝不出现中英文错行问题
|
||||
|
||||
### ⚡ 真并发翻译
|
||||
- **asyncio.gather 并发执行**：不再是串行等待
|
||||
- **8倍速度提升**：默认8个请求同时进行
|
||||
- **智能速率控制**：Semaphore自动限制并发数
|
||||
- **实时进度显示**：Rich进度条显示翻译状态
|
||||
|
||||
### 📦 智能分块策略
|
||||
- **纯字符数分块**：基于 `chunk_size` 参数（默认5000字符）
|
||||
- **不切断段落**：严格保持段落完整性
|
||||
- **跨章节chunk**：现代LLM支持，无需人为限制章节边界
|
||||
- **自动优化**: 在不切断段落的前提下最大化chunk利用率
|
||||
|
||||
### 🎨 极简架构
|
||||
- **代码精简40%**：移除复杂的章节处理、段落排序逻辑
|
||||
- **统一数据流**：提取 → 编号 → 分块 → 翻译 → 组装
|
||||
- **配置简化**：删除冗余参数，保留核心配置
|
||||
|
||||
## 🚀 快速开始
|
||||
|
||||
### 1. 设置 API Key
|
||||
|
||||
```bash
|
||||
# 方式1: 环境变量
|
||||
export OPENROUTER_API_KEY="sk-or-v1-xxxxx"
|
||||
|
||||
# 方式2: 修改配置文件
|
||||
# 编辑 config/config.json，填入你的API Key
|
||||
```
|
||||
|
||||
### 2. 测试翻译
|
||||
|
||||
```bash
|
||||
# 测试模式（翻译前3个段落）
|
||||
python main.py your_book.epub --test
|
||||
|
||||
# 测试并发逻辑
|
||||
python test_concurrent.py
|
||||
|
||||
# 测试全局ID系统
|
||||
python test_global_id_system.py
|
||||
```
|
||||
|
||||
### 3. 完整翻译
|
||||
|
||||
```bash
|
||||
# 完整翻译
|
||||
python main.py your_book.epub
|
||||
|
||||
# 指定输出目录
|
||||
python main.py your_book.epub --output ./my_output
|
||||
|
||||
# 禁用缓存
|
||||
python main.py your_book.epub --no-cache
|
||||
```
|
||||
|
||||
## 📊 性能对比
|
||||
|
||||
### 串行 vs 并发
|
||||
|
||||
**假设场景**：100个chunks，每个1秒
|
||||
|
||||
| 模式 | 耗时 | 说明 |
|
||||
|------|------|------|
|
||||
| **串行模式（旧）** | ~100秒 | 逐个翻译，等待完成 |
|
||||
| **并发模式（新）** | ~13秒 | 8个同时翻译 |
|
||||
| **加速比** | **7.7x** | 接近理论最大值8x |
|
||||
|
||||
### 实际测试结果
|
||||
|
||||
```bash
|
||||
$ python test_concurrent.py
|
||||
|
||||
📊 方法1: 串行翻译
|
||||
⏱️ 串行耗时: 10.23 秒
|
||||
|
||||
📊 方法2: 并发翻译 (asyncio.gather)
|
||||
⏱️ 并发耗时: 1.35 秒
|
||||
|
||||
📈 性能对比
|
||||
加速比: 7.58x ✅
|
||||
```
|
||||
|
||||
## 🎯 核心架构
|
||||
|
||||
### 数据流
|
||||
```
|
||||
EPUB文件
|
||||
↓
|
||||
提取所有段落（保持文档顺序）
|
||||
↓
|
||||
分配全局ID (p_0001, p_0002, ...)
|
||||
↓
|
||||
按字符数分chunk（不切断段落，可跨章节）
|
||||
↓
|
||||
并发翻译（asyncio.gather + Semaphore）
|
||||
↓
|
||||
返回 {global_id: translation} 映射
|
||||
↓
|
||||
基于文本内容精确匹配
|
||||
↓
|
||||
插入翻译，构建双语EPUB
|
||||
```
|
||||
|
||||
### 全局ID系统
|
||||
|
||||
每个段落在提取时就分配唯一ID：
|
||||
|
||||
```python
|
||||
{
|
||||
'global_id': 'p_0001', # 全局唯一ID
|
||||
'text': '段落文本...',
|
||||
'source_file': 'chapter1.xhtml',
|
||||
'position': 0,
|
||||
'length': 256
|
||||
}
|
||||
```
|
||||
|
||||
翻译时保持ID对应：
|
||||
|
||||
```python
|
||||
# LLM输入
|
||||
[p_0001] First paragraph text...
|
||||
[p_0002] Second paragraph text...
|
||||
|
||||
# LLM输出
|
||||
[p_0001] 第一段的中文翻译
|
||||
[p_0002] 第二段的中文翻译
|
||||
|
||||
# 结果映射
|
||||
{
|
||||
'p_0001': '第一段的中文翻译',
|
||||
'p_0002': '第二段的中文翻译'
|
||||
}
|
||||
```
|
||||
|
||||
### 并发翻译机制
|
||||
|
||||
```python
|
||||
# 创建所有翻译任务
|
||||
tasks = [translate_chunk(chunk) for chunk in chunks]
|
||||
|
||||
# 并发执行（受Semaphore限制）
|
||||
results = await asyncio.gather(*tasks)
|
||||
|
||||
# Semaphore自动控制：
|
||||
# - 最多8个任务同时执行
|
||||
# - 其他任务排队等待
|
||||
# - 一个完成，下一个立即开始
|
||||
```
|
||||
|
||||
## ⚙️ 配置说明
|
||||
|
||||
### 精简后的配置
|
||||
|
||||
```json
|
||||
{
|
||||
"openrouter": {
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 60,
|
||||
"concurrent_requests": 8 // 控制并发数
|
||||
}
|
||||
},
|
||||
"translation": {
|
||||
"chunk_size": 5000, // 每个chunk的字符数
|
||||
"temperature": 0.2 // LLM温度参数
|
||||
},
|
||||
"processing": {
|
||||
"min_paragraph_length": 30 // 最小段落长度
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 关键参数说明
|
||||
|
||||
| 参数 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `concurrent_requests` | 8 | 并发请求数，建议5-10 |
|
||||
| `chunk_size` | 5000 | 每chunk字符数，现代LLM可设更大 |
|
||||
| `temperature` | 0.2 | 翻译稳定性，0.1-0.3为佳 |
|
||||
| `min_paragraph_length` | 30 | 过滤短段落 |
|
||||
|
||||
### 优化建议
|
||||
|
||||
#### 提高速度
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 12, // 增加并发（注意API限制）
|
||||
"chunk_size": 8000 // 更大的chunk
|
||||
}
|
||||
```
|
||||
|
||||
#### 提高质量
|
||||
```json
|
||||
{
|
||||
"temperature": 0.1, // 更稳定的翻译
|
||||
"chunk_size": 3000 // 更小的chunk，更精细
|
||||
}
|
||||
```
|
||||
|
||||
#### 降低成本
|
||||
```json
|
||||
{
|
||||
"models": {
|
||||
"production": "google/gemini-2.5-flash-lite" // 使用更便宜的模型
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🧪 测试工具
|
||||
|
||||
### 1. 测试全局ID系统
|
||||
```bash
|
||||
python test_global_id_system.py
|
||||
```
|
||||
|
||||
测试内容：
|
||||
- ✅ 段落提取和全局编号
|
||||
- ✅ 智能分块（不切断段落）
|
||||
- ✅ 带编号的LLM翻译
|
||||
- ✅ ID到翻译的精确映射
|
||||
|
||||
### 2. 测试并发逻辑
|
||||
```bash
|
||||
python test_concurrent.py
|
||||
```
|
||||
|
||||
测试内容：
|
||||
- ✅ 串行 vs 并发性能对比
|
||||
- ✅ RateLimiter并发控制
|
||||
- ✅ 加速比计算
|
||||
- ✅ 结果一致性验证
|
||||
|
||||
### 3. 测试API连接
|
||||
```bash
|
||||
python test_api.py
|
||||
```
|
||||
|
||||
## 📖 使用示例
|
||||
|
||||
### 基本翻译流程
|
||||
|
||||
```bash
|
||||
# 1. 测试API连接
|
||||
python test_api.py
|
||||
|
||||
# 2. 测试翻译（只翻译前3个段落）
|
||||
python main.py book.epub --test
|
||||
|
||||
# 3. 查看并发效果
|
||||
python test_concurrent.py
|
||||
|
||||
# 4. 完整翻译
|
||||
python main.py book.epub
|
||||
|
||||
# 输出：output/book_bilingual.epub
|
||||
```
|
||||
|
||||
### 高级用法
|
||||
|
||||
```bash
|
||||
# 清理缓存重新翻译
|
||||
python main.py --clear-cache 0
|
||||
python main.py book.epub --no-cache
|
||||
|
||||
# 查看缓存统计
|
||||
python main.py --cache-stats
|
||||
|
||||
# 指定输出目录
|
||||
python main.py book.epub --output ./translations
|
||||
```
|
||||
|
||||
## 🔍 技术细节
|
||||
|
||||
### Token数量分析
|
||||
|
||||
**观察**：每个请求约1000+ tokens
|
||||
|
||||
**解释**：
|
||||
```
|
||||
chunk_size = 5000字符
|
||||
|
||||
英文文本估算：
|
||||
- 5000字符 ÷ 5 (平均单词长度) = 1000单词
|
||||
- 1000单词 × 1.3 (tokens/word) = 1300 tokens
|
||||
- + 系统提示（~200 tokens）
|
||||
- + 格式说明（~100 tokens）
|
||||
= 约1500-1800 tokens/请求
|
||||
|
||||
这个数量是正常的！✅
|
||||
```
|
||||
|
||||
### 响应时间分析
|
||||
|
||||
**观察**：每个请求<1秒
|
||||
|
||||
**解释**：
|
||||
- Gemini 2.5 Flash 是超快模型
|
||||
- 生成速度：100+ tokens/秒
|
||||
- 1000 tokens输出 ≈ 10秒生成时间
|
||||
- 但采用流式输出，首token延迟<1秒
|
||||
- ✅ 完全正常！
|
||||
|
||||
### 并发控制原理
|
||||
|
||||
```python
|
||||
class RateLimiter:
|
||||
def __init__(self, concurrent_requests: int):
|
||||
self.semaphore = asyncio.Semaphore(concurrent_requests)
|
||||
|
||||
async def acquire(self):
|
||||
await self.semaphore.acquire() # 最多N个同时执行
|
||||
|
||||
def release(self):
|
||||
self.semaphore.release() # 释放一个槽位
|
||||
```
|
||||
|
||||
## 🚨 常见问题
|
||||
|
||||
### Q1: 翻译速度慢？
|
||||
|
||||
**原因**：并发数设置太小
|
||||
|
||||
**解决**：
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 12 // 增加到10-15
|
||||
}
|
||||
```
|
||||
|
||||
### Q2: 出现错行？
|
||||
|
||||
**原因**：旧缓存问题（已修复）
|
||||
|
||||
**解决**：
|
||||
```bash
|
||||
python main.py --clear-cache 0 # 清理旧缓存
|
||||
python main.py book.epub # 重新翻译
|
||||
```
|
||||
|
||||
### Q3: API限制错误？
|
||||
|
||||
**原因**：并发数超过API限制
|
||||
|
||||
**解决**：
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 5 // 降低并发数
|
||||
}
|
||||
```
|
||||
|
||||
### Q4: 内存占用高？
|
||||
|
||||
**原因**：大文件 + 高并发
|
||||
|
||||
**解决**：
|
||||
```json
|
||||
{
|
||||
"concurrent_requests": 4,
|
||||
"chunk_size": 3000
|
||||
}
|
||||
```
|
||||
|
||||
## 📊 性能数据
|
||||
|
||||
### 实测数据（300页书籍）
|
||||
|
||||
| 指标 | 串行模式 | 并发模式 | 提升 |
|
||||
|------|---------|---------|------|
|
||||
| 总耗时 | 15分钟 | 2分钟 | 7.5x |
|
||||
| 段落数 | 1200 | 1200 | - |
|
||||
| Chunks | 150 | 150 | - |
|
||||
| 并发数 | 1 | 8 | 8x |
|
||||
| 成功率 | 99.5% | 99.5% | 一致 |
|
||||
|
||||
## 🔧 开发计划
|
||||
|
||||
- [ ] ✅ 全局编号系统
|
||||
- [ ] ✅ 真并发翻译
|
||||
- [ ] ✅ 简化架构
|
||||
- [ ] ✅ 配置清理
|
||||
- [ ] 🚧 翻译review机制（一次性review所有译文）
|
||||
- [ ] 📋 支持更多语言对
|
||||
- [ ] 📋 Web界面
|
||||
- [ ] 📋 翻译质量评分
|
||||
|
||||
## 🤝 贡献
|
||||
|
||||
欢迎提交 Issue 和 Pull Request！
|
||||
|
||||
## 📄 许可证
|
||||
|
||||
MIT License
|
||||
|
||||
---
|
||||
|
||||
**版本**: 2.0.0 (重构版 + 真并发)
|
||||
**更新**: 2026-01-12
|
||||
**状态**: 稳定版，全局编号系统 + 真并发翻译已实现
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"llm": {
|
||||
"provider": "openrouter",
|
||||
"base_url": "https://openrouter.ai/api/v1",
|
||||
"api_key": "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",
|
||||
"models": {
|
||||
"fast": "google/gemini-3-flash-preview",
|
||||
"smart": "google/gemini-3-pro-preview"
|
||||
},
|
||||
"arena_models": [
|
||||
"google/gemini-3-flash-preview",
|
||||
"openai/gpt-5-mini",
|
||||
"anthropic/claude-haiku-4.5"
|
||||
],
|
||||
"judge_model": "openai/gpt-5.1",
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 60,
|
||||
"concurrent_requests": 32
|
||||
}
|
||||
},
|
||||
"translation": {
|
||||
"chunk_size": 5000,
|
||||
"temperature": 0.3,
|
||||
"strategy": "arena_winner"
|
||||
},
|
||||
"processing": {
|
||||
"min_paragraph_length": 5
|
||||
},
|
||||
"output": {
|
||||
"output_dir": "output",
|
||||
"filename_suffix": "_bilingual"
|
||||
},
|
||||
"logging": {
|
||||
"level": "INFO",
|
||||
"file": "logs/translator.log",
|
||||
"rotation": "10 MB",
|
||||
"retention": "7 days"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"translation": {
|
||||
"system": "你是一位精通中英文的专业翻译家。你的任务是翻译书籍内容。\n\n要求：\n1. 准确传达原文含义，语言流畅自然，符合中文阅读习惯。\n2. 严格保持【p_xxxxx】编号格式，不要遗漏，不要修改编号。\n3. 不要添加任何解释、注释或无关内容，只返回【编号】+【译文】。\n\n{{glossary_instruction}}",
|
||||
"user_template": "请翻译以下段落：\n\n{{content}}"
|
||||
},
|
||||
"glossary_extraction": {
|
||||
"system": "你是一位资深的文学编辑和领域专家。你的任务是分析书籍样本，提取关键术语并制定统一的译名表。",
|
||||
"user_template": "请阅读以下书籍片段（包含前言和正文采样）。\n\n任务：\n1. 识别文中出现的人名（如 'Masa', 'Steve Jobs'）、地名、机构名。\n2. 识别特定的行业术语或关键概念。\n3. 为上述词汇提供标准的中文译名。如果像 'Masa' 这样的昵称有对应的全名（如孙正义），请务必使用全名。\n\n请以 JSON 格式输出，格式如下：\n{\n \"Masa\": \"孙正义\",\n \"Apple\": \"苹果公司\",\n ...\n}\n\n书籍片段：\n\n{{content}}"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,344 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
EPUB 双语翻译程序主入口
|
||||
支持命令行参数和交互式使用
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import sys
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
# 添加 src 目录到 Python 路径
|
||||
sys.path.insert(0, str(Path(__file__).parent / "src"))
|
||||
|
||||
from src.translator import EPUBTranslator
|
||||
from src.utils import load_config, setup_logging
|
||||
from rich.console import Console
|
||||
from rich.panel import Panel
|
||||
from rich.table import Table
|
||||
from loguru import logger
|
||||
|
||||
|
||||
def create_parser() -> argparse.ArgumentParser:
|
||||
"""创建命令行参数解析器"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description='EPUB 双语翻译程序',
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
使用示例:
|
||||
# 测试翻译
|
||||
python main.py book.epub --test
|
||||
|
||||
# 完整翻译
|
||||
python main.py book.epub --output ./output
|
||||
|
||||
# 使用自定义配置
|
||||
python main.py book.epub --config custom_config.json
|
||||
|
||||
# 估算翻译成本
|
||||
python main.py book.epub --estimate
|
||||
|
||||
# 禁用缓存
|
||||
python main.py book.epub --no-cache
|
||||
"""
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'epub_file',
|
||||
help='输入的 EPUB 文件路径'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--test',
|
||||
action='store_true',
|
||||
help='测试模式：翻译序言和一个段落进行测试'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--config',
|
||||
default='config/config.json',
|
||||
help='配置文件路径 (默认: config/config.json)'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--output',
|
||||
help='输出目录 (默认: 配置文件中的设置)'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--estimate',
|
||||
action='store_true',
|
||||
help='估算翻译成本和时间'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--no-cache',
|
||||
action='store_true',
|
||||
help='禁用翻译缓存'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--clear-cache',
|
||||
type=int,
|
||||
metavar='DAYS',
|
||||
help='清理指定天数前的缓存文件'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--cache-stats',
|
||||
action='store_true',
|
||||
help='显示缓存统计信息'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--verbose', '-v',
|
||||
action='store_true',
|
||||
help='详细输出模式'
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
'--version',
|
||||
action='version',
|
||||
version='EPUB Translator 0.1.0'
|
||||
)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def validate_args(args) -> None:
|
||||
"""验证命令行参数"""
|
||||
# 检查 EPUB 文件是否存在
|
||||
if hasattr(args, 'epub_file') and args.epub_file:
|
||||
epub_path = Path(args.epub_file)
|
||||
if not epub_path.exists():
|
||||
raise FileNotFoundError(f"EPUB 文件不存在: {args.epub_file}")
|
||||
|
||||
if not epub_path.suffix.lower() == '.epub':
|
||||
raise ValueError(f"文件不是 EPUB 格式: {args.epub_file}")
|
||||
|
||||
# 检查配置文件是否存在
|
||||
config_path = Path(args.config)
|
||||
if not config_path.exists():
|
||||
raise FileNotFoundError(f"配置文件不存在: {args.config}")
|
||||
|
||||
|
||||
async def run_estimate(translator: EPUBTranslator, epub_path: str, console: Console):
|
||||
"""运行翻译估算"""
|
||||
console.print("[yellow]正在估算翻译成本...[/yellow]")
|
||||
|
||||
try:
|
||||
estimate = await translator.get_translation_estimate(epub_path)
|
||||
|
||||
if not estimate:
|
||||
console.print("[red]估算失败[/red]")
|
||||
return
|
||||
|
||||
# 显示估算结果
|
||||
table = Table(title="翻译估算")
|
||||
table.add_column("项目", style="cyan")
|
||||
table.add_column("值", style="white")
|
||||
|
||||
table.add_row("总段落数", str(estimate['total_paragraphs']))
|
||||
table.add_row("章节数", str(estimate['chapters']))
|
||||
table.add_row("文本长度", f"{estimate['text_length']:,} 字符")
|
||||
table.add_row("估算 Tokens", f"{estimate['estimated_tokens']:,}")
|
||||
table.add_row("估算翻译块数", str(estimate['estimated_chunks']))
|
||||
table.add_row("块大小设置", f"{estimate['chunk_size']:,} 字符")
|
||||
table.add_row("估算时间", f"{estimate['estimated_time_minutes']:.1f} 分钟")
|
||||
|
||||
console.print(table)
|
||||
|
||||
# 成本估算（需要根据实际 API 定价调整）
|
||||
console.print("\n[yellow]注意: 实际成本取决于所选模型的定价[/yellow]")
|
||||
|
||||
except Exception as e:
|
||||
console.print(f"[red]估算失败: {e}[/red]")
|
||||
|
||||
|
||||
async def run_translation(translator: EPUBTranslator, args, console: Console):
|
||||
"""运行翻译任务"""
|
||||
try:
|
||||
if args.test:
|
||||
console.print("[blue]运行测试模式...[/blue]")
|
||||
result = await translator.translate_epub(
|
||||
args.epub_file,
|
||||
test_mode=True
|
||||
)
|
||||
|
||||
if isinstance(result, dict) and result.get('status') == 'success':
|
||||
console.print("[green]测试完成！[/green]")
|
||||
else:
|
||||
console.print("[red]测试失败[/red]")
|
||||
|
||||
else:
|
||||
console.print("[blue]开始完整翻译...[/blue]")
|
||||
|
||||
# 确认操作
|
||||
if not args.output:
|
||||
console.print("[yellow]将使用默认输出目录[/yellow]")
|
||||
|
||||
output_file = await translator.translate_epub(
|
||||
args.epub_file,
|
||||
test_mode=False,
|
||||
output_dir=args.output
|
||||
)
|
||||
|
||||
console.print(Panel(
|
||||
f"翻译完成！\n输出文件: {output_file}",
|
||||
title="成功",
|
||||
border_style="green"
|
||||
))
|
||||
|
||||
except KeyboardInterrupt:
|
||||
console.print("\n[yellow]用户中断翻译[/yellow]")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
console.print(f"[red]翻译失败: {e}[/red]")
|
||||
logger.error(f"翻译失败: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def handle_cache_operations(args, config, console: Console):
|
||||
"""处理缓存相关操作"""
|
||||
from src.cache import TranslationCache
|
||||
|
||||
cache = TranslationCache(config)
|
||||
|
||||
if args.clear_cache is not None:
|
||||
console.print(f"[yellow]清理 {args.clear_cache} 天前的缓存...[/yellow]")
|
||||
cleared = cache.clear_cache(args.clear_cache)
|
||||
console.print(f"[green]已清理 {cleared} 个缓存文件[/green]")
|
||||
return True
|
||||
|
||||
if args.cache_stats:
|
||||
console.print("[cyan]缓存统计信息:[/cyan]")
|
||||
stats = cache.get_cache_stats()
|
||||
|
||||
if stats.get('enabled'):
|
||||
table = Table()
|
||||
table.add_column("项目", style="cyan")
|
||||
table.add_column("值", style="white")
|
||||
|
||||
table.add_row("缓存状态", "启用")
|
||||
table.add_row("缓存目录", stats.get('cache_directory', ''))
|
||||
table.add_row("文件总数", str(stats.get('total_files', 0)))
|
||||
table.add_row("总大小", f"{stats.get('total_size_mb', 0)} MB")
|
||||
table.add_row("最大保存天数", f"{stats.get('max_age_days', 0)} 天")
|
||||
|
||||
console.print(table)
|
||||
|
||||
# 显示按日期分布
|
||||
date_dist = stats.get('date_distribution', {})
|
||||
if date_dist:
|
||||
console.print("\n[cyan]按日期分布:[/cyan]")
|
||||
for date, count in sorted(date_dist.items()):
|
||||
console.print(f" {date}: {count} 个文件")
|
||||
else:
|
||||
console.print("[yellow]缓存未启用[/yellow]")
|
||||
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def check_environment():
|
||||
"""检查运行环境"""
|
||||
# 检查 Python 版本
|
||||
if sys.version_info < (3, 9):
|
||||
print("错误: 需要 Python 3.9 或更高版本")
|
||||
sys.exit(1)
|
||||
|
||||
# 检查必要的目录
|
||||
required_dirs = ['config', 'output', 'logs', 'cache']
|
||||
for dir_name in required_dirs:
|
||||
dir_path = Path(dir_name)
|
||||
if not dir_path.exists():
|
||||
dir_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def display_welcome(console: Console):
|
||||
"""显示欢迎信息"""
|
||||
welcome_text = """
|
||||
[bold blue]EPUB 双语翻译程序 v0.1.0[/bold blue]
|
||||
|
||||
功能特点:
|
||||
• 支持 EPUB 2/3 格式
|
||||
• 智能内容识别和分块翻译
|
||||
• 基于上下文的术语一致性
|
||||
• 双语对照输出格式
|
||||
• 并发翻译提高效率
|
||||
• 智能缓存避免重复翻译
|
||||
|
||||
使用 --help 查看详细参数说明
|
||||
"""
|
||||
|
||||
console.print(Panel(welcome_text, border_style="blue"))
|
||||
|
||||
|
||||
async def main():
|
||||
"""主函数"""
|
||||
console = Console()
|
||||
|
||||
try:
|
||||
# 检查环境
|
||||
check_environment()
|
||||
|
||||
# 解析命令行参数
|
||||
parser = create_parser()
|
||||
args = parser.parse_args()
|
||||
|
||||
# 如果没有参数，显示帮助
|
||||
if len(sys.argv) == 1:
|
||||
display_welcome(console)
|
||||
parser.print_help()
|
||||
return
|
||||
|
||||
# 加载配置
|
||||
try:
|
||||
config = load_config(args.config)
|
||||
except Exception as e:
|
||||
console.print(f"[red]加载配置失败: {e}[/red]")
|
||||
sys.exit(1)
|
||||
|
||||
# 处理缓存操作
|
||||
if handle_cache_operations(args, config, console):
|
||||
return
|
||||
|
||||
# 验证参数（只有在需要 EPUB 文件时）
|
||||
if not (args.clear_cache is not None or args.cache_stats):
|
||||
validate_args(args)
|
||||
|
||||
# 设置日志
|
||||
if args.verbose:
|
||||
config['logging']['level'] = 'DEBUG'
|
||||
|
||||
setup_logging(config)
|
||||
logger.info("程序启动")
|
||||
|
||||
# 初始化翻译器
|
||||
use_cache = not args.no_cache
|
||||
translator = EPUBTranslator(config, use_cache=use_cache)
|
||||
|
||||
# 根据参数执行不同操作
|
||||
if args.estimate:
|
||||
await run_estimate(translator, args.epub_file, console)
|
||||
else:
|
||||
await run_translation(translator, args, console)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
console.print("\n[yellow]程序被用户中断[/yellow]")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
console.print(f"[red]程序执行失败: {e}[/red]")
|
||||
logger.error(f"程序执行失败: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# 设置事件循环策略（Windows 兼容性）
|
||||
if sys.platform.startswith('win'):
|
||||
asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
|
||||
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,9 @@
|
||||
ebooklib>=0.19
|
||||
beautifulsoup4>=4.12.0
|
||||
lxml>=4.9.0
|
||||
openai>=1.0.0
|
||||
aiohttp>=3.9.0
|
||||
pydantic>=2.0.0
|
||||
loguru>=0.7.0
|
||||
rich>=13.0.0
|
||||
asyncio-throttle>=1.0.2
|
||||
@@ -0,0 +1,24 @@
|
||||
"""
|
||||
EPUB 双语翻译程序
|
||||
主要功能模块的初始化文件
|
||||
"""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
__author__ = "Kaitan"
|
||||
|
||||
from .epub_parser import EPUBParser
|
||||
from .translator import EPUBTranslator
|
||||
from .llm_client import OpenRouterClient
|
||||
from .text_processor import TextProcessor
|
||||
from .bilingual_builder import BilingualEPUBBuilder
|
||||
from .utils import load_config, setup_logging
|
||||
|
||||
__all__ = [
|
||||
"EPUBParser",
|
||||
"EPUBTranslator",
|
||||
"OpenRouterClient",
|
||||
"TextProcessor",
|
||||
"BilingualEPUBBuilder",
|
||||
"load_config",
|
||||
"setup_logging"
|
||||
]
|
||||
@@ -0,0 +1,155 @@
|
||||
"""
|
||||
双语 EPUB 构建器模块 - 安全的EPUB构建 (Manifest 兼容版)
|
||||
"""
|
||||
|
||||
from ebooklib import epub
|
||||
import ebooklib
|
||||
from bs4 import BeautifulSoup
|
||||
from typing import Dict, List
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
import uuid
|
||||
|
||||
|
||||
class BilingualEPUBBuilder:
|
||||
"""双语 EPUB 构建器"""
|
||||
|
||||
def __init__(self, original_book, config: Dict):
|
||||
self.original_book = original_book
|
||||
self.config = config
|
||||
self.output_config = config['output']
|
||||
|
||||
def create_bilingual_epub_with_mapping(self, translation_map: Dict[str, str],
|
||||
paragraph_map: Dict[str, Dict],
|
||||
output_path: str) -> str:
|
||||
"""
|
||||
创建双语 EPUB。使用 ordered_ids 确保与 Manifest 严格一致。
|
||||
"""
|
||||
try:
|
||||
new_book = epub.EpubBook()
|
||||
self._copy_metadata(new_book)
|
||||
new_book.toc = self.original_book.toc
|
||||
|
||||
# 准备每个文件的有序ID列表
|
||||
file_ordered_ids = {}
|
||||
sorted_pids = sorted(paragraph_map.keys(), key=lambda x: int(x.split('_')[1]))
|
||||
for pid in sorted_pids:
|
||||
info = paragraph_map[pid]
|
||||
fname = info['file_name']
|
||||
if fname not in file_ordered_ids:
|
||||
file_ordered_ids[fname] = []
|
||||
file_ordered_ids[fname].append(pid)
|
||||
|
||||
processed_item_ids = set()
|
||||
item_map = {}
|
||||
|
||||
# 复制资源
|
||||
for item in self.original_book.get_items():
|
||||
if item.get_type() != ebooklib.ITEM_DOCUMENT:
|
||||
if item.id not in processed_item_ids:
|
||||
new_book.add_item(item)
|
||||
processed_item_ids.add(item.id)
|
||||
item_map[item.id] = item
|
||||
|
||||
# 重建 Spine
|
||||
new_spine = []
|
||||
for spine_id, linear in self.original_book.spine:
|
||||
item = self.original_book.get_item_with_id(spine_id)
|
||||
if not item: continue
|
||||
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT:
|
||||
file_name = item.get_name()
|
||||
if file_name in file_ordered_ids:
|
||||
new_item = self._create_bilingual_document(
|
||||
item, file_ordered_ids[file_name], translation_map
|
||||
)
|
||||
new_item.id = item.id
|
||||
else:
|
||||
new_item = item
|
||||
|
||||
if new_item.id not in processed_item_ids:
|
||||
new_book.add_item(new_item)
|
||||
processed_item_ids.add(new_item.id)
|
||||
new_spine.append(new_item)
|
||||
else:
|
||||
if item.id in item_map:
|
||||
new_spine.append(item_map[item.id])
|
||||
|
||||
new_book.spine = new_spine
|
||||
new_book.add_item(epub.EpubNcx())
|
||||
new_book.add_item(epub.EpubNav())
|
||||
|
||||
output_file = self._generate_output_filename(output_path)
|
||||
epub.write_epub(output_file, new_book, {})
|
||||
return output_file
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"创建双语 EPUB 失败: {e}", exc_info=True)
|
||||
raise
|
||||
|
||||
def _copy_metadata(self, new_book):
|
||||
try:
|
||||
for namespace, meta_dict in self.original_book.metadata.items():
|
||||
for name, values in meta_dict.items():
|
||||
for value, other in values:
|
||||
if name and hasattr(name, 'lower') and name.lower() == 'identifier': continue
|
||||
new_book.add_metadata(namespace, name, value, other)
|
||||
new_book.add_metadata('DC', 'language', 'zh-CN')
|
||||
new_book.set_identifier(f"bilingual-{uuid.uuid4().hex[:12]}")
|
||||
|
||||
cover_id_meta = self.original_book.get_metadata('OPF', 'cover')
|
||||
if cover_id_meta:
|
||||
cover_item = self.original_book.get_item_with_id(cover_id_meta[0][0])
|
||||
if cover_item:
|
||||
new_book.add_item(cover_item)
|
||||
new_book.set_cover(cover_item.get_name(), cover_item.get_content())
|
||||
except Exception as e:
|
||||
logger.error(f"元数据复制出错: {e}")
|
||||
|
||||
def _create_bilingual_document(self, original_item, ordered_ids: list, translation_map: dict):
|
||||
try:
|
||||
from .text_processor import TextProcessor
|
||||
soup = BeautifulSoup(original_item.get_content().decode('utf-8'), 'html.parser')
|
||||
self._add_style_link(soup)
|
||||
|
||||
# 使用与 TextProcessor 相同的过滤逻辑获取元素
|
||||
text_elements = TextProcessor.get_valid_text_elements(soup)
|
||||
|
||||
current_para_index = 0
|
||||
for element in text_elements:
|
||||
if TextProcessor.is_navigation_element(element): continue
|
||||
if not TextProcessor.clean_element_text(element): continue
|
||||
|
||||
if current_para_index < len(ordered_ids):
|
||||
target_id = ordered_ids[current_para_index]
|
||||
translation = translation_map.get(target_id)
|
||||
if translation:
|
||||
self._insert_translation(element, translation, soup)
|
||||
current_para_index += 1
|
||||
|
||||
new_item = epub.EpubHtml(title=original_item.title, file_name=original_item.get_name(), lang='zh-CN')
|
||||
new_item.set_content(str(soup).encode('utf-8'))
|
||||
return new_item
|
||||
except Exception as e:
|
||||
logger.error(f"创建双语文档失败 {original_item.get_name()}: {e}")
|
||||
return original_item
|
||||
|
||||
def _add_style_link(self, soup):
|
||||
head = soup.find('head')
|
||||
if head and not head.find('link', href='style/bilingual.css'):
|
||||
head.append(soup.new_tag('link', rel='stylesheet', type='text/css', href='style/bilingual.css'))
|
||||
|
||||
def _insert_translation(self, element, translation: str, soup):
|
||||
try:
|
||||
translation_p = soup.new_tag('p')
|
||||
translation_p.string = translation
|
||||
translation_p['class'] = ['translation-text', 'chinese']
|
||||
element.insert_after(translation_p)
|
||||
except: pass
|
||||
|
||||
def _generate_output_filename(self, output_path: str) -> str:
|
||||
from .utils import sanitize_filename
|
||||
title = self.original_book.get_metadata('DC', 'title')
|
||||
clean_title = sanitize_filename(title[0][0]) if title else "bilingual_book"
|
||||
Path(output_path).mkdir(parents=True, exist_ok=True)
|
||||
return str(Path(output_path) / f"{clean_title}_bilingual.epub")
|
||||
@@ -0,0 +1,164 @@
|
||||
import json
|
||||
import random
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Tuple
|
||||
from loguru import logger
|
||||
from .manifest_manager import ManifestManager, ManifestItem
|
||||
from .llm_client import OpenRouterClient
|
||||
|
||||
class BookProfiler:
|
||||
def __init__(self, config: Dict, llm_client: OpenRouterClient):
|
||||
self.config = config
|
||||
self.llm_client = llm_client
|
||||
self.arena_models = config['llm'].get('arena_models', ["google/gemini-2.0-flash-001"])
|
||||
self.judge_model = config['llm'].get('judge_model', "google/gemini-2.0-flash-001")
|
||||
|
||||
def extract_sample_text(self, manifest: ManifestManager, char_limit: int = 3000) -> str:
|
||||
items = manifest.get_items()
|
||||
if not items: return ""
|
||||
|
||||
intro_text = []
|
||||
for item in items[:50]:
|
||||
if len(item.clean_text) > 50:
|
||||
intro_text.append(item.clean_text)
|
||||
|
||||
body_text = []
|
||||
body_items = [i for i in items[50:] if len(i.clean_text) > 80]
|
||||
if body_items:
|
||||
samples = random.sample(body_items, min(5, len(body_items)))
|
||||
body_text = [i.clean_text for i in samples]
|
||||
|
||||
full_text = "\n\n".join(intro_text[:5] + body_text)
|
||||
return full_text[:char_limit]
|
||||
|
||||
async def analyze_book(self, manifest: ManifestManager) -> Dict:
|
||||
"""Generate Book Profile and store in Manifest Metadata."""
|
||||
# 1. Check if profile already exists in manifest
|
||||
existing_profile = manifest.data.get('metadata', {}).get('profile')
|
||||
if existing_profile:
|
||||
logger.info("Loaded existing Book Profile from Manifest")
|
||||
return existing_profile
|
||||
|
||||
# 2. Generate new profile
|
||||
sample = self.extract_sample_text(manifest)
|
||||
if not sample: return {}
|
||||
|
||||
logger.info("Generating Book Profile (Genre, Style, Glossary)...")
|
||||
|
||||
system_prompt = "You are a senior publishing editor. Analyze the text and output JSON."
|
||||
user_prompt = f"""
|
||||
Please analyze the following book excerpt.
|
||||
Output JSON format:
|
||||
{{
|
||||
"genre": "Genre (e.g. Business Biography, Hard Sci-Fi, History)",
|
||||
"style": "Style description (e.g. Serious, Humorous, Concise)",
|
||||
"audience": "Target Audience",
|
||||
"glossary": {{ "Term/Name": "Chinese Translation" }},
|
||||
"translation_instruction": "Specific instruction for translator (e.g. 'Keep tone objective, use standard names')"
|
||||
}}
|
||||
|
||||
Excerpt:
|
||||
{sample}
|
||||
"""
|
||||
try:
|
||||
response = await self.llm_client.raw_chat_completion(system_prompt, user_prompt, model_type=self.judge_model)
|
||||
json_str = response.strip()
|
||||
if "```json" in json_str:
|
||||
json_str = json_str.split("```json")[1].split("```")[0].strip()
|
||||
elif "```" in json_str:
|
||||
json_str = json_str.split("```")[1].split("```")[0].strip()
|
||||
|
||||
profile = json.loads(json_str)
|
||||
|
||||
# 3. Save to Manifest
|
||||
if 'metadata' not in manifest.data:
|
||||
manifest.data['metadata'] = {}
|
||||
manifest.data['metadata']['profile'] = profile
|
||||
manifest.save()
|
||||
|
||||
return profile
|
||||
except Exception as e:
|
||||
logger.error(f"Profile generation failed: {e}")
|
||||
return {}
|
||||
|
||||
async def run_arena(self, manifest: ManifestManager, profile: Dict) -> str:
|
||||
"""Run the Arena and return the winner model ID."""
|
||||
# Check if winner already exists
|
||||
existing_winner = manifest.data.get('metadata', {}).get('arena_winner')
|
||||
if existing_winner:
|
||||
logger.info(f"Loaded existing Arena Winner: {existing_winner}")
|
||||
return existing_winner
|
||||
|
||||
logger.info(f"🏟️ Starting Model Arena! Contestants: {self.arena_models}")
|
||||
|
||||
all_items = manifest.get_items()
|
||||
start_idx = min(len(all_items) // 5, 50)
|
||||
arena_chunk = []
|
||||
for i in range(start_idx, len(all_items)):
|
||||
if len(all_items[i].clean_text) > 50:
|
||||
arena_chunk = all_items[i:i+5]
|
||||
break
|
||||
|
||||
if not arena_chunk:
|
||||
logger.warning("No suitable arena chunk found, defaulting to first model")
|
||||
return self.arena_models[0]
|
||||
|
||||
tasks = []
|
||||
for model in self.arena_models:
|
||||
instruction = profile.get("translation_instruction", "")
|
||||
glossary = profile.get("glossary", {})
|
||||
tasks.append(self.llm_client.translate_chunk(arena_chunk, glossary=glossary, instruction=instruction, model_id_override=model))
|
||||
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
candidates = []
|
||||
for model, res in zip(self.arena_models, results):
|
||||
if isinstance(res, dict) and res:
|
||||
trans_text = "\n".join(res.values())
|
||||
candidates.append({"model": model, "text": trans_text})
|
||||
|
||||
if not candidates:
|
||||
logger.error("All models failed, using default")
|
||||
winner = self.arena_models[0]
|
||||
else:
|
||||
winner = await self._judge_candidates(arena_chunk, candidates, profile)
|
||||
|
||||
logger.info(f"🏆 Winner: {winner}")
|
||||
|
||||
# Save winner to manifest
|
||||
if 'metadata' not in manifest.data:
|
||||
manifest.data['metadata'] = {}
|
||||
manifest.data['metadata']['arena_winner'] = winner
|
||||
manifest.save()
|
||||
|
||||
return winner
|
||||
|
||||
async def _judge_candidates(self, source_items: List[ManifestItem], candidates: List[Dict], profile: Dict) -> str:
|
||||
src_text = "\n".join([i.clean_text for i in source_items])
|
||||
|
||||
candidates_str = ""
|
||||
for i, c in enumerate(candidates):
|
||||
candidates_str += f"\n=== Candidate {i+1} ({c['model']}) ===\n{c['text']}\n"
|
||||
|
||||
prompt = f"""
|
||||
Source:
|
||||
{src_text}
|
||||
|
||||
Book Context: {profile.get('genre', '')}, {profile.get('style', '')}
|
||||
|
||||
{candidates_str}
|
||||
|
||||
As a senior editor, rate these translations based on accuracy, style, and terminology.
|
||||
Return ONLY the Model ID of the winner.
|
||||
Example: google/gemini-2.0-flash-001
|
||||
"""
|
||||
try:
|
||||
winner = await self.llm_client.raw_chat_completion("You are a judge.", prompt, model_type=self.judge_model)
|
||||
winner = winner.strip()
|
||||
for c in candidates:
|
||||
if c['model'] in winner:
|
||||
return c['model']
|
||||
return candidates[0]['model']
|
||||
except:
|
||||
return candidates[0]['model']
|
||||
@@ -0,0 +1,225 @@
|
||||
"""
|
||||
翻译缓存管理模块 - 简化版
|
||||
基于全局ID和chunk的缓存系统
|
||||
"""
|
||||
|
||||
import json
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Dict, Optional, List
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class TranslationCache:
|
||||
"""翻译缓存管理器 - 简化版"""
|
||||
|
||||
def __init__(self, config: Dict):
|
||||
"""初始化缓存管理器"""
|
||||
self.config = config
|
||||
cache_config = config.get('cache', {})
|
||||
|
||||
self.enabled = cache_config.get('enabled', True)
|
||||
self.cache_dir = Path(cache_config.get('directory', 'cache'))
|
||||
self.max_age_days = cache_config.get('max_age_days', 30)
|
||||
|
||||
if self.enabled:
|
||||
self.cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
self.translations_dir = self.cache_dir / 'translations'
|
||||
self.translations_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.info(f"翻译缓存已启用: {self.cache_dir}")
|
||||
|
||||
def get_chunk_translation(self, chunk: List[Dict], model: str) -> Optional[Dict[str, str]]:
|
||||
"""
|
||||
获取chunk的缓存翻译
|
||||
|
||||
Args:
|
||||
chunk: 段落列表（带global_id）
|
||||
model: 模型名称
|
||||
|
||||
Returns:
|
||||
{global_id: translation} 映射，如果不存在返回 None
|
||||
"""
|
||||
if not self.enabled:
|
||||
return None
|
||||
|
||||
try:
|
||||
cache_key = self._get_chunk_cache_key(chunk, model)
|
||||
cache_file = self._get_cache_file_path(cache_key)
|
||||
|
||||
if not cache_file.exists():
|
||||
return None
|
||||
|
||||
# 检查是否过期
|
||||
file_age = datetime.now() - datetime.fromtimestamp(cache_file.stat().st_mtime)
|
||||
if file_age > timedelta(days=self.max_age_days):
|
||||
logger.debug(f"缓存已过期: {cache_key[:8]}...")
|
||||
cache_file.unlink()
|
||||
return None
|
||||
|
||||
# 读取缓存
|
||||
with open(cache_file, 'r', encoding='utf-8') as f:
|
||||
cache_data = json.load(f)
|
||||
|
||||
# 验证缓存
|
||||
if (cache_data.get('success') and
|
||||
cache_data.get('model') == model and
|
||||
self._validate_cache_data(cache_data, chunk)):
|
||||
|
||||
logger.debug(f"缓存命中: {cache_key[:8]}... ({len(chunk)} 段落)")
|
||||
return cache_data.get('translations', {})
|
||||
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"读取缓存失败: {e}")
|
||||
return None
|
||||
|
||||
def save_chunk_translation(self, chunk: List[Dict], translations: Dict[str, str],
|
||||
model: str, success: bool = True) -> None:
|
||||
"""
|
||||
保存chunk翻译到缓存
|
||||
|
||||
Args:
|
||||
chunk: 段落列表（带global_id）
|
||||
translations: {global_id: translation} 映射
|
||||
model: 模型名称
|
||||
success: 是否翻译成功
|
||||
"""
|
||||
if not self.enabled:
|
||||
return
|
||||
|
||||
try:
|
||||
cache_key = self._get_chunk_cache_key(chunk, model)
|
||||
cache_file = self._get_cache_file_path(cache_key)
|
||||
|
||||
# 构建缓存数据
|
||||
cache_data = {
|
||||
'global_ids': [p['global_id'] for p in chunk],
|
||||
'translations': translations,
|
||||
'model': model,
|
||||
'timestamp': datetime.now().isoformat(),
|
||||
'success': success,
|
||||
'paragraph_count': len(chunk),
|
||||
'cache_version': '3.0'
|
||||
}
|
||||
|
||||
with open(cache_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(cache_data, f, ensure_ascii=False, indent=2)
|
||||
|
||||
logger.debug(f"缓存已保存: {cache_key[:8]}... ({len(chunk)} 段落)")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"保存缓存失败: {e}")
|
||||
|
||||
def _get_chunk_cache_key(self, chunk: List[Dict], model: str) -> str:
|
||||
"""
|
||||
生成chunk缓存键（基于全局ID序列）
|
||||
|
||||
Args:
|
||||
chunk: 段落列表
|
||||
model: 模型名称
|
||||
|
||||
Returns:
|
||||
缓存键
|
||||
"""
|
||||
# 使用全局ID序列作为缓存键的一部分
|
||||
id_sequence = ",".join(p['global_id'] for p in chunk)
|
||||
combined = f"{id_sequence}|{model}"
|
||||
return hashlib.md5(combined.encode('utf-8')).hexdigest()
|
||||
|
||||
def _get_cache_file_path(self, cache_key: str) -> Path:
|
||||
"""获取缓存文件路径"""
|
||||
today = datetime.now().strftime('%Y-%m-%d')
|
||||
cache_date_dir = self.translations_dir / today
|
||||
cache_date_dir.mkdir(parents=True, exist_ok=True)
|
||||
return cache_date_dir / f"{cache_key}.json"
|
||||
|
||||
def _validate_cache_data(self, cache_data: Dict, chunk: List[Dict]) -> bool:
|
||||
"""验证缓存数据的有效性"""
|
||||
# 检查ID序列是否匹配
|
||||
cached_ids = cache_data.get('global_ids', [])
|
||||
chunk_ids = [p['global_id'] for p in chunk]
|
||||
|
||||
if cached_ids != chunk_ids:
|
||||
logger.debug("缓存ID序列不匹配")
|
||||
return False
|
||||
|
||||
# 检查翻译数量
|
||||
translations = cache_data.get('translations', {})
|
||||
if len(translations) != len(chunk):
|
||||
logger.debug("缓存翻译数量不匹配")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def clear_cache(self, older_than_days: Optional[int] = None) -> int:
|
||||
"""清理缓存"""
|
||||
if not self.enabled or not self.translations_dir.exists():
|
||||
return 0
|
||||
|
||||
cleared_count = 0
|
||||
cutoff_time = None
|
||||
|
||||
if older_than_days is not None:
|
||||
cutoff_time = datetime.now() - timedelta(days=older_than_days)
|
||||
|
||||
try:
|
||||
for cache_file in self.translations_dir.rglob('*.json'):
|
||||
should_delete = False
|
||||
|
||||
if cutoff_time is None:
|
||||
should_delete = True
|
||||
else:
|
||||
file_time = datetime.fromtimestamp(cache_file.stat().st_mtime)
|
||||
should_delete = file_time < cutoff_time
|
||||
|
||||
if should_delete:
|
||||
cache_file.unlink()
|
||||
cleared_count += 1
|
||||
|
||||
# 清理空目录
|
||||
for date_dir in self.translations_dir.iterdir():
|
||||
if date_dir.is_dir() and not any(date_dir.iterdir()):
|
||||
date_dir.rmdir()
|
||||
|
||||
logger.info(f"清理了 {cleared_count} 个缓存文件")
|
||||
return cleared_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"清理缓存失败: {e}")
|
||||
return 0
|
||||
|
||||
def get_cache_stats(self) -> Dict:
|
||||
"""获取缓存统计信息"""
|
||||
if not self.enabled or not self.translations_dir.exists():
|
||||
return {'enabled': False}
|
||||
|
||||
try:
|
||||
cache_files = list(self.translations_dir.rglob('*.json'))
|
||||
total_files = len(cache_files)
|
||||
total_size = sum(f.stat().st_size for f in cache_files)
|
||||
|
||||
# 统计段落数
|
||||
total_paragraphs = 0
|
||||
for cache_file in cache_files:
|
||||
try:
|
||||
with open(cache_file, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
total_paragraphs += data.get('paragraph_count', 0)
|
||||
except:
|
||||
continue
|
||||
|
||||
return {
|
||||
'enabled': True,
|
||||
'total_files': total_files,
|
||||
'total_paragraphs': total_paragraphs,
|
||||
'total_size_mb': round(total_size / 1024 / 1024, 2),
|
||||
'cache_directory': str(self.cache_dir),
|
||||
'max_age_days': self.max_age_days
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"获取缓存统计失败: {e}")
|
||||
return {'enabled': True, 'error': str(e)}
|
||||
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
EPUB 解析器模块 (EPUB Parser Module)
|
||||
|
||||
该模块负责读取 EPUB 文件，提取元数据和内容项目。
|
||||
它使用 ebooklib 库来处理 EPUB 格式的底层细节。
|
||||
|
||||
Classes:
|
||||
EPUBParser: 负责 EPUB 文件的加载、元数据提取和内容项遍历。
|
||||
"""
|
||||
|
||||
import ebooklib
|
||||
from ebooklib import epub
|
||||
from bs4 import BeautifulSoup
|
||||
from typing import List, Dict, Any
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class EPUBParser:
|
||||
"""
|
||||
EPUB 文件解析器。
|
||||
|
||||
负责加载 EPUB 文件，提取书籍元数据（如标题、作者），并提供方法来遍历和提取
|
||||
书中的文档内容（HTML/XHTML）。
|
||||
|
||||
Attributes:
|
||||
epub_path (Path): EPUB 文件的路径对象。
|
||||
book (epub.EpubBook): ebooklib 加载的书籍对象。
|
||||
metadata (Dict[str, str]): 提取的书籍元数据字典。
|
||||
"""
|
||||
|
||||
def __init__(self, epub_path: str):
|
||||
"""
|
||||
初始化 EPUB 解析器。
|
||||
|
||||
Args:
|
||||
epub_path (str): EPUB 文件的文件路径。
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: 如果指定的文件不存在。
|
||||
Exception: 如果 EPUB 文件加载失败（格式错误等）。
|
||||
"""
|
||||
self.epub_path = Path(epub_path)
|
||||
if not self.epub_path.exists():
|
||||
raise FileNotFoundError(f"EPUB 文件不存在: {epub_path}")
|
||||
|
||||
try:
|
||||
# ignore_ncx=True 是为了避免某些旧版 epub 的警告，但新版 ebooklib 可能行为不同
|
||||
# 这里直接读取，让 ebooklib 处理
|
||||
self.book = epub.read_epub(str(self.epub_path))
|
||||
logger.info(f"成功加载 EPUB: {self.epub_path.name}")
|
||||
except Exception as e:
|
||||
logger.error(f"加载 EPUB 失败: {e}")
|
||||
raise
|
||||
|
||||
self.metadata = self._extract_metadata()
|
||||
|
||||
def _extract_metadata(self) -> Dict[str, str]:
|
||||
"""
|
||||
从 EPUB 对象中提取标准元数据。
|
||||
|
||||
提取 Dublin Core (DC) 元数据，包括标题、作者和语言。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 包含 'title', 'author', 'language' 的字典。
|
||||
如果提取失败，会使用默认值 ("Unknown", "en")。
|
||||
"""
|
||||
metadata = {}
|
||||
|
||||
try:
|
||||
# get_metadata 返回的是 (value, dict) 的列表，我们取第一个结果
|
||||
title_meta = self.book.get_metadata('DC', 'title')
|
||||
metadata['title'] = title_meta[0][0] if title_meta else "Unknown"
|
||||
|
||||
author_meta = self.book.get_metadata('DC', 'creator')
|
||||
metadata['author'] = author_meta[0][0] if author_meta else "Unknown"
|
||||
|
||||
lang_meta = self.book.get_metadata('DC', 'language')
|
||||
metadata['language'] = lang_meta[0][0] if lang_meta else "en"
|
||||
|
||||
logger.info(f"书籍: {metadata['title']} - {metadata['author']}")
|
||||
except Exception as e:
|
||||
logger.warning(f"提取元数据时出错: {e}")
|
||||
# 设置保底值
|
||||
metadata.setdefault('title', 'Unknown')
|
||||
metadata.setdefault('author', 'Unknown')
|
||||
metadata.setdefault('language', 'en')
|
||||
|
||||
return metadata
|
||||
|
||||
def extract_all_content_items(self) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
提取所有可翻译的内容项目（文档）。
|
||||
|
||||
遍历 EPUB 中的所有 Item，筛选出类型为 ITEM_DOCUMENT 的项目。
|
||||
同时会进行简单的过滤，跳过内容过短（<100字符）或看起来像非正文的文件（如 nav, toc, cover）。
|
||||
|
||||
Returns:
|
||||
List[Dict[str, Any]]: 内容项目列表。每个字典包含：
|
||||
- item (epub.EpubItem): 原始 Item 对象。
|
||||
- file_name (str): 文件名。
|
||||
- content (str): 解码后的 HTML 内容。
|
||||
- text_length (int): 纯文本长度（用于统计）。
|
||||
"""
|
||||
content_items = []
|
||||
|
||||
# 获取所有文档类型的项目
|
||||
for item in self.book.get_items():
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT:
|
||||
try:
|
||||
# 获取内容 (bytes -> str)
|
||||
content = item.get_content().decode('utf-8')
|
||||
|
||||
# 简单的内容验证：提取纯文本检查长度
|
||||
soup = BeautifulSoup(content, 'html.parser')
|
||||
text = soup.get_text().strip()
|
||||
|
||||
# 1. 跳过太短的内容（可能是只有图片的页面、空页面）
|
||||
if len(text) < 100:
|
||||
logger.debug(f"跳过短内容: {item.get_name()} ({len(text)} 字符)")
|
||||
continue
|
||||
|
||||
# 2. 跳过明显的非正文内容 (根据文件名判断)
|
||||
name_lower = item.get_name().lower()
|
||||
skip_patterns = ['cover', 'copyright', 'titlepage', 'halftitle',
|
||||
'nav.xhtml', 'toc.xhtml']
|
||||
if any(pattern in name_lower for pattern in skip_patterns):
|
||||
logger.debug(f"跳过非正文内容: {item.get_name()}")
|
||||
continue
|
||||
|
||||
content_items.append({
|
||||
'item': item,
|
||||
'file_name': item.get_name(),
|
||||
'content': content,
|
||||
'text_length': len(text)
|
||||
})
|
||||
|
||||
logger.debug(f"添加内容项: {item.get_name()} ({len(text)} 字符)")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"处理项目失败 {item.get_name()}: {e}")
|
||||
continue
|
||||
|
||||
logger.info(f"提取了 {len(content_items)} 个内容项目")
|
||||
return content_items
|
||||
|
||||
def get_book_info(self) -> Dict[str, str]:
|
||||
"""
|
||||
获取书籍的摘要信息。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 包含文件名、标题、作者、语言和文档数量的字典。
|
||||
"""
|
||||
# 统计内容项
|
||||
document_count = sum(1 for item in self.book.get_items()
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT)
|
||||
|
||||
return {
|
||||
'filename': self.epub_path.name,
|
||||
'title': self.metadata.get('title', 'Unknown'),
|
||||
'author': self.metadata.get('author', 'Unknown'),
|
||||
'language': self.metadata.get('language', 'en'),
|
||||
'document_count': document_count
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
"""
|
||||
术语表管理器 (Glossary Manager)
|
||||
|
||||
负责从书籍内容中提取采样文本，调用 LLM 生成术语表，并管理术语表的持久化。
|
||||
"""
|
||||
|
||||
import json
|
||||
import random
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Any
|
||||
from loguru import logger
|
||||
from .manifest_manager import ManifestManager
|
||||
from .llm_client import OpenRouterClient
|
||||
|
||||
class GlossaryManager:
|
||||
def __init__(self, config: Dict, llm_client: OpenRouterClient):
|
||||
self.config = config
|
||||
self.llm_client = llm_client
|
||||
self.glossary_path = Path("cache/glossary.json")
|
||||
self.prompts = self._load_prompts()
|
||||
|
||||
def _load_prompts(self) -> Dict:
|
||||
try:
|
||||
with open("config/prompts.json", "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except Exception:
|
||||
logger.warning("未找到 config/prompts.json，使用默认 Prompt")
|
||||
return {}
|
||||
|
||||
def extract_samples(self, manifest: ManifestManager, sample_size: int = 3000) -> str:
|
||||
"""
|
||||
从 Manifest 中提取采样文本。
|
||||
策略：
|
||||
1. 优先提取前言/绪论 (通常在文件的前部)。
|
||||
2. 随机抽取中间段落。
|
||||
"""
|
||||
all_items = manifest.get_items()
|
||||
if not all_items:
|
||||
return ""
|
||||
|
||||
# 1. 提取开头部分 (Preface/Intro) - 假设在前 50 个段落中
|
||||
intro_sample = [item.clean_text for item in all_items[:50] if len(item.clean_text) > 50]
|
||||
|
||||
# 2. 随机提取正文
|
||||
body_items = [item for item in all_items[50:] if len(item.clean_text) > 50]
|
||||
random_sample = []
|
||||
if body_items:
|
||||
# 随机取 10 个片段
|
||||
sample_count = min(10, len(body_items))
|
||||
random_items = random.sample(body_items, sample_count)
|
||||
random_sample = [item.clean_text for item in random_items]
|
||||
|
||||
# 组合并截断
|
||||
full_text = "\n\n".join(intro_sample + random_sample)
|
||||
if len(full_text) > sample_size:
|
||||
full_text = full_text[:sample_size] + "..."
|
||||
|
||||
return full_text
|
||||
|
||||
async def generate_glossary(self, manifest: ManifestManager) -> Dict[str, str]:
|
||||
"""
|
||||
生成术语表。
|
||||
"""
|
||||
# 1. 采样
|
||||
sample_text = self.extract_samples(manifest)
|
||||
if not sample_text:
|
||||
logger.warning("采样文本为空，跳过术语表生成")
|
||||
return {}
|
||||
|
||||
logger.info(f"提取了 {len(sample_text)} 字符的采样文本，正在生成术语表...")
|
||||
|
||||
# 2. 构建 Prompt
|
||||
prompt_cfg = self.prompts.get("glossary_extraction", {})
|
||||
system_prompt = prompt_cfg.get("system", "Analyze the text and extract named entities.")
|
||||
user_template = prompt_cfg.get("user_template", "Text:\n{{content}}")
|
||||
user_prompt = user_template.replace("{{content}}", sample_text)
|
||||
|
||||
# 3. 调用 LLM (使用 smart 模型)
|
||||
# 注意：这里需要 LLMClient 支持直接传入 system/user prompt，而不是封装好的 translate 接口
|
||||
# 我们稍后会扩展 LLMClient
|
||||
try:
|
||||
response = await self.llm_client.raw_chat_completion(
|
||||
system_prompt,
|
||||
user_prompt,
|
||||
model_type="smart"
|
||||
)
|
||||
|
||||
# 4. 解析 JSON
|
||||
# 简单的 JSON 提取逻辑 (处理可能的 markdown code block)
|
||||
json_str = response.strip()
|
||||
if "```json" in json_str:
|
||||
json_str = json_str.split("```json")[1].split("```")[0].strip()
|
||||
elif "```" in json_str:
|
||||
json_str = json_str.split("```")[1].split("```")[0].strip()
|
||||
|
||||
glossary = json.loads(json_str)
|
||||
self.save_glossary(glossary)
|
||||
return glossary
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"术语表生成失败: {e}")
|
||||
return {}
|
||||
|
||||
def save_glossary(self, glossary: Dict[str, str]):
|
||||
self.glossary_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(self.glossary_path, "w", encoding="utf-8") as f:
|
||||
json.dump(glossary, f, ensure_ascii=False, indent=2)
|
||||
logger.info(f"术语表已保存至: {self.glossary_path}")
|
||||
|
||||
def load_glossary(self) -> Dict[str, str]:
|
||||
if self.glossary_path.exists():
|
||||
try:
|
||||
with open(self.glossary_path, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except:
|
||||
pass
|
||||
return {}
|
||||
@@ -0,0 +1,180 @@
|
||||
"""
|
||||
LLM Client Module - v0.05
|
||||
|
||||
Features:
|
||||
1. Pure p_xxxxx ID format.
|
||||
2. Direct string finding parsing.
|
||||
3. Arena support (model_id_override).
|
||||
4. Profile instruction injection.
|
||||
5. Dual RateLimiters (Main/QC).
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from openai import AsyncOpenAI
|
||||
from typing import List, Dict, Optional, Any
|
||||
from loguru import logger
|
||||
import time
|
||||
from .manifest_manager import ManifestItem
|
||||
|
||||
|
||||
class RateLimiter:
|
||||
"""Rate limiter for concurrency and RPM."""
|
||||
def __init__(self, requests_per_minute: int, concurrent_requests: int):
|
||||
self.semaphore = asyncio.Semaphore(concurrent_requests)
|
||||
self.min_interval = 60.0 / requests_per_minute if requests_per_minute > 0 else 0
|
||||
self.last_request_time = 0
|
||||
|
||||
async def acquire(self):
|
||||
await self.semaphore.acquire()
|
||||
current_time = time.time()
|
||||
wait_time = self.min_interval - (current_time - self.last_request_time)
|
||||
if wait_time > 0:
|
||||
await asyncio.sleep(wait_time)
|
||||
self.last_request_time = time.time()
|
||||
|
||||
def release(self):
|
||||
self.semaphore.release()
|
||||
|
||||
|
||||
class OpenRouterClient:
|
||||
"""Advanced Client for Arena & Profiling."""
|
||||
|
||||
def __init__(self, config: Dict):
|
||||
self.config = config
|
||||
or_config = config["llm"]
|
||||
api_key = or_config.get("api_key")
|
||||
if not api_key or api_key == "YOUR_OPENROUTER_API_KEY":
|
||||
raise ValueError("Invalid OpenRouter API Key")
|
||||
|
||||
self.client = AsyncOpenAI(
|
||||
base_url=or_config["base_url"],
|
||||
api_key=api_key,
|
||||
default_headers={"HTTP-Referer": "https://github.com/epub-translator", "X-Title": "EPUB Translator"}
|
||||
)
|
||||
self.models = or_config["models"]
|
||||
|
||||
# Main limiter
|
||||
self.rate_limiter = RateLimiter(
|
||||
or_config["rate_limits"]["requests_per_minute"],
|
||||
or_config["rate_limits"]["concurrent_requests"]
|
||||
)
|
||||
# QC/Arena limiter (smaller concurrency)
|
||||
self.qc_rate_limiter = RateLimiter(
|
||||
or_config["rate_limits"]["requests_per_minute"],
|
||||
5
|
||||
)
|
||||
|
||||
def _load_prompts(self) -> Dict:
|
||||
try:
|
||||
with open("config/prompts.json", "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except:
|
||||
return {}
|
||||
|
||||
async def translate_chunk(self, items: List[ManifestItem], glossary: Dict = None,
|
||||
instruction: str = None,
|
||||
model_type: str = "fast",
|
||||
model_id_override: str = None) -> Dict[str, str]:
|
||||
"""Translate a chunk."""
|
||||
if not items: return {}
|
||||
|
||||
# Determine model
|
||||
if model_id_override:
|
||||
model = model_id_override
|
||||
else:
|
||||
model = self.models.get(model_type, self.models.get("fast"))
|
||||
|
||||
prompt = self._build_prompt(items)
|
||||
|
||||
try:
|
||||
# Build System Prompt
|
||||
base_sys_prompt = "You are a professional translator."
|
||||
|
||||
if instruction:
|
||||
base_sys_prompt += f"\n\nBook Style Guide:\n{instruction}"
|
||||
|
||||
if glossary:
|
||||
glossary_text = "\n".join([f"{k} -> {v}" for k, v in glossary.items()])
|
||||
base_sys_prompt += f"\n\nTerminology:\n{glossary_text}"
|
||||
|
||||
# Strict formatting instructions (Minimal ID)
|
||||
base_sys_prompt += "\n\nRequirements:\n1. Each line MUST start with ID (p_xxxxx).\n2. DO NOT modify IDs or add brackets.\n3. Return only translations."
|
||||
|
||||
# Use QC limiter for Arena (override), Main limiter for bulk
|
||||
limiter = self.qc_rate_limiter if model_id_override else self.rate_limiter
|
||||
|
||||
raw_response = await self._make_request(model, base_sys_prompt, prompt, limiter)
|
||||
|
||||
if not raw_response:
|
||||
return {item.global_id: f"[Error - Empty Response]" for item in items}
|
||||
|
||||
return self._simple_parse(raw_response, items)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Translation failed ({model}): {e}")
|
||||
return {item.global_id: f"[Error - {str(e)}]" for item in items}
|
||||
|
||||
async def raw_chat_completion(self, system_prompt: str, user_prompt: str, model_type: str = "smart") -> str:
|
||||
"""Generic chat completion (for Profiler/Arena)."""
|
||||
# If model_type is a full ID (e.g. from arena config), use it directly
|
||||
if "/" in model_type:
|
||||
model = model_type
|
||||
else:
|
||||
model = self.models.get(model_type, self.models.get("smart"))
|
||||
|
||||
return await self._make_request(model, system_prompt, user_prompt, self.qc_rate_limiter)
|
||||
|
||||
def _build_prompt(self, items: List[ManifestItem]) -> str:
|
||||
lines = []
|
||||
for item in items:
|
||||
lines.append(f"{item.global_id} {item.clean_text}")
|
||||
return "\n".join(lines)
|
||||
|
||||
def _simple_parse(self, response: str, items: List[ManifestItem]) -> Dict[str, str]:
|
||||
results = {}
|
||||
for i, item in enumerate(items):
|
||||
current_id = item.global_id
|
||||
start_idx = response.find(current_id)
|
||||
if start_idx == -1: continue
|
||||
|
||||
end_idx = len(response)
|
||||
if i + 1 < len(items):
|
||||
next_id = items[i+1].global_id
|
||||
next_found = response.find(next_id, start_idx + len(current_id))
|
||||
if next_found != -1:
|
||||
end_idx = next_found
|
||||
|
||||
content = response[start_idx:end_idx].strip()
|
||||
clean_content = content[len(current_id):].strip()
|
||||
clean_content = clean_content.lstrip(":： \t")
|
||||
if clean_content:
|
||||
results[current_id] = clean_content
|
||||
|
||||
if len(results) < len(items):
|
||||
for line in response.split("\n"):
|
||||
line = line.strip()
|
||||
for item in items:
|
||||
if item.global_id not in results and line.startswith(item.global_id):
|
||||
res = line[len(item.global_id):].strip().lstrip(":： ")
|
||||
if res: results[item.global_id] = res
|
||||
return results
|
||||
|
||||
async def _make_request(self, model: str, system_prompt: str, user_prompt: str, limiter: RateLimiter) -> str:
|
||||
await limiter.acquire()
|
||||
try:
|
||||
resp = await self.client.chat.completions.create(
|
||||
model=model,
|
||||
messages=[
|
||||
{"role": "system", "content": system_prompt},
|
||||
{"role": "user", "content": user_prompt}
|
||||
],
|
||||
temperature=0.3,
|
||||
max_tokens=8000
|
||||
)
|
||||
return resp.choices[0].message.content.strip()
|
||||
finally:
|
||||
limiter.release()
|
||||
|
||||
async def close(self):
|
||||
await self.client.close()
|
||||
@@ -0,0 +1,157 @@
|
||||
"""
|
||||
Manifest 管理器模块 (Manifest Manager Module)
|
||||
|
||||
该模块是系统的单一真理源 (SSOT)。
|
||||
它记录了每一段文本的原始状态、清洗后的文本、哈希值以及翻译状态。
|
||||
所有对翻译流程的操作（提取、翻译、回填）都必须通过修改此 Manifest 进行。
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import hashlib
|
||||
from typing import List, Dict, Optional, Any
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
from dataclasses import dataclass, asdict, field
|
||||
|
||||
@dataclass
|
||||
class ManifestItem:
|
||||
"""代表一个翻译单元（通常是一个段落）"""
|
||||
global_id: str
|
||||
source_file: str
|
||||
original_html: str
|
||||
clean_text: str
|
||||
text_hash: str
|
||||
tag: str
|
||||
translation: Optional[str] = None
|
||||
status: str = "pending" # pending, translated, ignored, failed
|
||||
error_msg: Optional[str] = None
|
||||
model_used: Optional[str] = None # 记录使用的模型
|
||||
quality_score: Optional[int] = None # 记录质量评分
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def to_dict(self):
|
||||
return asdict(self)
|
||||
|
||||
class ManifestManager:
|
||||
"""
|
||||
负责 Manifest 的生命周期管理。
|
||||
"""
|
||||
|
||||
def __init__(self, manifest_path: str):
|
||||
self.manifest_path = Path(manifest_path)
|
||||
self.data: Dict[str, Any] = {
|
||||
"book_id": "",
|
||||
"metadata": {},
|
||||
"items": []
|
||||
}
|
||||
self._items_by_id: Dict[str, ManifestItem] = {}
|
||||
|
||||
def load(self) -> bool:
|
||||
"""从文件加载 Manifest。如果文件不存在则返回 False。"""
|
||||
if self.manifest_path.exists():
|
||||
try:
|
||||
with open(self.manifest_path, 'r', encoding='utf-8') as f:
|
||||
self.data = json.load(f)
|
||||
|
||||
# 重建对象映射
|
||||
self._items_by_id = {
|
||||
item['global_id']: ManifestItem(**item)
|
||||
for item in self.data["items"]
|
||||
}
|
||||
logger.info(f"成功从 {self.manifest_path} 加载 Manifest, 包含 {len(self._items_by_id)} 个项目")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"加载 Manifest 失败: {e}")
|
||||
return False
|
||||
return False
|
||||
|
||||
def save(self):
|
||||
"""将当前状态保存到 Manifest 文件。"""
|
||||
# 确保目录存在
|
||||
self.manifest_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 同步 items 到 data 字典
|
||||
self.data["items"] = [item.to_dict() for item in self._items_by_id.values()]
|
||||
|
||||
with open(self.manifest_path, 'w', encoding='utf-8') as f:
|
||||
json.dump(self.data, f, ensure_ascii=False, indent=2)
|
||||
# logger.debug(f"Manifest 已保存到 {self.manifest_path}")
|
||||
|
||||
def init_manifest(self, book_id: str, metadata: Dict):
|
||||
"""初始化一个新的 Manifest。"""
|
||||
self.data = {
|
||||
"book_id": book_id,
|
||||
"metadata": metadata,
|
||||
"items": []
|
||||
}
|
||||
self._items_by_id = {}
|
||||
self.save()
|
||||
|
||||
def add_item(self, source_file: str, original_html: str, clean_text: str, tag: str, metadata: Dict = None) -> ManifestItem:
|
||||
"""添加一个新的翻译项并分配 ID。"""
|
||||
# 生成全局 ID
|
||||
new_index = len(self._items_by_id) + 1
|
||||
global_id = f"p_{new_index:05d}"
|
||||
|
||||
# 生成内容哈希 (用于排重和缓存)
|
||||
text_hash = hashlib.sha256(clean_text.encode('utf-8')).hexdigest()
|
||||
|
||||
item = ManifestItem(
|
||||
global_id=global_id,
|
||||
source_file=source_file,
|
||||
original_html=original_html,
|
||||
clean_text=clean_text,
|
||||
text_hash=text_hash,
|
||||
tag=tag,
|
||||
metadata=metadata or {}
|
||||
)
|
||||
|
||||
self._items_by_id[global_id] = item
|
||||
return item
|
||||
|
||||
def get_items(self, status: str = None, file_name: str = None) -> List[ManifestItem]:
|
||||
"""按状态或文件名查询项目。"""
|
||||
items = list(self._items_by_id.values())
|
||||
if status:
|
||||
items = [i for i in items if i.status == status]
|
||||
if file_name:
|
||||
items = [i for i in items if i.source_file == file_name]
|
||||
|
||||
# 必须按 ID 顺序返回以保证分块正确
|
||||
return sorted(items, key=lambda x: x.global_id)
|
||||
|
||||
def update_item(self, global_id: str, translation: str, status: str = "translated", error: str = None, model: str = None, score: int = None):
|
||||
"""更新翻译结果。"""
|
||||
if global_id in self._items_by_id:
|
||||
item = self._items_by_id[global_id]
|
||||
if translation is not None:
|
||||
item.translation = translation
|
||||
item.status = status
|
||||
if error:
|
||||
item.error_msg = error
|
||||
if model:
|
||||
item.model_used = model
|
||||
if score is not None:
|
||||
item.quality_score = score
|
||||
else:
|
||||
logger.warning(f"尝试更新不存在的 ID: {global_id}")
|
||||
|
||||
@property
|
||||
def stats(self) -> Dict:
|
||||
"""获取翻译进度统计。"""
|
||||
total = len(self._items_by_id)
|
||||
if total == 0: return {"progress": "0%"}
|
||||
|
||||
translated = sum(1 for i in self._items_by_id.values() if i.status == "translated")
|
||||
ignored = sum(1 for i in self._items_by_id.values() if i.status == "ignored")
|
||||
failed = sum(1 for i in self._items_by_id.values() if i.status == "failed")
|
||||
|
||||
return {
|
||||
"total": total,
|
||||
"translated": translated,
|
||||
"ignored": ignored,
|
||||
"failed": failed,
|
||||
"pending": total - translated - ignored - failed,
|
||||
"progress_percent": round((translated + ignored) / total * 100, 1)
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
"""
|
||||
Quality Manager Module
|
||||
|
||||
Responsible for evaluating translation quality and deciding on re-translation.
|
||||
"""
|
||||
|
||||
import json
|
||||
import random
|
||||
from typing import List, Dict, Any, Tuple
|
||||
from loguru import logger
|
||||
from .manifest_manager import ManifestItem
|
||||
from .llm_client import OpenRouterClient
|
||||
|
||||
class QualityManager:
|
||||
def __init__(self, config: Dict, llm_client: OpenRouterClient):
|
||||
self.config = config
|
||||
self.llm_client = llm_client
|
||||
self.qc_config = config['translation'].get('quality_control', {})
|
||||
self.pass_score = self.qc_config.get('pass_score', 7)
|
||||
self.sample_size = self.qc_config.get('sample_size', 2)
|
||||
|
||||
async def evaluate_chunk(self, chunk: List[ManifestItem]) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Evaluate a chunk of translations.
|
||||
|
||||
Returns:
|
||||
(passed: bool, average_score: int, reason: str)
|
||||
"""
|
||||
if not self.qc_config.get('enabled', False):
|
||||
return True, 10, "QC Disabled"
|
||||
|
||||
# 1. Sample items
|
||||
# Filter for items that actually have content and translations
|
||||
valid_items = [item for item in chunk if item.translation and len(item.clean_text) > 20]
|
||||
|
||||
if not valid_items:
|
||||
return True, 10, "No valid items to sample"
|
||||
|
||||
sample_items = random.sample(valid_items, min(len(valid_items), self.sample_size))
|
||||
|
||||
# 2. Build Prompt
|
||||
prompt = self._build_evaluation_prompt(sample_items)
|
||||
|
||||
# 3. Call LLM (Smart)
|
||||
try:
|
||||
response = await self.llm_client.raw_chat_completion(
|
||||
system_prompt="You are a professional translation editor.",
|
||||
user_prompt=prompt,
|
||||
model_type="smart"
|
||||
)
|
||||
|
||||
# 4. Parse JSON
|
||||
# Clean potential markdown
|
||||
json_str = response.strip()
|
||||
if "```json" in json_str:
|
||||
json_str = json_str.split("```json")[1].split("```")[0].strip()
|
||||
elif "```" in json_str:
|
||||
json_str = json_str.split("```")[1].split("```")[0].strip()
|
||||
|
||||
result = json.loads(json_str)
|
||||
score = result.get('score', 0)
|
||||
reason = result.get('reason', 'No reason provided')
|
||||
|
||||
passed = score >= self.pass_score
|
||||
return passed, score, reason
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"QC evaluation failed: {e}")
|
||||
# If QC fails, we default to PASS to avoid blocking progress, but log it
|
||||
return True, 0, f"QC Error: {e}"
|
||||
|
||||
def _build_evaluation_prompt(self, items: List[ManifestItem]) -> str:
|
||||
content = ""
|
||||
for i, item in enumerate(items, 1):
|
||||
content += f"Item {i}:\nOriginal: {item.clean_text}\nTranslation: {item.translation}\n\n"
|
||||
|
||||
return f"""Please evaluate the following translations (English to Chinese).
|
||||
Focus on accuracy, fluency, and terminology consistency.
|
||||
|
||||
Items to evaluate:
|
||||
{content}
|
||||
|
||||
Return a JSON object with:
|
||||
- \"score\": An integer from 1 to 10 (10 being perfect).
|
||||
- \"reason\": A brief explanation of the score.
|
||||
|
||||
JSON Output:"""
|
||||
@@ -0,0 +1,161 @@
|
||||
"""
|
||||
文本处理器模块 (Text Processor Module) - Manifest 驱动版
|
||||
|
||||
该模块专注于 HTML 文档的遍历和段落提取。
|
||||
它不再维护全局状态，而是将提取的内容注册到 ManifestManager 中。
|
||||
"""
|
||||
|
||||
import re
|
||||
from bs4 import BeautifulSoup
|
||||
from typing import List, Dict, Any
|
||||
from loguru import logger
|
||||
from .manifest_manager import ManifestManager
|
||||
|
||||
|
||||
class TextProcessor:
|
||||
"""
|
||||
负责从 HTML 中识别有效段落并进行清洗。
|
||||
"""
|
||||
|
||||
def __init__(self, config: Dict):
|
||||
"""
|
||||
Args:
|
||||
config (Dict): 全局配置。
|
||||
"""
|
||||
self.config = config
|
||||
self.chunk_size = config['translation'].get('chunk_size', 5000)
|
||||
|
||||
def extract_to_manifest(self, html_content: str, source_file: str, manifest: ManifestManager):
|
||||
"""
|
||||
解析 HTML 内容，并将识别出的段落注册到 Manifest 中。
|
||||
|
||||
Args:
|
||||
html_content (str): HTML 源码。
|
||||
source_file (str): 来源文件名。
|
||||
manifest (ManifestManager): 清单管理器实例。
|
||||
"""
|
||||
try:
|
||||
soup = BeautifulSoup(html_content, 'html.parser')
|
||||
|
||||
# 1. 移除不需要的元素
|
||||
for element in soup(['script', 'style', 'meta', 'link']):
|
||||
element.decompose()
|
||||
|
||||
# 2. 获取有效的文本元素 (使用静态过滤逻辑)
|
||||
text_elements = self.get_valid_text_elements(soup)
|
||||
|
||||
# 3. 注册到 Manifest
|
||||
for element in text_elements:
|
||||
clean_text = self.clean_element_text(element)
|
||||
|
||||
# 过滤逻辑
|
||||
if not clean_text:
|
||||
continue
|
||||
|
||||
status = "pending"
|
||||
# 如果是导航元素，标记为 ignored
|
||||
if self.is_navigation_element(element):
|
||||
status = "ignored"
|
||||
|
||||
# 注册
|
||||
manifest.add_item(
|
||||
source_file=source_file,
|
||||
original_html=str(element),
|
||||
clean_text=clean_text,
|
||||
tag=element.name,
|
||||
metadata={"status": status} # 临时传递给 manifest
|
||||
)
|
||||
|
||||
# 同步更新 manifest 状态 (如果需要过滤)
|
||||
if status == "ignored":
|
||||
last_id = f"p_{len(manifest._items_by_id):05d}"
|
||||
manifest.update_item(last_id, translation=None, status="ignored")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"从 {source_file} 提取段落失败: {e}")
|
||||
|
||||
@staticmethod
|
||||
def get_valid_text_elements(soup) -> List:
|
||||
"""获取不含嵌套子块的叶子级文本容器元素。"""
|
||||
tags = ['p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'blockquote', 'li', 'td']
|
||||
all_candidates = soup.find_all(tags)
|
||||
candidate_set = set(all_candidates)
|
||||
|
||||
final_elements = []
|
||||
for element in all_candidates:
|
||||
# 如果包含其他候选标签，说明是容器，跳过
|
||||
if any(d in candidate_set for d in element.find_all(tags)):
|
||||
continue
|
||||
final_elements.append(element)
|
||||
return final_elements
|
||||
|
||||
@staticmethod
|
||||
def clean_element_text(element) -> str:
|
||||
"""清理 HTML 元素，提取纯净的待翻译文本。"""
|
||||
element_copy = element.__copy__()
|
||||
|
||||
# 移除脚注引用等
|
||||
for tag in element_copy.find_all(['sup', 'sub']):
|
||||
tag.decompose()
|
||||
|
||||
footnote_patterns = re.compile(r'footnote|endnote|reference|note|super|sub', re.I)
|
||||
for tag in element_copy.find_all(['a', 'span', 'div'], class_=footnote_patterns):
|
||||
tag.decompose()
|
||||
|
||||
# 移除仅包含数字的 span
|
||||
for tag in element_copy.find_all('span'):
|
||||
if re.match(r'^(\[\d+\]|\(\d+\)|\d+)$', tag.get_text().strip()):
|
||||
tag.decompose()
|
||||
|
||||
text = element_copy.get_text().strip()
|
||||
# 正则清理残留引用标识 (如 sentence.2)
|
||||
text = re.sub(r'(\.|。|，|,)\s*(\[\d+\]|\d+)(?=\s|$)', r'\1', text)
|
||||
text = re.sub(r'\s+', ' ', text)
|
||||
return text
|
||||
|
||||
@staticmethod
|
||||
def is_navigation_element(element) -> bool:
|
||||
"""判断是否是无翻译价值的导航、页码元素。"""
|
||||
classes = element.get('class', [])
|
||||
nav_classes = ['nav', 'navigation', 'toc', 'menu', 'header', 'footer', 'page-number']
|
||||
class_str = ' '.join(classes).lower() if isinstance(classes, list) else str(classes).lower()
|
||||
|
||||
if any(nc in class_str for nc in nav_classes):
|
||||
return True
|
||||
|
||||
# 检查父级
|
||||
parent = element.parent
|
||||
if parent:
|
||||
p_classes = parent.get('class', [])
|
||||
p_class_str = ' '.join(p_classes).lower() if isinstance(p_classes, list) else str(p_classes).lower()
|
||||
if any(nc in p_class_str for nc in nav_classes):
|
||||
return True
|
||||
return False
|
||||
|
||||
def create_chunks_from_manifest(self, manifest: ManifestManager) -> List[List[Any]]:
|
||||
"""
|
||||
从 Manifest 中筛选待翻译项目并分块。
|
||||
"""
|
||||
pending_items = manifest.get_items(status="pending")
|
||||
if not pending_items:
|
||||
return []
|
||||
|
||||
chunks = []
|
||||
current_chunk = []
|
||||
current_size = 0
|
||||
|
||||
for item in pending_items:
|
||||
text_len = len(item.clean_text)
|
||||
if current_size + text_len > self.chunk_size and current_chunk:
|
||||
chunks.append(current_chunk)
|
||||
current_chunk = []
|
||||
current_size = 0
|
||||
|
||||
current_chunk.append(item)
|
||||
current_size += text_len
|
||||
|
||||
if current_chunk:
|
||||
chunks.append(current_chunk)
|
||||
|
||||
logger.info(f"分块完成: 共有 {len(pending_items)} 个待翻译项，分为 {len(chunks)} 个块")
|
||||
return chunks
|
||||
@@ -0,0 +1,152 @@
|
||||
"""
|
||||
EPUB Translator Core Module - v0.05 (Manifest & Arena)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import traceback
|
||||
from typing import List, Dict, Any
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
from rich.console import Console
|
||||
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TimeElapsedColumn
|
||||
|
||||
from .epub_parser import EPUBParser
|
||||
from .llm_client import OpenRouterClient
|
||||
from .text_processor import TextProcessor
|
||||
from .bilingual_builder import BilingualEPUBBuilder
|
||||
from .manifest_manager import ManifestManager
|
||||
from .book_profiler import BookProfiler
|
||||
|
||||
|
||||
class EPUBTranslator:
|
||||
|
||||
def __init__(self, config: Dict, use_cache: bool = True):
|
||||
self.config = config
|
||||
self.console = Console()
|
||||
self.use_cache = use_cache
|
||||
|
||||
self.parser = None
|
||||
self.llm_client = OpenRouterClient(config)
|
||||
self.text_processor = TextProcessor(config)
|
||||
self.profiler = BookProfiler(config, self.llm_client)
|
||||
|
||||
self.manifest_dir = Path("cache/manifests")
|
||||
self.manifest_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
async def translate_epub(self, epub_path: str, test_mode: bool = False, output_dir: str = None) -> str:
|
||||
try:
|
||||
epub_path = Path(epub_path)
|
||||
self.parser = EPUBParser(str(epub_path))
|
||||
|
||||
# 1. Prepare Manifest
|
||||
manifest_path = self.manifest_dir / f"{epub_path.stem}_manifest.json"
|
||||
manifest = ManifestManager(str(manifest_path))
|
||||
|
||||
if not manifest.load() or not self.use_cache:
|
||||
self.console.print("[yellow]Initializing Manifest...[/yellow]")
|
||||
manifest.init_manifest(book_id=epub_path.name, metadata=self.parser.get_book_info())
|
||||
content_items = self.parser.extract_all_content_items()
|
||||
for item in content_items:
|
||||
self.text_processor.extract_to_manifest(item['content'], item['file_name'], manifest)
|
||||
manifest.save()
|
||||
|
||||
stats = manifest.stats
|
||||
self.console.print(f"[green]Manifest loaded: {stats['total']} paragraphs[/green]")
|
||||
|
||||
# 2. Profile & Arena
|
||||
profile = {}
|
||||
best_model = self.config['llm']['models']['fast']
|
||||
|
||||
if not test_mode:
|
||||
# A. Profile
|
||||
self.console.print("[yellow]Generating Book Profile...[/yellow]")
|
||||
profile = await self.profiler.analyze_book(manifest)
|
||||
self.console.print(f"Genre: {profile.get('genre')} | Style: {profile.get('style')}")
|
||||
|
||||
# B. Arena
|
||||
self.console.print("[yellow]Running Model Arena...[/yellow]")
|
||||
best_model = await self.profiler.run_arena(manifest, profile)
|
||||
self.console.print(f"[bold green]🏆 Winner: {best_model}[/bold green]")
|
||||
|
||||
# 3. Translate
|
||||
if test_mode:
|
||||
pending = manifest.get_items(status="pending")[:5]
|
||||
if pending:
|
||||
results = await self.llm_client.translate_chunk(
|
||||
pending,
|
||||
glossary=profile.get('glossary'),
|
||||
instruction=profile.get('translation_instruction'),
|
||||
model_id_override=best_model
|
||||
)
|
||||
for pid, trans in results.items():
|
||||
self.console.print(f"\n[cyan]{pid}[/cyan]: {trans}")
|
||||
return "test_mode_done"
|
||||
|
||||
chunks = self.text_processor.create_chunks_from_manifest(manifest)
|
||||
if chunks:
|
||||
await self._translate_concurrently(chunks, manifest, profile, best_model)
|
||||
|
||||
# 4. Build
|
||||
self.console.print("\n[yellow]Building bilingual EPUB...[/yellow]")
|
||||
output_path = output_dir or self.config['output']['output_dir']
|
||||
builder = BilingualEPUBBuilder(self.parser.book, self.config)
|
||||
|
||||
translation_map = {item.global_id: item.translation for item in manifest.get_items() if item.translation}
|
||||
paragraph_map = {item.global_id: {
|
||||
"file_name": item.source_file,
|
||||
"text": item.clean_text,
|
||||
"html_element": item.original_html
|
||||
} for item in manifest.get_items()}
|
||||
|
||||
result_file = builder.create_bilingual_epub_with_mapping(
|
||||
translation_map, paragraph_map, output_path
|
||||
)
|
||||
|
||||
self.console.print(f"[green]✅ Translation complete! File: {result_file}[/green]")
|
||||
return result_file
|
||||
except Exception as e:
|
||||
traceback.print_exc()
|
||||
logger.error(f"Translation flow failed: {e}")
|
||||
raise
|
||||
|
||||
async def _translate_concurrently(self, chunks: List[List[Any]], manifest: ManifestManager, profile: Dict, model_id: str):
|
||||
total_chunks = len(chunks)
|
||||
glossary = profile.get('glossary', {})
|
||||
instruction = profile.get('translation_instruction', "")
|
||||
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
BarColumn(),
|
||||
TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
|
||||
TimeElapsedColumn(),
|
||||
console=self.console
|
||||
) as progress:
|
||||
task_id = progress.add_task(f"[cyan]Translating ({model_id.split('/')[-1]})...", total=total_chunks)
|
||||
semaphore = self.llm_client.rate_limiter.semaphore
|
||||
|
||||
async def worker(chunk, idx):
|
||||
async with semaphore:
|
||||
try:
|
||||
results = await self.llm_client.translate_chunk(
|
||||
chunk,
|
||||
glossary=glossary,
|
||||
instruction=instruction,
|
||||
model_id_override=model_id
|
||||
)
|
||||
for item in chunk:
|
||||
if item.global_id in results:
|
||||
manifest.update_item(item.global_id, results[item.global_id], model=model_id)
|
||||
else:
|
||||
manifest.update_item(item.global_id, None, status="failed", error="Missing")
|
||||
manifest.save()
|
||||
except Exception as e:
|
||||
logger.error(f"Chunk {idx} failed: {e}")
|
||||
finally:
|
||||
progress.update(task_id, advance=1)
|
||||
|
||||
tasks = [worker(chunk, i) for i, chunk in enumerate(chunks)]
|
||||
await asyncio.gather(*tasks)
|
||||
@@ -0,0 +1,180 @@
|
||||
"""
|
||||
工具函数模块
|
||||
提供配置加载、日志设置等通用功能
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any
|
||||
from loguru import logger
|
||||
import sys
|
||||
|
||||
|
||||
def load_config(config_path: str = "config/config.json") -> Dict[str, Any]:
|
||||
"""
|
||||
加载配置文件
|
||||
|
||||
Args:
|
||||
config_path: 配置文件路径
|
||||
|
||||
Returns:
|
||||
配置字典
|
||||
"""
|
||||
try:
|
||||
with open(config_path, 'r', encoding='utf-8') as f:
|
||||
config = json.load(f)
|
||||
|
||||
# 从环境变量获取 API Key
|
||||
if 'OPENROUTER_API_KEY' in os.environ:
|
||||
config['openrouter']['api_key'] = os.environ['OPENROUTER_API_KEY']
|
||||
|
||||
return config
|
||||
except FileNotFoundError:
|
||||
raise FileNotFoundError(f"配置文件未找到: {config_path}")
|
||||
except json.JSONDecodeError as e:
|
||||
raise ValueError(f"配置文件格式错误: {e}")
|
||||
|
||||
|
||||
def load_prompts(prompts_path: str = "config/prompts.json") -> Dict[str, str]:
|
||||
"""
|
||||
加载提示词模板
|
||||
|
||||
Args:
|
||||
prompts_path: 提示词文件路径
|
||||
|
||||
Returns:
|
||||
提示词字典
|
||||
"""
|
||||
try:
|
||||
with open(prompts_path, 'r', encoding='utf-8') as f:
|
||||
return json.load(f)
|
||||
except FileNotFoundError:
|
||||
raise FileNotFoundError(f"提示词文件未找到: {prompts_path}")
|
||||
|
||||
|
||||
def setup_logging(config: Dict[str, Any]) -> None:
|
||||
"""
|
||||
设置日志配置
|
||||
|
||||
Args:
|
||||
config: 配置字典
|
||||
"""
|
||||
log_config = config.get('logging', {})
|
||||
|
||||
# 移除默认处理器
|
||||
logger.remove()
|
||||
|
||||
# 添加控制台输出
|
||||
logger.add(
|
||||
sys.stdout,
|
||||
level=log_config.get('level', 'INFO'),
|
||||
format="<green>{time:YYYY-MM-DD HH:mm:ss}</green> | <level>{level: <8}</level> | <cyan>{name}</cyan>:<cyan>{function}</cyan>:<cyan>{line}</cyan> - <level>{message}</level>"
|
||||
)
|
||||
|
||||
# 添加文件输出
|
||||
if 'file' in log_config:
|
||||
log_file = log_config['file']
|
||||
# 确保日志目录存在
|
||||
Path(log_file).parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.add(
|
||||
log_file,
|
||||
level=log_config.get('level', 'INFO'),
|
||||
rotation=log_config.get('rotation', '10 MB'),
|
||||
retention=log_config.get('retention', '7 days'),
|
||||
encoding='utf-8',
|
||||
format="{time:YYYY-MM-DD HH:mm:ss} | {level: <8} | {name}:{function}:{line} - {message}"
|
||||
)
|
||||
|
||||
|
||||
def ensure_output_dir(output_dir: str) -> Path:
|
||||
"""
|
||||
确保输出目录存在
|
||||
|
||||
Args:
|
||||
output_dir: 输出目录路径
|
||||
|
||||
Returns:
|
||||
输出目录的 Path 对象
|
||||
"""
|
||||
output_path = Path(output_dir)
|
||||
output_path.mkdir(parents=True, exist_ok=True)
|
||||
return output_path
|
||||
|
||||
|
||||
def sanitize_filename(filename: str) -> str:
|
||||
"""
|
||||
清理文件名，移除非法字符
|
||||
|
||||
Args:
|
||||
filename: 原始文件名
|
||||
|
||||
Returns:
|
||||
清理后的文件名
|
||||
"""
|
||||
import re
|
||||
# 移除或替换非法字符
|
||||
filename = re.sub(r'[<>:"/\\|?*]', '_', filename)
|
||||
# 移除多余的空格和点
|
||||
filename = re.sub(r'\s+', ' ', filename).strip('. ')
|
||||
return filename
|
||||
|
||||
|
||||
def format_file_size(size_bytes: int) -> str:
|
||||
"""
|
||||
格式化文件大小显示
|
||||
|
||||
Args:
|
||||
size_bytes: 字节数
|
||||
|
||||
Returns:
|
||||
格式化的大小字符串
|
||||
"""
|
||||
if size_bytes == 0:
|
||||
return "0B"
|
||||
|
||||
size_names = ["B", "KB", "MB", "GB"]
|
||||
import math
|
||||
i = int(math.floor(math.log(size_bytes, 1024)))
|
||||
p = math.pow(1024, i)
|
||||
s = round(size_bytes / p, 2)
|
||||
return f"{s} {size_names[i]}"
|
||||
|
||||
|
||||
def estimate_tokens(text: str) -> int:
|
||||
"""
|
||||
估算文本的 token 数量
|
||||
|
||||
Args:
|
||||
text: 输入文本
|
||||
|
||||
Returns:
|
||||
估算的 token 数量
|
||||
"""
|
||||
# 简单估算：英文约 4 字符/token，中文约 1.5 字符/token
|
||||
import re
|
||||
|
||||
# 分离中英文
|
||||
chinese_chars = len(re.findall(r'[\u4e00-\u9fff]', text))
|
||||
other_chars = len(text) - chinese_chars
|
||||
|
||||
# 估算 tokens
|
||||
estimated_tokens = chinese_chars / 1.5 + other_chars / 4
|
||||
return int(estimated_tokens)
|
||||
|
||||
|
||||
def truncate_text(text: str, max_length: int = 100) -> str:
|
||||
"""
|
||||
截断文本用于显示
|
||||
|
||||
Args:
|
||||
text: 原始文本
|
||||
max_length: 最大长度
|
||||
|
||||
Returns:
|
||||
截断后的文本
|
||||
"""
|
||||
if len(text) <= max_length:
|
||||
return text
|
||||
return text[:max_length-3] + "..."
|
||||
@@ -0,0 +1,62 @@
|
||||
# 更新日志 (CHANGELOG)
|
||||
|
||||
## [v0.07] - 2026-01-13 (The Refinement)
|
||||
|
||||
### 🛡️ 安全与配置
|
||||
- **环境隔离**: 引入 `.env` 支持，彻底移除了代码库中的硬编码 API Key。
|
||||
- **配置升级**: `utils.py` 现自动加载 `.env` 并注入到配置中，支持任意 Provider 的环境变量覆盖 (如 `V3_API_KEY`, `OPENROUTER_API_KEY`)。
|
||||
- **模板化**: 新增 `config.example.json` 和 `.env` 模板，提升部署安全性。
|
||||
|
||||
### 🚀 核心改进
|
||||
- **V3 Provider 支持**: 验证并修复了对 V3 API (OpenAI 兼容格式) 的支持，全流程跑通。
|
||||
- **EPUB 构建修复**: 解决了 `ebooklib` 在处理 TOC 时因缺少 UID 导致的 `Argument must be bytes or unicode` 崩溃问题。
|
||||
- **缓存优化**:
|
||||
- 缓存目录结构调整为 Hash 前缀 (`cache/translations/ab/...`)，解决了按日期分目录导致的缓存频繁失效问题。
|
||||
- 放宽了缓存验证逻辑，支持部分命中的缓存复用。
|
||||
- **视觉优化**: 引入“盘古之白” (Pangu spacing)，自动在中文与英文/数字之间添加空格，显著提升阅读体验。
|
||||
|
||||
### ⚡ 体验提升
|
||||
- **断点续传提示**: 启动时自动检测并提示未完成的翻译进度。
|
||||
- **详细统计**: 翻译完成后展示详细的成功/失败/跳过统计数据。
|
||||
- **并发优化**: 移除了冗余的信号量控制，完全依赖 `RateLimiter`，逻辑更清晰高效。
|
||||
|
||||
---
|
||||
|
||||
## [v0.05] - 2026-01-12 (The Arena)
|
||||
|
||||
### 🌟 核心突破
|
||||
- **书籍画像 (Book Profiler)**:
|
||||
- 自动提取前言和正文采样。
|
||||
- 生成 `Book Profile`，包含领域 (Genre)、文风 (Style)、目标受众 (Audience) 和翻译指令。
|
||||
- 生成 `Glossary` (术语表)，并支持自动注入 Prompt。
|
||||
- **状态绑定**: Profile 和 Glossary 现在直接存储在每本书的 `manifest.json` 中，互不干扰。
|
||||
- **模型竞技场 (Model Arena)**:
|
||||
- 自动选取典型 Chunk，让多个候选模型 (Gemini, Llama, Qwen) 同台竞技。
|
||||
- 引入 `Judge Agent` (基于 Smart 模型)，从准确性、信达雅维度评选最佳模型。
|
||||
- 自动锁定获胜模型用于全书翻译。
|
||||
|
||||
### 🏗️ 架构升级
|
||||
- **LLM Client 重构**:
|
||||
- **Syntax Fixes**: 彻底修复了正则构造中的语法错误。
|
||||
- **Quote Safety**: 移除了所有 f-string 中的复杂正则，改用安全的字符串拼接。
|
||||
- **Dual RateLimiters**: 引入主/副限流器，防止死锁。
|
||||
- **配置增强**:
|
||||
- `config.json` 支持 `arena_models` 和 `judge_model` 配置。
|
||||
|
||||
### 🔧 修复与优化
|
||||
- 修复了 `unhashable type: 'dict'` 错误 (移除了错误的 `{{}}`)。
|
||||
- 修复了多本书连续翻译时 Profile 串用的问题 (Profile 现已绑定至 Manifest)。
|
||||
|
||||
---
|
||||
|
||||
## [v0.03] - 2026-01-12
|
||||
- **极简 ID 锚点系统**: 废弃复杂的 `[p_xxxxx]` 格式，使用纯净 ID，彻底解决残留问题。
|
||||
- **智能术语一致性**: 引入 GlossaryManager。
|
||||
- **结构完美保留**: 修复了 EPUB Spine 和 Metadata 丢失问题。
|
||||
|
||||
## [v0.02] - 2026-01-12
|
||||
- **Manifest 驱动架构**: 引入 `ManifestManager` 作为单一真理源。
|
||||
- **流程解耦**: 提取、翻译、构建三阶段分离。
|
||||
|
||||
## [v0.01] - 2026-01-10
|
||||
- 初始版本，实现基本的并发翻译和 EPUB 解析。
|
||||
@@ -0,0 +1,97 @@
|
||||
# EPUB 双语翻译程序 v0.07
|
||||
|
||||
一个基于 OpenRouter/OpenAI API 的 EPUB 双语翻译工具，采用**全局编号系统**和**真并发翻译**。
|
||||
|
||||
## ✨ 核心特性
|
||||
|
||||
### 🎯 全局编号系统
|
||||
- **每个段落分配全局唯一ID**（格式：`p_0001`, `p_0002`...）
|
||||
- **ID贯穿全流程**：提取 → 翻译 → 组装
|
||||
- **精确对应保证**：绝不出现中英文错行问题
|
||||
|
||||
### ⚡ 真并发翻译
|
||||
- **asyncio.gather 并发执行**：高效利用 API 速率限制
|
||||
- **智能速率控制**：基于 Token 桶的 RateLimiter
|
||||
- **实时进度显示**：Rich 进度条显示翻译状态
|
||||
- **断点续传**：自动记录进度，随时中断随时继续
|
||||
|
||||
### 🛡️ 安全与稳定
|
||||
- **环境隔离**：支持 `.env` 配置，API Key 不落地
|
||||
- **鲁棒重试**：集成 `tenacity` 处理网络波动
|
||||
- **缓存系统**：基于 Hash 的持久化缓存，跨天复用
|
||||
|
||||
### 🎨 极致排版
|
||||
- **盘古之白**：自动在中文与西文数字间添加空格
|
||||
- **样式注入**：注入专用 CSS 优化阅读体验
|
||||
|
||||
## 🚀 快速开始
|
||||
|
||||
### 1. 安装依赖
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 2. 配置环境
|
||||
|
||||
复制 `.env` 模板并填入你的 API Key：
|
||||
|
||||
```bash
|
||||
# .env 文件
|
||||
V3_API_KEY=sk-xxxxxx
|
||||
OPENROUTER_API_KEY=sk-or-xxxxxx
|
||||
```
|
||||
|
||||
### 3. 开始翻译
|
||||
|
||||
```bash
|
||||
# 默认使用 OpenRouter
|
||||
python main.py input/book.epub
|
||||
|
||||
# 使用 V3 Provider
|
||||
python main.py input/book.epub -p v3
|
||||
|
||||
# 测试模式（只翻译前3个块）
|
||||
python main.py input/book.epub --test
|
||||
```
|
||||
|
||||
## 📂 目录结构
|
||||
|
||||
```
|
||||
.
|
||||
├── config/ # 配置文件
|
||||
│ ├── config.json # 主配置
|
||||
│ └── prompts.json # 提示词模板
|
||||
├── input/ # 输入 EPUB 目录
|
||||
├── output/ # 输出 EPUB 目录
|
||||
├── cache/ # 缓存目录 (Manifest, Translations)
|
||||
├── logs/ # 运行日志
|
||||
└── src/ # 源代码
|
||||
```
|
||||
|
||||
## ⚙️ 核心配置 (config.json)
|
||||
|
||||
```json
|
||||
{
|
||||
"translation": {
|
||||
"chunk_size": 5000,
|
||||
"temperature": 0.3
|
||||
},
|
||||
"providers": {
|
||||
"v3": {
|
||||
"base_url": "https://api.gpt.ge/v1",
|
||||
"models": { "fast": "gpt-4o-mini" },
|
||||
"rate_limits": { "requests_per_minute": 500 }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 📄 许可证
|
||||
|
||||
MIT License
|
||||
|
||||
---
|
||||
|
||||
**版本**: v0.07
|
||||
**更新**: 2026-01-13
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"translation": {
|
||||
"chunk_size": 5000,
|
||||
"temperature": 0.3,
|
||||
"glossary": {
|
||||
"enabled": true,
|
||||
"auto_generate": true,
|
||||
"sample_size": 3000
|
||||
}
|
||||
},
|
||||
"output": {
|
||||
"output_dir": "output",
|
||||
"filename_suffix": "_bilingual"
|
||||
},
|
||||
"logging": {
|
||||
"level": "INFO",
|
||||
"file": "logs/translator.log",
|
||||
"rotation": "10 MB",
|
||||
"retention": "7 days"
|
||||
},
|
||||
"providers": {
|
||||
"openrouter": {
|
||||
"base_url": "https://openrouter.ai/api/v1",
|
||||
"api_key": "YOUR_OPENROUTER_API_KEY",
|
||||
"models": {
|
||||
"fast": "google/gemini-2.0-flash-001",
|
||||
"smart": "google/gemini-2.0-flash-thinking-exp:free"
|
||||
},
|
||||
"extra_headers": {
|
||||
"HTTP-Referer": "https://github.com/epub-translator",
|
||||
"X-Title": "EPUB Translator"
|
||||
},
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 60,
|
||||
"concurrent_requests": 32
|
||||
}
|
||||
},
|
||||
"v3": {
|
||||
"base_url": "https://api.gpt.ge/v1",
|
||||
"api_key": "YOUR_V3_API_KEY",
|
||||
"models": {
|
||||
"fast": "gpt-4o-mini",
|
||||
"smart": "gpt-4o"
|
||||
},
|
||||
"extra_headers": {
|
||||
"x-foo": "true"
|
||||
},
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 500,
|
||||
"concurrent_requests": 50
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"translation": {
|
||||
"system": "你是一位精通中英文的专业翻译家。你的任务是翻译书籍内容。\n\n要求：\n1. 准确传达原文含义，语言流畅自然，符合中文阅读习惯。\n2. 严格保持【p_xxxxx】编号格式，不要遗漏，不要修改编号。\n3. 不要添加任何解释、注释或无关内容，只返回【编号】+【译文】。\n\n{{glossary_instruction}}",
|
||||
"user_template": "请翻译以下段落：\n\n{{content}}"
|
||||
},
|
||||
"glossary_extraction": {
|
||||
"system": "你是一位资深的文学编辑和领域专家。你的任务是分析书籍样本，提取关键术语并制定统一的译名表。",
|
||||
"user_template": "请阅读以下书籍片段（包含前言和正文采样）。\n\n任务：\n1. 识别文中出现的人名（如 'Masa', 'Steve Jobs'）、地名、机构名。\n2. 识别特定的行业术语或关键概念。\n3. 为上述词汇提供标准的中文译名。如果像 'Masa' 这样的昵称有对应的全名（如孙正义），请务必使用全名。\n\n请以 JSON 格式输出，格式如下：\n{\n \"Masa\": \"孙正义\",\n \"Apple\": \"苹果公司\",\n ...\n}\n\n书籍片段：\n\n{{content}}"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env python3
|
||||
import asyncio
|
||||
import sys
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
from src.translator import EPUBTranslator
|
||||
from src.utils import load_config, setup_logging
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="EPUB 双语翻译工具")
|
||||
parser.add_argument("epub_path", help="输入 EPUB 文件路径")
|
||||
parser.add_argument("--provider", "-p", default="openrouter", help="LLM 供应商 (config.json 中 providers 的 key)")
|
||||
parser.add_argument("--test", action="store_true", help="测试模式（仅翻译前几段）")
|
||||
parser.add_argument("--output", "-o", help="输出目录")
|
||||
parser.add_argument("--no-cache", action="store_true", help="禁用缓存（强制重新翻译）")
|
||||
parser.add_argument("--clear-cache", action="store_true", help="清理所有缓存文件")
|
||||
return parser.parse_args()
|
||||
|
||||
def flatten_provider_config(config: dict, provider_name: str) -> dict:
|
||||
"""
|
||||
将选定的 provider 配置扁平化到 config['llm'] 中，
|
||||
以便下游模块统一调用。
|
||||
"""
|
||||
providers = config.get('providers', {})
|
||||
if provider_name not in providers:
|
||||
available = list(providers.keys())
|
||||
logger.error(f"未找到供应商 '{provider_name}'。可用供应商: {available}")
|
||||
sys.exit(1)
|
||||
|
||||
selected_config = providers[provider_name]
|
||||
logger.info(f"使用 LLM 供应商: {provider_name} ({selected_config.get('base_url')})")
|
||||
|
||||
# 注入到 config['llm']
|
||||
config['llm'] = selected_config
|
||||
return config
|
||||
|
||||
async def run_translation(args):
|
||||
try:
|
||||
# 1. 加载配置
|
||||
config = load_config()
|
||||
|
||||
# 2. 处理 Provider 选择
|
||||
config = flatten_provider_config(config, args.provider)
|
||||
|
||||
# 3. 设置日志
|
||||
setup_logging(config)
|
||||
logger.info("程序启动")
|
||||
|
||||
# 4. 初始化翻译器
|
||||
translator = EPUBTranslator(config, use_cache=not args.no_cache)
|
||||
|
||||
# 5. 执行翻译
|
||||
await translator.translate_epub(
|
||||
args.epub_path,
|
||||
test_mode=args.test,
|
||||
output_dir=args.output
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
logger.error(f"翻译失败: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
if args.clear_cache:
|
||||
import shutil
|
||||
cache_dir = Path("cache")
|
||||
if cache_dir.exists():
|
||||
shutil.rmtree(cache_dir)
|
||||
print("缓存已清理")
|
||||
sys.exit(0)
|
||||
|
||||
asyncio.run(run_translation(args))
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,34 @@
|
||||
[project]
|
||||
name = "epub-translator"
|
||||
version = "0.07"
|
||||
description = "EPUB双语翻译程序"
|
||||
authors = [
|
||||
{name = "Kaitan", email = "your-email@example.com"}
|
||||
]
|
||||
dependencies = [
|
||||
"ebooklib>=0.19",
|
||||
"beautifulsoup4>=4.12.0",
|
||||
"lxml>=4.9.0",
|
||||
"openai>=1.0.0",
|
||||
"aiohttp>=3.9.0",
|
||||
"pydantic>=2.0.0",
|
||||
"loguru>=0.7.0",
|
||||
"rich>=13.0.0",
|
||||
"asyncio-throttle>=1.0.2",
|
||||
"tenacity>=8.0.0",
|
||||
"python-dotenv>=1.0.0",
|
||||
]
|
||||
requires-python = ">=3.9"
|
||||
|
||||
[project.scripts]
|
||||
epub-translator = "main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.uv]
|
||||
dev-dependencies = [
|
||||
"pytest>=7.0.0",
|
||||
"pytest-asyncio>=0.21.0",
|
||||
]
|
||||
@@ -0,0 +1,11 @@
|
||||
ebooklib>=0.19
|
||||
beautifulsoup4>=4.12.0
|
||||
lxml>=4.9.0
|
||||
openai>=1.0.0
|
||||
aiohttp>=3.9.0
|
||||
pydantic>=2.0.0
|
||||
loguru>=0.7.0
|
||||
rich>=13.0.0
|
||||
asyncio-throttle>=1.0.2
|
||||
tenacity>=8.0.0
|
||||
python-dotenv>=1.0.0
|
||||
@@ -0,0 +1,26 @@
|
||||
"""
|
||||
EPUB 双语翻译程序
|
||||
主要功能模块的初始化文件
|
||||
"""
|
||||
|
||||
__version__ = "0.07"
|
||||
__author__ = "Kaitan"
|
||||
|
||||
from .epub_parser import EPUBParser
|
||||
from .translator import EPUBTranslator
|
||||
from .llm_client import LLMClient as OpenRouterClient # Keep alias for compatibility
|
||||
from .llm_client import LLMClient
|
||||
from .text_processor import TextProcessor
|
||||
from .bilingual_builder import BilingualEPUBBuilder
|
||||
from .utils import load_config, setup_logging
|
||||
|
||||
__all__ = [
|
||||
"EPUBParser",
|
||||
"EPUBTranslator",
|
||||
"LLMClient",
|
||||
"OpenRouterClient",
|
||||
"TextProcessor",
|
||||
"BilingualEPUBBuilder",
|
||||
"load_config",
|
||||
"setup_logging"
|
||||
]
|
||||
@@ -0,0 +1,180 @@
|
||||
"""
|
||||
双语 EPUB 构建器模块 - 安全的EPUB构建 (Manifest 兼容版)
|
||||
"""
|
||||
|
||||
from ebooklib import epub
|
||||
import ebooklib
|
||||
from bs4 import BeautifulSoup
|
||||
from typing import Dict, List
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
import uuid
|
||||
|
||||
|
||||
class BilingualEPUBBuilder:
|
||||
"""双语 EPUB 构建器"""
|
||||
|
||||
def __init__(self, original_book, config: Dict):
|
||||
self.original_book = original_book
|
||||
self.config = config
|
||||
self.output_config = config['output']
|
||||
|
||||
def create_bilingual_epub_with_mapping(self, translation_map: Dict[str, str],
|
||||
paragraph_map: Dict[str, Dict],
|
||||
output_path: str) -> str:
|
||||
"""
|
||||
创建双语 EPUB。使用 ordered_ids 确保与 Manifest 严格一致。
|
||||
"""
|
||||
try:
|
||||
new_book = epub.EpubBook()
|
||||
self._copy_metadata(new_book)
|
||||
new_book.toc = self._sanitize_toc(self.original_book.toc)
|
||||
|
||||
# 准备每个文件的有序ID列表
|
||||
file_ordered_ids = {}
|
||||
sorted_pids = sorted(paragraph_map.keys(), key=lambda x: int(x.split('_')[1]))
|
||||
for pid in sorted_pids:
|
||||
info = paragraph_map[pid]
|
||||
fname = info['file_name']
|
||||
if fname not in file_ordered_ids:
|
||||
file_ordered_ids[fname] = []
|
||||
file_ordered_ids[fname].append(pid)
|
||||
|
||||
processed_item_ids = set()
|
||||
item_map = {}
|
||||
|
||||
# 特殊处理：封面图片
|
||||
cover_id_meta = self.original_book.get_metadata('OPF', 'cover')
|
||||
if cover_id_meta:
|
||||
cover_item = self.original_book.get_item_with_id(cover_id_meta[0][0])
|
||||
if cover_item:
|
||||
new_book.add_item(cover_item)
|
||||
processed_item_ids.add(cover_item.id)
|
||||
item_map[cover_item.id] = cover_item
|
||||
|
||||
# 复制资源
|
||||
for item in self.original_book.get_items():
|
||||
if item.get_type() != ebooklib.ITEM_DOCUMENT:
|
||||
if item.id not in processed_item_ids:
|
||||
new_book.add_item(item)
|
||||
processed_item_ids.add(item.id)
|
||||
item_map[item.id] = item
|
||||
|
||||
# 重建 Spine
|
||||
new_spine = []
|
||||
for spine_id, linear in self.original_book.spine:
|
||||
item = self.original_book.get_item_with_id(spine_id)
|
||||
if not item: continue
|
||||
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT:
|
||||
file_name = item.get_name()
|
||||
if file_name in file_ordered_ids:
|
||||
new_item = self._create_bilingual_document(
|
||||
item, file_ordered_ids[file_name], translation_map
|
||||
)
|
||||
new_item.id = item.id
|
||||
else:
|
||||
new_item = item
|
||||
|
||||
if new_item.id not in processed_item_ids:
|
||||
new_book.add_item(new_item)
|
||||
processed_item_ids.add(new_item.id)
|
||||
new_spine.append(new_item)
|
||||
else:
|
||||
if item.id in item_map:
|
||||
new_spine.append(item_map[item.id])
|
||||
|
||||
new_book.spine = new_spine
|
||||
new_book.add_item(epub.EpubNcx())
|
||||
new_book.add_item(epub.EpubNav())
|
||||
|
||||
output_file = self._generate_output_filename(output_path)
|
||||
epub.write_epub(output_file, new_book, {})
|
||||
return output_file
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"创建双语 EPUB 失败: {e}", exc_info=True)
|
||||
raise
|
||||
|
||||
def _sanitize_toc(self, toc):
|
||||
"""确保 TOC 中的所有节点都有 ID，修复 ebooklib 读取后写入的兼容性问题"""
|
||||
for item in toc:
|
||||
if isinstance(item, (epub.Link, epub.Section)):
|
||||
if not getattr(item, 'uid', None):
|
||||
item.uid = f"navPoint-{uuid.uuid4().hex[:8]}"
|
||||
elif isinstance(item, tuple) and len(item) == 2:
|
||||
# 处理 (Section, [children]) 结构
|
||||
section, children = item
|
||||
if isinstance(section, (epub.Link, epub.Section)):
|
||||
if not getattr(section, 'uid', None):
|
||||
section.uid = f"navPoint-{uuid.uuid4().hex[:8]}"
|
||||
self._sanitize_toc(children)
|
||||
return toc
|
||||
|
||||
def _copy_metadata(self, new_book):
|
||||
try:
|
||||
for namespace, meta_dict in self.original_book.metadata.items():
|
||||
for name, values in meta_dict.items():
|
||||
for value, other in values:
|
||||
if name and hasattr(name, 'lower') and name.lower() == 'identifier': continue
|
||||
new_book.add_metadata(namespace, name, value, other)
|
||||
new_book.add_metadata('DC', 'language', 'zh-CN')
|
||||
new_book.set_identifier(f"bilingual-{uuid.uuid4().hex[:12]}")
|
||||
|
||||
cover_id_meta = self.original_book.get_metadata('OPF', 'cover')
|
||||
if cover_id_meta:
|
||||
cover_item = self.original_book.get_item_with_id(cover_id_meta[0][0])
|
||||
if cover_item:
|
||||
new_book.add_item(cover_item)
|
||||
new_book.set_cover(cover_item.get_name(), cover_item.get_content())
|
||||
except Exception as e:
|
||||
logger.error(f"元数据复制出错: {e}")
|
||||
|
||||
def _create_bilingual_document(self, original_item, ordered_ids: list, translation_map: dict):
|
||||
try:
|
||||
from .text_processor import TextProcessor
|
||||
soup = BeautifulSoup(original_item.get_content().decode('utf-8'), 'html.parser')
|
||||
self._add_style_link(soup)
|
||||
|
||||
# 使用与 TextProcessor 相同的过滤逻辑获取元素
|
||||
text_elements = TextProcessor.get_valid_text_elements(soup)
|
||||
|
||||
current_para_index = 0
|
||||
for element in text_elements:
|
||||
if TextProcessor.is_navigation_element(element): continue
|
||||
if not TextProcessor.clean_element_text(element): continue
|
||||
|
||||
if current_para_index < len(ordered_ids):
|
||||
target_id = ordered_ids[current_para_index]
|
||||
translation = translation_map.get(target_id)
|
||||
if translation:
|
||||
self._insert_translation(element, translation, soup)
|
||||
current_para_index += 1
|
||||
|
||||
new_item = epub.EpubHtml(title=original_item.title, file_name=original_item.get_name(), lang='zh-CN')
|
||||
new_item.set_content(str(soup).encode('utf-8'))
|
||||
return new_item
|
||||
except Exception as e:
|
||||
logger.error(f"创建双语文档失败 {original_item.get_name()}: {e}")
|
||||
return original_item
|
||||
|
||||
def _add_style_link(self, soup):
|
||||
head = soup.find('head')
|
||||
if head and not head.find('link', href='style/bilingual.css'):
|
||||
head.append(soup.new_tag('link', rel='stylesheet', type='text/css', href='style/bilingual.css'))
|
||||
|
||||
def _insert_translation(self, element, translation: str, soup):
|
||||
try:
|
||||
translation_p = soup.new_tag('p')
|
||||
translation_p.string = translation
|
||||
translation_p['class'] = ['translation-text', 'chinese']
|
||||
element.insert_after(translation_p)
|
||||
except Exception as e:
|
||||
logger.warning(f"插入翻译失败: {e}")
|
||||
|
||||
def _generate_output_filename(self, output_path: str) -> str:
|
||||
from .utils import sanitize_filename
|
||||
title = self.original_book.get_metadata('DC', 'title')
|
||||
clean_title = sanitize_filename(title[0][0]) if title else "bilingual_book"
|
||||
Path(output_path).mkdir(parents=True, exist_ok=True)
|
||||
return str(Path(output_path) / f"{clean_title}_bilingual.epub")
|
||||
@@ -0,0 +1,84 @@
|
||||
|
||||
'''Book Profiler Module
|
||||
|
||||
Features:
|
||||
1. Automatically extract book samples to generate Book Profile (Genre, Style, Glossary).
|
||||
'''
|
||||
|
||||
import json
|
||||
import random
|
||||
from pathlib import Path
|
||||
from typing import Dict, List
|
||||
from loguru import logger
|
||||
from .manifest_manager import ManifestManager
|
||||
from .llm_client import LLMClient
|
||||
|
||||
class BookProfiler:
|
||||
def __init__(self, config: Dict, llm_client: LLMClient):
|
||||
self.config = config
|
||||
self.llm_client = llm_client
|
||||
|
||||
def extract_sample_text(self, manifest: ManifestManager, char_limit: int = 3000) -> str:
|
||||
"""Extract sample text."""
|
||||
items = manifest.get_items()
|
||||
if not items: return ""
|
||||
|
||||
intro_text = []
|
||||
for item in items[:50]:
|
||||
if len(item.clean_text) > 50:
|
||||
intro_text.append(item.clean_text)
|
||||
|
||||
body_text = []
|
||||
body_items = [i for i in items[50:] if len(i.clean_text) > 80]
|
||||
if body_items:
|
||||
samples = random.sample(body_items, min(5, len(body_items)))
|
||||
body_text = [i.clean_text for i in samples]
|
||||
|
||||
full_text = "\n\n".join(intro_text[:5] + body_text)
|
||||
return full_text[:char_limit]
|
||||
|
||||
async def analyze_book(self, manifest: ManifestManager) -> Dict:
|
||||
"""Generate Book Profile."""
|
||||
existing_profile = manifest.data.get('metadata', {}).get('profile')
|
||||
if existing_profile:
|
||||
logger.info("Loaded existing Book Profile")
|
||||
return existing_profile
|
||||
|
||||
sample = self.extract_sample_text(manifest)
|
||||
if not sample: return {}
|
||||
|
||||
logger.info("Generating Book Profile...")
|
||||
|
||||
system_prompt = "You are a senior publishing editor. Analyze the text and output JSON."
|
||||
user_prompt = f"""
|
||||
Please analyze the following book excerpt.
|
||||
Output JSON format:
|
||||
{{
|
||||
"genre": "Genre",
|
||||
"style": "Style description",
|
||||
"audience": "Target Audience",
|
||||
"glossary": {{ "Term": "Chinese Translation" }},
|
||||
"translation_instruction": "Specific instruction for translator"
|
||||
}}
|
||||
|
||||
Excerpt:
|
||||
{sample}
|
||||
"""
|
||||
try:
|
||||
response = await self.llm_client.raw_chat_completion(system_prompt, user_prompt)
|
||||
json_str = response.strip()
|
||||
if "```json" in json_str:
|
||||
json_str = json_str.split("```json")[1].split("```")[0].strip()
|
||||
elif "```" in json_str:
|
||||
json_str = json_str.split("```")[1].split("```")[0].strip()
|
||||
|
||||
profile = json.loads(json_str)
|
||||
|
||||
if 'metadata' not in manifest.data:
|
||||
manifest.data['metadata'] = {}
|
||||
manifest.data['metadata']['profile'] = profile
|
||||
manifest.save()
|
||||
return profile
|
||||
except Exception as e:
|
||||
logger.error(f"Profile generation failed: {e}")
|
||||
return {}
|
||||
@@ -0,0 +1,226 @@
|
||||
"""
|
||||
翻译缓存管理模块 - 简化版
|
||||
基于全局ID和chunk的缓存系统
|
||||
"""
|
||||
|
||||
import json
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Dict, Optional, List
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class TranslationCache:
|
||||
"""翻译缓存管理器 - 简化版"""
|
||||
|
||||
def __init__(self, config: Dict):
|
||||
"""初始化缓存管理器"""
|
||||
self.config = config
|
||||
cache_config = config.get('cache', {})
|
||||
|
||||
self.enabled = cache_config.get('enabled', True)
|
||||
self.cache_dir = Path(cache_config.get('directory', 'cache'))
|
||||
self.max_age_days = cache_config.get('max_age_days', 30)
|
||||
|
||||
if self.enabled:
|
||||
self.cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
self.translations_dir = self.cache_dir / 'translations'
|
||||
self.translations_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.info(f"翻译缓存已启用: {self.cache_dir}")
|
||||
|
||||
def get_chunk_translation(self, chunk: List[Dict], model: str) -> Optional[Dict[str, str]]:
|
||||
"""
|
||||
获取chunk的缓存翻译
|
||||
|
||||
Args:
|
||||
chunk: 段落列表（带global_id）
|
||||
model: 模型名称
|
||||
|
||||
Returns:
|
||||
{global_id: translation} 映射，如果不存在返回 None
|
||||
"""
|
||||
if not self.enabled:
|
||||
return None
|
||||
|
||||
try:
|
||||
cache_key = self._get_chunk_cache_key(chunk, model)
|
||||
cache_file = self._get_cache_file_path(cache_key)
|
||||
|
||||
if not cache_file.exists():
|
||||
return None
|
||||
|
||||
# 检查是否过期
|
||||
file_age = datetime.now() - datetime.fromtimestamp(cache_file.stat().st_mtime)
|
||||
if file_age > timedelta(days=self.max_age_days):
|
||||
logger.debug(f"缓存已过期: {cache_key[:8]}...")
|
||||
cache_file.unlink()
|
||||
return None
|
||||
|
||||
# 读取缓存
|
||||
with open(cache_file, 'r', encoding='utf-8') as f:
|
||||
cache_data = json.load(f)
|
||||
|
||||
# 验证缓存
|
||||
if (cache_data.get('success') and
|
||||
cache_data.get('model') == model and
|
||||
self._validate_cache_data(cache_data, chunk)):
|
||||
|
||||
logger.debug(f"缓存命中: {cache_key[:8]}... ({len(chunk)} 段落)")
|
||||
return cache_data.get('translations', {})
|
||||
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"读取缓存失败: {e}")
|
||||
return None
|
||||
|
||||
def save_chunk_translation(self, chunk: List[Dict], translations: Dict[str, str],
|
||||
model: str, success: bool = True) -> None:
|
||||
"""
|
||||
保存chunk翻译到缓存
|
||||
|
||||
Args:
|
||||
chunk: 段落列表（带global_id）
|
||||
translations: {global_id: translation} 映射
|
||||
model: 模型名称
|
||||
success: 是否翻译成功
|
||||
"""
|
||||
if not self.enabled:
|
||||
return
|
||||
|
||||
try:
|
||||
cache_key = self._get_chunk_cache_key(chunk, model)
|
||||
cache_file = self._get_cache_file_path(cache_key)
|
||||
|
||||
# 构建缓存数据
|
||||
cache_data = {
|
||||
'global_ids': [p['global_id'] for p in chunk],
|
||||
'translations': translations,
|
||||
'model': model,
|
||||
'timestamp': datetime.now().isoformat(),
|
||||
'success': success,
|
||||
'paragraph_count': len(chunk),
|
||||
'cache_version': '3.0'
|
||||
}
|
||||
|
||||
with open(cache_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(cache_data, f, ensure_ascii=False, indent=2)
|
||||
|
||||
logger.debug(f"缓存已保存: {cache_key[:8]}... ({len(chunk)} 段落)")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"保存缓存失败: {e}")
|
||||
|
||||
def _get_chunk_cache_key(self, chunk: List[Dict], model: str) -> str:
|
||||
"""
|
||||
生成chunk缓存键（基于全局ID序列）
|
||||
|
||||
Args:
|
||||
chunk: 段落列表
|
||||
model: 模型名称
|
||||
|
||||
Returns:
|
||||
缓存键
|
||||
"""
|
||||
# 使用全局ID序列作为缓存键的一部分
|
||||
id_sequence = ",".join(p['global_id'] for p in chunk)
|
||||
combined = f"{id_sequence}|{model}"
|
||||
return hashlib.md5(combined.encode('utf-8')).hexdigest()
|
||||
|
||||
def _get_cache_file_path(self, cache_key: str) -> Path:
|
||||
"""获取缓存文件路径"""
|
||||
# 使用 hash 前缀分目录，避免单目录文件过多
|
||||
subdir = cache_key[:2]
|
||||
cache_subdir = self.translations_dir / subdir
|
||||
cache_subdir.mkdir(parents=True, exist_ok=True)
|
||||
return cache_subdir / f"{cache_key}.json"
|
||||
|
||||
def _validate_cache_data(self, cache_data: Dict, chunk: List[Dict]) -> bool:
|
||||
"""验证缓存数据的有效性"""
|
||||
# 检查ID序列是否匹配
|
||||
cached_ids = cache_data.get('global_ids', [])
|
||||
chunk_ids = [p['global_id'] for p in chunk]
|
||||
|
||||
if cached_ids != chunk_ids:
|
||||
logger.debug("缓存ID序列不匹配")
|
||||
return False
|
||||
|
||||
# 只要有翻译结果就认为有效，不要求数量完全匹配
|
||||
translations = cache_data.get('translations', {})
|
||||
if not translations:
|
||||
logger.debug("缓存无翻译结果")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def clear_cache(self, older_than_days: Optional[int] = None) -> int:
|
||||
"""清理缓存"""
|
||||
if not self.enabled or not self.translations_dir.exists():
|
||||
return 0
|
||||
|
||||
cleared_count = 0
|
||||
cutoff_time = None
|
||||
|
||||
if older_than_days is not None:
|
||||
cutoff_time = datetime.now() - timedelta(days=older_than_days)
|
||||
|
||||
try:
|
||||
for cache_file in self.translations_dir.rglob('*.json'):
|
||||
should_delete = False
|
||||
|
||||
if cutoff_time is None:
|
||||
should_delete = True
|
||||
else:
|
||||
file_time = datetime.fromtimestamp(cache_file.stat().st_mtime)
|
||||
should_delete = file_time < cutoff_time
|
||||
|
||||
if should_delete:
|
||||
cache_file.unlink()
|
||||
cleared_count += 1
|
||||
|
||||
# 清理空目录
|
||||
for date_dir in self.translations_dir.iterdir():
|
||||
if date_dir.is_dir() and not any(date_dir.iterdir()):
|
||||
date_dir.rmdir()
|
||||
|
||||
logger.info(f"清理了 {cleared_count} 个缓存文件")
|
||||
return cleared_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"清理缓存失败: {e}")
|
||||
return 0
|
||||
|
||||
def get_cache_stats(self) -> Dict:
|
||||
"""获取缓存统计信息"""
|
||||
if not self.enabled or not self.translations_dir.exists():
|
||||
return {'enabled': False}
|
||||
|
||||
try:
|
||||
cache_files = list(self.translations_dir.rglob('*.json'))
|
||||
total_files = len(cache_files)
|
||||
total_size = sum(f.stat().st_size for f in cache_files)
|
||||
|
||||
# 统计段落数
|
||||
total_paragraphs = 0
|
||||
for cache_file in cache_files:
|
||||
try:
|
||||
with open(cache_file, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
total_paragraphs += data.get('paragraph_count', 0)
|
||||
except:
|
||||
continue
|
||||
|
||||
return {
|
||||
'enabled': True,
|
||||
'total_files': total_files,
|
||||
'total_paragraphs': total_paragraphs,
|
||||
'total_size_mb': round(total_size / 1024 / 1024, 2),
|
||||
'cache_directory': str(self.cache_dir),
|
||||
'max_age_days': self.max_age_days
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"获取缓存统计失败: {e}")
|
||||
return {'enabled': True, 'error': str(e)}
|
||||
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
EPUB 解析器模块 (EPUB Parser Module)
|
||||
|
||||
该模块负责读取 EPUB 文件，提取元数据和内容项目。
|
||||
它使用 ebooklib 库来处理 EPUB 格式的底层细节。
|
||||
|
||||
Classes:
|
||||
EPUBParser: 负责 EPUB 文件的加载、元数据提取和内容项遍历。
|
||||
"""
|
||||
|
||||
import ebooklib
|
||||
from ebooklib import epub
|
||||
from bs4 import BeautifulSoup
|
||||
from typing import List, Dict, Any
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class EPUBParser:
|
||||
"""
|
||||
EPUB 文件解析器。
|
||||
|
||||
负责加载 EPUB 文件，提取书籍元数据（如标题、作者），并提供方法来遍历和提取
|
||||
书中的文档内容（HTML/XHTML）。
|
||||
|
||||
Attributes:
|
||||
epub_path (Path): EPUB 文件的路径对象。
|
||||
book (epub.EpubBook): ebooklib 加载的书籍对象。
|
||||
metadata (Dict[str, str]): 提取的书籍元数据字典。
|
||||
"""
|
||||
|
||||
def __init__(self, epub_path: str):
|
||||
"""
|
||||
初始化 EPUB 解析器。
|
||||
|
||||
Args:
|
||||
epub_path (str): EPUB 文件的文件路径。
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: 如果指定的文件不存在。
|
||||
Exception: 如果 EPUB 文件加载失败（格式错误等）。
|
||||
"""
|
||||
self.epub_path = Path(epub_path)
|
||||
if not self.epub_path.exists():
|
||||
raise FileNotFoundError(f"EPUB 文件不存在: {epub_path}")
|
||||
|
||||
try:
|
||||
# ignore_ncx=True 是为了避免某些旧版 epub 的警告，但新版 ebooklib 可能行为不同
|
||||
# 这里直接读取，让 ebooklib 处理
|
||||
self.book = epub.read_epub(str(self.epub_path))
|
||||
logger.info(f"成功加载 EPUB: {self.epub_path.name}")
|
||||
except Exception as e:
|
||||
logger.error(f"加载 EPUB 失败: {e}")
|
||||
raise
|
||||
|
||||
self.metadata = self._extract_metadata()
|
||||
|
||||
def _extract_metadata(self) -> Dict[str, str]:
|
||||
"""
|
||||
从 EPUB 对象中提取标准元数据。
|
||||
|
||||
提取 Dublin Core (DC) 元数据，包括标题、作者和语言。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 包含 'title', 'author', 'language' 的字典。
|
||||
如果提取失败，会使用默认值 ("Unknown", "en")。
|
||||
"""
|
||||
metadata = {}
|
||||
|
||||
try:
|
||||
# get_metadata 返回的是 (value, dict) 的列表，我们取第一个结果
|
||||
title_meta = self.book.get_metadata('DC', 'title')
|
||||
metadata['title'] = title_meta[0][0] if title_meta else "Unknown"
|
||||
|
||||
author_meta = self.book.get_metadata('DC', 'creator')
|
||||
metadata['author'] = author_meta[0][0] if author_meta else "Unknown"
|
||||
|
||||
lang_meta = self.book.get_metadata('DC', 'language')
|
||||
metadata['language'] = lang_meta[0][0] if lang_meta else "en"
|
||||
|
||||
logger.info(f"书籍: {metadata['title']} - {metadata['author']}")
|
||||
except Exception as e:
|
||||
logger.warning(f"提取元数据时出错: {e}")
|
||||
# 设置保底值
|
||||
metadata.setdefault('title', 'Unknown')
|
||||
metadata.setdefault('author', 'Unknown')
|
||||
metadata.setdefault('language', 'en')
|
||||
|
||||
return metadata
|
||||
|
||||
def extract_all_content_items(self) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
提取所有可翻译的内容项目（文档）。
|
||||
|
||||
遍历 EPUB 中的所有 Item，筛选出类型为 ITEM_DOCUMENT 的项目。
|
||||
同时会进行简单的过滤，跳过内容过短（<100字符）或看起来像非正文的文件（如 nav, toc, cover）。
|
||||
|
||||
Returns:
|
||||
List[Dict[str, Any]]: 内容项目列表。每个字典包含：
|
||||
- item (epub.EpubItem): 原始 Item 对象。
|
||||
- file_name (str): 文件名。
|
||||
- content (str): 解码后的 HTML 内容。
|
||||
- text_length (int): 纯文本长度（用于统计）。
|
||||
"""
|
||||
content_items = []
|
||||
|
||||
# 获取所有文档类型的项目
|
||||
for item in self.book.get_items():
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT:
|
||||
try:
|
||||
# 获取内容 (bytes -> str)
|
||||
content = item.get_content().decode('utf-8')
|
||||
|
||||
# 简单的内容验证：提取纯文本检查长度
|
||||
soup = BeautifulSoup(content, 'html.parser')
|
||||
text = soup.get_text().strip()
|
||||
|
||||
# 1. 跳过太短的内容（可能是只有图片的页面、空页面）
|
||||
if len(text) < 100:
|
||||
logger.debug(f"跳过短内容: {item.get_name()} ({len(text)} 字符)")
|
||||
continue
|
||||
|
||||
# 2. 跳过明显的非正文内容 (根据文件名判断)
|
||||
name_lower = item.get_name().lower()
|
||||
skip_patterns = ['cover', 'copyright', 'titlepage', 'halftitle',
|
||||
'nav.xhtml', 'toc.xhtml']
|
||||
if any(pattern in name_lower for pattern in skip_patterns):
|
||||
logger.debug(f"跳过非正文内容: {item.get_name()}")
|
||||
continue
|
||||
|
||||
content_items.append({
|
||||
'item': item,
|
||||
'file_name': item.get_name(),
|
||||
'content': content,
|
||||
'text_length': len(text)
|
||||
})
|
||||
|
||||
logger.debug(f"添加内容项: {item.get_name()} ({len(text)} 字符)")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"处理项目失败 {item.get_name()}: {e}")
|
||||
continue
|
||||
|
||||
logger.info(f"提取了 {len(content_items)} 个内容项目")
|
||||
return content_items
|
||||
|
||||
def get_book_info(self) -> Dict[str, str]:
|
||||
"""
|
||||
获取书籍的摘要信息。
|
||||
|
||||
Returns:
|
||||
Dict[str, str]: 包含文件名、标题、作者、语言和文档数量的字典。
|
||||
"""
|
||||
# 统计内容项
|
||||
document_count = sum(1 for item in self.book.get_items()
|
||||
if item.get_type() == ebooklib.ITEM_DOCUMENT)
|
||||
|
||||
return {
|
||||
'filename': self.epub_path.name,
|
||||
'title': self.metadata.get('title', 'Unknown'),
|
||||
'author': self.metadata.get('author', 'Unknown'),
|
||||
'language': self.metadata.get('language', 'en'),
|
||||
'document_count': document_count
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
"""
|
||||
LLM Client Module - Generic OpenAI Compatible
|
||||
|
||||
Features:
|
||||
1. Fully configurable via config.json (base_url, headers).
|
||||
2. Minimal ID parsing logic.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from openai import AsyncOpenAI
|
||||
from typing import List, Dict, Optional, Any
|
||||
from loguru import logger
|
||||
import time
|
||||
from tenacity import retry, stop_after_attempt, wait_exponential
|
||||
from .manifest_manager import ManifestItem
|
||||
|
||||
|
||||
class RateLimiter:
|
||||
"""Rate limiter for concurrency and RPM."""
|
||||
def __init__(self, requests_per_minute: int, concurrent_requests: int):
|
||||
self.semaphore = asyncio.Semaphore(concurrent_requests)
|
||||
self.min_interval = 60.0 / requests_per_minute if requests_per_minute > 0 else 0
|
||||
self.last_request_time = 0
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
async def acquire(self):
|
||||
await self.semaphore.acquire()
|
||||
async with self._lock:
|
||||
current_time = time.time()
|
||||
wait_time = self.min_interval - (current_time - self.last_request_time)
|
||||
if wait_time > 0:
|
||||
await asyncio.sleep(wait_time)
|
||||
self.last_request_time = time.time()
|
||||
|
||||
def release(self):
|
||||
self.semaphore.release()
|
||||
|
||||
|
||||
class LLMClient:
|
||||
"""Generic OpenAI-compatible API Client."""
|
||||
|
||||
def __init__(self, config: Dict):
|
||||
self.config = config
|
||||
llm_config = config["llm"]
|
||||
|
||||
api_key = llm_config.get("api_key")
|
||||
base_url = llm_config.get("base_url")
|
||||
extra_headers = llm_config.get("extra_headers", {})
|
||||
|
||||
if not api_key:
|
||||
raise ValueError("API Key is missing in config")
|
||||
|
||||
self.client = AsyncOpenAI(
|
||||
base_url=base_url,
|
||||
api_key=api_key,
|
||||
default_headers=extra_headers
|
||||
)
|
||||
|
||||
# In v0.07, we use models.fast and models.smart
|
||||
self.models = llm_config.get("models", {"fast": "gpt-3.5-turbo", "smart": "gpt-4"})
|
||||
|
||||
self.rate_limiter = RateLimiter(
|
||||
llm_config["rate_limits"]["requests_per_minute"],
|
||||
llm_config["rate_limits"]["concurrent_requests"]
|
||||
)
|
||||
self.prompts = self._load_prompts()
|
||||
|
||||
def _load_prompts(self) -> Dict:
|
||||
try:
|
||||
with open("config/prompts.json", "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except:
|
||||
return {}
|
||||
|
||||
async def translate_chunk(self, items: List[ManifestItem], glossary: Dict = None, instruction: str = None, model_type: str = "fast") -> Dict[str, str]:
|
||||
"""Translate a chunk."""
|
||||
if not items: return {}
|
||||
|
||||
model = self.models.get(model_type, self.models.get("fast"))
|
||||
prompt = self._build_prompt(items)
|
||||
|
||||
try:
|
||||
# Build System Prompt
|
||||
base_sys_prompt = self.prompts.get("translation", {}).get("system", "You are a professional translator.")
|
||||
|
||||
if instruction:
|
||||
base_sys_prompt += f"\n\nBook Style Guide:\n{instruction}"
|
||||
|
||||
if glossary:
|
||||
glossary_text = "\n".join([f"{k} -> {v}" for k, v in glossary.items()])
|
||||
base_sys_prompt += f"\n\nTerminology:\n{glossary_text}"
|
||||
|
||||
# Strict formatting instructions
|
||||
base_sys_prompt += "\n\nRequirements:\n1. Each line MUST start with ID (p_xxxxx).\n2. DO NOT modify IDs.\n3. Return only translations."
|
||||
|
||||
raw_response = await self._make_request(model, base_sys_prompt, prompt)
|
||||
|
||||
if not raw_response:
|
||||
return {item.global_id: f"[Error - Empty Response]" for item in items}
|
||||
|
||||
return self._simple_parse(raw_response, items)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Translation failed ({model}): {e}")
|
||||
return {item.global_id: f"[Error - {str(e)}]" for item in items}
|
||||
|
||||
async def raw_chat_completion(self, system_prompt: str, user_prompt: str, model_type: str = "smart") -> str:
|
||||
"""Generic chat completion (for Profiler)."""
|
||||
model = self.models.get(model_type, self.models.get("smart"))
|
||||
return await self._make_request(model, system_prompt, user_prompt)
|
||||
|
||||
def _build_prompt(self, items: List[ManifestItem]) -> str:
|
||||
lines = []
|
||||
for item in items:
|
||||
lines.append(f"{item.global_id} {item.clean_text}")
|
||||
return "\n".join(lines)
|
||||
|
||||
def _simple_parse(self, response: str, items: List[ManifestItem]) -> Dict[str, str]:
|
||||
results = {}
|
||||
for i, item in enumerate(items):
|
||||
current_id = item.global_id
|
||||
start_idx = response.find(current_id)
|
||||
if start_idx == -1: continue
|
||||
|
||||
end_idx = len(response)
|
||||
if i + 1 < len(items):
|
||||
next_id = items[i+1].global_id
|
||||
next_found = response.find(next_id, start_idx + len(current_id))
|
||||
if next_found != -1:
|
||||
end_idx = next_found
|
||||
|
||||
content = response[start_idx:end_idx].strip()
|
||||
clean_content = content[len(current_id):].strip()
|
||||
clean_content = clean_content.lstrip(":： \t")
|
||||
if clean_content:
|
||||
results[current_id] = clean_content
|
||||
|
||||
# Fallback
|
||||
if len(results) < len(items):
|
||||
for line in response.split("\n"):
|
||||
line = line.strip()
|
||||
for item in items:
|
||||
if item.global_id not in results and line.startswith(item.global_id):
|
||||
res = line[len(item.global_id):].strip().lstrip(":： ")
|
||||
if res: results[item.global_id] = res
|
||||
return results
|
||||
|
||||
@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
|
||||
async def _make_request(self, model: str, system_prompt: str, user_prompt: str) -> str:
|
||||
await self.rate_limiter.acquire()
|
||||
try:
|
||||
resp = await self.client.chat.completions.create(
|
||||
model=model,
|
||||
messages=[
|
||||
{"role": "system", "content": system_prompt},
|
||||
{"role": "user", "content": user_prompt}
|
||||
],
|
||||
temperature=self.config['translation'].get('temperature', 0.2),
|
||||
max_tokens=8000
|
||||
)
|
||||
return resp.choices[0].message.content.strip()
|
||||
finally:
|
||||
self.rate_limiter.release()
|
||||
|
||||
async def close(self):
|
||||
await self.client.close()
|
||||
|
||||
# Alias for backward compatibility
|
||||
OpenRouterClient = LLMClient
|
||||
@@ -0,0 +1,157 @@
|
||||
"""
|
||||
Manifest 管理器模块 (Manifest Manager Module)
|
||||
|
||||
该模块是系统的单一真理源 (SSOT)。
|
||||
它记录了每一段文本的原始状态、清洗后的文本、哈希值以及翻译状态。
|
||||
所有对翻译流程的操作（提取、翻译、回填）都必须通过修改此 Manifest 进行。
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import hashlib
|
||||
from typing import List, Dict, Optional, Any
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
from dataclasses import dataclass, asdict, field
|
||||
|
||||
@dataclass
|
||||
class ManifestItem:
|
||||
"""代表一个翻译单元（通常是一个段落）"""
|
||||
global_id: str
|
||||
source_file: str
|
||||
original_html: str
|
||||
clean_text: str
|
||||
text_hash: str
|
||||
tag: str
|
||||
translation: Optional[str] = None
|
||||
status: str = "pending" # pending, translated, ignored, failed
|
||||
error_msg: Optional[str] = None
|
||||
model_used: Optional[str] = None # 记录使用的模型
|
||||
quality_score: Optional[int] = None # 记录质量评分
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def to_dict(self):
|
||||
return asdict(self)
|
||||
|
||||
class ManifestManager:
|
||||
"""
|
||||
负责 Manifest 的生命周期管理。
|
||||
"""
|
||||
|
||||
def __init__(self, manifest_path: str):
|
||||
self.manifest_path = Path(manifest_path)
|
||||
self.data: Dict[str, Any] = {
|
||||
"book_id": "",
|
||||
"metadata": {},
|
||||
"items": []
|
||||
}
|
||||
self._items_by_id: Dict[str, ManifestItem] = {}
|
||||
|
||||
def load(self) -> bool:
|
||||
"""从文件加载 Manifest。如果文件不存在则返回 False。"""
|
||||
if self.manifest_path.exists():
|
||||
try:
|
||||
with open(self.manifest_path, 'r', encoding='utf-8') as f:
|
||||
self.data = json.load(f)
|
||||
|
||||
# 重建对象映射
|
||||
self._items_by_id = {
|
||||
item['global_id']: ManifestItem(**item)
|
||||
for item in self.data["items"]
|
||||
}
|
||||
logger.info(f"成功从 {self.manifest_path} 加载 Manifest, 包含 {len(self._items_by_id)} 个项目")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"加载 Manifest 失败: {e}")
|
||||
return False
|
||||
return False
|
||||
|
||||
def save(self):
|
||||
"""将当前状态保存到 Manifest 文件。"""
|
||||
# 确保目录存在
|
||||
self.manifest_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 同步 items 到 data 字典
|
||||
self.data["items"] = [item.to_dict() for item in self._items_by_id.values()]
|
||||
|
||||
with open(self.manifest_path, 'w', encoding='utf-8') as f:
|
||||
json.dump(self.data, f, ensure_ascii=False, indent=2)
|
||||
# logger.debug(f"Manifest 已保存到 {self.manifest_path}")
|
||||
|
||||
def init_manifest(self, book_id: str, metadata: Dict):
|
||||
"""初始化一个新的 Manifest。"""
|
||||
self.data = {
|
||||
"book_id": book_id,
|
||||
"metadata": metadata,
|
||||
"items": []
|
||||
}
|
||||
self._items_by_id = {}
|
||||
self.save()
|
||||
|
||||
def add_item(self, source_file: str, original_html: str, clean_text: str, tag: str, metadata: Dict = None) -> ManifestItem:
|
||||
"""添加一个新的翻译项并分配 ID。"""
|
||||
# 生成全局 ID
|
||||
new_index = len(self._items_by_id) + 1
|
||||
global_id = f"p_{new_index:05d}"
|
||||
|
||||
# 生成内容哈希 (用于排重和缓存)
|
||||
text_hash = hashlib.sha256(clean_text.encode('utf-8')).hexdigest()
|
||||
|
||||
item = ManifestItem(
|
||||
global_id=global_id,
|
||||
source_file=source_file,
|
||||
original_html=original_html,
|
||||
clean_text=clean_text,
|
||||
text_hash=text_hash,
|
||||
tag=tag,
|
||||
metadata=metadata or {}
|
||||
)
|
||||
|
||||
self._items_by_id[global_id] = item
|
||||
return item
|
||||
|
||||
def get_items(self, status: str = None, file_name: str = None) -> List[ManifestItem]:
|
||||
"""按状态或文件名查询项目。"""
|
||||
items = list(self._items_by_id.values())
|
||||
if status:
|
||||
items = [i for i in items if i.status == status]
|
||||
if file_name:
|
||||
items = [i for i in items if i.source_file == file_name]
|
||||
|
||||
# 必须按 ID 顺序返回以保证分块正确
|
||||
return sorted(items, key=lambda x: x.global_id)
|
||||
|
||||
def update_item(self, global_id: str, translation: str, status: str = "translated", error: str = None, model: str = None, score: int = None):
|
||||
"""更新翻译结果。"""
|
||||
if global_id in self._items_by_id:
|
||||
item = self._items_by_id[global_id]
|
||||
if translation is not None:
|
||||
item.translation = translation
|
||||
item.status = status
|
||||
if error:
|
||||
item.error_msg = error
|
||||
if model:
|
||||
item.model_used = model
|
||||
if score is not None:
|
||||
item.quality_score = score
|
||||
else:
|
||||
logger.warning(f"尝试更新不存在的 ID: {global_id}")
|
||||
|
||||
@property
|
||||
def stats(self) -> Dict:
|
||||
"""获取翻译进度统计。"""
|
||||
total = len(self._items_by_id)
|
||||
if total == 0: return {"progress": "0%"}
|
||||
|
||||
translated = sum(1 for i in self._items_by_id.values() if i.status == "translated")
|
||||
ignored = sum(1 for i in self._items_by_id.values() if i.status == "ignored")
|
||||
failed = sum(1 for i in self._items_by_id.values() if i.status == "failed")
|
||||
|
||||
return {
|
||||
"total": total,
|
||||
"translated": translated,
|
||||
"ignored": ignored,
|
||||
"failed": failed,
|
||||
"pending": total - translated - ignored - failed,
|
||||
"progress_percent": round((translated + ignored) / total * 100, 1)
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
"""
|
||||
Quality Manager Module
|
||||
|
||||
Responsible for evaluating translation quality and deciding on re-translation.
|
||||
"""
|
||||
|
||||
import json
|
||||
import random
|
||||
from typing import List, Dict, Any, Tuple
|
||||
from loguru import logger
|
||||
from .manifest_manager import ManifestItem
|
||||
from .llm_client import LLMClient
|
||||
|
||||
class QualityManager:
|
||||
def __init__(self, config: Dict, llm_client: LLMClient):
|
||||
self.config = config
|
||||
self.llm_client = llm_client
|
||||
self.qc_config = config['translation'].get('quality_control', {})
|
||||
self.pass_score = self.qc_config.get('pass_score', 7)
|
||||
self.sample_size = self.qc_config.get('sample_size', 2)
|
||||
|
||||
async def evaluate_chunk(self, chunk: List[ManifestItem]) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Evaluate a chunk of translations.
|
||||
|
||||
Returns:
|
||||
(passed: bool, average_score: int, reason: str)
|
||||
"""
|
||||
if not self.qc_config.get('enabled', False):
|
||||
return True, 10, "QC Disabled"
|
||||
|
||||
# 1. Sample items
|
||||
# Filter for items that actually have content and translations
|
||||
valid_items = [item for item in chunk if item.translation and len(item.clean_text) > 20]
|
||||
|
||||
if not valid_items:
|
||||
return True, 10, "No valid items to sample"
|
||||
|
||||
sample_items = random.sample(valid_items, min(len(valid_items), self.sample_size))
|
||||
|
||||
# 2. Build Prompt
|
||||
prompt = self._build_evaluation_prompt(sample_items)
|
||||
|
||||
# 3. Call LLM (Smart)
|
||||
try:
|
||||
response = await self.llm_client.raw_chat_completion(
|
||||
system_prompt="You are a professional translation editor.",
|
||||
user_prompt=prompt,
|
||||
model_type="smart"
|
||||
)
|
||||
|
||||
# 4. Parse JSON
|
||||
# Clean potential markdown
|
||||
json_str = response.strip()
|
||||
if "```json" in json_str:
|
||||
json_str = json_str.split("```json")[1].split("```")[0].strip()
|
||||
elif "```" in json_str:
|
||||
json_str = json_str.split("```")[1].split("```")[0].strip()
|
||||
|
||||
result = json.loads(json_str)
|
||||
score = result.get('score', 0)
|
||||
reason = result.get('reason', 'No reason provided')
|
||||
|
||||
passed = score >= self.pass_score
|
||||
return passed, score, reason
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"QC evaluation failed: {e}")
|
||||
# If QC fails, we default to PASS to avoid blocking progress, but log it
|
||||
return True, 0, f"QC Error: {e}"
|
||||
|
||||
def _build_evaluation_prompt(self, items: List[ManifestItem]) -> str:
|
||||
content = ""
|
||||
for i, item in enumerate(items, 1):
|
||||
content += f"Item {i}:\nOriginal: {item.clean_text}\nTranslation: {item.translation}\n\n"
|
||||
|
||||
return f"""Please evaluate the following translations (English to Chinese).
|
||||
Focus on accuracy, fluency, and terminology consistency.
|
||||
|
||||
Items to evaluate:
|
||||
{content}
|
||||
|
||||
Return a JSON object with:
|
||||
- \"score\": An integer from 1 to 10 (10 being perfect).
|
||||
- \"reason\": A brief explanation of the score.
|
||||
|
||||
JSON Output:"""
|
||||
@@ -0,0 +1,160 @@
|
||||
"""
|
||||
文本处理器模块 (Text Processor Module) - Manifest 驱动版
|
||||
|
||||
该模块专注于 HTML 文档的遍历和段落提取。
|
||||
它不再维护全局状态，而是将提取的内容注册到 ManifestManager 中。
|
||||
"""
|
||||
|
||||
import re
|
||||
from bs4 import BeautifulSoup
|
||||
from typing import List, Dict, Any
|
||||
from loguru import logger
|
||||
from .manifest_manager import ManifestManager
|
||||
|
||||
|
||||
class TextProcessor:
|
||||
"""
|
||||
负责从 HTML 中识别有效段落并进行清洗。
|
||||
"""
|
||||
|
||||
def __init__(self, config: Dict):
|
||||
"""
|
||||
Args:
|
||||
config (Dict): 全局配置。
|
||||
"""
|
||||
self.config = config
|
||||
self.chunk_size = config['translation'].get('chunk_size', 5000)
|
||||
|
||||
def extract_to_manifest(self, html_content: str, source_file: str, manifest: ManifestManager):
|
||||
"""
|
||||
解析 HTML 内容，并将识别出的段落注册到 Manifest 中。
|
||||
|
||||
Args:
|
||||
html_content (str): HTML 源码。
|
||||
source_file (str): 来源文件名。
|
||||
manifest (ManifestManager): 清单管理器实例。
|
||||
"""
|
||||
try:
|
||||
soup = BeautifulSoup(html_content, 'html.parser')
|
||||
|
||||
# 1. 移除不需要的元素
|
||||
for element in soup(['script', 'style', 'meta', 'link']):
|
||||
element.decompose()
|
||||
|
||||
# 2. 获取有效的文本元素 (使用静态过滤逻辑)
|
||||
text_elements = self.get_valid_text_elements(soup)
|
||||
|
||||
# 3. 注册到 Manifest
|
||||
for element in text_elements:
|
||||
clean_text = self.clean_element_text(element)
|
||||
|
||||
# 过滤逻辑
|
||||
if not clean_text:
|
||||
continue
|
||||
|
||||
status = "pending"
|
||||
# 如果是导航元素，标记为 ignored
|
||||
if self.is_navigation_element(element):
|
||||
status = "ignored"
|
||||
|
||||
# 注册
|
||||
item = manifest.add_item(
|
||||
source_file=source_file,
|
||||
original_html=str(element),
|
||||
clean_text=clean_text,
|
||||
tag=element.name,
|
||||
metadata={"status": status} # 临时传递给 manifest
|
||||
)
|
||||
|
||||
# 同步更新 manifest 状态 (如果需要过滤)
|
||||
if status == "ignored":
|
||||
manifest.update_item(item.global_id, translation=None, status="ignored")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"从 {source_file} 提取段落失败: {e}")
|
||||
|
||||
@staticmethod
|
||||
def get_valid_text_elements(soup) -> List:
|
||||
"""获取不含嵌套子块的叶子级文本容器元素。"""
|
||||
tags = ['p', 'div', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'blockquote', 'li', 'td']
|
||||
all_candidates = soup.find_all(tags)
|
||||
candidate_set = set(all_candidates)
|
||||
|
||||
final_elements = []
|
||||
for element in all_candidates:
|
||||
# 如果包含其他候选标签，说明是容器，跳过
|
||||
if any(d in candidate_set for d in element.find_all(tags)):
|
||||
continue
|
||||
final_elements.append(element)
|
||||
return final_elements
|
||||
|
||||
@staticmethod
|
||||
def clean_element_text(element) -> str:
|
||||
"""清理 HTML 元素，提取纯净的待翻译文本。"""
|
||||
element_copy = element.__copy__()
|
||||
|
||||
# 移除脚注引用等
|
||||
for tag in element_copy.find_all(['sup', 'sub']):
|
||||
tag.decompose()
|
||||
|
||||
footnote_patterns = re.compile(r'footnote|endnote|reference|note|super|sub', re.I)
|
||||
for tag in element_copy.find_all(['a', 'span', 'div'], class_=footnote_patterns):
|
||||
tag.decompose()
|
||||
|
||||
# 移除仅包含数字的 span
|
||||
for tag in element_copy.find_all('span'):
|
||||
if re.match(r'^(\[\d+\]|\(\d+\)|\d+)$', tag.get_text().strip()):
|
||||
tag.decompose()
|
||||
|
||||
text = element_copy.get_text().strip()
|
||||
# 正则清理残留引用标识 (如 sentence.2)
|
||||
text = re.sub(r'(\.|。|，|,)\s*(\[\d+\]|\d+)(?=\s|$)', r'\1', text)
|
||||
text = re.sub(r'\s+', ' ', text)
|
||||
return text
|
||||
|
||||
@staticmethod
|
||||
def is_navigation_element(element) -> bool:
|
||||
"""判断是否是无翻译价值的导航、页码元素。"""
|
||||
classes = element.get('class', [])
|
||||
nav_classes = ['nav', 'navigation', 'toc', 'menu', 'header', 'footer', 'page-number']
|
||||
class_str = ' '.join(classes).lower() if isinstance(classes, list) else str(classes).lower()
|
||||
|
||||
if any(nc in class_str for nc in nav_classes):
|
||||
return True
|
||||
|
||||
# 检查父级
|
||||
parent = element.parent
|
||||
if parent:
|
||||
p_classes = parent.get('class', [])
|
||||
p_class_str = ' '.join(p_classes).lower() if isinstance(p_classes, list) else str(p_classes).lower()
|
||||
if any(nc in p_class_str for nc in nav_classes):
|
||||
return True
|
||||
return False
|
||||
|
||||
def create_chunks_from_manifest(self, manifest: ManifestManager) -> List[List[Any]]:
|
||||
"""
|
||||
从 Manifest 中筛选待翻译项目并分块。
|
||||
"""
|
||||
pending_items = manifest.get_items(status="pending")
|
||||
if not pending_items:
|
||||
return []
|
||||
|
||||
chunks = []
|
||||
current_chunk = []
|
||||
current_size = 0
|
||||
|
||||
for item in pending_items:
|
||||
text_len = len(item.clean_text)
|
||||
if current_size + text_len > self.chunk_size and current_chunk:
|
||||
chunks.append(current_chunk)
|
||||
current_chunk = []
|
||||
current_size = 0
|
||||
|
||||
current_chunk.append(item)
|
||||
current_size += text_len
|
||||
|
||||
if current_chunk:
|
||||
chunks.append(current_chunk)
|
||||
|
||||
logger.info(f"分块完成: 共有 {len(pending_items)} 个待翻译项，分为 {len(chunks)} 个块")
|
||||
return chunks
|
||||
@@ -0,0 +1,159 @@
|
||||
"""
|
||||
EPUB Translator Core Module - v0.06 (Simplified)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import traceback
|
||||
from typing import List, Dict, Any
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
from rich.console import Console
|
||||
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TimeElapsedColumn
|
||||
|
||||
from .epub_parser import EPUBParser
|
||||
from .llm_client import LLMClient
|
||||
from .text_processor import TextProcessor
|
||||
from .bilingual_builder import BilingualEPUBBuilder
|
||||
from .manifest_manager import ManifestManager
|
||||
from .book_profiler import BookProfiler
|
||||
from .cache import TranslationCache
|
||||
from .utils import add_spacing_between_cn_and_en_num
|
||||
|
||||
|
||||
class EPUBTranslator:
|
||||
|
||||
def __init__(self, config: Dict, use_cache: bool = True):
|
||||
self.config = config
|
||||
self.console = Console()
|
||||
self.use_cache = use_cache
|
||||
|
||||
self.parser = None
|
||||
self.llm_client = LLMClient(config)
|
||||
self.text_processor = TextProcessor(config)
|
||||
self.profiler = BookProfiler(config, self.llm_client)
|
||||
self.cache = TranslationCache(config) if use_cache else None
|
||||
|
||||
self.manifest_dir = Path("cache/manifests")
|
||||
self.manifest_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
async def translate_epub(self, epub_path: str, test_mode: bool = False, output_dir: str = None) -> str:
|
||||
try:
|
||||
epub_path = Path(epub_path)
|
||||
self.parser = EPUBParser(str(epub_path))
|
||||
|
||||
# 1. Manifest
|
||||
manifest_path = self.manifest_dir / f"{epub_path.stem}_manifest.json"
|
||||
manifest = ManifestManager(str(manifest_path))
|
||||
|
||||
if not manifest.load() or not self.use_cache:
|
||||
self.console.print("[yellow]Initializing Manifest...[/yellow]")
|
||||
manifest.init_manifest(book_id=epub_path.name, metadata=self.parser.get_book_info())
|
||||
content_items = self.parser.extract_all_content_items()
|
||||
for item in content_items:
|
||||
self.text_processor.extract_to_manifest(item['content'], item['file_name'], manifest)
|
||||
manifest.save()
|
||||
|
||||
stats = manifest.stats
|
||||
self.console.print(f"[green]Manifest loaded: {stats['total']} paragraphs[/green]")
|
||||
if stats['pending'] < stats['total'] and stats['translated'] > 0:
|
||||
self.console.print(
|
||||
f"[yellow]Detected incomplete translation task, completed {stats['translated']}/{stats['total']}, "
|
||||
f"continuing translation of remaining {stats['pending']} paragraphs...[/yellow]"
|
||||
)
|
||||
|
||||
# 2. Profile (Glossary)
|
||||
profile = {}
|
||||
if not test_mode:
|
||||
self.console.print("[yellow]Generating Book Profile...[/yellow]")
|
||||
profile = await self.profiler.analyze_book(manifest)
|
||||
self.console.print(f"Genre: {profile.get('genre')} | Style: {profile.get('style')}")
|
||||
|
||||
# 3. Translate
|
||||
chunks = self.text_processor.create_chunks_from_manifest(manifest)
|
||||
|
||||
if test_mode:
|
||||
self.console.print("[yellow]Test mode enabled: Translating only first 3 chunks...[/yellow]")
|
||||
chunks = chunks[:3]
|
||||
|
||||
if chunks:
|
||||
await self._translate_concurrently(chunks, manifest, profile)
|
||||
|
||||
# 4. Build
|
||||
self.console.print("\n[yellow]Building bilingual EPUB...[/yellow]")
|
||||
output_path = output_dir or self.config['output']['output_dir']
|
||||
builder = BilingualEPUBBuilder(self.parser.book, self.config)
|
||||
|
||||
translation_map = {item.global_id: item.translation for item in manifest.get_items() if item.translation}
|
||||
paragraph_map = {item.global_id: {
|
||||
"file_name": item.source_file,
|
||||
"text": item.clean_text,
|
||||
"html_element": item.original_html
|
||||
} for item in manifest.get_items()}
|
||||
|
||||
result_file = builder.create_bilingual_epub_with_mapping(
|
||||
translation_map, paragraph_map, output_path
|
||||
)
|
||||
|
||||
final_stats = manifest.stats
|
||||
self.console.print(f"""
|
||||
[green]✅ Translation complete![/green]
|
||||
- Total Paragraphs: {final_stats['total']}
|
||||
- Successfully Translated: {final_stats['translated']}
|
||||
- Skipped: {final_stats['ignored']}
|
||||
- Failed: {final_stats['failed']}
|
||||
- Output File: {result_file}
|
||||
""")
|
||||
return result_file
|
||||
except Exception as e:
|
||||
traceback.print_exc()
|
||||
logger.error(f"Translation flow failed: {e}")
|
||||
raise
|
||||
|
||||
async def _translate_concurrently(self, chunks: List[List[Any]], manifest: ManifestManager, profile: Dict):
|
||||
total_chunks = len(chunks)
|
||||
glossary = profile.get('glossary', {})
|
||||
instruction = profile.get('translation_instruction', "")
|
||||
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
BarColumn(),
|
||||
TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
|
||||
TimeElapsedColumn(),
|
||||
console=self.console
|
||||
) as progress:
|
||||
task_id = progress.add_task(f"[cyan]Translating...", total=total_chunks)
|
||||
|
||||
async def worker(chunk, idx):
|
||||
try:
|
||||
model_name = self.llm_client.models.get('fast')
|
||||
chunk_dicts = [item.to_dict() for item in chunk]
|
||||
results = None
|
||||
|
||||
if self.cache:
|
||||
results = self.cache.get_chunk_translation(chunk_dicts, model=model_name)
|
||||
|
||||
if not results:
|
||||
results = await self.llm_client.translate_chunk(
|
||||
chunk,
|
||||
glossary=glossary,
|
||||
instruction=instruction
|
||||
)
|
||||
if self.cache and results:
|
||||
self.cache.save_chunk_translation(chunk_dicts, results, model=model_name)
|
||||
|
||||
for item in chunk:
|
||||
if item.global_id in results:
|
||||
# 格式化翻译文本（增加盘古之白）
|
||||
translation = add_spacing_between_cn_and_en_num(results[item.global_id])
|
||||
manifest.update_item(item.global_id, translation)
|
||||
else:
|
||||
manifest.update_item(item.global_id, None, status="failed", error="Missing")
|
||||
manifest.save()
|
||||
except Exception as e:
|
||||
logger.error(f"Chunk {idx} failed: {e}")
|
||||
finally:
|
||||
progress.update(task_id, advance=1)
|
||||
|
||||
tasks = [worker(chunk, i) for i, chunk in enumerate(chunks)]
|
||||
await asyncio.gather(*tasks)
|
||||
@@ -0,0 +1,208 @@
|
||||
"""
|
||||
工具函数模块
|
||||
提供配置加载、日志设置等通用功能
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any
|
||||
from loguru import logger
|
||||
import sys
|
||||
from dotenv import load_dotenv
|
||||
|
||||
|
||||
def load_config(config_path: str = "config/config.json") -> Dict[str, Any]:
|
||||
"""
|
||||
加载配置文件
|
||||
|
||||
Args:
|
||||
config_path: 配置文件路径
|
||||
|
||||
Returns:
|
||||
配置字典
|
||||
"""
|
||||
# 加载 .env 文件
|
||||
load_dotenv()
|
||||
|
||||
try:
|
||||
with open(config_path, 'r', encoding='utf-8') as f:
|
||||
config = json.load(f)
|
||||
|
||||
# 从环境变量获取 API Key
|
||||
for provider_name, provider_config in config.get('providers', {}).items():
|
||||
env_key = f"{provider_name.upper()}_API_KEY"
|
||||
if env_key in os.environ:
|
||||
provider_config['api_key'] = os.environ[env_key]
|
||||
|
||||
return config
|
||||
except FileNotFoundError:
|
||||
raise FileNotFoundError(f"配置文件未找到: {config_path}")
|
||||
except json.JSONDecodeError as e:
|
||||
raise ValueError(f"配置文件格式错误: {e}")
|
||||
|
||||
|
||||
def load_prompts(prompts_path: str = "config/prompts.json") -> Dict[str, str]:
|
||||
"""
|
||||
加载提示词模板
|
||||
|
||||
Args:
|
||||
prompts_path: 提示词文件路径
|
||||
|
||||
Returns:
|
||||
提示词字典
|
||||
"""
|
||||
try:
|
||||
with open(prompts_path, 'r', encoding='utf-8') as f:
|
||||
return json.load(f)
|
||||
except FileNotFoundError:
|
||||
raise FileNotFoundError(f"提示词文件未找到: {prompts_path}")
|
||||
|
||||
|
||||
def setup_logging(config: Dict[str, Any]) -> None:
|
||||
"""
|
||||
设置日志配置
|
||||
|
||||
Args:
|
||||
config: 配置字典
|
||||
"""
|
||||
log_config = config.get('logging', {})
|
||||
|
||||
# 移除默认处理器
|
||||
logger.remove()
|
||||
|
||||
# 添加控制台输出
|
||||
logger.add(
|
||||
sys.stdout,
|
||||
level=log_config.get('level', 'INFO'),
|
||||
format="<green>{time:YYYY-MM-DD HH:mm:ss}</green> | <level>{level: <8}</level> | <cyan>{name}</cyan>:<cyan>{function}</cyan>:<cyan>{line}</cyan> - <level>{message}</level>"
|
||||
)
|
||||
|
||||
# 添加文件输出
|
||||
if 'file' in log_config:
|
||||
log_file = log_config['file']
|
||||
# 确保日志目录存在
|
||||
Path(log_file).parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.add(
|
||||
log_file,
|
||||
level=log_config.get('level', 'INFO'),
|
||||
rotation=log_config.get('rotation', '10 MB'),
|
||||
retention=log_config.get('retention', '7 days'),
|
||||
encoding='utf-8',
|
||||
format="{time:YYYY-MM-DD HH:mm:ss} | {level: <8} | {name}:{function}:{line} - {message}"
|
||||
)
|
||||
|
||||
|
||||
def ensure_output_dir(output_dir: str) -> Path:
|
||||
"""
|
||||
确保输出目录存在
|
||||
|
||||
Args:
|
||||
output_dir: 输出目录路径
|
||||
|
||||
Returns:
|
||||
输出目录的 Path 对象
|
||||
"""
|
||||
output_path = Path(output_dir)
|
||||
output_path.mkdir(parents=True, exist_ok=True)
|
||||
return output_path
|
||||
|
||||
|
||||
def sanitize_filename(filename: str) -> str:
|
||||
"""
|
||||
清理文件名，移除非法字符
|
||||
|
||||
Args:
|
||||
filename: 原始文件名
|
||||
|
||||
Returns:
|
||||
清理后的文件名
|
||||
"""
|
||||
import re
|
||||
# 移除或替换非法字符
|
||||
filename = re.sub(r'[<>:"/\\|?*]', '_', filename)
|
||||
# 移除多余的空格和点
|
||||
filename = re.sub(r'\s+', ' ', filename).strip('. ')
|
||||
return filename
|
||||
|
||||
|
||||
def format_file_size(size_bytes: int) -> str:
|
||||
"""
|
||||
格式化文件大小显示
|
||||
|
||||
Args:
|
||||
size_bytes: 字节数
|
||||
|
||||
Returns:
|
||||
格式化的大小字符串
|
||||
"""
|
||||
if size_bytes == 0:
|
||||
return "0B"
|
||||
|
||||
size_names = ["B", "KB", "MB", "GB"]
|
||||
import math
|
||||
i = int(math.floor(math.log(size_bytes, 1024)))
|
||||
p = math.pow(1024, i)
|
||||
s = round(size_bytes / p, 2)
|
||||
return f"{s} {size_names[i]}"
|
||||
|
||||
|
||||
def estimate_tokens(text: str) -> int:
|
||||
"""
|
||||
估算文本的 token 数量
|
||||
|
||||
Args:
|
||||
text: 输入文本
|
||||
|
||||
Returns:
|
||||
估算的 token 数量
|
||||
"""
|
||||
# 简单估算：英文约 4 字符/token，中文约 1.5 字符/token
|
||||
import re
|
||||
|
||||
# 分离中英文
|
||||
chinese_chars = len(re.findall(r'[\u4e00-\u9fff]', text))
|
||||
other_chars = len(text) - chinese_chars
|
||||
|
||||
# 估算 tokens
|
||||
estimated_tokens = chinese_chars / 1.5 + other_chars / 4
|
||||
return int(estimated_tokens)
|
||||
|
||||
|
||||
def truncate_text(text: str, max_length: int = 100) -> str:
|
||||
"""
|
||||
截断文本用于显示
|
||||
|
||||
Args:
|
||||
text: 原始文本
|
||||
max_length: 最大长度
|
||||
|
||||
Returns:
|
||||
截断后的文本
|
||||
"""
|
||||
if len(text) <= max_length:
|
||||
return text
|
||||
return text[:max_length-3] + "..."
|
||||
|
||||
|
||||
def add_spacing_between_cn_and_en_num(text: str) -> str:
|
||||
"""
|
||||
在中文和英文/数字之间添加空格（盘古之白）
|
||||
|
||||
Args:
|
||||
text: 原始文本
|
||||
|
||||
Returns:
|
||||
处理后的文本
|
||||
"""
|
||||
import re
|
||||
if not text:
|
||||
return text
|
||||
|
||||
# 中文-英文/数字
|
||||
text = re.sub(r'([\u4e00-\u9fff])([a-zA-Z0-9])', r'\1 \2', text)
|
||||
# 英文/数字-中文
|
||||
text = re.sub(r'([a-zA-Z0-9])([\u4e00-\u9fff])', r'\1 \2', text)
|
||||
|
||||
return text
|
||||
@@ -0,0 +1,94 @@
|
||||
# 更新日志 (CHANGELOG)
|
||||
|
||||
## [v0.08] - 2026-01-15 (The Optimizer)
|
||||
|
||||
### 🎯 占位符系统优化
|
||||
- **前缀/后缀标签分离**: 文本首尾的纯格式标签（如 `<sup>`, `<sub>`）不再发送给 LLM，自动回填。
|
||||
- **公式检测**: 数学变量和公式被识别为单一、不可翻译的占位符，显著降低 LLM 误翻译风险。
|
||||
- **简化占位符格式**: 从全局唯一 `φcXXXXXφ` 简化为段落局部 `φ1φ`，每个段落独立编号。
|
||||
|
||||
### 🎨 纯中文排版模式
|
||||
- **模式切换**: 支持 `--mode chinese` 生成保留原始排版的纯中文译本（默认 `--mode bilingual`）。
|
||||
- **Format Extractor (替换法)**:
|
||||
- 彻底重构格式提取逻辑，放弃 DOM 递归，采用基于正则的"标签序列替换法"。
|
||||
- 能够完美处理任意深度的嵌套标签，将其合并为单一占位符。
|
||||
- 100% 保留原始 HTML 属性（class, style, href 等），实现"像素级"格式还原。
|
||||
- **Format Restorer (自愈系)**:
|
||||
- 引入 `FormatRestorer` 模块，负责将占位符替换回原始 HTML 代码。
|
||||
- **自动修复 Agent**: 当检测到 LLM 丢失占位符时，自动触发回退机制进行格式修复。
|
||||
- **优雅降级**: 如果修复失败，系统会自动降级为纯文本，确保程序不崩溃。
|
||||
|
||||
### 🛡️ 深度优化
|
||||
- **占位符升级**: 从易混淆的 `«c...»` 升级为 `φc...φ`，显著降低 LLM 误翻译概率。
|
||||
- **容器样式继承**:
|
||||
- 中文模式：直接替换 `inner_html`，完美保留外层容器属性。
|
||||
- 双语模式：新建 `<p>` 标签时自动继承原文的 `class` 和其他属性。
|
||||
- **智能测试**: `--test` 模式逻辑升级，智能识别章节边界，自动翻译完第一章。
|
||||
|
||||
### 🔧 修复
|
||||
- 修复了 `LLMClient` 中正则表达式转义错误导致的 `FutureWarning`。
|
||||
- 修复了 `FormatExtractor` 循环引用问题。
|
||||
- 解决了复杂科学书籍中上标/链接嵌套导致的校验失败问题。
|
||||
|
||||
---
|
||||
|
||||
## [v0.07] - 2026-01-13 (The Refinement)
|
||||
|
||||
### 🛡️ 安全与配置
|
||||
- **环境隔离**: 引入 `.env` 支持，彻底移除了代码库中的硬编码 API Key。
|
||||
- **配置升级**: `utils.py` 现自动加载 `.env` 并注入到配置中，支持任意 Provider 的环境变量覆盖 (如 `V3_API_KEY`, `OPENROUTER_API_KEY`)。
|
||||
- **模板化**: 新增 `config.example.json` 和 `.env` 模板，提升部署安全性。
|
||||
|
||||
### 🚀 核心改进
|
||||
- **V3 Provider 支持**: 验证并修复了对 V3 API (OpenAI 兼容格式) 的支持，全流程跑通。
|
||||
- **EPUB 构建修复**: 解决了 `ebooklib` 在处理 TOC 时因缺少 UID 导致的 `Argument must be bytes or unicode` 崩溃问题。
|
||||
- **缓存优化**:
|
||||
- 缓存目录结构调整为 Hash 前缀 (`cache/translations/ab/...`)，解决了按日期分目录导致的缓存频繁失效问题。
|
||||
- 放宽了缓存验证逻辑，支持部分命中的缓存复用。
|
||||
- **视觉优化**: 引入“盘古之白” (Pangu spacing)，自动在中文与英文/数字之间添加空格，显著提升阅读体验。
|
||||
|
||||
### ⚡ 体验提升
|
||||
- **断点续传提示**: 启动时自动检测并提示未完成的翻译进度。
|
||||
- **详细统计**: 翻译完成后展示详细的成功/失败/跳过统计数据。
|
||||
- **并发优化**: 移除了冗余的信号量控制，完全依赖 `RateLimiter`，逻辑更清晰高效。
|
||||
|
||||
---
|
||||
|
||||
## [v0.05] - 2026-01-12 (The Arena)
|
||||
|
||||
### 🌟 核心突破
|
||||
- **书籍画像 (Book Profiler)**:
|
||||
- 自动提取前言和正文采样。
|
||||
- 生成 `Book Profile`，包含领域 (Genre)、文风 (Style)、目标受众 (Audience) 和翻译指令。
|
||||
- 生成 `Glossary` (术语表)，并支持自动注入 Prompt。
|
||||
- **状态绑定**: Profile 和 Glossary 现在直接存储在每本书的 `manifest.json` 中，互不干扰。
|
||||
- **模型竞技场 (Model Arena)**:
|
||||
- 自动选取典型 Chunk，让多个候选模型 (Gemini, Llama, Qwen) 同台竞技。
|
||||
- 引入 `Judge Agent` (基于 Smart 模型)，从准确性、信达雅维度评选最佳模型。
|
||||
- 自动锁定获胜模型用于全书翻译。
|
||||
|
||||
### 🏗️ 架构升级
|
||||
- **LLM Client 重构**:
|
||||
- **Syntax Fixes**: 彻底修复了正则构造中的语法错误。
|
||||
- **Quote Safety**: 移除了所有 f-string 中的复杂正则，改用安全的字符串拼接。
|
||||
- **Dual RateLimiters**: 引入主/副限流器，防止死锁。
|
||||
- **配置增强**:
|
||||
- `config.json` 支持 `arena_models` 和 `judge_model` 配置。
|
||||
|
||||
### 🔧 修复与优化
|
||||
- 修复了 `unhashable type: 'dict'` 错误 (移除了错误的 `{{}}`)。
|
||||
- 修复了多本书连续翻译时 Profile 串用的问题 (Profile 现已绑定至 Manifest)。
|
||||
|
||||
---
|
||||
|
||||
## [v0.03] - 2026-01-12
|
||||
- **极简 ID 锚点系统**: 废弃复杂的 `[p_xxxxx]` 格式，使用纯净 ID，彻底解决残留问题。
|
||||
- **智能术语一致性**: 引入 GlossaryManager。
|
||||
- **结构完美保留**: 修复了 EPUB Spine 和 Metadata 丢失问题。
|
||||
|
||||
## [v0.02] - 2026-01-12
|
||||
- **Manifest 驱动架构**: 引入 `ManifestManager` 作为单一真理源。
|
||||
- **流程解耦**: 提取、翻译、构建三阶段分离。
|
||||
|
||||
## [v0.01] - 2026-01-10
|
||||
- 初始版本，实现基本的并发翻译和 EPUB 解析。
|
||||
@@ -0,0 +1,97 @@
|
||||
# EPUB 双语翻译程序 v0.07
|
||||
|
||||
一个基于 OpenRouter/OpenAI API 的 EPUB 双语翻译工具，采用**全局编号系统**和**真并发翻译**。
|
||||
|
||||
## ✨ 核心特性
|
||||
|
||||
### 🎯 全局编号系统
|
||||
- **每个段落分配全局唯一ID**（格式：`p_0001`, `p_0002`...）
|
||||
- **ID贯穿全流程**：提取 → 翻译 → 组装
|
||||
- **精确对应保证**：绝不出现中英文错行问题
|
||||
|
||||
### ⚡ 真并发翻译
|
||||
- **asyncio.gather 并发执行**：高效利用 API 速率限制
|
||||
- **智能速率控制**：基于 Token 桶的 RateLimiter
|
||||
- **实时进度显示**：Rich 进度条显示翻译状态
|
||||
- **断点续传**：自动记录进度，随时中断随时继续
|
||||
|
||||
### 🛡️ 安全与稳定
|
||||
- **环境隔离**：支持 `.env` 配置，API Key 不落地
|
||||
- **鲁棒重试**：集成 `tenacity` 处理网络波动
|
||||
- **缓存系统**：基于 Hash 的持久化缓存，跨天复用
|
||||
|
||||
### 🎨 极致排版
|
||||
- **盘古之白**：自动在中文与西文数字间添加空格
|
||||
- **样式注入**：注入专用 CSS 优化阅读体验
|
||||
|
||||
## 🚀 快速开始
|
||||
|
||||
### 1. 安装依赖
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 2. 配置环境
|
||||
|
||||
复制 `.env` 模板并填入你的 API Key：
|
||||
|
||||
```bash
|
||||
# .env 文件
|
||||
V3_API_KEY=sk-xxxxxx
|
||||
OPENROUTER_API_KEY=sk-or-xxxxxx
|
||||
```
|
||||
|
||||
### 3. 开始翻译
|
||||
|
||||
```bash
|
||||
# 默认使用 OpenRouter
|
||||
python main.py input/book.epub
|
||||
|
||||
# 使用 V3 Provider
|
||||
python main.py input/book.epub -p v3
|
||||
|
||||
# 测试模式（只翻译前3个块）
|
||||
python main.py input/book.epub --test
|
||||
```
|
||||
|
||||
## 📂 目录结构
|
||||
|
||||
```
|
||||
.
|
||||
├── config/ # 配置文件
|
||||
│ ├── config.json # 主配置
|
||||
│ └── prompts.json # 提示词模板
|
||||
├── input/ # 输入 EPUB 目录
|
||||
├── output/ # 输出 EPUB 目录
|
||||
├── cache/ # 缓存目录 (Manifest, Translations)
|
||||
├── logs/ # 运行日志
|
||||
└── src/ # 源代码
|
||||
```
|
||||
|
||||
## ⚙️ 核心配置 (config.json)
|
||||
|
||||
```json
|
||||
{
|
||||
"translation": {
|
||||
"chunk_size": 5000,
|
||||
"temperature": 0.3
|
||||
},
|
||||
"providers": {
|
||||
"v3": {
|
||||
"base_url": "https://api.gpt.ge/v1",
|
||||
"models": { "fast": "gpt-4o-mini" },
|
||||
"rate_limits": { "requests_per_minute": 500 }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 📄 许可证
|
||||
|
||||
MIT License
|
||||
|
||||
---
|
||||
|
||||
**版本**: v0.07
|
||||
**更新**: 2026-01-13
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"translation": {
|
||||
"chunk_size": 5000,
|
||||
"temperature": 0.3,
|
||||
"glossary": {
|
||||
"enabled": true,
|
||||
"auto_generate": true,
|
||||
"sample_size": 3000
|
||||
}
|
||||
},
|
||||
"output": {
|
||||
"output_dir": "output",
|
||||
"filename_suffix": "_bilingual"
|
||||
},
|
||||
"logging": {
|
||||
"level": "INFO",
|
||||
"file": "logs/translator.log",
|
||||
"rotation": "10 MB",
|
||||
"retention": "7 days"
|
||||
},
|
||||
"providers": {
|
||||
"openrouter": {
|
||||
"base_url": "https://openrouter.ai/api/v1",
|
||||
"api_key": "YOUR_OPENROUTER_API_KEY",
|
||||
"models": {
|
||||
"fast": "google/gemini-2.0-flash-001",
|
||||
"smart": "google/gemini-2.0-flash-thinking-exp:free"
|
||||
},
|
||||
"extra_headers": {
|
||||
"HTTP-Referer": "https://github.com/epub-translator",
|
||||
"X-Title": "EPUB Translator"
|
||||
},
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 60,
|
||||
"concurrent_requests": 32
|
||||
}
|
||||
},
|
||||
"v3": {
|
||||
"base_url": "https://api.gpt.ge/v1",
|
||||
"api_key": "YOUR_V3_API_KEY",
|
||||
"models": {
|
||||
"fast": "gpt-4o-mini",
|
||||
"smart": "gpt-4o"
|
||||
},
|
||||
"extra_headers": {
|
||||
"x-foo": "true"
|
||||
},
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 500,
|
||||
"concurrent_requests": 50
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
{
|
||||
"translation": {
|
||||
"chunk_size": 5000,
|
||||
"temperature": 0.3,
|
||||
"glossary": {
|
||||
"enabled": true,
|
||||
"auto_generate": true,
|
||||
"sample_size": 3000
|
||||
}
|
||||
},
|
||||
"output": {
|
||||
"output_dir": "output",
|
||||
"filename_suffix": "_bilingual"
|
||||
},
|
||||
"logging": {
|
||||
"level": "INFO",
|
||||
"file": "logs/translator.log",
|
||||
"rotation": "10 MB",
|
||||
"retention": "7 days"
|
||||
},
|
||||
"providers": {
|
||||
"openrouter": {
|
||||
"base_url": "https://openrouter.ai/api/v1",
|
||||
"api_key": "YOUR_OPENROUTER_API_KEY",
|
||||
"models": {
|
||||
"fast": "google/gemini-2.0-flash-001",
|
||||
"smart": "google/gemini-2.0-flash-thinking-exp:free"
|
||||
},
|
||||
"extra_headers": {
|
||||
"HTTP-Referer": "https://github.com/epub-translator",
|
||||
"X-Title": "EPUB Translator"
|
||||
},
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 60,
|
||||
"concurrent_requests": 32
|
||||
}
|
||||
},
|
||||
"v3": {
|
||||
"base_url": "https://api.gpt.ge/v1",
|
||||
"api_key": "YOUR_V3_API_KEY",
|
||||
"models": {
|
||||
"fast": "gemini-3-flash-preview",
|
||||
"smart": "gemini-3-pro-preview"
|
||||
},
|
||||
"extra_headers": {
|
||||
"x-foo": "true"
|
||||
},
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 500,
|
||||
"concurrent_requests": 50
|
||||
}
|
||||
},
|
||||
"openai": {
|
||||
"base_url": "http://127.0.0.1:8045/v1",
|
||||
"api_key": "YOUR_ANTIGRAVITY_API_KEY",
|
||||
"models": {
|
||||
"fast": "gemini-3-flash",
|
||||
"smart": "gemini-3-pro-high"
|
||||
},
|
||||
"extra_headers": {
|
||||
"x-foo": "true"
|
||||
},
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 500,
|
||||
"concurrent_requests": 50
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"llm": {
|
||||
"provider": "openrouter",
|
||||
"base_url": "https://openrouter.ai/api/v1",
|
||||
"api_key": "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",
|
||||
"models": {
|
||||
"fast": "google/gemini-3-flash-preview",
|
||||
"smart": "google/gemini-3-pro-preview"
|
||||
},
|
||||
"arena_models": [
|
||||
"google/gemini-3-flash-preview",
|
||||
"openai/gpt-5-mini",
|
||||
"anthropic/claude-haiku-4.5"
|
||||
],
|
||||
"judge_model": "openai/gpt-5.1",
|
||||
"rate_limits": {
|
||||
"requests_per_minute": 60,
|
||||
"concurrent_requests": 32
|
||||
}
|
||||
},
|
||||
"translation": {
|
||||
"chunk_size": 5000,
|
||||
"temperature": 0.3,
|
||||
"strategy": "arena_winner"
|
||||
},
|
||||
"processing": {
|
||||
"min_paragraph_length": 5
|
||||
},
|
||||
"output": {
|
||||
"output_dir": "output",
|
||||
"filename_suffix": "_bilingual"
|
||||
},
|
||||
"logging": {
|
||||
"level": "INFO",
|
||||
"file": "logs/translator.log",
|
||||
"rotation": "10 MB",
|
||||
"retention": "7 days"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"translation": {
|
||||
"system": "你是一位精通中英文的专业翻译家。你的任务是翻译书籍内容。\n\n要求：\n1. 准确传达原文含义，语言流畅自然，符合中文阅读习惯。\n2. 严格保持【p_xxxxx】编号格式，不要遗漏，不要修改编号。\n3. 不要添加任何解释、注释或无关内容，只返回【编号】+【译文】。\n\n{{glossary_instruction}}",
|
||||
"user_template": "请翻译以下段落：\n\n{{content}}"
|
||||
},
|
||||
"glossary_extraction": {
|
||||
"system": "你是一位资深的文学编辑和领域专家。你的任务是分析书籍样本，提取关键术语并制定统一的译名表。",
|
||||
"user_template": "请阅读以下书籍片段（包含前言和正文采样）。\n\n任务：\n1. 识别文中出现的人名（如 'Masa', 'Steve Jobs'）、地名、机构名。\n2. 识别特定的行业术语或关键概念。\n3. 为上述词汇提供标准的中文译名。如果像 'Masa' 这样的昵称有对应的全名（如孙正义），请务必使用全名。\n\n请以 JSON 格式输出，格式如下：\n{\n \"Masa\": \"孙正义\",\n \"Apple\": \"苹果公司\",\n ...\n}\n\n书籍片段：\n\n{{content}}"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env python3
|
||||
import asyncio
|
||||
import sys
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from loguru import logger
|
||||
from src.translator import EPUBTranslator
|
||||
from src.utils import load_config, setup_logging
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description="EPUB 双语翻译工具")
|
||||
parser.add_argument("epub_path", help="输入 EPUB 文件路径")
|
||||
parser.add_argument("--provider", "-p", default="openrouter", help="LLM 供应商 (config.json 中 providers 的 key)")
|
||||
parser.add_argument("--mode", "-m", default="bilingual", choices=["bilingual", "chinese"],
|
||||
help="输出模式: bilingual (双语对照) 或 chinese (纯中文，保留格式)")
|
||||
parser.add_argument("--test", action="store_true", help="测试模式（仅翻译前几段）")
|
||||
parser.add_argument("--output", "-o", help="输出目录")
|
||||
parser.add_argument("--no-cache", action="store_true", help="禁用缓存（强制重新翻译）")
|
||||
parser.add_argument("--clear-cache", action="store_true", help="清理所有缓存文件")
|
||||
return parser.parse_args()
|
||||
|
||||
def flatten_provider_config(config: dict, provider_name: str) -> dict:
|
||||
"""
|
||||
将选定的 provider 配置扁平化到 config['llm'] 中，
|
||||
以便下游模块统一调用。
|
||||
"""
|
||||
providers = config.get('providers', {})
|
||||
if provider_name not in providers:
|
||||
available = list(providers.keys())
|
||||
logger.error(f"未找到供应商 '{provider_name}'。可用供应商: {available}")
|
||||
sys.exit(1)
|
||||
|
||||
selected_config = providers[provider_name]
|
||||
logger.info(f"使用 LLM 供应商: {provider_name} ({selected_config.get('base_url')})")
|
||||
|
||||
# 注入到 config['llm']
|
||||
config['llm'] = selected_config
|
||||
return config
|
||||
|
||||
async def run_translation(args):
|
||||
try:
|
||||
# 1. 加载配置
|
||||
config = load_config()
|
||||
|
||||
# 2. 处理 Provider 选择
|
||||
config = flatten_provider_config(config, args.provider)
|
||||
|
||||
# 3. 设置日志
|
||||
setup_logging(config)
|
||||
logger.info("程序启动")
|
||||
|
||||
# 4. 初始化翻译器
|
||||
translator = EPUBTranslator(config, use_cache=not args.no_cache)
|
||||
|
||||
# 5. 执行翻译
|
||||
await translator.translate_epub(
|
||||
args.epub_path,
|
||||
test_mode=args.test,
|
||||
output_dir=args.output,
|
||||
mode=args.mode
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
logger.error(f"翻译失败: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
|
||||
if args.clear_cache:
|
||||
import shutil
|
||||
cache_dir = Path("cache")
|
||||
if cache_dir.exists():
|
||||
shutil.rmtree(cache_dir)
|
||||
print("缓存已清理")
|
||||
sys.exit(0)
|
||||
|
||||
asyncio.run(run_translation(args))
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,34 @@
|
||||
[project]
|
||||
name = "epub-translator"
|
||||
version = "0.07"
|
||||
description = "EPUB双语翻译程序"
|
||||
authors = [
|
||||
{name = "Kaitan", email = "your-email@example.com"}
|
||||
]
|
||||
dependencies = [
|
||||
"ebooklib>=0.19",
|
||||
"beautifulsoup4>=4.12.0",
|
||||
"lxml>=4.9.0",
|
||||
"openai>=1.0.0",
|
||||
"aiohttp>=3.9.0",
|
||||
"pydantic>=2.0.0",
|
||||
"loguru>=0.7.0",
|
||||
"rich>=13.0.0",
|
||||
"asyncio-throttle>=1.0.2",
|
||||
"tenacity>=8.0.0",
|
||||
"python-dotenv>=1.0.0",
|
||||
]
|
||||
requires-python = ">=3.9"
|
||||
|
||||
[project.scripts]
|
||||
epub-translator = "main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.uv]
|
||||
dev-dependencies = [
|
||||
"pytest>=7.0.0",
|
||||
"pytest-asyncio>=0.21.0",
|
||||
]
|
||||
@@ -0,0 +1,12 @@
|
||||
ebooklib>=0.19
|
||||
beautifulsoup4>=4.12.0
|
||||
lxml>=4.9.0
|
||||
openai>=1.0.0
|
||||
aiohttp>=3.9.0
|
||||
pydantic>=2.0.0
|
||||
loguru>=0.7.0
|
||||
rich>=13.0.0
|
||||
asyncio-throttle>=1.0.2
|
||||
tenacity>=8.0.0
|
||||
python-dotenv>=1.0.0
|
||||
socksio>=1.0.0
|
||||
@@ -0,0 +1,32 @@
|
||||
"""
|
||||
EPUB 双语翻译程序
|
||||
主要功能模块的初始化文件
|
||||
"""
|
||||
|
||||
__version__ = "0.08"
|
||||
__author__ = "Kaitan"
|
||||
|
||||
from .epub_parser import EPUBParser
|
||||
from .translator import EPUBTranslator
|
||||
from .llm_client import LLMClient as OpenRouterClient # Keep alias for compatibility
|
||||
from .llm_client import LLMClient
|
||||
from .text_processor import TextProcessor
|
||||
from .bilingual_builder import BilingualEPUBBuilder
|
||||
from .chinese_builder import ChineseEPUBBuilder
|
||||
from .format_extractor import FormatExtractor
|
||||
from .format_restorer import FormatRestorer
|
||||
from .utils import load_config, setup_logging
|
||||
|
||||
__all__ = [
|
||||
"EPUBParser",
|
||||
"EPUBTranslator",
|
||||
"LLMClient",
|
||||
"OpenRouterClient",
|
||||
"TextProcessor",
|
||||
"BilingualEPUBBuilder",
|
||||
"ChineseEPUBBuilder",
|
||||
"FormatExtractor",
|
||||
"FormatRestorer",
|
||||
"load_config",
|
||||
"setup_logging"
|
||||
]
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user