feat: Release v0.10 - Modular Architecture & External Config

- Refactor codebase into src/ (preprocessing, translation, assembly) - Add pipeline/ scripts for individual stages - Externalize configuration to config/config.yaml - Fix Cover Image preservation - Update documentation and manuals
2026-01-31 22:49:44 +08:00
parent 9ef82393be
commit 7a93c52b42
306 changed files with 30313 additions and 1071 deletions
@@ -0,0 +1,113 @@
+# Operation Manual & Change Log
+
+## Core Principles
+
+1.  **Modularity**: The system is divided into three distinct phases (Preprocessing, Translation, Assembly) with clear boundaries.
+2.  **Immutability**: `book_structure.json` is generated once during preprocessing and should not be modified by subsequent steps.
+3.  **Source of Truth**: `manifest.json` is the single source of truth for translations.
+4.  **Idempotency**: Translation steps can be retried without side effects (existing translations are preserved).
+
+## Directory Structure
+
+*   `pipeline/`: Executable scripts for each stage.
+    *   `01_preprocess.py`: Clean EPUB, generate structure, extract text.
+    *   `02_translate.py`: Translate text in manifest.
+    *   `03_assemble.py`: Apply translations and build final EPUB.
+*   `src/`: Core logic modules.
+    *   `preprocessing/`: Cleaning, extraction, profiling.
+    *   `translation/`: LLM integration, manifest management.
+    *   `assembly/`: Backfilling, EPUB building.
+    *   `common/`: Shared data models and utils.
+*   `work/`: Working directory for intermediate files (ignored by git).
+
+## Pipeline Usage
+
+### Step 1: Preprocessing
+```bash
+python pipeline/01_preprocess.py inputs/my_book.epub
+```
+Generates `work/my_book/book_structure.json` and `manifest.json`.
+
+### Step 2: Translation
+```bash
+python pipeline/02_translate.py --input-epub inputs/my_book.epub
+```
+Translates entries in `manifest.json`. ensuring `.env` has `OPENAI_API_KEY`.
+
+### Step 3: Assembly
+### Step 3: Assembly
+```bash
+# Bilingual Output (Default: output/bilingual_my_book.epub)
+python pipeline/03_assemble.py inputs/my_book.epub --bilingual
+
+# Target Language Output (Default: output/translated_my_book.epub)
+python pipeline/03_assemble.py inputs/my_book.epub
+```
+**Note**: The Assembly step now includes an **LLM-based Placeholder Repair** mechanism. If `format_restorer` detects broken placeholders in the translation, it will query the LLM (using your configured credentials) to attempt an automatic fix. Ensure your `OPENAI_API_KEY` is set if you want this feature enabled.
+
+## Configuration
+
+System settings are managed via `config/config.yaml` and environment variables.
+
+### `config/config.yaml`
+Control LLM parameters and translation behavior:
+```yaml
+llm:
+  model: "gpt-3.5-turbo"           # LLM Model Name
+  base_url: "https://api.openai.com/v1" 
+  timeout: 60
+  requests_per_minute: 60          # Rate limiting
+  concurrent_requests: 5           # Parallel chunks
+
+translation:
+  chunk_size: 4000                 # Characters per chunk
+```
+
+### Environment Variables (`.env`)
+Security-sensitive credentials must be set here:
+```bash
+OPENAI_API_KEY=sk-...              # Required
+OPENAI_BASE_URL=...                # Optional override for config
+```
+
+## Known Issues & Troubleshooting
+
+### AuthenticationError (OpenRouter etc.)
+If you see `AuthenticationError` despite having the correct `base_url` in config:
+1.  Check if you have a stale `OPENAI_API_KEY` in your shell environment.
+2.  Environment variables **override** `.env` files.
+3.  Fix: Run `unset OPENAI_API_KEY` (and `OPENAI_BASE_URL`) before running the script.
+
+### Missing/Unknown Placeholders
+*   **Logs**: `WARNING - Restoration warning: missing placeholders...`
+*   **Cause**: The LLM translation didn't preserve the exact `φXφ` tags.
+*   **Fix**: 
+    1.  The system will now attempt to **auto-repair** using the LLM during Assembly.
+    2.  If that fails, check logs. In some cases (e.g., complex HTML entities like `&amp;`), the extractor might have degraded to plain text.
+    3.  (Fixed in v0.11) Enhanced `FormatExtractor` now handles HTML entities correctly, preventing phantom placeholder hallucinations.
+
+## Change Log
+
+### [2026-01-31] Robustness & Repair
+*   **Feature**: Added **LLM-based Placeholder Repair** in Assembly stage. If placeholders mismatch, the system asks the LLM to fix the tags without changing text.
+*   **Fix**: Solved `FormatExtractor` "phantom placeholders" issue by correctly unescaping HTML entities during integrity checks.
+*   **Fix**: Resolved **Duplicate ID** issue in LLM response parsing. Now recursively strips repeated headers (e.g., `#12: #12: ...`) to prevent them from leaking into the translation.
+*   **Tweak**: Updated `pipeline/03_assemble.py` to be async and load LLM config.
+
+### [2026-01-30] Performance Improvements
+*   **Concurrency Fix**: Resolved issue where `concurrent_requests` in `config.yaml` was ignored by the Translator engine. Now `main.py` and `pipeline/02_translate.py` correctly propagate this setting, allowing faster translation with higher limits (e.g., for local LLMs or high-rate-limit providers).
+
+### [2026-01-28] Bug Fixes
+*   **Fix Cover Image**: Resolved issue where book cover execution was missing in the final EPUB. Added `cover_image_id` tracking in `BookStructure` and restored proper OPF metadata in `BilingualBuilder`.
+
+### [2026-01-27] Externalized Configuration
+*   **Config**: Added `config/config.yaml` for tuning parameters (LLM model, RPM, Chunk Size).
+*   **Logic**: `pipeline/02_translate.py` now loads settings from `config.yaml`.
+*   **Dependency**: Added `PyYAML` to `requirements.txt`.
+
+### [2026-01-27] Architecture Refactoring
+*   **Restructured**: Moved source files into `src/preprocessing`, `src/translation`, `src/assembly`, `src/common`.
+*   **Pipeline**: Created individual pipeline scripts in `pipeline/`.
+*   **Refactor**: Renamed `fine_grained_extractor` to `text_extractor`, `translator` to `translator_engine`, etc.
+*   **Logic**: Enforced 100% text coverage check in `format_extractor.py` (removed 95% threshold).
+*   **Docs**: Created this Operation Manual.
@@ -0,0 +1,100 @@
+# ePub Bilingual Translator - Architecture & Data Flow (Revised)
+
+本文档详细描述了程序处理一个 EPUB 文件的完整生命周期。整个流程旨在实现**结构稳定性**（不丢段落）与**内容精细度**（不丢格式）的最佳平衡。
+
+## High Level Data Flow
+
+```mermaid
+graph TD
+    Input[Input EPUB] --> Cleaner[EpubCleaner]
+    Cleaner --> CleanedEPUB[1. Cleaned EPUB Temp]
+    
+    CleanedEPUB --> Profiler[Book Profiler]
+    Profiler -->|Identify| Profile[Book Profile / Style]
+    
+    CleanedEPUB --> Extractor[FineGrainedExtractor]
+    
+    subgraph Extraction
+        Extractor -->|Step 1: P/H Tags| P[Source Paragraph]
+        P -->|Step 2: FormatExtract| PhText[Analyzed Text]
+        PhText -->|Register| Manifest[Manifest DB]
+    end
+    
+    Manifest -->|Batch| LLM[LLM Translation]
+    Profile -->|Prompt Context| LLM
+    
+    LLM -->|Translation| Manifest
+    
+    Manifest --> Restorer[FormatRestorer]
+    Restorer -->|Reconstruct HTML| TargetHtml[Target HTML]
+    
+    CleanedEPUB --> Backfiller[Backfill Engine]
+    TargetHtml --> Backfiller
+    
+    Backfiller -->|Bilingual/Chinese Mode| DOM[Final DOM]
+    DOM --> Builder[BilingualBuilder]
+    Builder --> Output[Output EPUB]
+```
+
+## Detailed Workflow
+
+### 1. Preprocessing (预处理)
+**模块**: `src/epub_cleaner.py`
+*   **Flatten Structure**: 消除嵌套 `div`，统一转为 `<p>`，消除结构性漏译风险。
+*   **Auto-Fix**: 修复 TOC 死链、缺失 UID、由于 `ebooklib` bug 导致的样式丢失。
+*   **Result**: 产生一个标准的临时文件，后续所有操作基于此文件，不再受原始糟糕格式影响。
+
+### 2. Intelligent Extraction (智能提取)
+**模块**: `src/fine_grained_extractor.py` + `src/format_extractor.py`
+
+#### A. 结构层 (Macro)
+使用 `FineGrainedExtractor` 锁定所有正文元素 (`p`, `h1`-`h6`)。
+*   **Filter**: 排除页码、页眉脚。
+*   **Optimization**: 针对目录章节，识别 **罗马数字 (I, II)**、**单独数字 (1, 2)**、**修饰符 (***)**，这些内容**不送翻译**，直接在回填时保留原文，以维持原书排版美感。
+
+#### B. 内容层 (Micro)
+对每个提取的段落调用 `FormatExtractor`：
+*   **Inline Style**: 将 `<b>`, `<i>` 转为配对占位符 `φ1φ...φ/1φ`。
+*   **Formula Protection**: 识别 $E=mc^2$ 等数学公式，保护为不可变占位符。
+*   **Drop Cap Handling**: 
+    - 原始: `<span class="dropcap">T</span>he`
+    - 提取给 LLM: "The" (完整单词，无格式干扰)
+    - 记录: Prefix 包含 Drop Cap 样式。
+
+### 3. Manifest Management (清单管理)
+**模块**: `src/manifest_manager.py`
+Manifest 是系统的**核心状态中心 (Source of Truth)**。
+*   **作用**: 解耦提取和翻译。提取器只管往 Manifest 填数据，翻译器只管从 Manifest 取数据。
+*   **Persistence**: 支持中断续传，翻译进度实时保存。
+
+### 4. Translation with Profiling (翻译)
+**模块**: `src/book_profiler.py` & `src/translator.py`
+*   **Profiling**: 在翻译前，抽取部分文本分析书籍的类型（技术、小说、诗歌）、核心术语和语言风格，生成 `System Prompt`。
+*   **Translation**: 这是纯文本层面的转换，LLM 处理的是带有 `φ` 占位符的文本。
+
+### 5. Robust Restoration (健壮还原)
+**模块**: `src/format_restorer.py`
+负责将 LLM 返回的文本还原为 HTML。
+*   **Drop Cap Logic**: 
+    - **原文回填**: 需要 Prefix `<span class="dropcap">T</span>`。
+    - **译文回填**: **丢弃** Drop Cap Prefix。中文不需要首字母下沉，否则会出现 "T这本书..." 的怪诞结果。
+*   **Error Handling**:
+    - **Missing Placeholders**: 如果 LLM 丢了 `φ1φ`，自动在末尾补全或报错重试。
+    - **Hallucinated Placeholders**: 移除 LLM 臆造的不存在 ID。
+
+### 6. Backfill Strategy (回填策略)
+**模块**: `src/fine_grained_extractor.py` (backfill method)
+支持多种模式，且**严格遵循一对一 (One-to-One) 映射**，绝不依赖顺序，而是依赖元素的内存引用或唯一 ID。
+
+*   **Mode A: Bilingual (双语)**
+    - 保留原文 DOM。
+    - 在原文后 `append` 一个新元素 `<p class="translation">译文</p>`。
+    - 样式继承：译文元素复制原文的 `margin`, `text-align` 等关键样式。
+
+*   **Mode B: Chinese Only (仅中文)**
+    - **Replace**: 直接用译文元素替换原文元素。
+    - **Drop Cap 适配**: 此时译文通常为普通段落，不再保留首字母下沉样式，以符合中文排版习惯。
+
+### 7. Assembly (组装)
+**模块**: `src/bilingual_builder.py`
+将内存中的 DOM 序列化，重新打包资源，生成最终 EPUB。
@@ -0,0 +1,180 @@
+# 预处理与回填层技术文档
+
+本文档详细描述 EPUB 双语翻译器的预处理（Preprocessing）和回填（Backfill）层的技术方案、数据结构、问题解决方案，用于指导代码开发、调试和维护。
+
+---
+
+## 1. 架构概览
+
+```
+┌─────────────┐     ┌────────────────┐     ┌───────────────────┐
+│ EPUB 文件   │ ──► │ EpubCleaner    │ ──► │ book_structure.json│
+└─────────────┘     └────────────────┘     └───────────────────┘
+                           │
+                           ▼
+                    ┌────────────────┐     ┌───────────────────┐
+                    │FineGrainedExt. │ ──► │  manifest.json    │
+                    └────────────────┘     └───────────────────┘
+                                                   │
+                    ┌────────────────┐             │
+                    │ BackfillEngine │ ◄───────────┘
+                    └────────────────┘
+                           │
+                           ▼
+                    ┌────────────────┐     ┌───────────────────┐
+                    │BilingualBuilder│ ──► │  输出 EPUB        │
+                    └────────────────┘     └───────────────────┘
+```
+
+---
+
+## 2. 目录结构
+
+```
+work/
+  ├── {book_name}/              # 每本书独立的工作目录
+  │   ├── book_structure.json   # 书籍结构（会被复用）
+  │   ├── manifest.json         # 翻译清单（持久化，Source of Truth）
+  │   └── assets/               # 解压出的 EPUB 资源文件
+  │       └── OEBPS/images/...  # 保留原始路径结构
+  └── translations/             # (可选) 翻译记忆或其他中间文件
+```
+
+**关键设计**：`book_structure.json` **默认复用**。为避免 UUID 不一致导致回填失败，除非显式指定强制清理，否则程序优先读取现有的结构文件。
+
+---
+
+## 3. 数据结构定义
+
+### 3.1 BookStructure (book_structure.json)
+
+记录书籍的“骨架”，确保翻译后的章节能按正确顺序和层级重组。
+
+```json
+{
+  "metadata": {
+    "title": "书名",
+    "author": "作者",
+    "language": "en",
+    "identifier": "ISBN或UUID"
+  },
+  "spine": ["item_id_1", "item_id_2", ...],  // 阅读顺序
+  "resources": {
+    "item_id_1": {
+      "href": "OEBPS/chapter1.xhtml",
+      "media_type": "application/xhtml+xml",
+      "content": "<html>...</html>",  // 清理并注入ID后的HTML内容
+      "properties": "nav"
+    },
+    ...
+  }
+}
+```
+
+### 3.2 ManifestEntry (manifest.json)
+
+翻译清单是全生命周期的核心，记录了所有待翻译段落的状态。
+
+```json
+[
+  {
+    "entry_id": "OEBPS/c3Z.xhtml#uuid-603ff3fb",
+    "file_path": "OEBPS/c3Z.xhtml",
+    "element_id": "uuid-603ff3fb",
+    "original_text": "φ1φHelloφ/1φ world.",
+    "placeholders": {
+      "1": "<b>",
+      "/1": "</b>",
+      "_prefix": "<p>",
+      "_suffix": "</p>"
+    },
+    "translated_text": "φ1φ你好φ/1φ 世界。",
+    "context": "body"
+  }
+]
+```
+
+| 字段 | 类型 | 说明 |
+|------|------|------|
+| `entry_id` | str | **全局唯一ID** (`file_path#element_id`)，用于精准追踪。 |
+| `file_path` | str | 标识该段落属于哪一章，用于按章分组批处理。 |
+| `element_id` | str | HTML DOM 元素的 ID，回填时的锚点。 |
+| `original_text` | str | 经过智能抽提和占位符化后的文本，发送给 LLM。 |
+| `placeholders` | Dict | 格式映射表，用于还原 HTML 结构。 |
+| `translated_text`| str | 翻译结果（含占位符），初始为 null。 |
+
+---
+
+## 4. 预处理与抽提流程 (Extraction)
+
+流程由 `FormatExtractor` 驱动，分为三个阶段：
+
+### Step 1: 结构索引 (Structure Indexing)
+*   **动作**: 解析 OPF 文件，提取 Spine 和 Metadata。
+*   **目的**: 建立骨架，确定处理顺序。
+
+### Step 2: 语义识别 (Semantic Detection)
+*   **动作**: 使用 `HeadingDetector` 分析 DOM 节点。
+*   **逻辑**:
+    *   匹配 `h1`-`h6` 正则，区分 `Chapter`（章）与 `Section`（节）。
+    *   识别 `blockquote` 或特定 class 判定 `Epigraph`（引言）。
+*   **目的**: 为 LLM 提供差异化的 Prompt（例如翻译标题时不要加句号）。
+
+### Step 3: 智能抽提 (Smart Extraction - `_smart_extract_v3`)
+这是核心算法，将 HTML 转换为“纯文本+占位符”。
+
+1.  **首尾分离 (Prefix/Suffix Separation)**:
+    *   将包裹文本的外层标签（如 `<p>`, `div`）剥离到 `_prefix` 和 `_suffix`。
+    *   **目的**: 极大减少 LLM 输入 Token，且防止 LLM 随意修改外层布局。
+
+2.  **首字下沉处理 (Drop Cap Handling)**:
+    *   检测并合并被 `<span>` 单独包裹的首字母（如 `<span class="drop">O</span>` + `nce` → `Once`）。
+    *   **目的**: 修复语意割裂，让 LLM 看到完整的单词。
+
+3.  **占位符化 (Placeholder Mapping)**:
+    *   将内联标签（`<a>`, `<em>`）或公式替换为短码 `φIDφ`。
+    *   **目的**: 保护 HTML 属性不被“翻译”，降低噪声干扰。
+
+4.  **完整性校验 (Integrity Check)**:
+    *   **逻辑**: 抽提后的文本（去占位符）与原始纯文本进行归一化比对，要求覆盖率 **100%**。
+    *   **兜底**: 若校验失败（如误删内容），回退到简单模式（只剥离首尾标签）。
+
+---
+
+## 5. 构建与回填流程 (Backfill & Build)
+
+### 5.1 翻译回填
+*   根据 `entry_id` 找到对应的 DOM 节点。
+*   使用 `FormatRestorer` 将 `translated_text` 中的占位符（`φ1φ`）还原为原始 HTML 标签（`<b>`）。
+*   根据模式（双语/单语）决定将新节点插入到原文后还是替换原文。
+
+### 5.2 链接修复 (Link Repair)
+在 `BilingualBuilder` 中执行：
+1.  **ID 补全**: 为缺失 ID 的 TOC 节点自动生成 UUID。
+2.  **死链检测**: 检查 TOC/Nav 指向的文件是否存在。
+3.  **模糊修复**: 尝试通过文件名后缀匹配（解决路径前缀变更问题）或特定重定向（如 `c0.xhtml` -> `cover.xhtml`）。
+4.  **坏死剔除**: 无法修复的死链将从目录中移除。
+
+### 5.3 CSS 样式恢复
+`EbookLib` 默认可能会重写 `<head>` 导致样式丢失。
+*   **逻辑**:
+    *   收集所有 CSS 资源。
+    *   在构建每个 HTML Item 时，显式计算 HTML 到 CSS 的**相对路径**。
+    *   强制调用 `item.add_link(..., rel='stylesheet', type='text/css')` 注入引用。
+
+---
+
+## 6. 常见问题排查
+
+### 6.1 UUID 不匹配
+**现象**: `WARNING - Element uuid-xxx not found`.
+**原因**: 手动删除了 `book_structure.json` 但保留了 `manifest.json`，导致重新生成的 HTML ID 与清单记录不一致。
+**解决**: 清空 `work/BookName` 目录重新运行，或确保两个 JSON 文件版本一致。
+
+### 6.2 样式丢失
+**现象**: 打开书面目全非，只有黑白文字。
+**检查**: 解压 EPUB，查看 HTML `<head>` 是否有 `<link rel="stylesheet">`。如果没有，检查 `BilingualBuilder` 的 Step 4 逻辑。
+
+### 6.3 翻译错位
+**现象**: 译文出现在了错误的位置。
+**检查**: 确认 `entry_id` 生成逻辑是否包含文件名，且文件名在处理过程中未被意外修改。
@@ -0,0 +1,108 @@
+# Translation Layer Technical Documentation
+
+This document details the architecture of the **Translation Layer**, which operates primarily on `manifest.json`.
+
+> **Core Principle**: The Translation Layer is **decoupled** from the EPUB file format. It reads translatable units from `manifest.json`, processes them using an LLM, and writes translations back to `manifest.json`. It does **not** read or parse the EPUB file directly.
+
+---
+
+## 1. Architecture Overview
+
+### Data Flow
+
+```mermaid
+graph LR
+    A[manifest.json] -->|Load| B(ManifestManager)
+    B -->|Entries| C{Translator Orchestrator}
+    C -->|Sample Text| D[BookProfiler]
+    D -->|Style Guide| C
+    C -->|Chunks| E[LLMClient]
+    E -->|Translation| C
+    C -->|Update| B
+    B -->|Save| A[manifest.json]
+```
+
+1.  **Input**: `manifest.json` (Generated by Preprocessing Layer).
+2.  **Process**:
+    *   **Profiling**: Analyze text samples to generate a `BookProfile` (style, tone, terminology).
+    *   **Translation**: Batch entries into chunks, send to LLM, receive translations.
+3.  **Output**: `manifest.json` (Updated with `translated_text` fields).
+
+---
+
+## 2. Core Modules (`src/translation/`)
+
+### 2.1 ManifestManager (`manifest_manager.py`)
+*   **Role**: The interface for the "Source of Truth".
+*   **Responsibility**:
+    *   Load `manifest.json`.
+    *   Provide list of `ManifestEntry` objects.
+    *   Save updates back to disk.
+*   **Key Method**: `update_translation(entry_id, translation)`
+
+### 2.2 Translator Engine (`translator_engine.py`)
+*   **Role**: Orchestrates the translation process.
+*   **Responsibility**:
+    *   **Filtering**: Identify untranslated entries.
+    *   **Grouping**: Group entries by chapter (file path) to maintain context.
+    *   **Chunking**: Create character-based chunks (default ~5000 chars) that do not cross chapter boundaries.
+    *   **Concurrency**: Manage async workers (default 5 concurrent tasks).
+*   **Input**: `List[ManifestEntry]`, `BookProfile`.
+*   **Output**: Updates `ManifestEntry` objects in-place.
+
+### 2.3 LLM Client (`llm_client.py`)
+*   **Role**: Handles raw communication with the LLM Provider (OpenAI compatible).
+*   **Responsibility**:
+    *   **Prompt Engineering**: Construct Short-ID based prompts.
+    *   **Rate Limiting**: Control RPM (Requests Per Minute).
+    *   **Retry Logic**: Exponential backoff for API failures.
+    *   **Logging**: Save raw request/response pairs to `work/{book}/chunks/` for debugging.
+
+### 2.4 Book Profiler (`../preprocessing/profiler.py`)
+*   **Note**: While located in preprocessing, it is often invoked at the start of the translation phase.
+*   **Role**: Generates a style guide.
+*   **Mechanism**: Extracts a sample (Intro + Random segments) from `manifest.json` entries and asks the LLM to analyze author style.
+
+---
+
+## 3. Short ID Strategy
+
+To optimize token usage and ensuring mapping accuracy, we use a **Short ID** system for LLM interaction.
+
+**Prompt Format**:
+```text
+#1: First paragraph text...
+#2: Second paragraph text...
+```
+
+**Response Format**:
+```text
+#1: 第一段翻译...
+#2: 第二段翻译...
+```
+
+The `LLMClient` maintains a mapping of `Short ID (#N)` <-> `Entry ID (File#UUID)` for each chunk lifecycle.
+
+---
+
+## 4. Pipeline Usage
+
+The translation is executed via the standalone pipeline script:
+
+```bash
+python pipeline/02_translate.py --input-epub inputs/my_book.epub
+```
+*   **--input-epub**: Uses the filename to locate the `work/` directory.
+*   **--book-name**: Alternatively, specify the book folder name directly.
+
+### Dependencies
+*   Environment variables must be set in `.env`:
+    *   `OPENAI_API_KEY`
+    *   `OPENAI_BASE_URL` (Optional)
+
+---
+
+## 5. Development & Debugging
+
+*   **Chunk Logs**: Check `.work/{book}/chunks/` to see exactly what was sent to and received from the LLM.
+*   **Idempotency**: The translation script skips entries that already have `translated_text`. To re-translate, you must manually clear `translated_text` in `manifest.json` or delete the manifest (to restart from preprocessing).