Feat: v0.12 Pipeline Separation, Spacing Fix, and Idempotent Restoration
This commit is contained in:
谭凯
@@ -0,0 +1,131 @@
|
||||
# Operation Manual & Change Log
|
||||
|
||||
## Core Principles
|
||||
|
||||
1. **Modularity**: The system is divided into three distinct phases (Preprocessing, Translation, Assembly) with clear boundaries.
|
||||
2. **Immutability**: `book_structure.json` is generated once during preprocessing and should not be modified by subsequent steps.
|
||||
3. **Source of Truth**: `manifest.json` is the single source of truth for translations.
|
||||
4. **Idempotency**: Translation steps can be retried without side effects (existing translations are preserved).
|
||||
|
||||
## Directory Structure
|
||||
|
||||
* `pipeline/`: Executable scripts for each stage.
|
||||
* `01_preprocess.py`: Clean EPUB, generate structure, extract text.
|
||||
* `02_translate.py`: Translate text in manifest.
|
||||
* `03_assemble.py`: Apply translations and build final EPUB.
|
||||
* `src/`: Core logic modules.
|
||||
* `preprocessing/`: Cleaning, extraction, profiling.
|
||||
* `translation/`: LLM integration, manifest management.
|
||||
* `assembly/`: Backfilling, EPUB building.
|
||||
* `common/`: Shared data models and utils.
|
||||
* `work/`: Working directory for intermediate files (ignored by git).
|
||||
|
||||
## Pipeline Usage
|
||||
|
||||
### Step 1: Preprocessing
|
||||
```bash
|
||||
python pipeline/01_preprocess.py inputs/my_book.epub
|
||||
```
|
||||
Generates `work/my_book/book_structure.json` and `manifest.json`.
|
||||
|
||||
### Step 2: Translation
|
||||
```bash
|
||||
python pipeline/02_translate.py --input-epub inputs/my_book.epub
|
||||
```
|
||||
Translates entries in `manifest.json`. ensuring `.env` has `OPENAI_API_KEY`.
|
||||
|
||||
### Step 3: Restore Format (NEW)
|
||||
```bash
|
||||
python pipeline/03_restore_format.py inputs/my_book.epub
|
||||
```
|
||||
**Optimizes and validates translations**:
|
||||
1. Applies "Pangu" spacing (inserts space between Chinese and English/Numbers).
|
||||
2. Restores HTML tags using placeholders.
|
||||
3. Attempts **LLM Auto-Repair** if validation fails.
|
||||
4. Saves result to `manifest.json` (`translated_html` field).
|
||||
|
||||
### Step 4: Build EPUB
|
||||
```bash
|
||||
# Bilingual Output (Default)
|
||||
python pipeline/04_build_epub.py inputs/my_book.epub --bilingual
|
||||
|
||||
# Target Language Output
|
||||
python pipeline/04_build_epub.py inputs/my_book.epub
|
||||
```
|
||||
**Pure assembly**: Injects the pre-validated `translated_html` into the EPUB structure. Fast and deterministic.
|
||||
|
||||
## Configuration
|
||||
|
||||
System settings are managed via `config/config.yaml` and environment variables.
|
||||
|
||||
### `config/config.yaml`
|
||||
Control LLM parameters and translation behavior:
|
||||
```yaml
|
||||
llm:
|
||||
model: "gpt-3.5-turbo" # LLM Model Name
|
||||
base_url: "https://api.openai.com/v1"
|
||||
timeout: 60
|
||||
requests_per_minute: 60 # Rate limiting
|
||||
concurrent_requests: 5 # Parallel chunks
|
||||
|
||||
translation:
|
||||
chunk_size: 4000 # Characters per chunk
|
||||
```
|
||||
|
||||
### Environment Variables (`.env`)
|
||||
Security-sensitive credentials must be set here:
|
||||
```bash
|
||||
OPENAI_API_KEY=sk-... # Required
|
||||
OPENAI_BASE_URL=... # Optional override for config
|
||||
```
|
||||
|
||||
## Known Issues & Troubleshooting
|
||||
|
||||
### AuthenticationError (OpenRouter etc.)
|
||||
If you see `AuthenticationError` despite having the correct `base_url` in config:
|
||||
1. Check if you have a stale `OPENAI_API_KEY` in your shell environment.
|
||||
2. Environment variables **override** `.env` files.
|
||||
3. Fix: Run `unset OPENAI_API_KEY` (and `OPENAI_BASE_URL`) before running the script.
|
||||
|
||||
### Missing/Unknown Placeholders
|
||||
* **Logs**: `WARNING - Restoration warning: missing placeholders...`
|
||||
* **Cause**: The LLM translation didn't preserve the exact `φXφ` tags.
|
||||
* **Fix**:
|
||||
1. The system will now attempt to **auto-repair** using the LLM during Assembly.
|
||||
2. If that fails, check logs. In some cases (e.g., complex HTML entities like `&`), the extractor might have degraded to plain text.
|
||||
3. (Fixed in v0.11) Enhanced `FormatExtractor` now handles HTML entities correctly, preventing phantom placeholder hallucinations.
|
||||
|
||||
## Change Log
|
||||
|
||||
### [2026-02-01] Pipeline Separation & Formatting
|
||||
* **Architecture**: Decoupled "Restoration" from "Assembly" into a 4-step pipeline.
|
||||
* **New Step 3**: `03_restore_format.py` handles formatting, spacing, and repair. Saves to `translated_html`.
|
||||
* **New Step 4**: `04_build_epub.py` handles pure EPUB generation.
|
||||
* **Data Model**: Added `translated_html` to `ManifestEntry` as the "Gold Master" formatted content.
|
||||
* **UX**: Added **Pangu Spacing** (Auto-spacing between CJK and ASCII) in Restoration step.
|
||||
* **Optimization**: `RestorationEngine` is now idempotent (skips processing if `translated_html` exists). Added `--force-restore` flag.
|
||||
* **Fix**: `main.py` updated to orchestrate the new 4-stage pipeline.
|
||||
|
||||
### [2026-01-31] Robustness & Repair
|
||||
* **Feature**: Added **LLM-based Placeholder Repair** in Assembly stage. If placeholders mismatch, the system asks the LLM to fix the tags without changing text.
|
||||
* **Fix**: Solved `FormatExtractor` "phantom placeholders" issue by correctly unescaping HTML entities during integrity checks.
|
||||
* **Fix**: Resolved **Duplicate ID** issue in LLM response parsing. Now recursively strips repeated headers (e.g., `#12: #12: ...`) to prevent them from leaking into the translation.
|
||||
* **Tweak**: Updated `pipeline/03_assemble.py` to be async and load LLM config.
|
||||
|
||||
### [2026-01-30] Performance Improvements
|
||||
* **Concurrency Fix**: Resolved issue where `concurrent_requests` in `config.yaml` was ignored by the Translator engine. Now `main.py` and `pipeline/02_translate.py` correctly propagate this setting, allowing faster translation with higher limits (e.g., for local LLMs or high-rate-limit providers).
|
||||
|
||||
### [2026-01-28] Bug Fixes
|
||||
* **Fix Cover Image**: Resolved issue where book cover execution was missing in the final EPUB. Added `cover_image_id` tracking in `BookStructure` and restored proper OPF metadata in `BilingualBuilder`.
|
||||
|
||||
### [2026-01-27] Externalized Configuration
|
||||
* **Config**: Added `config/config.yaml` for tuning parameters (LLM model, RPM, Chunk Size).
|
||||
* **Logic**: `pipeline/02_translate.py` now loads settings from `config.yaml`.
|
||||
* **Dependency**: Added `PyYAML` to `requirements.txt`.
|
||||
|
||||
### [2026-01-27] Architecture Refactoring
|
||||
* **Restructured**: Moved source files into `src/preprocessing`, `src/translation`, `src/assembly`, `src/common`.
|
||||
* **Pipeline**: Created individual pipeline scripts in `pipeline/`.
|
||||
* **Refactor**: Renamed `fine_grained_extractor` to `text_extractor`, `translator` to `translator_engine`, etc.
|
||||
* **Logic**: Enforced 100% text coverage check in `format_extractor.py` (removed 95% threshold).
|
||||
* **Docs**: Created this Operation Manual.
|
||||
Reference in New Issue
Block a user