# Operation Manual & Change Log

## Core Principles

1.  **Modularity**: The system is divided into three distinct phases (Preprocessing, Translation, Assembly) with clear boundaries.
2.  **Immutability**: `book_structure.json` is generated once during preprocessing and should not be modified by subsequent steps.
3.  **Source of Truth**: `manifest.json` is the single source of truth for translations.
4.  **Idempotency**: Translation steps can be retried without side effects (existing translations are preserved).

## Directory Structure

*   `pipeline/`: Executable scripts for each stage.
    *   `01_preprocess.py`: Clean EPUB, generate structure, extract text.
    *   `02_translate.py`: Translate text in manifest.
    *   `03_assemble.py`: Apply translations and build final EPUB.
*   `src/`: Core logic modules.
    *   `preprocessing/`: Cleaning, extraction, profiling.
    *   `translation/`: LLM integration, manifest management.
    *   `assembly/`: Backfilling, EPUB building.
    *   `common/`: Shared data models and utils.
*   `work/`: Working directory for intermediate files (ignored by git).

## Pipeline Usage

### Step 1: Preprocessing
```bash
python pipeline/01_preprocess.py inputs/my_book.epub
```
Generates `work/my_book/book_structure.json` and `manifest.json`.

### Step 2: Translation
```bash
python pipeline/02_translate.py --input-epub inputs/my_book.epub
```
Translates entries in `manifest.json`. ensuring `.env` has `OPENAI_API_KEY`.

### Step 3: Restore Format (NEW)
```bash
python pipeline/03_restore_format.py inputs/my_book.epub
```
**Optimizes and validates translations**:
1.  Applies "Pangu" spacing (inserts space between Chinese and English/Numbers).
2.  Restores HTML tags using placeholders.
3.  Attempts **LLM Auto-Repair** if validation fails.
4.  Saves result to `manifest.json` (`translated_html` field).

### Step 4: Build EPUB
```bash
# Bilingual Output (Default)
python pipeline/04_build_epub.py inputs/my_book.epub --bilingual

# Target Language Output
python pipeline/04_build_epub.py inputs/my_book.epub
```
**Pure assembly**: Injects the pre-validated `translated_html` into the EPUB structure. Fast and deterministic.

## Configuration

System settings are managed via `config/config.yaml` and environment variables.

### `config/config.yaml`
Control LLM parameters and translation behavior:
```yaml
llm:
  model: "gpt-3.5-turbo"           # LLM Model Name
  base_url: "https://api.openai.com/v1" 
  timeout: 60
  requests_per_minute: 60          # Rate limiting
  concurrent_requests: 5           # Parallel chunks

translation:
  chunk_size: 4000                 # Characters per chunk
```

### Environment Variables (`.env`)
Security-sensitive credentials must be set here:
```bash
OPENAI_API_KEY=sk-...              # Required
OPENAI_BASE_URL=...                # Optional override for config
```

## Known Issues & Troubleshooting

### AuthenticationError (OpenRouter etc.)
If you see `AuthenticationError` despite having the correct `base_url` in config:
1.  Check if you have a stale `OPENAI_API_KEY` in your shell environment.
2.  Environment variables **override** `.env` files.
3.  Fix: Run `unset OPENAI_API_KEY` (and `OPENAI_BASE_URL`) before running the script.

### Missing/Unknown Placeholders
*   **Logs**: `WARNING - Restoration warning: missing placeholders...`
*   **Cause**: The LLM translation didn't preserve the exact `φXφ` tags.
*   **Fix**: 
    1.  The system will now attempt to **auto-repair** using the LLM during Assembly.
    2.  If that fails, check logs. In some cases (e.g., complex HTML entities like `&`), the extractor might have degraded to plain text.
    3.  (Fixed in v0.11) Enhanced `FormatExtractor` now handles HTML entities correctly, preventing phantom placeholder hallucinations.

## Change Log

### [2026-02-01] v0.13: Typography & Robustness
*   **Feature**: **Bilingual Typography V3 (Safe Mode)**.
    *   Resolved layout issues: **Drop Cap Overlap** (via `clear: both`) and **Double Indentation**.
    *   Preserved original alignment (e.g. centered titles) by relaxing default justification.
    *   Implemented heuristic CSS class injection (`translation-no-indent`) for Drop Cap paragraphs.
*   **Fix**: **Endnote/Link Extraction**.
    *   Fixed bug where citations like `[28]` were stripped because they were misidentified as translatable text.
    *   Update `FormatExtractor` to treat `[N]` patterns as structural anchors.
    *   Enhanced **Integrity Check** to "hydrate" placeholders during validation, preventing false-positive fallbacks to plain text.
    *   **Effect**: Links and formatting now correctly preserved in text containing citations.

### [2026-02-01] Pipeline Separation & Formatting
*   **Architecture**: Decoupled "Restoration" from "Assembly" into a 4-step pipeline.
    *   **New Step 3**: `03_restore_format.py` handles formatting, spacing, and repair. Saves to `translated_html`.
    *   **New Step 4**: `04_build_epub.py` handles pure EPUB generation.
*   **Data Model**: Added `translated_html` to `ManifestEntry` as the "Gold Master" formatted content.
*   **UX**: Added **Pangu Spacing** (Auto-spacing between CJK and ASCII) in Restoration step.
*   **Optimization**: `RestorationEngine` is now idempotent (skips processing if `translated_html` exists). Added `--force-restore` flag.
*   **Fix**: `main.py` updated to orchestrate the new 4-stage pipeline.

### [2026-01-31] Robustness & Repair
*   **Feature**: Added **LLM-based Placeholder Repair** in Assembly stage. If placeholders mismatch, the system asks the LLM to fix the tags without changing text.
*   **Fix**: Solved `FormatExtractor` "phantom placeholders" issue by correctly unescaping HTML entities during integrity checks.
*   **Fix**: Resolved **Duplicate ID** issue in LLM response parsing. Now recursively strips repeated headers (e.g., `#12: #12: ...`) to prevent them from leaking into the translation.
*   **Tweak**: Updated `pipeline/03_assemble.py` to be async and load LLM config.

### [2026-01-30] Performance Improvements
*   **Concurrency Fix**: Resolved issue where `concurrent_requests` in `config.yaml` was ignored by the Translator engine. Now `main.py` and `pipeline/02_translate.py` correctly propagate this setting, allowing faster translation with higher limits (e.g., for local LLMs or high-rate-limit providers).

### [2026-01-28] Bug Fixes
*   **Fix Cover Image**: Resolved issue where book cover execution was missing in the final EPUB. Added `cover_image_id` tracking in `BookStructure` and restored proper OPF metadata in `BilingualBuilder`.

### [2026-01-27] Externalized Configuration
*   **Config**: Added `config/config.yaml` for tuning parameters (LLM model, RPM, Chunk Size).
*   **Logic**: `pipeline/02_translate.py` now loads settings from `config.yaml`.
*   **Dependency**: Added `PyYAML` to `requirements.txt`.

### [2026-01-27] Architecture Refactoring
*   **Restructured**: Moved source files into `src/preprocessing`, `src/translation`, `src/assembly`, `src/common`.
*   **Pipeline**: Created individual pipeline scripts in `pipeline/`.
*   **Refactor**: Renamed `fine_grained_extractor` to `text_extractor`, `translator` to `translator_engine`, etc.
*   **Logic**: Enforced 100% text coverage check in `format_extractor.py` (removed 95% threshold).
*   **Docs**: Created this Operation Manual.