epub_bilingual_translator/doc/OPERATION_MANUAL.md

# Operation Manual & Change Log

## Core Principles

1.  **Modularity**: The system is divided into three distinct phases (Preprocessing, Translation, Assembly) with clear boundaries.
2.  **Immutability**: `book_structure.json` is generated once during preprocessing and should not be modified by subsequent steps.
3.  **Source of Truth**: `manifest.json` is the single source of truth for translations.
4.  **Idempotency**: Translation steps can be retried without side effects (existing translations are preserved).

## Directory Structure

*   `pipeline/`: Executable scripts for each stage.
    *   `01_preprocess.py`: Clean EPUB, generate structure, extract text.
    *   `02_translate.py`: Translate text in manifest.
    *   `03_assemble.py`: Apply translations and build final EPUB.
*   `src/`: Core logic modules.
    *   `preprocessing/`: Cleaning, extraction, profiling.
    *   `translation/`: LLM integration, manifest management.
    *   `assembly/`: Backfilling, EPUB building.
    *   `common/`: Shared data models and utils.
*   `work/`: Working directory for intermediate files (ignored by git).

## Pipeline Usage

### Step 1: Preprocessing
```bash
python pipeline/01_preprocess.py inputs/my_book.epub
```
Generates `work/my_book/book_structure.json` and `manifest.json`.

### Step 2: Translation
```bash
python pipeline/02_translate.py --input-epub inputs/my_book.epub
```
Translates entries in `manifest.json`. ensuring `.env` has `OPENAI_API_KEY`.

### Step 3: Assembly
### Step 3: Assembly
```bash
# Bilingual Output (Default: output/bilingual_my_book.epub)
python pipeline/03_assemble.py inputs/my_book.epub --bilingual

# Target Language Output (Default: output/translated_my_book.epub)
python pipeline/03_assemble.py inputs/my_book.epub
```
**Note**: The Assembly step now includes an **LLM-based Placeholder Repair** mechanism. If `format_restorer` detects broken placeholders in the translation, it will query the LLM (using your configured credentials) to attempt an automatic fix. Ensure your `OPENAI_API_KEY` is set if you want this feature enabled.

## Configuration

System settings are managed via `config/config.yaml` and environment variables.

### `config/config.yaml`
Control LLM parameters and translation behavior:
```yaml
llm:
  model: "gpt-3.5-turbo"           # LLM Model Name
  base_url: "https://api.openai.com/v1" 
  timeout: 60
  requests_per_minute: 60          # Rate limiting
  concurrent_requests: 5           # Parallel chunks

translation:
  chunk_size: 4000                 # Characters per chunk
```

### Environment Variables (`.env`)
Security-sensitive credentials must be set here:
```bash
OPENAI_API_KEY=sk-...              # Required
OPENAI_BASE_URL=...                # Optional override for config
```

## Known Issues & Troubleshooting

### AuthenticationError (OpenRouter etc.)
If you see `AuthenticationError` despite having the correct `base_url` in config:
1.  Check if you have a stale `OPENAI_API_KEY` in your shell environment.
2.  Environment variables **override** `.env` files.
3.  Fix: Run `unset OPENAI_API_KEY` (and `OPENAI_BASE_URL`) before running the script.

### Missing/Unknown Placeholders
*   **Logs**: `WARNING - Restoration warning: missing placeholders...`
*   **Cause**: The LLM translation didn't preserve the exact `φXφ` tags.
*   **Fix**: 
    1.  The system will now attempt to **auto-repair** using the LLM during Assembly.
    2.  If that fails, check logs. In some cases (e.g., complex HTML entities like `&`), the extractor might have degraded to plain text.
    3.  (Fixed in v0.11) Enhanced `FormatExtractor` now handles HTML entities correctly, preventing phantom placeholder hallucinations.

## Change Log

### [2026-01-31] Robustness & Repair
*   **Feature**: Added **LLM-based Placeholder Repair** in Assembly stage. If placeholders mismatch, the system asks the LLM to fix the tags without changing text.
*   **Fix**: Solved `FormatExtractor` "phantom placeholders" issue by correctly unescaping HTML entities during integrity checks.
*   **Fix**: Resolved **Duplicate ID** issue in LLM response parsing. Now recursively strips repeated headers (e.g., `#12: #12: ...`) to prevent them from leaking into the translation.
*   **Tweak**: Updated `pipeline/03_assemble.py` to be async and load LLM config.

### [2026-01-30] Performance Improvements
*   **Concurrency Fix**: Resolved issue where `concurrent_requests` in `config.yaml` was ignored by the Translator engine. Now `main.py` and `pipeline/02_translate.py` correctly propagate this setting, allowing faster translation with higher limits (e.g., for local LLMs or high-rate-limit providers).

### [2026-01-28] Bug Fixes
*   **Fix Cover Image**: Resolved issue where book cover execution was missing in the final EPUB. Added `cover_image_id` tracking in `BookStructure` and restored proper OPF metadata in `BilingualBuilder`.

### [2026-01-27] Externalized Configuration
*   **Config**: Added `config/config.yaml` for tuning parameters (LLM model, RPM, Chunk Size).
*   **Logic**: `pipeline/02_translate.py` now loads settings from `config.yaml`.
*   **Dependency**: Added `PyYAML` to `requirements.txt`.

### [2026-01-27] Architecture Refactoring
*   **Restructured**: Moved source files into `src/preprocessing`, `src/translation`, `src/assembly`, `src/common`.
*   **Pipeline**: Created individual pipeline scripts in `pipeline/`.
*   **Refactor**: Renamed `fine_grained_extractor` to `text_extractor`, `translator` to `translator_engine`, etc.
*   **Logic**: Enforced 100% text coverage check in `format_extractor.py` (removed 95% threshold).
*   **Docs**: Created this Operation Manual.