Synchronize chapter and navigation translations with Chinese prose indentation

This commit is contained in:
Kai Tan
2026-10-08 07:59:01 +08:00
parent caf9d8f4bb
commit a158b4e26c
18 changed files with 613 additions and 11 deletions
+8
View File
@@ -1,3 +1,11 @@
## 2026-10-08 — Chinese typography and shared navigation titles
- Indent Chinese body paragraphs by two em, including chapter openings and converted div prose; preserve heading/caption/note layouts.
- Bind NAV, NCX and visible contents links to shared chapter/part translations while preserving anchors, hierarchy and resource paths.
- Recognize adjacent number/title/subtitle blocks and normalize Chinese chapter/part numbers without merging their markup.
- Include heading context in translation cache fingerprints and update old manifest contexts during reconciliation.
- Add navigation regression tests and record the scanned PDF/OCR to EPUB roadmap.
## 2026-10-08 — translation and QA improvements ## 2026-10-08 — translation and QA improvements
- Replace regex tag extraction with balanced DOM placeholders and protect literal φ. - Replace regex tag extraction with balanced DOM placeholders and protect literal φ.
+12 -2
View File
@@ -26,9 +26,9 @@ cp .env.example .env
- DOM 遍历保留行内标签,图片、数学公式和脚注作为完整对象保护;占位符须次数正确、嵌套合法,还原后须是合法 XHTML。 - DOM 遍历保留行内标签,图片、数学公式和脚注作为完整对象保护;占位符须次数正确、嵌套合法,还原后须是合法 XHTML。
- 大段初译后,只重试未通过校验的条目。网络错误、重复输出 ID、缺失条目、截断和占位符错误保持待处理状态,不写入“错误译文”。正常运行有失败时终止,不静默生成不完整中文版。 - 大段初译后,只重试未通过校验的条目。网络错误、重复输出 ID、缺失条目、截断和占位符错误保持待处理状态,不写入“错误译文”。正常运行有失败时终止,不静默生成不完整中文版。
- 在原 EPUB 容器上修改正文;保留 OPF、目录、原 CSS 引用、图片、字体和其他资源。双语段落避免复制 ID,中文版保留原锚点;列表和表格不会插入无效兄弟节点。 - 在原 EPUB 容器上修改正文;保留 OPF、目录链接及层级、原 CSS 引用、图片、字体和其他资源。双语段落避免复制 ID,中文版保留原锚点;列表和表格不会插入无效兄弟节点。
- HTML5 修复处理不闭合标签和重复 ID,抽取不会重复处理嵌套列表/段落;支持裸 div、表格单元、图注和混合容器中的直接文本。 - HTML5 修复处理不闭合标签和重复 ID,抽取不会重复处理嵌套列表/段落;支持裸 div、表格单元、图注和混合容器中的直接文本。
- 已翻译条目按原文、占位符、术语表、模型及风格指南指纹复用;每个 chunk 完成就保存。源文件变更会重新预处理。旧版本缓存会升级,无法安全复用的译文需要重新翻译。 - 已翻译条目按原文、占位符、标题上下文、术语表、模型及风格指南指纹复用;每个 chunk 完成就保存。源文件变更会重新预处理。旧版本缓存会升级,无法安全复用的译文需要重新翻译。
`--skip-translation` 用于离线调试/部分样例,不代表全书已翻译。`--force-clean` 重新预处理;`--force-restore` 兼容旧用法,但现在每次回填前都会重新校验,旧的缓存 HTML 不再直接被信任。 `--skip-translation` 用于离线调试/部分样例,不代表全书已翻译。`--force-clean` 重新预处理;`--force-restore` 兼容旧用法,但现在每次回填前都会重新校验,旧的缓存 HTML 不再直接被信任。
@@ -64,3 +64,13 @@ Mac 沙箱中 Calibre 的 Qt 渲染可能需要允许进程运行。QA 使用项
``` ```
`build_sample.py` 只接受与当前原文和占位符完全对应的条目,输出明确标为部分样例。完整架构见 `doc/architecture_flow.md`;本次测试记录见 `doc/QA_2026-10-08.md`。 `build_sample.py` 只接受与当前原文和占位符完全对应的条目,输出明确标为部分样例。完整架构见 `doc/architecture_flow.md`;本次测试记录见 `doc/QA_2026-10-08.md`。
### 中文正文与标题目录
中文正文(包括章首段)使用 `2em` 首行缩进;标题、图注、引用和注释保持各自布局。原文段落保留出版社样式。中文段落清除原内联 `text-indent`,以免 `!important` 覆盖中文规则。
预处理以目录的目标文件/锚点建立标题映射。每个实际章标题只翻译一次,NAV、NCX、书内目录及阅读器文档标题使用同一结果;目录 href、章节 ID、层级和顺序保留。章号、主标题、副标题分成多个块时,精确匹配相邻标题块并分别回填,不合并 DOM。中文模式把独立章号统一成“第一章”,Part 对应“第一部”;双语模式保留英文标题和目录标签。
无法可靠绑定的目录条目仅翻译标签,不改写猜测的章节正文;具体条目记录在 `book_structure.json` 的 `navigation_titles[].binding_note`。这类条目需要检查章节标题是否一致,不能视为全书标题同步已经通过。索引、版权、注释全文仍遵守原有跳过策略;导航中这些页面的名称可以翻译。
后续扫描 PDF/OCR 大版本的图文、结构与目录方案见 [路线图](doc/OCR_EPUB_ROADMAP.md)。
+23
View File
@@ -0,0 +1,23 @@
# 扫描 PDF → EPUB:后续大版本
这是独立的大版本方向,本轮不实现。目标是把旧中文书的扫描 PDF 转成可重排、可搜索、可点击目录的 EPUB,同时保留原书中有阅读意义的结构与图文关系。
## 输入与评测样本
先收集少量代表样本:纯文字、竖排/繁体、双栏、跨页脚注、目录、表格、示意图、照片,以及歪斜和低清扫描。再盘点用户本地已部署的 OCR,用相同页面比较文字准确度、阅读顺序、版面坐标、图表识别、速度和资源占用。届时根据实际部署与测试结果选择模型,保留可替换的 OCR 接口。
## 中间结构
OCR 输出不能直接拼成 HTML。建立带页面来源、坐标、阅读顺序与置信度的版面中间表示,区分正文段落、章号、主标题、副标题、页眉页脚、脚注/尾注、表格、图片、图注。识别跨页段落合并与断词,保留人工修正记录,以便重复 OCR 时复用。
## 图文与样式
从原 PDF 页面或扫描图按区域提取图片、示意图,保留分辨率、裁切来源和对应图注,作为 EPUB 独立资源嵌入。表格优先转为语义表格;复杂或低置信度图表保留原图。可重排 EPUB 保留标题层级、强调、段落、引用和图文关系;精确页面位置有意义的材料另外评估固定版式 EPUB,不能要求同一个可重排版本在每种屏幕上保持 PDF 的坐标。
## 目录和注释
把目录条目、章标题、页码映射到中间结构的稳定 ID,采用本项目的“目标锚点 → 标题”映射生成 NAV、NCX 和书内目录。原 PDF 页码用作来源追踪,不直接充当 EPUB 页码。脚注建立正文引用、注释内容及返回链接;低置信度匹配进入待审核队列。
## QA 闭环与验收
对同一段原扫描、OCR 中间结果和 EPUB 阅读器截图进行对照。验证阅读顺序、图片/图注归属、标题层级、目录点击、注释双向跳转、字体与中文缩进。每次修复保留样例和回归用例;把文字识别质量、结构正确性与视觉版式分别报告。先完成一小本代表书的端到端试验,再扩展到批量转换。
+22
View File
@@ -26,3 +26,25 @@ CLI 主流程使用自制小 EPUB 完成无 API 冒烟验证;新增的输出
自动术语扫描依赖模型判断。完整名字和明确别名可约束;同姓人物、间接指称、未列入表的其他译名仍须语言 QA。`exclude_contexts` 与较长术语优先处理能减少误判,不能代替全书语义消歧。 自动术语扫描依赖模型判断。完整名字和明确别名可约束;同姓人物、间接指称、未列入表的其他译名仍须语言 QA。`exclude_contexts` 与较长术语优先处理能减少误判,不能代替全书语义消歧。
HTML5 解析可以修复结构,不能自动判断出版者本意;特别的固定版式、复杂 SVG/脚本内容和阅读器差异仍需具体样本。正常主流程遇到新增结构错误会停止并留下报告。原书问题单独留存,不用“没有新增问题”宣称原书符合所有 EPUB 标准。 HTML5 解析可以修复结构,不能自动判断出版者本意;特别的固定版式、复杂 SVG/脚本内容和阅读器差异仍需具体样本。正常主流程遇到新增结构错误会停止并留下报告。原书问题单独留存,不用“没有新增问题”宣称原书符合所有 EPUB 标准。
## 第二轮:中文缩进与标题目录同步
- 新增标题目标映射:NAV、NCX 和明确的书内目录链接解析到原文件/锚点;章号、标题、副标题的相邻块精确绑定,各块仍各自翻译和回填。目录复用实际标题译文,不另行生成章标题。
- 中文章号统一为“第一章”,Part 对应“第一部”。兼容数字、罗马数字和常见英文数字词,保留数字所在 span 的样式;双语模式保留原英文标题。
- 中文正文首行缩进 2em,包括原书使用 noindent 的章首段。清除中文段落的内联 text-indent,英文段落不修改;图注、引文、注释、居中段落和标题排除。Calibre 转换产生的正文 div 也适用。
- 41 项离线测试通过,覆盖双语/中文模式、拆分标题、NAV/NCX/书内目录的一致性、链接及层级、反复回填、内联 CSS、标题缓存上下文更新,以及标题首词 Two/I 的误判回归。
- 第二轮 11 本书共 27,219 单元、467 个目录目标:原文格式还原失败 0,新增结构问题 0。其中 403 个目标绑定实际标题,64 个不确定目标保留独立标签翻译并记录 binding_note。多数是封面、扉页或装饰标题,仍有复杂章节结构;不声称所有书的标题已经自动同步。Empire of AI 的原有 1 项源文件问题仍记录在报告中。
### Luna 样书与视觉范围
Luna 翻译了 Gambling Man 的 45 个标题单元,45/45 占位符校验通过;沿用第一轮 10 段正文,合计 55/1,582 单元。本轮没有做全书正文翻译,也没有验证实际外部 API 的整本吞吐。封面、扉页和版权这三个导航标签采用独立译文,其余目录目标均可绑定章节/部标题。
最终文件位于 `.work/navigation-final-v4/`:`bilingual-sample.epub`、`target_only-sample.epub`、完整 manifest,以及 `qa-chapter/` 和 `qa-bilingual/` 对照证据。这些局部 QA 样例不提交到 Git。
实际检查了中文版第 1–3 页、双语版第 1–4 页与相应原文页面:部标题、章号、主标题、中文首段/后续段缩进、英文 drop cap、上下标、强调与正文排列没有重叠或截断。PDF 页面是辅助视图;中文 EPUB 另外在 Calibre 9.15 阅读器中打开,确认中文目录“第一章:根源”跳转到对应章节、第一部到第五部层级保留、第一条注释弹窗内容正确。继续进入尾注页,点击第一章第 1 条尾注的返回链接,阅读器显示对应中文正文弹窗,再返回正文主视图;原锚点及返回链接正常。
脚注/尾注全文仍按原有提取策略跳过;本轮翻译的是其目录名称和目录所指的页面标题。尾注内部按章分组的标题仍可能是英文。这一范围不能误报为注释全文已译。
渲染期间有一次 Calibre PDF 进程 SIGSEGV,重新生成后的最终两组输出成功。自动报告仍为 visual_review: pending,人工/agent 观察记录单独保存为 visual-review.json;截图生成本身不表示整本视觉验收通过。
OCR、扫描页图文提取、版面中间结构与可点击目录的后续大版本方案见 `doc/OCR_EPUB_ROADMAP.md`。
+5 -1
View File
@@ -12,6 +12,7 @@ from src.common.config import load_global_config
from src.preprocessing.epub_cleaner import EpubCleaner from src.preprocessing.epub_cleaner import EpubCleaner
from src.preprocessing.profiler import BookProfiler from src.preprocessing.profiler import BookProfiler
from src.preprocessing.text_extractor import FineGrainedExtractor from src.preprocessing.text_extractor import FineGrainedExtractor
from src.preprocessing.navigation import prepare_navigation
from src.translation.manifest_manager import ManifestManager from src.translation.manifest_manager import ManifestManager
from src.translation.translator_engine import Translator from src.translation.translator_engine import Translator
from src.translation.glossary import prepare_glossary from src.translation.glossary import prepare_glossary
@@ -65,7 +66,8 @@ async def run_pipeline(args):
if work["structure"].exists() and not args.force_clean: if work["structure"].exists() and not args.force_clean:
logger.info(f"Reusing existing book_structure: {work['structure']}") logger.info(f"Reusing existing book_structure: {work['structure']}")
structure = BookStructure.load(work["structure"]) structure = BookStructure.load(work["structure"])
if structure.source_sha256 != hashlib.sha256(input_path.read_bytes()).hexdigest(): if (structure.preprocessing_version < 3 or
structure.source_sha256 != hashlib.sha256(input_path.read_bytes()).hexdigest()):
logger.info("Source changed or old preprocessing cache; re-cleaning") logger.info("Source changed or old preprocessing cache; re-cleaning")
structure = BookStructure.load(EpubCleaner(input_path, work["root"]).clean()) structure = BookStructure.load(EpubCleaner(input_path, work["root"]).clean())
else: else:
@@ -77,6 +79,8 @@ async def run_pipeline(args):
# 2. Extraction # 2. Extraction
extractor = FineGrainedExtractor() extractor = FineGrainedExtractor()
manifest_entries = extractor.extract(structure) manifest_entries = extractor.extract(structure)
prepare_navigation(input_path,structure,manifest_entries)
structure.save(work["structure"])
# 3. Manifest Management # 3. Manifest Management
manifest_manager = ManifestManager(work["manifest"]) manifest_manager = ManifestManager(work["manifest"])
+3
View File
@@ -9,6 +9,7 @@ sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
from src.preprocessing.epub_cleaner import EpubCleaner from src.preprocessing.epub_cleaner import EpubCleaner
from src.preprocessing.text_extractor import FineGrainedExtractor from src.preprocessing.text_extractor import FineGrainedExtractor
from src.preprocessing.navigation import prepare_navigation
from src.translation.manifest_manager import ManifestManager from src.translation.manifest_manager import ManifestManager
from src.common.data_model import BookStructure from src.common.data_model import BookStructure
from src.common.utils import setup_logger, ensure_directory from src.common.utils import setup_logger, ensure_directory
@@ -48,6 +49,8 @@ def run_preprocess(args):
logger.info("Extracting text segments...") logger.info("Extracting text segments...")
extractor = FineGrainedExtractor() extractor = FineGrainedExtractor()
entries = extractor.extract(structure) entries = extractor.extract(structure)
prepare_navigation(input_path,structure,entries)
structure.save(structure_path)
# 3. Update Manifest # 3. Update Manifest
logger.info(f"Updating manifest: {manifest_path}") logger.info(f"Updating manifest: {manifest_path}")
+2
View File
@@ -8,6 +8,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from src.common.data_model import BookStructure, ManifestEntry from src.common.data_model import BookStructure, ManifestEntry
from src.preprocessing.epub_cleaner import EpubCleaner from src.preprocessing.epub_cleaner import EpubCleaner
from src.preprocessing.text_extractor import FineGrainedExtractor from src.preprocessing.text_extractor import FineGrainedExtractor
from src.preprocessing.navigation import prepare_navigation
from src.assembly.restoration_engine import RestorationEngine from src.assembly.restoration_engine import RestorationEngine
from src.assembly.backfiller import BackfillEngine from src.assembly.backfiller import BackfillEngine
from src.assembly.builder import BilingualBuilder from src.assembly.builder import BilingualBuilder
@@ -17,6 +18,7 @@ async def run(args):
args.out.mkdir(parents=True,exist_ok=True) args.out.mkdir(parents=True,exist_ok=True)
structure=BookStructure.load(EpubCleaner(args.source,args.out/'source').clean()) structure=BookStructure.load(EpubCleaner(args.source,args.out/'source').clean())
entries=FineGrainedExtractor().extract(structure) entries=FineGrainedExtractor().extract(structure)
prepare_navigation(args.source,structure,entries)
by_id={e.entry_id:e for e in entries} by_id={e.entry_id:e for e in entries}
provided=[ManifestEntry.model_validate(e) for e in json.loads(args.manifest.read_text())] provided=[ManifestEntry.model_validate(e) for e in json.loads(args.manifest.read_text())]
for sample in provided: for sample in provided:
+4 -1
View File
@@ -6,6 +6,7 @@ from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1])) sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from src.preprocessing.epub_cleaner import EpubCleaner from src.preprocessing.epub_cleaner import EpubCleaner
from src.preprocessing.text_extractor import FineGrainedExtractor from src.preprocessing.text_extractor import FineGrainedExtractor
from src.preprocessing.navigation import prepare_navigation
from src.common.data_model import BookStructure from src.common.data_model import BookStructure
from src.assembly.format_restorer import FormatRestorer from src.assembly.format_restorer import FormatRestorer
from src.assembly.builder import BilingualBuilder from src.assembly.builder import BilingualBuilder
@@ -23,10 +24,12 @@ def main():
work=args.out/source.stem work=args.out/source.stem
structure=BookStructure.load(EpubCleaner(source,work).clean()) structure=BookStructure.load(EpubCleaner(source,work).clean())
entries=FineGrainedExtractor().extract(structure) entries=FineGrainedExtractor().extract(structure)
prepare_navigation(source,structure,entries)
bad=[e.entry_id for e in entries if not FormatRestorer().restore(e.original_text,e.placeholders,original_text=e.original_text)[1]] bad=[e.entry_id for e in entries if not FormatRestorer().restore(e.original_text,e.placeholders,original_text=e.original_text)[1]]
output=BilingualBuilder(work,source).build(structure,work/'roundtrip.epub') output=BilingualBuilder(work,source).build(structure,work/'roundtrip.epub')
before,after=audit_epub(source),audit_epub(output) before,after=audit_epub(source),audit_epub(output)
row.update(entries=len(entries),format_failures=bad,source_issues=len(before['issues']), row.update(entries=len(entries),navigation_titles=len(structure.navigation_titles),
unbound_titles=[r["source_label"] for r in structure.navigation_titles if r.get("binding_note")],format_failures=bad,source_issues=len(before['issues']),
new_issues=compare_audits(before,after)) new_issues=compare_audits(before,after))
except Exception as exc: except Exception as exc:
row['error']=str(exc) row['error']=str(exc)
+1 -1
View File
@@ -96,7 +96,7 @@ def main():
'scope': 'provided manifest only'} 'scope': 'provided manifest only'}
if errors and not args.allow_partial: if errors and not args.allow_partial:
report['new_issues'].append({'kind': 'incomplete_translation', 'detail': str(len(errors))}) report['new_issues'].append({'kind': 'incomplete_translation', 'detail': str(len(errors))})
chapter_hrefs = args.chapter or list(dict.fromkeys(e.file_path for e in entries if e.translated_text)) chapter_hrefs = args.chapter or list(dict.fromkeys(e.file_path for e in entries if e.translated_text and e.file_path != "__navigation__"))
if not args.chapter and args.max_chapters: if not args.chapter and args.max_chapters:
chapter_hrefs = chapter_hrefs[:args.max_chapters] chapter_hrefs = chapter_hrefs[:args.max_chapters]
report['sampled_chapters'] = chapter_hrefs report['sampled_chapters'] = chapter_hrefs
+27
View File
@@ -1,5 +1,7 @@
import copy import copy
import re
from src.common.html import parse_html from src.common.html import parse_html
from src.preprocessing.navigation import normalize_heading_html, sync_navigation
from src.assembly.format_restorer import FormatRestorer from src.assembly.format_restorer import FormatRestorer
from src.common.utils import setup_logger, add_spacing from src.common.utils import setup_logger, add_spacing
@@ -10,6 +12,7 @@ class BackfillEngine:
async def backfill(self, structure, manifest_entries, mode='bilingual', allow_partial=False): async def backfill(self, structure, manifest_entries, mode='bilingual', allow_partial=False):
if mode not in {'bilingual', 'target_only'}: if mode not in {'bilingual', 'target_only'}:
raise ValueError(f'Unknown output mode: {mode}') raise ValueError(f'Unknown output mode: {mode}')
titles = {r["entry_id"]: r for r in structure.navigation_titles}
by_file = {} by_file = {}
failures = [] failures = []
applied = 0 applied = 0
@@ -21,9 +24,16 @@ class BackfillEngine:
restored, ok = FormatRestorer().restore(add_spacing(entry.translated_text), entry.placeholders, restored, ok = FormatRestorer().restore(add_spacing(entry.translated_text), entry.placeholders,
original_text=entry.original_text) original_text=entry.original_text)
if not ok: if not ok:
entry.translated_html = None
entry.validation_errors = ["invalid restored XHTML"]
failures.append(entry.entry_id) failures.append(entry.entry_id)
continue continue
entry.translated_html = restored entry.translated_html = restored
record = titles.get(entry.entry_id)
if record:
entry.translated_html=normalize_heading_html(record,restored)
if entry.file_path=='__navigation__':
continue
by_file.setdefault(entry.file_path, []).append(entry) by_file.setdefault(entry.file_path, []).append(entry)
if failures and not allow_partial: if failures and not allow_partial:
raise ValueError(f"Backfill rejected {len(failures)} unvalidated entries: {failures[:3]}") raise ValueError(f"Backfill rejected {len(failures)} unvalidated entries: {failures[:3]}")
@@ -49,6 +59,21 @@ class BackfillEngine:
new_tag['xml:lang'] = 'zh-CN' new_tag['xml:lang'] = 'zh-CN'
classes = list(new_tag.get('class', [])) classes = list(new_tag.get('class', []))
new_tag['class'] = classes + ['translation'] new_tag['class'] = classes + ['translation']
heading = entry.context and (entry.context.startswith('navigation-heading:') or entry.context in {'chapter','section','subsection'})
special = re.search(r'caption|footnote|endnote|epigraph|verse|poem|center|attribution|dedication', ' '.join(element.get('class', [])), re.I)
prose = ((tag_name in {'p','div'} or (tag_name=='span' and element.get('data-translation-unit')))
and not heading and entry.context != 'epigraph'
and not special and not re.search(r'text-align\s*:\s*(?:center|right)', element.get('style',''),re.I)
and not element.find_parent(['blockquote', 'figure', 'aside'])
and not element.find_parent(attrs={'epub:type': re.compile(r'footnote|endnote')}))
# Chinese body paragraphs use two em even when the publisher's
# opening-paragraph class sets text-indent:0.
new_tag['data-translation-role'] = 'body' if prose else 'heading' if heading else 'other'
if prose:
# Publisher inline !important rules otherwise outrank CSS.
style = re.sub(r'(?:^|;)\s*text-indent\s*:[^;]*', '', new_tag.get('style',''),flags=re.I).strip('; ')
if style: new_tag['style'] = style
else: new_tag.attrs.pop('style', None)
applied += 1 applied += 1
fragment = parse_html(f'<div>{entry.translated_html}</div>').body.div fragment = parse_html(f'<div>{entry.translated_html}</div>').body.div
for node in list(fragment.contents): for node in list(fragment.contents):
@@ -63,6 +88,8 @@ class BackfillEngine:
else: else:
element.replace_with(new_tag) element.replace_with(new_tag)
resource.content = str(soup) resource.content = str(soup)
resource.translation_modified = True
sync_navigation(structure, manifest_entries, mode)
if mode == 'target_only' and applied and not allow_partial: if mode == 'target_only' and applied and not allow_partial:
structure.metadata.language = 'zh-CN' structure.metadata.language = 'zh-CN'
return structure return structure
+6 -1
View File
@@ -1,4 +1,4 @@
/* Preserve the book's alignment, indent and inline font choices. */ /* Preserve publisher alignment and inline fonts; apply Chinese prose indentation. */
.translation { .translation {
line-height: 1.6; line-height: 1.6;
clear: both; clear: both;
@@ -17,3 +17,8 @@ p.translation::first-letter {
color: inherit !important; color: inherit !important;
float: none !important; float: none !important;
} }
/* Prose gets two Chinese character widths, including chapter-opening prose.
Headings, captions, quotations and note markers keep their own layout. */
.translation[data-translation-role="body"] {
text-indent: 2em !important;
}
+7 -2
View File
@@ -37,15 +37,20 @@ def write_preserving_package(original, structure, output):
etree.SubElement(manifest, f'{{{OPF}}}item', id=css_id, href=css_href, **{'media-type': 'text/css'}) etree.SubElement(manifest, f'{{{OPF}}}item', id=css_id, href=css_href, **{'media-type': 'text/css'})
replacements[css_archive_path] = (Path(__file__).parent/'bilingual_style.css').read_bytes() replacements[css_archive_path] = (Path(__file__).parent/'bilingual_style.css').read_bytes()
for item_id, resource in structure.resources.items(): for item_id, resource in structure.resources.items():
if resource.media_type != 'application/xhtml+xml' or not resource.content: if resource.media_type not in {'application/xhtml+xml','application/x-dtbncx+xml'} or not resource.content:
continue continue
if 'nav' in (items.get(item_id).get('properties', '') if item_id in items else '').split() and 'translation-for' not in resource.content and 'class="translation"' not in resource.content: if 'nav' in (items.get(item_id).get('properties', '') if item_id in items else '').split() and not resource.translation_modified and 'translation-for' not in resource.content and 'class="translation"' not in resource.content:
continue continue
if item_id not in items: if item_id not in items:
raise ValueError(f'Unknown package item: {item_id}') raise ValueError(f'Unknown package item: {item_id}')
archive_path = posixpath.normpath(posixpath.join(base, unquote(items[item_id].get('href')))) archive_path = posixpath.normpath(posixpath.join(base, unquote(items[item_id].get('href'))))
if archive_path not in source.namelist(): if archive_path not in source.namelist():
raise ValueError(f'Missing source resource: {archive_path}') raise ValueError(f'Missing source resource: {archive_path}')
if resource.media_type=='application/x-dtbncx+xml':
if resource.translation_modified:
etree.fromstring(resource.content.encode(),etree.XMLParser(resolve_entities=False,no_network=True))
replacements[archive_path]=resource.content.encode()
continue
soup = parse_html(resource.content) soup = parse_html(resource.content)
soup.html['xmlns'] = 'http://www.w3.org/1999/xhtml' soup.html['xmlns'] = 'http://www.w3.org/1999/xhtml'
soup.html['xmlns:epub'] = 'http://www.idpf.org/2007/ops' soup.html['xmlns:epub'] = 'http://www.idpf.org/2007/ops'
+3 -1
View File
@@ -28,11 +28,13 @@ class ResourceItem(BaseModel):
content: Optional[str] = None # For text/html content: Optional[str] = None # For text/html
file_path: Optional[str] = None # For binary/assets (relative to assets dir) file_path: Optional[str] = None # For binary/assets (relative to assets dir)
properties: Optional[str] = None properties: Optional[str] = None
translation_modified: bool = False
class BookStructure(BaseModel): class BookStructure(BaseModel):
source_sha256: Optional[str] = None source_sha256: Optional[str] = None
preprocessing_version: int = 2 preprocessing_version: int = 3
metadata: BookMetaData metadata: BookMetaData
navigation_titles: List[Dict[str, Any]] = Field(default_factory=list)
spine: List[str] = Field(default_factory=list, description="Ordered list of item IDs in spine") spine: List[str] = Field(default_factory=list, description="Ordered list of item IDs in spine")
resources: Dict[str, ResourceItem] = Field(default_factory=dict, description="Map of item_id to ResourceItem") resources: Dict[str, ResourceItem] = Field(default_factory=dict, description="Map of item_id to ResourceItem")
+296
View File
@@ -0,0 +1,296 @@
"""One title translation per destination, shared by headings, NAV, NCX and TOC pages.
Original href/src, anchors and hierarchy are never rebuilt from translated text.
Uncertain heading matches are recorded rather than replacing arbitrary prose.
"""
import hashlib
import posixpath
import re
import zipfile
from urllib.parse import unquote, urlsplit
from bs4 import NavigableString
from lxml import etree
from src.common.html import parse_html
from src.common.data_model import ManifestEntry
from src.preprocessing.format_extractor import FormatExtractor
from src.assembly.package_writer import package_path
HEADINGS = ['h1', 'h2', 'h3', 'h4', 'h5', 'h6']
WORDS = {word: i for i, word in enumerate('zero one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen twenty'.split())}
NUMBER = r'(?:\d+|[ivxlcdm]+|' + '|'.join(WORDS) + ')'
def destination(document, href):
parts = urlsplit(href)
if parts.scheme or parts.netloc:
return None
path = (posixpath.normpath(posixpath.join(posixpath.dirname(document), unquote(parts.path)))
if parts.path else document)
return path + ('#' + unquote(parts.fragment) if parts.fragment else '')
def plain(text):
return re.sub(r'\s+', ' ', text).strip()
def is_toc(soup, href, properties=''):
return ('nav' in properties.split() or soup.find(attrs={'role': 'doc-toc'}) is not None
or any('toc' in str(t.get('epub:type', '')).split() for t in soup.find_all('nav'))
or re.search(r'(?:^|[/_\-\d])(toc|contents|nav)(?:[._\-]|$)', href, re.I) is not None
or any(plain(h.get_text()).casefold() in {'contents', 'table of contents', '目录'}
for h in soup.find_all(HEADINGS)))
def title_identity(label):
label = plain(label)
label, count = re.subn(r'^(?:chapter|chap\.?|part|book|section)\s+' + NUMBER + r'\b\s*[:.|\-–—]?\s*', '', label, count=1, flags=re.I)
if not count:
label = re.sub(r'^' + NUMBER + r'\b\s*[.:|\-–—]?\s*', '', label, count=1, flags=re.I)
return re.sub(r'[^\w]', '', label.casefold())
def number_info(label):
match = re.match(r'^(chapter|chap\.?|part|book)\s+(' + NUMBER + r')\b', plain(label), re.I)
if match:
role = 'chapter' if match[1].lower().startswith('chap') else 'part'
number = match[2]
else:
match = re.match(r'^(\d+)\b', plain(label))
if not match: return None, None
role, number = 'chapter', match[1]
if number.isdigit():
value = int(number)
elif number.casefold() in WORDS:
value = WORDS[number.casefold()]
else:
values = {'I':1,'V':5,'X':10,'L':50,'C':100,'D':500,'M':1000}
value, previous = 0, 0
for char in reversed(number.upper()):
current=values[char]; value += -current if current < previous else current; previous=max(previous,current)
return role, value
def chinese_number(number):
if number < 0 or number >= 10000:
return str(number)
if number == 0:
return '零'
digits='零一二三四五六七八九'; result=''; zero=False
for divisor,unit in [(1000,'千'),(100,'百'),(10,'十'),(1,'')]:
digit,number=divmod(number,divisor)
if digit:
if zero: result+='零'
if not (digit==1 and divisor==10 and not result): result+=digits[digit]
result+=unit;zero=False
elif result and number:
zero=True
return result
def ordinal(record):
if record.get('number') is None:
return None
return '第'+chinese_number(record['number'])+('章' if record['role']=='chapter' else '部')
def _headings(soup, target, label):
fragment = target.partition('#')[2]
anchor = soup.find(id=fragment) if fragment else None
if fragment and anchor is None:
return [], 'missing anchor'
identity = title_identity(label)
# Only inspect the opening leaf blocks or blocks at the explicit anchor.
# An exact match across adjacent blocks also recognizes Calibre-converted
# number/title/subtitle paragraphs without guessing from font sizes.
blocks = soup.find_all(HEADINGS + ['p', 'div'])
blocks = [t for t in blocks if not t.find(HEADINGS + ['p','div','section','blockquote','li'])]
if anchor:
nearby = [anchor] + anchor.find_all_next(HEADINGS + ['p','div'], limit=16)
block_ids = {id(t) for t in blocks}
blocks = [t for t in nearby if id(t) in block_ids]
blocks = [t for t in blocks if plain(t.get_text())][:12]
matches = []
for start in range(min(len(blocks), 6)):
for end in range(start+1, min(start+4, len(blocks))+1):
group = blocks[start:end]
if any(len(plain(t.get_text())) > 300 for t in group): break
combined = ' '.join(t.get_text(' ', strip=True) for t in group)
if title_identity(combined) == identity and identity:
matches.append(group)
# A number leaf and the subtitle alone can both match after stripping a
# numeric prefix. Prefer the complete adjacent title group.
if matches:
longest = max(len(g) for g in matches)
matches = [g for g in matches if len(g) == longest]
if len(matches) == 1: return matches[0], None
return [], 'ambiguous heading'
# An explicit heading anchor is authoritative even if the publisher uses
# an abbreviated TOC label; a prose anchor never receives this fallback.
if anchor and anchor.name in HEADINGS: return [anchor], None
return [], 'no matching heading'
def prepare_navigation(original, structure, entries):
"""Enrich the manifest with canonical heading entries, never translate TOC duplicates."""
soups={r.href:parse_html(r.content) for r in structure.resources.values()
if r.media_type=='application/xhtml+xml' and r.content}
resources={posixpath.normpath(unquote(r.href)):r for r in structure.resources.values()}
labels={}
def collect(document, href, label):
target=destination(document,href)
if target and plain(label):
record=labels.setdefault(target,{'target':target,'source_label':plain(label),'source_variants':[]})
if plain(label) not in record['source_variants']: record['source_variants'].append(plain(label))
# EPUB3 NAV and explicit contents pages take priority over legacy NCX names.
toc_files=set()
for href,soup in soups.items():
if not is_toc(soup,href,resources[posixpath.normpath(unquote(href))].properties or ''):
continue
toc_files.add(href)
toc_containers=[n for n in soup.find_all('nav') if 'toc' in str(n.get('epub:type','')).split() or n.get('role')=='doc-toc']
containers=toc_containers or [soup.body]
for container in containers:
for a in container.find_all('a',href=True): collect(href,a['href'],a.get_text(' ',strip=True))
with zipfile.ZipFile(original) as source:
base=posixpath.dirname(package_path(source))
for resource in structure.resources.values():
if resource.media_type!='application/x-dtbncx+xml': continue
raw=source.read(posixpath.normpath(posixpath.join(base,unquote(resource.href))))
doc=etree.fromstring(raw,etree.XMLParser(resolve_entities=False,no_network=True))
for point in doc.xpath('//*[local-name()="navMap"]//*[local-name()="navPoint"]'):
content=point.xpath('./*[local-name()="content"]/@src')
text=point.xpath('./*[local-name()="navLabel"]/*[local-name()="text"]')
if content and text: collect(resource.href,content[0],''.join(text[0].itertext()))
# NCX stays XML; it is not passed through an HTML parser.
resource.content=etree.tostring(doc,encoding='unicode')
# Ordinary manifest extraction must not independently translate visible
# TOC text. Those links are synchronized by target after title translation.
entries[:]=[e for e in entries if e.file_path not in toc_files]
by_id={e.entry_id:e for e in entries}; records=[]
for record in labels.values():
path=record['target'].partition('#')[0]; resource=resources.get(path)
role,number=number_info(record['source_label']);record.update(role=role,number=number)
headings=[];reason='missing document'
if resource and resource.href in soups:
headings,reason=_headings(soups[resource.href],record['target'],record['source_label'])
if headings:
ids=[]
for index,heading in enumerate(headings):
if not heading.get('id'):
heading['id']='title-'+hashlib.sha256((record['target']+str(index)).encode()).hexdigest()[:16]
entry_id=f'{resource.href}#{heading["id"]}'
ids.append(entry_id)
if entry_id not in by_id:
_,text,mapping,_,_=FormatExtractor().extract(str(heading))
by_id[entry_id]=ManifestEntry(entry_id=entry_id,file_path=resource.href,element_id=heading['id'],
original_text=text,placeholders=mapping)
entries.append(by_id[entry_id])
by_id[entry_id].context='navigation-heading: '+record['source_label']
record.update(entry_id=ids[0],entry_ids=ids,heading_id=headings[0]['id'],file_path=resource.href)
else:
entry_id='navigation:'+record['target']
record.update(entry_id=entry_id,heading_id=None,file_path=path,binding_note=reason)
by_id[entry_id]=ManifestEntry(entry_id=entry_id,file_path='__navigation__',element_id=entry_id,
original_text=record['source_label'],context='navigation-label: '+record['source_label'])
entries.append(by_id[entry_id])
records.append(record)
# Contents headings are not link labels but still need the same heading policy.
for href in toc_files:
for h in soups[href].find_all(HEADINGS):
if plain(h.get_text()).casefold() not in {'contents','table of contents','目录'}: continue
if not h.get('id'): h['id']='translation-contents-heading'
entry_id=f'{href}#{h["id"]}'
if entry_id in by_id: continue
_,text,mapping,_,_=FormatExtractor().extract(str(h))
entry=ManifestEntry(entry_id=entry_id,file_path=href,element_id=h['id'],original_text=text,
placeholders=mapping,context='navigation-heading: Contents')
entries.append(entry);by_id[entry_id]=entry
for href,soup in soups.items(): resources[posixpath.normpath(unquote(href))].content=str(soup)
structure.navigation_titles=records
return entries
def normalize_heading_html(record, html):
"""Convert separate Arabic/Roman chapter-number leaves without changing tags."""
prefix=ordinal(record)
if not prefix: return html
soup=parse_html('<div>'+html+'</div>');root=soup.body.div
nodes=[n for n in root.find_all(string=True) if plain(str(n))]
if not nodes: return html
node=nodes[0];text=plain(str(node));compact=re.sub(r'\s+', '', text).strip('::')
# Some publishers put the digit in its own styled span: 第<span>1</span>部.
if (compact=='第' and len(nodes)>=3 and re.fullmatch(r'\d+|[IVXLCDM]+',plain(str(nodes[1])),re.I)
and re.fullmatch(r'[章部篇][::]?',plain(str(nodes[2])))):
nodes[1].replace_with(chinese_number(record['number']))
nodes[2].replace_with('章' if record['role']=='chapter' else '部')
elif re.fullmatch(r'(?:\d+|[IVXLCDM]+|第[零一二三四五六七八九十百千万\d]+[章部篇])',compact,re.I):
node.replace_with(prefix)
if len(nodes)>1 and plain(str(nodes[1])) in {':',':'}: nodes[1].replace_with('')
elif re.match(r'^第[零一二三四五六七八九十百千万\d]+[章部篇]',compact):
node.replace_with(re.sub(r'^第[零一二三四五六七八九十百千万\d]+[章部篇]',prefix,compact))
return root.decode_contents()
def navigation_labels(structure, entries):
by_id={e.entry_id:e for e in entries}; translated={}
for record in structure.navigation_titles:
units=[by_id.get(key) for key in record.get('entry_ids',[record['entry_id']])]
if any(not e or not e.translated_html or e.validation_errors for e in units): continue
units[0].translated_html=normalize_heading_html(record,units[0].translated_html)
label=plain(' '.join(parse_html(e.translated_html).get_text(' ',strip=True) for e in units))
prefix=ordinal(record)
# Standardize punctuation only for the navigation label; the rich
# heading can keep its number and subtitle on separate lines.
if prefix:
label=re.sub(r'^第\s*[零一二三四五六七八九十百千万\d]+\s*[章部篇]',prefix,label)
if prefix and label.startswith(prefix):
tail=label[len(prefix):].strip(' ::.-–—')
label=prefix+(':'+tail if tail else '')
translated[record['target']]=label
return translated
def sync_navigation(structure, entries, mode):
translated=navigation_labels(structure,entries)
if not translated: return
records={r['target']:r for r in structure.navigation_titles}
def display(target):
target_text=translated.get(target)
if not target_text: return None
source=records[target]['source_label']
return source+' / '+target_text if mode=='bilingual' else target_text
for resource in structure.resources.values():
if not resource.content: continue
modified=False
if resource.media_type=='application/x-dtbncx+xml':
doc=etree.fromstring(resource.content.encode(),etree.XMLParser(resolve_entities=False,no_network=True))
for point in doc.xpath('//*[local-name()="navMap"]//*[local-name()="navPoint"]'):
refs=point.xpath('./*[local-name()="content"]/@src')
labels=point.xpath('./*[local-name()="navLabel"]/*[local-name()="text"]')
text=display(destination(resource.href,refs[0])) if refs else None
if labels and text:
labels[0].text=text;modified=True
if modified: resource.content=etree.tostring(doc,encoding='unicode')
elif resource.media_type=='application/xhtml+xml':
soup=parse_html(resource.content)
own = next((display(r['target']) for r in structure.navigation_titles
if r['file_path'] == resource.href and r.get('heading_id') and display(r['target'])), None)
if own and soup.head:
if not soup.head.title: soup.head.append(soup.new_tag('title'))
soup.head.title.string=own; modified=True
if not is_toc(soup,resource.href,resource.properties or ''):
if modified:
resource.content=str(soup); resource.translation_modified=True
continue
for link in soup.find_all('a',href=True):
text=display(destination(resource.href,link['href']))
if not text: continue
# Preserve the link node, its attributes and styled descendants.
nodes=[n for n in link.find_all(string=True) if plain(str(n))]
if nodes:
nodes[0].replace_with(text)
for n in nodes[1:]: n.replace_with('')
else: link.append(NavigableString(text))
link['data-navigation-translated']='true';modified=True
if modified: resource.content=str(soup)
if modified: resource.translation_modified=True
+11 -1
View File
@@ -144,6 +144,7 @@ Output Format:
""" """
id_map = {} # short_id -> entry_id id_map = {} # short_id -> entry_id
lines = [] lines = []
title_notes = []
for i, item in enumerate(items, 1): for i, item in enumerate(items, 1):
short_id = f"#{i}" short_id = f"#{i}"
@@ -151,9 +152,18 @@ Output Format:
# Clean text (remove extra whitespace) # Clean text (remove extra whitespace)
text = re.sub(r'\s+', ' ', item.original_text).strip() text = re.sub(r'\s+', ' ', item.original_text).strip()
if item.context and item.context.startswith('navigation-'):
title_notes.append(f"{short_id}: {item.context}")
lines.append(f"{short_id}: {text}") lines.append(f"{short_id}: {text}")
return id_map, "\n".join(lines) if title_notes:
instructions = ("Title context (instructions only; do not copy into translations):\n"
"Translate Chapter N as 第N章, Part N as 第N部, and translate the title itself. "
"Keep number/subtitle styling tokens in their original regions.\n"
+ "\n".join(title_notes) + "\n\nSource units:\n")
else:
instructions = ""
return id_map, instructions + "\n".join(lines)
def _parse_short_id_response(self, response: str, id_map: Dict[str, str]) -> Dict[str, str]: def _parse_short_id_response(self, response: str, id_map: Dict[str, str]) -> Dict[str, str]:
""" """
+1
View File
@@ -59,6 +59,7 @@ class ManifestManager:
for entry in new_entries: for entry in new_entries:
previous = self._entries_map.get(entry.entry_id) previous = self._entries_map.get(entry.entry_id)
if previous and previous.original_text == entry.original_text and previous.placeholders == entry.placeholders: if previous and previous.original_text == entry.original_text and previous.placeholders == entry.placeholders:
previous.context = entry.context
reconciled.append(previous) reconciled.append(previous)
else: else:
reconciled.append(entry) reconciled.append(entry)
+1 -1
View File
@@ -26,7 +26,7 @@ class Translator:
self.restorer = FormatRestorer() self.restorer = FormatRestorer()
def _fingerprint(self, entry, profile): def _fingerprint(self, entry, profile):
data = [entry.original_text, entry.placeholders, self.glossary.fingerprint(), data = [entry.original_text, entry.placeholders, entry.context, self.glossary.fingerprint(),
getattr(self.llm_client, 'model', ''), profile.style_guide if profile else ''] getattr(self.llm_client, 'model', ''), profile.style_guide if profile else '']
return hashlib.sha256(json.dumps(data, sort_keys=True, ensure_ascii=False).encode()).hexdigest() return hashlib.sha256(json.dumps(data, sort_keys=True, ensure_ascii=False).encode()).hexdigest()
+181
View File
@@ -0,0 +1,181 @@
import asyncio
import zipfile
from bs4 import BeautifulSoup
from ebooklib import epub
from lxml import etree
import pytest
from src.common.data_model import BookStructure
from src.preprocessing.epub_cleaner import EpubCleaner
from src.preprocessing.text_extractor import FineGrainedExtractor
from src.preprocessing.navigation import prepare_navigation, destination, chinese_number
from src.assembly.restoration_engine import RestorationEngine
from src.assembly.backfiller import BackfillEngine
from src.assembly.builder import BilingualBuilder
from src.qa.epub_audit import audit_epub, compare_audits
def fixture(tmp_path):
book = epub.EpubBook(); book.set_identifier('titles'); book.set_title('Title QA'); book.set_language('en')
chapter = epub.EpubHtml(uid='ch', file_name='Text/ch.xhtml', title='Roots')
chapter.content = '''<html><head></head><body><h2 id="ch1"><span class="line">1</span>
<span class="subtitle">ROOTS</span></h2><p id="opening" class="noindent" style="text-indent:0 !important; color:red">Opening prose.</p>
<p id="caption" class="caption">A caption.</p><blockquote><p id="quote">Quoted text.</p></blockquote>
<p id="second">Second paragraph.<a epub:type="noteref" href="notes.xhtml#note">1</a></p></body></html>'''
part = epub.EpubHtml(uid='part', file_name='Text/part.xhtml', title='Part 1')
part.content = '<html><body><h1 id="part"><span>PART 1</span><br/><em>BOY GENIUS</em></h1></body></html>'
contents = epub.EpubHtml(uid='contents', file_name='Text/Contents.xhtml', title='Contents')
contents.content = '''<html><body><h1 id="contents">Contents</h1><ol><li><a href="part.xhtml#part">Part 1: Boy Genius</a>
<ol><li><a href="ch.xhtml#ch1"><em>Chapter 1: Roots</em></a></li></ol></li></ol></body></html>'''
notes = epub.EpubHtml(uid='notes', file_name='Text/notes.xhtml', title='Notes')
notes.content = '<html><body><h1 id="notes">Notes</h1><aside epub:type="footnote" id="note"><p id="note-text">Note text. <a href="ch.xhtml#second">Back</a></p></aside></body></html>'
for item in [chapter,part,contents,notes,epub.EpubNav(),epub.EpubNcx()]: book.add_item(item)
book.toc = [(epub.Section('Part 1: Boy Genius', 'Text/part.xhtml#part'),
[epub.Link('Text/ch.xhtml#ch1','Chapter 1: Roots','ch1')]),
epub.Link('Text/notes.xhtml#notes','Notes','notes')]
book.spine = ['nav',contents,part,chapter,notes]
source=tmp_path/'source.epub';epub.write_epub(str(source),book)
work=tmp_path/'source'; structure=BookStructure.load(EpubCleaner(source,work).clean())
entries=FineGrainedExtractor().extract(structure)
prepare_navigation(source,structure,entries)
for e in entries:
e.translated_text=(e.original_text.replace('ROOTS','根源').replace('PART 1','第一部')
.replace('BOY GENIUS','天才少年').replace('Contents','目录').replace('Notes','注释')
.replace('Opening prose.','开篇正文。').replace('Second paragraph.','第二段正文。')
.replace('A caption.','图注。').replace('Quoted text.','引文。'))
assert asyncio.run(RestorationEngine().restore_entries(entries))==len(entries)
return source,work,structure,entries
@pytest.mark.parametrize('mode',['bilingual','target_only'])
def test_headings_nav_ncx_contents_share_translation_and_keep_links(tmp_path,mode):
source,work,structure,entries=fixture(tmp_path)
chapter_titles=[e for e in entries if e.element_id=='ch1']
assert len(chapter_titles)==1
assert not any('Chapter 1: Roots' in e.original_text for e in entries)
asyncio.run(BackfillEngine().backfill(structure,entries,mode))
# Repeat safely: no duplicate bilingual paragraph or translated TOC label.
asyncio.run(BackfillEngine().backfill(structure,entries,mode))
output=tmp_path/(mode+'.epub');BilingualBuilder(work,source).build(structure,output)
assert not compare_audits(audit_epub(source),audit_epub(output))
with zipfile.ZipFile(source) as before,zipfile.ZipFile(output) as after:
prefix=next(n for n in before.namelist() if n.endswith('/nav.xhtml')).removesuffix('nav.xhtml')
for relative in ['nav.xhtml','Text/Contents.xhtml']:
name=prefix+relative
a=BeautifulSoup(before.read(name),'html.parser');b=BeautifulSoup(after.read(name),'html.parser')
assert [x.get('href') for x in a.find_all('a')]==[x.get('href') for x in b.find_all('a')]
link=next(x for x in b.find_all('a') if 'ch.xhtml' in x.get('href',''))
assert '第一章:根源' in link.get_text()
assert link.get_text().count('根源')==1
assert link.find_parent('ol').find_parent('li') is not None
a=etree.fromstring(before.read(prefix+'toc.ncx'));b=etree.fromstring(after.read(prefix+'toc.ncx'))
assert a.xpath('//*[local-name()="content"]/@src')==b.xpath('//*[local-name()="content"]/@src')
assert a.xpath('//*[local-name()="navPoint"]/@id')==b.xpath('//*[local-name()="navPoint"]/@id')
assert '第一章:根源' in ''.join(b.itertext())
ch=BeautifulSoup(after.read(prefix+'Text/ch.xhtml'),'html.parser')
title=ch.find('h2',class_='translation')
assert title.find('span',class_='line').get_text()=='第一章'
assert title.find('span',class_='subtitle').get_text()=='根源'
assert title['data-translation-role']=='heading'
for key in ['opening','second']:
p=ch.find(attrs={'data-translation-for':key}) if mode=='bilingual' else ch.find(id=key)
assert p['data-translation-role']=='body'
assert 'text-indent' not in p.get('style','')
for key in ['caption','quote']:
p=ch.find(attrs={'data-translation-for':key}) if mode=='bilingual' else ch.find(id=key)
assert p['data-translation-role']=='other'
notes=BeautifulSoup(after.read(prefix+'Text/notes.xhtml'),'html.parser')
assert notes.find(id='note') and notes.find('a',href='ch.xhtml#second')
def test_uncertain_destination_translates_label_without_overwriting_prose(tmp_path):
source,work,structure,entries=fixture(tmp_path)
# A link to prose must not be treated as the chapter heading.
nav=next(r for r in structure.resources.values() if r.href=='nav.xhtml')
nav.content=nav.content.replace('Text/ch.xhtml#ch1','Text/ch.xhtml#opening')
entries=FineGrainedExtractor().extract(structure)
prepare_navigation(source,structure,entries)
record=next(r for r in structure.navigation_titles if r['target'].endswith('#opening'))
assert record['heading_id'] is None
assert record['binding_note']=='no matching heading'
entry=next(e for e in entries if e.entry_id==record['entry_id'])
assert entry.file_path=='__navigation__'
assert 'Opening prose.' in next(r for r in structure.resources.values() if r.href=='Text/ch.xhtml').content
def test_destination_preserves_canonical_fragment_and_ignores_external():
assert destination('Nav/toc.xhtml','../Text/ch%20one.xhtml#part%202')=='Text/ch one.xhtml#part 2'
assert destination('Text/ch.xhtml','#x')=='Text/ch.xhtml#x'
assert destination('toc.xhtml','https://example.com/') is None
assert [chinese_number(x) for x in [1,10,11,20,101,1001]]==['一','十','十一','二十','一百零一','一千零一']
@pytest.mark.parametrize('markup',[
'<h2 id="number">1</h2><h2 id="title">The Bookseller</h2><h2 id="subtitle">The Ren Family: 1937–1968</h2>',
'<p id="number" class="calibre1">1</p><p id="title" class="calibre2"><em>The Bookseller</em></p><p id="subtitle" class="calibre3">The Ren Family: 1937–1968</p>',
'<div id="number"><b>CHAPTER 1</b></div><div id="title"><b>The Bookseller</b></div><div id="subtitle">The Ren Family: 1937–1968</div>',
])
def test_separate_number_title_subtitle_are_shared_without_merging_dom(tmp_path,markup):
from src.preprocessing.navigation import navigation_labels
source,work,structure,entries=fixture(tmp_path)
chapter=next(r for r in structure.resources.values() if r.href=='Text/ch.xhtml')
chapter.content='<html><body>'+markup+'<p id="prose">A long unrelated opening paragraph.</p></body></html>'
nav=next(r for r in structure.resources.values() if r.href=='nav.xhtml')
nav.content=nav.content.replace('Text/ch.xhtml#ch1','Text/ch.xhtml').replace('Chapter 1: Roots','Chapter 1: The Bookseller | The Ren Family: 1937–1968')
entries=FineGrainedExtractor().extract(structure);prepare_navigation(source,structure,entries)
record=next(r for r in structure.navigation_titles if r['target']=='Text/ch.xhtml')
assert len(record['entry_ids'])==3
for e in entries:
if e.element_id in {'number','title','subtitle'}:
e.translated_text=e.original_text.replace('CHAPTER 1','第1章:').replace('The Bookseller','书商').replace('The Ren Family: 1937–1968','任家:1937—1968')
asyncio.run(RestorationEngine().restore_entries(entries))
labels=navigation_labels(structure,entries)
assert labels['Text/ch.xhtml']=='第一章:书商 任家:1937—1968'
asyncio.run(BackfillEngine().backfill(structure,entries,'target_only',allow_partial=True))
ch=BeautifulSoup(chapter.content,'html.parser')
assert all(ch.find(id=id) for id in ['number','title','subtitle','prose'])
assert ch.find(id='prose').get_text()=='A long unrelated opening paragraph.'
def test_heading_number_with_spaces_and_colon_normalizes():
from src.preprocessing.navigation import normalize_heading_html,number_info
assert normalize_heading_html({'role':'chapter','number':1},'<span>第 1 章:</span><em>根源</em>')=='<span>第一章</span><em>根源</em>'
assert number_info('Chapter Eleven: Rebels')==('chapter',11)
assert number_info('Part Two: Coffee')==('part',2)
def test_split_styled_digit_and_separate_separator():
from src.preprocessing.navigation import normalize_heading_html
assert normalize_heading_html({'role':'part','number':1},'第<span>1</span>部:<em>天才少年</em>')=='第<span>一</span>部<em>天才少年</em>'
assert normalize_heading_html({'role':'chapter','number':1},'<span>第1章</span>:<em>根源</em>')=='<span>第一章</span><em>根源</em>'
def test_numbered_titles_starting_with_two_or_i_do_not_lose_title_words():
from src.preprocessing.navigation import title_identity
assert title_identity('Chapter 22: Two Indians Are Better than One')==title_identity('22 TWO INDIANS ARE BETTER THAN ONE')
assert title_identity('Chapter 26: I Hope You Go Bust')==title_identity('26 I HOPE YOU GO BUST')
def test_manifest_reconciliation_updates_title_context(tmp_path):
from src.common.data_model import ManifestEntry
from src.translation.manifest_manager import ManifestManager
manager=ManifestManager(tmp_path/'manifest.json')
old=ManifestEntry(entry_id='x#h',file_path='x',element_id='h',original_text='Roots',context='section',translated_text='根源')
manager.add_entries([old])
new=old.model_copy(update={'context':'navigation-heading: Chapter 1: Roots','translated_text':None})
manager.add_entries([new])
assert manager.entries[0].context==new.context
assert manager.entries[0].translated_text=='根源'
def test_ncx_declared_utf16_is_parsed_without_utf8_assumptions(tmp_path):
source,work,structure,entries=fixture(tmp_path)
with zipfile.ZipFile(source) as z:
items=[(i,z.read(i.filename)) for i in z.infolist()]
converted=tmp_path/'utf16.epub'
with zipfile.ZipFile(converted,'w') as z:
for info,data in items:
if info.filename.endswith('.ncx'):
data=etree.tostring(etree.fromstring(data),encoding='utf-16',xml_declaration=True)
z.writestr(info,data)
prepare_navigation(converted,structure,entries)
assert any(r['source_label']=='Chapter 1: Roots' for r in structure.navigation_titles)