Synchronize chapter and navigation translations with Chinese prose indentation
This commit is contained in:
@@ -1,3 +1,11 @@
|
||||
## 2026-10-08 — Chinese typography and shared navigation titles
|
||||
|
||||
- Indent Chinese body paragraphs by two em, including chapter openings and converted div prose; preserve heading/caption/note layouts.
|
||||
- Bind NAV, NCX and visible contents links to shared chapter/part translations while preserving anchors, hierarchy and resource paths.
|
||||
- Recognize adjacent number/title/subtitle blocks and normalize Chinese chapter/part numbers without merging their markup.
|
||||
- Include heading context in translation cache fingerprints and update old manifest contexts during reconciliation.
|
||||
- Add navigation regression tests and record the scanned PDF/OCR to EPUB roadmap.
|
||||
|
||||
## 2026-10-08 — translation and QA improvements
|
||||
|
||||
- Replace regex tag extraction with balanced DOM placeholders and protect literal φ.
|
||||
|
||||
@@ -26,9 +26,9 @@ cp .env.example .env
|
||||
|
||||
- DOM 遍历保留行内标签,图片、数学公式和脚注作为完整对象保护;占位符须次数正确、嵌套合法,还原后须是合法 XHTML。
|
||||
- 大段初译后,只重试未通过校验的条目。网络错误、重复输出 ID、缺失条目、截断和占位符错误保持待处理状态,不写入“错误译文”。正常运行有失败时终止,不静默生成不完整中文版。
|
||||
- 在原 EPUB 容器上修改正文;保留 OPF、目录、原 CSS 引用、图片、字体和其他资源。双语段落避免复制 ID,中文版保留原锚点;列表和表格不会插入无效兄弟节点。
|
||||
- 在原 EPUB 容器上修改正文;保留 OPF、目录链接及层级、原 CSS 引用、图片、字体和其他资源。双语段落避免复制 ID,中文版保留原锚点;列表和表格不会插入无效兄弟节点。
|
||||
- HTML5 修复处理不闭合标签和重复 ID,抽取不会重复处理嵌套列表/段落;支持裸 div、表格单元、图注和混合容器中的直接文本。
|
||||
- 已翻译条目按原文、占位符、术语表、模型及风格指南指纹复用;每个 chunk 完成就保存。源文件变更会重新预处理。旧版本缓存会升级,无法安全复用的译文需要重新翻译。
|
||||
- 已翻译条目按原文、占位符、标题上下文、术语表、模型及风格指南指纹复用;每个 chunk 完成就保存。源文件变更会重新预处理。旧版本缓存会升级,无法安全复用的译文需要重新翻译。
|
||||
|
||||
`--skip-translation` 用于离线调试/部分样例,不代表全书已翻译。`--force-clean` 重新预处理;`--force-restore` 兼容旧用法,但现在每次回填前都会重新校验,旧的缓存 HTML 不再直接被信任。
|
||||
|
||||
@@ -64,3 +64,13 @@ Mac 沙箱中 Calibre 的 Qt 渲染可能需要允许进程运行。QA 使用项
|
||||
```
|
||||
|
||||
`build_sample.py` 只接受与当前原文和占位符完全对应的条目,输出明确标为部分样例。完整架构见 `doc/architecture_flow.md`;本次测试记录见 `doc/QA_2026-10-08.md`。
|
||||
|
||||
### 中文正文与标题目录
|
||||
|
||||
中文正文(包括章首段)使用 `2em` 首行缩进;标题、图注、引用和注释保持各自布局。原文段落保留出版社样式。中文段落清除原内联 `text-indent`,以免 `!important` 覆盖中文规则。
|
||||
|
||||
预处理以目录的目标文件/锚点建立标题映射。每个实际章标题只翻译一次,NAV、NCX、书内目录及阅读器文档标题使用同一结果;目录 href、章节 ID、层级和顺序保留。章号、主标题、副标题分成多个块时,精确匹配相邻标题块并分别回填,不合并 DOM。中文模式把独立章号统一成“第一章”,Part 对应“第一部”;双语模式保留英文标题和目录标签。
|
||||
|
||||
无法可靠绑定的目录条目仅翻译标签,不改写猜测的章节正文;具体条目记录在 `book_structure.json` 的 `navigation_titles[].binding_note`。这类条目需要检查章节标题是否一致,不能视为全书标题同步已经通过。索引、版权、注释全文仍遵守原有跳过策略;导航中这些页面的名称可以翻译。
|
||||
|
||||
后续扫描 PDF/OCR 大版本的图文、结构与目录方案见 [路线图](doc/OCR_EPUB_ROADMAP.md)。
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
# 扫描 PDF → EPUB:后续大版本
|
||||
|
||||
这是独立的大版本方向,本轮不实现。目标是把旧中文书的扫描 PDF 转成可重排、可搜索、可点击目录的 EPUB,同时保留原书中有阅读意义的结构与图文关系。
|
||||
|
||||
## 输入与评测样本
|
||||
|
||||
先收集少量代表样本:纯文字、竖排/繁体、双栏、跨页脚注、目录、表格、示意图、照片,以及歪斜和低清扫描。再盘点用户本地已部署的 OCR,用相同页面比较文字准确度、阅读顺序、版面坐标、图表识别、速度和资源占用。届时根据实际部署与测试结果选择模型,保留可替换的 OCR 接口。
|
||||
|
||||
## 中间结构
|
||||
|
||||
OCR 输出不能直接拼成 HTML。建立带页面来源、坐标、阅读顺序与置信度的版面中间表示,区分正文段落、章号、主标题、副标题、页眉页脚、脚注/尾注、表格、图片、图注。识别跨页段落合并与断词,保留人工修正记录,以便重复 OCR 时复用。
|
||||
|
||||
## 图文与样式
|
||||
|
||||
从原 PDF 页面或扫描图按区域提取图片、示意图,保留分辨率、裁切来源和对应图注,作为 EPUB 独立资源嵌入。表格优先转为语义表格;复杂或低置信度图表保留原图。可重排 EPUB 保留标题层级、强调、段落、引用和图文关系;精确页面位置有意义的材料另外评估固定版式 EPUB,不能要求同一个可重排版本在每种屏幕上保持 PDF 的坐标。
|
||||
|
||||
## 目录和注释
|
||||
|
||||
把目录条目、章标题、页码映射到中间结构的稳定 ID,采用本项目的“目标锚点 → 标题”映射生成 NAV、NCX 和书内目录。原 PDF 页码用作来源追踪,不直接充当 EPUB 页码。脚注建立正文引用、注释内容及返回链接;低置信度匹配进入待审核队列。
|
||||
|
||||
## QA 闭环与验收
|
||||
|
||||
对同一段原扫描、OCR 中间结果和 EPUB 阅读器截图进行对照。验证阅读顺序、图片/图注归属、标题层级、目录点击、注释双向跳转、字体与中文缩进。每次修复保留样例和回归用例;把文字识别质量、结构正确性与视觉版式分别报告。先完成一小本代表书的端到端试验,再扩展到批量转换。
|
||||
@@ -26,3 +26,25 @@ CLI 主流程使用自制小 EPUB 完成无 API 冒烟验证;新增的输出
|
||||
自动术语扫描依赖模型判断。完整名字和明确别名可约束;同姓人物、间接指称、未列入表的其他译名仍须语言 QA。`exclude_contexts` 与较长术语优先处理能减少误判,不能代替全书语义消歧。
|
||||
|
||||
HTML5 解析可以修复结构,不能自动判断出版者本意;特别的固定版式、复杂 SVG/脚本内容和阅读器差异仍需具体样本。正常主流程遇到新增结构错误会停止并留下报告。原书问题单独留存,不用“没有新增问题”宣称原书符合所有 EPUB 标准。
|
||||
|
||||
## 第二轮:中文缩进与标题目录同步
|
||||
|
||||
- 新增标题目标映射:NAV、NCX 和明确的书内目录链接解析到原文件/锚点;章号、标题、副标题的相邻块精确绑定,各块仍各自翻译和回填。目录复用实际标题译文,不另行生成章标题。
|
||||
- 中文章号统一为“第一章”,Part 对应“第一部”。兼容数字、罗马数字和常见英文数字词,保留数字所在 span 的样式;双语模式保留原英文标题。
|
||||
- 中文正文首行缩进 2em,包括原书使用 noindent 的章首段。清除中文段落的内联 text-indent,英文段落不修改;图注、引文、注释、居中段落和标题排除。Calibre 转换产生的正文 div 也适用。
|
||||
- 41 项离线测试通过,覆盖双语/中文模式、拆分标题、NAV/NCX/书内目录的一致性、链接及层级、反复回填、内联 CSS、标题缓存上下文更新,以及标题首词 Two/I 的误判回归。
|
||||
- 第二轮 11 本书共 27,219 单元、467 个目录目标:原文格式还原失败 0,新增结构问题 0。其中 403 个目标绑定实际标题,64 个不确定目标保留独立标签翻译并记录 binding_note。多数是封面、扉页或装饰标题,仍有复杂章节结构;不声称所有书的标题已经自动同步。Empire of AI 的原有 1 项源文件问题仍记录在报告中。
|
||||
|
||||
### Luna 样书与视觉范围
|
||||
|
||||
Luna 翻译了 Gambling Man 的 45 个标题单元,45/45 占位符校验通过;沿用第一轮 10 段正文,合计 55/1,582 单元。本轮没有做全书正文翻译,也没有验证实际外部 API 的整本吞吐。封面、扉页和版权这三个导航标签采用独立译文,其余目录目标均可绑定章节/部标题。
|
||||
|
||||
最终文件位于 `.work/navigation-final-v4/`:`bilingual-sample.epub`、`target_only-sample.epub`、完整 manifest,以及 `qa-chapter/` 和 `qa-bilingual/` 对照证据。这些局部 QA 样例不提交到 Git。
|
||||
|
||||
实际检查了中文版第 1–3 页、双语版第 1–4 页与相应原文页面:部标题、章号、主标题、中文首段/后续段缩进、英文 drop cap、上下标、强调与正文排列没有重叠或截断。PDF 页面是辅助视图;中文 EPUB 另外在 Calibre 9.15 阅读器中打开,确认中文目录“第一章:根源”跳转到对应章节、第一部到第五部层级保留、第一条注释弹窗内容正确。继续进入尾注页,点击第一章第 1 条尾注的返回链接,阅读器显示对应中文正文弹窗,再返回正文主视图;原锚点及返回链接正常。
|
||||
|
||||
脚注/尾注全文仍按原有提取策略跳过;本轮翻译的是其目录名称和目录所指的页面标题。尾注内部按章分组的标题仍可能是英文。这一范围不能误报为注释全文已译。
|
||||
|
||||
渲染期间有一次 Calibre PDF 进程 SIGSEGV,重新生成后的最终两组输出成功。自动报告仍为 visual_review: pending,人工/agent 观察记录单独保存为 visual-review.json;截图生成本身不表示整本视觉验收通过。
|
||||
|
||||
OCR、扫描页图文提取、版面中间结构与可点击目录的后续大版本方案见 `doc/OCR_EPUB_ROADMAP.md`。
|
||||
|
||||
@@ -12,6 +12,7 @@ from src.common.config import load_global_config
|
||||
from src.preprocessing.epub_cleaner import EpubCleaner
|
||||
from src.preprocessing.profiler import BookProfiler
|
||||
from src.preprocessing.text_extractor import FineGrainedExtractor
|
||||
from src.preprocessing.navigation import prepare_navigation
|
||||
from src.translation.manifest_manager import ManifestManager
|
||||
from src.translation.translator_engine import Translator
|
||||
from src.translation.glossary import prepare_glossary
|
||||
@@ -65,7 +66,8 @@ async def run_pipeline(args):
|
||||
if work["structure"].exists() and not args.force_clean:
|
||||
logger.info(f"Reusing existing book_structure: {work['structure']}")
|
||||
structure = BookStructure.load(work["structure"])
|
||||
if structure.source_sha256 != hashlib.sha256(input_path.read_bytes()).hexdigest():
|
||||
if (structure.preprocessing_version < 3 or
|
||||
structure.source_sha256 != hashlib.sha256(input_path.read_bytes()).hexdigest()):
|
||||
logger.info("Source changed or old preprocessing cache; re-cleaning")
|
||||
structure = BookStructure.load(EpubCleaner(input_path, work["root"]).clean())
|
||||
else:
|
||||
@@ -77,6 +79,8 @@ async def run_pipeline(args):
|
||||
# 2. Extraction
|
||||
extractor = FineGrainedExtractor()
|
||||
manifest_entries = extractor.extract(structure)
|
||||
prepare_navigation(input_path,structure,manifest_entries)
|
||||
structure.save(work["structure"])
|
||||
|
||||
# 3. Manifest Management
|
||||
manifest_manager = ManifestManager(work["manifest"])
|
||||
|
||||
@@ -9,6 +9,7 @@ sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
|
||||
|
||||
from src.preprocessing.epub_cleaner import EpubCleaner
|
||||
from src.preprocessing.text_extractor import FineGrainedExtractor
|
||||
from src.preprocessing.navigation import prepare_navigation
|
||||
from src.translation.manifest_manager import ManifestManager
|
||||
from src.common.data_model import BookStructure
|
||||
from src.common.utils import setup_logger, ensure_directory
|
||||
@@ -48,6 +49,8 @@ def run_preprocess(args):
|
||||
logger.info("Extracting text segments...")
|
||||
extractor = FineGrainedExtractor()
|
||||
entries = extractor.extract(structure)
|
||||
prepare_navigation(input_path,structure,entries)
|
||||
structure.save(structure_path)
|
||||
|
||||
# 3. Update Manifest
|
||||
logger.info(f"Updating manifest: {manifest_path}")
|
||||
|
||||
@@ -8,6 +8,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
from src.common.data_model import BookStructure, ManifestEntry
|
||||
from src.preprocessing.epub_cleaner import EpubCleaner
|
||||
from src.preprocessing.text_extractor import FineGrainedExtractor
|
||||
from src.preprocessing.navigation import prepare_navigation
|
||||
from src.assembly.restoration_engine import RestorationEngine
|
||||
from src.assembly.backfiller import BackfillEngine
|
||||
from src.assembly.builder import BilingualBuilder
|
||||
@@ -17,6 +18,7 @@ async def run(args):
|
||||
args.out.mkdir(parents=True,exist_ok=True)
|
||||
structure=BookStructure.load(EpubCleaner(args.source,args.out/'source').clean())
|
||||
entries=FineGrainedExtractor().extract(structure)
|
||||
prepare_navigation(args.source,structure,entries)
|
||||
by_id={e.entry_id:e for e in entries}
|
||||
provided=[ManifestEntry.model_validate(e) for e in json.loads(args.manifest.read_text())]
|
||||
for sample in provided:
|
||||
|
||||
@@ -6,6 +6,7 @@ from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
from src.preprocessing.epub_cleaner import EpubCleaner
|
||||
from src.preprocessing.text_extractor import FineGrainedExtractor
|
||||
from src.preprocessing.navigation import prepare_navigation
|
||||
from src.common.data_model import BookStructure
|
||||
from src.assembly.format_restorer import FormatRestorer
|
||||
from src.assembly.builder import BilingualBuilder
|
||||
@@ -23,10 +24,12 @@ def main():
|
||||
work=args.out/source.stem
|
||||
structure=BookStructure.load(EpubCleaner(source,work).clean())
|
||||
entries=FineGrainedExtractor().extract(structure)
|
||||
prepare_navigation(source,structure,entries)
|
||||
bad=[e.entry_id for e in entries if not FormatRestorer().restore(e.original_text,e.placeholders,original_text=e.original_text)[1]]
|
||||
output=BilingualBuilder(work,source).build(structure,work/'roundtrip.epub')
|
||||
before,after=audit_epub(source),audit_epub(output)
|
||||
row.update(entries=len(entries),format_failures=bad,source_issues=len(before['issues']),
|
||||
row.update(entries=len(entries),navigation_titles=len(structure.navigation_titles),
|
||||
unbound_titles=[r["source_label"] for r in structure.navigation_titles if r.get("binding_note")],format_failures=bad,source_issues=len(before['issues']),
|
||||
new_issues=compare_audits(before,after))
|
||||
except Exception as exc:
|
||||
row['error']=str(exc)
|
||||
|
||||
@@ -96,7 +96,7 @@ def main():
|
||||
'scope': 'provided manifest only'}
|
||||
if errors and not args.allow_partial:
|
||||
report['new_issues'].append({'kind': 'incomplete_translation', 'detail': str(len(errors))})
|
||||
chapter_hrefs = args.chapter or list(dict.fromkeys(e.file_path for e in entries if e.translated_text))
|
||||
chapter_hrefs = args.chapter or list(dict.fromkeys(e.file_path for e in entries if e.translated_text and e.file_path != "__navigation__"))
|
||||
if not args.chapter and args.max_chapters:
|
||||
chapter_hrefs = chapter_hrefs[:args.max_chapters]
|
||||
report['sampled_chapters'] = chapter_hrefs
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
import copy
|
||||
import re
|
||||
from src.common.html import parse_html
|
||||
from src.preprocessing.navigation import normalize_heading_html, sync_navigation
|
||||
from src.assembly.format_restorer import FormatRestorer
|
||||
from src.common.utils import setup_logger, add_spacing
|
||||
|
||||
@@ -10,6 +12,7 @@ class BackfillEngine:
|
||||
async def backfill(self, structure, manifest_entries, mode='bilingual', allow_partial=False):
|
||||
if mode not in {'bilingual', 'target_only'}:
|
||||
raise ValueError(f'Unknown output mode: {mode}')
|
||||
titles = {r["entry_id"]: r for r in structure.navigation_titles}
|
||||
by_file = {}
|
||||
failures = []
|
||||
applied = 0
|
||||
@@ -21,9 +24,16 @@ class BackfillEngine:
|
||||
restored, ok = FormatRestorer().restore(add_spacing(entry.translated_text), entry.placeholders,
|
||||
original_text=entry.original_text)
|
||||
if not ok:
|
||||
entry.translated_html = None
|
||||
entry.validation_errors = ["invalid restored XHTML"]
|
||||
failures.append(entry.entry_id)
|
||||
continue
|
||||
entry.translated_html = restored
|
||||
record = titles.get(entry.entry_id)
|
||||
if record:
|
||||
entry.translated_html=normalize_heading_html(record,restored)
|
||||
if entry.file_path=='__navigation__':
|
||||
continue
|
||||
by_file.setdefault(entry.file_path, []).append(entry)
|
||||
if failures and not allow_partial:
|
||||
raise ValueError(f"Backfill rejected {len(failures)} unvalidated entries: {failures[:3]}")
|
||||
@@ -49,6 +59,21 @@ class BackfillEngine:
|
||||
new_tag['xml:lang'] = 'zh-CN'
|
||||
classes = list(new_tag.get('class', []))
|
||||
new_tag['class'] = classes + ['translation']
|
||||
heading = entry.context and (entry.context.startswith('navigation-heading:') or entry.context in {'chapter','section','subsection'})
|
||||
special = re.search(r'caption|footnote|endnote|epigraph|verse|poem|center|attribution|dedication', ' '.join(element.get('class', [])), re.I)
|
||||
prose = ((tag_name in {'p','div'} or (tag_name=='span' and element.get('data-translation-unit')))
|
||||
and not heading and entry.context != 'epigraph'
|
||||
and not special and not re.search(r'text-align\s*:\s*(?:center|right)', element.get('style',''),re.I)
|
||||
and not element.find_parent(['blockquote', 'figure', 'aside'])
|
||||
and not element.find_parent(attrs={'epub:type': re.compile(r'footnote|endnote')}))
|
||||
# Chinese body paragraphs use two em even when the publisher's
|
||||
# opening-paragraph class sets text-indent:0.
|
||||
new_tag['data-translation-role'] = 'body' if prose else 'heading' if heading else 'other'
|
||||
if prose:
|
||||
# Publisher inline !important rules otherwise outrank CSS.
|
||||
style = re.sub(r'(?:^|;)\s*text-indent\s*:[^;]*', '', new_tag.get('style',''),flags=re.I).strip('; ')
|
||||
if style: new_tag['style'] = style
|
||||
else: new_tag.attrs.pop('style', None)
|
||||
applied += 1
|
||||
fragment = parse_html(f'<div>{entry.translated_html}</div>').body.div
|
||||
for node in list(fragment.contents):
|
||||
@@ -63,6 +88,8 @@ class BackfillEngine:
|
||||
else:
|
||||
element.replace_with(new_tag)
|
||||
resource.content = str(soup)
|
||||
resource.translation_modified = True
|
||||
sync_navigation(structure, manifest_entries, mode)
|
||||
if mode == 'target_only' and applied and not allow_partial:
|
||||
structure.metadata.language = 'zh-CN'
|
||||
return structure
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
/* Preserve the book's alignment, indent and inline font choices. */
|
||||
/* Preserve publisher alignment and inline fonts; apply Chinese prose indentation. */
|
||||
.translation {
|
||||
line-height: 1.6;
|
||||
clear: both;
|
||||
@@ -17,3 +17,8 @@ p.translation::first-letter {
|
||||
color: inherit !important;
|
||||
float: none !important;
|
||||
}
|
||||
/* Prose gets two Chinese character widths, including chapter-opening prose.
|
||||
Headings, captions, quotations and note markers keep their own layout. */
|
||||
.translation[data-translation-role="body"] {
|
||||
text-indent: 2em !important;
|
||||
}
|
||||
|
||||
@@ -37,15 +37,20 @@ def write_preserving_package(original, structure, output):
|
||||
etree.SubElement(manifest, f'{{{OPF}}}item', id=css_id, href=css_href, **{'media-type': 'text/css'})
|
||||
replacements[css_archive_path] = (Path(__file__).parent/'bilingual_style.css').read_bytes()
|
||||
for item_id, resource in structure.resources.items():
|
||||
if resource.media_type != 'application/xhtml+xml' or not resource.content:
|
||||
if resource.media_type not in {'application/xhtml+xml','application/x-dtbncx+xml'} or not resource.content:
|
||||
continue
|
||||
if 'nav' in (items.get(item_id).get('properties', '') if item_id in items else '').split() and 'translation-for' not in resource.content and 'class="translation"' not in resource.content:
|
||||
if 'nav' in (items.get(item_id).get('properties', '') if item_id in items else '').split() and not resource.translation_modified and 'translation-for' not in resource.content and 'class="translation"' not in resource.content:
|
||||
continue
|
||||
if item_id not in items:
|
||||
raise ValueError(f'Unknown package item: {item_id}')
|
||||
archive_path = posixpath.normpath(posixpath.join(base, unquote(items[item_id].get('href'))))
|
||||
if archive_path not in source.namelist():
|
||||
raise ValueError(f'Missing source resource: {archive_path}')
|
||||
if resource.media_type=='application/x-dtbncx+xml':
|
||||
if resource.translation_modified:
|
||||
etree.fromstring(resource.content.encode(),etree.XMLParser(resolve_entities=False,no_network=True))
|
||||
replacements[archive_path]=resource.content.encode()
|
||||
continue
|
||||
soup = parse_html(resource.content)
|
||||
soup.html['xmlns'] = 'http://www.w3.org/1999/xhtml'
|
||||
soup.html['xmlns:epub'] = 'http://www.idpf.org/2007/ops'
|
||||
|
||||
@@ -28,11 +28,13 @@ class ResourceItem(BaseModel):
|
||||
content: Optional[str] = None # For text/html
|
||||
file_path: Optional[str] = None # For binary/assets (relative to assets dir)
|
||||
properties: Optional[str] = None
|
||||
translation_modified: bool = False
|
||||
|
||||
class BookStructure(BaseModel):
|
||||
source_sha256: Optional[str] = None
|
||||
preprocessing_version: int = 2
|
||||
preprocessing_version: int = 3
|
||||
metadata: BookMetaData
|
||||
navigation_titles: List[Dict[str, Any]] = Field(default_factory=list)
|
||||
spine: List[str] = Field(default_factory=list, description="Ordered list of item IDs in spine")
|
||||
resources: Dict[str, ResourceItem] = Field(default_factory=dict, description="Map of item_id to ResourceItem")
|
||||
|
||||
|
||||
@@ -0,0 +1,296 @@
|
||||
"""One title translation per destination, shared by headings, NAV, NCX and TOC pages.
|
||||
|
||||
Original href/src, anchors and hierarchy are never rebuilt from translated text.
|
||||
Uncertain heading matches are recorded rather than replacing arbitrary prose.
|
||||
"""
|
||||
import hashlib
|
||||
import posixpath
|
||||
import re
|
||||
import zipfile
|
||||
from urllib.parse import unquote, urlsplit
|
||||
from bs4 import NavigableString
|
||||
from lxml import etree
|
||||
from src.common.html import parse_html
|
||||
from src.common.data_model import ManifestEntry
|
||||
from src.preprocessing.format_extractor import FormatExtractor
|
||||
from src.assembly.package_writer import package_path
|
||||
|
||||
HEADINGS = ['h1', 'h2', 'h3', 'h4', 'h5', 'h6']
|
||||
WORDS = {word: i for i, word in enumerate('zero one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen twenty'.split())}
|
||||
NUMBER = r'(?:\d+|[ivxlcdm]+|' + '|'.join(WORDS) + ')'
|
||||
|
||||
|
||||
def destination(document, href):
|
||||
parts = urlsplit(href)
|
||||
if parts.scheme or parts.netloc:
|
||||
return None
|
||||
path = (posixpath.normpath(posixpath.join(posixpath.dirname(document), unquote(parts.path)))
|
||||
if parts.path else document)
|
||||
return path + ('#' + unquote(parts.fragment) if parts.fragment else '')
|
||||
|
||||
|
||||
def plain(text):
|
||||
return re.sub(r'\s+', ' ', text).strip()
|
||||
|
||||
|
||||
def is_toc(soup, href, properties=''):
|
||||
return ('nav' in properties.split() or soup.find(attrs={'role': 'doc-toc'}) is not None
|
||||
or any('toc' in str(t.get('epub:type', '')).split() for t in soup.find_all('nav'))
|
||||
or re.search(r'(?:^|[/_\-\d])(toc|contents|nav)(?:[._\-]|$)', href, re.I) is not None
|
||||
or any(plain(h.get_text()).casefold() in {'contents', 'table of contents', '目录'}
|
||||
for h in soup.find_all(HEADINGS)))
|
||||
|
||||
|
||||
def title_identity(label):
|
||||
label = plain(label)
|
||||
label, count = re.subn(r'^(?:chapter|chap\.?|part|book|section)\s+' + NUMBER + r'\b\s*[:.|\-–—]?\s*', '', label, count=1, flags=re.I)
|
||||
if not count:
|
||||
label = re.sub(r'^' + NUMBER + r'\b\s*[.:|\-–—]?\s*', '', label, count=1, flags=re.I)
|
||||
return re.sub(r'[^\w]', '', label.casefold())
|
||||
|
||||
|
||||
def number_info(label):
|
||||
match = re.match(r'^(chapter|chap\.?|part|book)\s+(' + NUMBER + r')\b', plain(label), re.I)
|
||||
if match:
|
||||
role = 'chapter' if match[1].lower().startswith('chap') else 'part'
|
||||
number = match[2]
|
||||
else:
|
||||
match = re.match(r'^(\d+)\b', plain(label))
|
||||
if not match: return None, None
|
||||
role, number = 'chapter', match[1]
|
||||
if number.isdigit():
|
||||
value = int(number)
|
||||
elif number.casefold() in WORDS:
|
||||
value = WORDS[number.casefold()]
|
||||
else:
|
||||
values = {'I':1,'V':5,'X':10,'L':50,'C':100,'D':500,'M':1000}
|
||||
value, previous = 0, 0
|
||||
for char in reversed(number.upper()):
|
||||
current=values[char]; value += -current if current < previous else current; previous=max(previous,current)
|
||||
return role, value
|
||||
|
||||
|
||||
def chinese_number(number):
|
||||
if number < 0 or number >= 10000:
|
||||
return str(number)
|
||||
if number == 0:
|
||||
return '零'
|
||||
digits='零一二三四五六七八九'; result=''; zero=False
|
||||
for divisor,unit in [(1000,'千'),(100,'百'),(10,'十'),(1,'')]:
|
||||
digit,number=divmod(number,divisor)
|
||||
if digit:
|
||||
if zero: result+='零'
|
||||
if not (digit==1 and divisor==10 and not result): result+=digits[digit]
|
||||
result+=unit;zero=False
|
||||
elif result and number:
|
||||
zero=True
|
||||
return result
|
||||
|
||||
|
||||
def ordinal(record):
|
||||
if record.get('number') is None:
|
||||
return None
|
||||
return '第'+chinese_number(record['number'])+('章' if record['role']=='chapter' else '部')
|
||||
|
||||
|
||||
def _headings(soup, target, label):
|
||||
fragment = target.partition('#')[2]
|
||||
anchor = soup.find(id=fragment) if fragment else None
|
||||
if fragment and anchor is None:
|
||||
return [], 'missing anchor'
|
||||
identity = title_identity(label)
|
||||
# Only inspect the opening leaf blocks or blocks at the explicit anchor.
|
||||
# An exact match across adjacent blocks also recognizes Calibre-converted
|
||||
# number/title/subtitle paragraphs without guessing from font sizes.
|
||||
blocks = soup.find_all(HEADINGS + ['p', 'div'])
|
||||
blocks = [t for t in blocks if not t.find(HEADINGS + ['p','div','section','blockquote','li'])]
|
||||
if anchor:
|
||||
nearby = [anchor] + anchor.find_all_next(HEADINGS + ['p','div'], limit=16)
|
||||
block_ids = {id(t) for t in blocks}
|
||||
blocks = [t for t in nearby if id(t) in block_ids]
|
||||
blocks = [t for t in blocks if plain(t.get_text())][:12]
|
||||
matches = []
|
||||
for start in range(min(len(blocks), 6)):
|
||||
for end in range(start+1, min(start+4, len(blocks))+1):
|
||||
group = blocks[start:end]
|
||||
if any(len(plain(t.get_text())) > 300 for t in group): break
|
||||
combined = ' '.join(t.get_text(' ', strip=True) for t in group)
|
||||
if title_identity(combined) == identity and identity:
|
||||
matches.append(group)
|
||||
# A number leaf and the subtitle alone can both match after stripping a
|
||||
# numeric prefix. Prefer the complete adjacent title group.
|
||||
if matches:
|
||||
longest = max(len(g) for g in matches)
|
||||
matches = [g for g in matches if len(g) == longest]
|
||||
if len(matches) == 1: return matches[0], None
|
||||
return [], 'ambiguous heading'
|
||||
# An explicit heading anchor is authoritative even if the publisher uses
|
||||
# an abbreviated TOC label; a prose anchor never receives this fallback.
|
||||
if anchor and anchor.name in HEADINGS: return [anchor], None
|
||||
return [], 'no matching heading'
|
||||
|
||||
|
||||
def prepare_navigation(original, structure, entries):
|
||||
"""Enrich the manifest with canonical heading entries, never translate TOC duplicates."""
|
||||
soups={r.href:parse_html(r.content) for r in structure.resources.values()
|
||||
if r.media_type=='application/xhtml+xml' and r.content}
|
||||
resources={posixpath.normpath(unquote(r.href)):r for r in structure.resources.values()}
|
||||
labels={}
|
||||
def collect(document, href, label):
|
||||
target=destination(document,href)
|
||||
if target and plain(label):
|
||||
record=labels.setdefault(target,{'target':target,'source_label':plain(label),'source_variants':[]})
|
||||
if plain(label) not in record['source_variants']: record['source_variants'].append(plain(label))
|
||||
# EPUB3 NAV and explicit contents pages take priority over legacy NCX names.
|
||||
toc_files=set()
|
||||
for href,soup in soups.items():
|
||||
if not is_toc(soup,href,resources[posixpath.normpath(unquote(href))].properties or ''):
|
||||
continue
|
||||
toc_files.add(href)
|
||||
toc_containers=[n for n in soup.find_all('nav') if 'toc' in str(n.get('epub:type','')).split() or n.get('role')=='doc-toc']
|
||||
containers=toc_containers or [soup.body]
|
||||
for container in containers:
|
||||
for a in container.find_all('a',href=True): collect(href,a['href'],a.get_text(' ',strip=True))
|
||||
with zipfile.ZipFile(original) as source:
|
||||
base=posixpath.dirname(package_path(source))
|
||||
for resource in structure.resources.values():
|
||||
if resource.media_type!='application/x-dtbncx+xml': continue
|
||||
raw=source.read(posixpath.normpath(posixpath.join(base,unquote(resource.href))))
|
||||
doc=etree.fromstring(raw,etree.XMLParser(resolve_entities=False,no_network=True))
|
||||
for point in doc.xpath('//*[local-name()="navMap"]//*[local-name()="navPoint"]'):
|
||||
content=point.xpath('./*[local-name()="content"]/@src')
|
||||
text=point.xpath('./*[local-name()="navLabel"]/*[local-name()="text"]')
|
||||
if content and text: collect(resource.href,content[0],''.join(text[0].itertext()))
|
||||
# NCX stays XML; it is not passed through an HTML parser.
|
||||
resource.content=etree.tostring(doc,encoding='unicode')
|
||||
# Ordinary manifest extraction must not independently translate visible
|
||||
# TOC text. Those links are synchronized by target after title translation.
|
||||
entries[:]=[e for e in entries if e.file_path not in toc_files]
|
||||
by_id={e.entry_id:e for e in entries}; records=[]
|
||||
for record in labels.values():
|
||||
path=record['target'].partition('#')[0]; resource=resources.get(path)
|
||||
role,number=number_info(record['source_label']);record.update(role=role,number=number)
|
||||
headings=[];reason='missing document'
|
||||
if resource and resource.href in soups:
|
||||
headings,reason=_headings(soups[resource.href],record['target'],record['source_label'])
|
||||
if headings:
|
||||
ids=[]
|
||||
for index,heading in enumerate(headings):
|
||||
if not heading.get('id'):
|
||||
heading['id']='title-'+hashlib.sha256((record['target']+str(index)).encode()).hexdigest()[:16]
|
||||
entry_id=f'{resource.href}#{heading["id"]}'
|
||||
ids.append(entry_id)
|
||||
if entry_id not in by_id:
|
||||
_,text,mapping,_,_=FormatExtractor().extract(str(heading))
|
||||
by_id[entry_id]=ManifestEntry(entry_id=entry_id,file_path=resource.href,element_id=heading['id'],
|
||||
original_text=text,placeholders=mapping)
|
||||
entries.append(by_id[entry_id])
|
||||
by_id[entry_id].context='navigation-heading: '+record['source_label']
|
||||
record.update(entry_id=ids[0],entry_ids=ids,heading_id=headings[0]['id'],file_path=resource.href)
|
||||
else:
|
||||
entry_id='navigation:'+record['target']
|
||||
record.update(entry_id=entry_id,heading_id=None,file_path=path,binding_note=reason)
|
||||
by_id[entry_id]=ManifestEntry(entry_id=entry_id,file_path='__navigation__',element_id=entry_id,
|
||||
original_text=record['source_label'],context='navigation-label: '+record['source_label'])
|
||||
entries.append(by_id[entry_id])
|
||||
records.append(record)
|
||||
# Contents headings are not link labels but still need the same heading policy.
|
||||
for href in toc_files:
|
||||
for h in soups[href].find_all(HEADINGS):
|
||||
if plain(h.get_text()).casefold() not in {'contents','table of contents','目录'}: continue
|
||||
if not h.get('id'): h['id']='translation-contents-heading'
|
||||
entry_id=f'{href}#{h["id"]}'
|
||||
if entry_id in by_id: continue
|
||||
_,text,mapping,_,_=FormatExtractor().extract(str(h))
|
||||
entry=ManifestEntry(entry_id=entry_id,file_path=href,element_id=h['id'],original_text=text,
|
||||
placeholders=mapping,context='navigation-heading: Contents')
|
||||
entries.append(entry);by_id[entry_id]=entry
|
||||
for href,soup in soups.items(): resources[posixpath.normpath(unquote(href))].content=str(soup)
|
||||
structure.navigation_titles=records
|
||||
return entries
|
||||
|
||||
|
||||
def normalize_heading_html(record, html):
|
||||
"""Convert separate Arabic/Roman chapter-number leaves without changing tags."""
|
||||
prefix=ordinal(record)
|
||||
if not prefix: return html
|
||||
soup=parse_html('<div>'+html+'</div>');root=soup.body.div
|
||||
nodes=[n for n in root.find_all(string=True) if plain(str(n))]
|
||||
if not nodes: return html
|
||||
node=nodes[0];text=plain(str(node));compact=re.sub(r'\s+', '', text).strip('::')
|
||||
# Some publishers put the digit in its own styled span: 第<span>1</span>部.
|
||||
if (compact=='第' and len(nodes)>=3 and re.fullmatch(r'\d+|[IVXLCDM]+',plain(str(nodes[1])),re.I)
|
||||
and re.fullmatch(r'[章部篇][::]?',plain(str(nodes[2])))):
|
||||
nodes[1].replace_with(chinese_number(record['number']))
|
||||
nodes[2].replace_with('章' if record['role']=='chapter' else '部')
|
||||
elif re.fullmatch(r'(?:\d+|[IVXLCDM]+|第[零一二三四五六七八九十百千万\d]+[章部篇])',compact,re.I):
|
||||
node.replace_with(prefix)
|
||||
if len(nodes)>1 and plain(str(nodes[1])) in {':',':'}: nodes[1].replace_with('')
|
||||
elif re.match(r'^第[零一二三四五六七八九十百千万\d]+[章部篇]',compact):
|
||||
node.replace_with(re.sub(r'^第[零一二三四五六七八九十百千万\d]+[章部篇]',prefix,compact))
|
||||
return root.decode_contents()
|
||||
|
||||
|
||||
def navigation_labels(structure, entries):
|
||||
by_id={e.entry_id:e for e in entries}; translated={}
|
||||
for record in structure.navigation_titles:
|
||||
units=[by_id.get(key) for key in record.get('entry_ids',[record['entry_id']])]
|
||||
if any(not e or not e.translated_html or e.validation_errors for e in units): continue
|
||||
units[0].translated_html=normalize_heading_html(record,units[0].translated_html)
|
||||
label=plain(' '.join(parse_html(e.translated_html).get_text(' ',strip=True) for e in units))
|
||||
prefix=ordinal(record)
|
||||
# Standardize punctuation only for the navigation label; the rich
|
||||
# heading can keep its number and subtitle on separate lines.
|
||||
if prefix:
|
||||
label=re.sub(r'^第\s*[零一二三四五六七八九十百千万\d]+\s*[章部篇]',prefix,label)
|
||||
if prefix and label.startswith(prefix):
|
||||
tail=label[len(prefix):].strip(' ::.-–—')
|
||||
label=prefix+(':'+tail if tail else '')
|
||||
translated[record['target']]=label
|
||||
return translated
|
||||
|
||||
|
||||
def sync_navigation(structure, entries, mode):
|
||||
translated=navigation_labels(structure,entries)
|
||||
if not translated: return
|
||||
records={r['target']:r for r in structure.navigation_titles}
|
||||
def display(target):
|
||||
target_text=translated.get(target)
|
||||
if not target_text: return None
|
||||
source=records[target]['source_label']
|
||||
return source+' / '+target_text if mode=='bilingual' else target_text
|
||||
for resource in structure.resources.values():
|
||||
if not resource.content: continue
|
||||
modified=False
|
||||
if resource.media_type=='application/x-dtbncx+xml':
|
||||
doc=etree.fromstring(resource.content.encode(),etree.XMLParser(resolve_entities=False,no_network=True))
|
||||
for point in doc.xpath('//*[local-name()="navMap"]//*[local-name()="navPoint"]'):
|
||||
refs=point.xpath('./*[local-name()="content"]/@src')
|
||||
labels=point.xpath('./*[local-name()="navLabel"]/*[local-name()="text"]')
|
||||
text=display(destination(resource.href,refs[0])) if refs else None
|
||||
if labels and text:
|
||||
labels[0].text=text;modified=True
|
||||
if modified: resource.content=etree.tostring(doc,encoding='unicode')
|
||||
elif resource.media_type=='application/xhtml+xml':
|
||||
soup=parse_html(resource.content)
|
||||
own = next((display(r['target']) for r in structure.navigation_titles
|
||||
if r['file_path'] == resource.href and r.get('heading_id') and display(r['target'])), None)
|
||||
if own and soup.head:
|
||||
if not soup.head.title: soup.head.append(soup.new_tag('title'))
|
||||
soup.head.title.string=own; modified=True
|
||||
if not is_toc(soup,resource.href,resource.properties or ''):
|
||||
if modified:
|
||||
resource.content=str(soup); resource.translation_modified=True
|
||||
continue
|
||||
for link in soup.find_all('a',href=True):
|
||||
text=display(destination(resource.href,link['href']))
|
||||
if not text: continue
|
||||
# Preserve the link node, its attributes and styled descendants.
|
||||
nodes=[n for n in link.find_all(string=True) if plain(str(n))]
|
||||
if nodes:
|
||||
nodes[0].replace_with(text)
|
||||
for n in nodes[1:]: n.replace_with('')
|
||||
else: link.append(NavigableString(text))
|
||||
link['data-navigation-translated']='true';modified=True
|
||||
if modified: resource.content=str(soup)
|
||||
if modified: resource.translation_modified=True
|
||||
@@ -144,6 +144,7 @@ Output Format:
|
||||
"""
|
||||
id_map = {} # short_id -> entry_id
|
||||
lines = []
|
||||
title_notes = []
|
||||
|
||||
for i, item in enumerate(items, 1):
|
||||
short_id = f"#{i}"
|
||||
@@ -151,9 +152,18 @@ Output Format:
|
||||
|
||||
# Clean text (remove extra whitespace)
|
||||
text = re.sub(r'\s+', ' ', item.original_text).strip()
|
||||
if item.context and item.context.startswith('navigation-'):
|
||||
title_notes.append(f"{short_id}: {item.context}")
|
||||
lines.append(f"{short_id}: {text}")
|
||||
|
||||
return id_map, "\n".join(lines)
|
||||
if title_notes:
|
||||
instructions = ("Title context (instructions only; do not copy into translations):\n"
|
||||
"Translate Chapter N as 第N章, Part N as 第N部, and translate the title itself. "
|
||||
"Keep number/subtitle styling tokens in their original regions.\n"
|
||||
+ "\n".join(title_notes) + "\n\nSource units:\n")
|
||||
else:
|
||||
instructions = ""
|
||||
return id_map, instructions + "\n".join(lines)
|
||||
|
||||
def _parse_short_id_response(self, response: str, id_map: Dict[str, str]) -> Dict[str, str]:
|
||||
"""
|
||||
|
||||
@@ -59,6 +59,7 @@ class ManifestManager:
|
||||
for entry in new_entries:
|
||||
previous = self._entries_map.get(entry.entry_id)
|
||||
if previous and previous.original_text == entry.original_text and previous.placeholders == entry.placeholders:
|
||||
previous.context = entry.context
|
||||
reconciled.append(previous)
|
||||
else:
|
||||
reconciled.append(entry)
|
||||
|
||||
@@ -26,7 +26,7 @@ class Translator:
|
||||
self.restorer = FormatRestorer()
|
||||
|
||||
def _fingerprint(self, entry, profile):
|
||||
data = [entry.original_text, entry.placeholders, self.glossary.fingerprint(),
|
||||
data = [entry.original_text, entry.placeholders, entry.context, self.glossary.fingerprint(),
|
||||
getattr(self.llm_client, 'model', ''), profile.style_guide if profile else '']
|
||||
return hashlib.sha256(json.dumps(data, sort_keys=True, ensure_ascii=False).encode()).hexdigest()
|
||||
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
import asyncio
|
||||
import zipfile
|
||||
from bs4 import BeautifulSoup
|
||||
from ebooklib import epub
|
||||
from lxml import etree
|
||||
import pytest
|
||||
from src.common.data_model import BookStructure
|
||||
from src.preprocessing.epub_cleaner import EpubCleaner
|
||||
from src.preprocessing.text_extractor import FineGrainedExtractor
|
||||
from src.preprocessing.navigation import prepare_navigation, destination, chinese_number
|
||||
from src.assembly.restoration_engine import RestorationEngine
|
||||
from src.assembly.backfiller import BackfillEngine
|
||||
from src.assembly.builder import BilingualBuilder
|
||||
from src.qa.epub_audit import audit_epub, compare_audits
|
||||
|
||||
|
||||
def fixture(tmp_path):
|
||||
book = epub.EpubBook(); book.set_identifier('titles'); book.set_title('Title QA'); book.set_language('en')
|
||||
chapter = epub.EpubHtml(uid='ch', file_name='Text/ch.xhtml', title='Roots')
|
||||
chapter.content = '''<html><head></head><body><h2 id="ch1"><span class="line">1</span>
|
||||
<span class="subtitle">ROOTS</span></h2><p id="opening" class="noindent" style="text-indent:0 !important; color:red">Opening prose.</p>
|
||||
<p id="caption" class="caption">A caption.</p><blockquote><p id="quote">Quoted text.</p></blockquote>
|
||||
<p id="second">Second paragraph.<a epub:type="noteref" href="notes.xhtml#note">1</a></p></body></html>'''
|
||||
part = epub.EpubHtml(uid='part', file_name='Text/part.xhtml', title='Part 1')
|
||||
part.content = '<html><body><h1 id="part"><span>PART 1</span><br/><em>BOY GENIUS</em></h1></body></html>'
|
||||
contents = epub.EpubHtml(uid='contents', file_name='Text/Contents.xhtml', title='Contents')
|
||||
contents.content = '''<html><body><h1 id="contents">Contents</h1><ol><li><a href="part.xhtml#part">Part 1: Boy Genius</a>
|
||||
<ol><li><a href="ch.xhtml#ch1"><em>Chapter 1: Roots</em></a></li></ol></li></ol></body></html>'''
|
||||
notes = epub.EpubHtml(uid='notes', file_name='Text/notes.xhtml', title='Notes')
|
||||
notes.content = '<html><body><h1 id="notes">Notes</h1><aside epub:type="footnote" id="note"><p id="note-text">Note text. <a href="ch.xhtml#second">Back</a></p></aside></body></html>'
|
||||
for item in [chapter,part,contents,notes,epub.EpubNav(),epub.EpubNcx()]: book.add_item(item)
|
||||
book.toc = [(epub.Section('Part 1: Boy Genius', 'Text/part.xhtml#part'),
|
||||
[epub.Link('Text/ch.xhtml#ch1','Chapter 1: Roots','ch1')]),
|
||||
epub.Link('Text/notes.xhtml#notes','Notes','notes')]
|
||||
book.spine = ['nav',contents,part,chapter,notes]
|
||||
source=tmp_path/'source.epub';epub.write_epub(str(source),book)
|
||||
work=tmp_path/'source'; structure=BookStructure.load(EpubCleaner(source,work).clean())
|
||||
entries=FineGrainedExtractor().extract(structure)
|
||||
prepare_navigation(source,structure,entries)
|
||||
for e in entries:
|
||||
e.translated_text=(e.original_text.replace('ROOTS','根源').replace('PART 1','第一部')
|
||||
.replace('BOY GENIUS','天才少年').replace('Contents','目录').replace('Notes','注释')
|
||||
.replace('Opening prose.','开篇正文。').replace('Second paragraph.','第二段正文。')
|
||||
.replace('A caption.','图注。').replace('Quoted text.','引文。'))
|
||||
assert asyncio.run(RestorationEngine().restore_entries(entries))==len(entries)
|
||||
return source,work,structure,entries
|
||||
|
||||
|
||||
@pytest.mark.parametrize('mode',['bilingual','target_only'])
|
||||
def test_headings_nav_ncx_contents_share_translation_and_keep_links(tmp_path,mode):
|
||||
source,work,structure,entries=fixture(tmp_path)
|
||||
chapter_titles=[e for e in entries if e.element_id=='ch1']
|
||||
assert len(chapter_titles)==1
|
||||
assert not any('Chapter 1: Roots' in e.original_text for e in entries)
|
||||
asyncio.run(BackfillEngine().backfill(structure,entries,mode))
|
||||
# Repeat safely: no duplicate bilingual paragraph or translated TOC label.
|
||||
asyncio.run(BackfillEngine().backfill(structure,entries,mode))
|
||||
output=tmp_path/(mode+'.epub');BilingualBuilder(work,source).build(structure,output)
|
||||
assert not compare_audits(audit_epub(source),audit_epub(output))
|
||||
with zipfile.ZipFile(source) as before,zipfile.ZipFile(output) as after:
|
||||
prefix=next(n for n in before.namelist() if n.endswith('/nav.xhtml')).removesuffix('nav.xhtml')
|
||||
for relative in ['nav.xhtml','Text/Contents.xhtml']:
|
||||
name=prefix+relative
|
||||
a=BeautifulSoup(before.read(name),'html.parser');b=BeautifulSoup(after.read(name),'html.parser')
|
||||
assert [x.get('href') for x in a.find_all('a')]==[x.get('href') for x in b.find_all('a')]
|
||||
link=next(x for x in b.find_all('a') if 'ch.xhtml' in x.get('href',''))
|
||||
assert '第一章:根源' in link.get_text()
|
||||
assert link.get_text().count('根源')==1
|
||||
assert link.find_parent('ol').find_parent('li') is not None
|
||||
a=etree.fromstring(before.read(prefix+'toc.ncx'));b=etree.fromstring(after.read(prefix+'toc.ncx'))
|
||||
assert a.xpath('//*[local-name()="content"]/@src')==b.xpath('//*[local-name()="content"]/@src')
|
||||
assert a.xpath('//*[local-name()="navPoint"]/@id')==b.xpath('//*[local-name()="navPoint"]/@id')
|
||||
assert '第一章:根源' in ''.join(b.itertext())
|
||||
ch=BeautifulSoup(after.read(prefix+'Text/ch.xhtml'),'html.parser')
|
||||
title=ch.find('h2',class_='translation')
|
||||
assert title.find('span',class_='line').get_text()=='第一章'
|
||||
assert title.find('span',class_='subtitle').get_text()=='根源'
|
||||
assert title['data-translation-role']=='heading'
|
||||
for key in ['opening','second']:
|
||||
p=ch.find(attrs={'data-translation-for':key}) if mode=='bilingual' else ch.find(id=key)
|
||||
assert p['data-translation-role']=='body'
|
||||
assert 'text-indent' not in p.get('style','')
|
||||
for key in ['caption','quote']:
|
||||
p=ch.find(attrs={'data-translation-for':key}) if mode=='bilingual' else ch.find(id=key)
|
||||
assert p['data-translation-role']=='other'
|
||||
notes=BeautifulSoup(after.read(prefix+'Text/notes.xhtml'),'html.parser')
|
||||
assert notes.find(id='note') and notes.find('a',href='ch.xhtml#second')
|
||||
|
||||
|
||||
def test_uncertain_destination_translates_label_without_overwriting_prose(tmp_path):
|
||||
source,work,structure,entries=fixture(tmp_path)
|
||||
# A link to prose must not be treated as the chapter heading.
|
||||
nav=next(r for r in structure.resources.values() if r.href=='nav.xhtml')
|
||||
nav.content=nav.content.replace('Text/ch.xhtml#ch1','Text/ch.xhtml#opening')
|
||||
entries=FineGrainedExtractor().extract(structure)
|
||||
prepare_navigation(source,structure,entries)
|
||||
record=next(r for r in structure.navigation_titles if r['target'].endswith('#opening'))
|
||||
assert record['heading_id'] is None
|
||||
assert record['binding_note']=='no matching heading'
|
||||
entry=next(e for e in entries if e.entry_id==record['entry_id'])
|
||||
assert entry.file_path=='__navigation__'
|
||||
assert 'Opening prose.' in next(r for r in structure.resources.values() if r.href=='Text/ch.xhtml').content
|
||||
|
||||
|
||||
def test_destination_preserves_canonical_fragment_and_ignores_external():
|
||||
assert destination('Nav/toc.xhtml','../Text/ch%20one.xhtml#part%202')=='Text/ch one.xhtml#part 2'
|
||||
assert destination('Text/ch.xhtml','#x')=='Text/ch.xhtml#x'
|
||||
assert destination('toc.xhtml','https://example.com/') is None
|
||||
assert [chinese_number(x) for x in [1,10,11,20,101,1001]]==['一','十','十一','二十','一百零一','一千零一']
|
||||
|
||||
|
||||
@pytest.mark.parametrize('markup',[
|
||||
'<h2 id="number">1</h2><h2 id="title">The Bookseller</h2><h2 id="subtitle">The Ren Family: 1937–1968</h2>',
|
||||
'<p id="number" class="calibre1">1</p><p id="title" class="calibre2"><em>The Bookseller</em></p><p id="subtitle" class="calibre3">The Ren Family: 1937–1968</p>',
|
||||
'<div id="number"><b>CHAPTER 1</b></div><div id="title"><b>The Bookseller</b></div><div id="subtitle">The Ren Family: 1937–1968</div>',
|
||||
])
|
||||
def test_separate_number_title_subtitle_are_shared_without_merging_dom(tmp_path,markup):
|
||||
from src.preprocessing.navigation import navigation_labels
|
||||
source,work,structure,entries=fixture(tmp_path)
|
||||
chapter=next(r for r in structure.resources.values() if r.href=='Text/ch.xhtml')
|
||||
chapter.content='<html><body>'+markup+'<p id="prose">A long unrelated opening paragraph.</p></body></html>'
|
||||
nav=next(r for r in structure.resources.values() if r.href=='nav.xhtml')
|
||||
nav.content=nav.content.replace('Text/ch.xhtml#ch1','Text/ch.xhtml').replace('Chapter 1: Roots','Chapter 1: The Bookseller | The Ren Family: 1937–1968')
|
||||
entries=FineGrainedExtractor().extract(structure);prepare_navigation(source,structure,entries)
|
||||
record=next(r for r in structure.navigation_titles if r['target']=='Text/ch.xhtml')
|
||||
assert len(record['entry_ids'])==3
|
||||
for e in entries:
|
||||
if e.element_id in {'number','title','subtitle'}:
|
||||
e.translated_text=e.original_text.replace('CHAPTER 1','第1章:').replace('The Bookseller','书商').replace('The Ren Family: 1937–1968','任家:1937—1968')
|
||||
asyncio.run(RestorationEngine().restore_entries(entries))
|
||||
labels=navigation_labels(structure,entries)
|
||||
assert labels['Text/ch.xhtml']=='第一章:书商 任家:1937—1968'
|
||||
asyncio.run(BackfillEngine().backfill(structure,entries,'target_only',allow_partial=True))
|
||||
ch=BeautifulSoup(chapter.content,'html.parser')
|
||||
assert all(ch.find(id=id) for id in ['number','title','subtitle','prose'])
|
||||
assert ch.find(id='prose').get_text()=='A long unrelated opening paragraph.'
|
||||
|
||||
|
||||
def test_heading_number_with_spaces_and_colon_normalizes():
|
||||
from src.preprocessing.navigation import normalize_heading_html,number_info
|
||||
assert normalize_heading_html({'role':'chapter','number':1},'<span>第 1 章:</span><em>根源</em>')=='<span>第一章</span><em>根源</em>'
|
||||
assert number_info('Chapter Eleven: Rebels')==('chapter',11)
|
||||
assert number_info('Part Two: Coffee')==('part',2)
|
||||
|
||||
|
||||
def test_split_styled_digit_and_separate_separator():
|
||||
from src.preprocessing.navigation import normalize_heading_html
|
||||
assert normalize_heading_html({'role':'part','number':1},'第<span>1</span>部:<em>天才少年</em>')=='第<span>一</span>部<em>天才少年</em>'
|
||||
assert normalize_heading_html({'role':'chapter','number':1},'<span>第1章</span>:<em>根源</em>')=='<span>第一章</span><em>根源</em>'
|
||||
|
||||
|
||||
def test_numbered_titles_starting_with_two_or_i_do_not_lose_title_words():
|
||||
from src.preprocessing.navigation import title_identity
|
||||
assert title_identity('Chapter 22: Two Indians Are Better than One')==title_identity('22 TWO INDIANS ARE BETTER THAN ONE')
|
||||
assert title_identity('Chapter 26: I Hope You Go Bust')==title_identity('26 I HOPE YOU GO BUST')
|
||||
|
||||
|
||||
def test_manifest_reconciliation_updates_title_context(tmp_path):
|
||||
from src.common.data_model import ManifestEntry
|
||||
from src.translation.manifest_manager import ManifestManager
|
||||
manager=ManifestManager(tmp_path/'manifest.json')
|
||||
old=ManifestEntry(entry_id='x#h',file_path='x',element_id='h',original_text='Roots',context='section',translated_text='根源')
|
||||
manager.add_entries([old])
|
||||
new=old.model_copy(update={'context':'navigation-heading: Chapter 1: Roots','translated_text':None})
|
||||
manager.add_entries([new])
|
||||
assert manager.entries[0].context==new.context
|
||||
assert manager.entries[0].translated_text=='根源'
|
||||
|
||||
|
||||
def test_ncx_declared_utf16_is_parsed_without_utf8_assumptions(tmp_path):
|
||||
source,work,structure,entries=fixture(tmp_path)
|
||||
with zipfile.ZipFile(source) as z:
|
||||
items=[(i,z.read(i.filename)) for i in z.infolist()]
|
||||
converted=tmp_path/'utf16.epub'
|
||||
with zipfile.ZipFile(converted,'w') as z:
|
||||
for info,data in items:
|
||||
if info.filename.endswith('.ncx'):
|
||||
data=etree.tostring(etree.fromstring(data),encoding='utf-16',xml_declaration=True)
|
||||
z.writestr(info,data)
|
||||
prepare_navigation(converted,structure,entries)
|
||||
assert any(r['source_label']=='Chapter 1: Roots' for r in structure.navigation_titles)
|
||||
Reference in New Issue
Block a user