Add lightweight standalone PDF to EPUB import and source-page QA
This commit is contained in:
Kai Tan
@@ -1,3 +1,12 @@
|
||||
## 2026-10-28 — lightweight standalone PDF import preview
|
||||
|
||||
- Add PDFium/Pillow PDF import with native text, macOS system Vision OCR and an optional Tesseract adapter; no translation or heavyweight model dependency required.
|
||||
- Support Chinese reflow directly and English EPUB export for the existing translation pipeline, without automatically starting paid translation.
|
||||
- Cache source-bound page recognition, retain coordinates and raw OCR, reconstruct single-column paragraphs across pages, and preserve bookmark hierarchy and source-page anchors.
|
||||
- Preserve selected illustration/title/back-cover pages, export digital PDF image regions, and apply source-bound exact OCR corrections without rewriting raw checkpoints.
|
||||
- Add PDF/source-anchor desktop and narrow-screen QA with actual NAV clicks; validate all 476 pages and 87 bookmark targets on the user-provided sample.
|
||||
- Keep output explicitly marked as an OCR reflow draft; complex layouts, scanned-page figure segmentation and footnote reconstruction remain outside this release.
|
||||
|
||||
## 2026-10-10 — Highest Exam edition metadata
|
||||
|
||||
- Preserve OPF metadata XML comments while replacing edition values and their refinements.
|
||||
|
||||
@@ -109,7 +109,7 @@ Mac 沙箱中 Calibre 的 Qt 渲染可能需要允许进程运行。QA 使用项
|
||||
|
||||
无法可靠绑定的目录条目仅翻译标签，不改写猜测的章节正文；具体条目记录在 `book_structure.json` 的 `navigation_titles[].binding_note`。这类条目需要检查章节标题是否一致，不能视为全书标题同步已经通过。索引、版权、注释全文仍遵守原有跳过策略；导航中这些页面的名称可以翻译。
|
||||
|
||||
后续扫描 PDF/OCR 大版本的图文、结构与目录方案见 [路线图](doc/OCR_EPUB_ROADMAP.md)。
|
||||
轻量 PDF/OCR 首版见 [使用说明](doc/PDF_IMPORT.md)；后续困难版式见 [路线图](doc/OCR_EPUB_ROADMAP.md)。
|
||||
|
||||
### 中文封面与书库 metadata（v0.14）
|
||||
|
||||
@@ -149,3 +149,16 @@ Mac 沙箱中 Calibre 的 Qt 渲染可能需要允许进程运行。QA 使用项
|
||||
路径相对于 JSON 文件；常规译本保留，另生成 `edition_*.epub`。缺少已审封面会报错，不自动使用另一个版本的图片。基础封装不新增依赖、无网络请求；生图和书目检索由 skill 编排。
|
||||
|
||||
2026-10-08 接入核实：本机 Codex 使用 ChatGPT 登录，缓存目录没有可选择的 image 模型。官方模型目录提供 `gpt-image-2.5-sunburst`／`gpt-image-2.5-flare`，但[ChatGPT OAuth API 路径暂不支持图片生成](https://developers.openai.com/siwc/token-sharing-open-source/preview-limitations)。本轮实际使用 Codex 内置 imagegen 成功生成两版；该工具不公开模型 ID／选择参数，因此记录 `model: null`，不能宣称使用 GPT-Image-2.5。后续运行先检查当时的官方支持情况和工具能力；不要读取或转用私有 OAuth token 绕过限制，也不自动改用付费 API。
|
||||
|
||||
### 轻量 PDF → EPUB（独立模块）
|
||||
|
||||
新增 `scripts/pdf_to_epub.py`，仅安装 `requirements-pdf.txt` 即可独立使用。中文 PDF 直接重排；英文 PDF 先导出 EPUB，再进入现有翻译流程，转换阶段不调用翻译 API。支持文本层、macOS 系统 Vision OCR、可选 Tesseract、逐页恢复、原书签目录和来源锚点。首版限定横排单栏；样本已跑通476页全书，输出仍标为 OCR 重排初稿。
|
||||
|
||||
```bash
|
||||
.venv/bin/pip install -r requirements-pdf.txt
|
||||
.venv/bin/python scripts/pdf_to_epub.py input/book.pdf \
|
||||
--output output/pdf/book.epub --work .work/pdf-book \
|
||||
--language zh-CN --title '书名' --author '作者'
|
||||
```
|
||||
|
||||
独立环境、英文翻译衔接、图片页保留和视觉QA见 [PDF导入使用说明](doc/PDF_IMPORT.md)。
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# 扫描 PDF → EPUB：后续大版本
|
||||
|
||||
这是独立的大版本方向，本轮不实现。目标是把旧中文书的扫描 PDF 转成可重排、可搜索、可点击目录的 EPUB，同时保留原书中有阅读意义的结构与图文关系。
|
||||
2026-10-28 已开始独立模块首版，见 [PDF_IMPORT.md](PDF_IMPORT.md)。首版优先轻量方案和横排单栏；以下困难版式仍属后续范围。目标是把旧中文书的扫描 PDF 转成可重排、可搜索、可点击目录的 EPUB，同时保留原书中有阅读意义的结构与图文关系。
|
||||
|
||||
## 输入与评测样本
|
||||
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
# 轻量 PDF → EPUB 独立模块
|
||||
|
||||
首版只处理横排单栏、普通段落和简单图片。中文 PDF 转为中文重排 EPUB；英文 PDF 转为英文 EPUB，然后由现有翻译程序处理。PDF 模块不导入翻译客户端，不读取 API Key，不发起翻译请求。
|
||||
|
||||
## 安装和后端
|
||||
|
||||
只使用 PDF 模块时，可单独建环境，不安装原项目的翻译 API 依赖：
|
||||
|
||||
```bash
|
||||
python3 -m venv .work/pdf-env
|
||||
.work/pdf-env/bin/pip install -r requirements-pdf.txt
|
||||
```
|
||||
|
||||
当前项目环境也可以直接安装该文件。相对已有项目，新增 PDF 相关依赖仅 `pypdfium2`、`Pillow`；独立安装还包含 EPUB 封装／结构审计所需的 EbookLib、lxml、BeautifulSoup、html5lib。没有 Docling、MinerU、Torch 或额外模型权重。
|
||||
|
||||
- `--ocr auto`：正常文本 PDF 使用文本层；全页扫描图片重新 OCR，即使已有隐藏 OCR 层也不直接相信旧结果。Mac 默认使用系统 Apple Vision；其他系统使用可选 Tesseract。
|
||||
- `--ocr native`：只用已有文本层。扫描页会记录警告，不能把已有 OCR 质量视为已验证。
|
||||
- `--ocr vision`：强制系统 Vision，需要 macOS 和 Apple command line tools 的 `swiftc`。辅助程序在工作目录编译并复用，本地处理图片。Codex 沙箱内 Vision 如报 nilError，应允许本地 OCR 进程在沙箱外执行；不是 API 认证错误。
|
||||
- `--ocr tesseract`：强制可选 Tesseract CLI；需自行安装程序以及 `eng`／`chi_sim` 语言数据。适配器已实现，本轮机器没有安装 Tesseract，未进行真实 Tesseract 识别验收。
|
||||
|
||||
官方依据：[PDFium Python API](https://pypdfium2.readthedocs.io/en/stable/python_api.html)、[Apple Vision](https://developer.apple.com/documentation/vision/recognizing-text-in-images)、[Tesseract TSV](https://tesseract-ocr.github.io/tessdoc/Command-Line-Usage.html)。PDFium 在本模块中串行调用，避免跨线程调用原生库。
|
||||
|
||||
## 中文 PDF 直接重排
|
||||
|
||||
```bash
|
||||
.work/pdf-env/bin/python scripts/pdf_to_epub.py input/chinese.pdf \
|
||||
--output output/pdf/chinese.epub --work .work/pdf-chinese \
|
||||
--language zh-CN --title '书名' --author '作者' --cover-page 1
|
||||
```
|
||||
|
||||
中文正文使用两格缩进；保留原语言，不做繁简转换或自由改写。输出 metadata 明确标注“PDF重排初稿”，不误标为 AI 翻译版。
|
||||
|
||||
## 英文 PDF → EPUB → 翻译
|
||||
|
||||
```bash
|
||||
.work/pdf-env/bin/python scripts/pdf_to_epub.py input/english.pdf \
|
||||
--output output/pdf/english.epub --work .work/pdf-english \
|
||||
--language en --title 'Original Title' --author 'Original Author'
|
||||
|
||||
# 在已有翻译环境中执行，沿用它的模型、凭据和配置：
|
||||
.venv/bin/python main.py output/pdf/english.epub --complete-book --bilingual
|
||||
```
|
||||
|
||||
PDF 转换完成只打印可选的后续翻译命令，不执行。英文原生文本 PDF 已通过真实 PDF 夹具转换，并验证可由现有 EpubCleaner／FineGrainedExtractor 完整提取；没有调用模型翻译此夹具。
|
||||
|
||||
## 小样、图片与恢复
|
||||
|
||||
- `--pages 1-20,30`：按1起算、闭区间选页；未覆盖全书时，书名和报告明确标注部分样例。
|
||||
- `--cover-page 1`：显式指定封面，不默认把所有 PDF 的第一页当封面。
|
||||
- `--image-pages 2,13,476`：手动保留插图、扉页、原目录或封底整页原貌。少量文字／空白页也会保留为原页图片并列入报告，不声称所有页均已文字重排。
|
||||
- `--dpi 180`：扫描渲染分辨率，支持96–400。
|
||||
- 同一工作目录一次运行一个任务。再次运行相同命令会复用完成的逐页 OCR；源 SHA、引擎、OCR 实现／系统版本、语言或 DPI 变化时不复用不匹配结果。另一份源文件必须使用新目录。
|
||||
|
||||
工作目录保留 `source.json`、`pages/` 原始识别／坐标／置信度、`renders/` 原页图、`book.json` 段落与书签结构、`report.json`。OCR 失败保留已完成页面，禁止把仅完成部分的处理结果写成全书 EPUB。
|
||||
|
||||
EPUB 按原书签组织章节和目录层级；没有书签时使用识别出的简单章节标题，或明确的来源页导航。每个原页都有 `pdf-page-N` 来源锚点；跨页仅合并明显连续的正文，保留原页追踪。页眉删除记录和原 OCR 行仍在中间结果中。
|
||||
|
||||
普通文本 PDF 内的独立图片按页面区域渲染嵌入；扫描页中的混合图文首版不做自动精细分割，需要 `--image-pages` 保留原页。双栏、竖排、公式、复杂表格、脚注双向链接重建不在本轮范围，不作为已实现功能。
|
||||
|
||||
## QA
|
||||
|
||||
可选安装现有 `requirements-qa.txt` 和 Chromium 后：
|
||||
|
||||
```bash
|
||||
PLAYWRIGHT_BROWSERS_PATH="$PWD/.work/playwright" \
|
||||
.venv/bin/python scripts/pdf_qa.py output/pdf/chinese.epub \
|
||||
--work .work/pdf-chinese --pages 1,4,16 --out .work/pdf-chinese/qa
|
||||
```
|
||||
|
||||
生成原页／EPUB 900px／390px 对照，实际点击生成的 NAV 目录链接，并检查目标及来源锚点。HTML 报告和 JSON 分开保留自动结果与视觉待审状态。结构审计通过不等于 OCR 字词准确；Vision 置信分数不是字错率，也不能据此给全书准确率。
|
||||
|
||||
EPUB 在临时文件中构建并通过结构审计后才替换输出。没有依赖 Calibre 或桌面阅读器；Chromium 连续 XHTML 不能代表所有阅读器分页。
|
||||
|
||||
## 可追溯的局部校正
|
||||
|
||||
`--corrections corrections.json` 支持查看原扫描后确认的逐行校正：
|
||||
|
||||
```json
|
||||
{
|
||||
"source_sha256": "当前PDF的完整SHA256",
|
||||
"edits": [{
|
||||
"line_id": "p0101-l0011",
|
||||
"expected": "原始OCR整行原文",
|
||||
"replacement": "核对原扫描后的整行文字",
|
||||
"evidence": "核对依据"
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
源 SHA、行 ID 或 expected 不匹配就停止；重复条目和空替换也拒绝。原始 `pages/*.json` 不改写，校正写在派生 `book.json` 和报告里，可在重建时重复应用。识别内容变化会改变输出版本标识。此功能用于有证据的文字校对，不替代全书审读。
|
||||
@@ -0,0 +1,60 @@
|
||||
# 轻量 PDF 导入首版与样本验收
|
||||
|
||||
按本轮收窄的范围开发：不引入 Docling 或大型模型，不实现困难版式。新增独立 `src/pdf_import/`、`scripts/pdf_to_epub.py` 和可选 `scripts/pdf_qa.py`。中文直接重排为 EPUB；英文先生成英文 EPUB，再进入已有翻译程序。转换阶段不调用翻译 API。
|
||||
|
||||
## 已实现与依赖
|
||||
|
||||
新增 PDF 依赖为 PDFium Python 包与 Pillow。独立环境只装 `requirements-pdf.txt`，包含 EPUB 导出／审计的轻量基础库；实际验证未加载 OpenAI SDK、Torch、Transformers、Docling 或 Paddle。macOS 使用系统 Vision；Swift 辅助进程常驻批量识别，未额外下载 OCR 模型权重。Tesseract 适配器可选，本机未安装程序，未做真实 Tesseract 验收。
|
||||
|
||||
原生文本提取与扫描页识别分流，扫描页即使已有隐藏 OCR 层也重新识别。页面缓存按源 SHA、页号、实际引擎／实现与系统版本、语言、DPI 和 PDFium 版本匹配。失败只保留已完成页面；未完成全书时不写出冒充全书的结果。部分选页输出有明确样例标签。
|
||||
|
||||
根据来源坐标重建普通段落，修复首轮发现的页眉混入正文与连续缩进段落被逐行拆开的情况；明显的跨页连续正文合并，原页锚点保留。书签层级生成 NAV／NCX。保留指定整页图，数字 PDF 的独立图片按页面区域渲染嵌入。构建临时 EPUB，结构审计通过后才替换最终输出。
|
||||
|
||||
## 样本
|
||||
|
||||
用户指定下载目录中的《中国的世界高原 上》，沈卫平。476页，无密码；抽查为横排单栏扫描，已有 ABBYY 文本层，但可见漏字，不能当作可靠原文。
|
||||
|
||||
源 SHA256：`aebc9f263ab0599f6d11c7db6c5e89e1316f2a950f2faa9ddbc72a94fe3da335`。
|
||||
|
||||
```bash
|
||||
.venv/bin/python scripts/pdf_to_epub.py \
|
||||
'/Users/kaitan/Downloads/中国的世界高原 上 (沈卫平) (z-library.sk, 1lib.sk, z-lib.sk).pdf' \
|
||||
--output 'output/pdf/中国的世界高原_上_重排初稿.epub' \
|
||||
--work .work/pdf-highland --language zh-CN \
|
||||
--title '中国的世界高原（上）' --author 沈卫平 \
|
||||
--cover-page 1 --image-pages 2,13,476 \
|
||||
--corrections .work/pdf-highland/corrections.json
|
||||
```
|
||||
|
||||
476/476页处理，476个来源页锚点均存在。459页为重排文字，17页为封面、扉页、空白／短文字页等原页图片；报告逐项列明。第2、13、476页为显式指定保留原貌的扉页、纸本目录和带签名二维码的封底；其余少量空白／短文字页自动保留原页图，不称为可搜索的重排文字。
|
||||
|
||||
原87个书签对应87个有效目录目标，层级为10个一级、77个二级。浏览器实际点击87个 NAV 链接，目标均到达。509行页眉／页码候选从派生正文移除，原识别行和删除记录仍保存。
|
||||
|
||||
## 对照 QA 与修复
|
||||
|
||||
以原页序1、4、16、101、248、476生成原扫描／900px／390px对照，检查封面、普通段落、跨页衔接、引文和封底保留。自动结构审计零问题，抽样自动布局检查零问题。实际查看的图片及 SHA 写入 `.work/pdf-highland/review-record.json`。
|
||||
|
||||
原页对照确认第101页两行中的三个字形识别错误，使用带源 SHA、稳定行 ID、expected 原文和 replacement 的校正文件修正；原始逐页 OCR JSON 不变。校正不匹配或重复时拒绝应用，不做全书无证据替换。
|
||||
|
||||
OCR 准确率仍待全文校对，未计算人工真值字错率。Vision 的低置信提示不是准确率百分比。输出 metadata 明确标为“PDF重排初稿”；没有宣称所有文字或每个阅读器分页已验收。
|
||||
|
||||
## 独立与英文路径验证
|
||||
|
||||
在新环境 `.work/pdf-light-env` 仅安装 `requirements-pdf.txt`，实际将两页英文原生文本 PDF 夹具转换为 EPUB，并打印后续翻译命令，没有加载翻译 SDK 或发起翻译。另用现有 EpubCleaner 和 FineGrainedExtractor 验证生成英文 EPUB 可进入现有提取链路。后续翻译在原项目 `.venv` 中执行，沿用其模型与凭据选择。
|
||||
|
||||
测试：98 passed，2 skipped。新增覆盖页范围、页眉误识别、英文提取衔接、扫描页覆盖旧文本层、缓存复用／失效、失败恢复与禁止部分全书输出、连续缩进段落及源绑定校正。
|
||||
|
||||
## 范围限制
|
||||
|
||||
- 首版针对横排单栏普通书；没有实现双栏、竖排正文、公式、复杂表格或脚注双向链接重建。
|
||||
- 扫描页混合插图不自动精细切分；可指定整页保留。并非所有字体、粗体及页面坐标都被重建。
|
||||
- 来源锚点使用PDF文件页序，不冒充纸本页码。纸本目录页保留原图，交互目录由实际 PDF 书签生成。
|
||||
- headless 为连续 XHTML，不代表 Calibre、Apple Books 等所有阅读器的分页。全书文字校对没有完成。
|
||||
|
||||
## 本地证据
|
||||
|
||||
`.work/pdf-highland/`：`source.json`、原始 `pages/`、`renders/`、`book.json`、`corrections.json`、`report.json`、`qa-final/review.html`、`qa-final/report.json` 和 SHA 绑定的 `review-record.json`。原PDF不修改，书籍和识别文字不提交仓库。
|
||||
|
||||
输出：`output/pdf/中国的世界高原_上_重排初稿.epub`。
|
||||
|
||||
输出 SHA256：`9e2bece683420803d59292aa49fb44ee6e1cd002331364af7380ba0a5971b71d`。
|
||||
@@ -0,0 +1,8 @@
|
||||
# Optional PDF conversion only; no model weights or translation SDK required.
|
||||
pypdfium2>=5,<6
|
||||
Pillow>=10,<13
|
||||
# Minimal EPUB export/audit dependencies (the translation API stack is optional).
|
||||
ebooklib>=0.19,<1
|
||||
lxml>=4.9
|
||||
beautifulsoup4>=4.12
|
||||
html5lib>=1.1
|
||||
@@ -0,0 +1,67 @@
|
||||
"""Compare cached source PDF pages and actual EPUB source anchors locally."""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from urllib.parse import quote
|
||||
sys.path.insert(0,str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
|
||||
def main():
|
||||
parser=argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('epub',type=Path);parser.add_argument('--work',type=Path,required=True)
|
||||
parser.add_argument('--pages',default='1,4,16');parser.add_argument('--out',type=Path,required=True)
|
||||
args=parser.parse_args()
|
||||
from src.qa.headless import archive_server,book_inventory,LAYOUT_JS
|
||||
from src.qa.epub_audit import audit_epub
|
||||
from playwright.sync_api import sync_playwright
|
||||
data=json.loads((args.work/'book.json').read_text())
|
||||
pages={p['number']:p for p in data['pages']};selected=[int(p) for p in args.pages.split(',')]
|
||||
if any(p not in pages for p in selected):parser.error('Requested page not present in the conversion IR')
|
||||
args.out.mkdir(parents=True,exist_ok=True)
|
||||
inventory=book_inventory(args.epub);evidence=[];clicks=[];sections=[]
|
||||
with archive_server({'book':args.epub}) as base,sync_playwright() as pw:
|
||||
browser=pw.chromium.launch(headless=True)
|
||||
try:
|
||||
context=browser.new_context(java_script_enabled=False,service_workers='block')
|
||||
context.route('**/*',lambda route:route.continue_() if route.request.url.startswith(base+'/') else route.abort())
|
||||
tab=context.new_page()
|
||||
for number in selected:
|
||||
anchor=f'pdf-page-{number}'
|
||||
document=next((name for name,soup in inventory['documents'].items() if soup.find(id=anchor)),None)
|
||||
if not document:raise ValueError(f'EPUB lost source anchor {anchor}')
|
||||
source_name=f'pdf-{number:04d}.jpg';shutil.copy2(pages[number]['render'],args.out/source_name)
|
||||
rendered=[]
|
||||
for width in [900,390]:
|
||||
tab.set_viewport_size({'width':width,'height':1100})
|
||||
tab.goto(base+'/book/'+quote(document)+'#'+anchor,wait_until='networkidle')
|
||||
tab.evaluate('() => document.fonts.ready')
|
||||
tab.evaluate('(id) => {const e=document.getElementById(id); scrollTo(0,Math.max(0,e.getBoundingClientRect().top+scrollY-140));}',anchor)
|
||||
name=f'epub-{number:04d}-{width}.png';tab.screenshot(path=str(args.out/name));rendered.append(name)
|
||||
evidence.append({'source_page':number,'document':document,'viewport_width':width,'image':name,'layout_findings':tab.evaluate(LAYOUT_JS)})
|
||||
sections.append(f'<section><h2>PDF 原页序 {number}</h2><div class="pair"><img src="{source_name}" alt="原扫描页"/><img src="{rendered[0]}" alt="EPUB实际排版"/><img src="{rendered[1]}" alt="窄屏排版"/></div></section>')
|
||||
# Real browser navigation for every generated TOC target, including fragments.
|
||||
for entry in inventory['toc']:
|
||||
target=entry['target'];name,_,fragment=target.partition('#')
|
||||
nav=entry['file']
|
||||
nav_soup=inventory['documents'].get(nav)
|
||||
if nav_soup is not None:
|
||||
from src.qa.headless import destination
|
||||
anchors=nav_soup.find_all('a',href=True)
|
||||
link_index=next(i for i,a in enumerate(anchors) if destination(nav,a['href'])==target)
|
||||
tab.goto(base+'/book/'+quote(nav),wait_until='domcontentloaded')
|
||||
tab.locator('a[href]').nth(link_index).click(timeout=5000)
|
||||
else:
|
||||
tab.goto(base+'/book/'+quote(name)+(('#'+quote(fragment)) if fragment else ''),wait_until='domcontentloaded')
|
||||
reached=(tab.url.split('#')[0]==base+'/book/'+quote(name)) and (not fragment or tab.evaluate('(id) => !!document.getElementById(id)',fragment))
|
||||
clicks.append({'target':target,'destination_reached':reached})
|
||||
finally:browser.close()
|
||||
report={'source_sha256':data['source_sha256'],'epub_sha256':hashlib.sha256(args.epub.read_bytes()).hexdigest(),'epub_audit':audit_epub(args.epub),'evidence':evidence,'toc_navigation_checks':clicks,'visual_review':'pending','ocr_accuracy_review':'pending','limits':'Chromium continuous XHTML, not all reader pagination; source anchor may be inside a continued paragraph.'}
|
||||
(args.out/'report.json').write_text(json.dumps(report,ensure_ascii=False,indent=2)+'\n')
|
||||
(args.out/'review.html').write_text('<!doctype html><meta charset="utf-8"><title>PDF → EPUB QA</title><style>body{font:16px sans-serif;margin:24px}.pair{display:grid;grid-template-columns:1fr 1fr .5fr;gap:16px}img{width:100%;height:auto;align-self:start}section{margin-bottom:40px}</style><h1>PDF → EPUB 对照</h1><p>左：原扫描；中：EPUB；右：窄屏。截图生成不等于视觉或OCR验收。</p>'+''.join(sections))
|
||||
print(f'PDF source comparisons: {len(selected)}; TOC target navigation: {len(clicks)}')
|
||||
return int(bool(report['epub_audit']['issues']) or any(not c['destination_reached'] for c in clicks) or any(e['layout_findings'] for e in evidence))
|
||||
|
||||
if __name__=='__main__':sys.exit(main())
|
||||
@@ -0,0 +1,30 @@
|
||||
"""Independent lightweight PDF import. English output can enter the EPUB translator."""
|
||||
import argparse
|
||||
import json
|
||||
import shlex
|
||||
import sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0,str(Path(__file__).resolve().parents[1]))
|
||||
from src.pdf_import.converter import convert
|
||||
|
||||
|
||||
def main():
|
||||
parser=argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('source',type=Path);parser.add_argument('--output',required=True,type=Path)
|
||||
parser.add_argument('--work',required=True,type=Path);parser.add_argument('--language',choices=['zh-CN','en'],required=True)
|
||||
parser.add_argument('--title');parser.add_argument('--author');parser.add_argument('--pages',help='1-based inclusive selection, e.g. 1-30,40; output is explicitly partial')
|
||||
parser.add_argument('--ocr',choices=['auto','native','vision','tesseract'],default='auto')
|
||||
parser.add_argument('--dpi',type=int,default=180);parser.add_argument('--cover-page',type=int);parser.add_argument('--image-pages',help='Preserve chosen illustration/title/back-cover pages as images, e.g. 2,13,476')
|
||||
parser.add_argument('--corrections',type=Path,help='Source SHA-bound, exact OCR line corrections JSON')
|
||||
args=parser.parse_args()
|
||||
try:
|
||||
report=convert(args.source,args.output,args.work,args.language,args.title,args.author,args.ocr,args.pages,args.dpi,args.cover_page,image_pages=args.image_pages,corrections=args.corrections)
|
||||
except (ValueError,RuntimeError,ImportError,OSError) as exc:
|
||||
parser.exit(1,f'PDF import failed: {exc}\n')
|
||||
print(json.dumps({k:report[k] for k in ['output','partial','processed_pages','source_pages','engines']},ensure_ascii=False))
|
||||
if args.language=='en':
|
||||
translation_python=Path(__file__).resolve().parents[1]/'.venv/bin/python'
|
||||
print('Optional next step in the translation environment (not executed): '+shlex.join([str(translation_python) if translation_python.exists() else 'python',str(Path(__file__).resolve().parents[1]/'main.py'),str(args.output.resolve()),'--complete-book','--bilingual']))
|
||||
return 0
|
||||
|
||||
if __name__=='__main__':sys.exit(main())
|
||||
@@ -0,0 +1 @@
|
||||
"""Optional lightweight PDF import, independent of the translation pipeline."""
|
||||
@@ -0,0 +1,253 @@
|
||||
"""PDF -> source-traceable, reflowable EPUB. No translation calls or heavy models."""
|
||||
import hashlib
|
||||
import html
|
||||
import json
|
||||
import platform
|
||||
import re
|
||||
import sys
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from ebooklib import epub
|
||||
from .layout import paragraphs, repeated_furniture, join_text
|
||||
|
||||
SCHEMA = 1
|
||||
|
||||
|
||||
def digest(path):
|
||||
h = hashlib.sha256()
|
||||
with Path(path).open('rb') as f:
|
||||
for chunk in iter(lambda:f.read(1024*1024), b''): h.update(chunk)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def save_json(path, data):
|
||||
path = Path(path); path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temp = path.with_suffix('.tmp'); temp.write_text(json.dumps(data, ensure_ascii=False, indent=2)+'\n'); temp.replace(path)
|
||||
|
||||
|
||||
def page_selection(spec, count):
|
||||
if not spec: return list(range(count))
|
||||
selected = set()
|
||||
for item in spec.split(','):
|
||||
if not re.fullmatch(r'\d+(?:-\d+)?', item): raise ValueError('Pages must look like 1-20,25 (1-based inclusive)')
|
||||
pair = [int(x) for x in item.split('-')]; lo, hi = pair[0], pair[-1]
|
||||
if not 1 <= lo <= hi <= count: raise ValueError('Page range is outside the source PDF')
|
||||
selected.update(range(lo-1, hi))
|
||||
return sorted(selected)
|
||||
|
||||
|
||||
def native_lines(page):
|
||||
text = page.get_textpage()
|
||||
try:
|
||||
lines=[]; width,height=page.get_size()
|
||||
for i in range(text.count_rects()):
|
||||
left,bottom,right,top=text.get_rect(i)
|
||||
value=text.get_text_bounded(left,bottom,right,top).strip()
|
||||
if value:
|
||||
lines.append({'text':value,'bbox':[left/width,1-top/height,right/width,1-bottom/height], 'confidence':None})
|
||||
return lines
|
||||
finally: text.close()
|
||||
|
||||
|
||||
def image_boxes(page):
|
||||
import pypdfium2.raw as raw
|
||||
w,h=page.get_size(); boxes=[]
|
||||
for obj in page.get_objects(filter=[raw.FPDF_PAGEOBJ_IMAGE]):
|
||||
left,bottom,right,top=obj.get_bounds()
|
||||
boxes.append([max(0,left/w), max(0,1-top/h), min(1,right/w), min(1,1-bottom/h)])
|
||||
return boxes
|
||||
|
||||
|
||||
def validate_lines(lines):
|
||||
if not isinstance(lines,list): raise ValueError('OCR lines must be an array')
|
||||
for line in lines:
|
||||
box=line.get('bbox',[])
|
||||
if not isinstance(line.get('text'),str) or len(box)!=4 or not all(isinstance(v,(int,float)) and 0<=v<=1 for v in box) or box[0]>=box[2] or box[1]>=box[3]:
|
||||
raise ValueError('Invalid OCR text or normalized top-left bounding box')
|
||||
|
||||
|
||||
def apply_corrections(pages, correction, source_sha):
|
||||
"""Source-bound exact edits; never overwrite the raw OCR checkpoint."""
|
||||
if correction.get('source_sha256')!=source_sha or not isinstance(correction.get('edits'),list):
|
||||
raise ValueError('Corrections must be bound to this PDF SHA and contain an edits array')
|
||||
lines={line['id']:line for page in pages for line in page['lines']};applied=[];seen=set()
|
||||
for edit in correction['edits']:
|
||||
key=edit.get('line_id');line=lines.get(key)
|
||||
if key in seen or line is None or line['text']!=edit.get('expected') or not isinstance(edit.get('replacement'),str) or not edit['replacement'].strip():
|
||||
raise ValueError('Stale, duplicate or invalid correction: '+str(key))
|
||||
line['text']=edit['replacement'];seen.add(key);applied.append(edit)
|
||||
return applied
|
||||
|
||||
|
||||
def convert(source, output, work, language='zh-CN', title=None, author=None,
|
||||
engine='auto', pages=None, dpi=180, cover_page=None, ocr_timeout=120, image_pages=None, corrections=None):
|
||||
import pypdfium2 as pdfium
|
||||
source,output,work=Path(source),Path(output),Path(work)
|
||||
if not source.is_file(): raise ValueError('PDF source does not exist')
|
||||
if source.resolve()==output.resolve(): raise ValueError('Output must not overwrite source')
|
||||
if output.suffix.lower()!='.epub': raise ValueError('Output must have .epub suffix')
|
||||
if not 96<=dpi<=400: raise ValueError('DPI must be between 96 and 400')
|
||||
work.mkdir(parents=True,exist_ok=True); source_sha=digest(source)
|
||||
source_identity=work/'source.json'
|
||||
if source_identity.exists() and json.loads(source_identity.read_text())['sha256']!=source_sha:
|
||||
raise ValueError('Work directory belongs to another source; use a new directory')
|
||||
save_json(source_identity,{'path':str(source.resolve()),'sha256':source_sha})
|
||||
document=pdfium.PdfDocument(source); ocr=None
|
||||
try:
|
||||
selected=page_selection(pages,len(document)); partial=len(selected)!=len(document)
|
||||
preserved_images=set(i+1 for i in page_selection(image_pages,len(document))) if image_pages else set()
|
||||
if cover_page is not None and not 1<=cover_page<=len(document): raise ValueError('Cover page outside source PDF')
|
||||
backend=('vision' if platform.system()=='Darwin' else 'tesseract') if engine=='auto' else engine
|
||||
outlines=[]
|
||||
for bookmark in document.get_toc():
|
||||
dest=bookmark.get_dest()
|
||||
if dest and dest.get_index() in selected:
|
||||
outlines.append({'title':bookmark.get_title(),'page':dest.get_index()+1,'level':bookmark.level})
|
||||
results=[]; warnings=[]
|
||||
for index in selected:
|
||||
page=document[index]
|
||||
try:
|
||||
boxes=image_boxes(page)
|
||||
scan=any((b[2]-b[0])*(b[3]-b[1])>.65 for b in boxes)
|
||||
native=native_lines(page) if engine=='native' or (engine=='auto' and not scan) else []
|
||||
text_size=sum(len(l['text']) for l in native)
|
||||
use_ocr=engine not in {'native','auto'} or (engine=='auto' and (scan or (text_size==0 and bool(boxes))))
|
||||
is_cover=index+1==cover_page
|
||||
is_image=is_cover or index+1 in preserved_images
|
||||
if use_ocr and not is_image and ocr is None:
|
||||
from .ocr import VisionOCR,TesseractOCR
|
||||
ocr=(VisionOCR if backend=='vision' else TesseractOCR)(work,language,ocr_timeout)
|
||||
used='cover-image' if is_cover else 'page-image' if is_image else backend if use_ocr else 'native'
|
||||
stamp={'source':source_sha,'page':index+1,'engine':used,'ocr_identity':ocr.identity if use_ocr and not is_image else None,'language':language,'dpi':dpi,'pdfium':str(pdfium.PYPDFIUM_INFO),'schema':SCHEMA}
|
||||
cache=work/'pages'/f'{index+1:04d}.json'
|
||||
cached=json.loads(cache.read_text()) if cache.exists() else None
|
||||
render=work/'renders'/f'{index+1:04d}.jpg';render.parent.mkdir(exist_ok=True)
|
||||
if cached and cached.get('stamp')==stamp and render.exists():
|
||||
result=cached
|
||||
else:
|
||||
bitmap=page.render(scale=dpi/72)
|
||||
try: bitmap.to_pil().convert('RGB').save(render,quality=88)
|
||||
finally: bitmap.close()
|
||||
lines=[] if is_image else ocr.recognize(render) if use_ocr else native
|
||||
validate_lines(lines)
|
||||
for n,line in enumerate(lines): line['id']=f'p{index+1:04d}-l{n+1:04d}'
|
||||
result={'number':index+1,'width':page.get_width(),'height':page.get_height(),'engine':used,'scan':scan,'lines':lines,'image_boxes':boxes,'render':str(render.resolve()),'stamp':stamp}
|
||||
save_json(cache,result)
|
||||
results.append(result)
|
||||
print(f'PDF page {index+1}/{len(document)}: {used}, {len(result["lines"])} lines',file=sys.stderr,flush=True)
|
||||
finally: page.close()
|
||||
applied=apply_corrections(results,json.loads(Path(corrections).read_text()),source_sha) if corrections else []
|
||||
repeated=repeated_furniture(results)
|
||||
for page in results:
|
||||
blocks,removed=paragraphs(page,language,repeated)
|
||||
page['blocks']=blocks;page['removed_furniture']=removed
|
||||
count=sum(len(l['text']) for l in page['lines'])
|
||||
page['image_fallback']=page['engine'] in {'cover-image','page-image'} or count<30
|
||||
if page['image_fallback'] and page['engine'] not in {'cover-image','page-image'}:
|
||||
warnings.append({'page':page['number'],'kind':'page_image_fallback','detail':'Blank/illustrated/short-text page retained as an image; inspect source, not claimed as reflowed text.'})
|
||||
removed_ids={l['id'] for l in removed}
|
||||
low=[l['id'] for l in page['lines'] if l['id'] not in removed_ids and l.get('confidence') is not None and l['confidence']<.6]
|
||||
if low: warnings.append({'page':page['number'],'kind':'low_ocr_confidence','line_ids':low})
|
||||
if engine=='native' and page['scan']: warnings.append({'page':page['number'],'kind':'existing_scan_text_layer','detail':'Existing OCR text layer reused without verification.'})
|
||||
book={'schema':SCHEMA,'source_sha256':source_sha,'source':str(source.resolve()),'source_pages':len(document),'selected_pages':[i+1 for i in selected],'partial':partial,'language':language,'title':title or source.stem,'author':author or 'Unknown','outlines':outlines,'image_pages':sorted(preserved_images),'cover_page':cover_page,'pages':results,'warnings':warnings,'applied_corrections':applied,'scope':'horizontal single-column; no automatic footnote/table reconstruction'}
|
||||
save_json(work/'book.json',book)
|
||||
export_epub(book,output,work,cover_page)
|
||||
from src.qa.epub_audit import audit_epub
|
||||
audit=audit_epub(output)
|
||||
report={'source_sha256':source_sha,'output_sha256':digest(output),'output':str(output.resolve()),'partial':partial,'processed_pages':len(results),'source_pages':len(document),'image_only_pages':[p['number'] for p in results if p['image_fallback']],'removed_furniture_lines':sum(len(p['removed_furniture']) for p in results),'engines':{name:sum(p['engine']==name for p in results) for name in set(p['engine'] for p in results)},'warnings':warnings,'applied_corrections':applied,'epub_audit':audit,'visual_review':'pending','ocr_accuracy_review':'pending','translation':'not requested; English EPUB can be passed to main.py separately'}
|
||||
save_json(work/'report.json',report)
|
||||
if audit['issues']: raise ValueError('EPUB structural audit failed; inspect report.json')
|
||||
return report
|
||||
finally:
|
||||
if ocr: ocr.close()
|
||||
document.close()
|
||||
|
||||
|
||||
STYLE='''body {margin: 5%; line-height:1.65;} p {margin:.5em 0;} html[lang^="zh"] p {text-indent:2em;} h1,h2 {text-align:center;page-break-before:always;} img {max-width:100%;height:auto;} figure {margin:1em 0;text-align:center;} figcaption {font-size:.8em;} .source-page {font-size:.7em;color:#777;text-indent:0;} .image-page {text-align:center;text-indent:0;} .image-page img {max-height:90vh;width:auto;object-fit:contain;}'''
|
||||
|
||||
|
||||
def export_epub(data, output, work, cover_page):
|
||||
from PIL import Image
|
||||
import io
|
||||
book=epub.EpubBook(); identity={k:data[k] for k in ['source_sha256','language','title','author','selected_pages']}
|
||||
identity['content_sha256']=hashlib.sha256(json.dumps([[p['number'],p['image_fallback'],[(b['kind'],b['text']) for b in p['blocks']]] for p in data['pages']],ensure_ascii=False).encode()).hexdigest()
|
||||
book.set_identifier('urn:uuid:'+str(uuid.uuid5(uuid.NAMESPACE_URL,json.dumps(identity,sort_keys=True))))
|
||||
label='【部分样例】' if data['partial'] else ''
|
||||
book.set_title(label+data['title']+'【PDF重排初稿】');book.set_language(data['language']);book.add_author(data['author'])
|
||||
book.add_metadata('DC','source','urn:sha256:'+data['source_sha256'])
|
||||
book.add_metadata('DC','description','PDF结构重建初稿；OCR未逐字校对。'+('仅含选定页，非完整书。' if data['partial'] else ''))
|
||||
css=epub.EpubItem(uid='style',file_name='styles/book.css',media_type='text/css',content=STYLE.encode());book.add_item(css)
|
||||
chapters=[]; toc_records=[]; body=[]; current=None; previous=None; previous_page=None
|
||||
def finish():
|
||||
if current is not None:
|
||||
current.content=('<html xmlns:epub="http://www.idpf.org/2007/ops"><head></head><body>'+''.join(body)+'</body></html>').encode()
|
||||
current.add_link(href='../styles/book.css',rel='stylesheet',type='text/css')
|
||||
book.add_item(current);chapters.append(current)
|
||||
for page in data['pages']:
|
||||
number=page['number']; outlines=[o for o in data['outlines'] if o['page']==number]
|
||||
labels=[o['title'] for o in outlines]
|
||||
if current is None or labels or page['image_fallback'] or previous_page is None or number!=previous_page+1:
|
||||
finish();body=[];previous=None
|
||||
current=epub.EpubHtml(title=' / '.join(labels) or f'PDF {number}',file_name=f'chapters/p{number:04d}.xhtml',lang=data['language'])
|
||||
for outline in outlines:
|
||||
toc_records.append((epub.Link(current.file_name+f'#pdf-page-{number}',outline['title'],f'nav-{len(toc_records)+1}'),outline['level']))
|
||||
marker=f'<span id="pdf-page-{number}" epub:type="pagebreak" role="doc-pagebreak" aria-label="PDF {number}"></span>'
|
||||
if not page['image_fallback']:
|
||||
for heading in labels: body.append('<h1>'+html.escape(heading)+'</h1>')
|
||||
if page['image_fallback']:
|
||||
body.append(marker)
|
||||
image=Path(page['render']).read_bytes();name=f'images/page-{number:04d}.jpg'
|
||||
book.add_item(epub.EpubItem(uid=f'image-{number}',file_name=name,media_type='image/jpeg',content=image))
|
||||
body.append(f'<div class="image-page"><img src="../{name}" alt="PDF第{number}页原貌"/></div>')
|
||||
if number==cover_page: book.set_cover('images/cover.jpg',image,create_page=False)
|
||||
previous=None
|
||||
else:
|
||||
entries=[dict(b) for b in page['blocks']]
|
||||
for n,box in enumerate(page['image_boxes']):
|
||||
area=(box[2]-box[0])*(box[3]-box[1])
|
||||
if .02<area<.65: entries.append({'kind':'image','bbox':box,'image_index':n})
|
||||
entries.sort(key=lambda b:(b['bbox'][1],b['bbox'][0]))
|
||||
marker_pending=True
|
||||
for index,block in enumerate(entries):
|
||||
if block['kind']=='image':
|
||||
with Image.open(page['render']) as picture:
|
||||
box=block['bbox'];region=picture.crop(tuple(int(v*(picture.width if i%2==0 else picture.height)) for i,v in enumerate(box)))
|
||||
stream=io.BytesIO();region.save(stream,format='PNG')
|
||||
n=block['image_index'];name=f'images/p{number:04d}-{n}.png'
|
||||
book.add_item(epub.EpubItem(uid=f'figure-{number}-{n}',file_name=name,media_type='image/png',content=stream.getvalue()))
|
||||
if marker_pending:body.append(marker);marker_pending=False
|
||||
body.append(f'<figure><img src="../{name}" alt="PDF第{number}页图片"/></figure>');previous=None;continue
|
||||
plain=re.sub(r'\s','',block['text'])
|
||||
if any(plain==re.sub(r'\s','',label) or (index==0 and plain in {'序','序言','前言','自序','引言','后记','目录'} and re.sub(r'\s','',label).startswith(plain)) for label in labels):continue
|
||||
# Merge only obvious prose continuation across consecutive source pages.
|
||||
continuation=marker_pending and previous and not labels and previous_page==number-1 and previous['kind']=='paragraph' and block['kind']=='paragraph' and not block['indented'] and not re.search(r'[。！？.!?:：;；”’"]$',previous['text']) and body[-1].endswith('</p>')
|
||||
if continuation:
|
||||
extra=join_text(previous['text'],block['text'],data['language'])[len(previous['text']):]
|
||||
body[-1]=body[-1][:-4]+marker+f'<span id="{block["id"]}"></span>'+html.escape(extra)+'</p>'
|
||||
marker_pending=False;previous={**block,'text':join_text(previous['text'],block['text'],data['language'])};continue
|
||||
if marker_pending:body.append(marker);marker_pending=False
|
||||
tag='h2' if block['kind']=='heading' else 'p'
|
||||
body.append(f'<{tag} id="{block["id"]}">{html.escape(block["text"])}</{tag}>')
|
||||
if block['kind']=='heading' and not data['outlines']:
|
||||
toc_records.append((epub.Link(current.file_name+'#'+block['id'],block['text'],f'nav-{len(toc_records)+1}'),0))
|
||||
previous=block
|
||||
if marker_pending:body.append(marker)
|
||||
previous_page=number
|
||||
if page['image_fallback']:
|
||||
finish();current=None;body=[];previous=None
|
||||
finish()
|
||||
if not toc_records:toc_records=[(epub.Link(c.file_name,c.title,f'nav-{i+1}'),0) for i,c in enumerate(chapters)]
|
||||
roots=[];stack=[]
|
||||
for link,level in toc_records:
|
||||
node={'link':link,'children':[]}
|
||||
while stack and stack[-1][0]>=level:stack.pop()
|
||||
(stack[-1][1]['children'] if stack else roots).append(node);stack.append((level,node))
|
||||
def flatten(nodes):return [(node['link'],flatten(node['children'])) if node['children'] else node['link'] for node in nodes]
|
||||
book.toc=flatten(roots);book.add_item(epub.EpubNcx());nav=epub.EpubNav();nav.title='目录' if data['language'].startswith('zh') else 'Contents';book.add_item(nav);book.spine=['nav']+chapters
|
||||
output=Path(output);output.parent.mkdir(parents=True,exist_ok=True);temp=output.with_suffix('.tmp.epub')
|
||||
try:
|
||||
epub.write_epub(str(temp),book,{'epub3_pages':False})
|
||||
from src.qa.epub_audit import audit_epub
|
||||
if audit_epub(temp)['issues']:raise ValueError('Generated EPUB failed structural audit; existing output retained')
|
||||
temp.replace(output)
|
||||
finally:temp.unlink(missing_ok=True)
|
||||
@@ -0,0 +1,78 @@
|
||||
"""Conservative reconstruction for horizontal, single-column prose only."""
|
||||
import re
|
||||
import statistics
|
||||
from collections import Counter
|
||||
from difflib import SequenceMatcher
|
||||
|
||||
CJK = r'\u3400-\u9fff'
|
||||
|
||||
|
||||
def clean_text(text, language):
|
||||
text = re.sub(r'\s+', ' ', text).strip()
|
||||
if language.startswith('zh'):
|
||||
text = re.sub(r'(?<=['+CJK+r']) +(?=['+CJK+r'])', '', text)
|
||||
return text
|
||||
|
||||
|
||||
def join_text(left, right, language):
|
||||
if not left: return right
|
||||
if not right: return left
|
||||
if language.startswith('zh'):
|
||||
space = ' ' if left[-1].isascii() and left[-1].isalnum() and right[0].isascii() and right[0].isalnum() else ''
|
||||
else:
|
||||
space = '' if left.endswith('-') else ' '
|
||||
return left + space + right
|
||||
|
||||
|
||||
def furniture_key(text):
|
||||
return re.sub(r'[\W\d_]+', '', text).casefold()
|
||||
|
||||
|
||||
def repeated_furniture(pages):
|
||||
counts = Counter()
|
||||
for page in pages:
|
||||
counts.update({furniture_key(line['text']) for line in page['lines'] if line['bbox'][3] < .10 or line['bbox'][1] > .94})
|
||||
return {key for key, count in counts.items() if key and count >= 3}
|
||||
|
||||
|
||||
def paragraphs(page, language, repeated):
|
||||
lines = sorted(page['lines'], key=lambda line: (round(line['bbox'][1], 3), line['bbox'][0]))
|
||||
kept, removed = [], []
|
||||
for line in lines:
|
||||
text = clean_text(line['text'], language)
|
||||
if not text: continue
|
||||
edge = line['bbox'][3] < .10 or line['bbox'][1] > .94
|
||||
folio = len(text)<45 and bool(re.search(r'(?:^\d{1,4}\s*[|I丨!！/]|[|I丨!！/]\s*\d{1,4}$)', text) or re.match(r'^\d{3}\s+[^\d]', text))
|
||||
key=furniture_key(text)
|
||||
similar_header=len(key)>=4 and any(len(other)>=4 and SequenceMatcher(None,key,other).ratio()>=.72 for other in repeated)
|
||||
folio |= bool(re.search(r'\d{3}$',text))
|
||||
if edge and (folio or key in repeated or similar_header or re.fullmatch(r'[\W\d_]+', text)):
|
||||
removed.append(line); continue
|
||||
kept.append({**line, 'text': text})
|
||||
if not kept: return [], removed
|
||||
heights = [l['bbox'][3]-l['bbox'][1] for l in kept]
|
||||
height = statistics.median(heights)
|
||||
lefts = sorted(l['bbox'][0] for l in kept)
|
||||
baseline = lefts[max(0, len(lefts)//5)]
|
||||
right = statistics.median(l['bbox'][2] for l in kept)
|
||||
blocks = []
|
||||
for line_index, line in enumerate(kept):
|
||||
box = line['bbox']; text = line['text']
|
||||
# This is a heuristic heading proposal, not a claim of semantic certainty.
|
||||
heading = bool(re.fullmatch(r'第[一二三四五六七八九十百零〇两\d]+[章节部篇卷].{0,30}', text) or re.fullmatch(r'(?i)(chapter|part)\s+[\divxlc]+(?:\b.*)?', text))
|
||||
heading |= len(text) <= 10 and box[0] > .28 and box[2] < .75 and (bool(re.fullmatch(r'\d{1,3}', text)) or heights[line_index] > height*1.35)
|
||||
indented = box[0] > baseline + max(.015, height*.7)
|
||||
previous = blocks[-1] if blocks else None
|
||||
indent_start=indented and (line_index==0 or box[0]>kept[line_index-1]['bbox'][0]+max(.009,height*.3))
|
||||
split = heading or previous is None or previous['kind']=='heading' or indent_start
|
||||
if previous:
|
||||
gap = box[1]-previous['bbox'][3]
|
||||
split |= gap > height*.9 or previous['bbox'][2] < right-.10
|
||||
if split:
|
||||
blocks.append({'kind':'heading' if heading else 'paragraph', 'text':text, 'bbox':list(box), 'indented':indented, 'source_page':page['number'], 'line_ids':[line['id']]})
|
||||
else:
|
||||
previous['text'] = join_text(previous['text'], text, language)
|
||||
previous['bbox'] = [min(previous['bbox'][0],box[0]), previous['bbox'][1], max(previous['bbox'][2],box[2]), box[3]]
|
||||
previous['line_ids'].append(line['id'])
|
||||
for index, block in enumerate(blocks): block['id'] = f'p{page["number"]:04d}-b{index+1:04d}'
|
||||
return blocks, removed
|
||||
@@ -0,0 +1,92 @@
|
||||
"""Small OCR adapters. Vision uses macOS frameworks; Tesseract is an optional CLI."""
|
||||
import csv
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import platform
|
||||
import selectors
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class VisionOCR:
|
||||
name = 'vision'
|
||||
|
||||
def __init__(self, work, language, timeout=120):
|
||||
if platform.system() != 'Darwin':
|
||||
raise RuntimeError('Vision OCR requires macOS; select tesseract or native instead')
|
||||
compiler = shutil.which('swiftc')
|
||||
if not compiler:
|
||||
raise RuntimeError('Vision OCR requires Apple command line tools (swiftc)')
|
||||
source = Path(__file__).with_name('vision.swift')
|
||||
# Include OS version in cache identity: framework OCR can change between releases.
|
||||
self.identity = hashlib.sha256(source.read_bytes() + platform.mac_ver()[0].encode()).hexdigest()
|
||||
cache = Path(work)/'vision-runtime'; cache.mkdir(parents=True, exist_ok=True)
|
||||
binary = cache/('vision-' + self.identity[:16])
|
||||
if not binary.exists():
|
||||
module_cache = cache/'module-cache'; module_cache.mkdir(exist_ok=True)
|
||||
built = subprocess.run([compiler, '-O', '-module-cache-path', str(module_cache), str(source), '-o', str(binary)], capture_output=True, text=True, timeout=180)
|
||||
if built.returncode:
|
||||
raise RuntimeError('Vision helper compilation failed: ' + built.stderr[-3000:])
|
||||
self.languages = ['zh-Hans', 'en-US'] if language.startswith('zh') else ['en-US']
|
||||
self.timeout = timeout
|
||||
self.log = (cache/'runtime.log').open('a')
|
||||
self.process = subprocess.Popen([str(binary)], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=self.log, text=True, bufsize=1)
|
||||
|
||||
def recognize(self, image):
|
||||
self.process.stdin.write(json.dumps({'image': str(Path(image).resolve()), 'languages': self.languages})+'\n')
|
||||
self.process.stdin.flush()
|
||||
with selectors.DefaultSelector() as selector:
|
||||
selector.register(self.process.stdout, selectors.EVENT_READ)
|
||||
if not selector.select(self.timeout):
|
||||
self.process.kill()
|
||||
raise RuntimeError('Vision OCR timed out; completed page cache retained')
|
||||
raw = self.process.stdout.readline()
|
||||
if not raw:
|
||||
raise RuntimeError('Vision OCR stopped; inspect vision-runtime/runtime.log')
|
||||
result = json.loads(raw)
|
||||
if result.get('error'):
|
||||
raise RuntimeError('Vision OCR failed: ' + result['error'])
|
||||
return result['lines']
|
||||
|
||||
def close(self):
|
||||
if self.process.poll() is None:
|
||||
self.process.stdin.close()
|
||||
try: self.process.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
self.process.kill(); self.process.wait()
|
||||
self.log.close()
|
||||
|
||||
|
||||
class TesseractOCR:
|
||||
name = 'tesseract'
|
||||
|
||||
def __init__(self, work, language, timeout=120):
|
||||
self.binary = shutil.which('tesseract')
|
||||
if not self.binary:
|
||||
raise RuntimeError('Install tesseract and language data, or use Vision on macOS')
|
||||
self.language = 'chi_sim+eng' if language.startswith('zh') else 'eng'
|
||||
self.timeout = timeout
|
||||
version = subprocess.run([self.binary, '--version'], capture_output=True, text=True, check=True)
|
||||
self.identity = version.stdout.splitlines()[0] + ':' + self.language
|
||||
|
||||
def recognize(self, image):
|
||||
from PIL import Image
|
||||
with Image.open(image) as picture: width, height = picture.size
|
||||
output = subprocess.run([self.binary, str(image), 'stdout', '-l', self.language, 'tsv'], capture_output=True, text=True, check=True, timeout=self.timeout)
|
||||
groups = {}
|
||||
for word in csv.DictReader(io.StringIO(output.stdout), delimiter='\t', quoting=csv.QUOTE_NONE):
|
||||
if not (word.get('text') or '').strip(): continue
|
||||
key = (word['block_num'], word['par_num'], word['line_num'])
|
||||
groups.setdefault(key, []).append(word)
|
||||
lines = []
|
||||
for words in groups.values():
|
||||
left = min(int(w['left']) for w in words); top = min(int(w['top']) for w in words)
|
||||
right = max(int(w['left'])+int(w['width']) for w in words)
|
||||
bottom = max(int(w['top'])+int(w['height']) for w in words)
|
||||
text = (' ' if self.language == 'eng' else '').join(w['text'] for w in words)
|
||||
lines.append({'text': text, 'bbox': [left/width, top/height, right/width, bottom/height], 'confidence': min(float(w['conf'])/100 for w in words)})
|
||||
return lines
|
||||
|
||||
def close(self): pass
|
||||
@@ -0,0 +1,35 @@
|
||||
import Foundation
|
||||
import Vision
|
||||
import ImageIO
|
||||
import Darwin
|
||||
|
||||
struct Input: Decodable { let image: String; let languages: [String] }
|
||||
struct Line: Encodable { let text: String; let bbox: [Double]; let confidence: Float }
|
||||
struct Output: Encodable { let lines: [Line]; let engine: String; let error: String? }
|
||||
let decoder = JSONDecoder()
|
||||
let encoder = JSONEncoder()
|
||||
while let line = readLine() {
|
||||
autoreleasepool {
|
||||
let result: Output
|
||||
do {
|
||||
let job = try decoder.decode(Input.self, from: Data(line.utf8))
|
||||
let request = VNRecognizeTextRequest()
|
||||
request.recognitionLevel = .accurate
|
||||
request.recognitionLanguages = job.languages
|
||||
request.usesLanguageCorrection = true
|
||||
let handler = VNImageRequestHandler(url: URL(fileURLWithPath: job.image), options: [:])
|
||||
try handler.perform([request])
|
||||
let lines = (request.results ?? []).compactMap { observation -> Line? in
|
||||
guard let candidate = observation.topCandidates(1).first else { return nil }
|
||||
let r = observation.boundingBox
|
||||
return Line(text: candidate.string, bbox: [r.minX, 1-r.maxY, r.maxX, 1-r.minY].map(Double.init), confidence: candidate.confidence)
|
||||
}
|
||||
result = Output(lines: lines, engine: "apple-vision", error: nil)
|
||||
} catch {
|
||||
result = Output(lines: [], engine: "apple-vision", error: String(describing: error))
|
||||
}
|
||||
if let data = try? encoder.encode(result), let output = String(data: data, encoding: .utf8) {
|
||||
print(output); fflush(stdout)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
"""Lightweight import: selection, provenance, OCR routing, reflow and EPUB links."""
|
||||
import json
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
import pytest
|
||||
from src.pdf_import.converter import convert, page_selection, validate_lines
|
||||
from src.pdf_import.layout import paragraphs, repeated_furniture, join_text
|
||||
from src.qa.epub_audit import audit_epub
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def native_pdf(tmp_path):
|
||||
pytest.importorskip('pypdfium2')
|
||||
# Minimal native-text PDF fixture; no external PDF authoring dependency.
|
||||
def write(name='input.pdf', image=False):
|
||||
streams=[b'BT /F1 18 Tf 60 740 Td (Chapter 1: Start) Tj 0 -40 Td /F1 12 Tf (A paragraph with enough native English text to use without OCR.) Tj ET',b'BT /F1 12 Tf 60 740 Td (This is the second page with more native English prose.) Tj ET']
|
||||
if image:streams=[b'q 600 0 0 800 0 0 cm /Im1 Do Q '+s for s in streams]
|
||||
resources=b'<< /Font << /F1 7 0 R >>'+(b' /XObject << /Im1 8 0 R >>' if image else b'')+b' >>'
|
||||
objects=[b'<< /Type /Catalog /Pages 2 0 R >>',b'<< /Type /Pages /Kids [3 0 R 4 0 R] /Count 2 >>']
|
||||
objects.extend(b'<< /Type /Page /Parent 2 0 R /MediaBox [0 0 600 800] /Resources '+resources+b' /Contents '+str(n).encode()+b' 0 R >>' for n in [5,6])
|
||||
objects.extend(b'<< /Length '+str(len(s)).encode()+b' >>\nstream\n'+s+b'\nendstream' for s in streams)
|
||||
objects.append(b'<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>')
|
||||
if image:objects.append(b'<< /Type /XObject /Subtype /Image /Width 1 /Height 1 /ColorSpace /DeviceRGB /BitsPerComponent 8 /Length 3 >>\nstream\n\xff\xff\xff\nendstream')
|
||||
raw=b'%PDF-1.4\n';offsets=[0]
|
||||
for i,obj in enumerate(objects):offsets.append(len(raw));raw+=f'{i+1} 0 obj\n'.encode()+obj+b'\nendobj\n'
|
||||
xref=len(raw);raw+=f'xref\n0 {len(objects)+1}\n0000000000 65535 f \n'.encode()
|
||||
raw+=b''.join(f'{n:010d} 00000 n \n'.encode() for n in offsets[1:])
|
||||
raw+=f'trailer\n<< /Size {len(objects)+1} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF'.encode()
|
||||
path=tmp_path/name;path.write_bytes(raw);return path
|
||||
return write
|
||||
|
||||
|
||||
def test_page_ranges_do_not_silently_drop_pages():
|
||||
assert page_selection(None,3)==[0,1,2]
|
||||
assert page_selection('3,1-2,2',3)==[0,1,2]
|
||||
for value in ['0','3-2','1-4','-1','1,','abc']:
|
||||
with pytest.raises(ValueError):page_selection(value,3)
|
||||
|
||||
|
||||
def test_folio_ocr_variants_are_removed_but_body_years_survive():
|
||||
lines=[{'id':'header','text':'002 | 中国的世界系','bbox':[.15,.04,.4,.06]}, {'id':'body','text':'2019年发生的事。','bbox':[.15,.12,.8,.14]}]
|
||||
blocks,removed=paragraphs({'number':5,'lines':lines},'zh-CN',set())
|
||||
assert [b['text'] for b in blocks]==['2019年发生的事。']
|
||||
assert removed[0]['id']=='header'
|
||||
assert join_text('跨页','继续。','zh-CN')=='跨页继续。'
|
||||
assert join_text('Some','words','en')=='Some words'
|
||||
|
||||
|
||||
def test_native_english_import_is_independent_and_complete(native_pdf,tmp_path):
|
||||
source=native_pdf();out=tmp_path/'book.epub';work=tmp_path/'work'
|
||||
report=convert(source,out,work,language='en',engine='native')
|
||||
assert report['engines']=={'native':2} and not report['partial']
|
||||
assert audit_epub(out)['issues']==[]
|
||||
with zipfile.ZipFile(out) as z:
|
||||
chapters=[n for n in z.namelist() if '/chapters/' in n]
|
||||
assert len(chapters)==1 # Consecutive pages flow inside one reading document.
|
||||
text=z.read(chapters[0]).decode()
|
||||
assert 'native English' in text and 'second page' in text
|
||||
assert 'pdf-page-1' in text and 'pdf-page-2' in text
|
||||
assert '../styles/book.css' in text and 'lang="en"' in text
|
||||
from src.preprocessing.epub_cleaner import EpubCleaner
|
||||
from src.preprocessing.text_extractor import FineGrainedExtractor
|
||||
from src.common.data_model import BookStructure
|
||||
structure=BookStructure.load(EpubCleaner(out,tmp_path/'translation-input').clean())
|
||||
units=FineGrainedExtractor(complete_book=True).extract(structure)
|
||||
assert any('native English' in unit.original_text for unit in units)
|
||||
ir=json.loads((work/'book.json').read_text())
|
||||
assert len(ir['pages'])==2 and ir['pages'][0]['lines'][0]['bbox']
|
||||
partial=convert(source,tmp_path/'partial.epub',work,language='en',engine='native',pages='2')
|
||||
assert partial['partial'] and partial['processed_pages']==1
|
||||
with zipfile.ZipFile(tmp_path/'partial.epub') as z:
|
||||
opf=z.read('EPUB/content.opf').decode();assert '部分样例' in opf
|
||||
with pytest.raises(ValueError,match='another source'):
|
||||
convert(native_pdf('another.pdf',image=True),tmp_path/'other.epub',work,engine='native')
|
||||
|
||||
|
||||
def test_scanned_text_layer_is_replaced_and_ocr_cache_is_reused(native_pdf,tmp_path,monkeypatch):
|
||||
from src.pdf_import import ocr
|
||||
import src.pdf_import.converter as converter
|
||||
called=[]
|
||||
class FakeOCR:
|
||||
identity='fixture-v1'
|
||||
def __init__(self,*args):pass
|
||||
def recognize(self,image):
|
||||
called.append(image)
|
||||
return [{'text':'这是来自扫描图像的新识别结果，必须保留完整正文内容，不能使用已有错误文本层。','bbox':[.1,.2,.9,.25],'confidence':.9}]
|
||||
def close(self):pass
|
||||
monkeypatch.setattr(ocr,'VisionOCR',FakeOCR)
|
||||
monkeypatch.setattr(ocr,'TesseractOCR',FakeOCR)
|
||||
source=native_pdf(image=True);out=tmp_path/'scan.epub';work=tmp_path/'work'
|
||||
convert(source,out,work,language='zh-CN',engine='auto')
|
||||
assert len(called)==2
|
||||
convert(source,out,work,language='zh-CN',engine='auto')
|
||||
assert len(called)==2
|
||||
with zipfile.ZipFile(out) as z:
|
||||
body=b''.join(z.read(n) for n in z.namelist() if '/chapters/' in n).decode()
|
||||
assert '新识别结果' in body and 'native English' not in body
|
||||
assert 'lang="zh-CN"' in body
|
||||
FakeOCR.identity='fixture-v2'
|
||||
convert(source,out,work,language='zh-CN',engine='auto')
|
||||
assert len(called)==4
|
||||
|
||||
|
||||
def test_invalid_ocr_geometry_fails_closed():
|
||||
with pytest.raises(ValueError):validate_lines([{'text':'bad','bbox':[.8,.2,.1,.4]}])
|
||||
with pytest.raises(ValueError):validate_lines([{'text':'bad','bbox':[0,0,float('nan'),1]}])
|
||||
|
||||
|
||||
def test_ocr_failure_leaves_page_checkpoint_and_no_partial_final(native_pdf,tmp_path,monkeypatch):
|
||||
from src.pdf_import import ocr
|
||||
called=[]
|
||||
class FakeOCR:
|
||||
identity='resume-fixture'
|
||||
fail=True
|
||||
def __init__(self,*args):pass
|
||||
def recognize(self,image):
|
||||
called.append(Path(image).name)
|
||||
if Path(image).stem=='0002' and self.fail:raise RuntimeError('fixture OCR failure')
|
||||
return [{'text':'A sufficiently long recognized paragraph to remain searchable in the EPUB.', 'bbox':[.1,.2,.9,.3], 'confidence':1}]
|
||||
def close(self):pass
|
||||
monkeypatch.setattr(ocr,'VisionOCR',FakeOCR);monkeypatch.setattr(ocr,'TesseractOCR',FakeOCR)
|
||||
source=native_pdf(image=True);out=tmp_path/'output.epub';work=tmp_path/'work'
|
||||
with pytest.raises(RuntimeError,match='fixture OCR failure'):convert(source,out,work,language='en')
|
||||
assert not out.exists() and (work/'pages/0001.json').exists() and not (work/'pages/0002.json').exists()
|
||||
FakeOCR.fail=False
|
||||
convert(source,out,work,language='en')
|
||||
assert called==['0001.jpg','0002.jpg','0002.jpg']
|
||||
assert audit_epub(out)['issues']==[]
|
||||
|
||||
|
||||
def test_consistently_indented_quote_lines_stay_one_paragraph():
|
||||
lines=[{'id':'a','text':'这是正文段落。','bbox':[.1,.11,.9,.13]}, {'id':'b','text':'一段尚未结束的引文','bbox':[.2,.16,.8,.18]}, {'id':'c','text':'继续在相同缩进位置。','bbox':[.2,.19,.8,.21]}]
|
||||
blocks,_=paragraphs({'number':1,'lines':lines},'zh-CN',set())
|
||||
assert len(blocks)==2
|
||||
assert blocks[1]['text']=='一段尚未结束的引文继续在相同缩进位置。'
|
||||
|
||||
|
||||
def test_corrections_require_source_and_exact_expected_text():
|
||||
from src.pdf_import.converter import apply_corrections
|
||||
pages=[{'lines':[{'id':'p0001-l0001','text':'牲富'}]}]
|
||||
patch={'source_sha256':'source','edits':[{'line_id':'p0001-l0001','expected':'牲富','replacement':'牲畜'}]}
|
||||
with pytest.raises(ValueError):apply_corrections(pages,patch,'different-source')
|
||||
applied=apply_corrections(pages,patch,'source')
|
||||
assert pages[0]['lines'][0]['text']=='牲畜' and len(applied)==1
|
||||
with pytest.raises(ValueError,match='Stale'):apply_corrections(pages,patch,'source')
|
||||
Reference in New Issue
Block a user