kai/pi-agent-config
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
pi-agent-config/scenarios/curator/backend/README.md
T
Kai 88b06d782f feat(curator): vendor the application backend as the scenario's tracked source 
The curator Python backend (package, tests, systemd units, config templates, scripts) now lives under scenarios/curator/backend and is the single source of truth; the live checkout at the workspace path is a runtime copy. Exported from the app repo's tracked tree via git archive (no history, .pi/venv/caches excluded). 149 unit tests pass from the new location.

profile.toml backend is now repo-relative (scenarios/curator/backend); verify-generated.sh resolves a relative backend against REPO_ROOT. verify-no-secrets ASSIGN heuristic now requires value entropy so vendored kwargs like token=extraction_token no longer false-positive. README documents the backend/ layout and the operator-owned app rollout step.
2026-08-30 18:49:10 -07:00

9.7 KiB
Raw Permalink Blame History

Curator

Curator is the dedicated personal book, film, TV and music curation service. It uses federated catalogs: SQLite is authoritative for books, Radarr/Sonarr for video, and Plex for music. SQLite also stores Curator's intent, plan, job and audit ledger.

Production deployment, validation, backup, restore and troubleshooting are documented in docs/deployment.zh-CN.md. The production path is the host systemd --user service.

Phase 1 scope

  • SQLite Work -> Edition -> Asset catalog.
  • Control ledger with idempotent Intent -> Plan -> Command -> WorkflowJob -> Event records and TTL query caches.
  • EPUB/PDF validation, SHA-256 deduplication and deterministic storage layout.
  • Mobile-friendly LAN web upload, shelf, PDF viewer and EPUB chapter reader.
  • Wanted-book queue and empty provider/download-job contracts for later automation.
  • Optional dedicated Telegram Bot ingestion for EPUB/PDF and wanted-book messages.
  • Dedicated Pi Agent using zenmux/openai/gpt-5.6-luna as the natural-language planning and response layer, with per-chat persistent sessions and no host tool permissions.
  • Each link is treated only as a source: Curator extracts the books, films, TV series and music it substantively discusses, then evaluates each work separately.
  • Reconciliation against regular/4K Radarr and Sonarr, plus Curator's owned and wanted book records. Owned, tracked and already-wanted works are recorded but not presented as decisions. An explicit Telegram collect action prefers the uniquely matched 4K manager and requests a search; regular is only a configuration fallback.
  • A 120-second primary-model deadline, visible stage updates and automatic x-ai/grok-4.6 fallback.
  • Federated natural-language catalog questions: books query SQLite, movies/TV query regular/4K Radarr/Sonarr, and music queries Plex. The model resolves aliases and intent; deterministic adapters return current facts.
  • Bounded video metadata lookup through Radarr/Sonarr. Book evaluation combines matched public Douban/Goodreads pages with web review evidence and a source-bounded Luna synthesis. Online results are explicitly separated from owned-library facts.
  • Read-only questions execute immediately. Only explicit collect/wanted requests may invoke controlled writes; video acquisition remains 4K-first and deletion is unavailable from Telegram.
  • Daily online SQLite backup, integrity check and 14-day daily retention.
  • No automated Z-Library scraping/download, LazyLibrarian, gamdl download, EPUB translation or public exposure.

An experimental standalone downloader now lives at /home/claw/pi-workspaces/zlib-fetcher/. It provides persistent browser sessions, deterministic quota/login status handling, one conventional HTTP/SOCKS proxy and EPUB/PDF validation, but it is intentionally not connected to Curator's wanted queue or automatic import path yet.

Local deployment

The host mount at /mnt/truenas/multimedia is writable. A Codex sandbox may expose the same path through a read-only bind view; use a host-side findmnt and a disposable write probe when verifying NFS permissions.

mkdir -p ~/.config/curator ~/.local/share/curator
cp config/curator.env.example ~/.config/curator/curator.env
cp systemd/curator.service systemd/curator-maintenance.service systemd/curator-maintenance.timer ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now curator.service curator-maintenance.timer

Open http://192.168.50.145:8766/.

The LAN Web is organized by workflow: /candidates is the discovery and decision queue, /wanted merges selected candidates with wanted books and exposes manual EPUB acquisition, /library shows only verified files/owned backend media, /sources preserves provenance, and /activity shows import and analysis jobs. Book discovery uses responsive Goodreads/Douban-style bibliographic cards instead of a cross-media table.

Health check:

PYTHONPATH=. python3 -m curator health
curl -fsS http://127.0.0.1:8766/api/health | jq

Telegram

Create the dedicated Bot with BotFather, then place its token only in ~/.config/curator/curator.env and keep the file mode 0600:

CURATOR_TELEGRAM_BOT_TOKEN=<token>
CURATOR_TELEGRAM_ALLOWED_USERS=1093241065
CURATOR_RADARR_URL=http://192.168.50.10:7878
CURATOR_RADARR_API_KEY=<key>
CURATOR_RADARR_4K_URL=http://192.168.50.100:7878
CURATOR_RADARR_4K_API_KEY=<key>
CURATOR_SONARR_URL=http://192.168.50.10:8989
CURATOR_SONARR_API_KEY=<key>
CURATOR_SONARR_4K_URL=http://192.168.50.100:8989
CURATOR_SONARR_4K_API_KEY=<key>
CURATOR_RADARR_ROOT_FOLDER=/mnt/truenas/multimedia/movies
CURATOR_RADARR_QUALITY_PROFILE_ID=4
CURATOR_SONARR_ROOT_FOLDER=/mnt/truenas/multimedia/tv
CURATOR_SONARR_QUALITY_PROFILE_ID=4
CURATOR_RADARR_4K_ROOT_FOLDER=/mnt/unRaid/movie4k
CURATOR_RADARR_4K_QUALITY_PROFILE_ID=5
CURATOR_SONARR_4K_ROOT_FOLDER=/mnt/unRaid/tv4k
CURATOR_SONARR_4K_QUALITY_PROFILE_ID=7
CURATOR_PLEX_URL=http://192.168.50.100:32400
CURATOR_PLEX_TOKEN=<token>
CURATOR_PLEX_MUSIC_SECTION_ID=
CURATOR_TAVILY_API_KEY=<key>
CURATOR_BOOK_WEB_REVIEW_MAX_RESULTS=6
CURATOR_ZLIB_SEARCH_URL_TEMPLATE=https://zlib.li/s/{query}

Restart curator.service. The Bot accepts EPUB/PDF documents. A caption may contain:

书名：示例书名
作者：示例作者
语言：zh-Hans

EPUB/PDF files enter the import pipeline. A URL is fetched by the deterministic gateway (WeChat links reuse the local article archive service), then the isolated Curator Pi Agent extracts the media works discussed in its body. The Pi process explicitly loads the dedicated workspace skill .pi/skills/curator-media/SKILL.md; library state and writes remain deterministic backend responsibilities. The source article itself is never rated or collected. The Bot returns one recommendation and decision card per extracted work.

The Pi resource split is intentional: workspace/AGENTS.md contains the durable Curator identity and factual boundaries; workspace/.pi/skills/curator-media/SKILL.md contains reusable media identification and evaluation policy; request-specific inputs and JSON schemas stay in curator/pi_agent.py. Prompt templates remain disabled because the backend supplies complete non-interactive prompts, and no extension is loaded because Pi has no tools or direct write authority in this service.

Plain text is now handled as an agent conversation rather than a command form. Luna first emits a structured, side-effect-free intent plan, including normalized title, aliases, media type and whether the user explicitly requested a write. Curator then queries only the required backends and optionally *Arr online lookup. A second Luna pass receives the bounded query result and writes the final answer. This supports questions such as 权利的游戏，库里有什么版本, typo correction, follow-up questions and recommendation requests without inventing library state. Retry and simple acknowledgement remain deterministic fast paths. An explicit collect request may add a book to wanted or add a uniquely resolved movie/show to the 4K manager; all other natural-language messages are read-only. 再试一次 reruns the most recent source for that Telegram chat.

Movies/shows are queried across regular and 4K managers with external IDs, normalized titles, aliases and a one-year metadata tolerance. Existing 4K files win, otherwise new collection targets Radarr 4K/Sonarr 4K and requests a search. Regular managers are fallback targets only when the corresponding 4K service is not configured.

Plex is authoritative for music catalog queries, management and playback. gamdl is not a catalog: a future Curator Downloader will wrap it for controlled downloads, staging, validation and delivery to Plex. Until that downloader is enabled, music queries work when Plex credentials are configured, but automatic music acquisition remains unavailable.

Book candidates are enriched in two layers. Curator locates public Douban/Goodreads book pages through DuckDuckGo HTML, validates ISBN or title/author, then caches the attributed rating and cover locally; this path needs no account or API key. Google Books, Open Library and Hardcover are deliberately excluded. Tavily finds broader attributed review pages when configured; DuckDuckGo provides a zero-key fallback. Luna synthesizes a verdict from those snippets while preserving the supporting URLs and confidence level. Missing evidence is shown as insufficient rather than inferred. Candidate cards also expose a manual zlib.li search link. Curator does not scrape Z-Library result/detail pages or automate downloads; downloaded EPUB/PDF files enter through Web/Telegram/CLI import for validation and deduplication. The LAN upload form accepts multiple EPUB/PDF files; each file gets an independent import job so one failure does not abort the rest of the batch.

Refresh existing candidate review snapshots:

PYTHONPATH=. python3 -m curator refresh-book-reviews --candidate-id 12 --candidate-id 13

The production deployment is the host systemd --user service on the dedicated LLM VPS; Pi runs as a child of that service. No containerisation is used.

TrueNAS paths

The deployed env uses:

CURATOR_LIBRARY_ROOT=/mnt/truenas/multimedia/books
CURATOR_STAGING_ROOT=/mnt/truenas/multimedia/curator/staging/books
CURATOR_BACKUP_ROOT=/mnt/truenas/multimedia/curator/backup

The broader hierarchy is:

/mnt/truenas/multimedia/
├── books/
├── music/
└── curator/
    ├── archive/articles/YYYY/MM/<source-id>/
    ├── staging/{music,books,publish}/<job-id>/
    ├── imports/telegram/<job-id>/
    ├── exports/html/<work-id>/<edition-id>/
    ├── quarantine/{music,books}/
    └── backup/{database/{daily,weekly,monthly},config,manifests}/

Tests

PYTHONPATH=. python3 -m unittest discover -s tests -v
Reference in New Issue View Git Blame Copy Permalink