add isolated Supermemory POC for unRaid

2026-09-07 20:15:47 -07:00
parent 8289822ce7
commit ddfafca50f
9 changed files with 646 additions and 0 deletions
@@ -0,0 +1,122 @@
+# Supermemory POC on unRaid
+
+This project is an isolated evaluation service. It does not replace the
+production Hindsight service or change the default Hermes profile.
+
+## Fixed deployment choices
+
+- Supermemory Local `server-v0.0.8`, Linux x64 binary, verified by SHA-256.
+- One self-contained server; state, auth material, uploaded files, and model
+  cache live under `/mnt/user/appdata/supermemory-poc/data`.
+- A dedicated address on Docker `br0`; port `6767` is not published on the
+  unRaid host address.
+- Local multilingual `Xenova/bge-m3` embeddings at 1024 dimensions.
+- One embedding worker and ingest concurrency 1. This avoids treating known
+  concurrent-local-embedding instability as a retrieval-quality result.
+- Extraction/summarization uses an operator-supplied OpenAI-compatible LLM.
+- Resource ceiling: 4 CPUs and 8 GiB RAM. Supermemory Local keeps its corpus
+  in memory, so RSS must be watched as the sample grows.
+
+Do not change the embedding model or dimensions in place. Use a fresh data
+directory and re-ingest when comparing a different embedding plan.
+
+## Before deployment
+
+1. Confirm the chosen `SUPERMEMORY_IPV4_ADDRESS` is absent from both Arcane's
+   `br0` attachments and the LAN neighbor/DHCP tables.
+2. Add `SUPERMEMORY_IPV4_ADDRESS`, `OPENAI_API_KEY`, and optional model/base URL
+   overrides to this Project's Arcane environment. Do not put secrets in Git,
+   Compose, activity notes, or chat.
+3. Confirm `/mnt/user/appdata` has room for the 298 MiB server binary, the
+   multilingual model cache, data, and rollback copy.
+4. Confirm no other build, image pull, backup, or migration is active on the
+   unRaid Environment.
+
+The `supermemory-fetch` init service downloads the exact release asset once,
+checks its SHA-256, and stores it in the POC appdata directory. Subsequent
+starts only verify the existing binary. No custom image build is required.
+
+## First boot and authentication
+
+The server creates its client bearer token at:
+
+`/mnt/user/appdata/supermemory-poc/data/api-key`
+
+Read it through an authorized unRaid/Arcane console without printing it into
+logs. Save it only as `SUPERMEMORY_API_KEY` in the `supermemory-lab` Hermes
+profile. The extraction LLM key and Supermemory client key are different.
+
+The current candidate address is `192.168.50.13`: it is not attached to an
+Arcane-managed `br0` container and did not answer the pre-deployment neighbor
+or ICMP probes. Confirm it is outside the DHCP pool or reserve it before the
+Project is started.
+
+Expected Hermes profile file `$HERMES_HOME/supermemory.json` (also provided as
+`hermes-supermemory.json.example`):
+
+```json
+{
+  "base_url": "http://<dedicated-ip>:6767",
+  "container_tag": "hermes_supermemory_lab",
+  "auto_recall": true,
+  "auto_capture": true,
+  "max_recall_results": 10,
+  "profile_frequency": 50,
+  "capture_mode": "all",
+  "search_mode": "hybrid",
+  "api_timeout": 10.0
+}
+```
+
+Run `poc.py smoke` before importing any sampled production material. It checks
+the v3 document path plus the v4 search, profile, and conversation endpoints
+used by Hermes.
+
+## Evaluation guardrails
+
+- Score raw Supermemory API retrieval separately from the Hermes answer. In
+  Hermes Agent v0.21.1, the provider reads result metadata but omits it from
+  automatic prefetch text and from the explicit search tool response. A raw
+  hit with correct `source`/`original_id` followed by an uncited Hermes answer
+  is an adapter attribution gap, not a retrieval miss.
+- Do not run the fixed scored question set with automatic writes enabled. The
+  default Hindsight profile has `auto_retain: true`, and this lab profile has
+  `auto_capture: true`; earlier test answers could leak into later questions.
+  Use a read-only Hindsight snapshot/profile and set Supermemory
+  `auto_capture: false` for the retrieval benchmark. Re-enable capture only for
+  the separately scored session-experience phase.
+- Hermes built-in `MEMORY.md` and `USER.md` remain active independently of the
+  external provider. Record their presence and size for each arm so built-in
+  context is not credited to Hindsight or Supermemory.
+- Self-hosted Supermemory has no managed Google Drive, Gmail, Notion, or
+  OneDrive connectors. This POC evaluates explicit file/document ingestion,
+  not hosted connector sync.
+
+## Staged data flow
+
+1. `poc.py sample-hindsight` reads active observations from the supported
+   Hindsight 0.8.4 list API and writes a local JSONL artifact. It never mutates
+   the source bank.
+2. Manually review that artifact for scope and sensitive content.
+3. `poc.py ingest-jsonl` writes the approved rows serially to the isolated
+   Supermemory container using stable custom IDs and provenance metadata.
+4. `poc.py ingest-files` uploads a small, separately reviewed
+   DEVONthink/Obsidian manifest. Keep the original UUID/path in metadata and
+   treat the source system as authoritative. The default per-file cap is 50
+   MiB so an accidental library-wide import fails closed.
+5. Run the same questions against the default and `supermemory-lab` profiles;
+   record groundedness, source traceability, latency, tokens, and cost.
+
+Artifacts are ignored by Git. Do not put exported observations or document
+content in this configuration repository.
+
+Hermes sanitizes container tags by replacing hyphens with underscores. Keep
+the canonical `hermes_supermemory_lab` spelling in direct API imports so they
+land in the same container queried by the profile.
+
+## Rollback
+
+Stop/down only this Project. Keep `/mnt/user/appdata/supermemory-poc` for later
+inspection, or archive it before any separately approved deletion. The default
+Hermes profile and Hindsight service require no rollback because this project
+does not modify them.