151 lines
7.3 KiB
Markdown
151 lines
7.3 KiB
Markdown
# Supermemory POC on unRaid
|
|
|
|
This project is an isolated evaluation service. It does not replace the
|
|
production Hindsight service or change the default Hermes profile.
|
|
|
|
## Fixed deployment choices
|
|
|
|
- Supermemory Local `server-v0.0.8`, Linux x64 binary, verified by SHA-256.
|
|
- One self-contained server; state, auth material, uploaded files, and model
|
|
cache live under `/mnt/user/appdata/supermemory-poc/data`.
|
|
- A dedicated address on Docker `br0`; port `6767` is not published on the
|
|
unRaid host address.
|
|
- Native Gemini embeddings using `gemini-embedding-2` at 768 dimensions.
|
|
- Extraction, summarization, and other intelligent steps use Supermemory's
|
|
native Gemini provider default. This cleanly removes the LiteLLM and
|
|
OpenAI-compatible translation layer from the diagnostic path.
|
|
- Ingest concurrency remains 1 so the first POC emphasizes correctness and
|
|
observable cost over throughput.
|
|
- Resource ceiling: 4 CPUs and 8 GiB RAM. Supermemory Local keeps its corpus
|
|
in memory, so RSS must be watched as the sample grows.
|
|
|
|
Do not change the embedding model or dimensions in place. Use a fresh data
|
|
directory and re-ingest when comparing a different embedding plan.
|
|
|
|
## Before deployment
|
|
|
|
1. Confirm the chosen `SUPERMEMORY_IPV4_ADDRESS` is absent from both Arcane's
|
|
`br0` attachments and the LAN neighbor/DHCP tables.
|
|
2. Add `SUPERMEMORY_IPV4_ADDRESS` and `GEMINI_API_KEY` to this Project's
|
|
Arcane environment. Do not put the key in Git, Compose, activity notes, or
|
|
chat. Legacy `OPENAI_*` values may remain stored in Arcane for rollback,
|
|
but this Compose file does not inject them into the container.
|
|
3. Confirm `/mnt/user/appdata` has room for the 298 MiB server binary, the
|
|
multilingual model cache, data, and rollback copy.
|
|
4. Confirm no other build, image pull, backup, or migration is active on the
|
|
unRaid Environment.
|
|
|
|
The `supermemory-fetch` init service downloads the exact release asset once,
|
|
checks its SHA-256, and stores it in the POC appdata directory. Subsequent
|
|
starts only verify the existing binary. It also copies the CA bundle from the
|
|
pinned fetch image so the slim runtime can download local embedding model
|
|
artifacts over verified HTTPS. Both OpenSSL-style clients and the bundled Bun
|
|
runtime are pointed at that bundle, which is also mounted at Debian's standard
|
|
CA path for embedding workers. TLS verification remains enabled. No custom
|
|
image build is required.
|
|
|
|
## First boot and authentication
|
|
|
|
The server creates its client bearer token at:
|
|
|
|
`/mnt/user/appdata/supermemory-poc/data-gemini-embedding-2-768/api-key`
|
|
|
|
Read it through an authorized unRaid/Arcane console without printing it into
|
|
logs. Save it only as `SUPERMEMORY_API_KEY` in the `supermemory-lab` Hermes
|
|
profile. The extraction LLM key and Supermemory client key are different.
|
|
|
|
The current candidate address is `192.168.50.13`: it is not attached to an
|
|
Arcane-managed `br0` container and did not answer the pre-deployment neighbor
|
|
or ICMP probes. Confirm it is outside the DHCP pool or reserve it before the
|
|
Project is started.
|
|
|
|
Expected Hermes profile file `$HERMES_HOME/supermemory.json` (also provided as
|
|
`hermes-supermemory.json.example`):
|
|
|
|
```json
|
|
{
|
|
"base_url": "http://<dedicated-ip>:6767",
|
|
"container_tag": "hermes_supermemory_lab",
|
|
"auto_recall": true,
|
|
"auto_capture": true,
|
|
"max_recall_results": 10,
|
|
"profile_frequency": 50,
|
|
"capture_mode": "all",
|
|
"search_mode": "hybrid",
|
|
"api_timeout": 10.0
|
|
}
|
|
```
|
|
|
|
Run `poc.py smoke` before importing any sampled production material. It checks
|
|
the v3 document path plus the v4 search, profile, and conversation endpoints
|
|
used by Hermes.
|
|
|
|
### Embedding plan and smoke gate
|
|
|
|
The earlier local `Xenova/bge-m3` and LiteLLM-backed embedding experiments are
|
|
preserved under their old data directories because embedding plans cannot be
|
|
changed in place. This deployment uses the fresh
|
|
`data-gemini-embedding-2-768` directory. The one-shot `supermemory-smoke`
|
|
service verifies the locked embedding plan, tracks the exact document ID to a
|
|
terminal state, separately checks v3 search, v4 document search, and Chinese
|
|
hybrid search, checks provenance metadata, and exercises the profile and
|
|
conversation endpoints. It writes a sentinel after success so ordinary
|
|
redeploys do not add duplicate smoke documents. Do not import sampled
|
|
Hindsight or document material until this service exits successfully.
|
|
|
|
On 2026-09-08, the previous remote plan was accepted and locked as
|
|
`openai / google/gemini-embedding-2 / 1024`; the LiteLLM embedding and chat
|
|
endpoints also passed independently. Supermemory listed a completed smoke
|
|
document, but both v4 and v3 searches returned no hit. The old smoke mixed
|
|
list-wide status checks and did not track the newly created document, so this
|
|
native Gemini run uses a new store and a stricter document-specific gate.
|
|
|
|
## Evaluation guardrails
|
|
|
|
- Score raw Supermemory API retrieval separately from the Hermes answer. In
|
|
Hermes Agent v0.21.1, the provider reads result metadata but omits it from
|
|
automatic prefetch text and from the explicit search tool response. A raw
|
|
hit with correct `source`/`original_id` followed by an uncited Hermes answer
|
|
is an adapter attribution gap, not a retrieval miss.
|
|
- Do not run the fixed scored question set with automatic writes enabled. The
|
|
default Hindsight profile has `auto_retain: true`, and this lab profile has
|
|
`auto_capture: true`; earlier test answers could leak into later questions.
|
|
Use a read-only Hindsight snapshot/profile and set Supermemory
|
|
`auto_capture: false` for the retrieval benchmark. Re-enable capture only for
|
|
the separately scored session-experience phase.
|
|
- Hermes built-in `MEMORY.md` and `USER.md` remain active independently of the
|
|
external provider. Record their presence and size for each arm so built-in
|
|
context is not credited to Hindsight or Supermemory.
|
|
- Self-hosted Supermemory has no managed Google Drive, Gmail, Notion, or
|
|
OneDrive connectors. This POC evaluates explicit file/document ingestion,
|
|
not hosted connector sync.
|
|
|
|
## Staged data flow
|
|
|
|
1. `poc.py sample-hindsight` reads active observations from the supported
|
|
Hindsight 0.8.4 list API and writes a local JSONL artifact. It never mutates
|
|
the source bank.
|
|
2. Manually review that artifact for scope and sensitive content.
|
|
3. `poc.py ingest-jsonl` writes the approved rows serially to the isolated
|
|
Supermemory container using stable custom IDs and provenance metadata.
|
|
4. `poc.py ingest-files` uploads a small, separately reviewed
|
|
DEVONthink/Obsidian manifest. Keep the original UUID/path in metadata and
|
|
treat the source system as authoritative. The default per-file cap is 50
|
|
MiB so an accidental library-wide import fails closed.
|
|
5. Run the same questions against the default and `supermemory-lab` profiles;
|
|
record groundedness, source traceability, latency, tokens, and cost.
|
|
|
|
Artifacts are ignored by Git. Do not put exported observations or document
|
|
content in this configuration repository.
|
|
|
|
Hermes sanitizes container tags by replacing hyphens with underscores. Keep
|
|
the canonical `hermes_supermemory_lab` spelling in direct API imports so they
|
|
land in the same container queried by the profile.
|
|
|
|
## Rollback
|
|
|
|
Stop/down only this Project. Keep `/mnt/user/appdata/supermemory-poc` for later
|
|
inspection, or archive it before any separately approved deletion. The default
|
|
Hermes profile and Hindsight service require no rollback because this project
|
|
does not modify them.
|