Knowtation Hub on ICP

This folder contains the ICP canister implementation of the Knowtation Hub API. The same contract as the Node server (see docs/HUB-API.md) is implemented here so that the Hub UI and CLI can talk to either self-hosted (Docker) or hosted (ICP) deployment.

Contract

• Auth: For dev, use header X-Test-User or X-User-Id. In production the gateway sends x-user-id derived from the JWT that the canister trusts; see HUB-API.md.
• Endpoints: GET /health, GET /api/v1/operator/export (paginated user-index for backups; header X-Operator-Export-Key; secret set by controllers via admin_set_operator_export_secret — see OPERATOR-BACKUP.md), GET /api/v1/notes, GET /api/v1/notes/:path, DELETE /api/v1/notes/:path, POST /api/v1/notes, POST /api/v1/notes/batch (bulk write, single stable save), POST /api/v1/notes/delete-by-prefix (bulk delete by vault-relative path prefix), GET /api/v1/export, GET /api/v1/vaults, GET/POST /api/v1/proposals, GET /api/v1/proposals/:id, POST /api/v1/proposals/:id/evaluation, POST /api/v1/proposals/:id/review-hints, POST /api/v1/proposals/:id/enrich (stores LLM enrich fields; gateway runs the model on hosted), POST /api/v1/proposals/:id/approve, POST /api/v1/proposals/:id/discard. POST /api/v1/notes/delete-by-project and POST /api/v1/notes/rename-project are not canister routes — on hosted, the gateway implements them by calling the endpoints above (HUB-METADATA-BULK-OPS.md). Notes and export are scoped by X-Vault-Id (default default). Search and settings are not in the canister (gateway/bridge in hosted mode).
• Storage: Vault (path → frontmatter/body) and proposals per user in canister stable memory.
• Proposal enrich JSON: src/hub/JsonValidate.mo validates suggested_labels_json (must be a JSON array) and assistant_suggested_frontmatter_json (must be a JSON object) on POST …/enrich; invalid payloads coerce to [] / {}, oversized valid payloads return 400. GET …/proposals/:id normalizes those fragments so the response body is always valid JSON (protects all clients, including backup).

Pre-deploy safety (recommended)

Before dfx deploy --network ic, from the repository root:

All-in-one (recommended): loads .env if present, can sync main, defaults backup URL from canister_ids.json when you set only KNOWTATION_CANISTER_BACKUP_USER_ID:

npm run canister:release-prep
# Include: git checkout main && git pull (requires clean working tree):
npm run canister:release-prep -- --sync-main

Lower-level (same checks, no git / .env / URL defaulting):

npm run canister:preflight
# or: bash scripts/canister-predeploy.sh

Both run migration shape checks (npm run canister:verify-migration), npm test, and dfx build hub --network ic (matches canister_ids.json; plain dfx build hub targets local and fails with “Cannot find canister id” until you run dfx canister create hub on a local replica). Override: DFX_PREFLIGHT_NETWORK=local after local create. Optional JSON backup: set KNOWTATION_CANISTER_BACKUP_USER_ID (and optionally KNOWTATION_CANISTER_URL; omitted URL is derived from canister_ids.json). Preflight delegates to scripts/canister-export-backup.mjs via npm run canister:export-backup (notes + proposals for one X-User-Id partition; optional encrypt + S3 — see script header). Exports land under backups/ (gitignored). If dfx crashes with ColorOutOfRange, use SKIP_DFX_BUILD=1 after you have built successfully elsewhere, or upgrade dfx.

Full canister state (hub + attestation): see OPERATOR-BACKUP.md; npm run canister:snapshot-backup (controller dfx; downtime during stop).

Build and deploy

1. Install DFX (includes the Motoko compiler).
2. From this directory:

   dfx start   # optional: local replica
   dfx deploy  # or dfx deploy --network ic
   

3. After deploy, the canister is callable at:
   • Local: http://localhost:4943/?canisterId=<canister-id>
   • IC: https://<canister-id>.ic0.app
4. CORS: The canister sets Access-Control-Allow-Origin: * and allows GET, POST, OPTIONS and headers Authorization, Content-Type, X-Vault-Id, X-User-Id, X-Test-User so the Hub UI (e.g. on 4Everland) can call it when configured with this API base URL.

Canister ID and URL

• After dfx deploy, run dfx canister id hub to get the canister ID.
• Use that ID in the URL above. For the Hub UI, set the API base to the gateway URL (which proxies to the canister with auth) or, for local dev, the canister URL with X-Test-User: default (or another user id) on requests.

Stable memory upgrades (mainnet)

If dfx deploy fails with M0170 / “new type of stable variable storage is not compatible”, the on-chain stable type no longer matches the migration hook’s input type in Migration.mo. The hub actor uses (with migration = Migration.migration); the hook’s parameter type is the previous on-chain StableStorage shape.

• Production (mainnet still V1 in 2026-03): Migration.migration maps StableStorageV1 → StableStorage: first migrateFromV1ToV2Eval (adds evaluation Text fields on each proposal), then v2ToV3 (adds review_queue, review_severity, auto_flag_reasons_json, review_hints, review_hints_at, review_hints_model). If dfx deploy failed with M0170 and “Unsupported additional field evaluated_at”, the hook had wrongly been typed as StableStorageV2 while the canister was still V1 — fixed in Migration.mo.
• V4 (LLM Enrich): Migration.migration maps StableStorageBeforeEnrich → StableStorage, adding per-proposal assistant_notes, assistant_model, assistant_at, suggested_labels_json (defaults empty / []). Earlier releases used an identity hook on StableStorage until enrich fields shipped.
• Stranded V0 canisters (pre–Phase 15.1 layout only): deploy an older git revision that still migrated from StableStorageV0, or reinstall an empty canister.

V0 meant one note map per user; V1 is multi-vault (userId, vaultId) + billingByUser + vault_id on proposals; migrated notes use vault id default. V2 adds human evaluation fields on each proposal. V3 adds review-routing and optional hint fields; see PROPOSAL-LIFECYCLE.md. V4 adds optional LLM Enrich fields (assistant_*, suggested_labels_json).

Plan any stable change with migration notes in Migration.mo and redeploy discipline. After a one-way upgrade has run on mainnet, a later release may only simplify migration if Motoko compatibility allows (see Motoko upgrades).

ICP HTTP gateway behavior (hosted)

• Every browser request is delivered to the canister’s http_request (query) first. POST mutations are not routed directly to http_request_update; the gateway only calls http_request_update after http_request returns upgrade = ?true. See Upgrading HTTP calls to update calls and HTTPS gateways and incoming requests.
• The gateway may set HttpRequest.url to a full URL (e.g. https://<canister>.icp0.io/api/v1/notes?...). Routing must normalize to the path (e.g. /api/v1/notes) before matching; pathOnly + parsePath in main.mo do that.
• Without upgrade on POST, the canister’s query handler falls through to the generic 404 body {"error":"Not found","code":"NOT_FOUND"} — the same JSON the Hub shows when listing or creating notes.

Implementation status

• Implemented: Motoko canister in src/hub/main.mo: vault (notes list/get/write/delete, bulk delete by prefix), proposals (list/get/create/evaluation/review-hints/enrich/approve/discard with evaluation gate + waiver on approve), health, CORS. User from header; stable storage. HTTP upgrade for POST and URL path normalization as above. Hosted gateway runs the LLM for Enrich and POSTs stored fields to the canister; it adds evaluation_checklist_json from the UI checklist array before proxying POST …/evaluation to the canister, and merges policy + review triggers on POST …/proposals.
• Not in canister: Search, settings, vault sync — handled by gateway/bridge in the hosted product (see plan phases 2–4). POST /api/v1/notes/delete-by-project and POST /api/v1/notes/rename-project (bulk ops by frontmatter/path-inferred project slug) are Node Hub only in-repo; see HUB-METADATA-BULK-OPS.md for the hosted strategy and parity notes.

Reference

• HUB-API.md — full API and auth
• OPERATOR-BACKUP.md — exports and backups