Key Material Security Audit — Muse + MuseHub

Scope: muse and musehub repos only. Agentception, Stori, Maestro deferred.
Status: Phases 1–8 complete.

The Target Architecture

OS Keychain  ←→  one master BIP39 mnemonic per machine (not per hub)
                  │
                  ▼  mnemonic_to_seed()   [in memory, never logged]
                  64-byte seed
                  │
                  ▼  derive_identity_key(seed, ...)   SLIP-0010 Ed25519
                  DerivedKey  →  Ed25519PrivateKey   [in memory]
                  │              │
                  ▼              ▼
               dk.zero()     sign(canonical_message)   [in memory]
                              │
                              ▼
                           sig bytes  →  base64url  →  Authorization header

NO PEM EVER WRITTEN TO DISK.
NO INTERMEDIATE KEY BYTES OUTSIDE MEMORY.
ONE MNEMONIC. MANY HUBS.

Current State — What Exists Today

muse/core/keychain.py

• Stores mnemonic in OS Keychain via keyring library. ✅
• Key: service="muse", username="mnemonic" (single global entry — not per-hub). ✅ (Phase 1)
• Legacy per-hub "{hostname}/mnemonic" entries migrated transparently on first load(). ✅
• MUSE_KEYCHAIN_BACKEND=disabled for CI is correct. ✅
• load() / store() / delete() API is clean. ✅

muse/core/identity.py

• Mnemonic never written to TOML — stripped before _save_all(). ✅
• Injected at load time via kc_load() in load_identity(). ✅
• resolve_signing_identity() derives key from keychain mnemonic + hd_path — no PEM read. ✅ (Phase 2)
• TOML written atomically with fchmod(0o600) before data. ✅
• Symlink guard on write. ✅
• Advisory fcntl.flock on read-modify-write. ✅
• key_path: str still in IdentityEntry TypedDict and _dump_identity() serialiser. ⚠️ (Post-Phase-8 finding — see audit section below)
• _load_all() still parses key_path from TOML — intentional forward-compat. ✅

muse/core/keypair.py

• generate_hd_keypair, load_private_key, load_private_key_from_pem, _write_private_key_pem, _key_path, _hostname_key, _SAFE_AGENT_ID, _ensure_keys_dir — all deleted. ✅ (Post-Phase-8 cleanup)
• derive_hd_public_info() is the only key derivation path — no file writes. ✅ (Phase 4)
• _KEYS_DIR sentinel kept — used by cleanup-keys and security-check commands. ✅
• DerivedKey.zero() wrapped in try/finally at all call sites. ✅ (Phase 6)
• ~/.muse/keys/ contains no PEM files — all orphans destroyed by Phase 5. ✅

muse/core/hdkeys.py

• Derivation path structure is correct and well-documented. ✅
• derive_agent_sub_seed() returns SecretByteArray (auto-zeroes on GC). ✅ (Phase 6)
• dk.zero() called in derive_agent_sub_seed() inside try/finally. ✅ (Phase 6)
• public_bytes_from_seed() calls dk.zero() inside try/finally. ✅ (Phase 6)
• All intermediate DerivedKey objects passed through child_key() loop are zeroed. ✅

muse/core/msign.py

• build_msign_header() takes a SigningIdentity (handle + private key). ✅
• Signing happens in memory. ✅
• No key material logged or included in exceptions. ✅
• DEFAULT_SIGN_ALGO used correctly (not KeyAlgorithm enum). ✅ (fixed in previous session)

muse/core/transport.py — SigningIdentity

• Holds handle: str and private_key: Ed25519PrivateKey. ✅
• private_key now derived from keychain mnemonic via resolve_signing_identity() — no disk read. ✅ (Phase 2)

muse/cli/commands/auth.py

• run_keygen (human): generates mnemonic → keychain → derives in memory → writes pubkey only. No PEM write. ✅ (Phase 4)
• run_register: derives key from keychain mnemonic + hd_path, writes no key_path to entry. ✅ (Phase 4)
• run_recover: derives in memory from stdin mnemonic → stores in keychain; no PEM write. ✅ (Phase 8)
• run_rotate: reads mnemonic from keychain; derives in memory; no PEM write. ✅ (Phase 8)
• Mnemonic passed via --mnemonic-fd N (fd 0/1/2 reserved). ✅
• Mnemonic never echoed to stdout/stderr. ✅
• Keychain storage is now machine-global ("mnemonic" key, not per-hub). ✅ (Phase 1)

musehub/auth/request_signing.py

• Server-side verification: reads public key bytes from DB, calls verify_signature(). ✅
• No private key material on the server at all. ✅
• Streaming push (application/x-muse-wire) signs with b"" body. ✅ (fixed in previous session)
• DEFAULT_SIGN_ALGO used for canonical message default. ✅ (fixed in previous session)

Identified Issues (Ranked by Severity)

CRITICAL

HIGH

MEDIUM

LOW

Implementation Plan

Phase 1 — Fix the Mnemonic Keychain Key (C3) ✅ COMPLETE

Goal: One master mnemonic per machine, not one per hub.

• [x] Change _username() to return "mnemonic" (constant, no hub in key)
• [x] Write migration: if "{hostname}/mnemonic" exists and "mnemonic" does not, copy it over and delete the old entry
• [x] Run migration in load() transparently (one-time, idempotent)
• [x] Update store() and delete() — hub_url param removed
• [x] Update all callers of kc_store / kc_load in auth.py
• [x] Update tests in tests/test_core_keychain.py

Phase 2 — Derive and Sign in Memory (C1, C2, H1) ✅ COMPLETE

Goal: resolve_signing_identity() derives the Ed25519 key from the mnemonic at call time. No PEM file read. No PEM file write.

Call chain (implemented):

resolve_signing_identity(hub_url)
  → load_identity(hub_url)            # reads identity.toml (handle, hd_path, fingerprint)
  → kc_load()                         # mnemonic from OS keychain
  → mnemonic_to_seed(mnemonic)        # BIP39 PBKDF2
  → derive_path(seed, hd_path)        # SLIP-0010 Ed25519
  → to_ed25519_private_key(dk)        # materialise key
  → dk.zero()                         # zero DerivedKey immediately
  → Ed25519PrivateKey                 # sign, then let GC handle it

• [x] Rewrite resolve_signing_identity() to derive from keychain — no PEM read
• [x] Delete generate_hd_keypair() from keypair.py — done in post-Phase-8 cleanup
• [x] Delete load_private_key() / load_private_key_from_pem() / _write_private_key_pem() from keypair.py — done in post-Phase-8 cleanup
• [x] Delete _check_key_file_permissions() and _load_private_key_from_path() from identity.py — done in post-Phase-8 cleanup

Phase 3 — Remove key_path from identity.toml Schema (C4) ⚠️ PARTIAL

Goal: The key_path field is meaningless once Phase 2 lands. Remove it from the type, the serialiser, and the parser.

• [ ] Remove key_path from IdentityEntry TypedDict — ⚠️ still present (post-Phase-8 audit finding)
• [x] _load_all() parser still reads key_path from TOML for forward-compat — acceptable
• [ ] Remove key_path from _dump_identity() serialiser — ⚠️ still writes it if present in entry
• [x] save_identity() silently drops key_path if present in a loaded file — ⚠️ actually NOT dropped; _dump_identity re-serialises it
• [ ] Updated all tests that set or assert on key_path — ⚠️ many tests still set key_path in fixture entries; 4 tests fail because key_set now uses hd_path not key_path

Phase 4 — Update auth.py CLI Commands (H2, H4) ✅ COMPLETE

Goal: keygen and register no longer write PEM files or read them.

• [x] run_keygen: generate mnemonic → kc_store() → derive in memory → store pubkey/fingerprint/hd_path. No PEM write.
• [x] run_register: derive key from keychain mnemonic + hd_path — no load_private_key, no key_path in entry.
• [x] run_keygen reuses existing mnemonic from keychain unless --force
• [x] Data-integrity invariants tested: hd_path preserved, no key_path, fingerprint matches mnemonic, round-trip works (test_auth_register_integrity.py)
• [x] run_recover: derives in memory from stdin mnemonic → stores mnemonic in keychain; no PEM write (Phase 8)
• [x] run_rotate: reads mnemonic from keychain; derives in memory; no PEM write (Phase 8)

Phase 5 — Orphan PEM Cleanup ✅ COMPLETE

Goal: Remove existing PEM files from disk. They are now vestigial and represent unprotected key material.

• [x] muse auth cleanup-keys: overwrites each *.pem with os.urandom bytes, fsyncs, unlinks; JSON output
• [x] muse auth security-check: four invariant checks; exits 1 on any failure; JSON output
• [x] 6 stale PEM files destroyed from real ~/.muse/keys/ on first run
• [x] Tests: C1–C5 (cleanup), S1–S5 (security-check) — all green (test_cmd_auth_phase5.py)

Phase 6 — DerivedKey Zeroing Hardening (M1, M2) ✅ COMPLETE

Goal: Best-effort zeroing of sensitive bytes in Python's heap. We cannot zero cryptography's C-heap allocations, but we can minimize exposure window.

Gaps identified (all code paths, as of Phase 5):

Implementation targets:

• [x] DerivedKey.__del__ added — calls self.zero() as GC safety net
• [x] try/finally wrapping all dk.zero() sites:
  • identity.py::resolve_signing_identity._derive — exception now returns None, dk always zeroed
  • keypair.py::derive_hd_public_info
  • keypair.py::generate_hd_keypair
  • hdkeys.py::public_bytes_from_seed
  • auth.py::run_register inline derivation
• [x] SecretByteArray added to slip010.py — bytearray subclass with zero() method and __del__ auto-zero
• [x] derive_agent_sub_seed() return type changed from bytearray to SecretByteArray
• [x] Tests: Z1–Z7 all green (test_security_zeroing.py)

Phase 7 — MuseHub Server Audit ✅ COMPLETE

Goal: Confirm server never touches private key material.

• [x] Verify musehub/auth/request_signing.py only reads public keys from DB — confirmed
• [x] Verify no private key material in musehub/crypto/keys.py — confirmed
• [x] Verify MusehubAuthKey DB model stores only public_key_b64 and fingerprint — confirmed
• [x] Fixed stale ~/.muse/keys/{{hostname}}.pem reference in musehub/mcp/prompts.py
• [x] Fixed key_path / ~/.muse/keys/ references in docs_muse_identity.html template
• [x] 10 tests P7-1 through P7-6 all green (tests/test_security_server_audit.py)

Phase 8 — Migrate run_recover and run_rotate off PEM files ✅ COMPLETE

Goal: The last two CLI commands that wrote PEM files now derive keys in memory only. No PEM files written anywhere in the CLI.

• [x] run_recover: replaced generate_hd_keypair with derive_hd_public_info; removed key_path from entry and JSON; stores mnemonic in OS keychain; guards on identity entry (not PEM file) for --force check; removed key_path from _RecoverJson TypedDict
• [x] run_rotate: reads mnemonic from keychain (kc_load()) — no _read_mnemonic_securely stdin call; replaced generate_hd_keypair with derive_hd_public_info; removed key_path from entry and JSON; removed key_path from _RotateJson TypedDict
• [x] test_cmd_auth_phase8.py — 12 tests (REC-1 through REC-7, ROT-1 through ROT-4) all green; written TDD-first
• [x] test_auth_rotate.py — keychain patching added to isolated fixture; _rotate() no longer passes stdin; test_III2_new_pem_is_valid_ed25519 replaced with test_III2_rotate_writes_no_pem; all 10 green
• [x] test_hd_keygen_unified.py::TestRunRecover — test_recover_writes_pem replaced with test_recover_writes_no_pem; test_recover_pem_mode_600 deleted; keychain patching added to _do_recover; all green

Test Checklist

• [x] test_core_keychain.py — stale key_path fields removed from test entries
• [x] test_hd_keygen_unified.py — all PEM assertion tests replaced; keychain patching added; no-PEM + fingerprint-based tests green
• [x] test_cmd_auth_keygen_hd.py — PEM write tests replaced with derive_hd_public_info tests; stale imports removed
• [x] test_agent_signing.py — TestLoadPrivateKeyFromPem deleted; stale key_path removed; Phase 2 in-memory signing tests green
• [x] test_security_key_permissions.py — deleted; replaced by test_security_no_pem_on_disk.py (NP-1 through NP-5)
• [x] test_auth_rotate.py — PEM assertions replaced with no-PEM assertions; keychain patching added; all 10 green (Phase 8)
• [x] test_resolve_signing_identity_keychain_path.py — new; full chain KC-1 through KC-7 all green

Migration Path (Zero-Downtime)

1. Phase 1 lands first — mnemonic consolidation. Existing staging.musehub.ai/mnemonic is migrated to mnemonic transparently on first load(). Localhost entry (missing) is created via muse auth recover using the staging mnemonic.
2. Phase 2 + 3 land together — resolve_signing_identity reads from keychain. PEM files still on disk but no longer read. Identity.toml drops key_path.
3. Phase 4 lands — CLI commands stop writing PEMs.
4. Phase 5 lands — PEM files actively overwritten and deleted.
5. Phase 6 — ongoing hardening, no user-visible change.

Post-Phase-8 Completeness Audit

Findings recorded as discovered. Each item is either clean ✅, needs a doc/comment fix 📝, or is a real code issue ⚠️.

Deleted functions — still referenced

key_path still in production code

Test files calling deleted _key_path function

key_set logic broken — 4 test failures

_display_entry was updated to use key_set = bool(hd_path) (not bool(key_path)). Test fixtures that set key_path but not hd_path now get key_set = false, breaking tests that expected key_set = true.

Test files using key_path in identity entries (stale fixture data)

Docs and comments with stale key_path / PEM content

Files to Delete (deferred cleanup)

All deferred deletions are now complete as of the post-Phase-8 cleanup commit.