Third-party domain plugins: entry_points discovery, install/uninstall UX, upgrade durability, and marketplace distribution

Why

Every domain today — code, midi, mist, todo, chess (Episode 22) — requires editing muse/plugins/registry.py directly and shipping it inside muse itself. That's correct for domains muse wants to bundle for everyone, but it's the wrong model for someone else's domain (genomics, circuit design, a hobby chess plugin) — they shouldn't need a PR into muse to use their own VCS domain, and we definitely don't want to ship every third party's domain to every muse user.

Raised by gabriel after Episode 22; this issue is the concrete plan, grounded in a full architecture read (not guessed):

• muse/plugins/registry.py's _REGISTRY is a static dict, eagerly instantiated at import time, with one env-var-gated precedent (MUSE_ENABLE_MIDI) and one private cross-module import (domains.py:73 reaches into _REGISTRY directly for the dashboard).
• Zero use of importlib.metadata/entry_points/pkg_resources anywhere in muse, musehub, or agentception today. No [plugins] config section, no .muse/plugins/ convention. This is a from-scratch mechanism, not a wire-up of something half-built.
• Two independent venvs matter a lot here: stable muse (~/.local/share/muse/venv, real non-editable install) and muse-dev (~/.local/share/muse-dev/venv, editable, points at ~/ecosystem/muse). importlib.metadata.entry_points() only sees whatever's installed in the currently running interpreter's venv — a plugin installed into one is invisible to the other. Any install UX has to say which venv it targeted, not assume one.
• install.sh/uninstall.sh (generated by MuseHub at musehub/api/routes/musehub/install.py, served at /install.sh) fully recreate/destroy ~/.local/share/muse/venv on every install and upgrade. Anything pip installed into that venv outside of muse itself — i.e. any third-party plugin — gets silently wiped the next time someone runs the installer. This has to be solved, not just noted.
• MuseHub already has half of a marketplace: muse domains publish/update/ delete/info and MusehubDomain/MusehubDomainInstall tables (musehub/db/musehub_domain_models.py) with real metadata (capabilities, viewer_type, install_count) and an explicit docstring anticipating @alice/genomics-style third-party domains. But there is no package name, download URL, or distribution artifact field anywhere in that schema — the publish/discover half exists; the fetch-and-install half is completely missing. This issue is what completes it.
• Found and worth fixing as part of this work, not a separate ticket, because it directly affects how a third party's plugin errors surface: muse init --domain <anything> never validates against the registry (muse/core/validation.py::validate_domain_name is a pure regex/format check, never consults _REGISTRY) — it silently writes an unregistered domain to repo.json. Every subsequent command then raises MuseCLIError from registry.py::resolve_plugin(), but grep -rn "except MuseCLIError" muse/ returns zero hits anywhere in the codebase, so it surfaces as a raw, unhandled Python traceback on every single command, including --json invocations (which get a traceback on stderr and no JSON envelope on stdout at all).
• muse/cli/commands/domains.py::_scaffold_new_domain (the existing muse domains new / scaffold command) writes the generated plugin into the installed package's own site-packages directory — it cannot target an out-of-tree, standalone package today. New scaffold target needed for phase 4.
• docs/guide/plugin-authoring-guide.md only documents in-tree authoring and is independently stale (references a StructuredMergePlugin that doesn't exist — the real name is AddressedMergePlugin — and an op_transform module that's actually op_merge). Needs a rewrite regardless; this is the natural place to land the new instructions.

Design

Use Python's standard entry_points mechanism — the same pattern pytest, flake8, and Sphinx use for third-party plugins — rather than a bespoke loader:

# a third-party plugin package's own pyproject.toml
[project.entry-points."muse.domains"]
genomics = "musedomain_genomics.plugin:GenomicsPlugin"

registry.py merges the built-in _REGISTRY dict with whatever importlib.metadata.entry_points(group="muse.domains") finds, lazily (resolved/imported on first use, not at module import — so a third party's broken or slow-importing plugin doesn't tax every muse invocation) and cached after the first successful build. Built-in names win on collision, loudly (a warning naming the shadowed third-party plugin), never silently.

Phases (each independently shippable and testable)

Phase 0 — Two pre-existing bugs this work depends on

• Wrap main() in muse/cli/app.py with a real except MuseCLIError as exc: handler that prints exc cleanly to stderr (respecting --json — emit the JSON error envelope, not a traceback) and exits with exc.exit_code, not Python's default 1.
• muse init --domain X must validate X against the merged registry (built-in + entry-point-discovered) at creation time, the same way domains.py's existing use subcommand already does (domains.py:1532-1539 — a real, working pattern to copy, not invent), with a clean error naming known domains.
• Regression tests for both: a bad --domain value gets a clean one-line error, not a traceback, in both plain and --json modes.

Phase 1 — Core discovery mechanism

• registry.py: merge _REGISTRY with importlib.metadata.entry_points(group="muse.domains"), lazily, cached, with loud collision handling as described above.
• Fix the private cross-module import at domains.py:73 (from muse.plugins.registry import _REGISTRY) to go through a proper public accessor — it can no longer assume the dict is the complete picture.
• registered_domains(), schema_for(), resolve_plugin(), resolve_plugin_by_domain() all reflect entry-point-discovered plugins, not just built-ins.
• Unit tests using a fake/fixture distribution (importlib.metadata supports this cleanly) — no real third-party package needed to test discovery.
• Update the (already-stale) module docstring's 'midi' fallback claim (registry.py:12-13, 60 — the real fallback is _DEFAULT_DOMAIN = "code") while touching this file.

Phase 2 — Install / uninstall CLI UX

• muse domains install <pip-spec-or-path>: detects the current interpreter's venv (mirror the diagnostic approach in scripts/dev/which-muse.sh — report which venv, don't assume), runs pip install into it, prints a clear "you are installing and running third-party code with full process privileges, not sandboxed" warning before proceeding.
• --dev/editable flag for plugin authors iterating locally (pip install -e), meant to pair with muse-dev.
• muse domains uninstall <name>.
• muse domains list --installed (or extend the existing domains dashboard) to distinguish built-in vs. entry-point-discovered domains.
• Respect dev_guard.py's existing editable-build detection — decide explicitly whether muse domains install --dev should be gated by the same guard rail as other mutating commands, and document the decision (the guard's docstring already scopes hub subcommands as out of scope for a reason — this is a new category, not obviously either bucket).

Phase 3 — Durability across muse upgrades (the "load-bearing" part)

• A plugin manifest recording installed third-party packages (name + version + install mode), stored outside the venv (~/.local/share/muse/ alongside, not inside, venv/) so it survives uninstall.sh's rm -rf "$VENV_DIR".
• install.sh/uninstall.sh (generated server-side by musehub/api/routes/musehub/install.py — this phase touches musehub, not just muse) need a hook: after recreating the venv, reinstall everything the manifest lists, or at minimum warn loudly that plugins were wiped and how to restore them.
• muse domains doctor (or fold into an existing health-check command): detect drift between the manifest and what's actually importable, offer to repair.
• Test: install a fixture plugin, simulate an upgrade (venv recreation), confirm the plugin is either preserved or the user gets an actionable, specific warning — never silent loss.

Phase 4 — Out-of-tree authoring support

• New scaffold target (muse domains new --out-of-tree <path> or similar) generating a standalone package — its own pyproject.toml with the muse.domains entry point pre-filled, plugin.py from the existing scaffold template, a tests directory — since _scaffold_new_domain today can only write into the installed package's own site-packages dir (domains.py::_scaffold_new_domain, computes its target from pathlib.Path(__file__).parents[2]) and that's fundamentally wrong for this use case.
• Rewrite docs/guide/plugin-authoring-guide.md: fix the stale StructuredMergePlugin/op_transform references while rewriting, add the out-of-tree path as a first-class option alongside in-tree, cover HarmonyPlugin/StagePlugin (currently missing from the guide's protocol ladder entirely), update the version header (says "v0.1.2").
• A real worked example: consider moving (or duplicating) the chess plugin from Episode 22 out-of-tree as the reference package for this guide — dogfooding the exact path being documented.

Phase 5 — Close the loop: MuseHub marketplace → real installs

• Add distribution-artifact fields to MusehubDomain (musehub/db/musehub_domain_models.py) — package name + index (PyPI or a private index) or a source URL. Today capabilities is pure descriptive metadata; nothing points at installable code.
• muse domains install @author/slug resolves the real package coordinates via the hub (extends muse domains info's existing lookup) and installs it — the fetch-and-load half connecting to the publish/discover half that already exists.
• Decide how is_verified (already a real column) factors into the install warning from Phase 2 — an unverified third-party domain should probably warn harder than a verified one.

Explicitly out of scope for this issue (flag, don't solve here)

• Sandboxing/isolating third-party plugin code — plugins run with full same-process privileges today and will continue to; this issue is about discovery and distribution, not a security boundary. Worth its own issue if it becomes a real ask.
• Signing/provenance verification of published domain packages beyond the existing is_verified flag.

Assigned to gabriel. Multi-phase, load-bearing — expect this to take real design review at each phase boundary, not a single sitting.