# musehub — Agent Configuration

This repository is a member of a workspace.
Shared workspace rules live in the parent ``.muse/agent.md``.
This file contains only musehub-specific additions.

Managed by `muse agent-config` — regenerate adapters with `muse agent-config sync`.

---

## Proposal Titles

Proposal titles must be plain English — not branch-name style.

The branch (`feat/auth-v2`, `task/proposal-models-v2`) already encodes the type prefix.
The title is what a human reads in the list; it should describe the change, not echo the branch.

| Branch | Bad title | Good title |
|--------|-----------|------------|
| `feat/auth-v2` | `feat: auth v2` | `Ed25519 key rotation and MSign auth v2` |
| `task/proposal-models-v2` | `feat: proposal models v2` | `Proposal type badges, 7-state tabs, and ghost object integrity fix` |
| `fix/wire-timeout` | `fix: wire timeout` | `Increase push stream timeout to prevent drops on large repos` |

Rule: if the title could be mistaken for a branch name or a commit message subject line, rewrite it.

---

## Localhost Containers — ALWAYS Restart BOTH After Code Changes

Two containers run this repo's Python code, and **both** run without hot-reload —
code changes to the live-mounted volume are not picked up until each is restarted:

| Container | Compose service | Runs |
|---|---|---|
| `musehub` | `musehub` | the API (uvicorn, no `--reload`) |
| `musehub_worker` | `worker` | the background job processor (`mpack.index`, `fetch.mpack.prebuild`) |

`musehub-runner` (docker-in-docker sandbox) and `musehub_postgres`/`musehub_minio`
(datastores) do **not** run this repo's Python and never need restarting for a code
change.

**A real incident that cost real time:** `musehub_worker` was found running a
Docker image dated **over a month old** while `musehub` itself had been restarted
recently — because only `musehub`'s restart was a documented habit. Every
job-side observation made during that window (prebuild behavior, cache
invariants, mpack consolidation) was **silently wrong**, and it took a deep,
multi-hour investigation to notice the container itself was stale rather than
the code. Restarting only `musehub` and assuming the worker is also fresh is a
trap — **always restart both, together, every time:**

```bash
docker restart musehub musehub_worker && sleep 5 && curl -sk https://localhost:1337/healthz
```

### Verify freshness before trusting any test result

A restart alone does not guarantee correctness — confirm it actually worked:

```bash
# 1. Confirm both containers actually restarted (not just "Up", but recently)
docker ps --format "{{.Names}}\t{{.Status}}" | grep -E "^musehub(\s|_worker)"
# "Up X seconds" — if either shows days/weeks, the restart didn't take or targeted the wrong name

# 2. Confirm the API booted clean — look for these exact lines, not just "started"
docker logs musehub --tail 20 | grep -i "schema check passed\|Application startup failed"
# ✅ "schema check passed — ORM and DB are in sync (N tables)"
# ❌ "Application startup failed" — see schema-drift section below, do not proceed until fixed

# 3. Confirm the worker booted clean
docker logs musehub_worker --tail 20 | grep -i "worker started\|Application startup failed"
# ✅ "MuseHub worker started"
```

**If you're mid-investigation and something doesn't match your mental model of
current code — check container freshness FIRST, before re-reading source or
building new theories.** Compare image build time against your last relevant
commit:

```bash
docker inspect musehub --format '{{.Image}}' | xargs docker image inspect --format '{{.Created}}'
docker inspect musehub_worker --format '{{.Image}}' | xargs docker image inspect --format '{{.Created}}'
```

If either predates a commit you're relying on, restart doesn't help — see
"When you need a rebuild, not just a restart" below.

### When you need a rebuild, not just a restart

Plain `docker restart` is enough for `.py` edits under the bind-mounted
`musehub/` directory (see `docker inspect musehub --format '{{range .Mounts}}...'`
for the exact list — `musehub/musehub`, `musehub/alembic`, `musehub/tests`, etc.
are all live bind mounts). It is **not** enough when:

- `Dockerfile` changed
- `requirements*.txt` / dependency pins changed
- `docker-compose.yml` / `docker-compose.override.yml` changed

In those cases, rebuild the image before restarting:

```bash
cd ~/ecosystem/musehub
docker compose build musehub worker
docker compose up -d musehub worker
```

### Schema drift and missing migrations — two distinct failure modes

Both show up as `Application startup failed` in `docker logs musehub`, but need
different fixes. **Read the actual error before acting** — do not assume
`alembic upgrade head` fixes every schema error:

**Mode 1 — DB behind the migrations that already exist.** Error names a table/
column the ORM expects. Fix:
```bash
docker exec musehub sh -c "cd /app && alembic upgrade head"
```

**Mode 2 — an ORM model was added with no migration ever generated for it.**
`alembic current` already reports `(head)` — there is nothing to "catch up" on;
the migration simply doesn't exist. Confirm which mode you're in:
```bash
docker exec musehub sh -c "cd /app && alembic current"
docker exec musehub sh -c "cd /app && alembic heads"
# If current == heads and the app still fails on a missing table → Mode 2
```
Fix by generating the missing migration, then **rename it to match this repo's
numbered convention** (`00NN_description.py`, not the default timestamp/hex
name) before applying:
```bash
docker exec musehub sh -c "cd /app && alembic revision --autogenerate -m 'add <table>'"
# rename the generated file: alembic/versions/<timestamp>_<hex>_....py → 00NN_description.py
# edit revision/down_revision inside to '00NN'/'00NN-1' matching the new filename
docker exec musehub sh -c "cd /app && alembic upgrade head"
docker restart musehub musehub_worker
```
If the container already auto-applied the pre-rename autogenerated revision at
startup (common — many app entrypoints run migrations before the schema check),
re-stamp the DB to the renamed revision id instead of re-running upgrade:
```bash
docker exec musehub_postgres psql -U musehub -d musehub -c "UPDATE alembic_version SET version_num = '00NN';"
```

---

## Repo-Specific Notes

### Deploying MuseHub and publishing a new Muse CLI build

Full runbook: `docs/deploy.md`. Quick reference:

```bash
# Deploy MuseHub server (staging)
cd ~/ecosystem/musehub
bash deploy/push.sh staging

# Publish a new Muse CLI tarball (always AFTER deploying the server)
cd ~/ecosystem/musehub
bash deploy/publish_muse_release.sh
```

**Standard release flow (muse and musehub version independently — see below):**
1. Bump `version` in whichever of `muse/pyproject.toml` / `musehub/pyproject.toml` actually changed this cycle — bump both only if both genuinely changed. Commit each bump on its own repo's `dev`.
2. `muse -C ~/ecosystem/muse push local dev && muse -C ~/ecosystem/muse push staging dev`
3. `bash deploy/push.sh staging` — server must be at the new version before the tarball is published (only needed if musehub changed)
4. `bash deploy/publish_muse_release.sh` (only needed if muse changed)
5. Verify: `curl -fsSL https://staging.musehub.ai/install.sh | sh && muse --version`

**Muse and musehub versions are independent, not lockstep.** This mirrors the
industry-standard pattern for a CLI/server pair (git/GitHub, kubectl/the
Kubernetes API server, Terraform CLI/Terraform Cloud) — the client and
server evolve on their own cadence, and a fix confined to one repo doesn't
force a meaningless version bump (and empty-diff "release") on the other.
Prior to 2026-09-18 this file said both repos shared one version string;
that convention was dropped because it kept forcing exactly this kind of
no-op bump. If the two ever need an explicit compatibility signal (e.g. a
minimum muse CLI version a given musehub server requires), add that as its
own documented check — don't reach back for forced lockstep versioning to
get it.

### Recovering a Down Staging or Production Instance

**Production and staging live in two separate, isolated AWS accounts as of the
2026-08-25 rebuild — always set `AWS_PROFILE` (or `--profile`) explicitly, never
rely on the default profile.** The default/legacy `musehub-infra` credentials
only reach the Nonproduction account (staging, plus the retired legacy prod
box below) — they cannot see or touch the real production account at all, and
will not error loudly if you point them at the wrong instance ID; they'll just
silently operate on whatever exists in Nonproduction, which cost real time
during the 2026-09-17 production outage. Use the SSO profiles:

```bash
export AWS_PROFILE=musehub-staging      # or pass --profile per-command
export AWS_PROFILE=musehub-production
aws sso login --profile musehub-production   # if the session expired
```

If musehub.ai or staging.musehub.ai returns 502/522 or times out:

1. **Restart the container** — containers use `--restart unless-stopped` so a reboot brings them back, but a manual `docker stop` does not:
   ```bash
   sudo docker start musehub-blue   # or musehub-green, whichever is active
   ```

2. **Switch the active slot** — the ONLY correct way to change which slot nginx points to:
   ```bash
   sudo musehub-set-slot blue    # blue  = port 1337
   sudo musehub-set-slot green   # green = port 1338
   ```
   This script lives at `/usr/local/bin/musehub-set-slot` on the instance. It is the only sanctioned way to write `/etc/nginx/musehub-active-port`. **Never write to that file directly** — it must contain a full nginx upstream directive (`server 127.0.0.1:1337;`), not a bare port number. Writing a bare port breaks nginx config parsing and causes 502s.

3. **Check current state**:
   ```bash
   cat /opt/musehub/.active-slot           # which slot is active (blue or green)
   cat /etc/nginx/musehub-active-port      # what nginx is pointing at
   sudo docker ps --format "{{.Names}} {{.Status}}"  # container health
   ```

4. **If SSM commands are stuck Pending, or `describe-instance-status` shows
   `InstanceStatus: impaired` with `SystemStatus: ok`** (guest OS network stack
   wedged, hypervisor fine — confirmed the real signature of the 2026-09-17
   production outage) — reboot the instance. Containers with
   `--restart unless-stopped` come back automatically:
   ```bash
   aws ec2 reboot-instances --region us-east-1 --instance-ids <instance-id>
   ```
   Then poll until SSM is back before sending new commands:
   ```bash
   aws ssm describe-instance-information --region us-east-1 \
     --filters "Key=InstanceIds,Values=<instance-id>" \
     --query 'InstanceInformationList[0].PingStatus' --output text
   ```

### Instance IDs

| Environment | AWS account | AWS profile | Instance ID |
|-------------|-------------|-------------|--------------|
| staging | Nonproduction (`992382692655`) | `musehub-nonproduction` (or legacy default `musehub-infra` creds) | `i-07547cd20bee2dea5` |
| **production** | **Production (`672469410277`)** | **`musehub-production`** | **`i-043aaed71bef11903`** |
| ~~prod (legacy)~~ | Nonproduction (`992382692655`) | n/a — no IAM instance profile ever attached, unreachable via SSM by design | ~~`i-0855d6efe7fa1a49d`~~ — retired 2026-08-25, not decommissioned yet, retirement deliberately deferred. **Do not use this for production incidents** — it hasn't served real traffic since the rebuild. |

Full account/instance detail, ECR registries, IAM roles, and the DNS/TLS setup
for both environments: `docs/infrastructure.md`.