# musehub — Agent Configuration

This repository is a member of a workspace.
Shared workspace rules live in the parent ``.muse/agent.md``.
This file contains only musehub-specific additions.

Managed by `muse agent-config` — regenerate adapters with `muse agent-config sync`.

---

## Proposal Titles

Proposal titles must be plain English — not branch-name style.

The branch (`feat/auth-v2`, `task/proposal-models-v2`) already encodes the type prefix.
The title is what a human reads in the list; it should describe the change, not echo the branch.

| Branch | Bad title | Good title |
|--------|-----------|------------|
| `feat/auth-v2` | `feat: auth v2` | `Ed25519 key rotation and MSign auth v2` |
| `task/proposal-models-v2` | `feat: proposal models v2` | `Proposal type badges, 7-state tabs, and ghost object integrity fix` |
| `fix/wire-timeout` | `fix: wire timeout` | `Increase push stream timeout to prevent drops on large repos` |

Rule: if the title could be mistaken for a branch name or a commit message subject line, rewrite it.

---

## Localhost Containers — ALWAYS Restart BOTH After Code Changes

Two containers run this repo's Python code, and **both** run without hot-reload —
code changes to the live-mounted volume are not picked up until each is restarted:

| Container | Compose service | Runs |
|---|---|---|
| `musehub` | `musehub` | the API (uvicorn, no `--reload`) |
| `musehub_worker` | `worker` | the background job processor (`mpack.index`, `fetch.mpack.prebuild`) |

`musehub-runner` (docker-in-docker sandbox) and `musehub_postgres`/`musehub_minio`
(datastores) do **not** run this repo's Python and never need restarting for a code
change.

**A real incident that cost real time:** `musehub_worker` was found running a
Docker image dated **over a month old** while `musehub` itself had been restarted
recently — because only `musehub`'s restart was a documented habit. Every
job-side observation made during that window (prebuild behavior, cache
invariants, mpack consolidation) was **silently wrong**, and it took a deep,
multi-hour investigation to notice the container itself was stale rather than
the code. Restarting only `musehub` and assuming the worker is also fresh is a
trap — **always restart both, together, every time:**

```bash
docker restart musehub musehub_worker && sleep 5 && curl -sk https://localhost:1337/healthz
```

### Verify freshness before trusting any test result

A restart alone does not guarantee correctness — confirm it actually worked:

```bash
# 1. Confirm both containers actually restarted (not just "Up", but recently)
docker ps --format "{{.Names}}\t{{.Status}}" | grep -E "^musehub(\s|_worker)"
# "Up X seconds" — if either shows days/weeks, the restart didn't take or targeted the wrong name

# 2. Confirm the API booted clean — look for these exact lines, not just "started"
docker logs musehub --tail 20 | grep -i "schema check passed\|Application startup failed"
# ✅ "schema check passed — ORM and DB are in sync (N tables)"
# ❌ "Application startup failed" — see schema-drift section below, do not proceed until fixed

# 3. Confirm the worker booted clean
docker logs musehub_worker --tail 20 | grep -i "worker started\|Application startup failed"
# ✅ "MuseHub worker started"
```

**If you're mid-investigation and something doesn't match your mental model of
current code — check container freshness FIRST, before re-reading source or
building new theories.** Compare image build time against your last relevant
commit:

```bash
docker inspect musehub --format '{{.Image}}' | xargs docker image inspect --format '{{.Created}}'
docker inspect musehub_worker --format '{{.Image}}' | xargs docker image inspect --format '{{.Created}}'
```

If either predates a commit you're relying on, restart doesn't help — see
"When you need a rebuild, not just a restart" below.

### When you need a rebuild, not just a restart

Plain `docker restart` is enough for `.py` edits under the bind-mounted
`musehub/` directory (see `docker inspect musehub --format '{{range .Mounts}}...'`
for the exact list — `musehub/musehub`, `musehub/alembic`, `musehub/tests`, etc.
are all live bind mounts). It is **not** enough when:

- `Dockerfile` changed
- `requirements*.txt` / dependency pins changed
- `docker-compose.yml` / `docker-compose.override.yml` changed

In those cases, rebuild the image before restarting:

```bash
cd ~/ecosystem/musehub
docker compose build musehub worker
docker compose up -d musehub worker
```

### Schema drift and missing migrations — two distinct failure modes

Both show up as `Application startup failed` in `docker logs musehub`, but need
different fixes. **Read the actual error before acting** — do not assume
`alembic upgrade head` fixes every schema error:

**Mode 1 — DB behind the migrations that already exist.** Error names a table/
column the ORM expects. Fix:
```bash
docker exec musehub sh -c "cd /app && alembic upgrade head"
```

**Mode 2 — an ORM model was added with no migration ever generated for it.**
`alembic current` already reports `(head)` — there is nothing to "catch up" on;
the migration simply doesn't exist. Confirm which mode you're in:
```bash
docker exec musehub sh -c "cd /app && alembic current"
docker exec musehub sh -c "cd /app && alembic heads"
# If current == heads and the app still fails on a missing table → Mode 2
```
Fix by generating the missing migration, then **rename it to match this repo's
numbered convention** (`00NN_description.py`, not the default timestamp/hex
name) before applying:
```bash
docker exec musehub sh -c "cd /app && alembic revision --autogenerate -m 'add <table>'"
# rename the generated file: alembic/versions/<timestamp>_<hex>_....py → 00NN_description.py
# edit revision/down_revision inside to '00NN'/'00NN-1' matching the new filename
docker exec musehub sh -c "cd /app && alembic upgrade head"
docker restart musehub musehub_worker
```
If the container already auto-applied the pre-rename autogenerated revision at
startup (common — many app entrypoints run migrations before the schema check),
re-stamp the DB to the renamed revision id instead of re-running upgrade:
```bash
docker exec musehub_postgres psql -U musehub -d musehub -c "UPDATE alembic_version SET version_num = '00NN';"
```

---

## Repo-Specific Notes

### Deploying MuseHub and publishing a new Muse CLI build

Full runbook: `docs/deploy.md`. Quick reference:

```bash
# Deploy MuseHub server (staging)
cd ~/ecosystem/musehub
bash deploy/push.sh staging

# Publish a new Muse CLI tarball (always AFTER deploying the server)
cd ~/ecosystem/musehub
bash deploy/publish_muse_release.sh
```

**Standard release flow (both repos share the same version string):**
1. Bump `version` in `muse/pyproject.toml` AND `musehub/pyproject.toml`, commit each on `dev`
2. `muse -C ~/ecosystem/muse push local dev && muse -C ~/ecosystem/muse push staging dev`
3. `bash deploy/push.sh staging` — server must be at the new version before the tarball is published
4. `bash deploy/publish_muse_release.sh`
5. Verify: `curl -fsSL https://staging.musehub.ai/install.sh | sh && muse --version`

### Recovering a Down Staging Instance

If staging returns 502 or the webserver is down:

1. **Restart the container** — containers use `--restart unless-stopped` so a reboot brings them back, but a manual `docker stop` does not:
   ```bash
   sudo docker start musehub-blue   # or musehub-green, whichever is active
   ```

2. **Switch the active slot** — the ONLY correct way to change which slot nginx points to:
   ```bash
   sudo musehub-set-slot blue    # blue  = port 1337
   sudo musehub-set-slot green   # green = port 1338
   ```
   This script lives at `/usr/local/bin/musehub-set-slot` on the instance. It is the only sanctioned way to write `/etc/nginx/musehub-active-port`. **Never write to that file directly** — it must contain a full nginx upstream directive (`server 127.0.0.1:1337;`), not a bare port number. Writing a bare port breaks nginx config parsing and causes 502s.

3. **Check current state**:
   ```bash
   cat /opt/musehub/.active-slot           # which slot is active (blue or green)
   cat /etc/nginx/musehub-active-port      # what nginx is pointing at
   sudo docker ps --format "{{.Names}} {{.Status}}"  # container health
   ```

4. **If SSM commands are stuck as Pending** — reboot the instance. Containers with `--restart unless-stopped` come back automatically:
   ```bash
   aws ec2 reboot-instances --region us-east-1 --instance-ids i-07547cd20bee2dea5
   ```
   Then poll until SSM is back before sending new commands:
   ```bash
   aws ssm describe-instance-information --region us-east-1 \
     --filters "Key=InstanceIds,Values=i-07547cd20bee2dea5" \
     --query 'InstanceInformationList[0].PingStatus' --output text
   ```

### Instance IDs

| Environment | Instance ID |
|-------------|-------------|
| staging | `i-07547cd20bee2dea5` |
| prod | `i-0855d6efe7fa1a49d` |