musehub — Agent Configuration

This repository is a member of a workspace. Shared workspace rules live in the parent .muse/agent.md. This file contains only musehub-specific additions.

Managed by muse agent-config — regenerate adapters with muse agent-config sync.

Proposal Titles

Proposal titles must be plain English — not branch-name style.

The branch (feat/auth-v2, task/proposal-models-v2) already encodes the type prefix. The title is what a human reads in the list; it should describe the change, not echo the branch.

Rule: if the title could be mistaken for a branch name or a commit message subject line, rewrite it.

Localhost Containers — ALWAYS Restart BOTH After Code Changes

Two containers run this repo's Python code, and both run without hot-reload — code changes to the live-mounted volume are not picked up until each is restarted:

musehub-runner (docker-in-docker sandbox) and musehub_postgres/musehub_minio (datastores) do not run this repo's Python and never need restarting for a code change.

A real incident that cost real time: musehub_worker was found running a Docker image dated over a month old while musehub itself had been restarted recently — because only musehub's restart was a documented habit. Every job-side observation made during that window (prebuild behavior, cache invariants, mpack consolidation) was silently wrong, and it took a deep, multi-hour investigation to notice the container itself was stale rather than the code. Restarting only musehub and assuming the worker is also fresh is a trap — always restart both, together, every time:

docker restart musehub musehub_worker && sleep 5 && curl -sk https://localhost:1337/healthz

Verify freshness before trusting any test result

A restart alone does not guarantee correctness — confirm it actually worked:

# 1. Confirm both containers actually restarted (not just "Up", but recently)
docker ps --format "{{.Names}}\t{{.Status}}" | grep -E "^musehub(\s|_worker)"
# "Up X seconds" — if either shows days/weeks, the restart didn't take or targeted the wrong name

# 2. Confirm the API booted clean — look for these exact lines, not just "started"
docker logs musehub --tail 20 | grep -i "schema check passed\|Application startup failed"
# ✅ "schema check passed — ORM and DB are in sync (N tables)"
# ❌ "Application startup failed" — see schema-drift section below, do not proceed until fixed

# 3. Confirm the worker booted clean
docker logs musehub_worker --tail 20 | grep -i "worker started\|Application startup failed"
# ✅ "MuseHub worker started"

If you're mid-investigation and something doesn't match your mental model of current code — check container freshness FIRST, before re-reading source or building new theories. Compare image build time against your last relevant commit:

docker inspect musehub --format '{{.Image}}' | xargs docker image inspect --format '{{.Created}}'
docker inspect musehub_worker --format '{{.Image}}' | xargs docker image inspect --format '{{.Created}}'

If either predates a commit you're relying on, restart doesn't help — see "When you need a rebuild, not just a restart" below.

When you need a rebuild, not just a restart

Plain docker restart is enough for .py edits under the bind-mounted musehub/ directory (see docker inspect musehub --format '{{range .Mounts}}...' for the exact list — musehub/musehub, musehub/alembic, musehub/tests, etc. are all live bind mounts). It is not enough when:

• Dockerfile changed
• requirements*.txt / dependency pins changed
• docker-compose.yml / docker-compose.override.yml changed

In those cases, rebuild the image before restarting:

cd ~/ecosystem/musehub
docker compose build musehub worker
docker compose up -d musehub worker

Schema drift and missing migrations — two distinct failure modes

Both show up as Application startup failed in docker logs musehub, but need different fixes. Read the actual error before acting — do not assume alembic upgrade head fixes every schema error:

Mode 1 — DB behind the migrations that already exist. Error names a table/ column the ORM expects. Fix:

docker exec musehub sh -c "cd /app && alembic upgrade head"

Mode 2 — an ORM model was added with no migration ever generated for it. alembic current already reports (head) — there is nothing to "catch up" on; the migration simply doesn't exist. Confirm which mode you're in:

docker exec musehub sh -c "cd /app && alembic current"
docker exec musehub sh -c "cd /app && alembic heads"
# If current == heads and the app still fails on a missing table → Mode 2

Fix by generating the missing migration, then rename it to match this repo's numbered convention (00NN_description.py, not the default timestamp/hex name) before applying:

docker exec musehub sh -c "cd /app && alembic revision --autogenerate -m 'add <table>'"
# rename the generated file: alembic/versions/<timestamp>_<hex>_....py → 00NN_description.py
# edit revision/down_revision inside to '00NN'/'00NN-1' matching the new filename
docker exec musehub sh -c "cd /app && alembic upgrade head"
docker restart musehub musehub_worker

If the container already auto-applied the pre-rename autogenerated revision at startup (common — many app entrypoints run migrations before the schema check), re-stamp the DB to the renamed revision id instead of re-running upgrade:

docker exec musehub_postgres psql -U musehub -d musehub -c "UPDATE alembic_version SET version_num = '00NN';"

Repo-Specific Notes

Deploying MuseHub and publishing a new Muse CLI build

Full runbook: docs/deploy.md. Quick reference:

# Deploy MuseHub server (staging)
cd ~/ecosystem/musehub
bash deploy/push.sh staging

# Publish a new Muse CLI tarball (always AFTER deploying the server)
cd ~/ecosystem/musehub
bash deploy/publish_muse_release.sh

Standard release flow (both repos share the same version string):

1. Bump version in muse/pyproject.toml AND musehub/pyproject.toml, commit each on dev
2. muse -C ~/ecosystem/muse push local dev && muse -C ~/ecosystem/muse push staging dev
3. bash deploy/push.sh staging — server must be at the new version before the tarball is published
4. bash deploy/publish_muse_release.sh
5. Verify: curl -fsSL https://staging.musehub.ai/install.sh | sh && muse --version

Recovering a Down Staging Instance

If staging returns 502 or the webserver is down:

1. Restart the container — containers use --restart unless-stopped so a reboot brings them back, but a manual docker stop does not:

   sudo docker start musehub-blue   # or musehub-green, whichever is active
   

2. Switch the active slot — the ONLY correct way to change which slot nginx points to:

   sudo musehub-set-slot blue    # blue  = port 1337
   sudo musehub-set-slot green   # green = port 1338
   

   This script lives at /usr/local/bin/musehub-set-slot on the instance. It is the only sanctioned way to write /etc/nginx/musehub-active-port. Never write to that file directly — it must contain a full nginx upstream directive (server 127.0.0.1:1337;), not a bare port number. Writing a bare port breaks nginx config parsing and causes 502s.

3. Check current state:

   cat /opt/musehub/.active-slot           # which slot is active (blue or green)
   cat /etc/nginx/musehub-active-port      # what nginx is pointing at
   sudo docker ps --format "{{.Names}} {{.Status}}"  # container health
   

4. If SSM commands are stuck as Pending — reboot the instance. Containers with --restart unless-stopped come back automatically:

   aws ec2 reboot-instances --region us-east-1 --instance-ids i-07547cd20bee2dea5
   

   Then poll until SSM is back before sending new commands:

   aws ssm describe-instance-information --region us-east-1 \
     --filters "Key=InstanceIds,Values=i-07547cd20bee2dea5" \
     --query 'InstanceInformationList[0].PingStatus' --output text
   

Instance IDs