Closed #160 Enhancement production-readiness

filed by gabriel human · 68 days ago

Production readiness — Section 12: Observability

0 Anchors

— Blast radius

— Churn 30d

0 Proposals

Better than initially thought: structured JSON logs (with PII/secret scrubbing already implemented) are live in CloudWatch (/musehub/staging, ~1.7GB and growing). What is missing is entirely on top of that existing data: metric filters, alarms, SNS topics, dashboards, external uptime checks, and error tracking.

Quick wins: fix deploy/cloudwatch-alerts.sh (targets the wrong log group name, /musehub/app instead of the real /musehub/staging), set an explicit log retention period (currently unlimited).

◎ Activity2

●

gabriel opened this issue 68 days ago

○

gabriel 51 days ago

Update 2026-08-25: Fixed the exact issue this ticket flagged — /musehub/staging had unlimited retention and was already at ~1.9GB. Set explicit 30-day retention (matching the policy already applied to /musehub/production from day one). AWS prunes existing data older than the new retention window asynchronously (not instant, typically within 24h), so this both stops future unbounded growth and will shrink the current stored bytes down without further action.

Alarms/dashboards/metric-filters/SNS topics on top of this data — and fixing deploy/cloudwatch-alerts.sh's wrong log-group-name reference — remain open.

○

gabriel 37 days ago

Wired real alerting on top of the already-solid logging foundation.

Fixed real bugs in deploy/cloudwatch-alerts.sh, beyond the known log-group-name issue:

Wrong default region (eu-west-1 — everything here runs in us-east-1)
A metric-alarm call that would fail at runtime (--statistic + --extended-statistic are mutually exclusive) — found by actually running the script
A DB-connections alarm pointed at AWS/RDS that could never fire — this project uses self-hosted Postgres on EC2, not RDS. Removed rather than left silently broken.
CloudWatch Agent install instructions used yum (Amazon Linux) — these instances run Ubuntu. Corrected.
Single-email alerting → now routes to both Gabriel and Aaron by default

Ran it for real against both environments, verified via aws logs/aws cloudwatch describe-* directly: 30-day log retention set, SNS topics with both maintainers subscribed, 3 metric filters and 2 alarms (5xx rate, p99 latency) live on each. You'll each need to confirm the SNS email subscription AWS sent.

Added environment/release_version to every log line — verified end-to-end locally, takes effect on the next deploy to each environment.

Remaining work (CloudWatch Agent install, external uptime, error tracking, dashboards) needs real tool/cost decisions — split to #181.

Closing.

✓

closed this issue 37 days ago

Assignee

gabriel human

Release

no commits linked to this issue

create

muse hub issue create \
  --title "..." \
  --body "..." \
  --label bug \
  --anchor path/to/file.py::Symbol \
  --commit-anchor <sha> \
  --repo gabriel/musehub

read

muse hub issue get 160 --json
muse hub issue list --state open --json

update

muse hub issue edit 160 \
  --anchor path/to/file.py::Symbol \
  --repo gabriel/musehub

comment

muse hub issue comment 160 \
  --body "Fixed in <sha>" \
  --repo gabriel/musehub

reopen

muse hub issue reopen 160 \
  --repo gabriel/musehub

create

create_issue({
  repo_id: "sha256:a265796360c3b1b8700b5682ced5f6b044a2c0d3a2c58918892a5aa494db6c75",
  title: "...",
  body: "...",
  labels: ["bug"],
  symbol_anchors: [
    "path/to/file.py::Symbol"
  ],
  commit_anchors: ["<sha>"]
})

read

get_issue({
  repo_id: "sha256:a265796360c3b1b8700b5682ced5f6b044a2c0d3a2c58918892a5aa494db6c75",
  issue_number: 160
})

list_issues({
  repo_id: "sha256:a265796360c3b1b8700b5682ced5f6b044a2c0d3a2c58918892a5aa494db6c75",
  state: "open"
})

update

edit_issue({
  repo_id: "sha256:a265796360c3b1b8700b5682ced5f6b044a2c0d3a2c58918892a5aa494db6c75",
  issue_number: 160,
  symbol_anchors: ["path/to/file.py::Symbol"]
})

comment

create_issue_comment({
  repo_id: "sha256:a265796360c3b1b8700b5682ced5f6b044a2c0d3a2c58918892a5aa494db6c75",
  issue_number: 160,
  body: "..."
})

reopen

reopen_issue({
  repo_id: "sha256:a265796360c3b1b8700b5682ced5f6b044a2c0d3a2c58918892a5aa494db6c75",
  issue_number: 160
})

create

curl -X POST \
  http://localhost:10003/api/repos/sha256:a265796360c3b1b8700b5682ced5f6b044a2c0d3a2c58918892a5aa494db6c75/issues \
  -H "Content-Type: application/json" \
  -H "Authorization: MSign handle=\"...\" ts=... sig=\"...\"" \
  -d '{
    "title": "...",
    "body": "...",
    "labels": ["bug"],
    "symbol_anchors": ["path/to/file.py::Symbol"]
  }'

read

# get one issue
curl http://localhost:10003/api/repos/sha256:a265796360c3b1b8700b5682ced5f6b044a2c0d3a2c58918892a5aa494db6c75/issues/160

# list open issues
curl "http://localhost:10003/api/repos/sha256:a265796360c3b1b8700b5682ced5f6b044a2c0d3a2c58918892a5aa494db6c75/issues?state=open"

update

curl -X PATCH \
  http://localhost:10003/api/repos/sha256:a265796360c3b1b8700b5682ced5f6b044a2c0d3a2c58918892a5aa494db6c75/issues/160 \
  -H "Content-Type: application/json" \
  -H "Authorization: MSign handle=\"...\" ts=... sig=\"...\"" \
  -d '{"title": "...", "body": "..."}'

comment

curl -X POST \
  http://localhost:10003/api/repos/sha256:a265796360c3b1b8700b5682ced5f6b044a2c0d3a2c58918892a5aa494db6c75/issues/160/comments \
  -H "Content-Type: application/json" \
  -H "Authorization: MSign handle=\"...\" ts=... sig=\"...\"" \
  -d '{"body": "Fixed in <sha>"}'

reopen

curl -X POST \
  http://localhost:10003/api/repos/sha256:a265796360c3b1b8700b5682ced5f6b044a2c0d3a2c58918892a5aa494db6c75/issues/160/reopen \
  -H "Authorization: MSign handle=\"...\" ts=... sig=\"...\""