Section 12 — Observability

Companion to musehub-production-readiness-checklist.md. ⚠️ Correction to earlier sections in this sweep: Section 0 originally said "no logging and monitoring exists." That was wrong — I'd only checked describe-alarms/list-topics, not log groups themselves. Verified properly now: logging is further along than initially reported.

What's actually live — verified directly against CloudWatch

• /musehub/staging log group exists and is actively receiving structured JSON logs right now (~1.7 GB stored, confirmed via aws logs tail showing real-time traffic during this session).
• Structured logging is genuinely well-built (musehub/logging_config.py): every line is a JSON object with timestamp (ISO-8601 UTC), level, logger, message, request_id, user_id; per-request access logs add method/path/status/duration_ms.
• PII/secret scrubbing is already implemented — a PiiFilter regex-scrubs Bearer tokens, token=, password=, secret= patterns from formatted messages before they're logged. This directly satisfies "prevent credentials/tokens from entering logs," which is usually a gap.
• Memory-pressure warnings already fire — saw live WARNING entries like [memory] HIGH RSS 408.3 MiB after GET ... from a musehub.debug.memory logger. There's already a signal here; it's just not wired to an alert.

What's missing

• No metric filters, no alarms, no SNS topics, no dashboards. aws logs describe-metric-filters on the real log group returns empty; describe-alarms/list-topics are empty account-wide. The data to build all of this already exists in the log stream — this is genuinely "just" wiring, not "build from nothing."
• cloudwatch-alerts.sh targets the wrong log group (/musehub/app, hardcoded) — it would create a second, empty log group rather than attaching to the real one (/musehub/staging) if run as-is today. Needs a one-line fix before it's usable.
• No log retention limit set — logs accumulate forever. Not a missing-observability problem, but a cost and compliance one (Section 15).
• No external uptime checks.
• No error tracking tool (Sentry or similar) — confirmed absent in Section 0.
• No release/deployment markers on any dashboard — moot until a dashboard exists.
• Environment/service/release-version fields aren't in the JSON log schema — logging_config.py has timestamp/level/logger/request_id/user_id/method/path/status/duration_ms, but not environment or release_version explicitly. Worth adding since staging and prod currently share no field that would let a single dashboard distinguish them other than the log group name.

Checklist assessment

• [x] Centralize application logs. — done, CloudWatch Logs /musehub/staging
• [x] Use structured logs. — done, JSON with rich per-request fields
• [ ] Include: timestamp/environment/service/request-id/release-version/severity — mostly done; missing environment and release_version fields specifically
• [x] Prevent credentials/tokens/PII from entering logs. — done, PiiFilter
• [ ] Set explicit log-retention periods. — not done, currently unlimited retention
• [ ] Encrypt logs. — CloudWatch Logs encrypts at rest by default (AWS-managed key) unless configured otherwise; not verified whether a customer-managed KMS key is used
• [x] Collect infrastructure and application metrics. — partial: application-level (via structured logs, which could feed metric filters) exists; true infrastructure metrics (CPU/disk/memory at the instance level via CloudWatch Agent) not confirmed installed
• [ ] Build dashboards (request rate, error rate, latency, saturation, instance health, DB capacity/connections, queue depth, background-job failures) — none exist, but the raw data for most of these (status, duration_ms) is already flowing into the log group
• [ ] Add external uptime checks. — not done
• [ ] Configure alerts (site down, 5xx rate, latency, unhealthy deploy, DB storage, CPU/memory, backup failure, cert problems, queue backlog, security findings, cost growth) — none exist
• [ ] Route alerts to both Gabriel and Aaron. — N/A, no alerts exist
• [ ] Define warning vs. page-worthy severity. — not done
• [ ] Test alert delivery. — N/A
• [ ] Every urgent alert links to a runbook. — N/A
• [ ] Configure error tracking with release correlation. — not done, no tool exists
• [ ] Define SLIs/initial SLOs. — not done; blocked on Section 0's undefined availability targets
• [ ] Add deployment markers to dashboards. — N/A, no dashboards exist
• [ ] Verify logs/metrics remain available during an application outage. — not tested; CloudWatch Logs being a separate AWS service from the EC2 instance itself is a good sign this would likely hold, but untested

The actual work items here

This section is closer to done than any other section so far — the foundation (structured, scrubbed, centralized logging) is solid. What's left is genuinely "wire alarms on top of data that already exists":

1. Fix cloudwatch-alerts.sh's log group name, then run it for real.
2. Set an explicit retention period on /musehub/staging (and whatever prod's log group ends up being once prod deploy is fixed).
3. Add environment/release_version fields to logging_config.py.
4. Build the actual alarms + SNS subscriptions + a basic dashboard.
5. Add external uptime monitoring (e.g. a simple periodic health-check ping from outside AWS).