# Section 12 — Observability

> Companion to [`musehub-production-readiness-checklist.md`](../musehub-production-readiness-checklist.md).
> ⚠️ **Correction to earlier sections in this sweep**: Section 0 originally said "no logging and
> monitoring exists." That was wrong — I'd only checked `describe-alarms`/`list-topics`, not log
> groups themselves. Verified properly now: **logging is further along than initially reported.**

## What's actually live — verified directly against CloudWatch

- **`/musehub/staging` log group exists and is actively receiving structured JSON logs right now**
  (~1.7 GB stored, confirmed via `aws logs tail` showing real-time traffic during this session).
- **Structured logging is genuinely well-built** (`musehub/logging_config.py`): every line is a
  JSON object with `timestamp` (ISO-8601 UTC), `level`, `logger`, `message`, `request_id`,
  `user_id`; per-request access logs add `method`/`path`/`status`/`duration_ms`.
- **PII/secret scrubbing is already implemented** — a `PiiFilter` regex-scrubs `Bearer` tokens,
  `token=`, `password=`, `secret=` patterns from formatted messages before they're logged. This
  directly satisfies "prevent credentials/tokens from entering logs," which is usually a gap.
- **Memory-pressure warnings already fire** — saw live `WARNING` entries like `[memory] HIGH RSS
  408.3 MiB after GET ...` from a `musehub.debug.memory` logger. There's already a signal here;
  it's just not wired to an alert.

## What's missing

- **No metric filters, no alarms, no SNS topics, no dashboards.** `aws logs
  describe-metric-filters` on the real log group returns empty; `describe-alarms`/`list-topics`
  are empty account-wide. The data to build all of this already exists in the log stream — this
  is genuinely "just" wiring, not "build from nothing."
- **`cloudwatch-alerts.sh` targets the wrong log group** (`/musehub/app`, hardcoded) — it would
  create a second, empty log group rather than attaching to the real one (`/musehub/staging`) if
  run as-is today. Needs a one-line fix before it's usable.
- **No log retention limit set** — logs accumulate forever. Not a missing-observability problem,
  but a cost and compliance one (Section 15).
- **No external uptime checks.**
- **No error tracking tool** (Sentry or similar) — confirmed absent in Section 0.
- **No release/deployment markers on any dashboard** — moot until a dashboard exists.
- **Environment/service/release-version fields aren't in the JSON log schema** — `logging_config.py`
  has timestamp/level/logger/request_id/user_id/method/path/status/duration_ms, but not
  `environment` or `release_version` explicitly. Worth adding since staging and prod currently
  share no field that would let a single dashboard distinguish them other than the log group name.

## Checklist assessment

- [x] Centralize application logs. — done, CloudWatch Logs `/musehub/staging`
- [x] Use structured logs. — done, JSON with rich per-request fields
- [ ] Include: timestamp/environment/service/request-id/release-version/severity — mostly done;
  missing `environment` and `release_version` fields specifically
- [x] Prevent credentials/tokens/PII from entering logs. — done, `PiiFilter`
- [ ] Set explicit log-retention periods. — **not done**, currently unlimited retention
- [ ] Encrypt logs. — CloudWatch Logs encrypts at rest by default (AWS-managed key) unless
  configured otherwise; not verified whether a customer-managed KMS key is used
- [x] Collect infrastructure and application metrics. — partial: application-level (via structured
  logs, which *could* feed metric filters) exists; true infrastructure metrics (CPU/disk/memory at
  the instance level via CloudWatch Agent) not confirmed installed
- [ ] Build dashboards (request rate, error rate, latency, saturation, instance health, DB
  capacity/connections, queue depth, background-job failures) — **none exist**, but the raw data
  for most of these (status, duration_ms) is already flowing into the log group
- [ ] Add external uptime checks. — not done
- [ ] Configure alerts (site down, 5xx rate, latency, unhealthy deploy, DB storage, CPU/memory,
  backup failure, cert problems, queue backlog, security findings, cost growth) — none exist
- [ ] Route alerts to both Gabriel and Aaron. — N/A, no alerts exist
- [ ] Define warning vs. page-worthy severity. — not done
- [ ] Test alert delivery. — N/A
- [ ] Every urgent alert links to a runbook. — N/A
- [ ] Configure error tracking with release correlation. — not done, no tool exists
- [ ] Define SLIs/initial SLOs. — not done; blocked on Section 0's undefined availability targets
- [ ] Add deployment markers to dashboards. — N/A, no dashboards exist
- [ ] Verify logs/metrics remain available during an application outage. — not tested; CloudWatch
  Logs being a separate AWS service from the EC2 instance itself is a good sign this would likely
  hold, but untested

## The actual work items here

This section is closer to done than any other section so far — the foundation (structured,
scrubbed, centralized logging) is solid. What's left is genuinely "wire alarms on top of data
that already exists":
1. Fix `cloudwatch-alerts.sh`'s log group name, then run it for real.
2. Set an explicit retention period on `/musehub/staging` (and whatever prod's log group ends up
   being once prod deploy is fixed).
3. Add `environment`/`release_version` fields to `logging_config.py`.
4. Build the actual alarms + SNS subscriptions + a basic dashboard.
5. Add external uptime monitoring (e.g. a simple periodic health-check ping from outside AWS).