# Section 12 — Observability

> **Status: CLOSED** — production-readiness issue #160, closed 2026-09-08. Alarms, SNS alerting,
> log retention, and environment/release fields all live on both environments. Remaining work
> (CloudWatch Agent install for real infra metrics, external uptime checks, error tracking,
> dashboards) tracked in **#181**.
>
> Companion to [`musehub-production-readiness-checklist.md`](../musehub-production-readiness-checklist.md).

## Update (2026-09-08) — wired real alarms on top of the existing logging foundation

The foundation (structured, PII-scrubbed, centralized logging) was already solid. This pass did
the actual wiring work the original doc identified as the path to closing this section.

### Fixed real bugs in `deploy/cloudwatch-alerts.sh`, beyond just the log group name

- **Wrong default region** (`eu-west-1` — this project runs entirely in `us-east-1`)
- **Hardcoded single log group** (`/musehub/app`) — now takes `--env staging|production` and
  targets the real per-environment log group and its own SNS topic, matching the same
  per-environment pattern already used in `push.sh`/`deploy.sh`
- **A metric-alarm bug that would have failed at runtime**: passed both `--statistic p99` and
  `--extended-statistic p99` — these are mutually exclusive; percentile statistics can only use
  `--extended-statistic`. Found by actually running the script, not just reading it.
- **A completely inapplicable alarm removed**: the original script had a DB-connections alarm
  pointed at `AWS/RDS DatabaseConnections` — this project uses self-hosted Postgres in a Docker
  container on EC2, not RDS. That alarm could never have fired; left in, it would have looked like
  monitoring existed when it didn't. Removed with a note explaining what a real replacement needs
  (CloudWatch Agent's procstat plugin or a custom pushed metric).
- **Wrong CloudWatch Agent install instructions**: said `sudo yum install` (Amazon Linux) — these
  instances run Ubuntu. Corrected to the actual `.deb` install path, though the agent itself isn't
  installed yet (see Remaining work).
- **Single-email alerting → dual**: now subscribes both `gabriel@musehub.ai` and
  `aaronrene@musehub.ai` by default, satisfying "route alerts to both maintainers."

### Ran it for real against both environments — not just fixed and left unverified

```
staging:    30-day log retention set, SNS topic + 2 email subs, 3 metric filters, 2 alarms (OK)
production: 30-day log retention set, SNS topic + 2 email subs, 3 metric filters, 2 alarms (INSUFFICIENT_DATA — expected, low traffic volume)
```

Verified via `aws logs describe-log-groups` and `aws cloudwatch describe-alarms` directly, not
just trusting the script's own success output. **Gabriel and Aaron each need to confirm the SNS
email subscription** (AWS sends a confirmation link) before alerts actually deliver.

### Added `environment` and `release_version` to every log line

`musehub/logging_config.py`'s `JsonFormatter` now stamps `environment` (from the existing
`muse_env` setting) and `release_version` (a new setting, sourced from a `RELEASE_VERSION` env var
that `deploy/deploy.sh` now passes into the container from the same `IMAGE_TAG` it already uses).
Verified working end-to-end locally (`configure_logging` → real log line → both fields present
with correct values). Takes effect for real traffic on the next deploy to each environment.

## Checklist assessment — updated

- [x] Centralize application logs
- [x] Use structured logs
- [x] **Include timestamp/environment/service/request-id/release-version/severity** — now
  complete; `environment` and `release_version` were the only missing fields
- [x] Prevent credentials/tokens/PII from entering logs
- [x] **Set explicit log-retention periods** — 30 days on both environments, done for real
- [ ] Encrypt logs with a customer-managed KMS key — still using the AWS-managed default; not
  revisited, low priority
- [ ] Collect true infrastructure metrics (CPU/disk/memory at the instance level) — still not
  done; needs the CloudWatch Agent actually installed (#181)
- [ ] Build dashboards — still none exist, though metric filters now exist to build one from
- [ ] Add external uptime checks — still not done (#181)
- [x] **Configure alerts for 5xx rate and latency** — done, live on both environments
- [ ] Configure remaining alert types (unhealthy deploy, DB storage, CPU/memory, backup failure,
  cert problems, security findings, cost growth) — most need the CloudWatch Agent or other data
  sources that don't exist yet (#181)
- [x] **Route alerts to both Gabriel and Aaron** — done, both subscribed on both environments
- [ ] Define warning vs. page-worthy severity — not done
- [ ] Test alert delivery — not done (would need to actually trigger a 5xx spike or induce latency)
- [ ] Every urgent alert links to a runbook — not done, no runbooks exist yet
- [ ] Configure error tracking with release correlation — not done, no tool chosen (#181)
- [ ] Define SLIs/initial SLOs — not done
- [ ] Add deployment markers to dashboards — N/A, no dashboards exist
- [ ] Verify logs/metrics remain available during an application outage — not tested

## Remaining work — split to #181

CloudWatch Agent installation (for real disk/memory/infra metrics), external uptime monitoring,
error tracking tool selection, dashboard construction, alert-delivery testing, and severity/runbook
definitions are all real, separate pieces of work needing their own decisions (tool choice, cost,
alarm thresholds) rather than being quick wins on top of existing data — unlike what this pass
actually closed.