Observability follow-up: CloudWatch Agent, external uptime, error tracking, dashboards
0
Anchors
—
Blast radius
—
Churn 30d
0
Proposals
Context
Split out from #160, which is now closed. #160 fixed real bugs in the alerting script and got 5xx-rate/p99-latency alarms genuinely live on both environments, with alerts routed to both Gabriel and Aaron, plus explicit log retention and environment/release_version log fields.
Remaining items — each needs a real decision, not a quick toggle
- Install the CloudWatch Agent on both instances — needed for real infrastructure metrics
(CPU/disk/memory at the instance level). The install command is documented in
deploy/cloudwatch-alerts.sh's comments (corrected for Ubuntu/apt, was wrong before), but actually installing, configuring, and verifying it wasn't done in this pass. - A real self-hosted-Postgres connection alarm — the previous alarm pointed at
AWS/RDS DatabaseConnections, which could never fire (no RDS in this architecture). A real replacement needs either the CloudWatch Agent's procstat plugin or a custom metric pushed from inside the app/DB container. - External uptime monitoring — needs a tool decision (CloudWatch Synthetics canary, a third-party service, or a simple external cron ping) and its own cost/complexity tradeoff.
- Error tracking tool (Sentry or similar) — needs a tool + cost decision, not something to pick unilaterally.
- Dashboards — the metric filters now exist to build one from (request rate, error rate, latency), just not built yet.
- Alert-delivery testing — would need to actually trigger a 5xx spike or induced latency to confirm the SNS→email path works end-to-end, not just that the alarm exists.
- Warning vs. page-worthy severity definitions, and runbook links on alerts — neither exists yet.
- SLIs/SLOs — not defined.
- KMS encryption for logs — currently AWS-managed default key; not revisited, low priority.
- Verify logs/metrics remain available during an application outage — not tested.
Activity
No activity yet. Use the CLI to comment.