gabriel / musehub public
filed by gabriel human · 37 days ago

Observability follow-up: CloudWatch Agent, external uptime, error tracking, dashboards

0 Anchors
— Blast radius
— Churn 30d
0 Proposals

Context

Split out from #160, which is now closed. #160 fixed real bugs in the alerting script and got 5xx-rate/p99-latency alarms genuinely live on both environments, with alerts routed to both Gabriel and Aaron, plus explicit log retention and environment/release_version log fields.

Remaining items — each needs a real decision, not a quick toggle

  • Install the CloudWatch Agent on both instances — needed for real infrastructure metrics (CPU/disk/memory at the instance level). The install command is documented in deploy/cloudwatch-alerts.sh's comments (corrected for Ubuntu/apt, was wrong before), but actually installing, configuring, and verifying it wasn't done in this pass.
  • A real self-hosted-Postgres connection alarm — the previous alarm pointed at AWS/RDS DatabaseConnections, which could never fire (no RDS in this architecture). A real replacement needs either the CloudWatch Agent's procstat plugin or a custom metric pushed from inside the app/DB container.
  • External uptime monitoring — needs a tool decision (CloudWatch Synthetics canary, a third-party service, or a simple external cron ping) and its own cost/complexity tradeoff.
  • Error tracking tool (Sentry or similar) — needs a tool + cost decision, not something to pick unilaterally.
  • Dashboards — the metric filters now exist to build one from (request rate, error rate, latency), just not built yet.
  • Alert-delivery testing — would need to actually trigger a 5xx spike or induced latency to confirm the SNS→email path works end-to-end, not just that the alarm exists.
  • Warning vs. page-worthy severity definitions, and runbook links on alerts — neither exists yet.
  • SLIs/SLOs — not defined.
  • KMS encryption for logs — currently AWS-managed default key; not revisited, low priority.
  • Verify logs/metrics remain available during an application outage — not tested.
◎ Activity
●
gabriel opened this issue 37 days ago
◎
No activity yet. Use the CLI to comment.