Dev API 503 — consolidated EC2 wedge & self-heal¶

Incident: 2026-08-14, ~10:33–11:57 UTC (~84 min). Dev only. Impact: the entire dev API returned 503 Service Unavailable; tradai-dashboard unusable. Trigger: the single consolidated EC2 instance wedged at the OS level. Not rate limiting, a bad deploy, or a downstream (MLflow/RDS) outage.

What happened¶

Dev runs backend-api, data-collection, mlflow, and strategy-service as four Docker containers on one 2 GB t3.small behind ASG tradai-consolidated-asg-dev (min=max=desired=1). After ~14 h uptime the instance froze:

• backend stopped answering — even the unauthenticated /api/v1/health hung with no response
• container logs went silent (nothing for the final period)
• SSM PingStatus=ConnectionLost — the OS itself was unreachable, not just the app
• CPU held ~30% (that metric comes from the hypervisor, so it reports even when the OS is wedged)

With no healthy backend target, the ALB served 503 for every route.

Why it didn't self-heal (the real bug)¶

The ASG used health_check_type="EC2". That only reacts to EC2 status checks (system + instance reachability) — which stayed green the whole time (StatusCheckFailed = 0). So although the ALB target was unhealthy / Target.Timeout for 12+ minutes, the ASG never knew the app was dead and never replaced the box. A dead backend could not recover on its own.

How it was diagnosed (repeatable)¶

Distinguishing this from rate-limiting / a downstream outage:

* the Cognito authorizer rejects at the gateway before reaching the dead backend — a useful "is the edge up but the backend down?" tell.

Immediate mitigation (what to do if it recurs before the fix ships)¶

The box is unreachable, so SSM / docker restart won't work — replace the instance:

AWS_PROFILE=tradai aws autoscaling terminate-instance-in-auto-scaling-group \
  --region eu-central-1 \
  --instance-id <wedged-id> \
  --no-should-decrement-desired-capacity

--no-should-decrement-desired-capacity keeps desired=1, so the ASG launches a fresh instance immediately. Healthy in ~3 min. Verify:

curl -sk https://tradai-dev-1942285475.eu-central-1.elb.amazonaws.com/api/v1/health | jq .

The durable fix (this PR)¶

1. ASG self-heals on app health. health_check_type EC2 → ELB + grace period 300 → 600s to cover a full cold start (~190 s observed) without a boot-kill loop. An ALB-unhealthy backend is now replaced automatically.
2. Only the backend-api target group gates ASG health. The ASG fails an instance if any attached target group is unhealthy, so the ASG attaches the backend TG alone. Attaching mlflow's TG too would let a crash-looping mlflow (recurring RDS-auth failures — see the catalog/MLflow RDS outage) fail ELB health and drive an endless terminate/relaunch of the whole box, taking the healthy backend down with it. mlflow stays reachable in-cluster via Cloud Map (http://mlflow:5000); its ALB /mlflow route is a non-critical dev convenience and is intentionally not wired to ASG health.
3. Instance-refresh deploy race is workflow-bounded, not Pulumi-bounded. ELB health means a refresh waits for the new instance to pass the backend TG within the 600 s grace. The routine image-pull refresh is driven by docker-build.yml, which cancels any in-flight refresh before starting a new one (#632/#663) — that cancel-before-start is what prevents a hung refresh from blocking the next deploy (InstanceRefreshInProgress, PR #690). Gating on the backend TG only also means a slow non-critical service can no longer stall the refresh health-wait. If a refresh ever wedges, cancel it: aws autoscaling cancel-instance-refresh --auto-scaling-group-name tradai-consolidated-asg-dev --region eu-central-1.
4. Memory is now observable. The CloudWatch agent publishes mem_used_percent / swap_used_percent to the CWAgent namespace, with an AutoScalingGroupName rollup (stable across replacement). Two alarms page the alerts SNS topic, and which one fires tells you what to do:

High-memory treats missing data as missing, not breaching: docker-build.yml refreshes this single-instance ASG on every dev push, so a 5-10 min metric gap is routine and paging on it just gets the channel muted. Silence is the not-reporting alarm's job, on a window long enough to clear a refresh.

1. One agent config, and a deploy gate that proves it works. The agent merges the canonical etc/amazon-cloudwatch-agent.json with everything in the .d directory and rejects the whole merge when two configs disagree on a value. That is #1196: #1129 appended a second config whose metrics.namespace and metrics_collected.mem conflicted with the AMI-baked one, the append was discarded, and the alarm sat in ALARM on zero datapoints for days while every deploy stayed green.

So the slim userdata clears .d, overwrites the canonical config, and fetch-config -s that single file. Two things enforce it:

• Grep the boot log for the marker. Widen the window past the instance's launch: this is written once, at boot, and you reach this step after ≥30 min of silence on a box that may have been up for days.

  aws logs tail /tradai/consolidated/user-data --region eu-central-1 --since 7d \
    | grep -E 'CWAGENT_CONFIG_FAILED|Different values are specified'
  

  Different values are specified is the merge conflict itself, in the agent's own words. An empty result is only meaningful if that log group has recent lines from this instance at all — the marker is shipped by the very agent whose health it reports on. If the group is silent, read it on the box instead:

  aws ssm start-session --region eu-central-1 --target <instance-id>
  sudo grep -E 'CWAGENT_CONFIG_FAILED|Different values are specified' /var/log/user-data.log
  

• In CI, infra/asg-verify.sh fails the deploy red if the exact series the alarms watch has no fresh datapoint within 10 min of the ASG converging. It uses get-metric-statistics, not list-metrics, because list-metrics --dimensions is a subset filter and would pass on the per-instance series the alarms cannot match. Mind the scope: that gate runs from deploy-infra.yml, which is path-filtered to infra/**. The image-pull ASG refreshes docker-build.yml fires on every dev push are not gated — the not-reporting alarm is the backstop there. Running just asg-verify by hand skips the metric gate by default (it is the incident path); opt in with SKIP_METRIC_VERIFY=0 just asg-verify.

Still open — root cause of the wedge¶

Why the instance exhausted memory after ~14 h is not yet proven (no memory metric existed at the time; the console log was gone post-termination). The strong hypothesis is a slow leak in one of the four services. The box has since been resized 2 GB t3.small → 8 GB t3.large and has not wedged again, but that neither proves nor disproves the leak — it only buys time. The memory metrics and alarms exist to catch the next occurrence with evidence (and since #1196 they are actually wired to data). If it recurs, identify the leaking service (per-container docker stats shipped to CloudWatch) before resizing again — the box is already at 8 GB, so another bump is not the cheap answer it was at 2 GB.

Tracking: see the "consolidated EC2 memory leak" follow-up issue.