Dev Environment Health Audit¶

Date: 2026-04-27 Account: 600802701449 | Region: eu-central-1 | Profile: tradai

Summary¶

1. PERSISTENT Stack — ALL OK¶

S3 Buckets (5/5)¶

DynamoDB Tables (12/12) — All exist¶

ECR Repositories (26 total, 24 expected + 2 strategy)¶

All service repos have images tagged v0.1.1 + latest (pushed 2026-04-25). All lambda repos have images tagged v0.1.1.

Cognito — OK¶

• User Pool: tradai-users-dev (eu-central-1_dHPbxry36)

2. FOUNDATION Stack — ALL OK¶

VPC & Networking¶

RDS — OK¶

• ID: tradai-mlflow-dev
• Status: available
• Engine: PostgreSQL (db.t4g.micro)
• Endpoint: tradai-mlflow-dev.crcc44o2kjg3.eu-central-1.rds.amazonaws.com

SQS — OK¶

• tradai-backtest-queue-dev.fifo — exists, 0 messages
• tradai-backtest-dlq-dev.fifo — exists

SNS — OK¶

• tradai-alerts-dev
• tradai-registration-dev

3. COMPUTE Stack¶

EC2 Consolidated — OK¶

Docker containers on consolidated EC2:

No container errors in CloudWatch logs for the last hour.

ALB — OK¶

Target Group Health:

Health endpoint responses:

GET /api/v1/health → 200 OK (0.13s)
  Backend: healthy
  DynamoDB: healthy (11ms)
  SQS: healthy (8ms, 0 msgs)
  data-collection: healthy (14ms)
  strategy-service: healthy (11ms)

GET /mlflow/health → 200 OK (0.08s)

ECS Services¶

Note: Main services (backend, data-collection, strategy-service, mlflow) run on consolidated EC2, NOT as ECS services. Only live/dry-run-trading have ECS task definitions.

Lambda Functions (18 total) — ALL EXIST¶

Scheduled Lambdas execution status:

Step Functions Lambdas:

Step Functions — OK¶

Service Discovery — DEGRADED¶

Namespace: tradai-dev.local (ns-upztkwfphvzqzn2j)

Stale entries detected. Only i-0d81ba92e56c19946 (10.0.11.129) is the current consolidated instance. Old entries from terminated instances remain:

Impact: Lambda health-check resolves DNS and sometimes hits stale IPs, causing 10-30s timeouts per service. This explains the 20-60s total duration and frequent 60s timeouts.

Stale instance IDs: - i-0036ccac4827f82ca — terminated - i-09ff8e414fe16f7e7 — does not exist - i-0abb996a2c9885150 — does not exist - i-0aefcaeee239c441d — terminated

4. EDGE Stack¶

API Gateway — OK¶

GET /api/v1/health via API Gateway → 200 OK

WAF — EXISTS, NOT ASSOCIATED¶

• WebACL: tradai-waf-dev (0bffab6d-0f5c-47b5-b9a3-6f96061034b2)
• Log group: aws-waf-logs-tradai-dev (0 bytes — never received traffic)
• Known bug: WAFv2 cannot parse $default in HTTP API stage ARN

CloudWatch Alarms (28 total)¶

ISSUES TO FIX (Priority Order)¶

P0 — Critical¶

1. Stale Service Discovery entries causing health-check timeouts¶

• Problem: 9 stale SD entries from terminated EC2 instances
• Impact: health-check Lambda times out ~50% of invocations (60s timeout), services sometimes unreachable via SD DNS
• Fix: Deregister stale instances from Service Discovery
• Stale instances to remove:
• i-0036ccac4827f82ca from backend-api, mlflow
• i-09ff8e414fe16f7e7 from backend-api
• i-0abb996a2c9885150 from strategy-service, mlflow
• i-0aefcaeee239c441d from all 4 services
• Root cause: ASG lifecycle hook or cloud-init cleanup not deregistering on termination

P1 — Important¶

2. backtest-consumer Lambda never invoked¶

• Problem: SQS event source mapping is Enabled, but Lambda has 0 log bytes ever
• Impact: POST /api/v1/backtests via API Gateway → SQS → backtest-consumer → Step Functions pipeline untested
• Diagnosis needed: Either API Gateway SQS integration never receives requests, or the integration is misconfigured
• Action: Submit a test backtest through API Gateway POST /api/v1/backtests to verify the full pipeline

3. WAF not associated with API Gateway¶

• Problem: WebACL exists but not attached to the HTTP API $default stage
• Impact: No WAF protection on API Gateway
• Known issue: WAFv2 cannot parse $default stage ARN

P2 — Monitor¶

4. drift-monitor and pulumi-drift-detector running as no-ops¶

• Problem: Both execute in 3-5ms (just init+return), not doing real work
• Impact: No actual drift detection happening
• Likely cause: Missing configuration or empty state — need to check if MLflow has models to monitor

5. promote-model, model-rollback, sqs-consumer never invoked¶

• Problem: These Lambdas have 0 bytes in logs (never executed)
• Impact: Model promotion and rollback pipeline untested
• Note: Acceptable if no models have been promoted yet in dev

6. DynamoDB health_state missing timestamps¶

• Problem: last_checked field is None for all 4 services
• Impact: Health history not persisted correctly
• Likely cause: health-check Lambda may be using different attribute names, or timeouts prevent writing

P3 — Cleanup¶

7. Container Insights log group empty¶

• /aws/ecs/containerinsights/tradai-dev/performance — 0 bytes (1 day retention)
• Consolidated EC2 mode may not emit Container Insights metrics

8. ECS log group has minimal data¶

• /aws/ecs/tradai-dev — 444 KB, /ecs/tradai/dev — 558 KB
• Expected: most logs go to /tradai/consolidated/containers (~97 MB)

RESOURCE INVENTORY CHECKLIST¶

Persistent Stack¶

• S3: tradai-configs-dev
• S3: tradai-results-dev
• S3: tradai-arcticdb-dev
• S3: tradai-mlflow-dev
• S3: tradai-logs-dev
• DynamoDB: 12/12 tables
• ECR: 24+ repositories with images
• Cognito: tradai-users-dev
• CodeArtifact: (not checked individually)
• CloudTrail: logs flowing (543 MB)

Foundation Stack¶

• VPC: 10.0.0.0/16
• Subnets: 6/6 (2 public, 2 private, 2 database)
• NAT Instance: running (t4g.nano)
• RDS: available (PostgreSQL 15)
• SQS: 2 queues (backtest + DLQ)
• SNS: 2 topics (alerts + registration)
• Security Groups: (checked via VPC)
• VPC Endpoints: (not checked individually)

Compute Stack¶

• ECS Cluster: tradai-dev
• EC2 Consolidated: running, 4 containers healthy
• ALB: active, targets healthy
• Lambda: 18/18 functions exist
• EventBridge: 7 rules, all ENABLED
• Step Functions: 2 workflows
• IAM Roles: (implicit — services functioning)
• Service Discovery: namespace exists, BUT stale entries

Edge Stack¶

• API Gateway: 28 routes, health OK
• WAF: exists (NOT associated)
• CloudWatch Alarms: 28 alarms
• CloudWatch Dashboard: (not checked)