Pulumi Infrastructure Deployment Guide¶

Comprehensive guide for deploying and managing TradAI infrastructure using Pulumi with AWS.

Architecture Overview¶

TradAI infrastructure is split into 4 independent Pulumi stacks. The key design principle is separating data-bearing resources (never destroyed in dev cycles) from ephemeral compute (freely destroyed and redeployed):

persistent → foundation → (lambda-bootstrap) → compute → edge

Stack	Resources	Destroy in dev?
persistent	S3×5, DynamoDB×12, ECR×22, Cognito, CodeArtifact, CloudTrail, Pulumi backend	Never
foundation	VPC, subnets, NAT, SGs, NACLs, VPC endpoints, RDS, SQS, SNS	Rarely
compute	IAM roles, ALB, ECS, Lambda, Step Functions, EC2 Consolidated	Freely
edge	API Gateway, WAF, CloudWatch Alarms/Dashboard	Freely

Stacks share data via pulumi.StackReference: - compute reads from persistent (ECR URLs, S3 IDs, DynamoDB names) and foundation (VPC, SGs, RDS, SQS) - edge reads from persistent (Cognito), foundation (VPC, SGs, SQS), and compute (ALB endpoints)

Key Files¶

File	Purpose
`infra/shared/tradai_infra_shared/config.py`	Canonical source of truth for all configuration
`infra/shared/tradai_infra_shared/core/`	Shared builders (PolicyBuilder, SecurityRuleBuilder, etc.)
`infra/persistent/__main__.py`	Persistent stack entry point (data resources)
`infra/foundation/__main__.py`	Foundation stack entry point (networking)
`infra/compute/__main__.py`	Compute stack entry point (ECS, Lambda, IAM)
`infra/edge/__main__.py`	Edge stack entry point (API GW, WAF)
`infra/{stack}/Pulumi.yaml`	Per-stack Pulumi project definition
`infra/{stack}/Pulumi.{env}.yaml`	Per-stack environment configuration
`infra/{stack}/modules/`	Per-stack infrastructure modules

Prerequisites¶

# Required tools
python --version          # 3.11+
uv --version              # UV package manager
pulumi version            # Pulumi CLI
aws --version             # AWS CLI

# AWS credentials
aws configure list        # Verify credentials

Install Pulumi¶

# macOS
brew install pulumi

# Linux
curl -fsSL https://get.pulumi.com | sh

# Verify
pulumi version

Quick Start¶

For Existing Team Members¶

# 1. Setup environment
just infra-setup
# Edit infra/.env with S3_PULUMI_BACKEND_URL, PULUMI_CONFIG_PASSPHRASE, AWS_PROFILE, AWS_REGION

# 2. Preview all stacks
just infra-preview dev

# 3. Deploy all stacks in order
just infra-bootstrap dev

# Or deploy individually (persistent first):
just infra-up-persistent dev    # S3, DynamoDB, ECR, Cognito (no deps)
just infra-up-foundation dev    # VPC, RDS, SQS, SNS
just lambda-bootstrap            # Push Lambda images to ECR
just infra-up-compute dev        # IAM, ECS, ALB, Lambda (images exist now)
just infra-up-edge dev           # API Gateway, WAF, CloudWatch

First-Time Bootstrap¶

# 1. Setup environment
just infra-setup
# Edit infra/.env with AWS credentials and Pulumi passphrase

# 2. Full bootstrap (deploys foundation → pushes images → compute → edge)
just infra-bootstrap dev

# 3. Verify deployment
just infra-outputs persistent dev
just infra-outputs foundation dev
just infra-outputs compute dev
just infra-outputs edge dev

Stack Environments¶

Stack	Description	Key Differences
dev	Development	Single-AZ, t4g.micro RDS, FARGATE_SPOT, 7-day logs
staging	Pre-production	Production-like, single-AZ, db.t4g.micro RDS
prod	Production	Multi-AZ, full redundancy, 90-day logs, no Spot for live trading

Selecting a Stack¶

# List available stacks
pulumi stack ls

# Select a stack
pulumi stack select dev

# Create new stack
pulumi stack init staging

Stack Modules¶

Persistent Stack (`infra/persistent/`)¶

Data-bearing resources. Never destroyed in dev cycles.

Module	Description	Key Resources
`s3.py`	5 S3 buckets (configs, results, arcticdb, logs, mlflow)	Buckets, lifecycle rules
`dynamodb.py`	12 DynamoDB tables	Tables, GSIs
`ecr.py`	22 ECR repositories (4 service + 18 lambda)	Repos, lifecycle policies
`cognito.py`	Cognito user pool, M2M client	User pool, app client
`codeartifact.py`	CodeArtifact for Python packages	Domain, repository
`cloudtrail.py`	CloudTrail audit logging	Trail, S3 policy
`pulumi_backend.py`	Pulumi state S3 bucket	S3 bucket, IAM roles

Foundation Stack (`infra/foundation/`)¶

Networking and supporting infrastructure. No container image dependencies.

Module	Description	Key Resources
`vpc.py`	VPC (10.0.0.0/16), 6 subnets (2 AZs, 3 tiers)	VPC, subnets, route tables
`security_groups.py`	5 security groups	SGs, ingress/egress rules
`nat_instance.py`	NAT instance with ASG (cost-optimized)	ASG, launch template
`nacl.py`	Network ACLs for all subnets	NACLs, rules
`vpc_endpoints.py`	Gateway + Interface VPC Endpoints	S3, DynamoDB, ECR, STS, SSM, SQS endpoints
`vpc_flow_logs.py`	VPC Flow Logs to CloudWatch	Log group, IAM role
`rds.py`	PostgreSQL for MLflow	RDS instance, subnet group
`sqs.py`	Backtest queue + DLQ	Queues, redrive policy
`sns.py`	SNS topics (alerts, registration)	Topics, subscriptions
`secret_rotation.py`	RDS secret rotation (30-day)	Lambda, rotation schedule

Compute Stack (`infra/compute/`)¶

Needs ECR images. Deploy after just lambda-bootstrap.

Module	Description	Key Resources
`iam.py`	ECS execution, task, and consolidated roles	IAM roles, policies, instance profile
`ecs.py`	ECS Fargate cluster	Cluster, capacity providers
`ecs_services.py`	4 ECS services (or consolidated EC2)	Services, task definitions
`alb.py`	Application Load Balancer	ALB, listeners, target groups
`lambda_funcs.py`	Lambda functions	Functions, event sources
`step_functions.py`	Backtest workflow state machine	State machine, IAM role
`ec2_consolidated.py`	Consolidated EC2 for dev/staging	ASG, launch template, Cloud Map
`ec2_userdata.py`	EC2 instance bootstrap script	Userdata templates

Edge Stack (`infra/edge/`)¶

Needs compute resources (ALB, Cognito).

Module	Description	Key Resources
`api_gateway.py`	HTTP API Gateway with routes	API, routes, integrations
`waf.py`	WAF WebACL for API Gateway	WebACL, rules
`cloudwatch_alarms.py`	Composite alarm, heartbeat detection	Alarms, SNS actions
`cloudwatch_dashboard.py`	Trading platform dashboard	Dashboard widgets

Configuration Management¶

config.py - Single Source of Truth¶

All infrastructure values are defined in infra/shared/tradai_infra_shared/config.py:

# Core configuration
PROJECT_NAME = "tradai"
ENVIRONMENT = pulumi.get_stack()  # dev, staging, prod
AWS_REGION = "eu-central-1"

# VPC CIDR
VPC_CIDR = "10.0.0.0/16"

# Services definition
SERVICES = {
    "backend-api": {"port": 8000, "cpu": 512, "memory": 1024},
    "data-collection": {"port": 8002, "cpu": 256, "memory": 512},
    "mlflow": {"port": 5000, "cpu": 512, "memory": 1024},
    "strategy-service": {"port": 8003, "cpu": 512, "memory": 1024},
    "strategy-container": {"port": None, "cpu": 1024, "memory": 2048},
}

Note: SERVICES defines only the shared platform services. Live/dry-run trading runs on per-strategy ECS services (strategy-{slug}) created by create_strategy_services() in infra/compute/modules/ecs.py and driven by the backend (strategy_operations.py scales them between 0 and 1). There is no generic live-trading/dry-run-trading service (removed in #529).

# DynamoDB tables
DYNAMODB_TABLES = {
    "workflow_state": "tradai-workflow-state",
    "health_state": "tradai-health-state",
    "trading_state": "tradai-trading-state",
    # ...
}

Stack Configuration Overrides¶

Override values per stack in Pulumi.{stack}.yaml:

config:
  aws:region: eu-central-1
  tradai-infrastructure:environment: dev
  tradai-infrastructure:certificate_arn: "arn:aws:acm:..."  # Optional for HTTPS
  tradai-infrastructure:api_domain: "api.tradai.io"        # Optional custom domain

Common Configuration Commands¶

# Set certificate for HTTPS
pulumi config set certificate_arn "arn:aws:acm:eu-central-1:123:certificate/abc"

# Set alert email
pulumi config set --path 'alert_emails[0]' 'alerts@example.com'

# Set CORS origins
pulumi config set --path 'cors_allowed_origins[0]' 'https://app.tradai.io'

# Configure CloudWatch alarm thresholds
pulumi config set alarm_latency_threshold 5000
pulumi config set alarm_min_strategies 1  # For prod

# Configure Lambda schedules
pulumi config set drift_monitor_schedule "rate(1 day)"
pulumi config set retraining_scheduler_schedule "rate(7 days)"

Helper Functions¶

config.py provides helper functions for consistent naming:

# Standard resource name
get_resource_name("backend")  # tradai-backend-dev

# ECR repository name
get_ecr_repo_name("backend")  # tradai/backend

# Standard tags
get_tags(service="backend")
# {
#   "Project": "tradai",
#   "Environment": "dev",
#   "Service": "backend",
#   "ManagedBy": "pulumi",
# }

# Environment short code (for name length limits)
get_env_short()  # "d" for dev, "s" for staging, "p" for prod

Daily Operations¶

Preview and Deploy¶

# Preview all stacks
just infra-preview dev

# Preview a single stack
just infra-preview-persistent dev
just infra-preview-foundation dev
just infra-preview-compute dev
just infra-preview-edge dev

# Deploy all stacks in order
just infra-bootstrap dev

# Deploy a single stack (persistent first)
just infra-up-persistent dev
just infra-up-foundation dev
just infra-up-compute dev
just infra-up-edge dev

View Outputs¶

# Stack outputs (resource IDs, ARNs, endpoints)
just infra-outputs persistent dev
just infra-outputs foundation dev
just infra-outputs compute dev
just infra-outputs edge dev

# Filter a specific output
just infra-outputs compute dev | jq -r '.alb_dns_name'

Service Operations¶

# ECS service status
just ecs-status

# Force ECS redeployment (after image push)
just ecs-force-deploy backend
just ecs-force-deploy-all

# ASG refresh (consolidated EC2 — picks up new launch template)
just asg-refresh
just asg-status

# Push service images to ECR
just service-ecr-login
just service-push backend
just service-push-all

# Check Lambda images
just lambda-check-images

State Management¶

# Export state backup (run from specific stack dir)
cd infra/foundation && pulumi stack export > state-backup-$(date +%Y%m%d).json

# Refresh state from AWS
cd infra/foundation && set -a && source ../.env && set +a && pulumi refresh

# View resource URNs
cd infra/foundation && set -a && source ../.env && set +a && pulumi stack --show-urns

Rollback¶

# Pulumi doesn't have native rollback. Options:

# 1. Redeploy previous code from git
git checkout HEAD~1 -- infra/
just infra-bootstrap dev

# 2. Import previous state backup
cd infra/foundation && pulumi stack import --file state-backup.json
just infra-up-foundation dev

# 3. Manual ECS service rollback
aws ecs update-service --cluster tradai-dev --service tradai-backend-api-dev \
  --task-definition tradai-backend-api-dev:PREVIOUS_REVISION \
  --profile tradai --region eu-central-1

Removing Resources from the Persistent Stack¶

just infra-bootstrap deploys persistent first — that order is for creating resources. When a change removes persistent resources that compute/edge still reference (e.g. decommissioning an ECR repo whose image backs an ECS service), deploy the stacks individually in reverse order so consumers are torn down before the resource they depend on:

just infra-up-compute <stack>   # 1. tear down ECS services / task defs
just infra-up-edge <stack>      # 2. tear down alarms / listener rules
just infra-up-persistent <stack> # 3. delete the now-unreferenced repos

Prod only: persistent resources are created with protect=True and force_delete=False (infra/persistent/modules/ecr.py), so the persistent apply will fail until each removed resource is unprotected and (for ECR) empty:

cd infra/persistent && set -a && source ../.env && set +a

# 1. Confirm the repo is empty (delete images first if not)
aws ecr list-images --repository-name tradai/<repo> --query 'imageIds | length(@)'

# 2. Unprotect the resource in Pulumi state (find URN via: pulumi stack --show-urns)
pulumi state unprotect '<urn of the removed repo>'

# 3. Now the persistent apply can delete it
just infra-up-persistent prod

In dev/staging, force_delete=True and no protection — the reverse-order deploy alone is sufficient.

Adding New Modules¶

Module Template¶

"""New Module Description.

Task ID: XX001
Dependencies: vpc, security_groups
"""

import pulumi
import pulumi_aws as aws
from tradai_infra_shared.config import (
    ENVIRONMENT,
    PROJECT_NAME,
    get_tags,
)


class NewModule:
    """Creates new infrastructure component."""

    def __init__(
        self,
        vpc_id: pulumi.Input[str],
        security_group_id: pulumi.Input[str],
    ) -> None:
        """Initialize NewModule.

        Args:
            vpc_id: VPC ID for resource placement
            security_group_id: Security group to attach
        """
        self.resource = aws.service.Resource(
            f"{PROJECT_NAME}-component",
            vpc_id=vpc_id,
            security_groups=[security_group_id],
            tags=get_tags("component")
            | {"Name": f"{PROJECT_NAME}-component-{ENVIRONMENT}"},
        )

    @property
    def resource_id(self) -> pulumi.Output[str]:
        """Get resource ID."""
        return self.resource.id

Integration Checklist¶

Create module file in the appropriate stack's modules/ directory
Add import to the stack's __main__.py
Choose the right stack: foundation (no image deps), compute (needs ECR images), edge (needs compute)
Export outputs via pulumi.export() if other stacks need the values
Update shared config if new canonical values needed (infra/shared/tradai_infra_shared/config.py)
Add tests in the stack's tests/ directory
Update documentation (this guide)

Example: Adding to a Stack¶

# In compute/__main__.py
from modules.new_module import NewModule

# Read from foundation stack via StackReference
foundation_ref = pulumi.StackReference("tradai-foundation/dev")
vpc_id = foundation_ref.get_output("vpc_id")

new_module = NewModule(
    vpc_id=vpc_id,
    security_group_id=foundation_ref.get_output("ecs_sg_id"),
)
pulumi.export("new_resource_id", new_module.resource_id)

Troubleshooting¶

Common Issues¶

Error	Cause	Solution
`error: no Pulumi.yaml`	Wrong directory	`cd infra/foundation/` (or compute/edge)
`error: stack 'dev' not found`	Stack not initialized	`pulumi stack init dev`
`error: passphrase must be set`	Missing passphrase	`source infra/.env` or set `PULUMI_CONFIG_PASSPHRASE`
`error: unable to deserialize`	State corruption	Restore from backup
`error: resource already exists`	Orphaned resource	Import or delete manually
SSM agent not registering	Custom AMI missing agent	Redeploy compute stack (installs from S3 RPM)
S3/dnf 403 errors on EC2	VPC endpoint policy too restrictive	Check foundation `vpc_endpoints.py` AllowSystemPackageBuckets

State Lock Issues¶

# Check for lock
aws s3api head-object --bucket $BUCKET --key .pulumi/locks/...

# Force unlock (use with caution)
pulumi cancel

Import Existing Resources¶

# Import existing resource
pulumi import aws:s3:Bucket my-bucket existing-bucket-name

# Import with specific URN
pulumi import aws:ecs:Service backend arn:aws:ecs:eu-central-1:123:service/cluster/service

Debug Mode¶

# Verbose logging
PULUMI_DEBUG_COMMANDS=1 pulumi up

# Trace logging
pulumi up --logtostderr -v=9

# Dry run with detailed output
pulumi preview --diff --debug

Accessing Services¶

Dev/Staging: Consolidated EC2¶

In dev/staging, all services run on a single EC2 instance via Docker Compose (cost optimization). Access via SSM Session Manager (no SSH keys needed):

# Find the consolidated instance
aws ec2 describe-instances \
  --filters "Name=tag:Name,Values=tradai-consolidated-dev" "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[].InstanceId' --output text \
  --profile tradai --region eu-central-1

# Interactive shell
aws ssm start-session --target <instance-id> --profile tradai --region eu-central-1

# On the instance:
sudo docker ps                                   # Running containers
sudo docker logs backend-api --tail 50 -f        # Service logs
cd /opt/tradai && sudo docker-compose restart     # Restart services

Production: ECS Fargate¶

In production, services run on ECS Fargate. Use ECS Exec for debugging:

# List running tasks
aws ecs list-tasks --cluster tradai-prod --service-name tradai-backend-api-prod \
  --profile tradai --region eu-central-1

# Interactive shell into a container
aws ecs execute-command --cluster tradai-prod --task <task-arn> \
  --container backend-api --interactive --command "/bin/sh" \
  --profile tradai --region eu-central-1

Service Endpoints¶

All services are fronted by an Application Load Balancer:

Service	Port	Path	Purpose
backend-api	8000	`/api/*`	API gateway, orchestration
data-collection	8002	`/data/*`	Market data fetching
strategy-service	8003	`/strategy/*`	Strategy execution, backtesting
mlflow	5000	`/mlflow/*`	ML experiment tracking

# Get ALB DNS
just infra-outputs compute dev | jq -r '.alb_dns_name'

# Test health
curl -s http://<ALB_DNS>/api/health | jq .

Logs¶

# CloudWatch (consolidated EC2)
aws logs tail /tradai/consolidated/containers --since 30m \
  --profile tradai --region eu-central-1

# CloudWatch (follow in real time)
aws logs tail /tradai/consolidated/containers --follow \
  --profile tradai --region eu-central-1

CI/CD Integration¶

GitHub Actions¶

Infrastructure deployment is managed via the deploy-infra.yml workflow. On PRs touching infra/**, it runs pulumi preview for all environments. Manual dispatch allows deploying to any environment.

Required Secrets¶

Secret	Description	Secured
`PULUMI_CONFIG_PASSPHRASE`	Stack encryption	Yes
`S3_PULUMI_BACKEND_URL`	Backend URL	No
`AWS_ACCESS_KEY_ID`	AWS credentials	Yes
`AWS_SECRET_ACCESS_KEY`	AWS credentials	Yes
`AWS_REGION`	AWS region	No

Cost Optimization¶

Optimization	Savings	Implementation
NAT Instance vs Gateway	~$32/month	`nat_instance.py` uses t4g.nano
FARGATE_SPOT	~70% on backtests	Enabled in `ecs.py`
Single-AZ (dev/staging)	~50% on RDS	`rds.py` `multi_az=False`
db.t4g.micro	~$12/month	Dev/staging RDS instance
On-demand DynamoDB	Pay per use	Default in `dynamodb.py`
S3 lifecycle rules	Auto-cleanup	`s3.py` lifecycle policies

Security Best Practices¶

Secrets: Never in config, always AWS Secrets Manager
RDS: Private subnets only, security group restricted
IAM: Least-privilege roles in iam.py
Security Groups: Minimal ingress rules
WAF: Rate limiting on API Gateway
CloudTrail: Audit logging enabled
VPC Flow Logs: Network traffic logging
Encryption: S3, RDS, DynamoDB at rest

Pulumi Modules Reference - Full module inventory
Infrastructure Issues Runbook - Operational procedures
Database Operations Runbook - RDS failover, backup/restore
Pulumi Code Architecture - Detailed code reference