TradAI Deployment Pipeline¶

Version: 1.0.0 | Date: 2026-03-28 | Status: CURRENT Source: justfile, infra/, scripts/, .github/workflows/

1. TL;DR¶

4-stage deployment: Lambda images → Pulumi infrastructure (4 stacks in order) → Service containers → ECS force-deploy. Full bootstrap via just infra-bootstrap. Pulumi CI via infra/pulumi-ci.sh.

2. Deployment Architecture¶

flowchart TD
    CA[CodeArtifact Login] --> BW[Build Library Wheels]
    BW --> BLB[Build Lambda Base Image]
    BLB --> BLA[Build All Lambdas<br/><i>auto-discovers lambdas/*/Dockerfile</i>]
    BLA --> ECR_L[Push Lambda Images to ECR]

    ECR_L --> P1[Pulumi: persistent<br/><i>S3, DynamoDB, ECR, Cognito, CodeArtifact</i>]
    P1 --> P2[Pulumi: foundation<br/><i>VPC, RDS, SQS, SNS, Security Groups</i>]
    P2 --> P3[Pulumi: compute<br/><i>ALB, ECS, Lambda, Step Functions</i>]
    P3 --> P4[Pulumi: edge<br/><i>API Gateway, WAF, CloudWatch</i>]

    P4 --> DBS[Docker Build Services<br/><i>backend, data-collection,<br/>strategy-service, mlflow</i>]
    DBS --> ECR_S[Push Service Images to ECR]
    ECR_S --> ECS[ECS Force Deploy<br/><i>Rolling update</i>]

    style P1 fill:#f9f,stroke:#333
    style P2 fill:#f9f,stroke:#333
    style P3 fill:#f9f,stroke:#333
    style P4 fill:#f9f,stroke:#333
    style ECR_L fill:#bbf,stroke:#333
    style ECR_S fill:#bbf,stroke:#333
    style ECS fill:#bfb,stroke:#333

3. First-Time Bootstrap¶

Step-by-step for a new environment:

What just infra-bootstrap Runs¶

The infra-bootstrap recipe executes the following sequence:

1. just infra-up-persistent dev -- S3 buckets, DynamoDB, ECR repos, Cognito, CodeArtifact
2. just infra-up-foundation dev -- VPC, subnets, NAT, RDS, SQS/SNS, security groups
3. just lambda-bootstrap -- Build wheel + base image + all lambdas + push to ECR
4. just service-push-all -- Build all 4 service Docker images + push to ECR
5. just infra-up-compute dev -- ALB, ECS, Lambda functions, Step Functions
6. just infra-up-edge dev -- API Gateway, WAF, CloudWatch dashboards and alarms

4. Lambda Deployment Pipeline¶

4.1 Build Chain¶

sequenceDiagram
    participant W as Wheel Build
    participant B as Base Image
    participant L as Lambda Images
    participant E as ECR
    participant A as AWS Lambda

    W->>W: uv build libs/tradai-common --out-dir dist
    W->>B: tradai_common*.whl
    B->>B: docker build -f lambdas/base/Dockerfile
    B->>L: tradai/lambda-base:latest
    L->>L: docker build -f lambdas/<name>/Dockerfile<br/>(22 lambdas in parallel)
    L->>E: docker push tradai/lambda-<name>:tag
    E->>A: aws lambda update-function-code --image-uri

The base image (lambdas/base/Dockerfile) uses public.ecr.aws/lambda/python:3.11, installs gcc, g++, make, tar, gzip, and then installs the tradai-common wheel plus boto3, pydantic, pydantic-settings, and httpx. Each individual Lambda extends this base image with its own handler.py.

4.2 Commands¶

4.3 Auto-Discovery¶

Lambda build discovers all directories in lambdas/ automatically. Any directory containing a Dockerfile (excluding base/) is treated as a Lambda function.

Current Lambda functions (17):

5. Infrastructure Deployment (Pulumi)¶

5.1 Stack Order¶

The 4 Pulumi stacks must be deployed in strict dependency order:

flowchart LR
    P[persistent] --> F[foundation] --> C[compute] --> E[edge]

    P:::persistent
    F:::foundation
    C:::compute
    E:::edge

    classDef persistent fill:#e1bee7,stroke:#333
    classDef foundation fill:#bbdefb,stroke:#333
    classDef compute fill:#c8e6c9,stroke:#333
    classDef edge fill:#ffe0b2,stroke:#333

5.2 Commands¶

5.3 Pulumi CI Script¶

infra/pulumi-ci.sh is used by GitHub Actions for automated deployments.

Usage:

./infra/pulumi-ci.sh <layer> <stack> <command> [s3-backend-url]

Arguments:

Required environment variables: - PULUMI_CONFIG_PASSPHRASE -- Stack encryption passphrase - S3_PULUMI_BACKEND_URL -- S3 backend for Pulumi state - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY -- AWS credentials - AWS_REGION -- (optional, defaults to eu-central-1)

When layer=all, the script runs all stacks in order: persistent → foundation → compute → edge.

5.4 Preview Before Deploy¶

# Preview all stacks for dev environment
just infra-preview dev

# Preview individual stacks
just infra-preview-persistent dev
just infra-preview-foundation dev
just infra-preview-compute dev
just infra-preview-edge dev

On pull requests that touch infra/**, the deploy-infra.yml workflow automatically runs pulumi preview across all stacks for dev, staging, and prod, posting a summary table as a PR comment.

6. Service Container Deployment¶

6.1 Docker Build¶

Four services are containerized for ECS deployment:

# Build all 4 service images (linux/amd64)
just docker-build

# Build a single service
just docker-build-service backend

6.2 ECR Push¶

# Login to ECR + build + push all services
just service-push-all

# Push a single service
just service-push backend

# Verify images exist in ECR
just service-check-images

6.3 ECS Redeployment¶

# Force rolling update on all ECS services
just ecs-force-deploy-all

# Force rolling update on a specific service
just ecs-force-deploy strategy-service

# Check ECS service status
just ecs-status

# View recent ECS events
just ecs-events strategy-service

ECS services targeted by ecs-force-deploy-all: strategy-service. (The former live-trading/dry-run-trading decoys were removed in #529; per-strategy tradai-strategy-{slug} services redeploy via their own start/stop lifecycle.)

The docker-build.yml GitHub Actions workflow triggers on a push to dev (in parallel with CI) and on version tags after CI passes, builds all 4 service images in parallel, pushes to ECR with both the version tag and latest, then forces a new ECS deployment for backend-api, data-collection, strategy-service, and mlflow.

7. Library Publishing¶

Libraries are published to AWS CodeArtifact for consumption by the separate tradai-strategies repository.

The normal path is a merge to dev, not a local command. Since #942 a green dev CI run from a push invokes publish-libs.yml, which publishes all seven packages, reads the registry back to verify them, and notifies the strategies repo when a wheel was new. The local flow below is an escape hatch: it has no read-back, no CI/ref validation and no audit trail.

flowchart LR
    M[Merge to dev] --> CI[CI green]
    CI -->|workflow_run, event=push| PL[publish-libs<br/>publish + registry read-back]
    PL -->|only if a wheel was new| ST[notify tradai-bot/strategies]

    L[Local escape hatch:<br/><code>just codeartifact-login dev</code>] --> B[Build Wheels<br/><code>just build-libs</code>]
    B --> P[<code>just publish-all-libs dev</code><br/>no read-back, no audit trail]

    style L fill:#fff3e0
    style P fill:#fff3e0
    style PL fill:#c8e6c9

The publish-libs.yml GitHub Actions workflow triggers after every green CI run on dev, and on v* tags (#942), publishing all seven Python packages to CodeArtifact. Note CodeArtifact does not honour --skip-existing; the workflow inspects twine's output to tell a 409 from a real error, then verifies each published wheel by sha256. CodeArtifact's rule is asset/checksum based — an identical re-upload is accepted with 2xx and only same-filename/different-bytes 409s — so the sha256 read-back, not the 409 handling, is what proves the registry serves this pipeline's build.

8. Rollback Procedures¶

Service Rollback¶

Redeploy the previous working image by forcing a new ECS deployment. ECS uses the :latest tag, so push a known-good image and redeploy:

# Rebuild from a known-good commit
git checkout <good-commit>
just docker-build-service backend
just service-push backend
just ecs-force-deploy backend

Lambda Rollback¶

Rebuild Lambda images from the previous commit and push:

git checkout <good-commit>
just lambda-bootstrap

The deploy-lambdas.yml workflow updates each Lambda function code to the new image URI. AWS Lambda also supports version aliases for instant rollback without rebuilding.

Infrastructure Rollback¶

# Recover from a failed deployment (cancel pending ops, refresh state)
just infra-recover <layer> <stack>

# Re-deploy with current code (Pulumi converges to desired state)
just infra-up-<layer> <stack>

Pulumi stores state in S3. Each pulumi up converges the actual cloud state to match the declared code. Rolling back infrastructure means running pulumi up with the previous code version.

9. Environment Configuration¶

Local Development¶

The infra/.env file is sourced automatically by all just infra-* recipes via:

set -a && source ../.env && set +a

Required variables in infra/.env:

CI/CD (GitHub Actions)¶

GitHub Actions workflows use these repository secrets, one repository variable, and one environment secret pair:

CI/CD Workflow Triggers¶

10. Changelog¶

Dependencies¶

This document references: