TradAI CI/CD Pipeline Architecture¶

Version: 1.0.0 | Date: 2026-03-28 | Status: CURRENT Source: .github/workflows/, justfile

TL;DR: GitHub Actions (sole CI/CD). 16 workflows with path-based change detection and dynamic test matrix. Deployment gates: a green dev CI run publishes libraries (#942); v* tags additionally trigger docker-build and Lambda deploy -- all gated behind CI success via workflow_run. 4-stack Pulumi deployment via just infra-bootstrap. Manual workflow_dispatch available for emergency bypasses.

1. Pipeline Overview¶

flowchart LR
    subgraph Triggers
        PR[Pull Request]
        Push[Push to main]
        Tag["Tag v*"]
        Manual[workflow_dispatch]
        Schedule[Weekly / Monday 06:00 UTC]
    end

    subgraph CI["CI Gate (ci.yml)"]
        Changes[Detect Changes]
        Lint[Lint & Format]
        Type[Type Check]
        Test[Test Matrix]
        Security[Security Scan]
        Perf[Performance Tests]
        Contract[Contract Tests]
    end

    subgraph Deploy["Deployment Workflows"]
        Docker[Docker Build & Push]
        Lambda[Deploy Lambdas]
        Publish[Publish Libraries]
        Infra[Deploy Infrastructure]
        Docs[Deploy Documentation]
    end

    subgraph Targets
        ECR[Amazon ECR]
        ECS[ECS Services]
        LambdaFn[Lambda Functions]
        CA[CodeArtifact]
        CF[Cloudflare Pages]
        Pulumi[Pulumi Stacks]
    end

    PR --> CI
    Push --> CI
    Tag --> CI
    Schedule --> CI

    CI -->|workflow_run + v* tag| Docker
    CI -->|workflow_run + v* tag| Lambda
    CI -->|workflow_run: dev or v* tag| Publish
    Manual --> Infra
    Manual --> Lambda
    Push -->|docs paths| Docs

    Docker --> ECR --> ECS
    Lambda --> ECR --> LambdaFn
    Publish --> CA
    Infra --> Pulumi
    Docs --> CF

2. GitHub Actions Workflows¶

16 workflow files in .github/workflows/:

3. CI Gate Structure¶

The CI workflow (ci.yml) is the quality gate. Since #942, publish-libs fires whenever CI passes on dev (or on a v* tag); docker-build and deploy-lambdas also deploy on every dev push.

sequenceDiagram
    participant Dev as Developer
    participant GH as GitHub
    participant CI as CI Workflow
    participant Deploy as Deployment Workflows

    Dev->>GH: Push PR
    GH->>CI: Trigger CI (PR)
    CI->>CI: Detect Changes (dorny/paths-filter)
    CI->>CI: Build Test Matrix
    par Parallel Jobs
        CI->>CI: Lint & Format (Ruff)
        CI->>CI: Type Check (MyPy, 7 packages)
        CI->>CI: Security Scan (pip-audit + Bandit)
        CI->>CI: Performance Tests (PR only)
        CI->>CI: Contract Tests (PR + schedule)
    end
    CI->>CI: Test Matrix (affected legs only, parallel)
    CI-->>Dev: Status checks on PR

    Dev->>GH: Merge + Tag v1.2.3
    GH->>CI: Trigger CI (tag)
    CI->>CI: Full matrix (all 10 legs)
    CI-->>GH: CI completed (success)

    GH->>Deploy: workflow_run event
    par Tag Deployments
        Deploy->>Deploy: Docker Build & Push (4 services)
        Deploy->>Deploy: Deploy Lambdas (23 Dockerfile-based functions + shared base)
        Deploy->>Deploy: Publish Libraries (CodeArtifact)
    end
    Deploy->>Deploy: Redeploy ECS Services

4. Change Detection and Test Matrix¶

The CI workflow uses path-based change detection (dorny/paths-filter@v3) to avoid running all 8 package test suites on every PR. Each filter includes the package's own source plus the specific tradai-common submodules it imports. Heavy consumers (backend, strategy-service, cli) watch all of tradai-common/**; light consumers (data, strategy, data-collection) watch only their specific deps.

28 filters. Eight drive the package legs (deps — workspace config, triggers all — plus common, data, strategy, backend, data-collection, strategy-service, cli); the rest gate the suite legs (lambdas, root-unit, integration-tests, mlflow-service, root-suites-{financial,regression,data-quality,backtest-validation}) and the non-matrix jobs (any-code, api-contract, version-guard, infra, aws-emulator, guard-surface, security-inputs, image-inputs, ci-config, renovate-config, ledger, lock-inputs).

On PRs, only affected packages run. On push/schedule/dispatch, all 10 matrix entries run: six packages at a 60% coverage threshold, cli at 45%, mlflow-service at 0, and two test-suite legs at 0 (integration, and root-suites — one leg covering tests/unit/*.py, tests/unit/lambdas, tests/financial, tests/regression, tests/data_quality and tests/backtest_validation, which used to be six separate legs). Integration tests run when common changes or 2+ packages are affected.

5. Deployment Flows¶

5.1 Lambda Deployment (5-Stage Pipeline)¶

The deploy-lambdas.yml workflow builds and deploys 22 Lambda functions as container images.

flowchart TB
    subgraph Stage1["Stage 1: Version"]
        V[Calculate Version<br/>tag or manual-YYYYMMDDHHMMSS]
    end

    subgraph Stage2["Stage 2: Wheel"]
        W[Build tradai-common wheel<br/>uv build libs/tradai-common]
    end

    subgraph Stage3["Stage 3: Base Image"]
        B[Build lambda-base<br/>lambdas/base/Dockerfile]
    end

    subgraph Stage4["Stage 4: Individual Lambdas"]
        direction LR
        L1[backtest-consumer]
        L2[drift-monitor]
        L3[health-check]
        L4[update-status]
        LN["... 14 more"]
    end

    subgraph Stage5["Stage 5: Update Functions"]
        U[aws lambda update-function-code<br/>for each function]
    end

    V --> Stage2
    Stage2 --> Stage3
    Stage3 --> Stage4
    Stage4 --> Stage5

22 Lambda functions auto-discovered from lambdas/*/Dockerfile (backtest-consumer, drift-monitor, health-check, update-status, validate-strategy, and 17 more -- plus base shared image). The 23^rd Lambda (update-nat-routes) is an inline Python handler deployed directly via Pulumi (no Dockerfile), so it is not part of the lambda-bootstrap pipeline.

Local equivalent:

just lambda-bootstrap          # Full pipeline: wheel -> base -> all lambdas -> ECR push
just lambda-build-all          # Build only (no push)
just lambda-push-all           # Push pre-built images to ECR

5.2 Service Deployment (Docker Build & Push)¶

The docker-build.yml workflow builds 4 service images and redeploys ECS.

flowchart LR
    subgraph Build["Parallel Builds"]
        B1[backend<br/>services/backend/Dockerfile]
        B2[data-collection<br/>services/data-collection/Dockerfile]
        B3[strategy-service<br/>services/strategy-service/Dockerfile]
        B4[mlflow<br/>services/mlflow/Dockerfile]
    end

    subgraph Push["ECR Push"]
        ECR["ECR Registry<br/>:version + :latest tags"]
    end

    subgraph Redeploy["ECS Redeploy"]
        ECS["aws ecs update-service<br/>--force-new-deployment"]
    end

    Build --> Push --> Redeploy

ECS service names follow the pattern tradai-{service}-{env} with backend-api (not backend) matching infra/shared/tradai_infra_shared/config.py.

Local equivalent:

just docker-build              # Build all 4 service images (linux/amd64)
just service-push-all          # Build + push all to ECR
just ecs-force-deploy-all      # Force ECS redeployment (strategy-service)

5.3 Infrastructure Deployment (4-Stack Pulumi)¶

The deploy-infra.yml workflow manages 4 Pulumi stacks deployed in strict order.

infra/ci (the RunsOn self-hosted runner stack) is intentionally outside this workflow — it is applied by hand only (just infra-up-ci), since deploy-infra.yml itself would run on the RunsOn runners that stack provisions once CI migrates to them.

flowchart TB
    subgraph Validate
        V[Lint + Unit Tests<br/>all 4 stacks]
    end

    subgraph PR["PR: Preview"]
        P1["Preview dev"]
        P2["Preview staging"]
        P3["Preview prod"]
        PS["Post PR Summary<br/>(create/update/delete counts)"]
    end

    subgraph Manual["Manual: Deploy"]
        D1["persistent<br/>S3, DynamoDB, ECR, Cognito, CodeArtifact"]
        D2["foundation<br/>VPC, RDS, SQS, SNS"]
        D3["Lambda Bootstrap<br/>(just lambda-bootstrap)"]
        D4["compute<br/>ALB, ECS, Lambda, Step Functions"]
        D5["edge<br/>API Gateway, WAF, CloudWatch"]
    end

    Validate --> PR
    Validate --> Manual

    P1 --> PS
    P2 --> PS
    P3 --> PS

    D1 --> D2 --> D3 --> D4 --> D5

Local equivalent:

just infra-bootstrap dev       # Full: persistent -> foundation -> lambda-bootstrap -> service-push -> compute -> edge
just infra-up-foundation dev   # Single stack
just infra-preview dev         # Preview all 4 stacks
just infra-recover foundation dev  # Recovery: cancel + refresh + preview drift

Deploy script: infra/pulumi-ci.sh handles layer iteration, backend login, stack selection, and pulumi preview/pulumi up for any combination of layers and environments.

5.4 Library Publishing¶

The publish-libs.yml workflow publishes all seven TradAI packages to AWS CodeArtifact for use by the separate tradai-strategies repository.

Publishing is continuous. Every green dev CI run from a push publishes — no release step, and no v* tag needed for Python packages. Two bounds the trigger does not advertise: ci.yml's paths-ignore means a docs-only merge starts no CI run at all, and the job requires workflow_run.event == 'push', so ci.yml's weekly cron and a manual CI dispatch produce a green dev run that publishes nothing.

v* tags are not obsolete — they still drive the Docker and Lambda release paths, and re-run library publishing. They are simply no longer the trigger Python packages depend on.

Three independent mechanisms cover three different failures, and conflating them is the mistake

942 was originally about:¶

CodeArtifact acceptance is asset/checksum based, which is subtler than "versions are immutable": an identical same-filename re-upload is accepted with 2xx, the same filename with different bytes 409s, and a new filename can still be added under an existing version. So without a version bump the changed wheel is rejected while the run keeps going — and it is the read-back, not the upload, that turns that into a failure. The guard is a required check because catching it on the PR is cheaper than catching it after a merge.

Cross-repo contract. On a run that publishes something new, notify-strategies mints a GitHub App installation token scoped to repositories: strategies and sends a repository_dispatch of type tradai-packages-published. nightly.yml in tradai-bot/strategies listens for exactly that type and validates every strategy against the new packages. POST /dispatches returns 204 whether or not anything listens, so a rename on either side is green and permanently inert. A test pins the string on this side only — it does not read the consumer's workflow, so a listener rename in tradai-bot/strategies remains undetectable from here.

This buys latency, not safety: that repo's 03:00 UTC cron already validates against whatever is latest in the registry, so a dead dispatch costs detection speed, not coverage.

Authentication. All three publishing workflows (publish-libs, publish-npm-types, registry-drift) assume an IAM role by OIDC rather than using stored access keys. The trust policy admits exactly one subject, repo:tradai-bot/tradai:environment:publish-dev, which is why the environment: line on those jobs is load-bearing: workflow_dispatch runs the workflow file as it exists at the dispatched ref, so any in-file guard can be deleted — but deleting environment: yields a ref: subject the policy rejects, and no credentials are ever issued. See §4 (Security).

# CI pipeline steps:
uv build <each of the 7 packages> --out-dir dist       # Build wheel
twine upload --repository-url $CODEARTIFACT_URL ... # Publish to CodeArtifact

Local equivalent:

just codeartifact-login dev    # Configure pip/twine auth (12h expiry)
just publish-all-libs dev      # Build + publish all seven packages

5.5 Documentation Deployment¶

Triggered by pushes to docs/**, mkdocs.yml, lib/service READMEs, and architecture reports. Builds MkDocs with Material theme and deploys to Cloudflare Pages.

6. Emergency Procedures¶

• Emergency Lambda deploy: Actions > Deploy Lambdas > Run workflow (select env). Or locally: just lambda-bootstrap.
• Emergency ECS hotfix: just ecs-force-deploy-all (or single service: just ecs-force-deploy strategy-service).
• Rollback: Check out previous tag, rebuild and push: git checkout v1.2.2 && just docker-build && just service-push-all && just ecs-force-deploy-all. For Lambdas: just lambda-bootstrap.
• Infrastructure recovery: just infra-recover foundation dev (cancels pending ops, refreshes state, previews drift). deploy-infra.yml also prints recovery instructions on failure.

7. Changelog¶

8. Dependencies¶

This document relates to:

• 02-ARCHITECTURE-OVERVIEW.md -- System architecture and service topology
• 04-SECURITY.md -- Security controls and secrets management
• 05-SERVICES.md -- Service definitions (backend, data-collection, strategy-service, mlflow)
• .github/workflows/ -- All 16 workflow definitions
• justfile -- Local development and deployment commands
• infra/pulumi-ci.sh -- Pulumi deployment script (4-stack orchestration)
• .github/actions/setup-workspace/ -- Shared composite action for CI setup