CI/CD Pipeline Guide¶

Complete guide covering the CI/CD pipelines for both the tradai-uv platform (this repo) and the tradai-strategies repository. All CI/CD runs on GitHub Actions.

Pipeline Overview¶

Platform Pipeline (tradai-uv)¶

graph TD
    T1["PR"] --> CI1["CI (ci.yml)<br/>changes → lint, typecheck,<br/>test, security, version-bump-guard"]
    T2["Push to dev"] --> CI2["CI (ci.yml)<br/>lint → typecheck → test → security-scan"]
    T3["Tag v*"] --> CI2
    T2 -->|"push trigger<br/>(parallel to CI)"| DB["docker-build<br/>(4 services)"]
    T2 -->|"push trigger<br/>(parallel to CI)"| DL["deploy-lambdas"]
    CI2 -->|"workflow_run<br/>(CI success, tag v*)"| DB
    CI2 -->|"workflow_run<br/>(CI success, tag v*)"| DL
    CI2 -->|"workflow_run<br/>(CI success, event=push)"| PL["publish-libs<br/>(CodeArtifact)"]
    PL -->|"repository_dispatch<br/>only if a wheel was new"| ST["tradai-bot/strategies<br/>validates"]

Strategy Pipeline (tradai-strategies)¶

┌─────────────────────────────────────────────────────────────────────┐
│                        Pull Request                                 │
├─────────────────────────────────────────────────────────────────────┤
│  lint  →  typecheck  →  unit-tests  →  smoke-backtest              │
└─────────────────────────────────────────────────────────────────────┘
                               ↓
┌─────────────────────────────────────────────────────────────────────┐
│                     Main Branch Merge                               │
├─────────────────────────────────────────────────────────────────────┤
│  lint  →  typecheck  →  unit-tests  →  smoke-backtest              │
└─────────────────────────────────────────────────────────────────────┘
                               ↓
┌─────────────────────────────────────────────────────────────────────┐
│                     Tag Release (*-v*)                              │
├─────────────────────────────────────────────────────────────────────┤
│  lint → typecheck → tests → full-backtest → docker → ECR → MLflow  │
└─────────────────────────────────────────────────────────────────────┘

GitHub Actions Workflows¶

All workflow files live in .github/workflows/. Reusable workflows are prefixed with _.

1. CI (ci.yml)¶

Triggers: Push to dev and main (with paths-ignore for docs/reports/markdown), tags v*, pull requests, weekly schedule, manual dispatch. The dev push path is what #942's publishing depends on.

Path filtering: Uses dorny/paths-filter@v3 to build a dynamic test matrix. Each package filter lists its own source plus the tradai-common submodules it imports. This avoids the "common changed so test everything" cascade.

graph LR
    changes --> test["test (dynamic matrix)"]
    changes --> lint["lint (_lint.yml reusable)"]
    changes --> typecheck
    changes --> sec["security-scan<br/>(pip-audit + Bandit SAST)"]
    changes --> static["static-checks<br/>(actionlint, shell-tests, deps-lock-check,<br/>renovate-config, ledger, infra-shellcheck,<br/>test-coverage-map)"]
    changes --> contract["contract-tests<br/>(PR + push + schedule)"]
    changes --> integ["integration tests<br/>(schedule/dispatch only)"]

Performance benchmarks are not on the PR path. They run enforced on a daily cron (.github/workflows/performance.yml, #862); the PR path keeps only a collection smoke. Previously they ran per-PR behind || true, which meant they could not fail — the last run before the move reported "11 passed, 2 errors" against a MiniStack that was never started, and still reported success.

Workflow Coordination (CI Gate)¶

Deployment workflows use workflow_run trigger to gate on CI success:

This gates the tag path on CI. Note docker-build and deploy-lambdas additionally run directly on a dev push, in parallel with CI and therefore ungated (see §2 and §3); publish-libs has no direct push trigger, so it is the one that is CI-gated on both paths.

sequenceDiagram
    actor Dev as Developer
    participant GH as GitHub
    participant CI as CI Workflow<br/>(ci.yml)
    participant Docker as Docker Build<br/>(docker-build.yml)
    participant Lambdas as Deploy Lambdas<br/>(deploy-lambdas.yml)
    participant Publish as Publish Libs<br/>(publish-libs.yml)

    Dev->>GH: git push origin v1.2.0 (tag)
    GH->>CI: Trigger on tag push (v*)

    CI->>CI: changes (path detection)
    CI->>CI: lint (Ruff)
    CI->>CI: typecheck (mypy)
    CI->>CI: test (pytest matrix)
    CI->>CI: security-scan (pip-audit + Bandit)

    CI-->>GH: CI completed (success)

    par workflow_run triggers (parallel)
        GH->>Docker: workflow_run: CI success + tag v*
        Docker->>Docker: Build 4 service images
        Docker->>Docker: Push to ECR
        Docker->>Docker: Force redeploy ECS services
    and
        GH->>Lambdas: workflow_run: CI success + tag v*
        Lambdas->>Lambdas: version (compute version + select lambdas)
        Lambdas->>Lambdas: build-images (base + every selected lambda)
        Lambdas->>Lambdas: update-functions
    and
        GH->>Publish: workflow_run: CI success + tag v*
        Publish->>Publish: Build all seven wheels
        Publish->>Publish: Publish to CodeArtifact
    end

2. Docker Build (docker-build.yml)¶

Triggers: push to dev, workflow_run on CI (tag v*), manual dispatch

Builds and pushes Docker images for all services to ECR, then redeploys ECS:

Images are tagged with the version (e.g., v1.2.3) and latest. After push, the workflow force-redeploys ECS services (backend-api, data-collection, strategy-service, mlflow).

Version extraction: Uses github.event.workflow_run.head_branch (not GITHUB_REF_NAME) because workflow_run context does not have the tag in GITHUB_REF_NAME.

3. Deploy Lambdas (deploy-lambdas.yml)¶

Triggers: push to dev, workflow_run on CI (tag v*), manual dispatch

Discovers and deploys Lambda functions in three jobs:

version ──→ build-images ──→ update-functions
   │              │
   │              └─ Base image + every selected lambda, in one job
   └─ Calculates version + selects which lambdas to build

On the dev push lane, lambda selection is incremental: only lambdas whose inputs changed since the last successful dev deploy are built. Tag runs, prod runs, and any run where a shared input changed (e.g. tradai-common, the base image) build every lambda.

Lambda Discovery: Automatically finds all directories in lambdas/ with a Dockerfile (excluding base/). No config update needed when adding new lambdas.

Manual Dispatch: Select environment (dev, staging, prod) for emergency deploys that bypass the CI gate.

4. Publish Libraries (publish-libs.yml)¶

Triggers: workflow_run on CI — every green run on dev, plus v* tags. Also workflow_dispatch for a manual publish of a validated ref (#942).

Publishes all seven TradAI workspace wheels to AWS CodeArtifact (tradai-python-dev repository) for consumption by the strategies repository:

# The strategies repo consumes this via:
pip install tradai-strategy --index-url https://...codeartifact.../pypi/tradai-python-dev/simple/

5. Deploy Infrastructure (deploy-infra.yml)¶

Triggers: Pull requests (paths: infra/**), manual dispatch

PR behavior: Runs pulumi preview on all stacks (dev, staging, prod) in parallel via matrix, then posts a summary comment on the PR showing create/update/delete/replace counts per stack.

Manual deployment: Select stack and command from the Actions UI. Uses infra/pulumi-ci.sh to orchestrate all Pulumi layers.

6. Other Workflows¶

Strategy CI/CD Pipeline (tradai-strategies)¶

Note: The CI/CD workflows below run in the tradai-strategies repository, not in the platform repo.

The proprietary strategies repository has its own CI/CD pipeline. This section documents the expected workflow for that separate repo.

Pipeline Stages¶

Stage 1: Lint¶

lint:
  script:
    - just lint $STRATEGY_NAME

Checks: Ruff formatting, linting, import sorting.

Failure fix: just fmt $STRATEGY_NAME

Stage 2: Type Check¶

typecheck:
  script:
    - just typecheck $STRATEGY_NAME

Checks: mypy strict type checking, Protocol compliance.

Failure fix: uv run mypy $STRATEGY_NAME --show-error-codes

Stage 3: Unit Tests¶

test:
  script:
    - just test $STRATEGY_NAME

Requirements: All tests pass, coverage >= 80%.

Stage 4: Smoke Backtest¶

smoke-backtest:
  script:
    - just backtest-smoke $STRATEGY_NAME

30-day timerange, single pair (BTC/USDT:USDT), basic validation, ~5 minutes.

Stage 5: Full Backtest (Release Only)¶

full-backtest:
  script:
    - just backtest-full $STRATEGY_NAME --timerange 20240101-20241201

12-month timerange, all configured pairs, full metrics validation:

Stage 6: Docker Build & ECR Push¶

The strategy is packaged as a Docker image using Freqtrade as the base:

FROM freqtradeorg/freqtrade:stable

COPY . /app
RUN pip install /app
COPY configs /freqtrade/user_data/configs

ENTRYPOINT ["freqtrade", "trade"]

Built and pushed to ECR with version and latest tags.

Stage 7: MLflow Registration¶

mlflow-register:
  script:
    - just mlflow-register $STRATEGY_NAME $TAG

Registers model version with backtest metrics, Docker image URI, and strategy metadata.

Strategy GitHub Actions Workflow¶

The tradai-strategies repo uses this workflow (in its own .github/workflows/strategy-ci.yml):

name: Strategy CI/CD

on:
  pull_request:
    paths:
      - 'strategies/**'
  push:
    branches: [main]
    tags:
      - '*-v*'

env:
  AWS_REGION: eu-central-1

jobs:
  detect-strategies:
    runs-on: ubuntu-latest
    outputs:
      strategies: ${{ steps.detect.outputs.strategies }}
    steps:
      - uses: actions/checkout@v4
      - id: detect
        run: |
          STRATEGIES=$(git diff --name-only ${{ github.event.before }} ${{ github.sha }} \
            | grep '^strategies/' | cut -d'/' -f2 | sort -u \
            | jq -R -s -c 'split("\n")[:-1]')
          echo "strategies=$STRATEGIES" >> $GITHUB_OUTPUT

  lint-test:
    needs: detect-strategies
    runs-on: ubuntu-latest
    strategy:
      matrix:
        strategy: ${{ fromJson(needs.detect-strategies.outputs.strategies) }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install uv
      - run: cd strategies/${{ matrix.strategy }} && uv sync
      - run: just lint ${{ matrix.strategy }}
      - run: just typecheck ${{ matrix.strategy }}
      - run: just test ${{ matrix.strategy }}

  smoke-backtest:
    needs: lint-test
    runs-on: ubuntu-latest
    strategy:
      matrix:
        strategy: ${{ fromJson(needs.detect-strategies.outputs.strategies) }}
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}
      - name: CodeArtifact Login
        run: |
          export CODEARTIFACT_TOKEN=$(aws codeartifact get-authorization-token \
            --domain tradai --query authorizationToken --output text)
          pip config set global.extra-index-url \
            "https://aws:${CODEARTIFACT_TOKEN}@tradai-${AWS_REGION}.d.codeartifact.${AWS_REGION}.amazonaws.com/pypi/tradai-python-dev/simple/"
      - run: just backtest-smoke ${{ matrix.strategy }}

  release:
    if: startsWith(github.ref, 'refs/tags/')
    needs: smoke-backtest
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Extract strategy from tag
        id: extract
        run: |
          TAG=${GITHUB_REF#refs/tags/}
          STRATEGY=$(echo $TAG | sed 's/-v[0-9].*//')
          VERSION=$(echo $TAG | sed 's/.*-v//')
          echo "strategy=$STRATEGY" >> $GITHUB_OUTPUT
          echo "version=$VERSION" >> $GITHUB_OUTPUT
          echo "tag=$TAG" >> $GITHUB_OUTPUT
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}
      - run: just backtest-full ${{ steps.extract.outputs.strategy }}
      - name: Build Docker
        run: |
          docker build \
            -t ${{ secrets.ECR_REGISTRY }}/${{ steps.extract.outputs.strategy }}:${{ steps.extract.outputs.tag }} \
            -t ${{ secrets.ECR_REGISTRY }}/${{ steps.extract.outputs.strategy }}:latest \
            strategies/${{ steps.extract.outputs.strategy }}
      - name: Push to ECR
        run: |
          aws ecr get-login-password | docker login --username AWS --password-stdin ${{ secrets.ECR_REGISTRY }}
          docker push ${{ secrets.ECR_REGISTRY }}/${{ steps.extract.outputs.strategy }}:${{ steps.extract.outputs.tag }}
          docker push ${{ secrets.ECR_REGISTRY }}/${{ steps.extract.outputs.strategy }}:latest
      - run: just mlflow-register ${{ steps.extract.outputs.strategy }} ${{ steps.extract.outputs.tag }}

Required Secrets¶

GitHub Actions Secrets (tradai-uv)¶

Configure in GitHub Settings, then Secrets and variables, then Actions:

GitHub Actions Secrets (tradai-strategies)¶

Versioning Convention¶

Tag Format¶

Platform (tradai-uv): v<semver> (e.g., v1.0.0, v0.0.0-test)

Strategies (tradai-strategies): <strategy-slug>-v<semver> (e.g., momentum-v1.0.0, trend-following-v2.1.3, ml-rsi-v1.0.0-rc1)

Version Incrementing¶

CodeArtifact Integration¶

Setup¶

# Login (12-hour token)
just codeartifact-login

# Or manually
export CODEARTIFACT_TOKEN=$(aws codeartifact get-authorization-token \
  --domain tradai \
  --query authorizationToken \
  --output text)

pip config set global.extra-index-url \
  "https://aws:${CODEARTIFACT_TOKEN}@tradai-eu-central-1.d.codeartifact.eu-central-1.amazonaws.com/pypi/tradai-python-dev/simple/"

Publishing tradai-strategy¶

Bump the version and merge. When you change tradai-strategy (or any publishable package) in tradai-uv, increment its version in pyproject.toml in the same PR — version-bump-guard is a required check and will block the PR otherwise. The merge to dev publishes; there is no release step, and no tag to cut for Python packages.

Publishing by hand is an escape hatch, not the normal path — it has no registry read-back, no CI/ref validation and no audit trail:

cd libs/tradai-strategy
uv build
twine upload --repository codeartifact dist/*

In CI, this is handled by publish-libs.yml, which publishes to the tradai-python-dev CodeArtifact repository. It does NOT pass --skip-existing — CodeArtifact does not honour that flag — so the workflow inspects twine's output to tell an expected 409 from a real error, then verifies each published wheel by sha256. Note a 409 is not the usual outcome of republishing: CodeArtifact accepts an identical asset with 2xx and 409s only on same-filename/different-bytes, so the sha256 read-back is what actually establishes the registry holds this pipeline's build.

Consuming in Strategies¶

# strategies/my-strategy/pyproject.toml
[project]
dependencies = [
    "tradai-strategy>=1.0.0",  # From CodeArtifact
    "freqtrade>=2026.4",
]

Adding New Services¶

1. Create service directory: services/<name>/
2. Add Dockerfile: services/<name>/Dockerfile
3. Update docker-build.yml matrix:

matrix:
  include:
    # ... existing services
    - service: new-service
      dockerfile: services/new-service/Dockerfile
      image_name: tradai/new-service

Adding New Lambdas¶

Lambdas are auto-discovered. Just create:

lambdas/
└── new-lambda/
    ├── Dockerfile
    └── handler.py

The next push to dev builds and deploys it — deploy-lambdas.yml triggers on the push directly, not only on a tag.

Manual Commands¶

Local Development¶

# Full CI locally
just check                    # lint + typecheck + test

# Individual stages
just lint
just typecheck
just test
just fmt                      # Auto-fix formatting

Strategy Development (in tradai-strategies)¶

# Full CI locally
just ci momentum-strategy

# Individual stages
just lint momentum-strategy
just typecheck momentum-strategy
just test momentum-strategy
just backtest-smoke momentum-strategy

Release Process (tradai-uv)¶

Tags are cut from dev, not main — main is dormant (see the branch table in the root AGENTS.md). A tag from main would publish a months-old tree.

# 1. Ensure dev is up to date
git checkout dev && git pull

# 2. Tag and push
git tag v1.1.0
git push origin v1.1.0

One v* tag publishes all seven workspace packages; there is no per-package release.

Since #942, a tag run normally re-uploads content the dev merge already published — and that is the healthy outcome, not a failure. It does not show up as a 409. CodeArtifact's rule is asset/checksum based:

So the healthy tag run is a series of successful no-op uploads, and Uploading … for all seven is what you should expect to see. (Verified on the v0.1.5 run: seven Uploading lines, zero 409s, including tradai_strategy_service-0.1.3 which was already in the registry.)

A 409 therefore means something real — the same version carries different bytes. The upload step tolerates it (it cannot tell the two 409 causes apart and prints "skipping"), so the run turns red later, at Verify what the registry actually serves — grepping for a red publish step and finding none does not mean there was no 409. What to confirm on a release is that that step is green: it hashes every wheel against what the registry now returns, which is the check that distinguishes "re-uploaded our own identical build" from "someone else's artifact is sitting on this version".

Release Process (tradai-strategies)¶

# 1. Ensure main is up to date
git checkout main && git pull

# 2. Update version in pyproject.toml
# version = "1.0.0" -> "1.1.0"

# 3. Commit
git add .
git commit -m "chore(momentum-strategy): bump version to 1.1.0"
git push

# 4. Create and push tag
git tag momentum-strategy-v1.1.0
git push --tags

E2E Testing the Deployment Pipeline¶

Push a version tag to trigger CI, then Docker Build and Deploy Lambdas.

Publish Libraries no longer needs a tag. Since #942 it runs after every green CI run on dev, so by the time you cut a tag its packages are normally already published. The tag run still re-uploads all seven, and those uploads still succeed — CodeArtifact accepts an identical asset with 2xx (see Release Process above), so seven Uploading … lines is the healthy output, not an anomaly. What the tag run adds is the sha256 read-back, so a red tag run is worth investigating. To exercise the publish path deliberately, use its workflow_dispatch instead — a tag push is no longer a meaningful test of it.

Why run E2E tests¶

docker-build and deploy-lambdas run on every push to dev (branches: [dev], no path filter), so their dev path is exercised constantly. Since #942 publish-libs has a dev path too, so all three now publish/deploy from dev. docker-build and deploy-lambdas compute versions differently on the tag path, so each retains release-only code a dev push never touches. publish-libs has NO version-extraction branch — versions come from each pyproject.toml via uv build — its only event-specific code is the dispatch-ref validation. Without periodic E2E testing, bugs in those paths go undetected until a real release.

Examples of bugs caught by E2E testing (issue #93):

• Version extraction bug: Workflows used GITHUB_REF_NAME which returns main instead of the tag in workflow_run context. Docker images were tagged main instead of the version.
• Non-existent environment: Workflows targeted prod resources that had never been provisioned. publish-libs failed with ResourceNotFoundException.

How to run¶

git tag v0.0.0-test
git push origin v0.0.0-test

CI runs first. When it passes, three workflows trigger automatically via workflow_run.

What to check¶

Open the Actions tab and verify each workflow:

publish-libs.yml:

deploy-lambdas.yml:

docker-build.yml:

What gets created in AWS¶

Cleanup¶

Delete the test git tag only:

git push origin --delete v0.0.0-test
git tag -d v0.0.0-test

Cost¶

publish-libs runs on every green dev CI run (#942); docker-build and deploy-lambdas run on every dev push and on v* tags. PRs only run CI with smart change detection. CI compute is RunsOn EC2 in eu-central-1 (spot for CI jobs, on-demand for deploy/release lanes; only glue jobs stay on GitHub-hosted runners) — see CI Runners. A full E2E test tag run takes ~15 minutes wall-clock (jobs run concurrently; aggregate runner-minutes are higher and billed per second by AWS).

Troubleshooting¶

Pipeline Failures¶

CodeArtifact Token Expired¶

# Re-login
just codeartifact-login

# CI: Token is auto-refreshed in pipeline

ECR Push Permission Denied¶

# Check AWS credentials
aws sts get-caller-identity

# Verify ECR permissions
aws ecr describe-repositories --repository-names tradai/$SERVICE

# Required IAM permissions
# ecr:GetAuthorizationToken, ecr:BatchCheckLayerAvailability, ecr:PutImage

Cache Issues¶

# GitHub Actions uses UV cache at ~/.cache/uv
# Cache key includes uv.lock hash -- changing uv.lock invalidates cache automatically

Pulumi State Lock¶

# If Pulumi reports state lock:
pulumi cancel  # Cancel stuck operation
# Or manually unlock in S3

VERSION=main instead of tag¶

In workflow_run context, GITHUB_REF_NAME returns main, not the tag. Workflows must use github.event.workflow_run.head_branch to extract the version. If you see images tagged main instead of a version, check this extraction logic.

Failed Lambda Discovery¶

# Check lambdas directory structure
ls -la lambdas/

# Each lambda needs:
# lambdas/<name>/Dockerfile
# lambdas/<name>/handler.py (or similar entry point)

Related Documentation¶

• Strategy Lifecycle - Full development workflow
• Strategy Repo Guide - Repository setup
• Pulumi Deployment - Infrastructure (ECR, CodeArtifact)
• Pulumi Operations - Day-to-day infrastructure operations