E2E Backtest Pipeline (AWS)¶

Operations guide for testing the full AWS backtest pipeline: building strategy Docker images, pushing to ECR, submitting backtests via the deployed API, and debugging with AWS tools.

Looking for the developer backtest guide?

This page covers the AWS infrastructure pipeline (Docker, ECR, ECS, Step Functions). For everyday backtest development using the CLI, see the Backtesting Guide which covers tradai backtest quick/show/compare/report.

	Backtesting Guide	This Guide
Audience	Developers doing local dev	DevOps testing deployed AWS
Commands	`tradai backtest quick/show/compare`	`curl`, `aws` CLI, `docker build/push`
Environment	Local or Docker Compose	Deployed AWS dev/staging/prod
Orchestration	Direct Freqtrade or Backend API	SQS, Lambda, Step Functions, ECS Fargate
Auth	None (localhost)	AWS SSO
Debugging	CLI output, local logs	CloudWatch, Step Functions, SSM

Prerequisites¶

AWS CLI installed, SSO profile tradai configured
Docker Desktop running (for building strategy images)
Access to tradai-strategies repo (for strategy source code)
UV installed (for building tradai library wheels)

Architecture Overview¶

graph TD
    A["POST /api/v1/backtests<br/>ALB → Backend on EC2"] --> B["SQS FIFO queue<br/>tradai-backtest-queue-dev.fifo"]
    B --> C["Lambda: sqs-consumer → Step Functions"]
    C --> D["Lambda: validate-strategy"]
    D --> E["Lambda: data-collection-proxy<br/>EnsureData from ArcticDB"]
    E --> F["Lambda: update-status<br/>DynamoDB → RUNNING"]
    F --> G["ECS Fargate: strategy container<br/>freqtrade backtest"]
    G --> H["Lambda: update-status<br/>DynamoDB → COMPLETED"]
    H --> I["Lambda: notify-completion (SNS)"]
    I --> J["GET /api/v1/backtests/{id}<br/>results"]
    I --> K["GET /api/v1/backtests/{id}/equity<br/>equity curve"]

Quick Start (Existing Strategy Image in ECR)¶

If a strategy image is already pushed to ECR, you only need steps 1 and 5.

1. Authenticate to AWS¶

aws sso login --profile tradai

2. Submit Backtest¶

ALB_URL="http://tradai-dev-1942285475.eu-central-1.elb.amazonaws.com"

curl -s -X POST $ALB_URL/api/v1/backtests \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "strategy": "StochRsiStrategy",
      "symbols": ["BTC/USDT:USDT"],
      "timeframe": "1h",
      "start_date": "2025-01-01",
      "end_date": "2025-02-01",
      "stake_amount": 1000.0,
      "stake_currency": "USDT",
      "exchange": "binance_futures",
      "task_definition": "tradai-strategy-stochrsi-dev"
    },
    "experiment_name": "my-backtest-experiment",
    "priority": 5
  }'

Save the job_id from the response.

3. Monitor Execution¶

# Check Step Functions
aws stepfunctions list-executions \
  --state-machine-arn "arn:aws:states:eu-central-1:600802701449:stateMachine:tradai-backtest-workflow-dev" \
  --max-results 1 --profile tradai --region eu-central-1

# Check DynamoDB status
aws dynamodb get-item \
  --table-name tradai-workflow-state-dev \
  --key '{"run_id":{"S":"<JOB_ID>"}}' \
  --profile tradai --region eu-central-1 \
  --query 'Item.{status:status.S,error:error_message.S}'

4. Retrieve Results¶

# Full results
curl -s $ALB_URL/api/v1/backtests/<JOB_ID> | python -m json.tool

# Equity curve
curl -s $ALB_URL/api/v1/backtests/<JOB_ID>/equity | python -m json.tool

# List all backtests
curl -s $ALB_URL/api/v1/backtests | python -m json.tool

Building and Deploying a New Strategy Image¶

When you need to test a strategy that doesn't have an ECR image yet.

Step 1: Build Tradai Library Wheels¶

cd /path/to/tradai
just build-libs

This produces wheels in dist/: - tradai_common-0.1.0-py3-none-any.whl - tradai_data-0.1.0-py3-none-any.whl - tradai_strategy-1.0.0-py3-none-any.whl

Step 2: Copy Wheels to Strategies Repo¶

mkdir -p /path/to/strategies/dist
cp /path/to/tradai/dist/tradai_*.whl /path/to/strategies/dist/

Step 3: Build Strategy Docker Image¶

cd /path/to/strategies

docker build \
  --build-arg USE_LOCAL_WHEELS=true \
  --platform linux/amd64 \
  -t tradai/stochrsistrategy:latest \
  -f strategies/stochrsi-strategy/Dockerfile .

Replace stochrsi-strategy and image name for other strategies.

Step 4: Test Locally¶

docker run --rm --platform linux/amd64 \
  -e JOB_ID=test-local-001 \
  -e TRACE_ID=trace-local-001 \
  -e STRATEGY=StochRsiStrategy \
  -e TIMEFRAME=1h \
  -e TIMERANGE=20231201-20240107 \
  -e PAIRS=BTC/USDT:USDT \
  -e TRADING_MODE=backtest \
  -e STAKE_AMOUNT=1000 \
  -e EXPERIMENT_NAME=local-test \
  -e CALLBACK_ENABLED=false \
  -v "./user_data/data:/freqtrade/user_data/data:ro" \
  tradai/stochrsistrategy:latest

Verify: container outputs JSON BacktestResult and exits 0.

Note: Local test requires mounted data files. In AWS, the EnsureData step fetches data from ArcticDB automatically.

Step 5: Push to ECR¶

# Login to ECR
aws ecr get-login-password --region eu-central-1 --profile tradai | \
  docker login --username AWS --password-stdin \
  600802701449.dkr.ecr.eu-central-1.amazonaws.com

# Create ECR repo (first time only)
aws ecr create-repository \
  --repository-name tradai/stochrsistrategy \
  --profile tradai --region eu-central-1

# Tag and push
docker tag tradai/stochrsistrategy:latest \
  600802701449.dkr.ecr.eu-central-1.amazonaws.com/tradai/stochrsistrategy:latest

docker push \
  600802701449.dkr.ecr.eu-central-1.amazonaws.com/tradai/stochrsistrategy:latest

Step 6: Register ECS Task Definition (First Time Only)¶

aws ecs register-task-definition \
  --family tradai-strategy-stochrsi-dev \
  --cpu 1024 --memory 2048 \
  --network-mode awsvpc \
  --requires-compatibilities FARGATE \
  --execution-role-arn "arn:aws:iam::600802701449:role/tradai-ecs-execution-dev" \
  --task-role-arn "arn:aws:iam::600802701449:role/tradai-ecs-task-dev" \
  --container-definitions '[{
    "name": "strategy",
    "image": "600802701449.dkr.ecr.eu-central-1.amazonaws.com/tradai/stochrsistrategy:latest",
    "cpu": 1024, "memory": 2048, "essential": true,
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/aws/ecs/tradai-dev",
        "awslogs-region": "eu-central-1",
        "awslogs-stream-prefix": "strategy"
      }
    },
    "environment": [{"name": "TRADING_MODE", "value": "backtest"}]
  }]' \
  --profile tradai --region eu-central-1

For subsequent image updates, just push the new image — the task definition uses :latest tag.

Step 7: Submit Backtest¶

Use the task_definition field matching the family name:

curl -s -X POST $ALB_URL/api/v1/backtests \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "strategy": "StochRsiStrategy",
      "symbols": ["BTC/USDT:USDT"],
      "timeframe": "1h",
      "start_date": "2025-01-01",
      "end_date": "2025-02-01",
      "task_definition": "tradai-strategy-stochrsi-dev"
    },
    "experiment_name": "my-experiment"
  }'

Debugging¶

Check ECS Container Logs¶

# Find the task ID from Step Functions execution or DynamoDB
MSYS_NO_PATHCONV=1 aws logs get-log-events \
  --log-group-name "/aws/ecs/tradai-dev" \
  --log-stream-name "strategy/strategy/<TASK_ID>" \
  --profile tradai --region eu-central-1 \
  --query 'events[*].message' --output json

Check Step Functions Execution History¶

aws stepfunctions get-execution-history \
  --execution-arn "arn:aws:states:eu-central-1:600802701449:execution:tradai-backtest-workflow-dev:backtest-<JOB_ID>" \
  --profile tradai --region eu-central-1 \
  --query 'events[*].[timestamp,type,stateEnteredEventDetails.name]' \
  --output table

Check Backend Health¶

curl -s http://tradai-dev-1942285475.eu-central-1.elb.amazonaws.com/api/v1/health | python -m json.tool

Check Services on EC2¶

aws ssm send-command \
  --instance-ids i-09ff8e414fe16f7e7 \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["docker ps --format \"table {{.Names}}\t{{.Status}}\""]' \
  --profile tradai --region eu-central-1

# Get output (replace COMMAND_ID)
aws ssm get-command-invocation \
  --command-id <COMMAND_ID> \
  --instance-id i-09ff8e414fe16f7e7 \
  --profile tradai --region eu-central-1 \
  --query StandardOutputContent --output text

Restart a Service on EC2¶

aws ssm send-command \
  --instance-ids i-09ff8e414fe16f7e7 \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["cd /opt/tradai && docker-compose restart <SERVICE_NAME>"]' \
  --profile tradai --region eu-central-1

Service names: backend_api, mlflow, data_collection, strategy_service

Troubleshooting¶

"Invalid parameter for query_by_status: index not found"¶

DynamoDB GSI name mismatch. Check that code uses status-created_at-index (not status-updated_at-index). See PR #295.

"Access denied to state machine"¶

EC2 role needs states:StartExecution on the backtest workflow state machine ARN.

"ResourceInitializationError: log group does not exist"¶

Create the log group:

MSYS_NO_PATHCONV=1 aws logs create-log-group \
  --log-group-name "/aws/ecs/tradai-dev" \
  --profile tradai --region eu-central-1

"TaskDefinition not found"¶

The task_definition in the request must match a registered ECS task definition family name. Register one per Step 6 above.

"Impossible to load Strategy"¶

The strategy container uses bare freqtradeorg/freqtrade:stable without strategy code. Build a strategy-specific image per steps 1-6.

"MLflow API error: 503"¶

MLflow may be running on SQLite fallback. Check:

# On EC2 via SSM
docker logs mlflow --tail 5
# Look for "Backend store: sqlite://..." vs "postgresql://..."

Fix: restart MLflow container to re-fetch RDS password from Secrets Manager.

"No data found" in ECS Container¶

The EnsureData step should populate ArcticDB. If it fails, check: - Lambda data-collection-proxy logs - ArcticDB S3 bucket tradai-arcticdb-dev - Data-collection service health

Runner Scripts¶

Convenience scripts to run E2E backtests from your local machine without manual curl commands.

First Time Setup¶

Configure the AWS SSO profile:

aws configure sso --profile tradai

Use these values:

SSO start URL:  https://d-99675533ed.awsapps.com/start
SSO Region:     eu-central-1
Account:        600802701449 (dev)
Default region: eu-central-1
Output:         json

Running the Script¶

# macOS / Linux
chmod +x scripts/e2e-backtest.sh
./scripts/e2e-backtest.sh

# Windows
scripts\e2e-backtest.bat

# Custom strategy and date range
./scripts/e2e-backtest.sh StochRsiStrategy 2025-03-01 2025-03-31

Arguments: e2e-backtest[.sh|.bat] [strategy] [start_date] [end_date]

What the Script Does¶

Checks AWS SSO authentication (logs in if session expired)
Verifies backend API is reachable
Submits backtest via POST /api/v1/backtests
Polls for completion every 10 seconds (5 min timeout)
Prints result summary: trades, win rate, profit, Sharpe ratio, max drawdown

Typical run takes about 60 seconds.

Output Example¶

========================================
 TradAI E2E Backtest Runner
========================================
 Strategy:   StochRsiStrategy
 Symbol:     BTC/USDT:USDT
 Timeframe:  1h
 Range:      2025-01-01 to 2025-02-01
========================================

[1/4] Checking AWS authentication...  OK
[2/4] Checking backend health...  OK
[3/4] Submitting backtest...
 Submitted! Job ID: 282f7b79-...

[4/4] Polling for results (timeout: 300s)...
 [20s] Status: running...
 COMPLETED in 60s

========================================
 RESULTS
========================================
  Trades: 73 (W:43 L:30)
  Win Rate: 58.9%
  Profit: -4.83%
  Sharpe: -18.27
  Max DD: $501.85

AWS Resources Reference¶

Resource	ARN / URL
ALB	`tradai-dev-1942285475.eu-central-1.elb.amazonaws.com`
EC2	`i-09ff8e414fe16f7e7` (tradai-consolidated-dev)
ECS Cluster	`tradai-dev`
SQS Queue	`tradai-backtest-queue-dev.fifo`
Step Functions	`tradai-backtest-workflow-dev`
DynamoDB Table	`tradai-workflow-state-dev`
S3 Results	`tradai-results-dev`
S3 ArcticDB	`tradai-arcticdb-dev`
S3 MLflow	`tradai-mlflow-dev`
RDS MLflow	`tradai-mlflow-dev.crcc44o2kjg3.eu-central-1.rds.amazonaws.com`
Cloud Map	`tradai-dev.local`
ECS Execution Role	`tradai-ecs-execution-dev`
ECS Task Role	`tradai-ecs-task-dev`
EC2 Role	`tradai-consolidated-dev`