TradAI Final Architecture - Step Functions Workflows¶

Version: 11.1.0 | Date: 2026-03-28 | Status: CURRENT

Last synced with: infra/compute/asl_templates/backtest_workflow.json.j2, infra/compute/asl_templates/retraining_workflow.json.j2

TL;DR: 2 Step Functions workflows: Backtest (12 states, 7200s timeout) and Retraining (10 states). Both use Jinja2 ASL templates with Lambda ARN injection. DLQ handling is at SQS level, not workflow level.

1. When to Use Step Functions¶

Step Functions is one of four execution modes for backtesting (see 05-SERVICES.md Section 4).

Use Step Functions when you need: - Data freshness validation before backtest - Strategy validation (ECR/S3 checks) - Multi-step workflows (data sync → backtest → analyze) - Visual debugging and execution history - Complex retry/catch logic

Skip Step Functions when: - Simple backtest execution (use ECS direct or SQS mode) - Development/testing (use local mode) - Container handles its own status updates (v9.2 architecture)

2. Workflow Overview¶

Critical: All workflows use STANDARD type (not EXPRESS) because: - Backtests can run 30-60+ minutes - EXPRESS has 5-minute maximum duration - STANDARD provides execution history for 90 days

3. Backtest Workflow (v11)¶

The v11 workflow replaced the earlier v8.0 "full" and v9.2 "simplified" workflows with a single, unified workflow. Key changes from earlier versions:

• Removed PrepareConfig (ECS Task) — task definition resolved dynamically from $.task_definition
• Removed TransformResults (Lambda) — container handles DynamoDB/MLflow/S3 directly
• Renamed CheckDataFreshness → EnsureData — now does DB-first coverage check, fetches only gaps
• Added UpdateStatusRunning — explicitly sets RUNNING before backtest via update-status Lambda
• Split UpdateStatus → UpdateStatusCompleted / UpdateStatusFailed / UpdateStatusValidationFailed
• Split HandleError → HandleTimeout (Pass) + CleanupResources (Lambda)
• Added HandleSuccess as a Parallel state (UpdateStatus + Notify run concurrently)

Workflow Diagram¶

Mermaid Diagram¶

stateDiagram-v2
    [*] --> ValidateStrategy
    ValidateStrategy --> EvaluateValidation

    state EvaluateValidation <<choice>>
    EvaluateValidation --> EnsureData: Valid
    EvaluateValidation --> UpdateStatusValidationFailed: Invalid

    UpdateStatusValidationFailed --> FailValidation
    FailValidation --> [*]

    EnsureData --> UpdateStatusRunning
    UpdateStatusRunning --> RunBacktest

    state RunBacktest_result <<choice>>
    RunBacktest --> RunBacktest_result
    RunBacktest_result --> HandleSuccess: Success
    RunBacktest_result --> HandleTimeout: Timeout
    RunBacktest_result --> CleanupResources: TaskFailed

    HandleTimeout --> CleanupResources
    CleanupResources --> UpdateStatusFailed
    UpdateStatusFailed --> NotifyFailure
    NotifyFailure --> [*]

    state HandleSuccess {
        [*] --> UpdateStatusCompleted
        [*] --> NotifySuccess
    }
    HandleSuccess --> [*]

State Reference¶

State Machine Definition¶

See the canonical ASL template at infra/compute/asl_templates/backtest_workflow.json.j2.

Key configuration parameters (Jinja2 variables):

| lambda_arns.validate_strategy | ValidateStrategy Lambda ARN | | lambda_arns.data_collection_proxy | EnsureData Lambda ARN | | lambda_arns.update_status | UpdateStatus Lambda ARN | | lambda_arns.cleanup_resources | CleanupResources Lambda ARN | | lambda_arns.notify_completion | NotifyCompletion Lambda ARN | | ecs.cluster_arn | ECS cluster ARN | | ecs.subnets | Private subnet list | | ecs.security_group_id | ECS security group | | dynamodb_table_name | Workflow state table | | mlflow_tracking_uri | MLflow tracking URI | | arctic_bucket | S3 bucket for ArcticDB data | | arctic_library | ArcticDB library name (default: ohlcv) |

Key Design Decisions (v11)¶

1. Sequential validation — ValidateStrategy runs first, then EnsureData. Earlier versions ran these in parallel, but v11 validates first to fail fast before expensive data operations.

2. Dynamic task definition — TaskDefinition.$: "$.task_definition" is resolved from execution input, not hardcoded. This allows different strategy containers per execution.

3. Explicit status updates via Lambda — Uses update-status Lambda instead of direct DynamoDB integration. This enforces state transition guards and publishes CloudWatch metrics.

4. Separate timeout handling — HandleTimeout (Pass state) formats the error before routing to CleanupResources, keeping timeout and task-failure paths distinct.

5. Parallel success handling — HandleSuccess runs UpdateStatusCompleted and NotifySuccess concurrently with independent catch-all states, so a notification failure doesn't block status updates.

Error Handling (RunBacktest)¶

RunBacktest uses Catch blocks only (no Retry) for ECS task errors:

Lambda states (ValidateStrategy, EnsureData) have Retry blocks for transient AWS errors:

Expected Flow (Happy Path)¶

1. ValidateStrategy
   └─ Output: {valid: true, resolved_version: "1.2.0"}

2. EvaluateValidation → EnsureData

3. EnsureData
   └─ Output: {coverage: {BTC/USDT: 100%}, gaps_filled: 0}

4. UpdateStatusRunning
   └─ Sets job status to RUNNING in DynamoDB

5. RunBacktest
   └─ Duration: 10-30 minutes
   └─ Container handles MLflow logging, S3 upload

6. HandleSuccess (Parallel)
   ├─ UpdateStatusCompleted → DynamoDB updated to COMPLETED
   └─ NotifySuccess → SNS notification sent

Version History¶

4. Retraining Workflow (MO005)¶

The retraining workflow orchestrates ML model retraining, comparing the challenger model against the champion, and optionally promoting the new version.

See the canonical ASL template at infra/compute/asl_templates/retraining_workflow.json.j2.

Workflow Diagram¶

stateDiagram-v2
    [*] --> CheckRetrainingNeeded

    CheckRetrainingNeeded --> EvaluateRetrainingNeed
    CheckRetrainingNeeded --> NotifyFailure: States.ALL (Catch)

    state EvaluateRetrainingNeed <<choice>>
    EvaluateRetrainingNeed --> RunRetraining: needs_retraining
    EvaluateRetrainingNeed --> SkipRetraining: otherwise (default)

    SkipRetraining --> [*]

    RunRetraining --> CompareModels
    RunRetraining --> NotifyFailure: States.ALL (Catch)

    CompareModels --> DecidePromotion
    CompareModels --> NotifyFailure: States.ALL (Catch)

    state DecidePromotion <<choice>>
    DecidePromotion --> PromoteModel: promote
    DecidePromotion --> KeepCurrentModel: otherwise (default)

    PromoteModel --> NotifyCompletion
    PromoteModel --> NotifyFailure: States.ALL (Catch)
    KeepCurrentModel --> NotifyCompletion

    NotifyCompletion --> [*]
    NotifyFailure --> [*]

    note right of CheckRetrainingNeeded: Lambda&#58; check drift state and schedule triggers
    note right of RunRetraining: ECS RunTask (FARGATE_SPOT)<br/>TimeoutSeconds&#58; training_timeout_seconds<br/>HeartbeatSeconds&#58; heartbeat_seconds
    note right of CompareModels: Lambda&#58; champion vs challenger metrics
    note right of PromoteModel: Lambda&#58; promote challenger to Production in MLflow

State Reference¶

Template Variables¶

Error Handling¶

All states use Catch blocks routing to NotifyFailure on States.ALL errors. RunRetraining has no Retry -- ECS task failures go directly to NotifyFailure.

5. CloudWatch Integration¶

Metrics Published¶

Note: CloudWatch alarms for Step Functions are defined in the edge stack (infra/edge/), not in the compute stack.

6. Cost Analysis¶

Step Functions Pricing (Standard)¶

Comparison: EXPRESS vs STANDARD¶

7. Testing Workflows¶

Test Input¶

{
  "run_id": "test-001",
  "trace_id": "trace-test-001",
  "strategy_name": "MomentumStrategy",
  "strategy_version": "1.0.0",
  "task_definition": "arn:aws:ecs:eu-central-1:123456:task-definition/momentum-strategy:5",
  "experiment_name": "test-experiment",
  "symbols": ["BTC/USDT:USDT"],
  "symbols_csv": "BTC/USDT:USDT",
  "exchange": "binance",
  "timeframe": "1h",
  "start_date": "20240101",
  "end_date": "20240601",
  "config_version_id": "abc123"
}

8. Next Steps¶

1. Review 07-COST-ANALYSIS.md for complete cost breakdown
2. Review 09-PULUMI-CODE.md for Step Functions deployment code

Changelog¶

Dependencies¶