TradAI Error Handling & Resilience¶

Version: 1.0.0 | Date: 2026-03-28 | Status: CURRENT

1. TL;DR¶

TradAI uses a layered error handling strategy spanning application code, AWS infrastructure, and workflow orchestration:

• Application layer -- with_retry decorator with exponential backoff and configurable RetryConfig (max attempts, backoff factor, delay caps). A CircuitBreaker prevents cascading failures by transitioning through CLOSED / OPEN / HALF_OPEN states.
• Unified policy -- ResiliencePolicy combines retry + circuit breaker into a single decorator (@policy.protect) with presets for HTTP, database, and external API use cases.
• SQS + DLQ -- FIFO queues with maxReceiveCount: 3 and 14-day DLQ retention. Lambda consumers return batchItemFailures for partial-batch retry.
• Step Functions -- Every state has Retry blocks for transient Lambda errors and Catch blocks that route to UpdateStatusFailed / CleanupResources / NotifyFailure.
• Exception hierarchy -- All errors descend from TradAIError. Retriable exceptions (ExternalServiceError and subclasses) are separated from non-retriable ones (ValidationError, NotFoundError).

2. Retry Patterns¶

2.1 Decision Flow¶

flowchart TD
    A[Call function] --> B{Exception raised?}
    B -- No --> C[Return result]
    B -- Yes --> D{Is exception retriable?}
    D -- No --> E[Raise immediately]
    D -- Yes --> F{Attempts remaining?}
    F -- No --> G[Log exhaustion & raise last exception]
    F -- Yes --> H[Call on_retry callback]
    H --> I[Log warning with attempt/delay]
    I --> J[Sleep: delay * backoff_factor]
    J --> K[Cap delay at max_delay]
    K --> A

2.2 RetryConfig Fields¶

2.3 Decorator API¶

from tradai.common.resilience import with_retry, with_retry_async

@with_retry(max_attempts=3, retriable_exceptions=(ConnectionError,))
def fetch_data():
    return requests.get("https://api.example.com/data")

@with_retry_async(max_attempts=5, initial_delay=0.5, max_delay=10.0)
async def fetch_data_async():
    return await client.get("https://api.example.com/data")

2.4 Manual Retry Control¶

For scenarios requiring fine-grained control, use RetryContext:

from tradai.common.resilience import RetryContext

with RetryContext(max_attempts=3) as retry:
    for attempt in retry:
        try:
            result = fetch_data()
            break
        except ConnectionError as e:
            if not retry.should_retry(e):
                raise

3. Circuit Breaker¶

3.1 State Machine¶

stateDiagram-v2
    [*] --> CLOSED

    CLOSED --> OPEN : failure_count >= failure_threshold (default 5)
    OPEN --> HALF_OPEN : recovery_timeout elapsed (default 60s)
    HALF_OPEN --> CLOSED : success_count >= success_threshold (default 1)
    HALF_OPEN --> OPEN : Any failure

    CLOSED --> CLOSED : Success (resets failure_count)
    OPEN --> OPEN : Requests rejected (CircuitOpenError)

3.2 CircuitBreakerConfig¶

3.3 Recorded Exceptions¶

The circuit breaker only records transient I/O exceptions by default:

• ConnectionError
• TimeoutError
• OSError

Programming errors (TypeError, ValueError) pass through without tripping the breaker.

3.4 Thread Safety¶

All state transitions are protected by threading.Lock. The state property checks for timeout-based OPEN to HALF_OPEN transitions on every access, so recovery is automatic.

4. Unified Resilience Policy¶

ResiliencePolicy combines retry and circuit breaker into a single configuration with preset profiles:

4.1 Presets¶

4.2 Execution Order¶

Circuit breaker check runs first (fail fast), then retry logic wraps the call:

1. Check circuit breaker -> if OPEN, raise CircuitOpenError
2. Execute function
3. On success -> record_success() on circuit breaker
4. On retriable failure -> record_failure() + retry with backoff
5. On retry exhaustion -> raise last exception

from tradai.common.resilience import ResiliencePolicy, ResilienceConfig

policy = ResiliencePolicy(ResilienceConfig.for_external_api(), name="binance")

@policy.protect
def fetch_ohlcv(symbol: str) -> dict:
    return exchange.fetch_ohlcv(symbol)

5. DLQ Handling¶

5.1 SQS Queue Configuration¶

5.2 Message Lifecycle¶

sequenceDiagram
    participant API as Backend API
    participant SQS as SQS FIFO Queue
    participant Lambda as Backtest Consumer
    participant DLQ as Dead Letter Queue
    participant Ops as Operator

    API->>SQS: Send backtest request
    SQS->>Lambda: Deliver batch of messages

    alt Processing succeeds
        Lambda->>Lambda: SQSJobProcessor.process_batch()
        Lambda-->>SQS: Message deleted (ack)
    else Individual message fails
        Lambda-->>SQS: Return batchItemFailures
        Note over SQS: Message becomes visible again<br/>after 900s timeout
        SQS->>Lambda: Re-deliver (attempt 2 of 3)
    else All 3 attempts exhausted
        SQS->>DLQ: Move message to DLQ<br/>(14-day retention)
        Ops->>DLQ: Inspect failed messages
        Ops->>SQS: Replay after fixing root cause
    end

5.3 Partial Batch Failure¶

The backtest-consumer Lambda uses SQS partial batch failure reporting. On failure, it returns specific messageId values in batchItemFailures, so only failed messages are retried -- successful ones in the same batch are not reprocessed.

# From lambdas/backtest-consumer/handler.py
return {
    "batchItemFailures": [
        {"itemIdentifier": mid} for mid in result.failed_message_ids
    ]
}

6. Step Functions Error Handling¶

6.1 Retry Blocks (ASL)¶

Every Lambda invocation in the backtest workflow includes a Retry block for transient AWS errors:

6.2 Catch Blocks and Error Routing¶

ValidateStrategy ──Catch(States.ALL)──> UpdateStatusFailed
EnsureData ──Catch(States.ALL)──> UpdateStatusFailed
RunBacktest ──Catch(States.TaskFailed)──> CleanupResources
RunBacktest ──Catch(States.Timeout)──> HandleTimeout ──> CleanupResources
RunBacktest ──Catch(States.ALL)──> CleanupResources
CleanupResources ──────────────────────> UpdateStatusFailed
UpdateStatusFailed ──────────────────> NotifyFailure
UpdateStatusFailed ──Catch(States.ALL)──> NotifyFailure  (continue even if status update fails)

6.3 Timeout Handling¶

6.4 Non-Critical Failure Tolerance¶

The UpdateStatusRunning state has a special Catch block with "Comment": "Continue even if status update fails". If the DynamoDB status update fails, the workflow proceeds to RunBacktest anyway -- the ECS container will update status itself. This prevents transient DynamoDB errors from blocking the entire backtest.

7. Failure Modes Table¶

8. Exception Hierarchy¶

TradAIError (base)
+-- ValidationError                    # Non-retriable: bad input
+-- FeatureValidationError             # Non-retriable: ML schema mismatch
+-- NotFoundError                      # Non-retriable: missing resource
|   +-- DataNotFoundError              # Missing data in storage
+-- ConfigurationError                 # Non-retriable: bad config
+-- ConflictError                      # Non-retriable: state conflict
+-- AuthenticationError                # Non-retriable: auth failure
+-- BacktestError                      # Non-retriable: execution failure
+-- TradingError                       # Non-retriable: live trading failure
+-- ExternalServiceError               # RETRIABLE: external dependency
    +-- DataFetchError                 # Exchange / storage read failure
    +-- StorageError                   # Storage write failure
    +-- CircuitOpenError               # Circuit breaker is open
    +-- WebSocketError                 # WebSocket base
        +-- StreamDisconnectedError    # Connection lost
        +-- StreamReconnectExhaustedError  # Max reconnects exceeded

Changelog¶

Dependencies¶