TradAI Configuration Management¶

Version: 1.0.0 | Date: 2026-03-28 | Status: CURRENT

1. TL;DR¶

TradAI uses two distinct configuration systems that operate at different lifecycle stages:

• Infrastructure configuration (infra/shared/config.py) defines AWS resources at deployment time via Pulumi. It sets VPC CIDRs, ECS task sizes, S3 bucket names, DynamoDB tables, and Lambda schedules. Values are environment-specific (dev/staging/prod) and resolved at pulumi up.
• Application configuration runs at service startup and runtime. YAML config files live in S3, loaded by S3ConfigRepository, merged via ConfigMergeService (OmegaConf deep merge), and versioned in DynamoDB through ConfigVersionService. Services bind environment variables to Pydantic BaseSettings subclasses with validation, type coercion, and immutability (frozen=True).
• Secrets are stored in AWS Secrets Manager with thread-safe retrieval, never hardcoded. RDS uses managed password rotation.

2. Configuration Systems Overview¶

graph TD
    subgraph "Infrastructure Config (Deploy Time)"
        A[infra/.env] -->|AWS_PROFILE, PULUMI_CONFIG_PASSPHRASE| B[Pulumi CLI]
        C[infra/shared/config.py] -->|VPC, Services, S3, DynamoDB| B
        D[pulumi config set] -->|Overrides| B
        B -->|pulumi up| E[AWS Resources]
    end

    subgraph "Application Config (Runtime)"
        F[S3 Bucket: tradai-configs-ENV] -->|YAML download| G[S3ConfigRepository]
        G -->|raw dict| H[ConfigMergeService]
        I[Runtime Overrides] -->|OmegaConf deep merge| H
        H -->|validated dict| J[StrategyConfigLoader]
        J -->|StrategyConfig entity| K[Services]

        L[.env files] -->|env vars| M[Pydantic BaseSettings]
        M -->|validated, frozen| K

        N[DynamoDB: config-versions] <-->|CRUD| O[ConfigVersionService]
        O -->|versioning, lifecycle| K
    end

    subgraph "Secrets (Runtime)"
        P[AWS Secrets Manager] -->|get_secret| Q[Thread-safe Client]
        Q -->|JSON dict| K
    end

    E -.->|bucket names, table names, ARNs| L

3. Infrastructure Configuration¶

Infrastructure configuration lives in infra/shared/tradai_infra_shared/config.py and is consumed by Pulumi stacks at deployment time.

3.1 Config Structure¶

The file defines constants, typed dictionaries, and helper functions consumed by four Pulumi stacks (persistent, foundation, compute, edge):

3.2 Environment-Specific Behavior¶

Infrastructure config uses ENVIRONMENT = pulumi.get_stack() for conditional logic:

# Example: prod gets higher specs
"desired_count": 1 if ENVIRONMENT != "prod" else 2,
"instance_class": "db.t4g.micro" if ENVIRONMENT != "prod" else "db.t4g.small",
"multi_az": ENVIRONMENT == "prod",

3.3 Pulumi Config Overrides¶

Runtime overrides via pulumi config set:

4. Application Configuration¶

Application configuration is loaded at runtime from S3 and optionally merged with overrides. Three layers work together.

4.1 S3ConfigRepository¶

File: libs/tradai-common/src/tradai/common/aws/s3_config_repository.py

Provides CRUD for YAML config files in S3. Uses OmegaConf for YAML parsing to maintain consistency with Hydra patterns.

Key design decisions:

• Temp file pattern: Downloads to tempfile, parses, then deletes (no local state)
• DI-friendly: Optional client parameter for test injection
• Sanitized errors: User input in error messages passes through sanitize_for_display to prevent XSS

4.2 ConfigMergeService¶

File: libs/tradai-common/src/tradai/common/config/merge.py

Orchestrates config loading, OmegaConf deep merge, and validation.

S3ConfigRepository.download() -> base dict
                                    |
                    OmegaConf.merge(base, overrides)
                                    |
                          validate_config()
                           /              \
                  required_fields     Pydantic schema
                  (presence check)    (type + constraint)
                                    |
                              merged dict

• Deep merge: Nested dictionaries merge recursively; override values take precedence
• Dual validation: Required-field presence check AND optional Pydantic schema validation
• Thread-safe: Lazy repository initialization with double-checked locking
• Convenience method: merge_configs() combines load + merge + validate in one call

4.3 StrategyConfigLoader¶

File: libs/tradai-common/src/tradai/common/config/loader.py

High-level facade wrapping ConfigMergeService with strategy-specific concerns:

StrategyConfig is a frozen Pydantic model with validated fields:

• name (min 1 char), version (semver pattern), timeframe (pattern ^\d+[mhd]$)
• pairs, stake_currency, exchange, parameters, buy_params, sell_params, freqai

MLflow integration uses a table-driven tag extractor pattern (_TAG_EXTRACTORS) to map model version tags to config overrides with type-specific parsers (str, csv_list, int, bool).

4.4 Config Versioning¶

File: libs/tradai-common/src/tradai/common/config/service.py

ConfigVersionService provides content-addressable versioning with DynamoDB persistence:

• Deduplication: SHA256 hash of normalized JSON; identical configs reuse existing versions
• Version IDs: Format v{number}-{hash[:8]} (e.g., v3-a1b2c3d4)
• Atomic activation: Activating a version atomically deprecates the previous active version
• Repository: ConfigVersionRepository uses composite key (strategy_name, config_id) with GSIs for hash lookup and status queries

5. Secret Management¶

File: libs/tradai-common/src/tradai/common/aws/secrets_manager.py

5.1 Secret Retrieval Flow¶

Environment Variable (e.g., MLFLOW_SECRET_NAME)
        |
        v
get_secret(secret_name, client=None)
        |
        v
get_secrets_client()  ──> get_aws_client("secretsmanager")
        |
        v
sm_client.get_secret_value(SecretId=...)
        |
        v
json.loads(response["SecretString"]) -> dict

5.2 Error Handling¶

5.3 Rotation Status¶

get_rotation_status(secret_id) queries Secrets Manager for rotation metadata, returning:

• rotation_enabled (bool), rotation_lambda_arn, rotation_interval_days
• last_rotated_date, next_rotation_date (ISO format strings)

5.4 RDS Managed Passwords¶

RDS credentials use AWS-managed password rotation configured in infrastructure:

# From infra config
RDS_CONFIG = {
    "engine": "postgres",
    "engine_version": "15.13",
    "backup_retention": 7 if ENVIRONMENT != "prod" else 14,
    # Password managed by Secrets Manager with automatic rotation
}

6. Service Settings Pattern¶

File: libs/tradai-common/src/tradai/common/base_settings.py

All services inherit from tradai.common.Settings (which extends Pydantic BaseSettings):

pydantic_settings.BaseSettings
        |
tradai.common.Settings (frozen, extra="ignore")
    - service_name, service_version, host, port, debug, log_level, cors_origins
    - prepare_config() for S3/local config loading
    - from_hydra_cfg() for Hydra integration
        |
    +-- MLflowSettingsMixin (mlflow_tracking_uri, credentials, verify_ssl)
    +-- CognitoSettingsMixin (user_pool_id, client_id, region, auth_enabled)
        |
BackendSettings (env_prefix="BACKEND_")
    - environment, data_collection_url, strategy_service_url
    - executor_mode, ecs_cluster, stepfunctions_state_machine_arn
    - s3_config_bucket, dynamodb_table_name, aws_region
    - Validators: URL format, SQS queue URL, S3 bucket name, ARN format

6.1 Key Design Decisions¶

7. Config Version Lifecycle¶

File: libs/tradai-common/src/tradai/common/entities/config_version.py

stateDiagram-v2
    [*] --> DRAFT: create_version()
    DRAFT --> ACTIVE: activate()
    DRAFT --> DEPRECATED: deprecate()
    ACTIVE --> DEPRECATED: activate(newer) / deprecate()
    DEPRECATED --> [*]: TTL auto-cleanup (90 days)

    note right of DRAFT
        Initial state.
        Content-addressable via SHA256.
        Deduplication prevents duplicate versions.
    end note

    note right of ACTIVE
        Only ONE active version per strategy.
        deployed_at timestamp set automatically.
        Activation atomically deprecates previous.
    end note

    note right of DEPRECATED
        Terminal state.
        deprecated_at + superseded_by recorded.
        DynamoDB TTL auto-deletes after 90 days.
    end note

7.1 ConfigVersion Entity¶

The entity uses DynamoDBSerializableMixin for automatic serialization and the immutable update pattern via with_status().

7.2 DynamoDB Schema¶

8. Environment-Specific Overrides Table¶

Changelog¶

Dependencies¶