retraining-scheduler¶

Schedules and triggers model retraining based on drift detection, scheduled intervals, or manual requests.

Overview¶

Input Schema¶

{
    "models": [                         # Optional, uses DEFAULT_MODELS if not provided
        {
            "name": "E2ETestStrategy",
            "strategy": "E2ETestStrategy",
            "freqai_model": "LightGBMRegressor",
            "train_period_days": 30,
            "pairs": ["BTC/USDT:USDT", "ETH/USDT:USDT"],
            "timeframe": "1h"
        }
    ],
    "force": false,                     # Force retraining regardless of state
    "trigger": "manual"                 # Optional: override trigger type
}

Output Schema¶

{
    "success": true,
    "data": {
        "summary": {
            "models_evaluated": 2,
            "retraining_triggered": 1,
            "skipped": 1,
            "errors": 0
        },
        "results": [
            {
                "model_name": "E2ETestStrategy",
                "status": "triggered",
                "trigger": "drift_detected",
                "task_arn": "arn:aws:ecs:...:task/abc123",
                "retraining_triggered": true,
                "timestamp": "2024-01-01T12:00:00+00:00"
            },
            {
                "model_name": "RadStrategy",
                "status": "skipped",
                "reason": "recently_retrained",
                "hours_since_last": 12.5,
                "retraining_triggered": false,
                "timestamp": "2024-01-01T12:00:00+00:00"
            }
        ]
    },
    "environment": "dev"
}

Environment Variables¶

Trigger Types¶

Uses RetrainingTrigger enum from tradai.common.entities.retraining:

Retraining Decision Flow¶

flowchart TD
    A[Evaluate Model] --> B{Force flag?}
    B -->|Yes| C[Trigger: manual]
    B -->|No| D{Recently retrained?}
    D -->|Yes, within MIN_HOURS| E[Skip: cooldown]
    D -->|No| F{Drift detected in DynamoDB?}
    F -->|Yes, is_drifted=true| G[Trigger: drift_detected]
    F -->|No| H{Scheduled interval due?}
    H -->|Yes, days >= RETRAINING_INTERVAL_DAYS| I[Trigger: scheduled]
    H -->|No| J[Skip: no trigger]
    C --> K[Launch ECS Task]
    G --> K
    I --> K
    K --> L[Update Retraining State in DynamoDB]
    L --> M{Drift triggered?}
    M -->|Yes| N[Reset drift state: is_drifted=false]
    M -->|No| O[Send Notification]
    N --> O

ECS Task Configuration¶

The Lambda launches Fargate tasks with these container environment overrides:

There is no per-field env any more. The dual-write was removed at cutover step 6 (#754 P2), once the canary proved reader-capable images were deployed. STRATEGY, MODEL_NAME, FREQAI_MODEL, TRAIN_PERIOD_DAYS, BACKTEST_PERIOD_DAYS, TRAINING_TIMEOUT_HOURS, PAIRS and TIMEFRAME are no longer emitted — the blob is the sole carrier, and a container that cannot parse it fails at launch rather than booting on defaults.

Two consequences worth knowing:

• The FreqAI model range is no longer pinned to three. While the dual-write stood, any model outside the old reader's 3-item allowlist was refused pre-lock, because emitting it would brick an image at settings construction. Every model FreqAIModelRegistry knows — get_all_known_models(), i.e. the Freqtrade built-ins plus the TradAI ones — is launchable now. _assert_known_freqai_model still rejects anything outside that set before the retraining lock is taken: nothing downstream validates the name (is_valid_model returns True on every path, by design), so an unverifiable value would otherwise fail inside the container with the lock already held. A genuinely custom model class is therefore not launchable via the scheduler — the Lambda cannot import it to check.
• A stale task-def baseline is now refused. The cutover's fail-loud property was real but accidental: a pre-reader image ignores RUN_CONFIG entirely and fails only because its _build_config raises on a missing PAIRS. Since ECS merges containerOverrides additively onto the task-def baseline, a revision that still bakes PAIRS in would let that image boot on stale values and silently train the wrong universe. _assert_no_legacy_ training_baseline refuses such a revision pre-lock. STRATEGY is deliberately not in that set — build_strategy_env_vars writes it on every live/dry-run deploy. Verified 2026-08-22: none of dev's three strategy families carries any of the gated keys, so the guard is a no-op today; it exists so the property stays true when a rollback or repin brings a stale revision back.

• The combined-override guard has slack it cannot use. _assert_override_budget still measures the real wire payload against ECS's 8192-char cap, but run_config's own 5000-char blob cap now binds first — roughly 245 pairs and ~5.8k chars combined for a typical strategy name, well under 8192. (Treat those as a measurement, not an invariant: both shift with the strategy/model name lengths and the container name.) The guard is kept as defence-in-depth for env vars added later.

• Task definition: resolved via get_strategy_task_definition(deployment_identity, environment) → tradai-strategy-{slug}-{env}. An explicit task_definition key on the model config overrides it. The family is pre-flighted with DescribeTaskDefinition before the retraining lock is taken, so a missing family fails with an actionable message rather than an opaque ECS error — the check fails open if the IAM grant is absent.

• Capacity provider: FARGATE_SPOT (weight=1, base=0) + FARGATE (weight=0) if USE_SPOT=true, otherwise launchType=FARGATE
• No command override: uses container ENTRYPOINT from image

DynamoDB State Management¶

Retraining State Table¶

Records retraining job state with 30-day TTL:

{
    "model_name": "E2ETestStrategy",     # Partition key
    "last_retraining": "2024-01-01T12:00:00+00:00",
    "task_arn": "arn:aws:ecs:...",
    "trigger": "drift_detected",
    "status": "running",
    "expires_at": 1706745600            # Unix timestamp, 30 days TTL
}

Drift State Table¶

Checked for is_drifted flag. Reset after drift-triggered retraining:

# Reset operation
table.update_item(
    Key={"model_name": model_name},
    UpdateExpression="SET is_drifted = :val, reset_at = :ts",
    ...
)

CloudWatch Metrics¶

Namespace suffix: ModelRetraining

EventBridge Schedule¶

{
  "ScheduleExpression": "rate(6 hours)",
  "Targets": [{
    "Arn": "arn:aws:lambda:...:retraining-scheduler",
    "Input": "{\"models\": [{\"name\": \"E2ETestStrategy\", \"strategy\": \"E2ETestStrategy\"}]}"
  }]
}

SNS Notification Format¶

TradAI Model Retraining Notification

Environment: prod
Model: E2ETestStrategy
Trigger: Drift Detected
Task ARN: arn:aws:ecs:...:task/abc123
Timestamp: 2024-01-01T12:00:00+00:00

A model retraining job has been triggered. You will receive another
notification when the training completes.

Reason: Significant drift was detected in model predictions.

See Also¶

Related Lambdas:

• drift-monitor - Detects model drift (triggers retraining)
• compare-models - Champion vs challenger comparison

Architecture:

• ML Lifecycle - Full ML training pipeline
• ML Lifecycle - Pipeline diagram

Services:

• Strategy Service - Model registry integration

CLI:

• CLI Reference - A/B testing commands