Skip to content

compare-models

Compares champion (Production) vs challenger model versions to determine if promotion is recommended.

Overview

Property Value
Trigger Step Functions state machine
Runtime Python 3.11
Timeout 120 seconds
Memory 256 MB

Input Schema

{
    "model_name": "PascalStrategy",     # Required
    "challenger_run_id": "abc123",      # Optional, reserved for future use
    "champion_version": "Production",   # Default: "Production"
    "challenger_version": "Staging"     # Default: "Staging"
}

Output Schema

{
    "decision": "PROMOTE",           # PROMOTE | KEEP | INVESTIGATE
    "confidence": 0.85,              # 0.0 - 1.0
    "profit_improvement": 15.5,      # Percentage improvement
    "recommendation": "Challenger shows 15.5% profit improvement with high confidence",
    "champion_version": "3",
    "challenger_version": "4"
}

Environment Variables

Variable Required Default Description
MLFLOW_TRACKING_URI Yes - MLflow tracking server URL
MLFLOW_TRACKING_USERNAME No - MLflow username
MLFLOW_TRACKING_PASSWORD No - MLflow password
MIN_CONFIDENCE_THRESHOLD No 0.7 Confidence threshold for decisions

Decision Logic

The ModelComparator from tradai-common evaluates:

Metric Weight Threshold
profit_total High Challenger > Champion
sharpe_ratio High Challenger >= Champion
max_drawdown Medium Challenger <= Champion
win_rate Low Challenger >= 0.9 * Champion

Decisions: - PROMOTE: Challenger is significantly better (high confidence) - KEEP: Champion is better or similar - INVESTIGATE: Mixed results, manual review needed

Key Features

  • Uses ModelComparator from tradai-common for consistent comparison logic
  • Calculates profit improvement between versions
  • Returns confidence score for decision validation
  • Defaults to KEEP on errors to prevent risky promotions

Step Functions Integration

{
  "CompareModels": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:...:compare-models",
    "Parameters": {
      "model_name.$": "$.model_name"
    },
    "Next": "EvaluateDecision"
  },
  "EvaluateDecision": {
    "Type": "Choice",
    "Choices": [
      {
        "Variable": "$.decision",
        "StringEquals": "PROMOTE",
        "Next": "PromoteModel"
      },
      {
        "Variable": "$.decision",
        "StringEquals": "INVESTIGATE",
        "Next": "NotifyForReview"
      }
    ],
    "Default": "KeepChampion"
  }
}

See Also

Related Lambdas:

Architecture:

SDK:

Services: