Validating Strategies¶

Guide to pre-backtest validation and post-backtest sanity checks.

Overview¶

TradAI provides automatic validation at two stages:

Preflight Validation - Before backtest starts (catches errors early)
Sanity Checks - After backtest completes (warns about suspicious results)

Preflight Validation¶

Preflight validation runs automatically before every backtest (unless skipped).

What's Checked¶

Check	Description
Strategy Class	Valid TradAIStrategy subclass
Metadata	Required fields present and valid
Look-Ahead Bias	Detects `shift(-N)`, global stats, future data access
Data Availability	Data exists for requested symbols and period
Indicator Warmup	`startup_candle_count` sufficient for indicators

Automatic Validation¶

Validation runs automatically with tradai backtest quick:

# Normal execution - validation runs automatically
tradai backtest quick MomentumV2

# Output:
# Preflight Validation
# PASS   Strategy Class    Valid TradAIStrategy
# PASS   Metadata          All required fields present
# WARN   Look-Ahead        strategies/momentum_v2/strategy.py:45: shift(-1) detected
# PASS   Data              BTC/USDT:USDT available
# PASS   Warmup            startup_candle_count=50 sufficient
#
# Preflight passed with 1 warning(s)

Validation Options¶

# Skip validation (not recommended)
tradai backtest quick MomentumV2 --skip-validation

# Strict mode - treat warnings as errors
tradai backtest quick MomentumV2 --strict

Understanding Results¶

Status	Meaning	Action
PASS	Check passed	Continue
WARN	Potential issue	Review, may continue
FAIL	Critical error	Must fix before backtest

Look-Ahead Bias Detection¶

Look-ahead bias is when your strategy "sees the future" during backtesting.

Detected Patterns¶

# BAD: shift(-N) accesses future data
df['signal'] = df['close'].shift(-1) > df['close']  # Uses tomorrow's price!

# BAD: Global statistics include future data
df['zscore'] = (df['close'] - df['close'].mean()) / df['close'].std()

# BAD: iloc[-1] comparison
df['signal'] = df['close'].iloc[-1] > df['close'].iloc[-2]

Correct Patterns¶

# GOOD: shift(1) uses past data
df['signal'] = df['close'] > df['close'].shift(1)  # Compare to yesterday

# GOOD: Rolling statistics (look-back only)
df['zscore'] = (df['close'] - df['close'].rolling(20).mean()) / df['close'].rolling(20).std()

# GOOD: Use expanding window
df['cum_mean'] = df['close'].expanding().mean()

Metadata Validation¶

Strategy metadata is validated for completeness:

class MyStrategy(TradAIStrategy):
    # Required metadata
    STRATEGY_NAME = "MyStrategy"  # Must match class name
    VERSION = "1.0.0"             # Semantic version
    CATEGORY = "momentum"          # Valid category
    TIMEFRAME = "1h"               # Valid timeframe

    # Optional but recommended
    AUTHOR = "Your Name"
    DESCRIPTION = "Strategy description"

Valid Categories¶

trend-following, momentum, mean-reversion, breakout, scalping, grid-trading, dca, arbitrage

Valid Timeframes¶

1m, 5m, 15m, 30m, 1h, 4h, 1d

Warmup Validation¶

Indicators need historical data to initialize. Warmup validation ensures you've set sufficient startup_candle_count.

Example¶

class MyStrategy(TradAIStrategy):
    # If you use EMA(50), you need at least 50 candles
    startup_candle_count = 50  # Validates this is sufficient

    def populate_indicators(self, dataframe, metadata):
        dataframe['ema50'] = ta.EMA(dataframe['close'], 50)
        return dataframe

What Happens Without Warmup¶

WARN   Warmup   startup_candle_count=0, but indicators need ~50
       → First candles may have NaN indicators

Post-Backtest Sanity Checks¶

After backtest completes, sanity checks warn about suspicious results.

Checks Performed¶

Check	Warning Condition	Meaning
Win Rate	> 95%	Likely look-ahead bias
Returns	> 500%	Unrealistic, check for bugs
Zero Trades	0 trades	Strategy never triggered
Negative Sharpe	Sharpe < 0 with positive returns	High volatility
Profit Factor	> 10	Small sample or overfitting
Drawdown	< -50%	Extremely risky

Example Output¶

Quick backtest: MomentumV2 on BTC/USDT:USDT (30d)
✓ Completed in 12.3s

Profit: +120.5% | Trades: 3 | Sharpe: 4.85

Sanity Check Warnings
WARN   Profit Factor (25.3) is extremely high - may indicate overfitting
       → Run with more data to verify stability
WARN   Only 3 trades in backtest period
       → Results may not be statistically significant

Strategy Linting¶

For detailed code analysis outside of backtesting:

tradai strategy lint MyStrategy

Checks for: - Look-ahead bias patterns - Missing startup_candle_count - Repainting indicators - Invalid parameter combinations - Unused imports

Example Output¶

Linting MyStrategy...

strategies/my_strategy/strategy.py:45: WARNING
  Pattern: df.shift(-1) - potential look-ahead bias

strategies/my_strategy/strategy.py:23: WARNING
  Missing startup_candle_count - indicators need warmup period

strategies/my_strategy/indicators.py:12: INFO
  Global .mean() detected - ensure this is intentional

Summary: 2 warnings, 1 info

Data Availability¶

Preflight checks verify data exists before backtesting:

# Check data manually
tradai data check-freshness BTC/USDT:USDT

# Sync if needed
tradai data sync BTC/USDT:USDT --start 2024-01-01 --end 2024-06-01

Validation Output¶

PASS   Data: BTC/USDT:USDT   Available from 2024-01-01 to 2024-12-22
WARN   Data: ETH/USDT:USDT   Partial data: 85% coverage
FAIL   Data: SOL/USDT:USDT   No data available
       → Run: tradai data sync SOL/USDT:USDT

Best Practices¶

1. Never Skip Validation¶

# DON'T do this unless debugging
tradai backtest quick MyStrategy --skip-validation  # ⚠️

2. Use Strict Mode for CI¶

# In CI pipelines, use strict to catch all issues
tradai backtest quick MyStrategy --strict

3. Lint Before Committing¶

# Add to pre-commit workflow
tradai strategy lint MyStrategy

4. Review All Warnings¶

Even if validation passes with warnings, review each one: - Look-ahead warnings → Could invalidate your backtest - Warmup warnings → First trades may use invalid indicators - Sanity warnings → Results may not be reliable

Quick Reference¶

Command	Description
`tradai backtest quick STRATEGY`	Auto-validates before backtest
`tradai backtest quick --strict`	Treat warnings as errors
`tradai backtest quick --skip-validation`	Skip validation (not recommended)
`tradai strategy lint NAME`	Detailed code analysis
`tradai data check-freshness SYMBOL`	Check data availability

Validation Modules¶

For programmatic access:

# Preflight validation
from tradai.strategy.preflight import PreflightValidationService

service = PreflightValidationService()
result = service.validate(
    strategy_name="MomentumV2",
    symbols=["BTC/USDT:USDT"],
    start_date=datetime(2024, 1, 1),
    end_date=datetime(2024, 6, 1),
)

if not result.valid:
    for check in result.checks:
        if not check.passed:
            print(f"{check.severity}: {check.message}")

# Sanity checks
from tradai.strategy.sanity import SanityCheckService

service = SanityCheckService()
result = service.check(backtest_metrics)

for warning in result.warnings:
    print(f"{warning.severity}: {warning.message}")