Backtesting Strategies¶

Complete guide to running, analyzing, and comparing backtests.

Local vs Service Backtesting¶

Mode	Command	When to Use
Local	`tradai backtest quick ... --local`	Dev iteration, no services needed, runs Freqtrade directly
Service	`tradai backtest quick ...`	Production validation, tracks via DynamoDB/MLflow, full metrics

Local mode runs Freqtrade as a subprocess — fast and requires no running services. Service mode sends the request to Backend → Strategy Service, providing comprehensive tracking and result storage.

flowchart TD
    User["User runs<br/><code>tradai backtest quick MyStrategy</code>"]
    Decision{{"--local flag<br/>or --strategy-dir?"}}

    User --> Decision

    subgraph local ["Local Execution Path"]
        FreqtradeCLI["FreqtradeCLIBuilder<br/>constructs command"]
        FreqtradeProc["FreqtradeBacktester<br/>subprocess"]
        LocalResult["Results printed<br/>to terminal"]
        FreqtradeCLI --> FreqtradeProc --> LocalResult
    end

    subgraph service ["Service Execution Path"]
        BackendAPI["Backend API<br/>POST /backtests"]
        SQS["SQS Queue<br/>BacktestJobMessage"]
        StepFn["Step Functions<br/>orchestration"]
        ECS["ECS Task<br/>runs Freqtrade"]
        DynamoDB["DynamoDB<br/>BacktestJobStatus"]
        MLflow["MLflow<br/>metrics + artifacts"]
        ECS --> DynamoDB
        ECS --> MLflow
    end

    Decision -- "Yes" --> FreqtradeCLI
    Decision -- "No" --> BackendAPI
    BackendAPI --> SQS --> StepFn --> ECS

Quick Backtest¶

For rapid iteration during development:

tradai backtest quick MyStrategy

This runs a focused 30-day backtest on BTC/USDT:USDT with minimal output:

Quick backtest: MyStrategy on BTC/USDT:USDT (30d)
✓ Completed in 12.3s

Profit: +5.23% | Trades: 15 | Sharpe: 1.45

Quick Backtest Options¶

# Custom period
tradai backtest quick MyStrategy --period 7d

# Different symbol
tradai backtest quick MyStrategy --symbol ETH/USDT:USDT

# With version
tradai backtest quick MyStrategy@1.0

# Non-blocking (returns job ID immediately)
tradai backtest quick MyStrategy --no-wait

# Strategy in external repo (forces local mode)
tradai backtest quick StochRsiStrategy --strategy-dir ../tradai-strategies

# Or set env var once
export TRADAI_STRATEGY_DIR=../tradai-strategies
tradai backtest quick StochRsiStrategy --period 30d

--strategy-dir forces local execution

The --strategy-dir flag (or TRADAI_STRATEGY_DIR env var) forces local execution via freqtrade subprocess. The backend service uses its own strategy path configuration.

Full Backtest¶

Run a comprehensive backtest through the strategy service:

# Quick backtest (recommended for development)
tradai backtest quick MyStrategy --timerange 20240101-20240331 --symbol BTC/USDT:USDT

# Full backtest via strategy service
just run-backtest MyStrategy 1.0

View Results¶

Show Backtest with Visualizations¶

tradai backtest show <job_id>

Output includes: - Performance summary (trades, profit, win rate) - Risk metrics (Sharpe, Sortino, Calmar ratios) - ASCII equity curve - Monthly returns bar chart

Example:

═══ BACKTEST RESULTS ═══

┌─────────────────────────────────────────────────┐
│              Performance Summary                 │
├─────────────────────────────────────────────────┤
│ Trades: 45 | Win Rate: 71.1%                    │
│ Profit: $1,234.56 (+12.35%)                     │
│                                                 │
│ Risk Metrics                                    │
│ Sharpe: 1.85 | Sortino: 2.41 | Calmar: 0.95    │
│ Max Drawdown: -8.23%                            │
│ Profit Factor: 1.85                             │
└─────────────────────────────────────────────────┘

Equity Curve
$11200 ┤                                     ██
$10800 ┤                               ██████
$10400 ┤                         ██████
$10000 ┤ ████████████████████████
       └─────────────────────────────────────────
         Jan      Feb      Mar      Apr      May

Monthly Returns
Jan: ████████████  +5.2%
Feb: ██████        +2.1%
Mar: ████          -1.5%
Apr: ██████████    +4.8%
May: ████          +1.7%

View in MLflow: http://localhost:5001
Generate report: tradai backtest report abc123

Disable Charts¶

For CI/scripting:

tradai backtest show <job_id> --no-chart

Compare Backtests¶

Compare two backtest runs side by side:

tradai backtest compare run1-abc123 run2-def456

Output:

                    Backtest Comparison
┌──────────────────┬─────────────┬─────────────┬──────────┐
│ Metric           │ run1-abc123 │ run2-def456 │ Diff     │
├──────────────────┼─────────────┼─────────────┼──────────┤
│ sharpe_ratio     │ 1.45        │ 1.82        │ +0.37    │
│ sortino_ratio    │ 2.10        │ 2.45        │ +0.35    │
│ total_profit_pct │ 12.35%      │ 15.20%      │ +2.85%   │
│ max_drawdown_pct │ -8.23%      │ -6.50%      │ +1.73%   │
│ win_rate         │ 71.1%       │ 68.0%       │ -3.1%    │
│ profit_factor    │ 1.85        │ 2.10        │ +0.25    │
└──────────────────┴─────────────┴─────────────┴──────────┘

Winner by Sharpe: run2-def456
Winner by Profit: run2-def456

Custom Metrics¶

Compare specific metrics:

tradai backtest compare run1 run2 --metrics sharpe_ratio,profit_factor

Generate HTML Report¶

Create an interactive HTML report with Plotly charts:

tradai backtest report <job_id>

This generates backtest_report.html with: - Interactive equity curve (zoom, pan, hover) - Drawdown chart - Monthly returns heatmap - Trade distribution analysis - Win/loss breakdown

Report Options¶

# Custom output path
tradai backtest report <job_id> --output my_analysis.html

# Don't open browser
tradai backtest report <job_id> --no-open

Export Trades¶

Get detailed trade log (no 100-trade limit):

# Table view
tradai backtest trades <job_id>

# Export to CSV
tradai backtest trades <job_id> --format csv --output trades.csv

# Export to JSON
tradai backtest trades <job_id> --format json --output trades.json

# Export to Excel
tradai backtest trades <job_id> --format excel --output trades.xlsx

Trade data includes: - Open/close date and price - Pair and side (long/short) - Profit/loss - Duration - Exit reason

List Backtests¶

View recent backtest jobs:

tradai backtest list

Output:

                      Backtest Jobs
┌──────────────┬───────────┬──────────┬────────┬────────┬─────────────────────┐
│ Job ID       │  Status   │ Profit % │ Sharpe │ Trades │ Created             │
├──────────────┼───────────┼──────────┼────────┼────────┼─────────────────────┤
│ abc123def456 │ completed │ +12.35%  │   1.45 │     45 │ 2024-12-22T10:30:00 │
│ xyz789uvw012 │  running  │        - │      - │      - │ 2024-12-22T10:45:00 │
│ mno345pqr678 │   failed  │        - │      - │      - │ 2024-12-22T09:15:00 │
└──────────────┴───────────┴──────────┴────────┴────────┴─────────────────────┘

Filter by Status¶

tradai backtest list --status completed
tradai backtest list --status running
tradai backtest list --limit 50

MLflow Integration¶

Backtests are automatically tracked in MLflow:

# View MLflow UI
open http://localhost:5001

MLflow stores: - Run metrics (Sharpe, Sortino, profit, etc.) - Parameters (stoploss, stake amount) - Artifacts (trade logs, equity curves)

Workflow Example¶

Complete development → backtest → analyze workflow:

# 1. Create strategy
tradai strategy new MomentumV2 --category momentum --timeframe 1h

# 2. Edit strategy logic (in the tradai-strategies repo)
code strategies/momentum_v2/src/momentum_v2/strategy.py

# 3. Test indicators
tradai indicator test "ta.RSI(df['close'], 14)" --symbol BTC/USDT:USDT

# 4. Lint strategy
tradai strategy lint ./momentum_v2/

# 5. Quick backtest
tradai backtest quick MomentumV2 --period 7d

# 6. Extended backtest (longer period)
tradai backtest quick MomentumV2 --period 90d

# 7. View results
tradai backtest show <job_id>

# 8. Compare with previous version
tradai backtest compare <new_job_id> <old_job_id>

# 9. Generate report
tradai backtest report <job_id>

Quick Reference¶

Command	Description
`tradai backtest quick STRATEGY`	Quick 30-day backtest
`tradai backtest show JOB_ID`	View results with charts
`tradai backtest compare RUN1 RUN2`	Compare two runs
`tradai backtest report JOB_ID`	Generate HTML report
`tradai backtest trades JOB_ID`	Export trade log
`tradai backtest list`	List recent backtests

Quality Thresholds¶

Use these benchmarks to decide whether results are ready for validation:

Metric	Minimum	Target	Overfitting Warning
Sharpe Ratio	> 1.0	> 1.5	> 4.0 (likely overfit)
Sortino Ratio	> 1.5	> 2.0	> 5.0
Max Drawdown	< -25%	< -15%	< -3% (unrealistic)
Win Rate	> 40%	> 55%	> 85% (likely overfit)
Profit Factor	> 1.2	> 1.5	> 3.0
Trade Count	> 30	> 100	— (more is better)

Overfitting Signs

Extremely good metrics (Sharpe > 4, Win Rate > 85%) often indicate overfitting. Validate with out-of-sample data, different time periods, and multiple symbols before proceeding to deployment.

Realism Controls (Issue #460)¶

Backtests systematically overstate live performance unless fees, slippage, and out-of-sample marking are made explicit. Every BacktestConfig carries four fields that govern realism:

Field	Type	Default	Purpose
`fee_rate`	`float \\| None` (0.0–0.1)	`None` (exchange default)	Per-side fee as decimal (e.g. `0.0007` = 0.07%). Passed to Freqtrade as the top-level `fee` config key.
`slippage_bps`	`float` (0–500)	`0.0`	Per-side slippage in basis points. Read by `TradAIStrategy.custom_entry_price` / `custom_exit_price` and applied via `apply_tradai_slippage(rate, is_long=…, is_entry=…)`.
`is_out_of_sample`	`bool \\| None`	`None` (unknown)	Caller-declared OOS intent. `True` means the dates are a holdout window never used in training or tuning.
`out_of_sample_period`	`str \\| None`	`None`	Human-readable label, e.g. `"2024-07-01..2024-12-31"`.

Every BacktestResult exposes the produced fee data so operators can audit it:

Field	Type	Meaning
`total_fees`	`float \\| None`	Sum across every trade's orders of `order.cost * trade.fee_{open\\|close}` plus per-trade `funding_fees` (futures). `None` when no trades.
`total_fee_pct`	`float \\| None`	`total_fees / stats["total_volume"] * 100`. Both numerator and denominator are round-trip notional, so the ratio equals the per-side fee rate when entry/exit notional are similar.
`is_out_of_sample`	`bool \\| None`	Stamped from the config by `FreqtradeBacktester.execute()`. Round-trips through MLflow tags.
`out_of_sample_period`	`str \\| None`	Stamped from the config.

Submit a realistic backtest¶

curl -X POST http://localhost:8000/api/v1/backtests \
  -H "Content-Type: application/json" \
  -d '{
    "experiment_name": "momentum-validation",
    "config": {
      "strategy": "MomentumStrategy",
      "timeframe": "1h",
      "start_date": "2024-07-01",
      "end_date": "2024-12-31",
      "symbols": ["BTC/USDT:USDT"],
      "stake_amount": 100,
      "exchange": "binance_futures",
      "fee_rate": 0.0007,
      "slippage_bps": 25.0,
      "is_out_of_sample": true,
      "out_of_sample_period": "2024-07-01..2024-12-31"
    }
  }'

Backtest slippage is candle-bounded

slippage_bps is fully effective in live trading (no candle constraint). In backtest mode, Freqtrade's get_valid_entry_price_and_stake clamps the bumped limit price to the simulated candle's [low, high] range:

propose_rate = min(propose_rate, row[HIGH_IDX])  # long
propose_rate = max(propose_rate, row[LOW_IDX])   # short

So a slippage_bps=25 long-entry bump above candle-high is silently clamped to candle-high. Read slippage_bps in backtest output as an upper bound on intent, not a literal applied cost. The live total_fees numerator is unaffected.

Strict OOS gate at promotion¶

PerformanceThresholds.validate_result() rejects any BacktestResult where is_out_of_sample is not True by default. The promotion-time validation backtest in ModelPromotionService verifies the validation window starts strictly after the model's training_end_date tag — if the tag is missing or the windows overlap, the result is marked is_out_of_sample=False and the strict gate fails loudly. See Phase 5: Register for the CLI --oos flag.

To opt out (NOT recommended for production promotion), construct PerformanceThresholds(require_out_of_sample=False) with an explicit justification in the call site.

Custom strategy hooks¶

If your strategy subclasses TradAIStrategy and overrides custom_entry_price or custom_exit_price, you must either:

Call super().custom_entry_price(...) (or super().custom_exit_price(...)) and return its result, OR
Call self.apply_tradai_slippage(rate, is_long=..., is_entry=...) on your computed price, OR
Set _tradai_slippage_managed = True on the subclass to opt out (you accept responsibility for fill realism).

TradAIStrategy.__init_subclass__ emits a RuntimeWarning at class-definition time if none of the above hold — silent slippage bypass would recreate the exact bug class Issue #460 fixes.

Metrics Reference¶

Metric	Description	Good Value
Sharpe Ratio	Risk-adjusted return	> 1.5
Sortino Ratio	Downside risk-adjusted	> 2.0
Calmar Ratio	Return vs max drawdown	> 1.0
Win Rate	% of winning trades	> 50%
Profit Factor	Gross profit / gross loss	> 1.5
Max Drawdown	Largest peak-to-trough	< -20%
CAGR	Compound annual return	Depends on risk

Environment Variables¶

Variable	Description	Default
`TRADAI_STRATEGY_DIR`	Directory containing strategies (forces local mode)	—
`MLFLOW_TRACKING_URI`	MLflow server URL	`http://localhost:5001`
`BACKEND_URL`	Backend service URL	`http://localhost:8000`