TradAI Testing Strategy¶

Version: 1.0.0 | Date: 2026-03-28 | Status: CURRENT Source: tests/, pyproject.toml, justfile, .devcontainer/

1. TL;DR¶

18 test suite categories, 14 pytest markers, 80% coverage minimum. Parallel execution via pytest-xdist. LocalStack for AWS mocking. Devcontainer for reproducible environments with weekly CI validation.

2. Test Pyramid¶

graph TB
    subgraph pyramid["Test Pyramid"]
        direction TB
        PERF["Performance / Load<br/><i>Locust, pytest-benchmark</i><br/>minutes"]
        E2E["E2E<br/><i>Full workflows, Docker services</i><br/>30-120s"]
        CONTRACT["Contract<br/><i>API schema, backward compat</i><br/>5-15s"]
        INTEGRATION["Integration<br/><i>Service-to-service, LocalStack</i><br/>5-30s"]
        UNIT["Unit<br/><i>Isolated, fast, many</i><br/>&lt;1s each"]
    end

    PERF --> E2E
    E2E --> CONTRACT
    CONTRACT --> INTEGRATION
    INTEGRATION --> UNIT

    style UNIT fill:#4CAF50,color:#fff
    style INTEGRATION fill:#8BC34A,color:#fff
    style CONTRACT fill:#FFC107,color:#000
    style E2E fill:#FF9800,color:#fff
    style PERF fill:#F44336,color:#fff

Guiding principle: Most tests should be unit tests (fast, isolated). Integration and E2E tests validate cross-boundary behavior. Performance and load tests run on demand or in scheduled CI.

3. Test Suite Categories¶

All suites live under tests/ at the repository root. Package-level tests live under libs/*/tests/ and services/*/tests/.

4. Pytest Markers¶

Registered in pyproject.toml under [tool.pytest.ini_options]. All markers are enforced via --strict-markers.

5. Quality Gates¶

6. Test Infrastructure¶

6.1 LocalStack¶

LocalStack provides local AWS service mocking for integration and unit tests that interact with AWS APIs.

Mocked services: S3, DynamoDB, SQS, SNS, ECR, Lambda

Start command:

just up-localstack
# or directly:
docker compose --profile localstack up -d

Environment variables: LOCALSTACK_URL (default http://localhost:4566), AWS_ENDPOINT_URL (set by just check to redirect boto3), AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY (testing), AWS_REGION (us-east-1).

The root conftest.py provides session-scoped fixtures that auto-start LocalStack and create test buckets (e.g., tradai-configs-test). Cleanup runs only on pytest-xdist worker gw0 to avoid race conditions during bucket teardown.

6.2 Devcontainer¶

The devcontainer provides a fully reproducible development environment with all tools pre-installed.

Configuration: .devcontainer/devcontainer.json

Base image: ghcr.io/astral-sh/uv:python3.11-bookworm-slim

What's included:

Pre-built images: The devcontainer-prebuild.yml workflow builds and pushes a cached image to GHCR on every push to main that touches .devcontainer/ files, plus weekly rebuilds (Sunday 00:00 UTC) for base-image security updates. Local builds pull cached layers via cacheFrom.

Weekly CI validation: The devcontainer-ci.yml workflow runs every Sunday at 02:00 UTC. It builds the devcontainer image, syncs dependencies, and runs the full test suite (excluding slow, e2e, performance, integration) inside the container. This catches environment drift between the devcontainer and bare-runner CI.

Forwarded ports: 8000 (Backend), 8002 (Data Collection), 8003 (Strategy), 5433 (PostgreSQL), 5001 (MLflow), 4566 (LocalStack), 6379 (Redis).

6.3 Git Worktree Testing¶

When running tests in a git worktree (e.g., .claude/worktrees/<name>), Python imports may resolve to the main repo instead of the worktree's modified source files due to UV's editable install .pth files.

Fix: Prepend worktree source paths to PYTHONPATH:

wt-test() {
    local WT="$(pwd)"
    PYTHONPATH="$WT/libs/tradai-common/src:$WT/libs/tradai-data/src:..." \
    uv run pytest "$@"
}

Full documentation: docs/ai/worktree-testing.md

7. Coverage Configuration¶

Coverage is configured in pyproject.toml under [tool.coverage.run] and [tool.coverage.report].

Target: 80% minimum (enforced in just test-strict via --cov-fail-under)

Source directories:

[tool.coverage.run]
source = ["libs", "services", "cli"]
omit = ["*/tests/*", "*/__pycache__/*", "*/.venv/*"]

Reports generated:

Excluded lines:

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "def __repr__",
    "if __name__ == .__main__.:",
    "raise AssertionError",
    "raise NotImplementedError",
    "if TYPE_CHECKING:",
]

Note: Most just test commands pass --no-cov to disable coverage for faster feedback. Use just test-cov or just test-strict explicitly when coverage data is needed.

8. Specialized Test Types¶

8.1 Financial Tests¶

Directory: tests/financial/ | Marker: financial

Financial domain tests validate monetary precision and correctness:

• P&L calculations using Decimal types (not float) for exact arithmetic
• Fee computations with configurable rate precision
• Trade entry/exit price validation
• Leverage and position sizing accuracy

The root conftest.py provides sample_trade (with Decimal prices, quantities, fees), sample_ohlcv_df, invalid_ohlcv_df (high < low), and ohlcv_with_gaps (missing candles) fixtures.

8.2 Backtest Validation¶

Directory: tests/backtest_validation/ | Marker: backtest_validation

Validates backtest integrity to prevent common quantitative research pitfalls:

• Look-ahead bias detection -- ensures strategies only use data available at decision time
• Reproducibility -- same config + data must produce identical results across runs
• Strategy compliance -- validates that strategy configs match expected schemas
• Result sanity -- checks profit factor, Sharpe ratio, drawdown within realistic bounds

8.3 Performance Regression Tests¶

Directory: tests/performance/ | Marker: performance

Uses pytest-benchmark and a custom Timer context manager to detect execution-time and memory regressions.

Baseline management: Baseline metrics are stored in tests/performance/baselines.json. The assert_no_regression fixture compares current measurements against baselines with a configurable threshold (default: 20% degradation tolerance).

Memory tracking: Uses memory-profiler (dev dependency) for memory usage assertions.

8.4 Load Testing¶

Directory: tests/load/ | Framework: Locust

Locust-based load tests simulate realistic traffic patterns against all three services.

Available user classes:

Commands:

just load-test                          # Web UI at localhost:8089
just load-test-ci 10 2 1m              # Headless: 10 users, 2/s spawn, 1 min
just load-test-all                      # All services sequentially
just load-test-backend 5 30s           # Backend only
just load-test-data-collection 5 30s   # Data collection only
just load-test-strategy 5 30s          # Strategy service only

Reports are written to reports/load-test-*.html.

8.5 Security Tests¶

Directory: tests/security/ | Marker: security

The root conftest.py provides injection payload fixtures:

• sql_injection_payloads -- 5 SQL injection vectors
• xss_payloads -- 5 XSS vectors
• path_traversal_payloads -- 4 path traversal vectors

Tests validate that all API endpoints reject malicious input without exposing internal errors.

9. Root Conftest Hooks¶

The root tests/conftest.py provides shared fixtures across all test categories.

HTTP client fixtures: backend_client, data_collection_client, strategy_service_client (sync httpx.Client), async_backend_client (async).

Skip conditions: skip_slow_tests (needs RUN_SLOW_TESTS=1), skip_if_no_services (Docker down), skip_if_ci (CI=true), skip_if_no_localstack (health endpoint unreachable).

Data generator: generate_ohlcv_data() produces synthetic OHLCV DataFrames with configurable symbol, periods, frequency, and base price. Uses numpy with seed 42 for reproducibility.

10. Changelog¶

11. Dependencies¶