TradAI Testing Strategy¶

Version: 1.0.0 | Date: 2026-03-28 | Status: CURRENT Source: tests/, pyproject.toml, justfile, .devcontainer/

1. TL;DR¶

18 test suite categories, 14 pytest markers, 60% coverage minimum (enforced via pyproject.toml fail_under = 60; roadmap target: 80%). Parallel execution via pytest-xdist. LocalStack for AWS mocking. Devcontainer for reproducible environments with weekly CI validation.

2. Test Pyramid¶

graph TB
    subgraph pyramid["Test Pyramid"]
        direction TB
        PERF["Performance / Load<br/><i>Locust, pytest-benchmark</i><br/>minutes"]
        E2E["E2E<br/><i>Full workflows, Docker services</i><br/>30-120s"]
        CONTRACT["Contract<br/><i>API schema, backward compat</i><br/>5-15s"]
        INTEGRATION["Integration<br/><i>Service-to-service, LocalStack</i><br/>5-30s"]
        UNIT["Unit<br/><i>Isolated, fast, many</i><br/>&lt;1s each"]
    end

    PERF --> E2E
    E2E --> CONTRACT
    CONTRACT --> INTEGRATION
    INTEGRATION --> UNIT

    style UNIT fill:#4CAF50,color:#fff
    style INTEGRATION fill:#8BC34A,color:#fff
    style CONTRACT fill:#FFC107,color:#000
    style E2E fill:#FF9800,color:#fff
    style PERF fill:#F44336,color:#fff

Guiding principle: Most tests should be unit tests (fast, isolated). Integration and E2E tests validate cross-boundary behavior. Performance and load tests run on demand or in scheduled CI.

3. Test Suite Categories¶

All suites live under tests/ at the repository root. Package-level tests live under libs/*/tests/ and services/*/tests/.

4. Pytest Markers¶

Registered in pyproject.toml under [tool.pytest.ini_options]. All markers are enforced via --strict-markers.

5. Quality Gates¶

6. Test Infrastructure¶

6.1 LocalStack¶

LocalStack provides local AWS service mocking for integration and unit tests that interact with AWS APIs.

Mocked services: S3, DynamoDB, SQS, SNS, ECR, Lambda

Start command:

just up-localstack
# or directly:
docker compose --profile localstack up -d

Environment variables: LOCALSTACK_URL (default http://localhost:4566), AWS_ENDPOINT_URL (set by just check to redirect boto3), AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY (testing), AWS_REGION (LocalStack's default test region).

The root conftest.py provides session-scoped fixtures that auto-start LocalStack and create test buckets (e.g., tradai-configs-test). Cleanup runs only on pytest-xdist worker gw0 to avoid race conditions during bucket teardown.

6.2 Devcontainer¶

The devcontainer provides a fully reproducible development environment with all tools pre-installed.

Configuration: .devcontainer/devcontainer.json

Base image: ghcr.io/astral-sh/uv:python3.11-bookworm-slim

What's included:

Pre-built images: The devcontainer-prebuild.yml workflow builds and pushes a cached image to GHCR on every push to main that touches .devcontainer/ files, plus weekly rebuilds (Sunday 00:00 UTC) for base-image security updates. Local builds pull cached layers via cacheFrom.

Weekly CI validation: The devcontainer-ci.yml workflow runs every Sunday at 02:00 UTC. It builds the devcontainer image, syncs dependencies, and runs the full test suite (excluding slow, e2e, performance, integration) inside the container. This catches environment drift between the devcontainer and bare-runner CI.

Forwarded ports: 8000 (Backend), 8002 (Data Collection), 8003 (Strategy), 5433 (PostgreSQL), 5001 (MLflow), 4566 (LocalStack), 6379 (Redis).

6.3 Git Worktree Testing¶

Each worktree gets its own environment (uv sync --all-packages), after which imports resolve correctly with no environment variables at all — uv writes per-venv _editable_impl_tradai_*.pth files scoped to that worktree.

Never export a PYTHONPATH that points at a specific checkout. An absolute PYTHONPATH precedes the venv on sys.path, so every other worktree then imports tradai.* from that first checkout and its tests silently exercise the wrong code.

just wt-doctor          # from any worktree: is it testing its own code?

Earlier revisions of this section blamed UV's .pth files and prescribed prepending PYTHONPATH. That root cause does not reproduce with current uv, and the workaround is what causes the problem it was meant to solve. Corrected 2026-08-23; see issue

1349.¶

Full documentation: docs/ai/worktree-testing.md

7. Coverage Configuration¶

Coverage is configured in pyproject.toml under [tool.coverage.run] and [tool.coverage.report].

Target: 60% minimum (enforced via pyproject.toml fail_under = 60). Roadmap target: 80%.

Source directories:

[tool.coverage.run]
source = ["libs", "services", "cli"]
omit = ["*/tests/*", "*/__pycache__/*", "*/.venv/*"]

Reports generated:

Excluded lines:

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "def __repr__",
    "if __name__ == .__main__.:",
    "raise AssertionError",
    "raise NotImplementedError",
    "if TYPE_CHECKING:",
]

Note: Most just test commands pass --no-cov to disable coverage for faster feedback. Use just test-cov or just test-strict explicitly when coverage data is needed.

8. Specialized Test Types¶

8.1 Financial Tests¶

Directory: tests/financial/ | Marker: financial

Financial domain tests validate monetary precision and correctness:

• P&L calculations using Decimal types (not float) for exact arithmetic
• Fee computations with configurable rate precision
• Trade entry/exit price validation
• Leverage and position sizing accuracy

The root conftest.py provides sample_trade (with Decimal prices, quantities, fees), sample_ohlcv_df, invalid_ohlcv_df (high < low), and ohlcv_with_gaps (missing candles) fixtures.

8.2 Backtest Validation¶

Directory: tests/backtest_validation/ | Marker: backtest_validation

Validates backtest integrity to prevent common quantitative research pitfalls:

• Look-ahead bias detection -- ensures strategies only use data available at decision time
• Reproducibility -- same config + data must produce identical results across runs
• Strategy compliance -- validates that strategy configs match expected schemas
• Result sanity -- checks profit factor, Sharpe ratio, drawdown within realistic bounds

8.3 Performance Regression Tests¶

Directory: tests/performance/ | Marker: performance

Uses pytest-benchmark and a custom Timer context manager to detect execution-time and memory regressions.

Baseline management: Baseline metrics are stored in tests/performance/baselines.json. The assert_no_regression fixture compares current measurements against baselines with a configurable threshold (default: 20% degradation tolerance).

Memory tracking: Uses memory-profiler (dev dependency) for memory usage assertions.

8.4 Load Testing¶

Directory: tests/load/ | Framework: Locust

Locust-based load tests simulate realistic traffic patterns against all three services.

Available user classes:

Commands:

just load-test                          # Web UI at localhost:8089
just load-test-ci 10 2 1m              # Headless: 10 users, 2/s spawn, 1 min
just load-test-all                      # All services sequentially
just load-test-backend 5 30s           # Backend only
just load-test-data-collection 5 30s   # Data collection only
just load-test-strategy 5 30s          # Strategy service only

Reports are written to reports/load-test-*.html.

8.5 Security Tests¶

Directory: tests/security/ | Marker: security

The root conftest.py provides injection payload fixtures:

• sql_injection_payloads -- 5 SQL injection vectors
• xss_payloads -- 5 XSS vectors
• path_traversal_payloads -- 4 path traversal vectors

Tests validate that all API endpoints reject malicious input without exposing internal errors.

9. Root Conftest Hooks¶

The root tests/conftest.py provides shared fixtures across all test categories.

HTTP client fixtures: backend_client, data_collection_client, strategy_service_client (sync httpx.Client), async_backend_client (async).

Skip conditions: skip_slow_tests (needs RUN_SLOW_TESTS=1), skip_if_no_services (Docker down), skip_if_ci (CI=true), skip_if_no_localstack (health endpoint unreachable).

Data generator: generate_ohlcv_data() produces synthetic OHLCV DataFrames with configurable symbol, periods, frequency, and base price. Uses numpy with seed 42 for reproducibility.

10. Changelog¶

11. Dependencies¶