RUN_CONFIG cutover runbook (#754)¶

STATUS — the training cutover (the addendum below) is COMPLETE. Steps 1-7 of the addendum have been executed: tradai-common published, e2e-test-strategy rebuilt onto the reader, both canary proofs passed against the deployed image, and the scheduler's dual-write is stripped (cutover step 6). Training now has no per-field env fallback at all.

The live/dry-run cutover in the first half of this document was completed earlier (#754 P0/P1); its per-service migration tooling remains the reference for future fleets.

The live/dry-run reader (TradingHandler) parses one RUN_CONFIG env blob and fails loud if it is absent — there is no per-field fallback (no dual-read, by design). The writer (StrategyOperationsService.run_strategy) only injects RUN_CONFIG (and strips the legacy TIMEFRAME/PAIRS/CONFIG_OVERRIDES/EXCHANGE_SECRET_NAME/MODEL_VERSION_ALIAS env) on an explicit launch.

The hazard this runbook prevents¶

A live/dry-run service whose task-def predates the cutover has no RUN_CONFIG. Two ways to crash a live trader during the transition:

1. The new (fail-loud) image reaches a RUN_CONFIG-less task-def → it raises in _parse_run_config and ECS crash-loops it.
2. The migration adds RUN_CONFIG + strips the legacy env, but the task-def still runs the old image → the old reader ignores RUN_CONFIG and, with EXCHANGE_SECRET_NAME stripped, a live trader can't load credentials.

Both are avoided by making the image repin and the RUN_CONFIG add the SAME task-def revision — the old task keeps running the old image+env until the new revision deploys, and the new revision is always self-consistent (new image AND RUN_CONFIG). run_strategy does this (image_uri + RUN_CONFIG in one revision); the migration drives it per service.

This is a CROSS-REPO cutover. The RUN_CONFIG reader lives in tradai-common, which is baked into each per-strategy image (tradai/strategies/<slug>) — built in the separate tradai-strategies repo from CodeArtifact — NOT the platform strategy-service image. So an already-deployed strategy image on an old tradai-common cannot read RUN_CONFIG. There is no single fleet image: every deployed strategy must be re-released from tradai-strategies against the new tradai-common, and the cutover repins EACH service to ITS OWN rebuilt image. This is a one-time cost of the breaking wire-contract change; steady-state tradai-common updates do not require it (only a breaking schema_version bump would).

Ordered steps (dev downtime cutover)¶

1. Disable triggers so nothing launches mid-cutover: pause the SFN start path, schedulers, and any SQS event-source mappings for the strategy path.
2. Publish tradai-common (with the reader) to CodeArtifact, then re-release every deployed strategy from tradai-strategies so each tradai/strategies/<slug> image is rebuilt against it. Push the rebuilt images to ECR and record each digest-pinned URI, but do NOT redeploy the ECS services to them yet — a bare image rollout is precisely the crash-loop in hazard (1). Assemble a strategy_id -> image URI JSON map. (A CI gate that blocks the ASL/infra apply until these images exist is a tracked follow-up; until it lands, confirm the images are in ECR manually here.)
3. Deploy the backend carrying the new run_strategy + migration.
4. Migrate every ACTIVE service atomically: uv run python scripts/ops/migrate_run_config.py --images strategy-images.json (a thin, exit-code-gated wrapper over StrategyOperationsService.migrate_all_to_run_config; exit 0 = all migrated/skipped, 1 = blocked remain, 2 = ECS not configured, 3 = bad map file). It enumerates non-terminal services from the trading-state repo and, per service, registers one task-def revision with that strategy's own rebuilt image (from the map) and RUN_CONFIG, then force-deploys. It is idempotent and safety-guarded:
5. a service already on a valid RUN_CONFIG (parses under its mode) is skipped;
6. a stopped service (desired_count == 0, e.g. emergency-stopped) is NEVER resumed by the migration. If it is already on a valid blob it is skipped; if it is still on a legacy / reader-less task-def it is stopped_legacy — a resume-time trap (a naive resume would trade base-config defaults live), reported with a RESUME GATE warning. This also covers the normal successful kill-switch, which persists a terminal STOPPED trading-state row the DB scan excludes: the fleet cross-check surfaces such a stopped legacy service by ECS service name. Repin it while stopped (or gate its resume) before scaling it back up;
7. a non-live/dry-run service is skipped;
8. a service is blocked (not migrated, not crashed) when its blob is present-but-invalid with the legacy env already gone, or its legacy max_leverage/max_drawdown_pct exceeds deployment-safe bounds. blocked is a hard stop for that service. stopped_legacy does NOT hard-fail the fleet gate (the service isn't running), but it is emphatically not "clean" — resolve every one before re-enabling triggers.
9. Resolve every blocked result before proceeding: the operator must fix the service's config (or stop it) and re-run step 4. migrate_all returning blocked_count == 0 is the gate to continue. Add --wait (H6) to block until each migrated service reaches a terminal ECS rollout state before the script returns. A service that does NOT stabilize — crash-looped on the new image, or circuit-breaker-rolled-back to its legacy task def — is reported BLOCKED (not a false migrated), so the exit code is non-zero and the cutover is treated as NOT done. --stable-timeout <s> sets the per-service deadline (default 300s).
10. Verify by re-running scripts/ops/migrate_run_config.py with no --images (the read-only verify pass — it calls verify_run_config_cutover() and never mutates ECS): a clean cutover exits 0, reporting every service skipped ("already on a valid RUN_CONFIG") and zero blocked/needs_migration. The verify pass is fleet-scoped: it enumerates the trading-state table AND cross-checks the actual ECS cluster (ecs:ListServices), so an ACTIVE service whose DynamoDB row is terminal/missing is surfaced as blocked rather than silently certified clean. It is still a MANUAL stabilization check, not an automated ECS-stable wait — update_service returns before ECS finishes deploying, so a service that circuit-breaker-rolled-back to the old (legacy) task-def surfaces here as needs_migration. Even with --wait used in step 4, run this final read-only pass as the independent confirmation that nothing has since rolled back. (An automated wait-until-services-stable gate is a tracked follow-up, H6.)
11. Re-enable triggers. Smoke-test: launch one dry-run via the API and confirm the container parses RUN_CONFIG and starts (no _parse_run_config raise in logs).

Rollback (H8 — one command per service)¶

The reader is fail-loud with no dual-read, so a rollback across the cutover boundary must roll the task-def env back too — not just repin the image. Rolling back by re-running the migration with the OLD image URI is unsafe (the old reader can't read RUN_CONFIG). A whole prior task-def revision is self-consistent (old image + its legacy per-field env travel as one unit), so restore that:

uv run python scripts/ops/migrate_run_config.py --rollback <strategy_id> --to-task-def <ARN>

This force-deploys the recorded pre-cutover revision atomically (single update_service, the single-writer deployment config applies). Do NOT repin only the image digest across this boundary. Keep the pre-cutover task-def revision ARNs recorded (step 2) so you have the <ARN> to restore; rollback is one service at a time (deliberate — you decide which to revert).

What is automated vs manual (do not assume more is automated than is)¶

Untracked / terminal-row service remediation (#810 item #3)¶

The fleet cross-check flags an ACTIVE service the DB scan missed (blocked) or a stopped pre-cutover one (stopped_legacy) — it does not auto-migrate it, and the migration script cannot be pointed at one. migrate_all_to_run_config iterates the trading-state table (list_non_terminal) and only appends the untracked services as flags after that loop; a --images entry never reaches them, and the read-only inspect path returns before repinning a stopped_legacy service. This is a deliberate decision, not a missing feature:

• The only key available for a table-invisible service is its ECS service name. The strategy_id → tradai-strategy-{slug} mapping (get_strategy_service_name) is lossy in reverse — the name cannot recover the strategy_id, so the flag records the service name in the strategy_id field purely for identification. Do not feed that value back into the migration: it re-derives the service via get_strategy_service_name(strategy_id), which would double-prefix the name and target a non-existent (or wrong) service.
• Even the task definition's own STRATEGY_ID env is not a safe migration key here: the CLI deploy path sets STRATEGY_ID = {strategy_name}-{env} while it derives the ECS service name from strategy_name alone (deploy/commands.py), so STRATEGY_ID does not round-trip through get_strategy_service_name. Guessing the image from a mis-derived identity is exactly the fail-loud hazard the cutover prevents.

Operator remediation (manual, safe):

1. Identify the strategy from the flagged service — describe its task definition and read the STRATEGY_ID / image for human identification only (aws ecs describe-task-definition), or match the service name to a tradai-strategies release.
2. stopped_legacy (desired_count=0): leave it stopped and gate its resume — do NOT scale it back up on the legacy task-def (a naive resume trades base-config defaults live). Roll it onto a self-consistent revision before any resume (a whole prior revision, per the Rollback section, or step 3).
3. blocked (ACTIVE): re-run tradai deploy strategy for it from tradai-strategies — the deploy path is the authoritative cutover writer (rebuilds the image, registers a task-def WITH RUN_CONFIG, strips legacy env, repins the digest, and writes a tracked trading-state row), so the service becomes visible to the DB scan and clean. This is preferred over any hand-rolled register-task-definition + update-service.
4. Re-run the read-only verify pass (step 6) until the fleet reports zero blocked / stopped_legacy. A targeted "migrate this exact service name" CLI is intentionally not provided — the safe path is a fresh deploy, not a reverse-lookup.

Strategy-image capability marker (cross-repo — tracked in #782)¶

The cutover writer strips the legacy env; a strategy image whose baked tradai-common can't read RUN_CONFIG then crash-loops, and capability is not derivable from an image digest alone.

Both halves of #782 are now built and merged. The producer stamps a tradai.runconfig.schema_version OCI label at build time in the (single-stage) cookiecutter Dockerfile, read from the ACTUALLY installed tradai-common (not a host /opt/venv build-arg, which fails on CI or stamps a divergent version); an in-image gate fails the build on drift. The consumer reads the label by digest via ECR and enforces it at ingest (registration) and at every forward ACTIVE-advancing site (promotion, forward repoint). Rollback deliberately bypasses the check — recovery must never be capability-blocked.

The gate does not protect anything yet, and this step is still a manual operator check. It rejects only on a label that is present and different from the platform's current version, and:

• the deployed fleet carries no capability label at all (no image was built through the labelling path before it shipped) — or the producer's fail-safe unknown default. Both normalize to unknown-grace in is_unknown_runconfig_capability, so neither is rejected;
• existing ReleaseRecord rows carry no capability — capability is snapshotted once at registration into an immutable record, so only a fresh registration can stamp one;
• TradingRunConfig.schema_version is still Literal[1], so no image could be rejected even with a fully labelled fleet.

Flipping unknown from graced to rejected, backfilling the fleet, and the re-release campaign that makes the gate bite are tracked in #1001. Until that lands, confirm image capability before the flip manually (step 2's re-release + ECR check).

scripts/ops/audit_runconfig_capability.py is advisory by default — it exits 0 on any completed audit. Run it with --strict to turn "every audited strategy is ok" into a precondition a runbook step or CI job can gate on: it exits 1 when any strategy is not ok, and ⅔/4 still mean unconfigured table / bad usage / operational failure, so a failed audit is never mistaken for a dirty fleet. Note the audit covers exactly the strategies you pass — the deployments table has no env-scoped index, so it is not a fleet enumeration and a short --strategies list still exits 0.

Addendum: TRAINING cutover (#754 P2)¶

Everything above is live/dry-run only. Its hazard model is persistent ECS services with task-def revisions, and its tooling (scripts/ops/migrate_run_config.py) is built around repinning a running fleet. Training has neither: every run is a one-shot RunTask, so there is no service to repin and no crash-loop to avoid — a bad launch fails one job.

That makes this cutover lighter, but it has one hazard the live path does not.

The hazard this addendum prevents¶

The training reader is fail-loud (StrategyEntrypoint.train_run_config), and the writer (retraining-scheduler) deploys on a different clock than the reader:

So the writer is always ahead. If it stripped the legacy per-field env at merge, every training run between the merge and the image rebuild would launch an old image with no readable config.

This is why the scheduler dual-writes. It emits RUN_CONFIG and a legacy env block, every value of which is projected from the validated contract (_legacy_env in the handler) rather than re-read from the raw model config — so the two cannot disagree. Old images read the env; new images read the blob. No container ever reads both, so this is not a dual-read and the "one format, parsed once" rule still holds.

FREQAI_MODEL is emitted only when its value is one the pre-cutover reader accepts. EntrypointSettings validates it against a 3-item allowlist while the contract permits every model in FreqAIModelRegistry (18), so echoing a legal-but-newer value would make an old image refuse to start at settings construction — before any status could be reported.

This is now history. The dual-write was removed in step 6 after the canary proved reader-capable images were live. _legacy_env, _LEGACY_READER_FREQAI_MODELS and _assert_legacy_safe_freqai_model no longer exist, and the FREQAI_MODEL allowlist note above no longer applies — every model FreqAIModelRegistry resolves is launchable.

Ordered steps¶

1. Freeze training launches.
2. Disable the EventBridge schedule for tradai-retraining-scheduler-<env>.
3. Confirm no manual entry point is about to fire.
4. Drain in-flight runs. Reuse the idempotency identity the scheduler already stamps — JOB_ID is the per-model lock token, also passed as the ECS clientToken. Poll ecs describe-tasks until no RUNNING task carries a scheduler-launched identity. Do not invent new tracking; do not release the per-model retraining lock by hand (a released lock plus an unwritten last_retraining launches a duplicate — EC9/#541).
5. Publish tradai-common — automatic since #942: merging to dev publishes it once CI is green. No tag needed.
6. Rebuild the training images — cross-repo. Both the generic and the per-strategy training images build in the sibling tradai-strategies repo, whose CI lives in its own docs.

   Name the owner before starting. This is the step that stalls: nothing in this repo can trigger it, and a half-done cutover leaves the writer ahead of the reader indefinitely.

7. Prove reader capability on one canary run. Note the scheduler now projects the legacy env FROM the contract, so a scheduler-launched run cannot have them disagree — you cannot prove the blob was read by comparing the two. Launch the canary directly instead:

aws ecs run-task --task-definition tradai-strategy-<slug>-<env> \
  --overrides '{"containerOverrides":[{"name":"strategy","environment":[
     {"name":"TRADING_MODE","value":"train"},
     {"name":"RUN_CONFIG","value":"<blob with values unlike every default>"},
     {"name":"PAIRS","value":"ZZZ/USDT:USDT"},
     {"name":"TRAIN_PERIOD_DAYS","value":"999"}]}]}' ...

with deliberately CONFLICTING legacy decoys, then confirm from the run's MLflow params and job-log feed that the window, timeframe, pairs and timeout all match the blob, not the decoys. A run that merely succeeds proves nothing. 6. Strip the dual-write. DONE. All six pieces were removed from lambdas/retraining-scheduler/handler.py — _legacy_env and its spread in _container_env; _LEGACY_READER_FREQAI_MODELS; _assert_legacy_safe_freqai_model and its call; the DUAL-WRITE comment block; and two dual-write-specific diagnostics that would otherwise have survived as lies (_assert_override_budget's "counted twice" docstring and its ValidationError text citing "cutover step 6"). The contract does not enforce a model whitelist of its own — TrainingRunConfig.freqai_model is a plain str and FreqAIModelRegistry accepts built-in, discovered and custom models by design — so the blocked set was "everything the registry resolves except those three", not a fixed count.

One consequence to know: the payload roughly halved, so the ECS 8192-char combined-override guard is no longer reachable through pairs at the current 5000-char blob cap. The guard is kept as defence-in-depth for env vars added later. 7. Re-enable the EventBridge schedule.

No dual-read at any point.

What merging the #754 P2 stack did NOT do — and what closed it¶

Merging shipped the writers only. #754 P2 did not close on merge; it closed on deployed canary evidence, tracked in #1187. The three gaps that merge left open, and how each was closed:

• The training container read legacy per-field env. The RUN_CONFIG reader reaches a container only via a tradai-common publish plus an image rebuild in the separate tradai-strategies repo. (Since #942 that publish is a dev merge with a version bump, not a tag; at the time of this cutover it required a tag.) Closed by publishing tradai-common, regenerating the strategy's uv.lock (the lock — not CodeArtifact's "latest" — governs what the image ships), and verifying the deployed task-def revision by image digest; nothing pushes :latest for strategy images.
• freqai_model was restricted to the old reader's three-item allowlist. Lifted by step 6.
• backtest_period_days and training_timeout_hours — the two fields #746 is named after — round-tripped only because the dual-write carried them explicitly. Now carried by the blob alone, proven by the canary and pinned by a test asserting both their absence from the env and their correct values in the contract (asserting only absence would pass just as well if they had been dropped outright — the original bug).

Rollback¶

Re-enable the schedule on the previous Lambda version. Because training is one-shot, rollback is per-run, not per-fleet: no task-def revision to revert, no running trader to protect. If the reader turns out to be broken, roll the image back (repin the previous digest in tradai-strategies).

The dual-write is no longer the safety net it was. Before step 6 an older image rolled back to would still find its per-field env. It will not now: the scheduler emits only RUN_CONFIG, so a pre-reader image fails loud at launch (its _build_config raises on the missing PAIRS). Rolling the image back therefore also requires rolling the Lambda back to a pre-step-6 version, or accepting that training is down until a reader-capable image is repinned. Training is one-shot, so this costs runs, not a running trader.

Deploy note — retiring sqs-consumer is DESTRUCTIVE¶

Removing it from LAMBDA_NAMES also removes it from the derived LAMBDA_ECR_REPOS (infra/shared/tradai_infra_shared/config.py), so the routine pulumi up on persistent deletes its ECR repository. This is not the manual delete-existing.sh helper — no operator action is required for it to happen.

Two stacks are involved, and neither one does both. persistent deletes the ECR repository; compute deletes the function and its log group. Applying only persistent leaves the function in place pointing at an image that no longer exists.

For a manual apply, the safe order is:

1. compute first (just infra-up-compute <env>) — deletes the Lambda and its log group.
2. persistent second — deletes the tradai/lambda-sqs-consumer ECR repository.

Doing it the other way round removes the image an existing function still references.

The dev auto-deploy does the reverse, and that is tolerated here. deploy-dev-on-push runs persistent → (waits up to 35 min for Deploy Lambdas) → foundation → compute → edge. So on merge the image is force-deleted while tradai-sqs-consumer-dev still references it, and the function stays orphaned pointing at a missing image until the compute apply lands — or indefinitely if that apply fails for an unrelated reason. For this Lambda the blast radius is nil: it has never been invoked and has no trigger, so a broken image URI harms nothing. Do not generalize that — for any Lambda that actually runs, follow the manual order above rather than letting the pipeline invert it.

prod caveat: infra/persistent/modules/ecr.py sets force_delete=ENVIRONMENT != "prod" and protect=ENVIRONMENT == "prod". In prod, pulumi state unprotect alone is therefore not sufficient — because force_delete is False there, the repository must also be emptied of all images before it can be deleted, or the apply fails. On dev, force_delete is True and it succeeds silently.

Not covered here¶

• A legacy CONFIG_OVERRIDES reader still exists — on the live/dry-run side, not this one. The container reader is gone (run_config.py is the sole allowlist gate), but StrategyOperationsService still parses legacy task-def env at services/backend/.../strategy_operations.py:1216 to reconstruct a launch during migration. "#754 removed every CONFIG_OVERRIDES env read" is therefore too broad; retire that reader with the live fleet cutover above, not with this addendum.
• The retraining state machine is not part of this cutover. Nothing starts it, and its RunRetraining state targets the tradai-strategy-generic-<env> task definition, which is the freqtradeorg/freqtrade:stable placeholder — it carries no TradAI entrypoint. It now forwards RUN_CONFIG (so the #754 gate is satisfied and a future producer inherits the contract), but making it runnable — a real task definition, and demoting the scheduler to a StartExecution submitter — belongs to #538.