Skip to content

Trader Ledger Persistence (TRADAI_PERSIST_DIR)

Operator reference for the durable Freqtrade trade ledger used by dry-run/live trading (#97 M1). Covers the TRADAI_PERSIST_DIR contract, the fail-fast startup tripwire, how db_url is pinned, and the single-writer caveat.

Why it exists

A paper/live trader's state (open trades, filled orders, PnL) lives in a Freqtrade SQLite file. On ECS that file must sit on the EFS mount, not on the container's ephemeral disk — otherwise every restart, redeploy, or circuit-breaker rollback silently starts the strategy from a blank ledger.

TRADAI_PERSIST_DIR is the env var that points the trader at the durable directory (the EFS mount, /data/persist in the task definition).

The startup tripwire

EntrypointSettings refuses to construct when all of these hold:

  • trading mode is dry-run or live (backtest/train/hyperopt are exempt), and
  • the process is running on ECS (Fargate or EC2 launch type), and
  • TRADAI_PERSIST_DIR is unset or blank/whitespace.

The container exits immediately with:

TRADAI_PERSIST_DIR is required for dry-run/live trading on ECS but is unset —
refusing to start on an ephemeral ledger (likely an EFS rollback; #97 M1).
Restore the EFS task definition or set TRADAI_PERSIST_DIR.

This is deliberate: failing closed beats trading on a ledger that vanishes on the next restart. The most common trigger is a circuit-breaker rollback to an older, EFS-less task definition.

A blank value (TRADAI_PERSIST_DIR=) counts as unset — a templating bug or an empty-string sentinel will trip the wire, not slip past it.

If you hit this: redeploy the strategy on a task definition that has the EFS volume + mount and sets TRADAI_PERSIST_DIR. Do not work around it by unsetting the check.

How db_url is pinned

When TRADAI_PERSIST_DIR is set, the trader writes (SQLite's sqlite: scheme + the absolute path's own leading /, so an absolute mount like /data/persist yields four slashes — sqlite:/// + /data/...):

db_url = sqlite:////data/persist/tradesv3.dryrun.sqlite   # dry-run, TRADAI_PERSIST_DIR=/data/persist
db_url = sqlite:////data/persist/tradesv3.sqlite          # live,    TRADAI_PERSIST_DIR=/data/persist

The ledger location is platform-owned:

  • A db_url in the strategy config is stripped (it is in the derived-key denylist) so a stored config can never repoint the ledger.
  • A db_url in the base config file is overridden when persist is set, and dropped when persist is unset (local/backtest). The base config is trusted platform config, but a stale db_url there would otherwise silently relocate the ledger to a relative tradesv3*.sqlite, so it is removed rather than honored.

TRADAI_PERSIST_DIR must be an absolute path on ECS. A relative path is rejected at settings construction (the container fails to start) because it would resolve against the container CWD onto ephemeral storage and defeat the tripwire. Off ECS (local dev/backtest) a relative path is tolerated — the trader anchors it to the current working directory.

Per-strategy isolation

TRADAI_PERSIST_DIR is expected to be already strategy-specific. Infra mounts a per-slug EFS access point rooted at /strategies/{slug}, so the fixed tradesv3[.dryrun].sqlite filename is unique per strategy. The trader does not add its own strategy_id subdir — that would double-namespace and break the access-point root contract.

Single-writer caveat

SQLite on NFS/EFS tolerates exactly one writer. Two Freqtrade processes against one ledger file corrupt it. This is a deployment property, not something the trader can enforce.

The strategy ECS services (the only writers of the EFS ledger; create_strategy_services in infra/compute/modules/ecs.py) deploy with a single-writer rolling-update policy (max 100% / min healthy 0%), introduced with the EFS-persistence work of #97 M1, which stops the old task before the new one starts so a redeploy does not run two writers against the ledger. (The ALB platform services in ecs_services.py keep the AWS-default max 200% / min healthy 100% — they do not touch the ledger, so that does not affect this.)

Two residual single-writer risks remain operational (they cannot be enforced in code):

  • A manual desired_count bump above 1 on a strategy service.
  • The runtime start/stop path overlapping an old container with a new one.

Treat any "database is locked" / ledger-corruption symptom as a possible two-writer event and confirm exactly one running task before restarting.

Troubleshooting

Symptom Cause Action
Container exits at boot with "TRADAI_PERSIST_DIR is required…" ECS dry-run/live with no/blank persist dir (often an EFS rollback) Redeploy on a task def with the EFS mount + TRADAI_PERSIST_DIR
"Could not create durable trade-DB directory … read-only/No such file" EFS mount missing or unwritable Check the task's EFS volume + access point; verify the SG/NACL NFS path
Ledger looks empty after a restart Ledger was on ephemeral disk, not EFS Confirm db_url points under /data/persist; check the mount
"TRADAI_PERSIST_DIR must be an absolute path … on ECS" (fails at startup) A relative path was supplied on ECS Set an absolute path (the EFS mount point)