Trading Session Lifecycle & Liveness¶

Operator reference for how a dry-run/live trading container runs, stops, and is kept honest about liveness (#97 M1). Covers the unbounded-by-default session, the shutdown grace watchdog, and the heartbeat liveness backstop.

Unbounded session by default¶

A paper/live trader runs until it is stopped — there is no default max-runtime timeout. A session cap can still be set explicitly with TRADING_TIMEOUT_HOURS (1–168); leaving it unset (or 0/blank) means "run continuously".

A genuine timeout exits with code 124 and the platform scales the service to 0 (a deliberate cost ceiling, not a crash-restart). A crash (any other non-zero exit) is left to ECS to restart.

Stopping a session (grace watchdog)¶

request_shutdown() sends SIGTERM to Freqtrade for a graceful stop (finish the cycle, flush SQLite, exit). Because the default session timeout is gone, a short grace watchdog is the only thing that still guarantees an eventual force-kill: if Freqtrade ignores SIGTERM past the grace period (env var SHUTDOWN_GRACE_SECONDS, default 30s) while the output stream is blocked, the watchdog force-kills the subprocess. This unblocks the wrapper so run() returns and the cleanup / scale-to-0 path executes — instead of hanging until the platform SIGKILLs the whole container ~30s later (after which our own cleanup never runs).

Liveness backstop (heartbeat attests the subprocess)¶

The HealthReporter heartbeat publishes container health to DynamoDB every ~30s. On its own it only proves DynamoDB is writable — an exited Freqtrade subprocess could otherwise keep publishing green heartbeats forever (no exit 124, no alert), the hardest failure to notice.

So the heartbeat now attests the subprocess has not exited. When the trader has started and the subprocess has since exited without a shutdown being requested (FreqtradeTrader.is_unexpectedly_dead), the reporter:

1. flips DynamoDB status to ERROR,
2. fires a one-shot SNS alert ("Trading subprocess dead: "),
3. invokes the fatal liveness callback, which SIGTERMs the subprocess and preserves the ERROR status (it does not take the graceful STOPPED path), and
4. stops the heartbeat loop.

The container then exits and ECS restarts the task (the service is still at desired_count=1). A liveness death is treated as a crash, so — unlike a genuine TRADING_TIMEOUT_HOURS timeout (exit 124) — it does not scale the service to 0; recovery is an ECS restart-on-death. (If the death is persistent, the restart loop is the signal to investigate, and the ERROR status + SNS alert make it visible.)

The probe is gated on "ever started", so the pre-launch window and deliberate graceful stops do not false-fire. A clean Freqtrade self-exit (e.g. operator /stopbuy, exit 0) is also not treated as a death.

Caveat — this detects an exited subprocess, not a wedged-but-alive one. is_unexpectedly_dead requires the process to have actually exited (poll() non-None). A Freqtrade that deadlocks or stops trading without exiting still reads as alive and keeps publishing green heartbeats. Detecting that liveness-vs-progress case (e.g. last-trade/heartbeat-advance from the API, or a no-activity-for-N-intervals check) is a follow-up beyond M1.

Troubleshooting¶

Related¶

• Trading Issues
• Service Recovery