CI Runners (RunsOn)¶

TradAI's CI compute runs on RunsOn — self-hosted GitHub Actions runners on ephemeral EC2 instances in our own AWS account, standing in for GitHub-hosted runners on migrated jobs. This page covers what it is, the runner classes, the label convention, the kill switch, and how to watch it.

What & where¶

RunsOn is deployed as a single aws.cloudformation.Stack wrapping RunsOn's own CloudFormation template, managed by the Pulumi project infra/ci/ in a dedicated AWS account (tradai-ci, eu-central-1). It is not applied by deploy-infra.yml — apply it by hand:

just infra-preview-ci   # dry-run
just infra-up-ci        # apply

Runner classes & presets¶

Presets live in .github/runs-on.yml, referenced from a workflow via the runs-on=<label>/runner=<preset> label described in Label convention below. All presets are x64 (ubuntu24-full-x64).

Glue jobs (changes, lint-required, test-required, version-bump-guard, version, report-*, notify-*) stay on ubuntu-latest.

deploy-* is deliberately not shared with ci-*: the release path must not consume a cache a PR run could have seeded (deploy-lambdas.yml:133-145 documents the same concern for GitHub's own cache), so deploy-* runners get no s3-cache extra at all.

Label convention¶

Every migrated job resolves its runs-on: label through the RUNS_ON_ENABLED repo variable, so the fleet can be killed with one variable flip. The label convention is run-id-only — no github.job discriminator. github.job renders as null in a runs-on: expression (GitHub documents github.job as unavailable outside job steps, and runs-on: is evaluated before the job starts); this was confirmed with live evidence — RunsOn canary run 32800592037 and a CI dispatch both showed the empty render. github.job must never appear in a label, and tests/unit/test_runs_on_labels.py rejects any label that contains it.

Plain job:

runs-on: ${{ vars.RUNS_ON_ENABLED == 'true' && format('runs-on={0}/runner=<preset>', github.run_id) || 'ubuntu-latest' }}

Matrix job — discriminate on strategy.job-index instead:

runs-on: ${{ vars.RUNS_ON_ENABLED == 'true' && format('runs-on={0}-{1}/runner=<preset>', github.run_id, strategy.job-index) || 'ubuntu-latest' }}

_test.yml reusable callers pass a runner: input built the same way, keyed off matrix.package:

runner: ${{ format('runs-on={0}-test-{1}/runner=ci-4', github.run_id, matrix.package) }}

The presets canary (fans out one job per preset) builds its label directly rather than through format(), discriminating on strategy.job-index the same way a matrix job does:

runs-on: runs-on=${{ github.run_id }}-${{ strategy.job-index }}/runner=${{ matrix.preset }}

Jobs within one run that end up sharing a plain runs-on=<run_id>/runner=<preset> label are interchangeable by design: RunsOn launches one runner per queued job, so two jobs with the same label just get two separate runners — only a matrix fan-out needs the extra discriminator to keep its parallel jobs distinct.

A reusable-workflow caller job (e.g. uses: ./.github/workflows/_lint.yml or _test.yml) has no runs-on: or steps: of its own, so it will never show runs-on: or runs-on/action — but its with: runner: input is where the kill-switch expression, the preset, and (for a matrix caller) the matrix discriminator are actually supplied. The called workflow's job is what consumes that value (runs-on: ${{ inputs.runner }}) and carries the runs-on/action first step. When auditing a job (e.g. against the PR template's "CI runner label" checklist item), review both: the caller's with: runner: for the label expression/preset/ matrix key, and the called workflow's runs-on:/runs-on/action for how it's consumed. (The runner input on _lint.yml/_test.yml lands with the light-jobs migration PR; until then those two reusable workflows run on ubuntu-latest and have nothing to audit.)

The first step of every migrated job is:

- uses: runs-on/action@46910bf61b41721b0579f237e186afb35477007a  # v2.3.0

This step is a no-op on GitHub-hosted runners, so it's safe to leave in place even when the kill switch has fallen back to ubuntu-latest.

The typo trap¶

A misspelled preset name doesn't fail the job outright — GitHub just never finds a runner that satisfies the label, so it queues until GitHub's ~24 h queued-job timeout cancels it — a typo surfaces as a failure only after a ~24 h delay, not immediately; tests/unit/test_runs_on_labels.py is the pre-merge guard. Actionlint cannot catch this: the label is a computed expression, not a static string, so there's nothing for a linter to check against .github/runs-on.yml. That test asserts every runs-on=.../runner=<preset> label in the workflow files names a preset that actually exists in .github/runs-on.yml.

Kill switch & rollback¶

Fleet outage (spot capacity, GitHub App broken, licence problem) — first response:

gh variable set RUNS_ON_ENABLED --body false

Every migrated label reads that variable, so the next run of every workflow lands on GitHub-hosted runners with no PR and no merge required. Unset (or never set) also means GitHub-hosted. Required checks keep flowing either way. Flip it back to true once the cause is fixed.

Per-job rollback: delete the ternary for that one job so it always resolves to ubuntu-latest.

Monitoring¶

The first canary ran all seven presets; queue + boot came in at 25–28 s per job across the board — use that as the baseline when judging whether a run's p50/p95 below looks off.

Queue + boot time (p50/p95/max) for a run:

gh api "repos/tradai-bot/tradai/actions/runs/$RUN/jobs?per_page=100" --jq '[.jobs[] | select(.labels[0] | startswith("runs-on=")) | ((.started_at|fromdate)-(.created_at|fromdate))] | sort | {n: length, p50: (if length == 0 then null else .[((length * 0.5 | ceil) - 1)] end), p95: (if length == 0 then null else .[((length * 0.95 | ceil) - 1)] end), max: max}'

Stuck jobs (queued > 10 min usually means a label typo or a capacity shortage). gh run list --status queued lists runs, not jobs, and a run can show in_progress while one of its jobs is still queued — query the jobs API instead:

for st in queued in_progress; do
  gh run list --status "$st" --limit 30 --json databaseId --jq '.[].databaseId'
done | while read -r r; do
  gh api "repos/tradai-bot/tradai/actions/runs/$r/jobs?per_page=100" \
    | jq -r --argjson now "$(date +%s)" --arg run "$r" \
      '.jobs[] | select(.status=="queued" and (.labels[0] // "" | startswith("runs-on=")) and (($now - (.created_at|fromdate)) > 600)) | "\($run)\t\(.name)\t\(($now - (.created_at|fromdate)))s"'
done

(gh api has no --argjson flag — that's plain jq's; pipe its output into a separate jq instead. gh run list --status also only accepts one value, so a second --status silently overrides the first — loop over the statuses instead of repeating the flag.)

Any row printed = label typo or capacity problem.

Spot interruptions: jobs with run_attempt > 1 on ci-* jobs, cross-checked against the RunsOn CloudWatch log group filtered for interrupt. The RunsOn service itself (not the ephemeral runners) logs to /aws/ecs/runs-on/runs-on-worker in the tradai-ci account:

aws logs tail /aws/ecs/runs-on/runs-on-worker --since 15m --profile tradai-ci

Get the log group name programmatically from the infra/ci Pulumi stack output — it's the same name, just resolved instead of hardcoded:

just infra-outputs-ci | jq -r .service_log_group_name

If the stack output isn't available, fall back to the raw CloudFormation output (RunsOnServiceLogGroupName):

aws cloudformation describe-stacks --stack-name runs-on --profile tradai-ci \
  --query "Stacks[0].Outputs[?OutputKey=='RunsOnServiceLogGroupName'].OutputValue" --output text

Cost: the per-job cost line the runs-on/action step prints in that step's own post-step log segment (gh run view <id> --log | grep -i cost), and AWS Cost Explorer grouped by the tradai-ci-class tag.

Operating the stack¶

The GitHub App used to register runners with GitHub is created once via the stack's RunsOnEntryPoint output. After initial setup, infra/ci config sets enable_admin_routes to false and protect to true, so routine operation doesn't expose the admin surface and the stack can't be destroyed by accident.

Besides service_log_group_name, the Pulumi stack also outputs entry_point (the RunsOn control-plane URL used for GitHub App setup) and license_status_parameter_name (the SSM parameter, /runs-on/license/status, the licence health check reads). Read any of them the same way:

just infra-outputs-ci | jq -r .entry_point
just infra-outputs-ci | jq -r .license_status_parameter_name