Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .github/workflows/coolify-token-check.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: Coolify token check

# Daily guard for the coolify-staging COOLIFY_TOKEN used by
# deploy-coolify.yml: fails when Coolify rejects the token or when the
# recorded expiry (COOLIFY_TOKEN_EXPIRES_AT) is within 30 days, so a token
# expiry or rotation shows up as a red run, not as a broken deploy.
# Rotation steps: _docs/runbooks/coolify-api-token.md

on:
schedule:
- cron: "17 6 * * *"
workflow_dispatch:

permissions:
contents: read

jobs:
check:
runs-on: ubuntu-24.04
environment: coolify-staging
steps:
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5
- name: Check the Coolify API token and its expiry
env:
COOLIFY_URL: ${{ secrets.COOLIFY_URL }}
COOLIFY_TOKEN: ${{ secrets.COOLIFY_TOKEN }}
COOLIFY_TOKEN_EXPIRES_AT: ${{ vars.COOLIFY_TOKEN_EXPIRES_AT }}
run: python3 -m deploy.coolify_token check --expires-at "${COOLIFY_TOKEN_EXPIRES_AT}" --warn-days 30
7 changes: 7 additions & 0 deletions .github/workflows/deploy-coolify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,13 @@ jobs:
--sha "$GITHUB_SHA" \
--timeout-seconds 1800 \
--interval-seconds 30
- name: Check the Coolify API token
env:
COOLIFY_URL: ${{ secrets.COOLIFY_URL }}
COOLIFY_TOKEN: ${{ secrets.COOLIFY_TOKEN }}
# An expired or revoked token otherwise surfaces as a bare curl 401
# mid-deploy. Fail here with the cause and the rotation runbook.
run: python3 -m deploy.coolify_token check
- name: Point staging at the pushed release
env:
COOLIFY_URL: ${{ secrets.COOLIFY_URL }}
Expand Down
64 changes: 64 additions & 0 deletions _docs/runbooks/coolify-api-token.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Coolify API tokens for GitHub deploys

The Contabo Coolify control plane runs on contabo-a. GitHub Actions deploy
through its REST API with Sanctum API tokens. Each consumer has its own scoped
token, so one token's expiry or rotation never breaks another repo:

| Consumer | Coolify token name | Abilities | GitHub secret | DataOps secret |
|----------|--------------------|-----------|---------------|----------------|
| `DataTalksClub/website` `deploy-coolify.yml` | `gha-website-deploy-<date>` | read, write, deploy | env `coolify-staging` → `COOLIFY_TOKEN` | `coolify.gha-website-deploy` |
| `DataTalksClub/course-management-platform` `deploy-coolify.yml` | `gha-cmp-deploy-<date>` | read, write, deploy, read:sensitive (reads `docker_compose_raw`) | env `coolify-staging` → `COOLIFY_TOKEN` | `coolify.gha-cmp-deploy` |
| `DataTalksClub/relay` deploy workflows | `gha-relay-deploy-<date>` | read, write, deploy, read:sensitive | repo and env `coolify-staging` → `COOLIFY_TOKEN` | `coolify.gha-relay-deploy` |
| Hetzner `coolify-rolling-web` (production follower), `cmp-release-poller`, aisl tools | `hetzner-ops-<date>` | read, write, deploy, read:sensitive, write:sensitive | `~/.secrets/` and `~/tmp/secrets/coolify-api-169.58.119.163.token` on Hetzner | `coolify.hetzner-ops` |

The current tokens were minted on 2026-11-21 and expire on 2027-11-20.

## Incident: 2026-11-20

The original shared token `agent-setup-2026-10-22` was created with a 30-day
expiry. It expired at 2026-11-20 22:12:50 UTC. The next website deploy (run
44830025858) failed at "Point staging at the pushed release" with a bare
`curl: (22) ... 401`. The production rolling follower and the CMP release
poller on Hetzner used the same token and failed from 22:13 UTC. Nothing
had revoked it; it just expired, and no check looked at the expiry.

## Guards

- `deploy-coolify.yml` runs `python3 -m deploy.coolify_token check` before its
first Coolify call. A rejected token fails with the cause and this runbook
instead of a bare curl error.
- `coolify-token-check.yml` runs daily (and on dispatch). It fails when Coolify
rejects the token, or when the `coolify-staging` variable
`COOLIFY_TOKEN_EXPIRES_AT` (YYYY-MM-DD) is within 30 days or missing. The red
run reaches the GitHub Actions failure routine weeks before a deploy breaks.

## Check token expiry

Token metadata only (never select the `token` column):

```bash
ssh contabo-a "sudo -n docker exec coolify-db psql -U coolify -d coolify -c \
'select id, name, abilities, last_used_at, expires_at from personal_access_tokens order by id;'"
```

## Rotate a token

All four tokens expire on the same day, so rotate them together.

Never print a token. Write it to a mode-600 file, pipe it into the stores, then
delete the file.

1. Mint a token in the Coolify UI (Keys & Tokens → API tokens) on the Root
Team, with only the abilities in the table and a 1-year expiry. Or use
`php artisan tinker` in the `coolify` container:
`session(['currentTeam' => App\Models\Team::find(0)]);
App\Models\User::find(0)->createToken('<name>', [...abilities], now()->addYear())->plainTextToken`.
2. Update the GitHub secret from the file:
`gh secret set COOLIFY_TOKEN --env coolify-staging -R <repo> < <file>`.
3. Update the DataOps secret with `dataops secrets set <name> --from-file -`.
Include every field (`token`, `url`, `notes`): `set` replaces the whole set.
4. For the website, record the expiry:
`gh variable set COOLIFY_TOKEN_EXPIRES_AT --env coolify-staging -R DataTalksClub/website --body <YYYY-MM-DD>`.
5. Dispatch `coolify-token-check.yml` and confirm it is green. Then rerun or
dispatch the deploy.
6. Delete the old token row once nothing uses it (check `last_used_at`).
84 changes: 84 additions & 0 deletions ci/tests/test_coolify_token_guard.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
"""The Coolify API token guard fails loudly on rejection and near expiry."""

from __future__ import annotations

import datetime as dt
from pathlib import Path
from typing import Any

import yaml

from deploy.coolify_token import RUNBOOK, check, decide_auth, decide_expiry

ROOT = Path(__file__).resolve().parents[2]
TODAY = dt.date(2026, 11, 21)


def workflow(name: str) -> dict[str, Any]:
return yaml.load(
(ROOT / ".github" / "workflows" / name).read_text(encoding="utf-8"),
Loader=yaml.BaseLoader,
)


def test_rejected_token_names_the_cause_and_runbook() -> None:
for status in (401, 403):
decision = decide_auth(status)
assert not decision.ok
assert "expired" in decision.message
assert RUNBOOK in decision.message


def test_accepted_token_and_other_failures() -> None:
assert decide_auth(200).ok
assert not decide_auth(500).ok
assert not decide_auth(None, "timed out").ok


def test_expiry_threshold() -> None:
assert decide_expiry("2027-11-20", today=TODAY, warn_days=30).ok
assert not decide_expiry("2026-12-15", today=TODAY, warn_days=30).ok
assert not decide_expiry("2026-11-20", today=TODAY, warn_days=30).ok
assert not decide_expiry("", today=TODAY, warn_days=30).ok
assert not decide_expiry("next year", today=TODAY, warn_days=30).ok


def test_check_skips_expiry_unless_requested_and_rejects_empty_inputs() -> None:
def ok_prober(url: str, token: str) -> tuple[int, str]:
return 200, ""

decisions = check(
url="https://c", token="t", expires_at=None, warn_days=30, today=TODAY, prober=ok_prober
)
assert [d.ok for d in decisions] == [True]
decisions = check(
url="https://c",
token="t",
expires_at="2026-12-01",
warn_days=30,
today=TODAY,
prober=ok_prober,
)
assert [d.ok for d in decisions] == [True, False]
assert not check(
url="", token="t", expires_at=None, warn_days=30, today=TODAY, prober=ok_prober
)[0].ok


def test_deploy_checks_the_token_before_the_first_coolify_call() -> None:
steps = workflow("deploy-coolify.yml")["jobs"]["deploy"]["steps"]
names = [step.get("name", "") for step in steps]
guard = names.index("Check the Coolify API token")
assert guard < names.index("Point staging at the pushed release")
assert "deploy.coolify_token check" in steps[guard]["run"]
assert steps[guard]["env"]["COOLIFY_TOKEN"] == "${{ secrets.COOLIFY_TOKEN }}"


def test_scheduled_check_watches_the_recorded_expiry() -> None:
wf = workflow("coolify-token-check.yml")
assert "schedule" in wf["on"]
job = wf["jobs"]["check"]
assert job["environment"] == "coolify-staging"
step = job["steps"][-1]
assert step["env"]["COOLIFY_TOKEN_EXPIRES_AT"] == "${{ vars.COOLIFY_TOKEN_EXPIRES_AT }}"
assert "--expires-at" in step["run"]
148 changes: 148 additions & 0 deletions deploy/coolify_token.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
"""Fail loudly when the Coolify API token stops working or nears expiry.

The Contabo staging deploy (``.github/workflows/deploy-coolify.yml``) drives
Coolify through its REST API with the ``coolify-staging`` environment secret
``COOLIFY_TOKEN``. Coolify API tokens are Laravel Sanctum tokens and can carry
an expiry: on 2026-11-20 the original token expired at 22:12 UTC and the next
deploy died with a bare ``curl: (22) ... 401`` halfway through the job.

This module turns that into two explicit checks:

* ``check`` asks Coolify whether the token authenticates (``GET
/api/v1/version``) and names the cause and the runbook on a 401/403.
* With ``--expires-at`` (the ``COOLIFY_TOKEN_EXPIRES_AT`` environment variable,
recorded when the token is minted) it also fails once the expiry is within
``--warn-days``, so the scheduled ``coolify-token-check.yml`` workflow goes
red weeks before a deploy can break.

Stdlib-only so the workflows can run it with the runner's ``python3``.
Rotation steps live in ``_docs/runbooks/coolify-api-token.md``.
"""

from __future__ import annotations

import argparse
import datetime as dt
import os
import sys
import urllib.error
import urllib.request
from collections.abc import Callable
from dataclasses import dataclass

RUNBOOK = "_docs/runbooks/coolify-api-token.md"
DEFAULT_WARN_DAYS = 30
TIMEOUT_SECONDS = 20


@dataclass(frozen=True)
class TokenDecision:
ok: bool
message: str


def decide_auth(status: int | None, error: str = "") -> TokenDecision:
"""Classify the HTTP status Coolify returned for an authenticated GET."""
if status is not None and 200 <= status < 300:
return TokenDecision(True, f"Coolify API token authenticates (HTTP {status}).")
if status in (401, 403):
return TokenDecision(
False,
f"Coolify rejected COOLIFY_TOKEN (HTTP {status}): the token expired, "
"was revoked, or lost its abilities. Mint a new scoped token and "
f"update the coolify-staging secret; see {RUNBOOK}.",
)
if status is None:
return TokenDecision(False, f"Coolify API unreachable: {error or 'no response'}.")
return TokenDecision(False, f"Coolify API returned HTTP {status} for the token check.")


def decide_expiry(expires_at: str, *, today: dt.date, warn_days: int) -> TokenDecision:
"""Fail when the recorded token expiry is missing, past, or within warn_days."""
value = expires_at.strip()
if not value:
return TokenDecision(
False,
"COOLIFY_TOKEN_EXPIRES_AT is not set; record the token expiry "
f"(YYYY-MM-DD) as a coolify-staging variable; see {RUNBOOK}.",
)
try:
expiry = dt.date.fromisoformat(value[:10])
except ValueError:
return TokenDecision(
False, f"COOLIFY_TOKEN_EXPIRES_AT={value!r} is not a YYYY-MM-DD date; see {RUNBOOK}."
)
days_left = (expiry - today).days
if days_left < 0:
return TokenDecision(
False, f"COOLIFY_TOKEN expired on {expiry}; rotate it now; see {RUNBOOK}."
)
if days_left <= warn_days:
return TokenDecision(
False,
f"COOLIFY_TOKEN expires on {expiry} ({days_left} days left, "
f"threshold {warn_days}); rotate it before deploys break; see {RUNBOOK}.",
)
return TokenDecision(True, f"COOLIFY_TOKEN expires on {expiry} ({days_left} days left).")


def probe(url: str, token: str) -> tuple[int | None, str]:
"""Return (HTTP status, error text) for GET {url}/api/v1/version."""
request = urllib.request.Request(
f"{url.rstrip('/')}/api/v1/version",
headers={"Authorization": f"Bearer {token}", "Accept": "application/json"},
)
try:
with urllib.request.urlopen(request, timeout=TIMEOUT_SECONDS) as response:
return response.status, ""
except urllib.error.HTTPError as exc:
return exc.code, ""
except (urllib.error.URLError, TimeoutError, OSError) as exc:
return None, str(exc)


def check(
*,
url: str,
token: str,
expires_at: str | None,
warn_days: int,
today: dt.date,
prober: Callable[[str, str], tuple[int | None, str]] = probe,
) -> list[TokenDecision]:
if not url or not token:
return [TokenDecision(False, f"COOLIFY_URL or COOLIFY_TOKEN is empty; see {RUNBOOK}.")]
decisions = [decide_auth(*prober(url, token))]
if expires_at is not None:
decisions.append(decide_expiry(expires_at, today=today, warn_days=warn_days))
return decisions


def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("command", choices=["check"])
parser.add_argument(
"--expires-at",
default=None,
help="Recorded token expiry (YYYY-MM-DD); also fail when within --warn-days.",
)
parser.add_argument("--warn-days", type=int, default=DEFAULT_WARN_DAYS)
args = parser.parse_args(argv)

decisions = check(
url=os.environ.get("COOLIFY_URL", ""),
token=os.environ.get("COOLIFY_TOKEN", ""),
expires_at=args.expires_at,
warn_days=args.warn_days,
today=dt.datetime.now(dt.UTC).date(),
)
for decision in decisions:
if decision.ok:
print(decision.message)
else:
print(f"::error::{decision.message}", file=sys.stderr)
return 0 if all(decision.ok for decision in decisions) else 1


if __name__ == "__main__":
sys.exit(main())
Loading