Unreachable gateways

Use this runbook when a customer-hosted gateway or gateway adapter stops reporting to Control. It explains what an unreachable gateway can still do, how to read its freshness, and how to move execution away from it safely.

Status

Control's side of customer-hosted gateways (registration, signed heartbeats, signed policy bundles, acknowledgements, executor ownership) is implemented. The customer-hosted gateway container and the Kong, Apigee and AWS API Gateway adapters are not part of this preview; adapters are candidates that need certification against the common tests.

How freshness works

A gateway authenticates with its own key. Every heartbeat (POST /v1/gateways/:gid/heartbeat, a compact JWS over { gateway_id, policy_version_acked, acks, iat, jti }, accepted within ±120 s and once per jti) returns a fresh bundle signed by Axiom's policy key: active policies, trust data (issuer keys, trust profiles, suspended agents, revoked credentials and grants) and every emergency deny that can affect the project. A bundle is valid for 300 seconds (fresh_until).

Freshness	Meaning
fresh	The last bundle it received is still valid.
stale	The bundle has lapsed, but the gateway made contact in the last 15 minutes.
offline	No contact for more than 15 minutes, or never.

During a partition a gateway may evaluate operations allowed by a still-valid bundle. Once the bundle lapses, sensitive writes stay blocked or pending within their deadlines. An unreachable gateway never receives new emergency denies, so it cannot acknowledge them.

1. Find the gateway and what it affects

Console: Fleet and gateways (/control/orgs/:oid/fleet). Each project row shows its gateways, freshness, last_contact_at, policy_version_acked, fresh_until, the project's current executor and unacknowledged changes. API:

axiom_get /v1/orgs/$ORG/fleet
axiom_get /v1/projects/prj_01…/gateways

If an incident is open, the Incident detail screen (GET /v1/incidents/:id) lists the gateway under offline_gateways[] with freshness, last_contact_at and cached_bundle_valid_until: the time until which the gateway may keep relying on the bundle it holds. The incident stays open (not contained) until that gateway acknowledges every deny.

2. Decide whether to move execution

Each project has exactly one executor: either one active gateway holding the executor flag, or the Sectigo-hosted gateway when no gateway holds it. Claims are per action and never move: actions an unreachable gateway already claimed stay with it.

• If the gateway is not the executor, there is nothing to transfer. Keep it registered until it returns, or retire it.
• If it is the executor and you need execution elsewhere, drain it:

# hand execution back to the Sectigo-hosted gateway
axiom_post /v1/gateways/gwy_01…/status '{"status":"draining"}'

# or to another active gateway in the same project
axiom_post /v1/gateways/gwy_01…/status '{"status":"draining","transfer_to":"gwy_02…"}'

The response includes project_executor (a gateway ID or "hosted"). Requires project.admin.

Every Axiom path honours the executor flag. While a customer gateway holds it, the REST resume, the hosted HTTP and MCP proxies and the Axiom MCP server refuse to claim (409 invalid_state, "execution for this project is owned by gateway …"), and an allowed submit is recorded as approved without immediate execution. Draining the gateway (above) is therefore what lets the hosted gateway execute again; embedded-SDK services are unaffected, because their provider executes.

3. Deal with work it claimed

Console: Project → Actions, state executing or outcome_unknown; Action detail shows claimed_by in the timeline. Work claimed by the unreachable gateway resolves when the gateway reports the outcome or reconciliation reaches the provider (see Missing receipts). Do not resubmit it.

4. Retire, or wait for it to return

A gateway can be retired only when it is not the executor and has no claimed actions still executing:

axiom_post /v1/gateways/gwy_01…/status '{"status":"retired"}'

Retired gateways cannot heartbeat or read bundles again; register a new gateway (new key) to replace one. If the gateway comes back instead, its first heartbeat acknowledges outstanding denies and returns it to fresh; then set it active (and executor: true if it should own execution again).

What users see

Agents talking to an unreachable gateway see whatever it returns locally, typically a pending result for sensitive writes once its bundle has lapsed. Through Axiom-hosted paths nothing changes. Control shows the gap between hosted enforcement (local_block.hosted_enforced) and acknowledged gateways separately, so an offline node is never reported as contained.