SIEM recovery

Use this runbook when a SIEM destination shows a delivery failure, falls behind, or is known to have missed events (for example a suspension during a SIEM outage). Axiom's audit stream is ordered and checkpointed, so recovery is a replay from a sequence number, and receivers that deduplicate on the event ID never count an event twice.

How delivery works

• Every audit event has a stable id (evt_…) and a global, monotonic seq.
• Each destination keeps a checkpoint: the highest seq it has accepted. Delivery sends batches of up to 100 events after the checkpoint and advances it only after the receiver answers 2xx.
• Delivery runs from the durable queue and from the five-minute maintenance sweep. Events less than 30 seconds old are held back if an earlier sequence number is still missing, so a slow writer cannot cause a gap to be skipped.
• On failure the destination becomes failing with last_error, the checkpoint stays where it was, and delivery retries after 60 seconds. Nothing after the failed batch is skipped.
• A new destination starts at the current head of the stream. It receives nothing from before it was created unless you replay.

The SIEM event shape is { id, seq, occurred_at, org_id, tenant_id, project_id, type, action_id, actor: { type, id }, policy_version, evidence_ref, outcome, data }. redaction.drop_fields can remove any dotted path except id and seq.

1. Check the destination

Console: SIEM delivery (/control/orgs/:oid/siem). API:

axiom_get /v1/orgs/$ORG/siem

(The shell helpers are defined in Runbooks.)

Field	Look for
status	active, failing (delivery errors) or disabled.
checkpoint	The last seq the receiver accepted.
lag	How many in-scope events are waiting after the checkpoint.
last_error	The receiver's HTTP status or the network error from the last failed batch.
recent_deliveries[]	Recent batches with their status and error.
scope, redaction	Which tenants and projects are sent, and which fields are dropped.

2. Fix connectivity, then test

Common causes: an expired token behind secret_ref, a rotated HEC token, a firewall change, or a receiver returning 4xx for a payload format change. Formats: webhook and chronicle receive { "events": [...] }, sentinel a JSON array, splunk_hec newline-delimited HEC events with Authorization: Splunk …; the others use Authorization: Bearer … when a secret is configured.

Send a connectivity test (requires org.admin):

axiom_post /v1/siem/sie_01…/test '{}'

The response is { ok, detail }. The test event has type siem.test and seq: 0; it does not move the checkpoint. Once the receiver is healthy, the next flush delivers everything after the checkpoint and the status returns to active on its own.

3. Replay missed events

If events were lost on the receiving side (a SIEM outage that accepted and then dropped data, an index restored from backup, or a new destination that needs history), rewind the checkpoint:

axiom_post /v1/siem/sie_01…/replay '{"from_seq": 18240}'

• The checkpoint is set to from_seq - 1 (it never moves forward through a replay) and a flush is queued. The response is { queued, checkpoint }, where queued counts in-scope events to resend.
• Choose from_seq from the last event the SIEM definitely has. When unsure, go earlier: the receiver must deduplicate on id, so repeated events are harmless.
• The replay is recorded as siem.replay_requested in the audit stream.

4. Verify a specific event arrived

For the case the brief calls out, a suspension missed during an outage:

1. Find the event in Axiom: the incident or agent timeline shows the time; the audit type is agent.suspended (or emergency_deny.published, incident.opened).
2. Search the SIEM for that event id. It should appear exactly once after deduplication.
3. If it is missing, replay from a seq at or before it and check again.

Receiver requirements

• Deduplicate on id, not on content.
• Order by seq if order matters; batches arrive in order, but a replay resends earlier events.
• Answer 2xx only after the batch is durable. A 2xx is what advances the checkpoint.