Post-Saturation Recovery

This runbook describes how an operator returns the OhShii Shield to its normal posture after a saturation event — a Sybil burst or cycle-drain attack that drove one or more protected endpoints into their high-load (Defcon) state, and possibly prompted an operator to engage the kill switch.

The single most important thing to know first: almost all of this is self-healing. The Shield’s high-load state, its rate-limit windows, and its pre-auth cache all converge back to normal on their own once attack traffic stops. The manual steps below exist only to accelerate return-to-normal so that legitimate callers are not left waiting out a window — they are not required for the system to recover.

The one exception is the kill switch: if an operator manually moved a canister into the TrustedOnly lockdown, that mode is sticky by design and stays engaged until an operator (or a governance vote) moves it back. Everything else clears automatically.

Step 0 — Read the current state before changing anything

Every Shield-bearing canister exposes a free public query that returns an aggregate-only snapshot (no per-caller data):

get_shield_rate_limit_status() -> PublicShieldStatus

Call it on each affected canister and note four things:

any_defcon_active — is any endpoint still in the high-load state?
preauth_mode — Enforce (normal), TrustedOnly (kill switch engaged), or Disabled.
rate_limits — per endpoint: global_window_count against its max, plus rate_limited_count, preauth_bypass_count, preauth_reject_count.
cache_stats / refresh_stats — per context cache size, hit/miss counts, and the outcome of recent refreshes.

This snapshot tells you which of the steps below you actually need. In the common case — the attack has stopped and no operator touched the kill switch — you will find any_defcon_active = false and preauth_mode = Enforce already, and there is nothing to do but confirm.

Step 1 — Confirm the high-load (Defcon) state has de-escalated on its own

The high-load state is automatic, per-endpoint, and self-clearing: it rises only from admitted load (never from rejections, so a rate-limit storm cannot pin it high) and falls on its own once an endpoint’s admitted rate drops back to/under the low watermark and the anti-flap hold has elapsed — applied on the next admitted call. So once attack traffic drops, the endpoint self-clears within the hold window with no operator action required. For the full mechanism (and why the thresholds are not published), see Defcon: the high-load state.

What to check:

In the status snapshot, any_defcon_active should return to false, and each endpoint’s global_window_count in rate_limits should fall well below its max as the rolling window drains.
If any_defcon_active is still true while global_window_count is already low, that is expected for a short period — the anti-flap hold has not elapsed yet, or no call has come through to apply the transition. Re-query after the hold window rather than intervening.

There is intentionally no operator command to force-clear the high-load state. It is meant to clear itself from observed load, and forcing it would defeat the anti-flap protection. If it does not clear after traffic has demonstrably stopped and the hold window has passed, treat that as a signal the attack is still ongoing, not as a stuck state.

Step 2 — If an operator engaged the kill switch, return the mode to `Enforce`

The pre-auth mode is the only lever that stays where an operator put it. During a severe event an operator may have moved one or more canisters into TrustedOnly, the lockdown mode in which non-trusted public callers are rejected cheaply before they can consume rate-limit, verification, or proof-of-work cycles. Trusted callers (governance, guardians, self-calls) keep flowing in TrustedOnly, so the ecosystem is never frozen — but public access is denied until the mode is reverted.

This mode is orthogonal to the automatic high-load state: clearing Defcon does not lift TrustedOnly, and you must move the mode back to Enforce deliberately.

There are two ways to do it.

Per canister (targeted, immediate):

shield_set_preauth_mode(Enforce)

Call it on each canister still reporting preauth_mode = TrustedOnly in its status snapshot.

Ecosystem-wide (one call across the trusted core): the ONS governance canister acts as a coordinator and fans the mode change out to the core canisters in a single call:

coordinate_shield_kill_switch(Enforce, opt "post-incident recovery")

The coordinator returns a per-target outcome vector and does not short-circuit on a partial failure, so re-issue only to the targets whose result was an error. The same fan-out is available through the DAO proposal path (SetShieldPreauthMode) when the reversal should be vote-paced rather than operator-immediate. Per-campaign (SONS) canisters are deliberately excluded from the coordinator fan-out and, if one was locked down individually, are reverted individually with shield_set_preauth_mode(Enforce).

After reverting, re-query get_shield_rate_limit_status() and confirm preauth_mode = Enforce on every affected canister.

Step 3 — (Optional) Reset the per-endpoint rate counters

Even after attack traffic stops, a saturated endpoint’s rolling global window can keep legitimate callers in a back-off state until the window naturally drains. To hand that window back immediately for a specific endpoint:

admin_reset_rate_limit_counter("<endpoint_id>", opt pow)

This clears both the global cross-caller counter and the per-caller counters for that endpoint id (for example purchase_tokens_icrc2, claim_tokens, prepare_ico_creation). It is a convenience that shortens the wait; the counters would drain on their own regardless.

This endpoint exists on the core Shield-bearing canisters. On the storage canisters, which run no Shield pipeline, it is a documented no-op and returns OHSHII_RESET_NO_SHIELD_PIPELINE — that is expected, not an error.

Step 4 — (Optional) Force a pre-auth cache refresh

The pre-auth cache is the mechanism that lets known-eligible callers — verified users, and LGE contributors with a contribution of record — skip the global cap during a saturation event instead of being caught behind it. The cache is heap-only and resets on upgrade; after an incident or a deploy it repopulates on its own through three paths: organic recording on each caller’s first successful call, the one-way trusted push from ONS when a user verifies, and the scheduled refresh. None of these require an operator.

To converge the cache immediately rather than waiting for those paths — for example right after an upgrade, so eligible callers regain the cap-skip without a cold-start retry — force a refresh of the relevant context:

refresh_preauth_cache(<context>)

A single refresh re-pulls and replaces the entire eligible set for that context from the source of truth in one shot, so one successful refresh re-admits the whole known-eligible population at once. Caller-side budgets bound how often a full snapshot can be retaken, not how many principals converge. If a refresh is being short-circuited by de-duplication and you need to force it unconditionally, use the guardian variant guardian_refresh_preauth_cache(<context>).

This step never grants access by itself: a cache entry only earns the global-cap skip. Every other Shield layer — per-caller limits, the high-load state, verification, and the endpoint’s own authorization — still runs under consensus.

Step 5 — Verify return to normal

Re-query the aggregate status on every canister you touched and confirm:

any_defcon_active = false.
preauth_mode = Enforce everywhere it should be.
In rate_limits, global_window_count is well below max and rate_limited_count has stopped climbing.
In cache_stats / refresh_stats, the expected contexts show a non-empty entries_count and a recent successful refresh.
recent_events shows the mode change(s) you issued and no fresh escalation markers.

Because the status query carries only aggregate counts and the reporting canister’s own id — no per-caller principals — it is safe to consult and to share in an incident write-up.

Summary

Symptom	Mechanism	Operator action
Endpoint still in high-load state	Automatic, load-driven, anti-flap hold	None — confirm it self-clears on the next admitted call after the hold elapses
Canister stuck in `TrustedOnly`	Sticky kill-switch mode, orthogonal to the high-load state	Revert to `Enforce` per canister, or ecosystem-wide via the ONS coordinator / DAO proposal
Legitimate callers backing off after traffic stopped	Rolling rate window still draining	Optional `admin_reset_rate_limit_counter("<endpoint_id>")`
Eligible callers hitting the global cap after an upgrade	Cache empty / not yet converged	Optional `refresh_preauth_cache(<context>)`
Confirm normal	Aggregate status query	`get_shield_rate_limit_status()`

The high-load state, the rate windows, and the cache all recover without intervention. The only step that is ever required is reverting the kill switch, because that is the one posture an operator sets by hand.