Post-Saturation Recovery

This runbook describes how an operator returns the OhShii Shield to its normal posture after a saturation event — a Sybil burst or cycle-drain attack that drove one or more protected endpoints into their high-load (Defcon) state, and possibly prompted an operator to engage the kill switch.

The single most important thing to know first: almost all of this is self-healing. The Shield’s high-load state, its rate-limit windows, and its pre-auth cache all converge back to normal on their own once attack traffic stops. The manual steps below exist only to accelerate return-to-normal so that legitimate callers are not left waiting out a window — they are not required for the system to recover.

The one exception is the kill switch: if an operator manually moved a canister into the TrustedOnly lockdown, that mode is sticky by design and stays engaged until an operator (or a governance vote) moves it back. Everything else clears automatically.

Step 0 — Read the current state before changing anything

Every Shield-bearing canister exposes a free public query that returns an aggregate-only snapshot (no per-caller data):

get_shield_rate_limit_status() -> PublicShieldStatus

Read `state` FIRST

state is the only field that can tell you the canister did not measure anything. Read it before the rest. If it is Disabled or Unknown, the fields below are not measurements on that canister — they are defaults, and a false or a 0 from them is not evidence of health.

That matters most on dao_storage and lock_storage, which run no Shield pipeline at all. Their any_defcon_active is a hardcoded false (src/storage/src/lib.rs:4589, locker src/storage/src/lib.rs:3120) sitting beside state: Unknown — an affirmative claim of health from a canister that did not look. Their admin_reset_rate_limit_counter / admin_flush_preauth_cache are documented no-ops for the same reason.

Then note these, with their current status — the annotations are as of 2026-08-24 and each carries a row id so you can check whether it has been wired since:

field	status today	source
`preauth_mode`	LIVE — `Enforce` / `TrustedOnly` / `Disabled`	—
`global_window_count` / `_max` / `_start_ns`	LIVE — real counters	`preauth_api.rs:329-331`
`any_defcon_active`	LIVE on Shield canisters; hardcoded `false` on the two storage canisters — they run no Shield pipeline	—
`rate_limited_count`	CONSTANT ZERO — literal `0` in the only constructor	`preauth_api.rs:333`
`preauth_bypass_count`	CONSTANT ZERO — same constructor	`preauth_api.rs:334`
`preauth_reject_count`	CONSTANT ZERO — same constructor	`preauth_api.rs:335`
`positive_hits` / `fresh_misses`	CONSTANT ZERO at the publisher	see the note below
`push_counters` (ONS / SONS)	ALL FIELDS ZERO — `PublicPushCounters::default()`	`ohshii_governance/src/lib.rs:10892`, `sons_governance/src/lib.rs:15008`

⚠ positive_hits / fresh_misses are not simply unmeasured — they are measured and not published. The canister does increment step_p_positive_hits / step_p_fresh_misses internally (ohshii_governance/src/preauth.rs:583, sons_governance/src/preauth.rs:558), but the public field is written as a literal 0 beside them (ohshii_governance/src/preauth.rs:1131, sons_governance/src/preauth.rs:1098, backend/src/preauth.rs:2012), and the internal counter appears in no .did. So during an incident you cannot reach the live number by any public call. Any hit-rate derived from these is 0/0.

⚠⚠ What this means for you at 3am: none of the CONSTANT-ZERO fields above can report a problem. A zero from them is indistinguishable from a healthy zero and from a field nobody wired. Do not conclude “no rate limiting occurred” or “the cache is cold” from them. The fields that can still tell you something are state, preauth_mode, global_window_count against its max, and any_defcon_active on a canister whose state is not Unknown.

ⓘ These fields are unpopulated, not broken — some have no producer, some are ::default(), and some are honest stubs on a canister with no Shield pipeline. Their disposition is an open owner decision (queue Q8); nothing here should be read as a decided remedy.

This snapshot tells you which of the steps below you actually need. In the common case — the attack has stopped and no operator touched the kill switch — you will find any_defcon_active = false and preauth_mode = Enforce already, and there is nothing to do but confirm.

Step 1 — Confirm the high-load (Defcon) state has de-escalated on its own

The high-load state is automatic, per-endpoint, and self-clearing: it rises only from admitted load (never from rejections, so a rate-limit storm cannot pin it high) and falls on its own once an endpoint’s admitted rate drops back to/under the low watermark and the anti-flap hold has elapsed — applied on the next admitted call. So once attack traffic drops, the endpoint self-clears within the hold window with no operator action required. For the full mechanism (and why the thresholds are not published), see Defcon: the high-load state.

What to check:

In the status snapshot, any_defcon_active should return to false, and each endpoint’s global_window_count in rate_limits should fall well below its max as the rolling window drains.
If any_defcon_active is still true while global_window_count is already low, that is expected for a short period — the anti-flap hold has not elapsed yet, or no call has come through to apply the transition. Re-query after the hold window rather than intervening.

There is intentionally no operator command to force-clear the high-load state. It is meant to clear itself from observed load, and forcing it would defeat the anti-flap protection. If it does not clear after traffic has demonstrably stopped and the hold window has passed, treat that as a signal the attack is still ongoing, not as a stuck state.

Step 2 — If an operator engaged the kill switch, return the mode to `Enforce`

The pre-auth mode is the only lever that stays where an operator put it. During a severe event an operator may have moved one or more canisters into TrustedOnly, the lockdown mode in which non-trusted public callers are rejected cheaply before they can consume rate-limit, verification, or proof-of-work cycles. Trusted callers (governance, guardians, self-calls) keep flowing in TrustedOnly, so the ecosystem is never frozen — but public access is denied until the mode is reverted.

This mode is orthogonal to the automatic high-load state: clearing Defcon does not lift TrustedOnly, and you must move the mode back to Enforce deliberately.

There are two ways to do it.

Per canister (targeted, immediate):

shield_set_preauth_mode(Enforce)

Call it on each canister still reporting preauth_mode = TrustedOnly in its status snapshot.

Ecosystem-wide (one call across the trusted core): the ONS governance canister acts as a coordinator and fans the mode change out to the core canisters in a single call:

coordinate_shield_kill_switch(Enforce, opt "post-incident recovery")

The coordinator returns a per-target outcome vector and does not short-circuit on a partial failure, so re-issue only to the targets whose result was an error. The same fan-out is available through the DAO proposal path (SetShieldPreauthMode) when the reversal should be vote-paced rather than operator-immediate. Per-campaign (SONS) canisters are deliberately excluded from the coordinator fan-out and, if one was locked down individually, are reverted individually with shield_set_preauth_mode(Enforce).

After reverting, re-query get_shield_rate_limit_status() and confirm preauth_mode = Enforce on every affected canister.

Step 3 — (Optional) Reset the per-endpoint rate counters

Even after attack traffic stops, a saturated endpoint’s rolling global window can keep legitimate callers in a back-off state until the window naturally drains. To hand that window back immediately for a specific endpoint:

admin_reset_rate_limit_counter("<endpoint_id>", opt pow)

This clears both the global cross-caller counter and the per-caller counters for that endpoint id (for example purchase_tokens_icrc2, claim_tokens, prepare_ico_creation). It is a convenience that shortens the wait; the counters would drain on their own regardless.

This endpoint exists on the core Shield-bearing canisters. On the storage canisters, which run no Shield pipeline, it is a documented no-op and returns OHSHII_RESET_NO_SHIELD_PIPELINE — that is expected, not an error.

Step 4 — (Optional) Force a pre-auth cache refresh

The pre-auth cache is the mechanism that lets known-eligible callers — verified users, and LGE contributors with a contribution of record — skip the global cap during a saturation event instead of being caught behind it. The cache is heap-only and resets on upgrade; after an incident or a deploy it repopulates on its own through three paths: organic recording on each caller’s first successful call, the one-way trusted push from ONS when a user verifies, and the scheduled refresh. None of these require an operator.

To converge the cache immediately rather than waiting for those paths — for example right after an upgrade, so eligible callers regain the cap-skip without a cold-start retry — force a refresh of the relevant context:

refresh_preauth_cache(<context>)

A single refresh re-pulls and replaces the entire eligible set for that context from the source of truth in one shot, so one successful refresh re-admits the whole known-eligible population at once. Caller-side budgets bound how often a full snapshot can be retaken, not how many principals converge. If a refresh is being short-circuited by de-duplication and you need to force it unconditionally, use the guardian variant guardian_refresh_preauth_cache(<context>).

This step never grants access by itself: a cache entry only earns the global-cap skip. Every other Shield layer — per-caller limits, the high-load state, verification, and the endpoint’s own authorization — still runs under consensus.

Step 5 — Verify return to normal

Re-query the aggregate status on every canister you touched and confirm:

any_defcon_active = false.
preauth_mode = Enforce everywhere it should be.
In rate_limits, global_window_count is well below max and rate_limited_count has stopped climbing.
In cache_stats / refresh_stats, the expected contexts show a non-empty entries_count and a recent successful refresh.
recent_events shows the mode change(s) you issued and no fresh escalation markers.

Because the status query carries only aggregate counts and the reporting canister’s own id — no per-caller principals — it is safe to consult and to share in an incident write-up.

Summary

Symptom	Mechanism	Operator action
Endpoint still in high-load state	Automatic, load-driven, anti-flap hold	None — confirm it self-clears on the next admitted call after the hold elapses
Canister stuck in `TrustedOnly`	Sticky kill-switch mode, orthogonal to the high-load state	Revert to `Enforce` per canister, or ecosystem-wide via the ONS coordinator / DAO proposal
Legitimate callers backing off after traffic stopped	Rolling rate window still draining	Optional `admin_reset_rate_limit_counter("<endpoint_id>")`
Eligible callers hitting the global cap after an upgrade	Cache empty / not yet converged	Optional `refresh_preauth_cache(<context>)`
Confirm normal	Aggregate status query	`get_shield_rate_limit_status()`

The high-load state, the rate windows, and the cache all recover without intervention. The only step that is ever required is reverting the kill switch, because that is the one posture an operator sets by hand.