Home / Notes / AC Failover Mass AP Offline Post-Mortem
NOTES · WIRELESS / CHASSIS FAILOVER TROUBLESHOOTING

Chassis Failover Knocked All APs Offline: A Post-Mortem

On an S7706 campus core cluster, one command meant as a same-site escape hatch — reboot chassis 1 — took every downstream AP offline at once. This is the forensic timeline, the exact commands that reconstructed it, the demote-then-repromote AC failover mechanism behind the batch delete, and the wiring change that stopped it recurring.

By the AtlasCommTech engineering team — 13 years of carrier & enterprise network deployments · Updated July 2026

All APs Dropping At Once Looks Like a Wireless Problem. It Wasn't.

The AC pair, the AP radios and the CAPWAP tunnels were never the fault — the trigger was two hops away, in a switch reboot.

On an S7706 V200R024C00SPC500 cluster acting as the campus core, an engineer ran reboot chassis 1 — a documented escape-hatch command for restarting one chassis of a stacked/clustered system when physically power-cycling it on site isn't an option. Every downstream AP dropped at once, and the logs recorded a wave of interface-Down events. Nothing about the wireless side had actually failed.

What follows: the forensic timeline reconstructed from display reset-reason and display ap offline-record, the mechanism — a VRRP demote-then-repromote blip that triggers a batch delete of AP sessions — and the topology change that lab-verified stops it recurring on the same reboot command.

The Forensic Timeline

Three separate log sources, cross-referenced, tell a story that none of them tells alone.

03:21:36 — reboot chassis 1 issued display reset-reason: chassis[1] board[4] reset, “Reset for reboot chassis command” 03:21:45–46 — Interface board restart, one at a time Standby's logs record the peer (master) interfaces going Down — the standby's own physical port state never actually changes. Same window — VRRP hellos leak through intermittently AC-master—S7706-master—S7706-standby—AC-standby path stays partially up during the staged reboot, instead of cutting cleanly. AC-standby: Backup → Master → Backup → Master A false promotion followed almost immediately by a demotion — it's specifically that demote step that triggers a batch delete. display ap offline-record all — reason: “Batch delete” Every AP the standby AC had begun adopting gets dropped and must reconnect — the CAPWAP heartbeat-timeout entries reflect the same window. 03:21:50–51 / 04:34:02 — Trunk members recover, chassis rejoins Eth-Trunk members “went Up” as the cluster settles; a second reset

Diagram labels are kept in English for engineering clarity.

Read separately, each log source tells only part of the story — reset-reason shows one clean event, the interface logs show a staged sequence, and ap offline-record shows a batch delete that neither of the other two explains on its own.

The Three Commands That Reconstructed It

None of these three outputs alone explains a mass AP drop — cross-referencing them does.

1. display reset-reason — Confirm What Actually Reset, and When

On the S7706 master, this shows exactly one relevant event: the interface board that reset, and why.

<HUAWEI> display reset-reason
... ...
Info: The LPU chassis[1] board[1] does not have reset records.
Info: The LPU chassis[1] board[2] does not have reset records.
Info: The LPU chassis[1] board[3] does not have reset records.
The LPU chassis[1] board[4]'s reset time is 1 in total. Detailed information:
-- 1. 2025/03/07 03:21:36, Reset No.: 1
        Reason: Reset for reboot chassis command
Info: The LPU chassis[1] board[5] does not have reset records.
Info: The LPU chassis[1] board[6] does not have reset records.
Info: The MPU chassis[1] board[8]'s reset time is 2 in total. Detailed information:
-- 1. 2025/03/07 04:34:02, Reset No.: 2
        Reason: Reset for chassis combine
-- 2. 2025/02/28 16:53:02, Reset No.: 1
        Reason: Reset for chassis combine

This is the piece that's easy to misread: one clean “Reset for reboot chassis command” event looks like proof the failover was equally clean. It isn't proof of anything on the AC side — reset-reason only records chassis and board resets, not VRRP state transitions.

2. The Interface Logs — Whose Interface Actually Went Down

The standby chassis's own log is explicit that its physical interfaces never changed state — what it logged was the peer's.

Mar 7 2025 03:21:45+08:00 TCLY-02G07-201.253 %%01IFPDT/4/IF_STATE(I)[3125]:Interface
XGigabitEthernet1/6/0/12 has turned into DOWN state.
Mar 7 2025 03:21:46+08:00 TCLY-02G07-201.253 %%01IFPDT/4/IF_STATE(I)[3128]:Interface
XGigabitEthernet1/6/0/24 has turned into DOWN state.
// recorded on the standby's log, but this is the master's interface state, propagated
// over the inter-chassis link -- the standby's own physical port state never went Down

Mar 7 2025 03:21:50+08:00 TCLY-02G07-201.253 %%01TRUNK/S/MEMBER_UP(I)[3191]:The status of the
trunk member went Up.(TrunkName=Eth-Trunk12, PortName=XGigabitEthernet2/6/0/1)
// Eth-Trunk members recorded "went Up" as an interface-state refresh once the
// standby had taken over -- not a sign the physical link had ever actually dropped

3. display ap offline-record all — The Fingerprint That Names the Mechanism

This is the command that actually explains the mass drop — the reason field says “Batch delete,” not a radio or heartbeat fault.

<HUAWEI> display ap offline-record all
... ...
MAC              Last offline time       Reason
------------------------------------------------------------------------------
0425-c58b-ea80 2025-03-07/04:24:10 Heartbeat packet transmission for the CAPWAP control tunnel
between the AC and AP times out.
0425-c58b-ea80 2025-03-07/03:15:36 Batch delete
04bd-702b-8aa0 2025-03-07/04:24:10 Heartbeat packet transmission for the CAPWAP control tunnel
between the AC and AP times out.
... ...

Two different reason codes on the same AP tell two different parts of the story: “Batch delete” is the signature of the AC-side demote-then-repromote event described below; the CAPWAP heartbeat-timeout entries are the ordinary consequence of the same window of interface instability upstream.

The Mechanism: Why a Staged Reboot Fools VRRP Into Flapping

reboot chassis 1 is an escape hatch, not a clean power-cycle — and the difference between the two is exactly what caused this.

reboot chassis 1 is documented as an escape-hatch command, meant specifically for situations where the chassis can't be power-cycled on site, and it carries its own usage restrictions. When it runs, the target chassis first notifies its peer that the stack/cluster is splitting, then restarts its interface boards one at a time — it cannot guarantee that all traffic stops simultaneously the way an actual power-cycle would.

The VRRP advertisement path in this topology runs AC-master — S7706-master — S7706-standby — AC-standby. Because the S7706 master's interface boards go Down one at a time during the staged reboot, rather than all at once, VRRP hello packets keep leaking through intermittently instead of stopping cleanly the instant the reboot begins. AC-standby sees the hellos break, promotes itself to Master — then, almost immediately, sees hellos resume and demotes back to Backup, before promoting to Master again once the failover genuinely completes. It's specifically that demote-right-after-a-false-promotion transition that triggers a batch delete of every AP session the standby AC had started adopting.

This same discipline — don't trust a clean top-level status until you've checked whether a control-plane process actually glitched underneath it — is the same one we walk through in our switch CPU-high troubleshooting note: a device can look structurally fine on the surface while a brief control-plane event underneath it does the real damage.

Underneath all of it sits a topology gap: the as-built network has no redundant link between AC-master/S7706-standby or AC-standby/S7706-master, so the VRRP hello path has exactly one route to leak through — and that route runs straight through the chassis being rebooted.

Four Things Worth Getting Right

The reset-reason output looked completely normal. That's exactly the trap.

1. reboot chassis N Is an Escape Hatch, Not a Routine Reload

SYMPTOMAn engineer runs reboot chassis 1 on a live clustered S7706 expecting the same behavior as a full device restart — instead, every downstream AP drops.

CAUSEThe source documentation confirms reboot chassis 1 is an escape-hatch command intended only for situations where physically power-cycling the chassis on site isn't possible, and it carries its own usage restrictions — its staged, one-board-at-a-time restart sequence is not equivalent to a real power-cycle, which drops all traffic simultaneously.

FIXPrefer an actual power-cycle, or a full-device reboot rather than a single-chassis reboot, inside a proper change window whenever site access allows it. Treat reboot chassis N as a last resort with its own distinct risk profile, not a routine substitute.

2. A Clean reset-reason Entry Doesn't Mean the Failover Was Clean Too

SYMPTOMdisplay reset-reason shows exactly one tidy event — “Reset for reboot chassis command” — yet the AC pair bounced Backup → Master → Backup → Master during the same window.

CAUSEBecause the primary S7706's interface boards restart sequentially rather than together, VRRP advertisements on the AC-master—S7706-master—S7706-standby—AC-standby path keep leaking through intermittently during the reboot instead of stopping the instant it starts — and reset-reason has no visibility into VRRP state at all.

FIXDon't read a clean reset-reason entry as proof the whole failover was clean — cross-check display ap offline-record all and the standby AC's own VRRP transition history for the same time window before closing out the incident.

3. The Batch Delete Comes From the Extra Demotion, Not From the AP or the Radio

SYMPTOMdisplay ap offline-record all shows “Batch delete” as the reason for the mass-offline event, not a heartbeat timeout or a radio fault.

CAUSEThe AC pair's VRRP state doesn't just fail over once — it demotes back to Backup almost immediately after promoting to Master, on the leaking hellos described above — and that specific demote-right-after-promote transition is what triggers a batch delete of every AP session the standby AC had begun to adopt.

FIXWhen troubleshooting an AP mass-offline event on a redundant AC pair, pull the full VRRP transition history, not just the final state — a single clean failover and a flapping one look identical if you only check where the state machine ended up.

4. No Cross-Link Between the Standby Pair Means One Chassis's Reboot Can Reach the Other AC

SYMPTOMLab reproduction confirms the failure on the documented topology, and the same reboot command stops reproducing it once one additional link is added on each side.

CAUSEThe as-deployed topology has no redundant path between AC-master/S7706-standby and AC-standby/S7706-master — VRRP hellos have exactly one route to leak through during a reboot, and that route runs straight through the chassis being restarted.

FIXAdd one link each between AC-master—S7706-standby and AC-standby—S7706-master, with both ACs and both S7706 chassis connected via Eth-Trunk aggregation on each side. Lab-verified: the same reboot chassis command no longer drops any AP once these links are in place.

Change Discipline Going Forward

Four changes, in order of how directly each one addresses what actually happened here.

  1. Add the cross-links between AC-master/S7706-standby and AC-standby/S7706-master described above — this closes the single leak path the whole incident depended on.
  2. Treat reboot chassis N on a live production cluster as an incident-level decision, not a quick fix — reserve it for cases where physically power-cycling the chassis genuinely isn't possible, exactly as the source documentation intends it.
  3. After any AC failover — planned or not — check display ap offline-record all for “Batch delete” entries. That reason code is the fingerprint of exactly this demote-then-repromote mechanism, and it's worth checking even when the failover appeared to go smoothly.
  4. Log VRRP state transitions continuously on both ACs, not just current state — a post-incident timeline that only has interface logs to work with will miss a bouncing failover entirely, the same way this one almost did.

Related solution designs

Five Questions That Come Up Constantly

Pulled straight from the field — the ones worth having an answer ready for.

The standby chassis's log shows interfaces going Down — so why does the case say its own ports never actually went Down?

Those Down events on the standby's log are about the peer's (the master's) interface state, propagated to the standby over the inter-chassis link — not the standby's own physical port state. The source case is explicit: the standby chassis's own interface physical state never changed throughout the incident.

Is reboot chassis 1 the same thing as pulling power on that chassis?

No. It's documented as an escape command specifically for when you can't power-cycle the chassis on site, and it behaves differently — restarting interface boards one at a time rather than dropping all traffic at once. The source recommendation is explicit that this exact issue would not occur under a genuine power-cycle reboot.

We only have a single AC and a single core switch — does this failure mode still apply to us?

The batch-delete trigger described here is specific to an AC pair's VRRP state bouncing Backup → Master → Backup → Master during a peer's staged reboot. A single-AC design doesn't have that failover path at all — but it loses redundancy entirely instead, which is a different risk, not a safer one.

Our display reset-reason only shows a single reset event — doesn't that rule out a flapping failover?

No. reset-reason only records chassis and board resets, not VRRP state transitions on the AC pair. A single clean reset event on the switch side is fully consistent with a bouncing failover happening on the AC side — that transition simply won't show up in this command's output at all.

Does adding the cross-links fix this permanently, or just make it less likely?

Lab reproduction after adding the links did not reproduce the AP drop under the same reboot command, because the cross-links remove the single path VRRP hellos had to leak through. The underlying staged, one-board-at-a-time restart behavior of reboot chassis N is unchanged — the fix removes the leak path, not the reboot's internal sequencing.

Honest Limits of This Note

Honest Limits of This Note

This post-mortem is built from a single documented S7706 V200R024C00SPC500 case in Huawei's own S-series campus switch maintenance manual. Different chassis models, software versions, or WLAN AC vendors may use different failover-detection timers and batch-delete triggers — treat the demote-then-repromote mechanism described here as a pattern worth checking for, not a guaranteed match for every mass-AP-offline event.

Just had every AP drop during a switch maintenance action?

Send us your display reset-reason and display ap offline-record all output, plus the AC pair's topology — we'll help you work out whether it's the same mechanism.

WhatsApp an engineer →

Related Reading